Study design

A single group quasi-experimental, post-intervention study aimed at evaluating the use of ChatGPT as an AI-simulated patient in teaching smoking cessation counseling. It was designed as an intervention where students practiced counseling scenarios with ChatGPT outside the classroom and completed a self-administered questionnaire afterward.

Participants and setting

The study included 145 fourth-year students enrolled in the smoking cessation counseling course during the first semester of 2024 academic year (July 2024) at the Faculty of Pharmacy, Chiang Mai University. These participants were students who were required to complete the course as part of their curriculum.

Development of AI-simulated patient

The development of the AI-simulated patient was guided by both the literature and the expertise of the research team, which included pharmacy educators with backgrounds in clinical communication and smoking cessation. The team created a standardized prompt for ChatGPT, instructing it to act as a smoker who wanted to quit smoking. The scenario followed the 5As counseling framework (Ask, Advise, Assess, Assist, Arrange), with specific patient details such as smoking habits, motivation level, and health concerns. It aimed to provide a scalable, interactive, and practical platform for enhancing pharmacy students’ counseling skills by simulating realistic patient interactions and offering structured, evidence-based learning opportunities. The development process of this intervention included three steps: designing five case scenarios, integrating the scenarios into ChatGPT, and validating the tool.

Designing five case scenarios involved developing meaningful and practical training situations for smoking cessation counseling, the scenarios were divided into five basic themes. The scenarios were developed from two main sources: 1) real cases of individuals who had sought smoking cessation services at community pharmacies, and 2) constructed cases designed specifically to suit the training context for pharmacy students. Each scenario was carefully designed to reflect common challenges that pharmacy students, who had not previously provided smoking cessation counseling, might face in real-life counseling situations.

The cases aimed to cover a wide range of patient motivations, demographics, and health backgrounds, ensuring that students would be able to practice counseling in varied contexts:

Once the case scenarios were developed, they were integrated into ChatGPT to simulate real-time counseling sessions. The final step involved content validation and pilot testing. Pharmacy practice lecturers reviewed the integrated ChatGPT case scenarios for accuracy, realism, and alignment with learning objectives, particularly the 5As framework. Based on their feedback, revisions were made to enhance educational value. A pilot test with five students was also conducted to assess functionality and user experience.

During the activity, students were required to engage with five AI-simulated smoking cessation cases. Using the 5As framework, they conducted verbal counseling with ChatGPT, which acted as a smoker. After each session, students prompted ChatGPT to provide written feedback on their performance. The prompts requested detailed feedback on strengths and areas for improvement, structured in a rubric scoring format. Five case scenarios and prompts are provided in Figures 1 and 2.

Figure 1

AI and smoking cessation prompt with case scenarios, a single quasi-experimental group at Chiang Mai University, 2024 (N=145)

Figure 2

Description of prompts in the role-play activity*, a single quasi-experimental group at Chiang Mai University, 2024 (N=145)

Intervention: AI-simulated patient

The smoking cessation topic was integrated into a structured learning sequence within a pharmacy course, consisting of a 2-hour lecture, a three-week counseling practice using ChatGPT, and a concluding 2-hour classroom session for discussion and skill application. Prior to interacting with ChatGPT, fourth-year pharmacy students attended a lecture on the pharmacist’s role in smoking cessation, delivered as part of the course on Pharmacotherapy for Pharmacy Students. The lecture covered both pharmacological and non-pharmacological approaches, including the 5As framework.

The intervention involved the use of ChatGPT as an AI-simulated patient, incorporating five smoking cessation case scenarios. Each student was randomly assigned one of the five prepared cases and practiced counseling with ChatGPT over a three-week period. They were free to conduct as many practice sessions as they wished, with no restrictions on frequency or duration. Students were free to use either the free ChatGPT 3.5 interface or the optional paid ChatGPT Plus subscription if they already had access. After completing their session, students submitted a video recording of their interaction with ChatGPT, along with a self-reflection analyzing their performance, challenges faced, and the educational value of the AI tool.

Following the three-week practice period, students participated in a 2-hour classroom session to apply their knowledge and discuss their experiences, including the benefits and limitations of AI-based counseling. While student videos and self-reflections were essential components of the learning process, they are not included in this manuscript.

Outcome measure

Four key outcomes were assessed using a self-administered questionnaire. This questionnaire consisted of both closed- and open-ended questions. It covered four key outcome areas: satisfaction, perceived learning impact, benefits, and difficulties encountered. In addition, students were asked to rate the naturalness of AI’s responses on a scale from 1 (not natural at all) to 10 (very natural and close to a real human interaction) as a supplementary item to explore their perception of how human-like the interaction felt.

First, satisfaction was measured using a Likert-scale question: ‘How satisfied were you with the AI-simulated patient activity?’ (1=least satisfied, 5=most satisfied). Second, impact on learning was assessed with the question: ‘How did the AI-simulated patient affect your learning?’ with response options (one answer): ‘Improved understanding’, ‘No impact’, ‘Made learning more difficult’, and ‘Other (please specify)’. Third, perceived benefits were evaluated using the question: ‘What benefits did you gain from using AI-simulated patient?’. Students could select more than one option, such as assessment of learning outcomes, adaptation in the learning process, immediate access to counseling and guidance, and other. Fourth, difficulties encountered were explored through the question: ‘What challenges or limitations did you face while using AI-simulated patient?’ with multiple response options including technical issues, complexity of use, lack of understanding of AI, and lack of instructor support, and other (please specify).

The questionnaire was developed based on the study’s objectives. The internal consistency of the questionnaire was not assessed in this study. Instead, content validity was evaluated through expert review. Specifically, three experts in education and smoking cessation counseling examined the questionnaire for relevance, clarity, and appropriateness of the items. Based on their feedback, necessary revisions were made to improve the quality and validity of the instrument. A pilot test was conducted with five students to ensure that the questions were clear and appropriate to the study’s objectives.

Data collection

At the end of the 2-hour classroom session on discussion and skill application, all students completed the post-AI interaction questionnaire via a secure online form. No identifiable personal information was collected.

Data analysis

Quantitative data from the questionnaire were analyzed using descriptive statistics, including frequencies and percentages, to summarize satisfaction, perceived learning impact, benefits, and difficulties encountered. Data analyses were performed using STATA Software, version 14.0. Responses from the open-ended fields were reviewed, categorized according to common content, and summarized descriptively to complement the quantitative findings.

Ethical considerations

The study protocol was approved by the Research Ethics Committee of Chiang Mai University, based on the Declaration of Helsinki, ICH GCP. Participants provided informed consent and participated voluntarily. The study carefully avoided collecting any personally identifiable information. Participants also had the option to withdraw from the study at any time.

Satisfaction, perceived learning impact, benefits, and difficulties encountered

Overall satisfaction with the AI-simulated patient was high, with two-thirds of participants reporting that they were most or very satisfied. This suggests that the tool was generally perceived as valuable for practicing smoking cessation counseling. Nonetheless, around one-third of students reported only moderate or lower satisfaction, reflecting mixed experiences. As noted in open-ended responses, factors such as technical instability, limited realism compared to human standardized patients, and reduced motivation for independent learning may explain lower satisfaction levels.

Regarding the perceived learning impact, the majority of students reported that AI activity improved their understanding, highlighting the educational value of simulated practice with AI. This suggests that AI provides a valuable practice environment, allowing students to improve their counseling skills in a safe, non-judgmental space. In line with other studies in healthcare education^17-19, these findings suggest that students had a favorable perception of AI enhanced learning. In addition, students provided open-ended responses that highlighted further perspectives; some students mentioned the value of practice and skill development that practicing with AI helped them better understand how to communicate with patients, use appropriate counseling techniques, and apply the 5As framework in real-time. Others mentioned the value of feedback and information, with detailed feedback perceived as useful and the AI system providing access to diverse information. Another theme concerned increased engagement and interest, with the AI tool described as making learning more stimulating. At the same time, some students raised technical and practical concerns, such as the challenge of preparing counseling sessions, occasional instability of the AI system, and the belief that human standardized patients remain more effective for skill development. Finally, some participants felt that the AI tool had both advantages and disadvantages, which varied with the learning situation. They appreciated the opportunities for frequent practice and diverse information, but also pointed out challenges such as technical instability and the absence of genuine human interaction.

Moreover, students reported several benefits of AI-simulated patient. The most frequently selected benefit was the ability to assess their own learning. This referred to students’ capacity to measure how well they met the specific learning outcomes, such as applying the 5As framework, demonstrating effective communication, and practicing empathy, which suggests that the AI activity supported formative self-assessment. This is considered one unique feature of the AI-simulated patient activity in this study, in which students were instructed to prompt the AI to evaluate their performance. Using structured messages, students were able to request customized feedback directly from ChatGPT, for example: ‘Could you please evaluate my performance based on the 5As approach (Ask, Advise, Assess, Assist, Arrange), empathy and rapport, communication skills, professionalism, and patient engagement? I would appreciate detailed feedback on what I did well and areas where I can improve. Please provide the feedback in a rubric score format’. This step promoted formative self-assessment, encouraging students to reflect on their performance and identify areas for improvement immediately after the counseling session. The use of a rubric-based approach helped students evaluate their performance against clear and structured criteria, which is often difficult to do without guidance. Many students reported that this feedback helped them recognize specific strengths (such as clarity or empathy) as well as gaps (such as incomplete use of the 5As model or limited patient engagement). This process aligns well with principles of self-regulated learning, where learners monitor their own progress and adjust strategies accordingly²⁰. It also supports Sustainable Development Goal 4 (SDG 4): Quality Education by encouraging active, learner-centered participation and the development of critical reflection skills^21,22. The ability to receive personalized feedback without delay makes AI a valuable tool for formative evaluation, especially in large classes where individual instructor feedback may not always be possible.

The second most reported benefit, adaptation in the learning process, reflects the flexibility that AI tools provide. Students were able to repeat sessions, experiment with different counseling strategies, and adjust their approaches based on real-time AI responses. This strongly aligns with the educational theory of ‘learning by doing’²³, which emphasizes that students learn best through active participation and hands-on experience.

Besides the mentioned benefits, immediate access to counseling and guidance demonstrates the role of AI in reducing dependence on instructor availability and supporting self-paced learning. This aligns with the principles of autonomy and just-in-time learning, which are increasingly emphasized in digital health education. The ability to practice with a variety of simulated clinical scenarios ‘anytime and anywhere’ provides more flexibility and opportunities for skill development²⁴. ChatGPT can provide scenario simulations leading to the learning opportunities for healthcare students²⁵. Moreover, this simulation can provide standardized patient interactions, ensuring all students receive similar foundational experiences. This consistency is often difficult to achieve with human standardized patients due to variability in performance and availability²⁶. Lastly, active participation in the learning process remains important benefit. The AI platform transformed passive content consumption into an experiential activity that required students to take initiative, solve problems, and respond dynamically.

Despite the overall positive response, students reported several barriers that affected their experience with AI-simulated patient. The most common issue was technical problems, such as slow system response, internet connectivity issues, and freezing or disconnection during sessions. These disruptions impacted the flow of the conversation and sometimes caused students to lose progress, leading to frustration and reduced engagement. The second most reported barrier was a lack of understanding about how AI functions. Some students were unsure how to effectively communicate with ChatGPT, structure their questions, or interpret its responses. These barriers reduced their confidence and made it difficult to use the tool to its full potential. Another barrier was that some students found the platform too complex or unfamiliar, especially those who had never used AI-based tools before. They felt unsure about how to begin, what to say, or how to guide the counseling session effectively. In addition, a few students mentioned a lack of instructor support during the activity. While the task was designed for self-directed learning, some students felt they needed more guidance, such as clear examples, demonstration, or technical assistance.

Other barriers were mentioned less frequently in open-ended responses. These included the AI’s limited understanding and ability to speak the Thai language, which caused minor confusion for some students who chose to interact in Thai. Several students reported that ChatGPT often misunderstood Thai input, responded with delays, or provided answers that did not match the context of the conversation. These issues disrupted the natural flow of the counseling interaction and, in some cases, caused frustration or confusion. On the other hand, students who spoke English reported smoother interactions and faster responses. ChatGPT was able to understand their input more accurately and respond in a way that felt more natural. Unfortunately, only a small number of students were confident enough in their English ability to conduct the session entirely in English. Most students lacked the fluency required for effective counseling conversations in a second language, which limited their use of the AI to its full potential.

While the findings highlight the educational potential of ChatGPT, the AI-simulated patient is not immune to errors. Because no accuracy checks were implemented, ChatGPT may have generated inaccurate or misleading counseling information. This risk was increased when students interacted in Thai, a language for which large language models have less training data and weaker contextual understanding, potentially leading to misinterpretation or inappropriate responses. Such factors could have affected both the quality of the simulated counseling sessions and the students’ perceived learning.

Students using the free version of ChatGPT reported several limitations, including restricted access time, disconnections, and content errors during extended conversations, which disrupted learning. Many felt the AI interaction was unnatural and lacked human-like qualities, reducing engagement. The responses were often repetitive, too formal, and lacked emotional tone, which made the interaction feel less natural and realistic. The absence of non-verbal communication cues, such as facial expressions, tone of voice, and gestures, also made it difficult to practice empathy, active listening, and rapport-building. While useful for practicing structured content, the AI fell short in simulating real human connection. These findings highlight the need for future improvements in AI design, including more dynamic, emotion-sensitive, and personalized communication, to enhance realism and support the development of advanced interpersonal skills.

Strengths and limitations

This study has several limitations that should be acknowledged. First, the participants were pharmacy students from a single institution, which may limit the generalizability of the findings. Second, the study relied mainly on self-reported data from the questionnaire, without external assessment of counseling performance by instructors or experts. Third, no pre–post evaluation was conducted, so the actual learning gains could not be measured objectively. Fourth, each outcome was assessed with a single-item Likert scale, which may not fully capture the complexity of students’ learning. Although open-ended responses were included to provide additional context, future research should consider using multi-item validated measures for a more comprehensive evaluation. Lastly, this study was conducted as a descriptive investigation without inferential statistical analyses, which restricts the ability to draw conclusions beyond the study sample. Moreover, it did not include a comparison group (e.g. traditional peer role-play), making it difficult to determine the relative usefulness of AI-simulated patient. Nevertheless, the insights gained from this work may serve as a useful foundation for further testing, adaptation, or application in other populations.

This study is one of the few in pharmacy education to explore the use of AI in smoking cessation training^14,27. A key strength is the structured use of ChatGPT as a simulated patient, combining real-time interaction, self-directed learning, and AI-generated feedback through a rubric. The study also highlights language-related challenges in a non-English-speaking context – an area rarely discussed in current literature.

Implications for pharmacy education and practice

This study suggests that AI-simulated patient can serve as a valuable supplement to traditional pharmacy education, particularly in teaching patient communication and smoking cessation skills. It offers scalable, individualized, and flexible training opportunities – making it ideal for large classes or programs with limited access to real patients. Educators may consider incorporating AI tools like ChatGPT into classroom activities, either as pre-lab preparation or follow-up reflection exercises, to reinforce practical counseling frameworks such as the 5As model. In addition, structured prompts and AI-generated feedback can support self-assessment and reflective learning, helping students monitor their progress and identify areas for improvement. However, successful implementation will require orientation sessions, language considerations, and ongoing instructor support to ensure accessibility and effectiveness.