National Health Service England’s 2023 Long Term Workforce Plan outlines the need to develop an increasing number of skilled health care professionals to address the projected workforce shortfall from imminent “demographic pressures” and “changing burden of disease” [1]. Subsequently, education and training needs are predicted to increase between 50% and 65% by 2030 and 2031 across all health care professions [1], which necessitates a “high-quality educator workforce” [2]. More recently, the 10 Year Health Plan for England highlighted plans to “modernize postgraduate medical education,” as well as work with educational institutes to “overhaul education and training curricula” for health care professions, including training in artificial intelligence (AI) use [3]. This increasing demand for modern educators means that health professions education (HPE) programs must be confident they are appropriately assessing whether learners achieve required professional education standards in the light of a constantly changing world.

AI plays a crucial role in this process, both by driving change and providing advanced tools and technologies that support personalized learning, efficient assessment, and continuous improvement in educational practices. More specifically, “generative AI” (GenAI) is now widely used and rapidly developing, capable of creating new content, including text, images, and other media. Within GenAI, natural language processing allows computers to understand human language; analyze, process, and interpret text; extract meaning; and perform tasks previously thought to require real intelligence [4]. Large language models (LLMs) are a further type of GenAI that perform natural language processing tasks by using “artificial intelligence algorithms to generate language that resembles that produced by humans” [4].

GenAI has significant ramifications within education, including HPE, which have become noticeable since the increased use following the release of the updated LLM by OpenAI in November 2022, ChatGPT-3.5. There has been subsequent development of LLMs internationally. Concerns arise regarding inequity for learners in barriers accessing GenAI, but these will likely diminish with their ongoing development, driving down costs and increasing accessibility. Crucially, the use of GenAI has triggered concerns among many HPE educators around how AI will influence teaching and learning and, especially, assessments. Current forms of assessment in HPE, especially at master’s level, are traditionally essay-based, incorporating reports, literature reviews, reflective pieces, and theses. By contrast, clinical qualifications tend to use “objective” examinations and work-based assessments. Hence, within HPE, there are increased concerns regarding potential academic misconduct in essay-type assessments if students use AI [5] and the implications for the quality and competence of these future educators when they enter the workforce.

Concerns about cheating in assessments are not new. Essay-based assessments have always been open to “contract cheating,” when a student submits work written by someone else, but GenAI has made this more accessible [6] and has blurred the lines between cheating, plagiarism, and a “helping hand.” Among academics, vacillation is rife over whether GenAI is a “game-changer” and represents an end to the essay-based assessment or adds little to the multiple ways students can already cheat [7]. Besides potential academic misconduct, there are other considerations related to the use of AI within HPE assessment: privacy risks involving sensitive data [8], bias arising from the data on which the AI systems are trained [9], and “hallucinations” whereby AI creates false information and fabricates references [10].

“AI-output detectors” detect the use of AI in academic work, with varying results against humans in detecting AI-generated questions [11], but they have successfully discriminated between most original and AI-generated abstracts [12]. Indeed, there are platforms to help bypass AI-output detectors, so in this escalating arms race, the speed of AI’s sophisticated evolution makes it almost impossible for output detectors to stay ahead. This increases concerns about academic misconduct or misapplication in HPE assessments and questions whether educators or software will be able to detect its use.

AI literacy is “the ability to understand, use, monitor, and critically reflect on AI applications without necessarily being able to develop AI models themselves” [13] and is necessary for both learners and educators to effectively use AI within assessments. Although self-ratings for AI-knowledge in health professions students are low [14], learner perceptions toward AI are positive [15], wanting AI to be incorporated into medical school curricula [14,16,17]. If AI is formally incorporated into the curriculum, this will affect both how and what we assess, and educators will need to feel confident in AI literacy, as there are a multitude of ways to incorporate AI into assessment [18]. Tlili et al [19] identified the need to “upskill” educators on AI’s practicalities and how to design and teach it in curricula. Educator training would improve AI literacy and address faculty concerns related to using AI, such as misinformation or academic misconduct [20,21]. Current assessments within HPE are unlikely to be suitable to assess AI literacy, yet not incorporating AI in learning and assessment will likely impede future educators, learners, and patients, given AI’s growing presence.

Communities of practice (CoP) are “groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly” [22], and here it is used as a framework to consider how HPE forms CoP to co-construct knowledge and stay “ahead of the curve” in the face of the complex challenges, changes, and opportunities AI presents.

This research focuses on the views of opportunities and challenges within HPE assessment of GenAI (AI here refers to GenAI unless specified otherwise), specifically LLMs. The approach analyzes participants’ perspectives on the concept of LLMs in the context of HPE assessments. Discourse within the educator community is currently evolving and full of uncertainty, particularly in exploring how AI may specifically influence written HPE assessments, so we ask the question: what are educators’ and learners’ perspectives on the opportunities and challenges presented by GenAI in postgraduate HPE assessments?

This research focused on exploring perspectives from a constructivist paradigm, using qualitative methodology. It aimed to explore educators’ and learners’ perceptions of how AI may influence HPE assessments, rather than testing knowledge of AI. Participants’ experiences are explored with the idea of effecting potential change—whether in an approach to future assessment or attitudes to AI within the HPE community. Methods are reported in line with the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist (Checklist 1) [23].

Educators currently teaching or assessing a postgraduate HPE course and current learners enrolled in a postgraduate HPE course were invited via purposive sampling to participate in semistructured interviews exploring educators’ and learners’ perspectives. Educators and students from different HPE courses across the United Kingdom were invited to avoid response bias specific to perspectives from a single institution. As GenAI was a relatively new concept at the time of interviews (January to March 2024), the inclusion criteria specified that only educators and postgraduate learners currently involved in a course were invited to participate. Exclusion criteria were any undergraduate students or individuals either teaching or being taught by CP. As there was no 1 overall group from which to recruit participants, purposive sampling ensured the invitation reached those who met the criteria. Participants were recruited through an email invitation sent by CP, and participants were invited to forward the invitation to colleagues to encourage snowball sampling. CP and administrators of appropriate HPE social media groups distributed the invitations through their respective health profession education social media channels (Microsoft Teams and WhatsApp) to widen the invitation’s reach. It was unclear how many were approached via this method, as there are multiple educators and learners on such channels.

CP held roles as clinician, researcher, and educator. She was both an educator and learner on different postgraduate HPE courses. The research was for her master’s thesis (for MSc in medical education at University College London/Royal College of Physicians), with prior experience in qualitative research. DH was her supervisor and educationalist.

Specific ethical implications were considered, as recruitment methods included inviting some participants who were colleagues of the interviewer (CP). The participant information leaflet clearly outlined the intent of the research, and alongside the email invitation, it was made clear that the study was part of a master’s thesis and participation was voluntary. The invitation was also disseminated by administrators via wider social media channels to increase the uptake of participants, especially those not known to the interviewer.

Reflexively, CP was aware that their dual role as a current learner and as an educator with an interest in assessment and AI could potentially be intimidating in interviews if participants did not feel knowledgeable about AI, thereby creating a barrier to honesty. However, the dual role could also have been advantageous if interviewees felt that CP had an understanding of their role and, therefore, may have felt more comfortable.

DH was the supervisor for this research and an educationalist with assessment responsibilities and an interest in how AI is impacting assessment. DH was very aware of the potential for a conflict of interest—particularly if current assessment methods were perceived as no longer fit for purpose. DH had no knowledge or contact with the participants, and none of them were involved in the assessment process of any programs involving DH.

Semistructured interviews were conducted by CP as the sole interviewer on Microsoft Teams and auto-transcribed. The transcripts were then pseudonymized and edited by the interviewer to ensure they were transcribed verbatim. Interviews were audio-video recorded with the camera off to encourage participants to speak freely and to increase confidentiality, and recordings were deleted at the end of the withdrawal period. Only the interviewer (CP) and the participant were present during the interview, and there were no repeat interviews. CP made research notes during the interviews to refer back to during analysis. The length of interviews can be found in the Results section. The interview schedule (Multimedia Appendix 1) was developed by CP to answer the research questions while allowing flexibility, with initial questions being reviewed and revised based on feedback from DH. The interview schedule started with more open-ended questions on the participants’ current roles and HPE assessments, before exploring their knowledge, perceptions, and use of AI both personally and within their communities. Subsequent questions were more structured to address the research questions and examined opportunities and challenges of AI within HPE assessments for both educators and learners. The interview schedule was not piloted due to concerns of low recruitment but was revised based on feedback from DH, and the semistructured nature of the interview allowed for sufficient flexibility. There was no member checking of transcripts, both for feasibility and to avoid participants reconsidering their original views.

Coding was undertaken by 1 data coder (CP) using NVivo 14 (Lumivero), as part of an independent master’s thesis. Data were analyzed using Braun and Clark’s thematic analysis [24], following the 6-phase process and using a predominantly inductive approach to reflect the relative novelty of the topic at the time.

In phase 1, CP immersed herself in the data by editing and rereading the transcripts. In phase 2, in keeping with Braun and Clarke’s [24] thematic analysis, “codes” were given to lengths of text relevant to the research question. Data that captured similar meanings were assigned to the same code. In phase 3, codes were then constructed into subthemes and then grouped under themes. There was no participant checking of codes. Associated codes from themes and subthemes helped identify illustrative quotes outlined in the Results section. In phase 4, themes were reviewed, and as CP was the only coder, there was no interresearcher triangulation of themes and, therefore, no disagreements. The limitations of having a single coder in triangulation and the potential for bias were acknowledged. However, data analysis started after a couple of interviews had been conducted and transcribed, and emerging themes and subthemes were continually revised and reviewed as further interviews were conducted and subsequently analyzed. Research notes made during interviews allowed for further refining of themes.

In phase 5, the datasets were reviewed, analyzed, and visualized on a thematic map (Figure 1), enabling further refinement of coherent themes and subthemes without repetition. Phase 6 involved writing up the analysis. Reflexively, CPs’ lived experience straddling educator and learner roles while undertaking HPE assessments in an evolving AI era may have helped co-construct knowledge and understanding and enabled development and depth to the codes and themes.

Data were stored securely on a university password-protected platform. Ethics approval was granted by the University College London Research Ethics Committee (ethics 23511/011). Participation was voluntary and without compensation, and informed consent was obtained from all participants. Consent for publication was obtained from all participants; no identifiable data were used.

Eight semistructured interviews were conducted via Microsoft Teams between January and March 2024 (4 educators [E] and 4 learners [L]), lasting 30 to 60 minutes (Table 1). Six doctors and 2 nurses across 6 different higher institutes within the United Kingdom were interviewed. Data saturation was reached after 8 interviews, reaching informational redundancy. The participants had a range of prior AI knowledge; however, the distinction between learners and educators was blurred, as all educators had previously been learners in HPE, and all learners were currently educators in some capacity, which may have led to more analogous perspectives. The dual role of interviewee as both educators and learners may, therefore, have led to theme convergence.

The iterative process of data analysis reduced 15 subthemes to 10 and 6 themes to 4. The four themes are (1) AI as an opportunity to rethink HPE assessments; (2) AI is here, students are at a disservice if we do not embrace it; (3) AI is a gray area; and (4) AI is fallible.

As seen in the thematic map (Figure 1), some themes and subthemes have strong associations (bidirectional arrows) and affiliations (straight lines), demonstrating the interconnection between concepts.

The following are the actionable strategies in HPE to guide assessment redesign:

The following are the future research needs:

There was potential educator and learner response bias among those who felt more confident discussing AI. The research was specific to postgraduate HPE, so the opinions of undergraduates and students from different courses may differ. Interviews were conducted by a single interviewer (CP) as part of her master's thesis, and despite efforts at reflexivity, there may have been interviewer bias. The study is based on 8 interviews from institutions across the United Kingdom, predominantly doctors, so generalizability to different countries and courses may be limited. Furthermore, it is difficult to extrapolate group comparison due to the role duality of learner-educators, as well as the small sample size of only 4 participants per group. While it was not possible to obtain the assessment format from all HPE courses, as participants were from different HPE courses, they were able to provide an overview of various formats.

Key findings indicate that educator and learner perspectives agree on the opportunities and challenges presented by GenAI: AI is here, students are at a disservice if we do not embrace it, AI as an opportunity to rethink HPE assessments, AI is a gray area, and AI is fallible. Participants consider AI to offer the opportunity for an overdue reevaluation of predominantly written-based HPE assessments to better reflect real clinical and academic life and develop educators who are going to enter the modern workforce in which AI is ever present. Developing an assessor CoP will allow for shared AI practice and knowledge cocreation, building both educators’ and learners’ confidence in this new field.