Background: Medical students are at higher risk of burnout than the general population. Interventions that facilitate adaptive coping behaviors (eg, Psychological Flexibility) in the context of inherent stressors associated with medical training could mitigate burnout risk and improve well-being. Delivering these interventions using smartphone apps offers advantages such as accessibility, scalability, mitigation of time and stigma barriers, and facilitation of individual tailoring (individualization). There is a need for feasibility trials with medical students in this emerging field. Formal evaluations of user experiences of app-based psychological skill training are required to identify barriers to and facilitators of engagement and optimize intervention development before implementation in efficacy trials and real-world settings.
Objective: This study aimed to assess the feasibility of delivering an individualized Psychological Flexibility skill training intervention (Acceptance and Commitment Training [ACTraining]) to medical students using an app-based delivery format. We further aimed to explore how formal evaluation of user experiences might inform and guide the development of this app before implementation in an efficacy trial and future research involving app-delivered psychological skill training for medical students.
Methods: This single-arm study was an early-phase feasibility trial of a stand-alone ACTraining app conducted with a sample of Australian medical students (n=11). We collected app usability and user experience data across a broad range of domains (eg, perceived helpfulness and relevance, learning experiences, and self-efficacy) using self-report questionnaires (quantitative and qualitative) and behavioral engagement outcomes.
Results: Behavioral engagement data demonstrated that the app delivered the assessment procedures and individualized ACTraining intervention to medical students as intended. The subjective feedback provided by students who actively engaged with the app was generally positive across several indicators, including usability, perceived relevance and helpfulness, accessibility, maintenance of privacy, and opportunity for self-reflection. Disengagement from the app was an identified challenge throughout the trial. Participant feedback identified several factors that may have affected engagement, such as time, expectations regarding app interface functioning, and individual differences in confidence and self-efficacy when implementing skills.
Conclusions: This study reports user experience data that have been largely absent from the literature on digital psychological interventions for medical students. Our findings demonstrate the preliminary feasibility of an app-delivered ACTraining intervention for medical student well-being and burnout and support the value of future assessment of the efficacy of this approach with larger samples. We consider subjective feedback from medical students in relation to observed engagement and propose how this information might be used to inform the development of this app and future research in this nascent field.
Medical Student Stress and Psychological Health
The stress of medical training can have adverse effects on students’ psychological health [- ]. Worldwide, medical students are at an elevated risk of psychological ill health compared with the general population [ ], demonstrating high prevalence of burnout (44.2% [ ]), depression (27.2% [ ]), and suicidal ideation (11.1% [ ]). Burnout risk increases as training progresses [ ] and can persist throughout a physician’s professional life [ ]. The detrimental consequences of burnout and psychological ill health among medical students and physicians are well documented and include poor physical health [ ], diminished life satisfaction [ ], and disruption to health care service delivery (eg, impaired performance [ ] and reduced empathy [ ] and work engagement [ ]).
Adaptive Psychological Skill Training
Medical educators report increasing concern regarding the psychological health of students  and recognize the need for early intervention strategies that can prevent burnout rather than treat it after it has already emerged [ - ]. In addition to organizational approaches that mitigate modifiable external stressors (eg, improving learning environments and addressing student mistreatment [ ]), there is support for initiatives that build individual psychological and behavioral resources for coping with unmodifiable stressors encountered during medical education and practice [ , , , ] (eg, academic pressures and exposure to illness and death [ , ]). This intervention strategy has the potential to buffer medical students against burnout and psychological ill health by facilitating adaptive responses to stressful experiences [ , , ]. Beyond simply preventing psychological ill health, adaptive psychological skill training can improve well-being [ , ]. Well-being is a state of personal thriving and vitality that is an essential component of complete psychological health [ , ] and a key outcome of interest to medical educators [ ]. There is support for the acceptability of adaptive psychological skill-building interventions among medical students [ ] and physicians experiencing burnout, with the latter having rated such training as being as important as learning clinical skills [ ].
The increased interest in medical student psychological health has resulted in a growing number of studies evaluating individual skill-building interventions . A notable gap in this emerging field is the absence of studies evaluating the delivery of such interventions using stand-alone smartphone apps [ , ]. Although the preliminary feasibility of app-delivered interventions (eg, mindfulness) has been demonstrated among small samples of medical residents [ , ], there are no studies evaluating feasibility among medical students. Medical students use smartphones frequently and are accustomed to engaging with technology [ , , ] and, thus, app-delivered interventions have the potential to be accessible [ , ] and scalable [ ]. Apps could facilitate medical students’ access to psychological skill training anytime and anywhere [ ], which is particularly important when the availability of psychological health care personnel is limited or face-to-face delivery is not practical. App-delivered interventions may also offset known barriers to medical student engagement in psychological interventions by maintaining anonymity and privacy for students concerned about mental health stigma [ , ] and delivering brief intervention components that minimize participation-related time pressures [ ]. Finally, app-delivered interventions provide a potential solution to the problem of heterogeneity and variation in individual responses to psychological interventions [ ] by facilitating the tailoring of training to individual medical student needs (ie, individualization) [ ]. Individualized psychological apps have demonstrated greater effectiveness than nonindividualized apps with respect to well-being outcomes among heterogeneous university student samples [ ].
Despite the potential advantages of app-delivered interventions for medical student psychological health, there are several challenges that could affect the feasibility of this strategy. Maintaining individual participation in psychological health apps is a known problem generally, with real-world user engagement diminishing substantially within 2 to 4 weeks of sign-up . Engagement can be affected by a range of usability and acceptability factors, including the quality and functionality of the interface [ , ], training content relevance [ ], the context in which the intervention is experienced [ ], time burden [ ], and user experiences [ ] (eg, affective responses, perceived effectiveness, and self-efficacy [ ]).
Given the infancy of this emerging field and intervention delivery mode, feasibility assessment is an important precursor to successful implementation in larger efficacy trials . There is a need for studies that formally evaluate medical students’ subjective experiences of engaging in app-delivered psychological interventions as well as their perceptions of interface usability [ , ]. The absence of this information from the literature has been noted as a hindrance to progressing digitally delivered psychological skill interventions beyond the prototype phase to effective, fully developed products that could be successfully implemented in medical education settings [ ]. The collection of these data can provide information about how psychological skill training apps are experienced in real-world contexts among medical student end users [ , ], unexpected methodological issues and barriers [ ], and factors that may affect engagement [ , ].
This study is a small-scale feasibility assessment of an adaptive psychological skill training app for medical student burnout prevention and well-being using an evidence-informed behavioral model (Psychological Flexibility).
Intervention Model: Psychological Flexibility
There are no standardized psychological skill training interventions available for medical student burnout and psychological well-being . However, Psychological Flexibility is an adaptive behavioral skill set that has been shown to protect against adverse mental health outcomes in stressful situations [ , - ] and is emerging as a promising intervention target [ , , , ]. The model encompasses 6 modifiable behavioral flexibility (and corresponding inflexibility) processes: present-moment awareness (nonawareness), experiential acceptance (avoidance), cognitive defusion (fusion), self-as-context (self-as-content), contact with values (noncontact with values), and committed action toward values (inaction) [ ]. When exposed to stressful situations, individuals who are high in Psychological Inflexibility tend to respond in ways that are rigidly driven by their internal experiences (eg, fusion with unhelpful thoughts, rules, or self-stories, and avoidance of emotional discomfort). This can increase their risk of adverse psychological outcomes by restricting coping repertoires, disrupting engagement in personally meaningful behaviors, and exacerbating distress [ , , ]. Conversely, as psychologically flexible individuals are more able to adopt an open and nonevaluative stance toward internal experiences, they are less likely to interpret emotionally uncomfortable stressors as threatening to their well-being [ ]. By focusing less on avoiding or reducing discomfort and more on connecting with and accepting their direct experiences, psychologically flexible individuals are able to recognize a broader range of situational opportunities available to them when responding to stressful conditions [ , ]. This facilitates purposeful action toward personally held values and related goals [ ]. Early evidence supports the potential relevance of Psychological Flexibility skills to medical students, who are less likely to experience burnout when they adopt experiential acceptance [ , ] and engage in values-driven actions [ ]. Conversely, low Psychological Flexibility among medical students is associated with an elevated risk of burnout [ ], reduced life satisfaction, and greater personal distress when seeing others in harm [ ].
Acceptance and Commitment Training
Psychological Flexibility skills can be developed and strengthened using Acceptance and Commitment Training (ACTraining) . ACTraining interventions implemented in medical education settings have demonstrated beneficial impacts such as reduced burnout among distressed medical students [ ] and improved well-being and diminished psychological distress among female medical students [ ]. ACTraining is well suited to an app-based format as it teaches Psychological Flexibility skills using metaphors and experiential activities that can be delivered as brief and practical training components [ ]. By facilitating opportunities to practice during the course of medical students’ everyday lives, app-delivered ACTraining interventions could strengthen and generalize Psychological Flexibility skills [ ]. Furthermore, as ACTraining components have functional links with their corresponding Psychological Flexibility processes [ , , ], interventions can be individualized within an app by identifying which skill a student requires the most assistance with in a particular moment and aligning the delivery of targeted components with these needs [ ]. This is important given the heterogeneity of Psychological Flexibility skill profiles between individuals and the fact that the 6 processes can vary independently of one another [ , , ], suggesting that medical students are likely to require training in different skills at different times and in different situations [ ].
At the content level, ACTraining offers potential feasibility benefits with respect to its acceptability and relevance to medical students. ACTraining adopts a normalizing stance toward uncomfortable internal responses to stressors (ie, thoughts, emotions, and physical sensations) [, ], providing opportunities to learn adaptive behavioral skills in a nonstigmatizing way. Furthermore, as ACTraining encourages action driven by awareness of an individual’s unique experiences, values, and goals [ ], students can learn to apply the skills in ways that are personally relevant. However, aspects of ACTraining (eg, self-reflection and bringing attention to uncomfortable thoughts and emotions) may be challenging when experienced in a self-directed app-based context without guided support [ ]. Evaluating medical students’ subjective experiences of engaging with ACTraining content is an important aspect of feasibility assessment, particularly given the potential impact on app engagement [ ]. To our knowledge, no feasibility data of this nature have been reported previously.
Study Aims and Objectives
We developed an individualized Psychological Flexibility skill training app (using an ACTraining intervention approach) for medical student burnout and well-being. The aims of this study were to assess the feasibility of delivering an ACTraining intervention to medical students using an app-based delivery format, and explore how formal evaluation of subjective user experiences might inform and guide the development of this app before implementation in a planned efficacy trial as well as future research regarding psychological skill training apps for medical students. This study was an early implementation trial conducted with a small sample of Australian medical students who were interested in engaging in an app-based burnout prevention and well-being intervention. We collected behavioral engagement data and detailed student feedback regarding the ACTraining intervention content, usability of the app interface, and experiences of engaging with the intervention across a range of acceptability domains (eg, affective responses, perceived learning, self-efficacy, and engagement barriers and facilitators).
This study was a single-arm feasibility trial of a 2-stage individualized ACTraining app.
Participants, Recruitment, and Study Setting
We aimed to recruit approximately 10 to 15 participants, consistent with sample size recommendations for feasibility studies . As the app had not been trialed previously, we limited the number of participants to allow for a smaller-scale, detailed end-user evaluation before broader implementation in a planned larger efficacy trial [ ]. Recruitment occurred over a 3-week period in June 2021.
The sampling frame was all students enrolled in year 3 of the Joint Medical Program at the University of Newcastle and University of New England (Australia) during semester 1 of 2021 (N=170). Year 3 students were selected for pragmatic reasons (eg, timing of the study in relation to academic demands and the impact of the COVID-19 pandemic), as recommended by the University of Newcastle head of medical student well-being (GH). Students were eligible for inclusion if they had regular access to a reliable internet connection and an electronic device compatible with app use (smartphone or tablet). Although the ability to understand English was a requirement for using the app, this was not stated explicitly as all members of the sampling frame met this criterion. There were no ineligibility criteria.
All aspects of the study were implemented on the web. A digital invitation was sent from a Joint Medical Program administrative account to the students’ university email accounts, and a follow-up invitation was sent a few days before the final enrollment date. Students also received a verbal invitation at the end of a web-based class delivered by a member of the research team. Participation in the study was voluntary. Students were given a URL and QR code that allowed them to access the enrollment website (hosted on a secure web-based survey and database platform, REDCap [Research Electronic Data Capture; Vanderbilt University]) that provided information about the study and allowed students to enroll using an e-consent process. After completing a brief demographic questionnaire, consented students were allocated a unique (4-digit) participant ID and provided links to download the app via the App Store (Apple) and Play Store (Android). After downloading the app, students created an account. All further assessment, data collection, and intervention procedures were conducted via the app, which students accessed independently during the study period. Self-report outcomes were measured at 3 time points: baseline (t0), following stage 1 of the intervention (t1), and following stage 2 of the intervention (t2). Owing to the time commitment involved in the study, an Aus $30 (US $20.43) gift voucher was offered to participants who completed the intervention and self-report outcome measures.shows the participant flow diagram.
Ethics approval for this study was granted by the University of Newcastle Human Research Ethics Committee on January 21, 2021 (approval ID: H-2020-0311) and ratified by the University of New England Human Research Ethics Committee on February 11, 2021. To ensure the privacy and anonymity of participant data, 2 password-protected databases were used to store personal and demographic information (in an identifiable Participant Information Database) separately from the outcome data (stored in an anonymous Study Database). Outcome data were deidentified, with participant ID used as a linkage key. Only the lead author had access to the Participant Information Database. Members of the research team did not have access to identifying participant information. To protect privacy while using the app, students input their unique participant ID as their username when creating and accessing their accounts.
Before providing consent, students were informed that (1) data collected during the study would be anonymous and retained and stored for a minimum period of 5 years from completion of the research, in accordance with the University of Newcastle research data management policies; (2) deidentified data would be analyzed, summarized, and presented in various formats (eg, peer-reviewed journal publications); (3) nonidentifiable data may be used for future ethically approved research; and (4) if students elected to withdraw from the study, data provided up to the point of withdrawal would be retained and included in the study in a deidentified and anonymized form.
The intervention involved the delivery of individualized ACTraining via an app (“BiSi: Build it. Sustain it.”) developed by clinical psychologists (ED and BK). Existing Psychological Flexibility training activities and concepts were adapted to suit the context and target participant group. The app was created in a low-code platform (Cogniss) that facilitated the delivery of the outcome measures and intervention and the collection of study data (including participants’ responses to outcome measures and use data). Intervention content was delivered in audio format with accompanying images and some supporting written text. The ACTraining intervention was delivered in 2 stages. Further information regarding the Build it. Sustain it. app and intervention protocol has been published elsewhere .
Stage 1: Introductory Module
During stage 1, participants used the app to complete an introductory module that provided a conceptual framework from which to understand the full Psychological Flexibility skill set. The module was divided into 7 sections (<10 min each), which were presented in a fixed order and took approximately 1 hour to complete. Progress was saved at the end of each section, allowing participants to pause and resume the module if they wished to complete it over multiple sittings. Section 1 content included an introduction to burnout (definition, recognizing the signs, and normalizing and destigmatizing) and well-being (definition and importance of choosing behaviors that support career sustainability). Sections 2 to 7 provided education about each Psychological Flexibility skill set (see the screenshot in), including the role of psychologically inflexible behaviors in burnout and psychological distress and how flexibility processes may facilitate more adaptive responses and psychologically healthy outcomes. Although the focus of stage 1 was predominantly conceptual, each Psychological Flexibility process was accompanied by at least one skill practice activity (including personal value identification, metaphors, mindfulness practice, awareness practice, thought defusion, perspective-shifting techniques, and behavioral activities). Participants were given 1 week to complete stage 1.
Stage 2: Individualized Skill Training
Stage 2 was designed to provide students with on-demand access to a library of short (3-8 min each) practical Psychological Flexibility skill activities (20-25 activities per process), which were targeted to their specific training needs each time they accessed the app. Completion of stage 1 unlocked access to the stage 2 dashboard (see the screenshot in), which students could access at any time. Although there was no requirement to complete a specific number of activities, participants were sent daily reminder emails and encouraged to practice regularly to optimize skill learning. Activities focused on normalizing challenging internal experiences, strengthening present-moment awareness, recognizing the influence of internal experiences (including thoughts and emotions) on choices and actions, learning to alter unworkable responses to internal experiences, setting values-based goals, and developing more flexible and value-aligned behavioral repertoires [ , , ]. provides a summary of the aims and an example activity for each Psychological Flexibility skill set.
|Skill set||Aim||Example activity|
|Present-moment awareness||Teach students to bring purposeful awareness to their present-moment experiences and recognize how these experiences influence their choices (in everyday life and stressful situations)||Guided mindfulness activity in which students practice attending to specific aspects of their present-moment experience (eg, sounds, physical sensations, thoughts, emotions, and breathing)|
|Defusion||Teach students to distinguish between their direct experiences and their thoughts about their experiences and respond more flexibly to the latter by changing the contexts in which they experience their thoughts||Students practice responding flexibly to a thought by choosing an action that contradicts a thought they are having (eg, they are asked to think “I can’t lift my arm” and to also lift their arm).|
|Self-as-context (perspective taking)||Teach students to recognize that their thoughts and emotions do not define their identity but are transient aspects of their experience, which they can observe from a more stable and constant sense of “self”||Students identify a “self-story” that causes distress or limits their response options in certain situations and practice choosing actions that expand their behavioral repertoires outside of these stories.|
|Acceptance (willingness)||Teach students to identify unhelpful avoidance behaviors and practice open, flexible, and adaptive ways of responding to internal discomfort||Students practice observing an uncomfortable sensation (eg, an urge or an itch) with openness, curiosity, and willingness (eg, notice the components of the sensation and allow them to come and go without buying into evaluations of the sensations or attempting to alter them).|
|Values||Teach students to identify their own personally held values and notice how connection or disconnection with these values can contribute to well-being or burnout experiences, respectively||Students choose one of their own personally held values and briefly reflect on how their behaviors have aligned with that value during the day.|
|Committed action||Teach students to choose effective and purposeful behaviors that align with personally held values even when internal discomfort is present; use direct experiences to recognize the difference between well-being–oriented action and inflexible persistence that could increase burnout risk||Students select a personally held value that feels most important to them in that moment and commit to taking one small and achievable action toward that value during the day.|
Individual tailoring of Psychological Flexibility skill activity delivery was implemented using a method adapted from a similar previous study . At each app log-in, participants responded to a single-item screening question: which of the following are you having the most difficulty with today? (see the screenshot in ). Each response option corresponded to 1 of the 6 Psychological Flexibility processes ( ). After selecting a response, participants were directed to a dashboard that provided access to all skill activities available for the Psychological Flexibility process they had identified as having the most difficulty with on that occasion (presented in a list). Students were then able to practice one of the available skill activities, either by choosing from the list or by allowing the app to make a random selection for them (see the screenshot in ). Upon completion of an activity, participants had the option to complete another activity within the same Psychological Flexibility skill set. These steps were repeated until the participants elected to discontinue. Participants had access to stage 2 for 3 weeks.
Stage 2 individualization screening question options and corresponding Psychological Flexibility processes.
- Struggling with your feelings: acceptance (willingness)
- Unable to do what matters to you: committed action
- Stuck in your thoughts: defusion
- Stuck in autopilot or struggling to stay in the present moment: present-moment awareness
- Disconnected from a sense of meaning or purpose: values
- Stuck in stories about who you are or who you should be: self-as-context (perspective taking)
We integrated design elements aimed at enhancing medical students’ engagement with the app-based intervention. In addition to individualization , this included the provision of daily reminder emails [ ], offering a variety of training activities for each skill set [ , ], and providing opportunities to engage in self-reflection while engaging in the educational and skill practice components of the app [ ].
Psychological Outcome Measures
We administered self-report psychological outcome measures () at t0 only. The purpose of administering these outcome measures in this study was to evaluate the feasibility of using the app delivery method to assess psychological outcomes of interest in the planned efficacy trial and explore the potential assessment burden to participants. Psychological outcome data were also used to describe the baseline psychological characteristics of this feasibility trial sample.
|Outcome measure||Items, N||Measure characteristics||Cronbach α|
|Psychological Flexibility and Inflexibility—MPFI-SFi ||24||.89|
|Depression, anxiety, and stress—DASS-21l ||21||.91|
|App usability—SUSm ||10||.81|
|User Experience Assessment part 1||32||.93|
|User Experience Assessment part 2||8||N/An|
|Learning experiences assessment||13||.90|
aMBI-GS(S): Maslach Burnout Inventory General Survey for Students.
dAE: academic efficacy.
eMHC-SF: Mental Health Continuum-Short Form.
fEWB: emotional well-being.
gSWB: social well-being.
hPWB: psychological well-being.
iMPFI-SF: Multidimensional Psychological Flexibility Index-Short Form.
jPF: Psychological Flexibility.
kPI: Psychological Inflexibility.
lDASS-21: Depression, Anxiety, and Stress Scale-21.
mSUS: System Usability Scale.
nN/A: not applicable.
Improving burnout outcomes is a core goal of this intervention, and burnout is the primary outcome for the planned efficacy trial. We assessed burnout using the gold-standard measure, the Maslach Burnout Inventory . Of the range of versions available, the Maslach Burnout Inventory General Survey for Students [ ] was considered the most appropriate for this cohort.
Improving well-being is a core goal of this intervention, and well-being is an outcome of interest to the planned efficacy trial. We assessed this outcome using the short form of the Mental Health Continuum .
Psychological Flexibility and Inflexibility
In the planned efficacy trial, we aim to explore whether engagement with the intervention app improves medical students’ Psychological Flexibility and Inflexibility and evaluate whether changes in these processes mediate burnout and well-being outcomes . As there is growing evidence that Psychological Flexibility and Inflexibility are “conceptually distinct” processes that can exert independent effects on outcomes and may respond differently to interventions [ , ], we selected the Multidimensional Psychological Flexibility Index-Short Form [ ], which assesses Psychological Flexibility and Inflexibility as separate processes rather than as a single dimension [ ]. This will facilitate the evaluation of potential differential intervention and mediation effects in the future efficacy study.
Depression, Anxiety, and Stress
In addition to burnout and well-being, the planned efficacy trial aims to explore the potential benefits of the intervention app to other psychological health outcomes relevant to medical students. As such, the Depression, Anxiety, and Stress Scale-21  was administered to evaluate medical students’ psychological health outcomes more broadly.
Feasibility Outcome Measures
Behavioral Use Data
The app collected participants’ use data, including which assessment and intervention components were accessed and in which order. This allowed us to observe whether the app facilitated participants’ progress through the study protocol in the intended way. We defined the app as functional if the outcome measures and intervention components were delivered to all participants at the intended time and in the intended order, if individualization procedures were implemented as intended, if at least one participant completed the study protocol in its entirety, and if the app generated data output required to evaluate intervention outcomes.
Engagement at key points in the study was defined as the number of participants who (1) downloaded the app, (2) completed all outcome measures at t0, (3) completed the introductory module at stage 1, (4) completed all outcome measures at t1, (5) completed at least one skill training activity at stage 2, (6) completed all outcome measures at t2, and (7) were still using the app at least 2 weeks after their initial sign-up.
Frequency of App Use
Frequency of app use was defined as the number of log-ins during the study and the number of skill activities accessed and completed.
Time of App Use
The app captured the time of day of each log-in, which we categorized using the following definitions: early morning (5 AM to 9 AM), morning (9 AM to noon), afternoon (noon to 5 PM), evening (5 PM to 9 PM), late night (9 PM to midnight), and after midnight (midnight to 5 AM).
Self-report Measures of Usability and User Experiences
Self-report usability and user experience measures () were administered at t1 (to obtain students’ feedback on the intervention following the stage 1 introductory module) and t2 (to obtain feedback following the stage 2 individualized skill training access period).
App Interface Usability
We assessed medical students’ satisfaction with the usability of the app as subjective perceptions of the functionality of a digital psychological intervention can influence engagement [, ]. The System Usability Scale (SUS) is a valid and reliable measure of this outcome [ ] and is the most frequently used subjective measure of program usability [ ].
We used the theoretical framework of acceptability (TFA)  to guide the assessment of user experiences with the intervention. The TFA is a comprehensive research-based model that recommends evaluating participant-rated acceptability of health care interventions using 7 emotional and cognitive indicators: affective attitude, perceived effectiveness, intervention coherence, self-efficacy, opportunity costs, burden, and ethicality [ ]. As there were no available standardized TFA measures, we used the definitions of the 7 indicators to inform the development of a 2-part User Experience Assessment for the purpose of this study (see for indicator definitions and assessment questions). The development of the questions was further guided by research related to factors that affect the acceptability of psychological interventions among medical students (eg, time [ ], privacy [ , ], and accessibility and flexibility [ ]), health care interventions (eg, affective experiences and satisfaction [ ]), and digital interventions (eg, self-guided [ ], expectation of benefit [ ], quality, and content relevance [ ]). Where appropriate, questions were framed with respect to the outcomes targeted by the intervention (ie, burnout and well-being) to evaluate factors such as the perceived effectiveness and relevance of this approach among medical students. Part 1 asked participants to indicate their level of agreement with statements about their experiences of using the Psychological Flexibility app. We defined items with an average score of <3.8 as aspects of the intervention that might require revision before the planned efficacy trial. This value was selected as it represented an average rating of a little below agree. We defined items with an average score of >4 (“agree”) as intervention strengths. Part 2 consisted of 8 short-answer questions, which were included to collect feedback regarding facilitators of and barriers to engagement as well as other information that we may not have considered.
As part of assessing the feasibility of adapting ACTraining into a digital learning format, we identified key learning goals and asked medical students to rate the degree to which they felt that these had been achieved by engaging with the app. A learning experiences assessment previously used in a similar web-based intervention study  was adapted to align with the focus of this app ( [ ]). Questions related to medical students’ self-reported ability to identify Psychological Inflexibility processes and apply Psychological Flexibility skills across each of the core processes.
Skill Activity Feedback
After completing a skill activity during stage 2, participants were asked to rate whether they liked the activity using a single-item measure assessed on a binary scale (using a thumbs-up or thumbs-down icon). This method of evaluating user experiences of intervention content has been used in previous digital intervention studies .
Participants were invited to report concerns or harms experienced during the study using contact links provided within the app and via email communications sent during the trial (eg, daily reminders).
Descriptive analyses (including mean, SD, mode, minimum, and maximum) of quantitative demographic, psychological, feasibility, and behavioral engagement outcome data were performed using the Jamovi statistical software (version 2.3.0) .
Participants’ short-answer responses to the open-ended user experience questions were evaluated qualitatively. Responses were independently reviewed by 2 members of the research team (ED and BK) and coded thematically according to their relevance to usability, accessibility, external factors (eg, time), behavioral factors (eg, practice), or any of the TFA indicators (affective attitude, perceived effectiveness, intervention coherence, self-efficacy, opportunity costs, burden, and ethicality). Each response was coded as a barrier to or facilitator of engagement or a positive or negative evaluation of an aspect of the intervention. The coding was then reviewed together to determine common themes and subthemes in a reiterative process.
Participant Characteristics at Baseline (t0)
shows that 19 medical students enrolled and 11 (58%) actively engaged in the study and provided user experience feedback. Of the 11 actively engaged participants, 10 (91%) identified as female, 10 (91%) were enrolled as domestic students, and 2 (18%) were Aboriginal individuals. Participants were aged between 20 and 23 (mean 21, SD 1.2) years. The average time engaged in the workforce was 4.5 years (SD 2.3), and medicine was the first career for most (10/11, 91%) participants. In total, 73% (8/11) of the participants reported having experienced burnout in the past. Actively engaged participants rated their physical health as good (mean 3.8, SD 0.4). Diet (mean 3.6, SD 0.5) and self-care (mean 3.5, SD 0.7) ratings fell between average and good.
Of the participants lost to follow-up, 50% (4/8) identified as male, reflecting that most of the enrolled male students (4/5, 80%) discontinued the study before engaging in or providing feedback on the intervention. Although the demographic characteristics of the students lost to follow-up were generally similar to those of the students who engaged in the intervention, we noted that reported self-care was lower for students who discontinued early (mean 2.8, SD 0.7).
shows baseline psychological characteristics of the participants who actively engaged in the study.
|Emotional exhaustion||16.00 (6.24)||7-24|
|Academic efficacy||24.82 (5.90)||18-32|
|Psychological Flexibility||3.71 (0.67)||2.67-5|
|Psychological Inflexibility||3.03 (0.47)||2.25-3.92|
Mean total scores for exhaustion (mean 16.00, SD 6.24) and cynicism (mean 8.45, SD 3.50) for actively engaged students were comparable with those of a recently published sample of medical students who, by subjective self-report, were not experiencing burnout (exhaustion mean 14.96; cynicism mean 7.59) . Conversely, our sample’s mean total score for academic efficacy (mean 24.82, SD 5.90) was comparable with that of a sample of medical students who believed themselves to be experiencing burnout (mean 24.81) [ ].
On average, participants’ emotional (mean 11.36, SD 2.54), social (mean 13.45, SD 4.06), and psychological (mean 19.55, SD 4.57) well-being scores were slightly lower than those of a comparable published sample of medical students (emotional well-being mean 12.79; social well-being mean 14.84; psychological well-being mean 24.42) .
Psychological Flexibility and Inflexibility
Multidimensional Psychological Flexibility Index-Short Form scores indicated that, on average, participants had had experiences consistent with Psychological Flexibility often to very often during the previous 2 weeks (mean 3.71, SD 0.67), which was comparable with a general population sample (mean 3.83) . On average, students reported that they had often had experiences consistent with Psychological Inflexibility during the previous 2 weeks (mean 3.03, SD 0.47), which was slightly more frequent than the general population sample (mean 2.73) [ ].
Depression, Anxiety, and Stress
The average Depression, Anxiety, and Stress Scale-21 scores indicated that medical students in this trial were experiencing depression (mean 8.55, SD 6.82), anxiety (mean 8.18, SD 7.12), and stress (mean 10.36, SD 10.23) at levels higher than the general population averages (depression: mean 6.34, SD 6.97; anxiety: mean 4.7, SD 4.91; stress: mean 10.11, SD 7.91) .
Behavioral Use Data
The app generally functioned as intended during the trial. Use data demonstrated that outcome assessments and intervention components were delivered in the intended order and at the intended time, individualization procedures were implemented as planned, 16% (3/19) of the participants successfully completed the intervention protocol in its entirety, and the app generated the data output required to evaluate intervention outcomes. We noted that a system glitch resulted in participants being sent the daily reminder email up to 4 times each day.
Of the 19 participants who were enrolled in the study, 15 (79%) downloaded the app and created an account, 14 (74%) completed all t0 outcome measures, 11 (58%) completed stage 1 and the t1 outcome measures, 8 (42%) engaged in stage 2, and 3 (16%) completed the t2 outcome measures (). Of the participants who downloaded the app and created an account, 47% (7/15) continued to use the app for at least 2 weeks from the date of their initial log-in.
Frequency of App Use
Students who were actively engaged in the study (11/19, 58%) logged into the app between 2 and 10 times (total=70; mean 6.36, SD 2). Log-in frequency diminished between stage 1 (total=41; mean 3.73, SD 1) and stage 2 (total=29; mean 2.64, SD 2). It took students between 1 and 5 sittings to complete all the stage 1 activities. Participants who logged in during stage 2 (8/11, 73%) accessed a total of 24 skill activities (mean 2.18, SD 1), of which 16 (67%) were completed (mean 1.45, SD 1).
Time of App Use
shows that 56% (39/70) of app use was distributed fairly evenly throughout the day (between 9 AM and 9 PM) and that medical students used the app most frequently between 9 PM and midnight (21/70, 30%). Although reasonably infrequent (5/70, 7%), medical students occasionally used the app after midnight.
|Time of log-in||Log-in frequency, n (%)|
|Early morning (5 AM to 9 AM)||5 (7)|
|Morning (9 AM to noon)||12 (17)|
|Afternoon (noon to 5 PM)||14 (20)|
|Evening (5 PM to 9 PM)||13 (19)|
|Late night (9 PM to midnight)||21 (30)|
|After midnight (midnight to 6 AM)||5 (7)|
At t1, self-report measures of user experiences were completed by 100% (11/11) of the participants who completed stage 1. At t2, these measures were completed by 38% (3/8) of the participants who engaged in stage 2. Owing to the low response rate at t2, we report the minimum and maximum scores for each outcome instead of the average scores.
App Interface Usability
The average total SUS score at t1 (mean 85.0, SD 9.3) indicates excellent usability . Individual participant SUS scores ranged between 70 (low marginal) and 96 (excellent), indicating that there were substantial differences in participants’ satisfaction with the usability of the app interface. Similarly, at t2, individual usability ratings varied, ranging between 72 and 96.
The qualitative feedback provided by participants at t1 offers insights into the factors that may have influenced this variability. Many students reported feeling satisfied by the ease of use and navigation, logical structure and flow of the app interface, provision of simple and clear instructions, and continuation of audio play when their device auto-locked. However, app usability elements that generated dissatisfaction included lack of access to a dashboard or home screen during stage 1, insufficient feedback regarding how far students had progressed through the introductory module, having to complete module components in a fixed order rather than being able to navigate flexibly through the concepts, and the delivery of reminders via email rather than in-app push notifications. Although some students favored the audio delivery of content, others reported that having a written version of all content would have improved accessibility (eg, audio content was a barrier to usability when in public, and audio content delivery did not match personal learning styles for some).
At t1, the average total user experience ratings ranged between slightly agree (3.4) and strongly agree (4.8; mean 4.05, SD 0.34), indicating that participants who engaged in stage 1 generally found the intervention to be acceptable with respect to the experiences assessed. Similar findings were observed at t2 (range 3.3-4.6).
The average participant scores for each user experience question at t1 are presented in. We identified several intervention strengths (average item scores of >4). Strengths related to the app delivery format and interface included quality, visual appearance, web-based availability, ability to access at a personally convenient time, maintenance of privacy, and did not get in the way of other important life activities. Strengths related to the ACTraining intervention content included opportunities for self-reflection, content relevance, anticipated benefits of continued skill use, anticipated benefits to well-being during career, concepts were easy to understand, felt confident performing the skill activities while using the app, felt capable of dealing with any challenging internal experiences that arose while using the app, aligned with personal values, and the time and effort invested in the Psychological Flexibility training was worthwhile. After completing the introductory module, most students (8/11, 73%) agreed or strongly agreed that they would like to use the app again in the future (mean 3.91, SD 0.94).
We identified 5 user experience items with average scores below the 3.8 cutoff at t1, indicating that these intervention elements might require additional attention before the larger trial. These items were enjoyed using the app, held interest and attention, received amount of training needed to achieve the goals that are important to me, felt confident using the skills in everyday life, and I believe these skills will help me prevent burnout. Qualitative feedback provided by 18% (2/11) of the participants indicated that they did not feel that burnout prevention should be the focus of the intervention as the techniques were “helpful for many different things.”
Most participants (9/11, 82%) indicated that the 1-hour introductory module was slightly too long, whereas the remaining 18% (2/11) of participants felt that the duration was just right. However, qualitative feedback indicated that some participants felt that the content was as concise as it could be without compromising the value of the intervention. At both time points, participants indicated that they did not find completing the outcome questionnaires to be burdensome.
The qualitative user experience feedback at t1 generally reflected the quantitative findings. Most students who completed stage 1 provided favorable feedback about the Psychological Flexibility intervention content, describing the app as “great,” “useful,” and “helpful” and suggesting that the training “needs to be widely available.” One of the reported benefits of the app-based delivery was that students liked being able to engage in short sections and pick up where they left off rather than having to complete the entire intervention at once. Students reported enjoying the skill activities, stating that they “provided context and demonstrated how to use [the skills],” and some students requested “more interactive components and practice activities” and more “clinical examples.” T1 qualitative data also highlighted the self-efficacy challenges faced by students when accessing Psychological Flexibility skill training via an app, including feeling “too overwhelmed” in real-world situations to use the skills, not recognizing opportunities to use the skills when they arose, and not knowing which skills to apply in which situations. A participant reported feeling that they had learned the Psychological Flexibility “concepts” but not the “behavior.” These findings align with the fact that stage 1 focused on conceptual learning and participants had only engaged in minimal skill training at this stage of the intervention. Students also reported that a key barrier to using the app was having time available to do so.
Qualitative data collected at t2 provided further insights into the perceptions of a small selection of participants regarding the stage 2 individualized skill training. Students liked the app interface and the short duration of skill activities. However, the high frequency of reminders resulting from an app glitch was reported as unsatisfactory and frustrating. A participant suggested that reminders should be delivered at the end of the workday as this would align better with medical students’ schedules and reduce burden. Barriers to engagement in skill training included life commitments, time, forgetting, and the fact that the training had not yet become a priority or a habit.
At t1, participants’ learning experience assessment scores ranged from 2.92 (slightly agree) to 4.1 (agree; mean 3.7, SD 0.51). The most frequently endorsed rating for 92% (12/13) of the learning experiences was agree (mode=4). At t2, scores ranged from 2.62 to 4, with a participant indicating that they did not learn the skills targeted by the intervention and 18% (2/11) of the participants agreeing that they had learned the skills.
Skill Activity Feedback
Students “liked” 88% (14/16) of the activities completed during stage 2.
No harms were reported during this trial.
Feasibility, Engagement, and User Experiences
This early implementation trial with a small group of Australian medical students provided preliminary support for the feasibility of an individualized ACTraining intervention app for well-being and burnout as well as user experience feedback that may be used to guide the ongoing development of the app and future research. In relation to our first study aim, we demonstrated that the app could successfully administer outcome assessments, ACTraining intervention components, and Psychological Flexibility skill individualization procedures. These functional feasibility data were supported by subjective user experience feedback provided by medical students who actively engaged with the app. The interface was, on average, rated as having excellent usability, and students responded favorably to accessibility factors related to the app-based delivery (eg, maintenance of privacy and ability to access at any time). Furthermore, we demonstrated the feasibility of using an app to deliver ACTraining content and skill training to a sample of medical students in a way that they found easy to understand and acceptable.
More broadly, the feasibility of delivering psychological skill training via an app is contingent on the degree to which this method can facilitate and maintain end-user engagement with the intervention content. We found that 42% (8/19) of the medical students who enrolled in the study disengaged without completing the introductory module, and 50% (4/8) of those students did not proceed as far as downloading or signing up to the app. These students reported lower average self-care ratings than those who completed the introductory module, and future research should explore whether existing self-care behavioral patterns could impede early engagement in psychological intervention apps. Regarding maintenance of engagement, the frequency of app log-ins diminished between stages 1 and 2. Less than half (7/15, 47%) of the students who downloaded the app were still engaged after 2 weeks. Those who used the app during stage 2 completed <2 skill activities on average despite “liking” most activities completed. Although the observed pattern of disengagement over time is common for digital and app-based psychological interventions [, ], it is incongruent with the fact that students who completed the introductory module generally reported that they would like to use the app in the future. These findings reinforce the need for a detailed formal evaluation of end-user experiences to better understand factors that may affect feasibility [ ] and guide app intervention development, which was the second aim of this study.
Previous research evaluating an app-delivered ACTraining intervention indicated that disengagement is not necessarily an unfavorable outcome and might reflect the successful achievement of an individual’s learning goals . User experience findings suggest that this was not the case in this sample. Learning experience scores showed that medical students who completed the introductory module agreed that they had learned most of the Psychological Flexibility skills targeted by the ACTraining app to some extent. However, on average, participants only slightly agreed that they had received the amount of training needed to achieve the outcomes that were important to them, and qualitative feedback indicated that they wanted more opportunities for skill practice. Thus, declining engagement following the introductory module was likely related to other factors.
Medical students who completed the introductory module predominantly reported finding the Psychological Flexibility concepts and skills relevant, helpful, and worthwhile. Students tended to perceive that the skills had greater benefits and relevance to well-being than burnout prevention. Given that the intervention was intended to have relevance to both outcomes, this feedback suggests that content might not have aligned sufficiently with factors that participants perceived to be relevant to their burnout experiences. A future step in the development process of this app will involve consultation with medical student end users regarding which aspects of their academic and clinical experiences they consider to be most associated with burnout risk. Intervention content can then be adjusted to demonstrate how Psychological Flexibility skills might be implemented to mitigate these risk factors, which could increase perceived relevance and engagement. Some students astutely recognized the potential for Psychological Flexibility to offer benefits beyond burnout prevention and proposed that the intervention shift its focus and messaging to the broader relevance of these skills. This approach may increase the engagement of medical students interested in well-being and other outcomes. However, removing the burnout prevention focus could also result in failure to engage students who self-identify as being at risk as well as missing opportunities to train important risk-mitigating applications of Psychological Flexibility skills. Future research might explore whether engagement is improved by using the app to assess which outcomes are most important to individual medical students and individualizing the focus of intervention content based on these preferences.
Performing self-reflection and skill-learning tasks without in-person guidance is a potential challenge for individuals engaging in a psychological intervention using a stand-alone app . Medical students who actively trialed this intervention reported both enjoyment and confidence with respect to these activities while using the app. However, lower levels of confidence and self-efficacy were reported with respect to implementing the skills in real-world situations, with some students not knowing when or how to apply specific skills and others feeling “too overwhelmed” to do so. Previous research has demonstrated that perceived self-efficacy can affect engagement in digital health interventions [ ]. Although stage 2 of the app was intended to strengthen and generalize skill acquisition by providing access to targeted training in real-world contexts, clearer guidance may have been required regarding when and how to practice the skills. Future research may also consider how to use in-app assessment of students’ confidence and self-efficacy to tailor intervention delivery to their current skill level and explore whether this enhances engagement.
We identified enjoyment of the intervention and its capacity to hold medical students’ interest and attention as aspects of the app requiring further attention. Most participants who trialed the app (9/11, 82%) reported finding the 1-hour introductory module slightly too long. Given that this was very brief compared with face-to-face ACTraining interventions [- ] and that most students reported wanting further training, reducing the volume of introductory content could affect future efficacy and is not an ideal solution. Rather, qualitative user experience feedback highlighted aspects of the app interface that may have contributed to this module feeling subjectively long. Delivery of content in a fixed order was a frustration for many students, who reported finding it difficult to track how far they had progressed through the module and would have preferred the autonomy to move between topics using a home screen. Students indicated that they would have liked more interactive components and opportunities to practice skills and self-reflection, suggesting that the introductory content could have been delivered in a more engaging format. Furthermore, although many students responded favorably to the audio-delivered content, others reported that the availability of a written version would have increased accessibility and engagement. Medical students’ preferences for interacting with app-based media appeared to affect their experiences of participating in the intervention. When adapting established psychological intervention models to app-based intervention models, future research may explore whether evaluating and accommodating medical students’ interface-related preferences strengthens interest and engagement.
Feedback provided by medical students who actively participated in the intervention indicated that the short and resumable nature of the app-delivered skill activities facilitated engagement and that the time invested was worthwhile. However, time factors and demands related to medical studies were reported barriers to app use, which is consistent with previous research evaluating an app-delivered psychological intervention for medical residents . Participant feedback suggested that this might be mitigated by amending the format and timing of reminders, such as push notifications at the end of the day, when there are fewer competing demands associated with medical studies. This approach is supported by the observed timing of app engagement occurring frequently after 9 PM. Although this intervention focused on strategies to support well-being behaviors in the context of time pressures and demands, future interventions should introduce these components during the early stages of training, when app engagement is at its highest.
Medical students in this sample reported depression, anxiety, and stress experiences at levels higher than those observed in general population samples , and previous research indicates that these factors can affect engagement in digital interventions [ ]. Furthermore, this sample reported high baseline Psychological Inflexibility compared with a general population sample [ ]. Although the ACTraining app was intended to improve this outcome, if existing inflexible behavioral repertoires function as barriers to engagement, then steps would need to be taken to mitigate this during the early stages of the intervention. Future research should explore whether baseline flexibility and inflexibility predict app engagement.
Although this trial had the full support of the universities in which it was conducted, the intervention was intentionally distanced from students’ medical education programs to prevent perceived participation coercion. Although important from a research ethics perspective, this meant that the app was delivered in isolation without external contextual factors that might have facilitated engagement. For example, delivery as a formal part of medical education training could normalize and legitimize the value of well-being skill training, which may strengthen students’ justification for prioritizing and allocating time for engagement [, ]. There is support for individual skill training interventions to be formally incorporated into broader well-being and burnout prevention initiatives within medical education [ ], and future research should explore whether this approach could strengthen engagement in app-based training.
Engagement rates for this trial should be interpreted with caution. The volume of study-related outcome measures that were extraneous to the intervention itself was high and might have contributed to disengagement, particularly given that time was reported as a barrier to app engagement even among those who participated for the full duration of the study. Although medical students who completed the outcome measures did not find them to be burdensome, this may not have been the case for those who disengaged. The user experience findings were biased toward the opinions of medical students who were actively engaged in the intervention. Future feasibility studies should adopt a secondary method of follow-up data collection to assess the experiences of students who discontinue app use to facilitate a better understanding of engagement barriers.
The collection of user experience data using a nonstandardized questionnaire written for the purpose of this study was a limitation. Although the User Experience Assessment was theory-driven, it is possible that researcher subjectivity might have biased its development and response interpretation. Given the lack of quantitative scales based on the TFA , we have made this assessment available ( ) to facilitate further validation and refinement of feasibility assessment strategies for psychological app development studies.
The skew toward female participants and small sample size prevent the generalizability of the observed findings to medical students more broadly. Furthermore, although subjective feedback was informative regarding what was important to participants while using the app, we were unable to confirm whether these factors affected actual behavioral engagement because of the sample size and study design, and further research is required to explore the proposed relationships. Finally, the low level of engagement with the stage 2 individualized skill training component meant that most of the user experience data collected pertained to the introductory module only.
Theoretical Contributions and Conclusions
Despite its limitations, this study reports detailed user experience data that have been largely absent from the literature on digital psychological interventions for medical students  and outlines a process for the important step of assessing feasibility outcomes as part of app development. The findings support the feasibility of implementing this app in a larger efficacy trial and provide user experience feedback that could be used to improve the intervention and strengthen engagement. Intervention development was driven by research suggesting the relevance of Psychological Flexibility skills to medical students with respect to burnout and well-being [ , - ] and the potential benefits of app-delivered psychological skill training for this group [ , , , - ]. Although this is early-phase work, the study provides preliminary support for the feasibility of delivering an ACTraining intervention to medical students using a stand-alone app and supports the value of further evaluation of this approach in burnout prevention and well-being research. Psychological Flexibility has recently been described as “the cornerstone of psychological health and resiliency” [ ] because of its broad well-being benefits. With continued evaluation and refinement, ACTraining apps offer a promising approach to facilitating medical students’ development of these important skills, which could be useful in the context of unavoidable academic and occupational stressors.
This study is the first to report detailed feedback from medical student end users regarding their experience of engaging in an app-delivered psychological intervention. Although specific findings may not be generalizable beyond this sample and app, our observations and evaluation process have potential relevance to researchers in the emerging field of app-delivered psychological interventions for medical student well-being. We demonstrated that making an intervention accessible to medical students via an app does not guarantee that they will access it, even when their self-reported intention is to do so, and that formal evaluation of user experiences across a range of domains may facilitate the identification of contributing factors. Our findings suggest that it may be useful for intervention developers to assess medical students’ preferences regarding the functionality and usability aspects of app interfaces as well as individual differences in learning goals, baseline skills, and confidence. Failure to understand user experiences could result in disengagement for reasons unrelated to the intervention model, leading to missed opportunities to fully develop and deploy potentially valuable psychological skill training . We recommend that future researchers incorporate formal evaluation of medical students’ subjective user experiences into the intervention development process (including quantitative and qualitative assessments) to better understand and address factors that might affect engagement and disengagement.
This research project was funded by the Centre for Rehab Innovations and the Priority Research Centre for Stroke and Brain Injury at the University of Newcastle. The authors wish to acknowledge the Joint Medical Program (University of Newcastle and University of New England) for supporting this research and the medical students who participated in the trial, the Cogniss team for their commitment to fine-tuning the app throughout the development process, and Lucy Bailey for providing instrumental support in the administrative aspects of the study.
The data sets generated and analyzed during this study are not publicly available for the protection of participant privacy, but deidentified outcome data sets are available from the corresponding author upon reasonable request.
ED wrote the manuscript, designed the study protocol, and created the app. ED and BK wrote and compiled the intervention content. NH provided consultation on the study protocol design, data analyses, and app usability. GH led stakeholder engagement and provided consultation on the translation of various aspects of the protocol and intervention to the cohort context. MN and RW oversaw the project. All authors reviewed and contributed to the manuscript.
Conflicts of Interest
Stage 1 screenshot.PNG File , 395 KB
App home screen.PNG File , 371 KB
App screening questions.PNG File , 433 KB
Stage 2 skill set dashboard.PNG File , 119 KB
Acceptability assessment and mean participant data at t1 (n=11).DOCX File , 24 KB
Learning experiences assessment.DOCX File , 15 KB
- Yogeswaran V, El Morr C. Effectiveness of online mindfulness interventions on medical students' mental health: a systematic review. BMC Public Health 2021 Dec 18;21(1):2293 [FREE Full text] [CrossRef] [Medline]
- Rotenstein LS, Ramos MA, Torre M, Segal JB, Peluso MJ, Guille C, et al. Prevalence of depression, depressive symptoms, and suicidal ideation among medical students: a systematic review and meta-analysis. JAMA 2016 Dec 06;316(21):2214-2236 [FREE Full text] [CrossRef] [Medline]
- Seo C, Corrado M, Fournier K, Bailey T, Haykal KA. Addressing the physician burnout epidemic with resilience curricula in medical education: a systematic review. BMC Med Educ 2021 Feb 01;21(1):80 [FREE Full text] [CrossRef] [Medline]
- Frajerman A, Morvan Y, Krebs MO, Gorwood P, Chaumette B. Burnout in medical students before residency: a systematic review and meta-analysis. Eur Psychiatry 2019 Jan;55:36-42. [CrossRef] [Medline]
- Bugaj T, Cranz A, Junne F, Erschens R, Herzog W, Nikendei C. Psychosocial burden in medical students and specific prevention strategies. Ment Health Prevent 2016 Mar;4(1):24-30. [CrossRef]
- Fares J, Al Tabosh H, Saadeddin Z, El Mouhayyar C, Aridi H. Stress, burnout and coping strategies in preclinical medical students. N Am J Med Sci 2016 Feb;8(2):75-81 [FREE Full text] [CrossRef] [Medline]
- Maslach C, Leiter MP. Understanding the burnout experience: recent research and its implications for psychiatry. World Psychiatry 2016 Jun;15(2):103-111 [FREE Full text] [CrossRef] [Medline]
- Shanafelt TD, Boone S, Tan L, Dyrbye LN, Sotile W, Satele D, et al. Burnout and satisfaction with work-life balance among US physicians relative to the general US population. Arch Intern Med 2012 Oct 08;172(18):1377-1385. [CrossRef] [Medline]
- Shanafelt TD, Mungo M, Schmitgen J, Storz KA, Reeves D, Hayes SN, et al. Longitudinal study evaluating the association between physician burnout and changes in professional work effort. Mayo Clin Proc 2016 Apr;91(4):422-431. [CrossRef] [Medline]
- Kemp S, Hu W, Bishop J, Forrest K, Hudson JN, Wilson I, et al. Medical student wellbeing - a consensus statement from Australia and New Zealand. BMC Med Educ 2019 Mar 04;19(1):69 [FREE Full text] [CrossRef] [Medline]
- Walsh AL, Lehmann S, Zabinski J, Truskey M, Purvis T, Gould NF, et al. Interventions to prevent and reduce burnout among undergraduate and graduate medical education trainees: a systematic review. Acad Psychiatry 2019 Aug;43(4):386-395. [CrossRef] [Medline]
- Harvey SB, Epstein RM, Glozier N, Petrie K, Strudwick J, Gayed A, et al. Mental illness and suicide among physicians. Lancet 2021 Sep 04;398(10303):920-930 [FREE Full text] [CrossRef] [Medline]
- Dyrbye LN, Satele D, West CP. Association of characteristics of the learning environment and US medical student burnout, empathy, and career regret. JAMA Netw Open 2021 Aug 02;4(8):e2119110 [FREE Full text] [CrossRef] [Medline]
- Amanullah S, Ramesh Shankar R. The impact of COVID-19 on physician burnout globally: a review. Healthcare (Basel) 2020 Oct 22;8(4):421 [FREE Full text] [CrossRef] [Medline]
- Ishak W, Nikravesh R, Lederer S, Perry R, Ogunyemi D, Bernstein C. Burnout in medical students: a systematic review. Clin Teach 2013 Aug;10(4):242-245. [CrossRef] [Medline]
- Arif NM, Roslan NS, Ismail SB, Nayak RD, Jamian MR, Mohamad Ali Roshidi AS, et al. Prevalence and associated factors of psychological distress and burnout among medical students: findings from two campuses. Int J Environ Res Public Health 2021 Aug 10;18(16):8446 [FREE Full text] [CrossRef] [Medline]
- Wersebe H, Lieb R, Meyer AH, Hofer P, Gloster AT. The link between stress, well-being, and psychological flexibility during an Acceptance and Commitment Therapy self-help intervention. Int J Clin Health Psychol 2018;18(1):60-68 [FREE Full text] [CrossRef] [Medline]
- Puolakanaho A, Tolvanen A, Kinnunen SM, Lappalainen R. A psychological flexibility -based intervention for burnout:a randomized controlled trial. J Contextual Behav Sci 2020 Jan;15:52-67. [CrossRef]
- Keyes CL. Mental illness and/or mental health? Investigating axioms of the complete state model of health. J Consult Clin Psychol 2005 Jun;73(3):539-548 [FREE Full text] [CrossRef] [Medline]
- Lamers SM, Westerhof GJ, Bohlmeijer ET, ten Klooster PM, Keyes CL. Evaluating the psychometric properties of the Mental Health Continuum-Short Form (MHC-SF). J Clin Psychol 2011 Jan;67(1):99-110. [CrossRef] [Medline]
- Lenoir AL, De Troyer C, Demoulin C, Gillain I, Bayot M. Challenges in treating physician burnout: the psychologist's perspective. Presse Méd 2021 Jun;2:100006. [CrossRef]
- Taylor M, Hageman JR, Brown M. A mindfulness intervention for residents: relevance for pediatricians. Pediatr Ann 2016 Oct 01;45(10):e373-e376. [CrossRef] [Medline]
- Wen L, Sweeney TE, Welton L, Trockel M, Katznelson L. Encouraging mindfulness in medical house staff via smartphone app: a pilot study. Acad Psychiatry 2017 Oct;41(5):646-650. [CrossRef] [Medline]
- Awasthi S, Kaur A, Solanki H, Pamei G, Bhatt M. Smartphone use and the quality of life of medical students in the Kumaun Region, Uttarakhand. J Family Med Prim Care 2020 Aug;9(8):4252-4258 [FREE Full text] [CrossRef] [Medline]
- Singh K, Sarkar S, Gaur U, Gupta S, Adams OP, Sa B, et al. Smartphones and educational apps use among medical students of a smart university campus. Front Commun 2021 Nov 2;6:49102. [CrossRef]
- Patel S, Akhtar A, Malins S, Wright N, Rowley E, Young E, et al. The acceptability and usability of digital health interventions for adults with depression, anxiety, and somatoform disorders: qualitative systematic review and meta-synthesis. J Med Internet Res 2020 Jul 06;22(7):e16228 [FREE Full text] [CrossRef] [Medline]
- Levin ME, Haeger J, Cruz RA. Tailoring acceptance and commitment therapy skill coaching in the moment through smartphones: results from a randomized controlled trial. Mindfulness 2018 Jul 21;10(4):689-699. [CrossRef]
- Amanullah S, McNally K, Zelin J, Cole J, Cernovsky Z. Are burnout prevention programs for hospital physicians needed? Asian J Psychiatr 2017 Apr;26:66-69. [CrossRef] [Medline]
- Kalani SD, Azadfallah P, Oreyzi H, Adibi P. Interventions for physician burnout: a systematic review of systematic reviews. Int J Prev Med 2018 Sep 17;9:81 [FREE Full text] [CrossRef] [Medline]
- Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [CrossRef] [Medline]
- Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017 Jan 26;17(1):88 [FREE Full text] [CrossRef] [Medline]
- Blatch-Jones AJ, Pek W, Kirkpatrick E, Ashton-Key M. Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study. BMJ Open 2018 Sep 25;8(9):e022233 [FREE Full text] [CrossRef] [Medline]
- Yogeswaran V, El Morr C. Mental Health for Medical Students, what do we know today? Proc Comput Sci 2022;198:307-310 [FREE Full text] [CrossRef]
- Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 2010 Jan 06;10:1 [FREE Full text] [CrossRef] [Medline]
- Bell ML, Whitehead AL, Julious SA. Guidance for using pilot studies to inform the design of intervention trials with continuous outcomes. Clin Epidemiol 2018 Jan 18;10:153-157 [FREE Full text] [CrossRef] [Medline]
- Onwezen MC, van Veldhoven MJ, Biron M. The role of psychological flexibility in the demands–exhaustion–performance relationship. Eur J Work Organ Psychol 2014;23(2):163-176. [CrossRef]
- Lloyd J, Bond FW, Flaxman PE. The value of psychological flexibility: examining psychological mechanisms underpinning a cognitive behavioural therapy intervention for burnout. Work Stress 2013 Apr;27(2):181-199. [CrossRef]
- Vilardaga R, Luoma JB, Hayes SC, Pistorello J, Levin ME, Hildebrandt MJ, et al. Burnout among the addiction counseling workforce: the differential roles of mindfulness and values-based processes and work-site factors. J Subst Abuse Treat 2011 Jun;40(4):323-335 [FREE Full text] [CrossRef] [Medline]
- Veage S, Ciarrochi J, Deane FP, Andresen R, Oades LG, Crowe TP. Value congruence, importance and success and in the workplace: links with well-being and burnout amongst mental health practitioners. J Contextual Behav Sci 2014 Oct;3(4):258-264. [CrossRef]
- Bond FW, Bunce D. The role of acceptance and job control in mental health, job satisfaction, and work performance. J Appl Psychol 2003 Dec;88(6):1057-1067. [CrossRef] [Medline]
- Gloster AT, Meyer AH, Lieb R. Psychological flexibility as a malleable public health target: evidence from a representative sample. J Contextual Behav Sci 2017 Apr;6(2):166-171. [CrossRef]
- Kinnunen SM, Puolakanaho A, Mäkikangas A, Tolvanen A, Lappalainen R. Does a mindfulness-, acceptance-, and value-based intervention for burnout have long-term effects on different levels of subjective well-being? Int J Stress Manag 2020 Feb;27(1):82-87. [CrossRef]
- Hayes SC, Strosahl KD, Wilson KG. Acceptance and Commitment Therapy: The Process and Practice of Mindful Change. 2nd edition. New York, NY, USA: The Guilford Press; 2016.
- Stabbe OK, Rolffs JL, Rogge RD. Flexibly and/or inflexibly embracing life: identifying fundamental approaches to life with latent profile analyses on the dimensions of the Hexaflex model. J Contextual Behav Sci 2019 Apr;12:106-118. [CrossRef]
- Moran DJ. ACT for leadership: using acceptance and commitment training to develop crisis-resilient change managers. Int J Behav Consult Ther 2010;6(4):341-355. [CrossRef]
- Ramaci T, Bellini D, Presti G, Santisi G. Psychological flexibility and mindfulness as predictors of individual outcomes in hospital health workers. Front Psychol 2019 Jun 12;10:1302 [FREE Full text] [CrossRef] [Medline]
- Thompson G, McBride RB, Hosford CC, Halaas G. Resilience among medical students: the role of coping style and social support. Teach Learn Med 2016;28(2):174-182. [CrossRef] [Medline]
- Kroska EB, Calarge C, O’Hara MW, Deumic E, Dindo L. Burnout and depression in medical students: relations with avoidance and disengagement. J Contextual Behav Sci 2017 Oct;6(4):404-408. [CrossRef]
- Johns G, Waddington L, Samuel V. Prevalence and predictors of mental health outcomes in UK doctors and final year medical students during the COVID-19 pandemic. J Affect Disord 2022 Aug 15;311:267-275 [FREE Full text] [CrossRef] [Medline]
- Palladino CL, Ange B, Richardson DS, Casillas R, Decker M, Gillies RA, et al. Measuring psychological flexibility in medical students and residents: a psychometric analysis. Med Educ Online 2013 Aug 13;18:20932 [FREE Full text] [CrossRef] [Medline]
- Szarko AJ, Houmanfar RA, Smith GS, Jacobs NN, Smith BM, Assemi K, et al. Impact of Acceptance and Commitment Training on psychological flexibility and burnout in medical education. J Contextual Behav Sci 2022 Jan;23:190-199. [CrossRef]
- Shahsavari Googhari Z, Hafezi F, Asgari P, Heidari A. The effectiveness of mindfulness-based cognitive therapy and acceptance and commitment therapy on medical science students’ subjective well-being, psychological distress, and emotion regulation. J Shahrekord Univ Med Sci 2022 Jan 26;24(1):35-41. [CrossRef]
- Villatte JL, Vilardaga R, Villatte M, Plumb Vilardaga JC, Atkins DC, Hayes SC. Acceptance and Commitment Therapy modules: differential impact on treatment processes and outcomes. Behav Res Ther 2016 Feb;77:52-61 [FREE Full text] [CrossRef] [Medline]
- Rolffs JL, Rogge RD, Wilson KG. Disentangling components of flexibility via the hexaflex model: development and validation of the multidimensional psychological flexibility inventory (MPFI). Assessment 2018 Jun;25(4):458-482. [CrossRef] [Medline]
- Rogge RD, Daks JS, Dubler BA, Saint KJ. It's all about the process: examining the convergent validity, conceptual coverage, unique predictive validity, and clinical utility of ACT process measures. J Contextual Behav Sci 2019 Oct;14:90-102. [CrossRef]
- Reeve A, Tickle A, Moghaddam N. Are acceptance and commitment therapy-based interventions effective for reducing burnout in direct-care staff? A systematic review and meta-analysis. Ment Health Rev J 2018 Jul 04;23(3):131-155. [CrossRef]
- Ditton E, Knott B, Hodyl N, Horton G, Walker FR, Nilsson M. Assessing the efficacy of an individualized psychological flexibility skills training intervention app for medical student burnout and well-being: protocol for a randomized controlled trial. JMIR Res Protoc 2022 Feb 04;11(2):e32992 [FREE Full text] [CrossRef] [Medline]
- Larmar S, Wiatrowski S, Lewis-Driver S. Acceptance and Commitment Therapy: an overview of techniques and applications. J Serv Sci Manag 2014 Jun;07(03):216-221. [CrossRef]
- Viskovich S, Pakenham KI. Pilot evaluation of a web-based acceptance and commitment therapy program to promote mental health skills in university students. J Clin Psychol 2018 Dec;74(12):2047-2069. [CrossRef] [Medline]
- Firestone J, Cardaciotto L, Levin ME, Goldbacher E, Vernig P, Gambrel LE. A web-based self-guided program to promote valued-living in college students: a pilot study. J Contextual Behav Sci 2019 Apr;12:29-38. [CrossRef]
- Schaufeli WB, Martínez IM, Pinto AM, Salanova M, Bakker AB. Burnout and engagement in university students: a cross-national study. J Cross Cult Psychol 2002 Sep;33(5):464-481. [CrossRef]
- Maslach C, Jackson SE, Leiter MP. Maslach Burnout Inventory Manual. 3rd edition. Mountain View, CA, USA: CPP Incorporated; 1996.
- Obregon M, Luo J, Shelton J, Blevins T, MacDowell M. Assessment of burnout in medical students using the Maslach Burnout Inventory-Student Survey: a cross-sectional data analysis. BMC Med Educ 2020 Oct 21;20(1):376 [FREE Full text] [CrossRef] [Medline]
- Keyes CL. Brief description of the mental health continuum short form (MHC-SF). The University of North Carolina. Chapel Hill, NC, USA: The University of North Carolina; 2009. URL: https://peplab.web.unc.edu/wp-content/uploads/sites/18901/2018/11/MHC-SFoverview.pdf [accessed 2021-01-30]
- Seidler D, Stone B, Clark BE, Koran J, Drake CE. Evaluating the factor structure of the Multidimensional Psychological Flexibility Inventory: an independent replication and extension. J Contextual Behav Sci 2020 Jul;17:23-31. [CrossRef]
- Lovibond SH, Lovibond PF. Manual for the Depression Anxiety Stress Scales. 2nd edition. Sydney, Australia: Psychology Foundation of Australia; 1995.
- Henry JD, Crawford JR. The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br J Clin Psychol 2005 Jun;44(Pt 2):227-239. [CrossRef] [Medline]
- Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009 May;4(3):114-123.
- Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Human Comput Interact 2008 Jul 30;24(6):574-594. [CrossRef]
- Kinnunen SM, Puolakanaho A, Tolvanen A, Mäkikangas A, Lappalainen R. Does mindfulness-, acceptance-, and value-based intervention alleviate burnout?—A person-centered approach. Int J Stress Manag 2019 Feb;26(1):89-101. [CrossRef]
- Lewis JR. The system usability scale: past, present, and future. Int J Human Comput Interact 2018 Mar 30;34(7):577-590. [CrossRef]
- Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland L, Weerdmeester B, editors. Usability Evaluation in Industry. London, UK: CRC Press; 1996:189-194.
- Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017 Jun;7(2):254-267 [FREE Full text] [CrossRef] [Medline]
- Prochaska JJ, Vogel EA, Chieng A, Kendra M, Baiocchi M, Pajarito S, et al. A therapeutic relational agent for reducing problematic substance use (Woebot): development and usability study. J Med Internet Res 2021 Mar 23;23(3):e24850 [FREE Full text] [CrossRef] [Medline]
- The Jamovi Project. Jamovi (Version 2.3). Jamovi. 2022. URL: https://www.jamovi.org/ [accessed 2022-12-08]
- Michalec B, Keyes CL. A multidimensional perspective of the mental health of preclinical medical students. Psychol Health Med 2013;18(1):89-97. [CrossRef] [Medline]
- Grégoire S, Gagnon J, Lachance L, Shankland R, Dionne F, Kotsou I, et al. Validation of the english and french versions of the multidimensional psychological flexibility inventory short form (MPFI-24). J Contextual Behav Sci 2020 Oct;18:99-110. [CrossRef]
- Wang J, Song B, Shao Y, Zhu J. Effect of online psychological intervention on burnout in medical residents from different majors: an exploratory study. Front Psychol 2021 May 7;12:632134 [FREE Full text] [CrossRef] [Medline]
- Dindo L. One-day acceptance and commitment training workshops in medical populations. Curr Opin Psychol 2015 Apr 01;2:38-42 [FREE Full text] [CrossRef] [Medline]
- Kroska EB, Roche AI, O'Hara MW. How much is enough in brief Acceptance and Commitment Therapy? A randomized trial. J Contextual Behav Sci 2020 Jan;15:235-244. [CrossRef]
- Towey-Swift KD, Lauvrud C, Whittington R. Acceptance and commitment therapy (ACT) for professional staff burnout: a systematic review and narrative synthesis of controlled trials. J Ment Health (forthcoming) 2022 Feb 02:1-13. [CrossRef] [Medline]
- Pakenham KI, Landi G, Boccolini G, Furlani A, Grandi S, Tossani E. The moderating roles of psychological flexibility and inflexibility on the mental health impacts of COVID-19 pandemic and lockdown in Italy. J Contextual Behav Sci 2020 Jul;17:109-118 [FREE Full text] [CrossRef] [Medline]
|ACTraining: Acceptance and Commitment Training|
|REDCap: Research Electronic Data Capture|
|SUS: System Usability Scale|
|TFA: theoretical framework of acceptability|
Edited by A Mavragani; submitted 06.10.22; peer-reviewed by C Jacob, L Grepo; comments to author 07.11.22; revised version received 22.11.22; accepted 28.11.22; published 10.01.23Copyright
©Elizabeth Ditton, Brendon Knott, Nicolette Hodyl, Graeme Horton, Frederick Rohan Walker, Michael Nilsson. Originally published in JMIR Formative Research (https://formative.jmir.org), 10.01.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.