This study retrospectively evaluated data from patients whose treatment had ended primarily to inform the improvement of ongoing programing. Patients give authorization for their surveys and recorded clinical data to be used for ongoing program evaluation when they agree to treatment; they are provided this information again each time they are provided a survey, and have the option to skip the full survey and/or any individual questions they choose.

Given that the client data used for the analyses in this paper were observational and pulled from our existing electronic health record system and that the primary purpose was QI rather than research, the initiative was reviewed and approved by the NorthStar Institutional Review Board as exempt from the common rule definition of research (NB400174). The primary client consent covers secondary analysis without additional consent. Patients received no compensation for their participation in the study.

Three key decisions were made at the outset concerning the individuals who would be involved in the initiative (CFIR implementation leads construct). First, an external subject matter expert (SME), the second author, was engaged to provide consultation and the necessary clinician training. The second decision was to assemble a task group with relevant areas of expertise across the organization, including 15 leaders from the following departments: (1) clinical; (2) research and clinical outcomes; (3) care delivery; (4) compliance; (5) utilization review; (6) engineering, product, and design; and (7) growth strategy. The third decision involved appointing a project manager with a background in clinical research to lead the task group and oversee the implementation from start to finish.

In determining the measurement set, we selected standardized measures appropriate for a younger population, targeting the most commonly experienced symptoms among our clients. We also prioritized measures that were brief, free of cost, easily understood and explained, and available in the public domain [31]. In collaboration with the chief clinical officer and cofounder, 2 clinical directors, and the research and clinical outcomes team, we chose the following instruments: the (1) WHO-5 (psychological well-being) [32]; (2) PROMIS (Patient-Reported Outcomes Measurement Information System) Pediatric Peer Relationships Scale [36]; (3) PROMIS Pediatric Family Relationships Scale [37]; (4) PROMIS Pediatric Depressive Symptoms Short Form [38]; and (5) PROMIS Pediatric Anxiety Symptoms Short Form [38]. To cater to the diverse age groups within our client base, we opted for the pediatric versions of the 4 PROMIS scales for adolescents (<18 years) and the adult versions for young adult clients (≥18 years).

Given that the total number of items across the 5 measures was 33, resulting in a lengthy and potentially burdensome weekly survey, we split the measures into 2 different surveys administered on alternating weeks. The WHO-5 and Peer and Family Relationships Scales (survey 1) were administered on odd weeks, while the depression and anxiety measures (survey 2) were administered on even weeks.

Next, in collaboration with our engineering, product, and clinical teams, technical and workflow components were established. Surveys were administered using Qualtrics (Qualtrics, LLC) alongside an internal solution for real-time display of responses within sessions. For each measure, 2 graphs were generated—a sum score graph and an individual items graph—to showcase historical data, encompassing all previous weeks, alongside the most recent set of scores. A set of sample graphs for the WHO-5 scale is included in Figure 1. Graphs were designed to be intuitive and easy to follow, so clinicians and clients could view and discuss the results on their screen during the session and collaboratively make decisions based on the data.

In pilot study 1, we selected 2 clinical cohorts totaling 27 clinicians, each working with around 20 to 25 clients. This resulted in approximately 600 clients in the study at any given time. As mentioned, clinical cohorts were divided by region; the 2 clinical cohorts chosen for this study were selected to allow for the broadest geographical representation of clients.

The inclusion of the clinical directors from both clinical cohorts in the task group proved pivotal during the planning phase, as they provided valuable insights into their respective team cultures and practices (CFIR culture construct). To further understand clinician attitudes and inform the training approach, we conducted the 15-item Evidence-Based Practice Attitudes Scale (EBPAS-15) [39] before developing the trainings. This 15-item tool assesses clinician attitudes regarding the implementation of EBPs in mental health practice. Along with a total score, the tool yields four domain scores: (1) appeal or the extent to which a clinician may adopt an EBP because they find it appealing or if trusted peers and colleagues are using the intervention; (2) requirements or the likelihood of adopting the EBP given the requirements to do so; (3) openness or the extent to which the clinician is open to new clinical interventions; and (4) divergence or the clinician’s perception of the degree to which the novel EBP differs from their current clinical practice. Items are rated on a Likert scale from 0 (not at all) to 4 (very great extent).

EBPAS-15 scores for this group of clinicians (Table 1) were 2.8 (moderate extent) in the appeal domain; 2.9 (moderate extent) in the requirements domain; 3.0 (great extent) in the openness domain; and 0.7 (not at all) in the divergence domain. Average domain scores revealed similar attitudes toward the implementation of EBPs between this cohort of clinicians and previously published samples of clinicians [40,41]. Internal consistency (Cronbach α) for the EBPAS-15 global scale was good (α=0.81) and subscales ranged from acceptable to excellent (required=0.98; appeal=0.88; openness=0.85; and divergence=0.73). These values indicate similar or slightly stronger internal consistency than in previously reported scores, such as globally α=0.77, 0.79 [40], and 0.82 [41].

The assessment revealed positive attitudes toward implementing EBPs, which lent itself well to the implementation of MBC with this group of clinicians. Given that the openness domain score averaged 3.0, indicating a high level of receptiveness to new clinical interventions, we opted to present the trainings as opportunities to acquire valuable new clinical skills, rather than emphasizing their mandatory nature.

Before the start of MBC training, the clinical directors emphasized the importance of socializing MBC to the team, aiming to foster enthusiasm and create a positive mindset for clinicians going into the training series. To do this, our cofounder and chief clinical officer attended weekly team meetings 2 weeks before the trainings began, providing a preview and building excitement for the upcoming training series.

The training spanned 4 hours, divided into two 2-hour sessions, and was facilitated live by the external SME. Mandatory attendance for clinicians was closely monitored by the clinician experience team. Both segments of the training, part 1 and part 2, were recorded for future reference and for those unable to attend either portion of the training. Individuals who did not attend the live training were assigned to watch the recording before the official launch date. Part 1 of the training covered the first 2 components of the MBC process, namely “collect” and “share.” Part 2 covered the last component, “act,” along with the technical workflow. Both sessions included lecture segments, interactive group polls and quizzes, practice and role-plays in smaller groups, and opportunities for questions and answers.

Clinicians were trained at the end of the MBC core trainings to execute four main steps: (1) at the start of each session, send the client their designated MBC survey link (survey 1 or survey 2, depending on the treatment week) for private completion on their own screens (“collect”); (2) upon the client’s survey submission, access the data display interface and load the client’s graphs; (3) share the graphs with the client during the session, using the screenshare feature on Zoom, and discuss the various components (“share”); and (4) determine next steps in collaboration with the client and document the conversation, along with any actions or adjustments to the treatment plan influenced by the MBC scores in the client chart after the session (“act”). A key point of emphasis throughout the training was the importance of elevating the client’s voice throughout the MBC process, particularly in the “share” and “act” steps. This involved gaining a deep understanding of the client’s perspectives on their own data, ensuring their insights and experiences were integral to decision-making.

Once clinicians were trained and ready to start using MBC in their individual sessions, pilot study 1 was officially launched on February 13, 2023.

Given that a major goal of this study was collecting and implementing feedback to refine the process before its widespread implementation across all care teams, consistent feedback collection played a significant role in the months following the launch (CFIR reflecting and evaluating construct). This regular and reciprocal feedback was a key driver of the evaluation and reflection process, as well as the adaptation process (CFIR innovation adaptability construct).

Feedback was obtained in 2 main ways. First, the clinical directors and supervisors of each clinical cohort, who met with each clinician on their team on a weekly basis, would check-in with each clinician regarding their experience with MBC, including any successes, challenges, and opportunities for improvement. These clinical leaders would then convey this feedback in weekly task group meetings, allowing the team to discuss possible changes and iterations. Second, and perhaps most impactfully, the project manager met with the entire pilot cohort on a monthly basis to facilitate open dialogue regarding the pilot. In each of these feedback sessions, quantitative and qualitative feedback was collected through a short survey at the start of the session and key points from the group conversations were noted for further examination. Since each clinical cohort already held weekly team meetings, no additional sessions needed scheduling, making it convenient for all pilot clinicians to participate.

Several data sources were analyzed to determine implementation success and outcomes.

To apply the learnings from study 1 before initiating widespread implementation across all care teams, a second study was initiated with a new clinical cohort, which officially launched on September 25, 2023. Most of the methods from study 1 were replicated, with key modifications outlined in Table 2 (key learnings 1-4). Notably, clinicians in study 2 used the updated measurement set and administered the same survey on a weekly basis.

A single clinical cohort was chosen, encompassing 25 clinicians led by 1 clinical director and 2 supervisors. This cohort, the largest of the clinical teams, was chosen to capture the largest client population possible.

EBPAS-15 data were once again collected to assess clinician attitudes going into study 2 and design the trainings accordingly. Average domain scores for this cohort were similar to scores from the first cohort of clinicians in study 1, indicating consistent attitudes toward EBP implementation in both groups (Table 1). Internal consistency (Cronbach α) for the EBPAS-15 global scale was good (α=0.81) and subscales ranged from acceptable to excellent (required=0.97; appeal=0.79; and openness=0.85) with somewhat lower reliability estimates for divergence (α=0.44).

The second clinician cohort received 4 hours of live training from the external SME, with adjustments, including training on optimized measures and survey cadence, live consultation for common concerns regarding MBC, and provision of MBC companion materials immediately after training. Given that the updated training also included content from the refresher training conducted with study 1’s clinicians, no additional refresher trainings by the SME were necessary. In addition, 3 individuals were identified as “MBC champions”—key leaders and MBC enthusiasts—from the first study (CFIR innovation recipients construct). Champions described their experiences with MBC at the start of the training, including stories of their clients who greatly benefitted from using MBC during treatment, to facilitate positive peer perspectives and generate enthusiasm among the new clinician group. In support of key learning 4 (Table 2), the desire for ongoing MBC consultation, MBC champions were also available to the new clinician group for any ongoing consultation needs. The new clinicians were able to access support from the MBC champions as needed throughout the duration of study 2, generally by contacting them directly for peer-to-peer support.

The same metrics and data analysis methods as those used in study 1 were used in study 2 to determine implementation success and outcomes, including survey completion rate, the monthly clinician feedback survey, and client outcomes.

Four major learnings emerged from study 1, with relevant changes implemented in the months following the launch. Notably, we learned the following: (1) the measurement set needed further refinements, (2) the survey cadence needed simplification, (3) clinicians required support resources, and (4) clinicians required ongoing training and case consultation. All key learnings (detailed in Table 2) came from the live feedback session discussions and qualitative questions on the monthly feedback survey.

The average survey completion rate was 80.08% (3216/4016) in the 4 months following the launch—in other words, an MBC survey was completed in around 80% of therapy sessions. From the monthly feedback survey, the average score for each quantitative item (shown in Table 3) indicated neutral attitudes toward MBC, with average responses in the “neither agree nor disagree” category. These scores were recorded approximately 1 month after the launch. Qualitative data from the survey were promising, with sample clinician quotes organized by theme in Textbox 1.

The final client dataset consisted of 3723 clients who had all been discharged, with admission dates between February 13, 2023 (the launch date for study 1), and December 31, 2023. Out of 3723 clients, for the 2 comparison groups, 1576 (42.3%) clients belonged to the MBC group, while 2147 (57.3%) clients belonged to the TAU group. The ages of this sample ranged from 10 to 33 years, with an average age of 19.4 years. Around 52.99% (1973/3723) of clients identified as female, 24.98% (930/3723) identified as male, and 13.98% (520/3723) identified as a gender minority group.

In comparing routine discharge rates of MBC versus TAU clients, we found that MBC clients had a 22% higher likelihood of discharging due to successful treatment completion, a statistically significant finding (P<.001).

In addition, in all 3 outcome measures (PHQ-9, GAD-7, and WHO-5), we observed statistically significant differences in symptom reduction between MBC and TAU clients (detailed in Table 4). MBC clients experienced statistically significantly greater improvements in symptoms by discharge. However, though these results hold statistical significance, they do not appear to represent clinically significant symptom change, as both the MBC and TAU groups had average intake and discharge scores within the same clinical categories for each measure (eg, for the PHQ-9, both groups averaged “moderate” depression at intake and “mild” depression at discharge).

Clients with missing data for any outcome measure (PHQ-9, GAD-7, and WHO-5) were excluded from the respective measure’s analysis but included in analyses for other measures if their data were available.

Study 2 clinicians shared fewer overarching concerns about the implementation of MBC and instead provided feedback to support smaller adjustments. Three key learnings arose and are included in Table 2 (key learnings 5-7). Notably, (1) some language on the survey needed improvement, (2) specific aspects of the data display graphs needed optimization, and (3) a more streamlined communication channel for all-things MBC was needed.

The average survey completion rate for study 2 was 86.01% (4218/4904) in the 3 months following launch. As for the quantitative items on the monthly feedback survey, average scores for study 2 clinicians improved in comparison to study 1 clinician scores, with average scores for almost all questions falling in the “agree” category (Table 3). Representative quotes from the qualitative data collected in study 2 are available in Textbox 1.

The client outcomes analysis for study 2 was done with a much smaller sample size, given that fewer clients had been discharged since the launch of study 2, just 4.5 months before the analysis. The final dataset consisted of 405 clients who had all been discharged, with admission dates between September 25, 2023 (launch date of study 2), and February 1, 2024. Out of 405 clients, for the 2 comparison groups, 315 (77.8%) clients belonged to the MBC group, while 90 (22.2%) clients belonged to the same clinical team but did not participate in MBC (TAU group). The ages of this sample ranged from 10 to 34 years, with an average age of 17.5 years. Around 51.1% (207/405) of the clients identified as female, 20% (81/405) identified as male, and 15.1% (61/405) identified as a gender minority group.

In comparing routine discharge rates of MBC versus TAU clients, the MBC clients showed a 29% higher likelihood of discharging because of successful treatment completion than TAU clients, a statistically significant finding (P<.001).

Score improvements from intake to discharge for the PHQ-9, GAD-7, and WHO-5 (Table 4) were statistically significantly greater for MBC clients in the GAD-7 and WHO-5. The PHQ-9 score difference bordered on statistical significance. Similar to study 1, the results do not appear to represent clinically significant symptom change, as both the MBC and TAU groups had average intake and discharge scores within the same clinical categories for each measure.

This study sought to add to the literature by documenting the design and deployment of an MBC QI initiative within a youth-serving, remote IOP. Recognizing the large size of our clinical workforce and the number of clients served, conducting pilot testing before any clinical implementation is crucial to gather learnings and evaluate success. Thus, we conducted 2 pilot studies with several clinical cohorts, aimed at understanding and adapting the MBC process to best fit the needs of our clinical population before a full-scale rollout. In both studies, several critical iterations were made to streamline and improve the process as much as possible. The increasingly positive clinician feedback ultimately led to our confidence in planning a full-scale clinical rollout following the conclusion of study 2.

Across both studies, results supported the feasibility and clinician acceptability of a rigorous MBC process in a youth-serving, remote-first intermediate care setting.

While ideally surveys would be completed in every session, achieving a 100% completion rate is often impractical due to the acuity of the client population, where crisis de-escalation may prevent survey completion. Although no universal benchmark exists for satisfactory MBC survey completion rates in IOP settings, the literature suggests that ≥80% indicates success [45]. Both study cohorts exceeded this benchmark, suggesting strong clinician buy-in and adherence to the “collect” portion of the model. The nearly 5% improvement in completion rate in study 2 possibly stems from refinements made during study 1.

We designed the monthly feedback survey to track clinician attitudes toward MBC after the launch. Comparing week 4 of study 1 to week 4 of study 2, we saw higher scores in the second cohort, indicating more positive attitudes and perceptions at the same relative time point. This improvement likely also results from the iterative changes made during study 1, which helped to streamline the MBC process. Pilot clinicians also seemed to appreciate the task group’s responsiveness to their feedback, highlighting the value of an iterative approach and ongoing provider feedback in any clinical implementation.

The significant increase in routine discharge rate for the MBC groups in both studies (22% and 29%) was the most promising finding, suggesting that MBC is positively correlated with increased treatment completion. The PHQ-9, GAD-7, and WHO-5 outcomes were also promising, suggesting that MBC may be positively correlated with symptom reduction. However, despite the statistical significance of these results, clinical significance was not observed. Since this was a series of pilots, we were aiming for high-quality implementation and not necessarily outcome improvement, but the statistical significance observed in these pilots was promising and we hope to see clinical significance at full launch.

Overall, these results suggest that MBC may be more effective than usual practice. Our work aligns with previous literature, where meta-analyses have discussed that MBC leads to faster client improvement, greater symptom reduction, and lower dropout rates [17,24]. While we did not examine treatment dropout rates, we did see promising results regarding treatment completion rates and symptom reduction. A key distinction is that while previous studies largely focused on traditional in-person individual psychotherapy [46,47], our research was conducted within the context of a remote IOP, specifically geared toward youth.