This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Youth and young adults face barriers to mental health care, including a shortage of programs that accept youth and a lack of developmentally sensitive programming among those that do. This shortage, along with the associated geographically limited options, has contributed to the health disparities experienced by youth in general and by those with higher acuity mental health needs in particular. Although intensive outpatient programs can be an effective option for youth with more complex mental health needs, place-based intensive outpatient programming locations are still limited to clients who have the ability to travel to the clinical setting several days per week.
The objective of the analysis reported here was to assess changes in depression between intake and discharge among youth and young adults diagnosed with depression attending remote intensive outpatient programming treatment. Analysis of outcomes and the application of findings to programmatic decisions are regular parts of ongoing quality improvement efforts of the program whose results are reported here.
Outcomes data are collected for all clients at intake and discharge. The Patient Health Questionnaire (PHQ) adapted for adolescents is used to measure depression, with changes between intake and discharge regularly assessed for quality improvement purposes using repeated measures
Clients ranged in age from 11 to 25 years, with an average of 16 years. Almost one-quarter (23%) identified as nongender binary and 60% identified as members of the lesbian, gay, bisexual, transgender, queer (LGBTQ+) community. Significant decreases (mean difference –6.06) were seen in depression between intake and discharge (
Findings support the use of remote intensive outpatient programming to treat depression among youth and young adults, suggesting that it may be a modality that is an effective alternative to place-based mental health treatment. Additionally, findings suggest that the remote intensive outpatient program model may be an effective treatment approach for youth from marginalized groups defined by gender and sexual orientation. This is important given that youth from these groups tend to have poorer outcomes and greater barriers to treatment compared to cisgender, heterosexual youth.
Approximately 3 million youths were diagnosed with major depressive disorder (MDD) in 2019 [
IOP is an alternative model to inpatient care for youth with needs greater than those that can be served in traditional outpatient services. Youth attending IOP spend upwards of 15 hours per week in individual, family, and group treatment sessions with a multidisciplinary team of professionals, including mental health clinicians (psychiatrists, psychologists, counselors, and social workers), medical professionals (nurses and dietitians), and specialists trained in a variety of therapies such as art and recreational therapy [
Although the IOP model can be an effective alternative to inpatient care for youth, place-based IOP locations are still limited to clients who live close enough to travel to the clinical setting several days per week. This can be particularly problematic for adolescents who depend on adults to drive them or provide the resources to take public transportation (bus or train fare) to the IOP location, assuming that public transportation is available. Furthermore, even shorter distances coupled with the expected frequency of attendance, can become burdensome on parents or caregivers and have negative impacts on the ability of parents to work and care for other children. Thus, adherence to IOP may be compromised.
Such geographic limitations restrict both the availability of services to youth living far from treatment settings as well as the degree to which treatment can be targeted to particular developmental and clinical needs, given that treatment groups by necessity reflect the conditions of the majority of the population in the particular geographic location. For example, a place-based IOP may not have enough similarly presenting clients to create a treatment group specifically for younger adolescents seeking treatment for eating disorders who identify as nongender conforming if most of the clients in that geographic area are older adolescents seeking treatment for anxiety who identify as cisgender.
Remote IOPs are intensive outpatient programs that conduct 100% of their services over Health Insurance Portability Accountability Act (HIPAA)–compliant video software, with patients and families typically attending sessions from home or school. Similar to place-based IOPs, clients attend 3-4 hours of treatment per day, 3-5 days per week. The primary benefit of remote IOP is the removal of geographic barriers inherent to place-based programs, which increases access to care for adolescents and young adults as well as allowing for the provision of more appropriate treatments specific to individual client needs. Because remote IOP can draw clients from an unlimited distance, it affords the opportunity to provide developmentally targeted treatment for youth at various points on the adolescent or young adult spectrum and specialized treatment for those whose demographic or clinical characteristics are unique. Due to its remote modality, populating a treatment group to the ideal therapeutic size is not dependent on the number of clients at the same stage of development with similar diagnoses, symptoms, and identities living in the same geographic area.
Previous research suggests that remote and internet-based interventions are effective in treating a variety of clinical diagnoses, with effect sizes equivalent to those reported for place-based or traditional face-to-face therapy [
The aim of this report is to present the findings of quality improvement (QI) analysis using data collected by program staff of a remote intensive outpatient program serving youth and young adults. The QI analysis presented here focused on changes in depression among clients diagnosed with depressive disorder to assess both overall treatment effectiveness and differences by subgroups, including those defined by gender and sexual orientation. The goal of the program’s ongoing QI efforts is to use outcomes data to inform clinical and programmatic decisions that will enable the program to better serve the needs of all program youth.
Outcomes data are regularly collected as a part of the program’s ongoing QI procedures and are reported both monthly and quarterly to their payors and providers. The data are also used to inform program and clinical changes to enhance the delivery of effective services. Data collected as part of the QI process were used to perform a repeated-measures, longitudinal assessment of changes in depression as measured by the Patient Health Questionnaire Modified for Adolescents (PHQ-A) at intake, discharge, and follow-up. Analyses tested changes in depression among all clients with depressive disorder and among subgroups. Threats to the validity of findings due to history and spontaneous remission were also explored.
This project was reviewed and determined by the University of Pennsylvania Institutional Review Board to qualify as QI, indicating that these activities are not human subject research.
The intensive outpatient program,
The primary diagnoses served by the program include anxiety, depression, posttraumatic stress disorder, and bipolar disorder, with secondary diagnoses that include severe emotional disturbance, substance use disorders, and eating disorders. The program specializes in complex cases, including youth with severe trauma, neglect, juvenile justice, or foster care experience; gender and sexual minoritized youth; youth who reside in rural or native communities; and youth who have had multiple admissions to inpatient or hospital settings in the previous year.
A biopsychosocial assessment is completed for each client at intake after which they are placed in treatment tracks that include groups and therapeutic approaches identified as best practices for their developmental stage and specific diagnostic, identity, and behavioral health needs. Developmentally determined groups include those targeted to youth in early adolescence, middle adolescence, late adolescence, young adulthood, and emerging adulthood. Identity focused tracks and groups include those targeted to youth from minoritized gender and sexual orientation populations. Treatment modalities include variations of cognitive behavioral therapy, dialectical behavior therapy, and trauma-focused treatment. Client enrollment is ongoing and new clients may be placed into newly developed or already established groups. Additionally, because adjustments are made throughout the treatment process, clients may begin in one track but be reassigned to new groups based on their response to treatment.
Group therapies are provided in the following three 50-minute blocks, 3 days per week: evidence-based skill building interventions (ie, dialectical behavior therapy and cognitive behavioral therapy), general therapeutic processing, and experiential therapy (ie, art, music, and journaling). Individual and family therapy groups are also offered each week with masters-level, licensed clinicians. Parents and others involved in the client’s care are provided psychoeducation and mutual aid groups that deliver support and guidance in such topics as mindful communication and sibling support. Parents and caregivers are also provided weekly “IOP Roadmap(s),” which impart information on the skills their children are learning with tips on how to support the practice of those skills at home.
Data for clients who were in treatment between June 2021 and October 2022 were reviewed for inclusion. Those clients with a diagnosis of depressive disorder who also passed engagement and completion criteria, defined as a minimum of 18 hours of treatment (7 sessions) and at least two weeks in care, were included. The cutoff at 18 hours was informed by research on neurological indicators of brain changes in response to cognitive and behavioral therapies [
Cases in which the client did not meet this minimum standard of engagement were not included. Clients who did not complete treatment (discharged due to lack of engagement or insurance denial, transfer to lower or higher level of care, left against clinical advice, etc) but were deemed as having passed the engagement criteria (at least 18 hours of IOP sessions) were included despite discharge status. Cases in which clients were absent from group for extended lengths of time (ie, 2-4 weeks), regardless of the reason (vacation, admission to higher level of care, disengaged, etc) were also included as long as they either met the minimum level of engagement prior to departure or, upon their return to treatment, attained a level of engagement that passed engagement criteria.
A majority of the clients included in the analysis lived in Washington State (n=303), Texas (n=163), Arizona (n=106), Montana (n=88), California (n=58), Idaho (n=55), and Pennsylvania (n=45). A smaller number were from Delaware (n=27), Illinois (n=21), Utah (n=18), New York (n=14), New Jersey (n=12), and Florida (n=10). Five or fewer clients lived in each of the following: Colorado, Wyoming, New Mexico, Ohio, Alabama, Georgia, Kentucky, Michigan, Nevada, Indiana, Louisiana, Minnesota, and North Carolina.
Baseline measures of depression are collected by IOP staff after intake and prior to the first group session. Upon joining their first group session, clients are moved to a breakout room within the video platform where they meet with an outcomes coordinator who shares the link to the web-based intake survey with them. If a client is unable to open the link within the platform, they are sent the link via email or text messaging. The process, repeated for the discharge survey, is scheduled to occur at each client’s final group session.
If clients are unable to complete the survey prior to discharge, they are sent the link via email or text messaging and offered a US $25 Amazon gift card for their time. The decision to offer a gift card was made in recognition of the time and effort the program was asking of the clients in completing the survey on their own time, outside of program hours. The amount offered is within ethical standards that would not be considered to be unduly influential [
IOP staff download the completed survey responses at the end of each month for internal reports and to share with the analysis team. The data are deidentified by IOP staff prior to uploading the file to a secure folder shared with the analysis team by removing names and assigning unique ID numbers.
Demographic characteristics are collected from respondents, including age, gender identity, and sexual orientation. Treatment characteristics are collected from administrative records and include diagnoses, intake and discharge dates, number of weeks in treatment, total number of sessions attended, and discharge status (ie, completed treatment, administrative discharge, transferred to a lower or higher level of care, or left against clinical advice).
The PHQ-A is used to assess depression prior to and after treatment. PHQ-A is a self-report measure that uses a 9-item depression severity scale asking how bothered respondents have been in the past 2 weeks by symptoms such as “feeling down, depressed, irritable, or hopeless” and having “thoughts that you would be better off dead, or of hurting yourself in some way.” Response options range from 0 (not at all) to 3 (nearly every day). The PHQ-A has been found to have good diagnostic validity [
Descriptive statistics were used to summarize the client group as a whole. Means, SDs, minimums, and maximums were calculated for the following continuous variables: age, PHQ-A scores at intake, number of weeks, and treatment sessions attended. Frequencies were computed for the following categorical variables: comorbidity, symptom severity at intake, gender identity, sexual orientation, and discharge status.
To test the overall effectiveness of the remote intensive outpatient program to reduce depression, overall PHQ-A scores at intake and discharge were used as repeated measures and to calculate difference scores. Symptom severity scores based on overall PHQ-A scores at intake and discharge were used to identify movement across the clinical threshold associated with MDD. These 2 metrics, difference scores and clinical thresholds, are considered indices of clinically meaningful change as defined and studied by Wolpert et al [
Changes in depression scores between intake and discharge were analyzed employing a repeated-measures
One-way ANOVA was used to compare differences in treatment effects among self-reported gender and sexual orientation groups. Difference scores were calculated for use as the dependent variable by subtracting PHQ-A intake scores from PHQ-A discharge scores so that treatment gains would be demonstrated as negative scores, reflecting a reduction in depression.
Because adolescence and young adulthood encompass several developmental periods [
To test for possible effects of history, particularly those from seasonal differences, that could explain changes in depression between intake and discharge, an ANOVA was conducted that compared mean change scores by season of intake. To separate treatment experience into seasons, clients’ month of intake were used to assign them to 1 of 4 seasons as defined by the National Geographic Society [
A total of 1062 cases met inclusion criteria and were included in analyses. The average treatment engagement lasted 10 (SD 4.12) weeks with a mean of 27 (SD 12.03) sessions. The average age of the clients was 16.2 (SD 3.3) years with a range of 11-25 years. As shown in
Based on inclusion criteria, all clients had a diagnosis of depression and as a group entered treatment with an average depression score of 14.90 (SD 7.13). Three quarters of the clients (73%) entered treatment experiencing moderate to severe symptoms, 22% entered with minimal to mild symptoms, and 2% reported no symptoms (see
Client characteristics at intake.
|
Participants, n (%) | ||
|
|||
|
Female | 473 (44.5) | |
|
Gender-fluid | 69 (6.5) | |
|
Gender-neutral | 15 (1.4) | |
|
Gender-questioning | 25 (2.4) | |
|
Genderqueer | 15 (1.4) | |
|
Male | 261 (24.6) | |
|
Nonconforming | 19 (1.8) | |
|
Nonbinary | 104 (9.8) | |
|
Missing | 81 (7.6) | |
|
|||
|
Asexual or graysexuality | 49 (4.6) | |
|
Bisexual | 231 (21.8) | |
|
Gay | 34 (3.2) | |
|
Heterosexual or straight | 319 (30) | |
|
Lesbian | 55 (5.2) | |
|
Pansexual | 164 (15.4) | |
|
Queer | 58 (5.5) | |
|
Questioning | 53 (5) | |
|
Missing | 99 (9.3) |
Severity of depression symptoms at intake.
|
Participants, n (%) |
None to minimal | 103 (9.7) |
Mild | 151 (14.2) |
Moderate | 212 (20.0) |
Moderately severe | 255 (24.0) |
Severe | 309 (29.1) |
Missing | 32 (3.0) |
As summarized in
Analysis of differences in depression between intake and discharge.
|
Mean (SD) | SE mean | Mean difference | Cohen |
|||||||||
|
–6.06 | –24.68 (967) | <.001 | –0.79 | |||||||||
|
Intake | 15.01 (7.11) | 0.23 |
|
|
|
|
||||||
|
Discharge | 8.95 (6.60) | 0.21 |
|
|
|
|
Crossing the clinical threshold using symptom severity at intake versus discharge.a
|
Discharge, n | Total, n | |||||
|
None to mild | Moderate to severe |
|
||||
|
|||||||
|
None to mild | 184 | 48 | 232 | |||
|
Moderate to severe | 388 | 348 | 736 | |||
Total | 572 | 396 | 968 |
aMcNemar 2 (N=968):
There were no significant differences found in analyses of change scores among subgroups of gender (
Change scores were significantly different among clients by the season in which they started CH (
One-way ANOVA of change scores between intake and discharge.
|
Mean (SD) | Minimum | Maximum | Participants, n |
Winter | –5.90 (8.47) | –27 | 23 | 183 |
Spring | –7.46a (7.47) | –26 | 17 | 304 |
Summer | –5.06a (7.18) | –27 | 24 | 354 |
Fall | –5.70 (7.55) | –23 | 13 | 127 |
June 2021 | –5.38 (6.70) | –19 | 3 | 13 |
July 2021 | –4.61 (8.00) | –27 | 9 | 28 |
August 2021 | 0.10 (7.19) | –10 | 24 | 30 |
September 2021 | –4.39 (8.89) | –23 | 13 | 33 |
October 2021 | –6.26b (7.49) | –22 | 8 | 43 |
November 2021 | –6.02 (6.63) | –19 | 7 | 49 |
December 2021 | –2.86 (9.79) | –27 | 23 | 43 |
January 2022 | –6.26b (9.12) | –25 | 18 | 58 |
February 2022 | –7.23b (6.79) | –21 | 8 | 82 |
March 2022 | –7.34b (8.63) | –26 | 17 | 83 |
April 2022 | –6.72b (6.93) | –25 | 9 | 95 |
May 2022 | –8.10b (7.03) | –25 | 8 | 126 |
June 2022 | –5.68b (7.24) | –24 | 12 | 137 |
July 2022 | –5.91b (7.06) | –24 | 17 | 94 |
August 2022 | –5.00 (5.90) | –17 | 10 | 52 |
September 2022 | –7.50 (10.61) | –15 | 0 | 2 |
aSignificantly different.
bSignificantly different than August 2021.
Analysis of differences in depression between discharge and 3-month follow-up.
Depression score | Mean (SD) | SE mean | Mean difference | ||||||
|
–4.61 | –6.64 (128) | <.001 | ||||||
|
Intake | 14.66 (7.32) | 0.65 |
|
|
|
|||
|
Discharge | 10.05 (6.50) | 0.57 |
|
|
|
|||
|
–0.09 | –0.16 (128) | .88 | ||||||
|
Discharge | 10.05 (6.50) | 0.57 |
|
|
|
|||
|
3-month follow-up | 9.95 (6.86) | 0.60 |
|
|
|
The aim of the QI analysis undertaken here was to assess the effectiveness of a remote intensive outpatient program to treat depression in adolescents and young adults. Program effectiveness was supported such that depression was lower at discharge compared to intake as tested by significant changes in both PHQ-A scores and symptom severity. This aligns with and provides additional support for previous research suggesting that mental health needs can be effectively met using remote treatment models [
Subgroup comparisons provide support for remote IOP as a treatment option that is equally effective with adolescents and young adults who identify across gender and sexual orientation populations. This is important given that marginalized populations, including those who do not identify in the gender binary and those who select as members of LGBTQ+ communities, are at higher risk of mental health disorders due to experiences of marginalization [
The equivalent treatment effects determined across subgroups are likely the result of both program factors that are common across all clients as well as those that address the unique needs of individuals. All clients experience the same treatment platform (remote) and model (intensive outpatient program) that increase their access to intensive services while lowering burdens associated with travel time and cost. At the same time, individual treatment plans are specific to the developmental, psychological, and psychosocial needs of each client, meaning that each client gets to experience treatment groups designed to address their unique intersection of needs related to diagnoses, symptoms, developmental period (age), and identity (gender and sexual orientation).
One threat to the validity of findings is history or exposure to factors outside of the IOP treatment process that could have reduced depression between intake and discharge [
In a clinical setting, spontaneous remission is the tendency for symptoms, regardless of severity, to improve over time independent of treatment [
Since the COVID-19 pandemic, telemental health care treatment options have become far more acceptable and viable. Although they do not alleviate all disparities, there are certain disparities that they can reduce. The remote platform decreases younger clients’ dependency on others, such as parents for transport or the availability of public transportation for their attendance.
Providing treatment remotely in the manner described here may have also attracted and helped engage LGBTQ+ youth. Previous research has found that LGBTQ+ youth are more likely than heterosexual cisgender youth to participate in web-based treatment [
Such services may also be easier to establish parent involvement due to decreased time commitment required without the travel to a physical location. Additionally, such involvement and the ability to pull from a wide geographical range makes it feasible for programs to hold targeted parent groups, such as the LGBTQ+ parent group offered by this program. These factors may have contributed to the effectiveness of this service for LGBTQ+ youth.
This treatment modality can also serve a wide range of ages without necessity for establishing separate facilities or specialized units for youth and young adults that require drawing on a broad geographical area. Consequently, it is far more feasible to develop groups remotely with youth who have similar developmental and demographic needs, as again these unique groups are not constrained by geographical location which by its very nature may not have the needed diversity. Given the comfort and technical savvy common in youth and young adults, this treatment modality has the potential to not only be a comfortable treatment alternative but one that is more attractive because it is tech-heavy.
A primary strength of this analysis is the use of outcomes data collected as part of program administration by program staff, which increases the likelihood that findings reflect the actual remote IOP process uninfluenced by the presence of research staff and additional resources frequently infused into organizations during the research process. The presence of research staff during the investigation of controlled interventions has implications for fidelity and the ability of an intervention to continue to engender the same treatment effects once organizations are left to implement interventions on their own [
The gold standard in mental health treatment is personalized treatment to address each client’s unique intersection of mental health and developmental and psychosocial needs. This makes good mental health treatment inherently heterogeneous. The value that QI analysis brings to the field is the degree to which outcomes reflect what is happening in a real-world treatment setting. QI cannot have the precision of a randomized controlled trial as delivered by a research team, because it is based on clinical and administrative data, but it nonetheless extends the knowledge base.
Another strength of this analysis is the use of 2 metrics to assess clinically meaningful change [
A primary limitation of this study is the lack of a comparison group with which to compare those in treatment to assure that changes over time were not due to factors other than or outside of the treatment experience. To account for this limitation, the following 2 threats to validity of the findings were explored in analyses: history effects and spontaneous remission.
These analyses help to fill the gap in the literature regarding the use of remote mental health services (telehealth) to treat adolescents and young adults with more complex mental health needs by providing evidence for the ability of remote IOP to reduce depression in high acuity youth without hospitalization and with treatment effects that remained stable for up to 3 months post discharge. The metrics used to assess treatment effects reflect both changes in clinical diagnoses as well as changes in real-world symptoms, including changes in sleeping and eating patterns, the ability to concentrate, and having interest or taking pleasure in everyday activities, all of which have direct implications for the physical and mental well-being of clients in everyday life.
This study also provides evidence for the ability of remote intensive outpatient program to effectively treat those youth who identify as members of marginalized gender and sexual orientation populations. The opportunity to work outside of the limitations of place and local demographics allows clinicians to create specialized treatment groups sensitive to identity-specific needs and, given the disproportionately high percentage of clients who identified as a member of an LGBTQ+ community, may provide a safer space to self-disclose than those based in local facilities. The remote intensive outpatient program model as delivered here could move the field forward in providing more opportunities and safer spaces for youth from LGBTQ+ communities to receive identity-sensitive treatment and in doing so address the disparities that lead to the poorer outcomes and greater treatment barriers experienced by LGBTQ+ youth compared to their cisgender, heterosexual peers.
Finally, providing evidence for the effectiveness of remote IOP engenders support for another level within the spectrum of care that allows patients whose needs go beyond that which can be addressed in regular outpatient settings to receive treatment in their homes rather than inpatient treatment settings or hospitalization. This in turn enables youth to receive a higher level of intensive treatment without removing them from their everyday supports and the healthy, normative activities that are developmentally essential during adolescence and young adulthood.
Charlie Health, Inc
intensive outpatient programming
lesbian, gay, bisexual, transgender, queer
major depressive disorder
Patient Health Questionnaire
Patient Health Questionnaire Modified for Adolescents
quality improvement
This quality improvement initiative was funded by Charlie Health, Inc.
The data used for this report were collected by program staff as part of ongoing quality improvement efforts and are therefore not available for public use.
MEC was involved in conceptualizations, formal analysis, investigation, methodology, supervision, and the writing of the original draft. PS was responsible for conceptualizations, supervision, investigation, and the writing of the original draft. BP performed data curation, project administration, writing, review, and editing. RK carried out data curation, writing, review, and editing. CF was responsible for funding acquisition, resources, writing, review, and editing.
MEC and RK report consulting fees from Charlie Health. PS has no disclosures to report. At the time of writing, BP was an employee of Charlie Health. CF is a founder and Chief Clinical Officer of Charlie Health.