This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Existing mental health treatments are insufficient for addressing mental health needs at scale, particularly for teenagers, who now seek mental health information and support on the web. Single-session interventions (SSIs) may be particularly well suited for dissemination as embedded web-based support options that are easily accessible on popular social platforms.
We aimed to evaluate the acceptability and effectiveness of three SSIs, each with a duration of 5 to 8 minutes (Project Action Brings Change, Project Stop Adolescent Violence Everywhere, and REFRAME)—embedded as Koko
We used quantitative data (ie, star ratings and SSI completion rates) to evaluate acceptability and short-term utility of all 3 SSIs. Paired 2-tailed
The SSIs were completed 6179 times between March 2021 and February 2022. All 3 SSIs generated high star ratings (>4 out of 5 stars), with high completion rates (approximately 25%-57%) relative to real-world completion rates among other digital self-help interventions. Paired 2-tailed
Very brief SSIs, when embedded within popular social platforms, are one promising and acceptable method for providing free, scalable, and potentially helpful mental health support on the web. Considering the unique barriers to mental health treatment access that many teenagers face, this approach may be especially useful for teenagers without access to other mental health supports.
Existing mental health treatments have long been inaccessible due to well-established structural (eg, cost, transportation, and time) and individual (eg, stigma and distrust of providers) barriers [
To fill the gap between the
Initial efforts suggest that it is
Despite the great
SSIs designed for these web-based, embedded contexts should be streamlined to improve engagement while retaining their therapeutic value. Once disseminated in the real world, digital interventions often encounter issues with user engagement (ie, low uptake and low completion) [
Several intervention design principles, drawn from basic research in social psychology, education, and marketing [
As a majority of existing mental health treatments remain inaccessible, many young people seek mental health support via social platforms on the web. SSIs are uniquely well positioned for integration within web-based social platforms, providing free, brief, and anonymous mental health support options at scale. However, little research has formally evaluated the acceptability and effectiveness of very brief SSIs designed for this context. This study adapted 3 web-based SSIs (5-8 minutes) that were offered as Koko minicourses on Tumblr, a microblogging and social networking website with 135 million active users monthly [
For this study, we collected anonymous data exclusively from individuals on Tumblr—all of whom were introduced to the service by either (1) clicking on a featured advertisement from Tumblr (eg, “take control by taking this mood-boosting minicourse”) or (2) direct referral from the platform. As all data were part of a completely anonymous program evaluation, this study was deemed as nonhuman subjects research in consultation with the institutional review board at Stony Brook University. In addition, Koko’s privacy policy and terms of service acknowledge that anonymized data may be shared for research purposes.
The direct referral pathway for each of the 3 SSIs was similar. Users who searched for mental health topics on Tumblr were shown an in-app overlay with links to various resources, such as The National Suicide Prevention Lifeline. A set of over 1300 keywords and their derivations were used to detect terms such as “self-harm” or “depression” as well as slang and obfuscations, such as “sewer-slide” and “'s3lf h@rm.” In addition to links to crisis lines, users were also sent a direct message from Koko through the Tumblr direct message channel. Specifically, they were sent the following automated message from a chatbot called “Kokobot”:
Hi! I’m Kokobot [wave emoji]. I’m working with Tumblr to connect people who are interested in mental health topics. Type “hi” to get started...
Next, users were onboarded to the service and asked to describe a recent negative situation that they have been facing, along with any associated negative thoughts. From there, a set of text-based classifiers for mental health [
The 3 SSIs were initially introduced as Koko minicourses at 3 separate times: March 2021 (Project ABC), June 2021 (REFRAME), and July 2021 (Project SAVE). Across all 3 SSIs, the data for this study were collected through February 2022. Preintervention data were collected immediately before beginning each SSI (ie, each Koko minicourse), and postintervention data were collected immediately following the completion of a program.
Project ABC minicourse was a briefer 5- to 8-minute version of the original 20- to 30-minute Project ABC SSI evaluated in earlier randomized trial research [
Before disseminating as a Koko minicourse, an abbreviated (8-minute) version of the Project SAVE SSI was adapted from an original 30-minute program [
The REFRAME SSI (5 minutes) teaches cognitive reappraisal, an emotion regulatory strategy that involves modifying one’s interpretation of stressful situations [
Demographic information (age, gender, and race and ethnicity) was collected as part of the preintervention measures for individuals completing Project SAVE.
The Beck Hopelessness Scale-4 is a brief and reliable measure used to assess hopelessness in young people [
The Self-Hate Scale is a reliable 7-item measure used to assess self-hatred in young people [
Desire to discontinue self-harm behavior was indexed using a single item adapted from the Self-Injurious Thoughts and Behaviors Interview–Revised [
After completing all 3 SSIs, users were prompted to provide a quantitative “star rating” of the program, from 1 to 5 stars, where higher star values indicate higher ratings or more positive feedback. In addition, after the intervention, all individuals were asked if they would like to provide qualitative feedback via an optional writing prompt (eg, “Do you have any feedback for us?”).
Previous web-based SSI research suggests small to large within-group effect sizes for key outcomes (including hopelessness and self-hatred) using more naturalistic study designs (ie, not randomized controlled trials) [
We assessed the total number of views, starts, and completions for each SSI, as well as item-level drop-off data within each SSI, to describe broad usage patterns without excluding any data. Demographic and outcome data were only recorded and made available once individuals advanced through to the end of a program and clicked “submit.” All pre-post analyses, star ratings, and qualitative data were therefore conducted and reported within program “completers” (ie, individuals who completed an SSI). In addition, where demographic data were available (Project SAVE SSI data), we restricted our analysis to teenagers (ie, excluding individuals who reported ages outside of 13-19 years). As Project SAVE was designed for teenagers engaging in self-harm, only individuals who self-disclosed recently engaging in self-harm (via the Koko onboarding pathways described earlier in this section) were directed to complete this program.
For each SSI, we evaluated usage patterns and feedback, including the number of people who viewed (ie, opened the first page of the program), started (ie, advanced past the first page), completed (ie, advanced through the entire program and beyond the questions), and provided “star ratings” (ie, quantitative ranking of 1-5 stars, with higher stars reflecting a higher rating) for each SSI. In addition, we calculated the program completion rates (percentage completed out of those who started) and average star ratings for each SSI. To illustrate the types of qualitative feedback each SSI received, we extracted specific examples of positive and negative feedback (see the Results section).
Aggregate-level data were available for views and user dropout, for every page within each SSI, among the entire sample (ie, without focusing solely on program completers). We reported these results by plotting the percentage dropout from the number of views for each page within each SSI.
We evaluated pre- to postintervention changes for 4 outcomes: hopelessness (via two separate tests, 1 for Project ABC and 1 for REFRAME) as well as self-hate and desire to discontinue self-harm (Project SAVE only). After checking appropriate assumptions (ie, verifying that pre-post difference scores for each outcome were approximately normally distributed) [
As Koko provides anonymous mental health support, individuals are not required to provide potentially identifiable demographic information (eg, age, gender, and race and ethnicity) to complete minicourses. Specific demographic information for this study (age, gender, and race and ethnicity) was only available for the Project SAVE SSI, as this minicourse was introduced to Koko
The Project SAVE analyses excluded individuals who were not teenagers between the ages of 13 and 19 years, resulting in a final sample of 1194 individuals (72.28% of the total Project SAVE data). Among this group, the average age of the individuals who completed Project SAVE was 15.71 (SD 1.83) years. The top three most commonly endorsed gender identities were female (419/1194, 35.09%), nonbinary (189/1194, 15.83%), and not sure (98/1194, 8.21%). The top three most commonly endorsed racial or ethnic identities were White (607/1194, 50.84%); Asian (172/1194, 14.41%); and Hispanic or Latinx (112/1194, 9.38%).
Gender identity and race and ethnicity for Project Stop Adolescent Violence Everywhere (N=1194).
Demographics | Values, n (%) | |
|
||
|
Agender | 35 (2.93) |
|
Androgynous | 13 (1.09) |
|
Female | 419 (35.09) |
|
Female to male transgender | 74 (6.20) |
|
Gender expansive | 15 (1.26) |
|
Gender identity not listed | 18 (1.51) |
|
Gender information missing | 219 (18.34) |
|
Intersex | 3 (0.25) |
|
Male | 27 (2.26) |
|
Male to female transgender | 2 (0.17) |
|
Nonbinary | 189 (15.83) |
|
Not sure | 98 (8.21) |
|
Prefer not to say | 26 (2.18) |
|
Transfeminine gender | 2 (0.17) |
|
Trans man | 10 (0.84) |
|
Transmasculine gender | 44 (3.69) |
|
Transgender | 15 (1.26) |
|
Two-spirited | 2 (0.17) |
|
||
|
Asian | 172 (14.41) |
|
Black or African American | 66 (5.53) |
|
Hispanic or Latinx | 112 (9.38) |
|
Native American or Alaska Native | 23 (1.93) |
|
Native Hawaiian or other Pacific Islander | 12 (1.01) |
|
White | 607 (50.84) |
|
Prefer not to answer | 125 (10.47) |
aIndividuals could select multiple racial and ethnic identities.
Power analyses indicated that we had 99% power to detect a small effect size (Cohen
Project ABC was viewed 17,620 times, started 14,434 times, and completed 3679 times—with a 25.49% (3679/14,434) completion rate among starters across the 12-month study period. Among the Project ABC completers, 3412 (92.74%) provided a star rating with an average star rating of 4.27 (SD 0.94; median 5) stars. In total, 1217 (33.08%) provided qualitative feedback on Project ABC (see
Finally, Project ABC received far more views, starts, and completions than either of the other two SSI programs (4065 and 2174 views for Project SAVE and REFRAME, respectively), as Tumblr advertised the Project ABC SSI as a featured minicourse between December 2021 and February 2022, resulting in higher traffic to this SSI.
Examples of positive and critical feedback for all single-session interventions (SSIs).
SSI | Positive feedback | Critical feedback |
ABCa | “Thanks y’all! been dealing w some serious mental health issues and having places to remind me of my agency and joy is really helpful.” | “Despite the great intentions and work I think there is situations that are very complicated and having this intermediate bot, very few info of the person you are helping it’s too simplistic.” |
|
“this was really really helpful and i’m seriously going to try my goal/plan. I also feel awake and motivated enough to study. I’d love to see more in the future.” | “This helped me see get through my rain cloud but now I kinda feel stressed.” |
|
“Hey, this was surprisingly well done. I went in expecting it would be terrible. But you’re making mental health a really approachable topic for people. Thanks for working on reaching out to others. I’d love to see you continue with these mini-courses.” | “This is a good idea, but only works for the things one has control over. If you are terminally ill, just lost a loved one or in another uncontrollable challenge, none of these can help.” |
REFRAMEb | “This is such a great way to de-stress, I mean re-frame your stress, it’s definitely a bit helpful.” | “Less simplification” |
|
“I decided to pick up my phone and do this while I was procrastinating. It’s so crazy how this seemingly small task changed my perspective.” | “i think that one of the problems i see is that this requires people to be more specific and there isn’t a sense of connection.” |
|
“I love this so much!! It actually made me feel better, which I didn’t think it would! Thank you <3.” | “Please include physical stress reducers, as well. It is difficult to focus on reducing stress when I can’t properly string thoughts together.” |
SAVEc | “This is amazing. My thoughts of self-harm faded a bit, and prompted me to do the things alternative to when I have self-hate thoughts.” | “It would’ve been more helpful if reasons for self-harm outside of self-hatred were explored. I’m currently dealing with external circumstances that are overwhelming me and my response is to harm myself. It feels like the only way to release my emotions.” |
|
“This is the most convincing thing I’ve ever heard as to why not to self harm. Thank you so much this is so helpful.” | “I have dealed with this so long that there wasn’t really anything new to me, so it really didn’t help.” |
|
“Thank you. This course has definitely calmed me down when I was having a breakdown and thinking of hurting myself.” | “Talk about slowly building up to recovery instead of just jumping right in. Talk about how to deal with intense emotions and more.” |
aProject Action Brings Change (ABC) SSI.
bREFRAME SSI.
cProject Stop Adolescent Violence Everywhere (SAVE) SSI.
Percentage dropout on each page of all 3 single-session interventions (SSIs), out of the number of individuals who viewed that page. Arrows reflect points where writing prompts were introduced, in each of the 3 SSIs. Spikes in dropout tended to occur after initially opening each SSI, as well as on pages requesting written responses. ABC: Action Brings Change; REFRAME; SAVE: Stop Adolescent Violence Everywhere.
In 12 months, Project SAVE was viewed 4065 times, started 2961 times, and completed 1652 times; 55.79% (1652/2961) of those who started Project SAVE completed it. After excluding individuals with ages outside our desired range (13-19 years), 1194 observations remained for analysis. Among those who completed the minicourse, 954 (79.90%) provided a star rating for Project SAVE, with an average rating of 4.22 (SD 0.97; median 5) stars. A total of 209 (17.50%) participants provided qualitative feedback on Project SAVE (
REFRAME was viewed 2174 times, started 1498 times, and completed 848 times within 12 months (848/1498, 56.60% completion rate among the starters). Among REFRAME completers, 732 (86.32%) provided a star rating for the REFRAME SSI, with an average rating of 4.31 (SD 0.93; median 5) stars; 246 (29.01%) provided qualitative feedback on the REFRAME minicourse (
Among individuals who completed Project ABC, hopelessness significantly decreased from before to after the SSI (
Means, SDs, and effect sizes for all single-session intervention (SSI) outcomes.
Outcome and SSI | Before the SSI, mean (SD) | After the SSI, mean (SD) | Cohen |
|
|
||||
|
ABCa | 2.60 (0.78) | 2.16 (0.80) | −0.81 (−0.85 to −0.77) |
|
REFRAMEb | 2.86 (0.74) | 2.31 (0.78) | −0.88 (−0.96 to −0.80) |
|
||||
|
SAVEc | 5.69 (1.27) | 5.07 (1.64) | −0.67 (−0.74 to −0.60) |
|
||||
|
SAVE | 2.63 (1.20) | 2.97 (1.32) | 0.40 (0.33 to 0.47) |
aProject Action Brings Change (ABC) SSI.
bREFRAME SSI.
cProject Stop Adolescent Violence Everywhere (SAVE) SSI.
dDesire to discontinue self-harm behavior.
In addition, Wilcoxon signed-rank tests were performed as sensitivity analyses for all outcomes due to the relatively nonnormal distributions of difference scores. For all outcomes, results were consistent with
Hopelessness ratings before and after the Project Action Brings Change (ABC) single-session intervention (SSI; left) and before and after the REFRAME SSI (right). Higher scores reflect higher levels of hopelessness.
Self-hate ratings before and after the Project Stop Adolescent Violence Everywhere single-session intervention (left), where higher scores reflect higher levels of self-hatred. (Right) Desire to stop future self-harm behavior, where higher scores reflect higher desire to stop future self-harm.
Within 12 months, 3 very brief (5-8 minutes) web-based SSIs were viewed >18,800 times and completed >6100 times. Offered as Koko minicourses and embedded within a popular social platform, all 3 SSIs received high-quality ratings (average ratings >4 out of 5 stars). Of the participants who started an SSI, between 25% and 57% of them completed one. Among those who completed Project ABC and REFRAME minicourses, individuals reported a decrease in hopelessness from before to after the SSI. For those who completed Project SAVE, individuals reported decreased self-hatred and increased desire to stop future self-harm behavior from before to after the intervention. This study represents a real-world evaluation of the acceptability and short-term utility of web-based SSIs as very brief, anonymous, “in-the-moment” mental health supports that can be integrated within major social platforms such as Tumblr.
Consistent with existing research on web-based SSIs [
One possible reason for the relative similarity of postintervention effect sizes observed in this study (5-8–minute SSIs) versus within-group effects in earlier randomized trials (30-minute SSIs) [
Our results may also indicate the
Low real-world completion rates for self-help digital interventions, especially relative to guided digital supports [
Embedding SSIs within a popular social platform (Tumblr) also likely increased the
Notably, existing research identifies safety as a primary ethical concern for researchers and stakeholders interested in using digital mental health tools among youth [
This study has several strengths. Although randomized trial research overestimates user engagement for digital interventions [
In addition to the aforementioned strengths, this study has some limitations. First, demographic information was not available for all individuals included in this study. Demographic information that
In addition, as is often the case in web-based mental health support research [
Finally, given that this study represents an unmasked evaluation of within-group intervention effects in a completers-only sample, our ability to draw causal inferences was limited. For example, those who completed an SSI may have provided more positive star ratings for the programs than those who exited before finishing the program. However, 2 of the 3 SSIs featured in this study have demonstrated positive effects on identical outcomes in large-scale, triple-masked, and randomized research [
Very brief SSIs (5-8 minutes) can be embedded within web-based social platforms as anonymous, in-the-moment mental health supports capable of reaching many individuals within months. Among those who complete these SSIs, individuals generally rate them as acceptable, and pre-post evaluations suggest that they may be helpful in reducing hopelessness and self-hate as well as in increasing the desire to stop self-harm. These pre-post findings, combined with results of earlier randomized trial research, suggest that SSIs delivered in this context may be a sustainable approach for providing the much-needed mental health resources, particularly for teenagers who may not have access to other mental health supports. Considering the substantial unmet need for mental health care among teenagers in the United States [
Action Brings Change
Stop Adolescent Violence Everywhere
single-session intervention
JLS received funding from the National Institute of Health Office of the Director (DP5OD028123), National Institute of Mental Health (R43MH128075), National Science Foundation (2141710), Health Research and Services Association (U3NHP45406-01-00), Upswing Fund for Adolescent Mental Health, Society for Clinical Child and Adolescent Psychology, and the Klingenstein Third Generation Foundation.
MLD received funding from the Graduate Council Fellowship at Stony Brook University and has previously received research funding from the Psi Chi International Honors Society and the University of Denver Faculty Research Fund.
JLS, RM, and MLD conceptualized the project, contributed to the study design, and developed the original versions of the single-session interventions. RM adapted the original intervention materials for integration into the Koko. MLD performed data analyses and wrote the initial draft of the manuscript. All the authors contributed to the review and editing of the final manuscript.
MLD receives book royalties from New Harbinger and funding from the Graduate Council Fellowship at Stony Brook University. She has previously received research funding from Psi Chi International Honors Society and University of Denver Faculty Research Fund. RM is the cofounder and chief executive officer of Koko, a nonprofit organization that receives financial aid from industry partners, private donors, and Hopelab Ventures. He also serves as a scientific advisor for Homecoming, a digital platform to help support psychedelic-assisted psychotherapy. JLS serves on the Scientific Advisory Board for Walden Wise and the Clinical Advisory Board for Koko; is the cofounder and codirector of Single Session Support Solutions, Inc.