Background: An increasing number of mental health apps (MHapps) are being developed for youth. In addition, youth are high users of both technologies and MHapps. However, little is known about their perspectives on MHapps. MHapps might be particularly well suited to reach the youth underserved by traditional mental health resources, and incorporating their perspectives is especially critical to ensure such tools are useful to them.
Objective: The goal of this study was to develop and pilot a process for eliciting youth perspectives on MHapps in a structured and collaborative way. We also sought to generate learnings on the perspectives of Latinx youth on MHapps and their use in ways that might facilitate discovery, activation, or engagement in MHapps, especially in Latinx populations.
Methods: We created a series of focus groups consisting of 5 sessions. The groups introduced different categories of MHapps (cognitive behavioral therapy apps, mindfulness apps, and miscellaneous apps). Within each category, we selected 4 MHapps that participants chose to use for a week and provided feedback through both between-session and in-session activities. We recruited 5 youths ranging in age from 15 to 21 (mean 18, SD 2.2) years. All the participants identified as Hispanic or Latinx. After completing all 5 focus groups, the participants completed a brief questionnaire to gather their impressions of the apps they had used.
Results: Our focus group methodology collected detailed and diverse information about youth perspectives on MHapps. However, we did identify some aspects of our methods that were less successful at engaging the youth, such as our between-session activities. The Latinx youth in our study wanted apps that were accessible, relatable, youth centric, and simple and could be integrated with their offline lives. We also found that the mindfulness apps were viewed most favorably but that the miscellaneous and cognitive behavioral therapy apps were viewed as more impactful.
Conclusions: Eliciting youth feedback on MHapps is critical if these apps are going to serve a role in supporting their mental health and well-being. We refined a process for collecting feedback from the youth and identified factors that were important to a set of Latinx youth. Future work could be broader, that is, recruit larger samples of more diverse youth, or deeper, that is, collect more information from each youth around interests, needs, barriers, or facilitators or better understand the various impacts of MHapps by using qualitative and quantitative measures. Nevertheless, this study advances the formative understanding of how the youth, particularly Latinx youth, might be viewing these tools.
Numerous health apps exist, with estimates of just over 350,000 health apps in 2020, with 18% of those being for stress and wellness and 5% for mental health conditions . Despite this large number of available mental health apps (MHapps), few are downloaded, and those that are downloaded rarely have long-term, sustained use. A review of MHapps targeting depression and anxiety found that the top 3 most downloaded apps accounted for 90% of all downloads and that 63% of depression apps and 56% of anxiety apps had no regular users over a month period [ ]. Another study found that the median retention rate for MHapps 15 days after downloading was 3.9%, and by 30 days, it dropped to 3.3% [ ]. Thus, app discovery (ie, finding useful MHapps), app activation (ie, initiating the use of an MHapp), and app engagement (ie, sustained and effective use of an MHapp over time) are all major barriers to ensuring that MHapps can provide the most impact on consumers’ lives.
It is often proposed that MHapps could have a significant impact on youth [- ]. Most mental health disorders emerge in adolescence or early adulthood, suggesting that youth is a critical transitory period for mental health [ ]. Youth face many barriers to care that include structural and perceptual barriers, including perceptions of both mental health problems and services [ ]. Youth of color experience even more barriers to care. They are less likely to live in communities with mental health providers [ ] and to receive traditional mental health care than their White counterparts [ , ]. Even when providers and treatments are available, providers are less likely to match their racial and ethnic backgrounds, and culturally relevant care is rarely available [ , ]. For youth with mental health needs, parents and other adults are often the gatekeepers to care. As such, youth often seek care through informal sources first, which includes family and friends but increasingly includes internet-based information and interventions [ ]. Seeking informal sources of support before engaging in traditional care might be even more common among Black and Latinx youth than among their White counterparts [ ].
MHapps are a potentially useful way to reach youth and provide mental health support and interventions. Smartphone ownership increases substantially from age 11 to 18 years. With over half of youth aged 11 years and 91% of those aged 18 years owning smartphones . As such, youth are quite comfortable using technology, and most report using them almost constantly to access the internet and communicate with peers [ ]. Many youth also report that they are using or have used health apps. A recent representative survey of those aged 14 to 22 years found that 69% report having used a health app, a number that climbs to 75% among the youth with moderate to severe symptoms of depression [ ]. In this survey, many of the categories of apps youth reported using were related to mental health, including sleep (27%), meditation or mindfulness (17%), stress reduction (14%), mood tracking (10%), depression (9%), and alcohol or substance use (5%). Thus, MHapp use among youth seems common.
MHapps might be particularly useful to provide mental health resources for traditionally underserved groups, such as racial and ethnic minorities [, ]. First, as mentioned earlier, a sufficient workforce that mirrors the racial and ethnic diversity of the population in need does not exist. As an example, in California, Latinxs represent 38% of the state’s population but only 4% of its psychiatrists and 8% of its psychologists [ ]. Investment in developing a better pipeline to train and retain diverse mental health providers should be a priority, although this process will take time and resources are needed today to help those in need. Second, MHapps can overcome geographic barriers and can be deployed more effectively where gaps in service provision exist. More than 80% of counties in the United States are designated as mental health professional shortage areas [ ], and again, youth of color are less likely to live in areas with access to care [ ]. Third, technology might be seen as a more desirable resource either because of the ease of access, overcoming barriers such as transportation or stigma [ ], or just because some people might be more inclined to use an MHapp rather than traditional care. Calls to use technology for mental health service delivery have emphasized the need to broaden the portfolio of available mental health resources to promote market segmentation; that is, identifying targeted groups of consumers to better tailor products, such as mental health resources, to those populations. Even when racial and ethnic minorities receive traditional mental health services, they tend to receive fewer sessions [ ]. Therefore, in addition to considering how to better engage racial and ethnic minorities in care, we must also consider how to design care that will better serve them when they get there. MHapps might be one such solution to help address this challenge.
Although robust literature exists to suggest that MHapps are effective [, ], reviews specifically focused on apps for youth have shown that research in this space is sparse [ , ] but have demonstrated some indications of positive benefits [ ]. In addition to there being fewer studies that address the effectiveness of MHapps for youth, most of the studies conducted focus on the early feasibility or acceptability of apps that never make it into the hands of consumers. For example, Grist et al [ ] found in their systematic review that of the 15 apps described in 24 eligible studies, only 2 were available to download. Studies on MHapps for youth have been characterized by poor uptake and engagement, lack of specification of procedures such as human support, and a lack of research on younger children or traditionally marginalized populations [ ]. Given the lack of empirical evidence to speak of the effectiveness of MHapps for racial and ethnic minority youth, processes to better elicit their perspectives and views could be useful to inform the adoption and deployment of MHapps among such populations.
Thus, a gap exists between the use and enthusiasm of MHapps for youth and our understanding of the actual impact on this group. It is worthwhile to better understand youth’s impressions, especially youth of color, of MHapps to understand what they would use and why, potentially leading to better design and dissemination of MHapps for youth and youth of color.
Including Youth Perspectives in the Research Process
One way to address this gap and provide information to better design studies to investigate the potential effectiveness of MHapps for youth is to incorporate youth’s perspectives on MHapps, especially those currently available to youth, into research . Indeed, research has taken different approaches to consider the youth perspective, including expert opinions, app reviews, and working with youth themselves. One study conducted a workshop with psychologists with experience working with youth to identify different youth media preferences and then analyzed app features to identify the presence of these media preferences [ ]. Media preferences included strategies of social connectivity, use of videos, tailorability, and rich interactions. However, they found that the use of these strategies was limited in apps available for youth, with social connectivity being the most used but with few examples of interactive multimedia experiences. Several studies have worked with youth to co-design digital mental health interventions [ - ], including MHapps, but these studies have often resulted in early-stage ideas or functional prototypes that, although sometimes are evaluated, rarely make it to app stores or other places where youth could access them. Finally, many studies have focused on trying to understand MHapps from app store user reviews [ , ]. Although these studies have not focused specifically on MHapps for youth people or on reviews coming only from youth, given that youth are using these tools, it is possible that some of these reviews come from youth. Overall, these studies demonstrate findings similar to other evaluations that leverage expert opinions or content analysis. Users appreciate apps that are esthetically pleasing and functionally diverse while allowing opportunities for personalization and tailoring. In addition, users dislike apps with poor usability and pricing models that focus on freemium models or emergent costs. However, a limitation of these reviews is that these prompts are general and may not elicit feedback on the aspects of MHapps that might best support their design and implementation.
Diverse methods exist to gather people’s impressions of digital products and to team with potential users in ways to better create technologies that meet their needs. These methods, leveraged from the field of human-centered design, have been used in various projects working to develop MHapps for the youth or to understand the needs and interests of the youth when interacting with technologies. Applying such techniques to better understand the impressions of existing apps is also useful to understand what are the affordances of these technologies as well as where such technologies are lacking. In our study, we drew on synchronous methods, used focus groups, and adapted asynchronous remote community (ARC) methods to inform the aspects of our study design. ARC includes a collection of methods intended to facilitate group-based research at a given distance . ARC has been used in various apps, especially for working with populations in which in-person coordination might be challenging, such as populations with rare diseases [ ] or stigmatized populations [ ]. ARC methods have also been used with teens to leverage digital communication using platforms familiar to teens (eg, social media and asynchronous communication) and to balance their busy schedules [ ]. Despite the creation of ARC methods before the pandemic, these methods are also useful for conducting remote design work that is necessary, given the need to follow physical distancing protocols.
In this study, we attempted to develop a process for incorporating youth perspectives on MHapps in a structured and collaborative way. We sought to pilot our methods with a small group of youth while also attempting to learn their perspectives on available MHapps. We were particularly interested in the views of youth from traditionally underserved and marginalized populations (given the potential of MHapps to overcome barriers they face and the relative lack of knowledge of MHapps for these populations in comparison with their White counterparts) and therefore worked with a sample of youth with a Latinx background. We conducted a 5-week focus group with a set of youth aged 15 to 21 years to refine a process for incorporating app feedback into their views on MHapps and to identify some characteristics of MHapps that might serve as barriers to or facilitators of discovery, activation, or engagement. As such, we framed our work in contributing both our methodologies and some formative learnings from our small pilot of these methods.
We recruited participants through a local nonprofit team that works with youth from underrepresented groups, especially Latinx youth, to accelerate readiness in Science, Technology, Engineering, Arts, and Math. Recruitment occurred via email and recruitment messages posted on the nonprofit team’s Discord channel. We compensated participants with a US $25 Amazon gift card for each of the 5 focus group sessions that a participant attended, with a maximum compensation of US $125 for the entire study.
Informed consent was obtained from all participants aged ≥18 years. For participants <18 years of age, informed consent was obtained from their parent or guardian and assent was obtained from the participants. Researchers had an existing partnership with this nonprofit team, which facilitated consenting participants and their parents or guardians for participants <18 years. A member of our research team provided a digital consent or assent form to the participants and a digital consent form to parents for participants <18 years of age. A member of the nonprofit team helped facilitate contact with parents or guardians when necessary. Institutional review board approval was obtained from the University of California, Irvine (#2019-5609).
We conducted 5 focus group sessions, 1 per week, totaling 5 weeks. For the first 3 weeks, each week, participants used a new category of MHapps and then discussed the app they used during the following session. In the fourth week, participants were asked to choose 1 app to use from all the apps we introduced across all the weeks. In each session, we collected feedback about the apps participants used in the past week. In the fifth session, we discussed feedback across all the apps participants used.
The app categories were cognitive behavioral therapy (CBT) apps (week 1), mindfulness apps (week 2), and miscellaneous apps, which did not pertain to 1 category, including a coping app, journaling app, mood-tracking app, and peer support app (week 3). Each category contained 4 apps. These apps were selected to represent the general categories of MHapps that the youth use and the features present in popular MHapps. Furthermore, our research team used our experience of identifying and evaluating apps and considered apps that were used by youth based on previous analyses . Each of the 4 apps discussed in the week was assigned to a different participant (2 participants used the same app).
We selected focus groups for our method because focus groups enabled us to contrast between the different apps by engaging in group discussions. We used this approach to enable discussions about the different types of functionalities of related apps. Focus groups also enabled us to address known challenges in working with teens  by making it easier to balance power dynamics between researchers and teens. We drew on design theory that has established that discussion around different prototypes can result in increased participant rapport and sharing about the prototypes they each engaged with [ ]. We also built on prior work through which exposing participants to different types of designs and functionalities can elicit feedback on functionalities that are most important to users [ ].
Focus Group Session Structure
Each focus group session was approximately 90 minutes and was organized around a similar schedule (): icebreakers to build rapport between participants and researchers, discussion and activities about the apps assigned to participants in the past week (week 2-5), a brief presentation of a core concept for the apps in the upcoming week (eg, CBT or mindfulness), and assignment of apps that participants would use in the following week. To ensure that all the apps were evaluated, each participant chose a different app until all the apps were chosen by at least one person. We then asked the participants to use the app of their choice throughout the next week. In the week 4 session, the participants could choose an app that they had already used in the past few weeks or they could choose a completely new app that they were interested in.
|Week||Activities in focus group sessions||MHappsa assigned|
aMHapp: mental health app.
bCBT: cognitive behavioral therapy.
cN/A: not applicable.
Focus Group Activities for Collecting Feedback
We collected feedback about participants’ experiences using the assigned apps in the past week for approximately 50 of the total 90 minutes of sessions in weeks 2 to 5. Participants filled out sticky notes on an interactive remote whiteboard named Miro  in response to guided questions. Participants then shared their experiences with the app they used orally with the rest of the group. We used the following questions to guide the discussion: “What did you like most about the app?”, “What did you like least?”, “How do you think apps like this could be useful for youth like you?”, “What do you think apps like this are missing for youth like you?”, and “If you could design a [category, i.e., CBT, mindfulness, mental health]-app for youth like you, what would you do differently?”
We used a web-based community platform to communicate with participants and elicit further information from participants about their experiences using the app they chose to interact with for the week. We chose Discord  because the participants were already using it as part of the community from which we recruited. The goal of these activities was to capture participant perspectives as they were using the apps in their lives, rather than reflecting on their use in the focus group sessions. We planned to deliver 3 questions every week. We used a generic user profile called “wellness-bot,” which would post the questions in Discord ( ). The profile was controlled by a research team member. We named the account wellness-bot to encourage users to interact with the account as if it was a chatbot and feel more comfortable with the anonymity of sharing with a chatbot instead of a person. Some questions were delivered in a channel created for this study that all participants shared, while others were delivered through direct messages. The questions prompted the participants to use the app and provide feedback on their experiences using the app.
Engagement in Discord started immediately after the first focus group. We continued that week with the second question:
Now that you’ve set up the app, try out an activity! Some examples include logging how you’re feeling today, reviewing past entries, listening to a video, etc. What is something about the app that you find useful or not useful? Share in the channel.
At the end of participation, the participants received a follow-up survey about the apps they used. They were asked to indicate the app they used within each category. For each app they indicated, they were asked the following questions (response options) shown in.
This survey was created by adapting the One Mind PsyberGuide Consumer Review Questionnaire  and questions from the Mobile App Rating Scale [ ], which have been recommended by service users for consumer evaluation in another study [ ]. This included star ratings, satisfaction as indicated by likelihood to recommend the product, and open-ended questions about what they liked the most and least about the app. These questions also align with those used to evaluate technologies and apps in other settings, such as star ratings from app stores or the Net Promoter Score.
Follow-up survey about the apps they used.
- During the week you used the app, how often did you use it? (1, One day of the week; 2, A few days of the week; 3, Once a day; 4, Multiples times per day)
- What did you like most about the app? (Free-text response)
- What did you like least about the app? (Free-text response)
- What impact, if any, did this app have on your wellness or how you felt? (1, Negative impact; 2, No impact; 3, Small positive impact; 4, Moderate to large positive impact)
- Did you continue to use this app after the week you were asked to use it? (1-No; 2-Yes)
- Would you recommend this app to people who might benefit from it? (1, Not at all--I would not recommend this app to anyone; 2, There are very few people I would recommend this app to; 3, Maybe -- There are several people whom I would recommend it to; 4, There are many people I would recommend this app to; 5, Definitely -- I would recommend this app to everyone)
- What is your overall star rating of the app? (1, ☆ -- One of the worst apps I’ve used; 2, ☆☆; 3, ☆☆☆ -- Average; 4, ☆☆☆☆; 5, ☆☆☆☆☆ -- One of the best apps I’ve used)
Qualitative Analysis of Feedback From the Youth
We used thematic analysis to analyze the focus group data, using an inductive approach  to conceptualize themes. A senior (EA) and junior (SP) researcher open coded the focus group transcripts using a descriptive coding approach [ ]. The codes described the topics mentioned by the participants on a line-by-line basis. The codes were identified inductively based on the data. Given the small sample of data, we did not follow a fixed codebook; rather, we coded for the different topics that participants discussed and then conceptualized themes. Examples of codes identified include easy-to-consume content, personalization, social interaction, and relatability of content. The 2 researchers worked closely together, discussing data from all the focus groups. On the basis of the resultant codes, the 2 researchers conceptualized the data into themes related to each other through brief memos summarizing the coded data. During weekly meetings, the researchers discussed codes and themes in the data. The 2 researchers grouped subthemes together into themes such as personalization, connecting with others, and reliability of content. The other 4 researchers in the team assessed the subthemes and provided feedback, which led to a slight reorganization of the themes as presented in the results. Because the data collected centered on feedback about the functionalities of the apps, the themes that were conceptualized are also connected to app functionalities (eg, content type, features, and design of the app). We did not conduct interrater reliability, as any misalignments in coding supported our process of synthesizing data, understanding themes, and revising memos [ ]. The themes presented are comprehensive of our sample, but given the small sample size and formative nature of this research, we did not intend to reach data saturation.
Analysis of Research Approach
The research team reflected on the stages of the research design and assessed which ones were successful at engaging the youth: the way we engaged and built rapport with youth, how we engaged with the group of participants, and the activities we designed during and between the focus group sessions. Because this was a preliminary and formative study, our research team met on a weekly basis to discuss that week’s session, make any modifications to the next week’s activities, and discuss lessons learned. We note that our team conducting the research procedures did not include any members that identified as Latinx. However, this study was part of an existing research partnership with a nonprofit team from which participants were recruited, and members of the nonprofit team shaped the research design and recruitment considerations. For example, together we agreed that focus groups were suitable and to use the nonprofit team’s existing Discord infrastructure for between-session interaction. The nonprofit team’s members also directed our recruitment to students who were part of the program, influencing who was willing to participate. The research team included 1 researcher who had been conducting qualitative studies for >10 years (EA) and another for >5 years (SMS). In addition to collaboration with this nonprofit team for this project, our team also had multiple years of research and clinical training with Latinx communities and collaboration with Latinx investigators.
Five participants attended each of the focus groups. The participants ranged in age from 15 to 21 (mean 18, SD 2.2) years. Of 5 participants, 4 (80%) identified as women and 1 (20%) as men. All the participants identified as Hispanic or Latinx, and 1 (20%) participant also identified as Black.
Focus Group Insights
Participants desired a range of features that would tailor the apps to the needs and practices of the youth, such as topically relevant content to youth issues, lightweight and playful content, and brief interactive exercises or content. The youth noted that the content of the apps came from authoritative sources. The youth also emphasized the importance of authentic social connections with others.
Personalizing Content With Youth’s Needs
Participants shared that they often found the content of these apps not specific enough to the problems that youth encountered. Users mentioned wanting apps specifically designed for their demographics, age, challenges, and stage of life. One of the participants currently enrolled in high school felt “different timelines or milestones that youth deal with, [specifically] graduating from high school, and navigating adult life” could be helpful for others in the youth community.
Customizing Content and App Display
The participants found that tailoring the amount of content to their particular interests would help them engage with the apps. Such useful tailoring occurred when participants could customize the topics shown in the app so that they were better aligned with their interests (P4). The participants also wanted to customize the look and feel of the app by changing the colors used in the app (P4 and P1). They also preferred topics customized to the user, for example, by avoiding sensitive topics:
I took into consideration that some people might have trauma or something. And it was like, oh, would you rather not see posts related to certain topics. So they pretty much asked you that, before you even get to see anything, which I really liked.
The participants valued activities that could capture their personal situations. For example, P4 valued having the opportunity to create a plan to manage anxiety that was personal to themselves, not just to listen to or read generic content about the topic. When the content fit the participant’s life, they felt the app was more relatable:
I clicked on one of the manage stress [short audio clips] and they all let me see, oh, that’s my story too.
Content Aligned to Youth Use of Technology: “Lighten the Mood”
The participants hoped that MHapps would complement the types of activities they engaged in when using technology. They wanted the duration of the content to be aligned with their use of technology for entertainment:
Nowadays, people want something entertaining over something that takes more time. So in order for...youth to want to use a wellness app, they would have to bring in entertainment...maybe...including games, like something to make it more fun.
Another participant mentioned that it was important to have content to “lighten the mood” and make it “light on emotion,” such as comedic content, or visual and interactive elements, such as emojis and stickers (P2). They thought this could be more aligned with a positive interaction with technology:
I feel like this type of app should feel more like, I want to do this [as] part of my daily routine that’s actually going to bring me joy.
They thought that content related to mental health could be alternated with other types of engaging content:
Maybe I don’t want to be asked how I’m doing. Maybe I just want to play a game.
A different way of making content more lightweight was through encouraging messages:
As soon as I woke up the notification with words of encouragement was already there. And I would just have to click on it, and it would take me to a meditation.
Integration With Offline Activities
Several participants mentioned that they did not carry their phones with them all the time or that they preferred to do certain activities supported by the apps in a nondigital context; for example, journaling on paper. Some participants (P1 and P2) did not foresee using the digital tool for journaling:
I don’t really use my phone for any other apps, mainly just internet browsing and YouTube.
One participant thought it could be useful to get complementary support in their journaling activities; for example, by getting journaling templates that can guide the activity (P2).
Valuing Diversity in Content Over Single-Focus Apps
Participants found it valuable to have access to a range of content because it could give them control over what features to use:
There was a lot of different categories, and I just chose the ones that I was, more like I needed help with.
The type of content that interested participants included informational articles, audio content, meditation videos, or social content to help them relate to other people’s experiences (P4 and P5). The participants valued seeing content on different topics, such as work, exercise, sleep, nutrition, mindfulness, emotions, or managing the pandemic (P3, P4, and P5) and expressed reluctance in using an app that was focused on only 1 topic, such as anxiety (P3 and P4). They preferred entertaining features in apps. They wanted access to music playlists to boost their mood (P4).
Connecting With Others: Valuing Receiving and Providing Authentic Support
The participants valued apps that had a social component, making them feel less alone (P2, P4, and P5). They were more interested in fostering a community with other users than in the general content of the apps (P5). They identified 4 different needs for feeling connected to others: being able to share challenging content, seeing positive or motivational content, receiving support from others, and offering help to others.
Participants wanted to share their experiences and struggles with others who have similar experiences:
A safe space in which people who are experiencing the same feelings can talk to one another and just know that they’re not alone.
[Participant not identifiable from the transcript]
Participants valued having a space on the web to talk about things they would not be able to speak about with other people:
I feel like it’s for people that don’t have someone to talk to so they come on this app to like, I guess vent about their lives.
P1 felt it was useful for them to receive support from a peer who was going through similar challenges than through generic content in the app:
Links to people who could help you out... who know what they’re doing...
The participants also thought that offering support to others can bring them satisfaction as well. One participant posted a positive message on a community forum in the hope that it might help someone else who needed support (P5).
The participants valued authentic connections. They were reluctant to connect with others whose identities they could not recognize. For example, participants could not recognize if some of the social profiles of other people on the app were actual people or some type of automated account not associated with a person (P4).
Participants valued seeing their growth through tracking features, such as mood tracking or saved journal entries (P2). P2 noted that they felt they could not personally assess their emotional growth, and the app allowed them to reflect on their progress:
When you don’t track your progress, you don’t realize how far you come. But when you can physically see it, it can either be a positive thing or a negative, but sometimes is usually helps you adjust your mindset.
Quick Engagement With Content
Some participants found it important to get value out of the apps quickly through short periods of engagement and lightweight interactions (P1 and P4). P4 mentioned that they valued audio content when they wanted to relax because they were able to get the support they needed to unwind quickly, or to complete an activity quickly, such as recognizing anxiety (P4). They did not like engaging with conversational apps because it took too much time to arrive at a helpful solution. They found it valuable when the apps were interactive; for example, conversational interfaces that felt like talking to a therapist (P4) or to another person (P3).
Ease of Use
Participants valued simplicity in apps: “simple and easy to use...you just know where to go, almost like you’re walking in your house” (P2) reflected in simple layouts, esthetics, and functionality. Participants wanted content to be easy to navigate and for them to quickly know how to use the app, “you just know...where to find everything” (P2). Participants thought that too many features were overwhelming:
There’s a lot of different features that pop up in the app...might feel overwhelmed.
When apps contained too many features, it can make it difficult to find the desired content or even the content the user previously visited, which led to frustration:
Everything was all over the place in that app. It was hard to find a topic that I had seen earlier…couldn’t find it anymore.
The participants felt meditation apps did this particularly well:
Especially meditation. And I feel like they’re simple enough…someone wouldn’t feel like they’re going out of their way.
Match App Content With the User’s Context of Use
The participants “were more inclined to actually use the app” if the content was relevant to the need they had at the moment. In 1 instance, a participant used their app to wind down because the app had categories relevant to their intended use, to wind down at the end of the day. Because the user’s interests aligned with the app, they were able to engage with the app more meaningfully:
I was seeing an article about difficulty falling asleep sometimes. So, I was really interested in the article about sleep.
Reliability and Accessibility of Content
Accessibility of Apps and of Free Content
The participants identified certain aspects that made it hard for them to engage with the app. For example, 1 app included a community feature where people posted in foreign languages. P5 was concerned that some people might be expressing a need for help through their posts but that they might not be able to provide help because they do not understand the language.
The participants valued the availability of free content in the app and found frequent requests for upgrading to a paid version disruptive:
It was really pushing…need to buy…the premium….every time I go on the app.
The participants preferred using apps that were richer in free content:
This app was definitely way better. I feel like there was more content that was free and available to public use.
[P3 and P5]
The trial version of apps also made participants reluctant about the benefits of the app after the trial period.
Reliability of Content
Some participants found it important to know that the content of the apps was validated by professionals and that it was not biased (P1). They wanted to ensure that people could connect with professions if they wish to:
I kind of feel like at that stage, it should bring up links to professionals, because we’re dealing with mental health problems here.
Once we began the study, the number of student responses to between-session activities over Discord was disproportionately low, yielding only 2 user responses. During this time, we first posted messages on Discord using an account that presented as a bot sending updates. This did not receive responses, so we changed our updates to be posted by one of our undergraduate researchers. We also asked about Discord engagement during our focus group session and encouraged participants to respond to prompts. This prompted responses from another participant. However, the responses were consistent and overlapped with what we were learning in the focus groups. Therefore, after the second week, we discontinued the between-session activities because engagement was low, and we were not receiving additional information in the feedback sessions. We also thought it would be more beneficial to keep participant feedback confined to focus groups where it could be discussed more in depth.
In the follow-up survey, we identified 2 themes related to overall feedback on the apps. Organization of the app was identified as an important aspect across all app categories (CBT, mindfulness, and miscellaneous). The participants reported that they preferred apps with a simple interface such that they were “to the point,” organized, and easy to navigate. They disliked the presence of too many tabs, features, or steps to work through to get to the support they wanted. Payment was the second theme. In general, the participants reported that they disliked when most features were only available in a premium, paid version of the app, and they also disliked seeing promotions of paid content. This aligned with comments made throughout the sessions that participants thought that these types of apps should be free or indicated that they would be unlikely to download an app that required payment when free versions exist.
On average, the participants reported that the apps had a small positive impact on their mental well-being across all apps. No one indicated that the apps had a negative impact, although “no impact” was reported for 1 CBT app, 2 mindfulness apps, and 1 miscellaneous app. Miscellaneous apps were ranked as the most impactful, followed by CBT apps and then mindfulness apps.
When asked how often they used their app during the week they were assigned to use it, the participants were most likely to report using the app for “a few days of the week.” Participants were slightly more likely to continue using mindfulness apps after the study compared with CBT or miscellaneous apps. Of the 5 participants, 2 (40%) reported that they continued to use their CBT and miscellaneous apps after the week they were asked to use it, and 3 (60%) participants reported continuing to use their mindfulness app after they were asked to do so. The average star ratings assigned to the app categories by the participants were 3 for CBT apps and 3.6 for both mindfulness and miscellaneous apps out of a total of 5 stars.
In this study, we developed and refined a process for obtaining feedback on various MHapps. Our resultant series of focus groups allowed us to gain initial insights about Latinx youth’s preferences for and within our selected set of MHapps and their views on the benefits and use of the apps, albeit from a small sample of participants. As we gained experience with our procedure, we made a few changes; for example, removing the between-session activities and adding a final session to get impressions across all the MHapps. Despite refining the methods over the course of the 5 focus groups, we still identified some consistent preliminary themes across these sessions that provide some impressions of Latinx youth’s views on MHapps.
Latinx youth wanted apps that were youth centric in terms of content and functionality. However, youth-centric content meant not just making age-appropriate content in terms of language, visuals, or examples, but it also meant making the interactions brief and fun. Our participants also emphasized the importance of MHapps being able to transcend their digital components and facilitate meaningful offline interactions. Our follow-up survey also showed that ratings of MHapps might differ based on the types of questions asked. Although the CBT apps had lower star ratings than the other apps (3.0 vs 3.6), they were rated as being more impactful than the mindfulness apps. We discuss our findings in subsequent sections, both insights gained from what we learned from the participants and reflections on our methods.
The youth wanted MHapps that were reflective of youth problems, interests, and technology use. Many existing MHapps, even those that claim to be focused on the youth, often contain minimal surface-level tailoring, such as changing examples or esthetics . Other approaches include using techniques such as gamification that might appeal to some youth. On the other hand, others might find that games trivialize mental health issues and would appreciate a more serious approach [ ]. Indeed, another study that used a workshop approach to ideate multiple ideas for well-being technologies with the youth identified interests in diverse engagement strategies and types of technologies [ ]. Thus, it is worth noting that “youth” is not a homogenous population, and differences exist among different subpopulations (eg, specific age ranges, gender or gender identity, race, or ethnic background).
Our study focused on the subpopulation of Latinx youth. Although we cannot comment on which themes identified were specific to Latinx youth, as we did not have a comparison group, it is worth reflecting on some things noted. Our participants noted the importance of social connectedness through MHapps. The importance of social connection has been noted in other mobile health studies comparing Spanish and English speakers, finding that Spanish speakers emphasized feelings of social support, whereas English speakers emphasized introspection and self-awareness . It is hard to separate how much of these findings are related to youth generally or Latinx youth specifically, and further work could replicate these methods across different subgroups to help lead to more specific and definitive design recommendations.
Youth also go through different developmental processes. Therefore, adopting a developmental science lens might introduce affordances for MHapps and other youth-focused technologies . One proposed solution to better reflect youth preferences in MHapps aimed at youth is to include youth and various youth stakeholders in the design process [ ]. Indeed, various co-design methods (eg, questionnaires, interviews, focus groups, interactive workshops, and meetings) might be helpful to solicit input from youth and can be selected based on the stage of development and type of information needed [ ]. We used some co-design methods in this project to solicit information regarding the developed MHapps, but of course, these methods should precede fully developed products.
It is also worth reflecting on considerations for evaluation methods for MHapps. Several evaluation frameworks or systems exist, such as the American Psychiatric Association framework , One Mind PsyberGuide, and Framework to Assist Stakeholders in Technology Evaluation for Recovery to Mental Health and Wellness [ ]. Although these frameworks are intended to assist decision-making by diverse stakeholders, including consumers, providers, advocacy organizations, payers, and health systems, they often require specialized knowledge. This results in evaluations being conducted by individuals with expertise to apply such evaluation frameworks. Therefore, although consumer input might have been solicited in the development of these frameworks [ , ], methods for incorporating bottom-up input from consumers, especially consumers from diverse backgrounds, are needed. Some measures are relevant for consumers, such as the user version of the Mobile App Rating Scale [ ] or the System Usability Scale [ ], but these are mostly questions, rather than processes, for soliciting feedback. Our approach had both processes and questions, and many of the questions we used for our follow-up survey were derived from these measures [ , ].
We also found discrepancies between the youth feedback depending on how questions were asked. For example, we found differences between apps that were viewed as most liked (ie, mindfulness), using a 5-star rating scale, and those seen as more impactful (ie, CBT and miscellaneous). Given the vast amount of information available through app stores, many researchers have used ratings to understand the user perspectives of MHapps [, ]. However, this information might obscure more nuanced perspectives on apps. The specific discrepancies identified in our study, for example, for mindfulness apps and CBT, might reflect how this content is delivered to youth. It is worth noting that we did not collect formal outcome measures regarding the impact of these apps; therefore, our findings are entirely based on the youth’s subjective perspective. Research on traditional mental health services has found that youth preferences do not always align with better outcomes [ ]. As such, it is possible that these apps might have meaningful impacts on wellness and clinical targets or that satisfaction and clinical outcomes are not aligned, but additional research would be needed to verify this. Consistent with our previous points about the importance of separating app discovery (identifying apps), activation (initiating use), and engagement (repeated and sustained use), it is possible that elements that contribute to satisfaction, such as esthetics and learnability, might lead to people starting with an MHapp but that clinical effectiveness (ie, benefiting) might lead to sustained use. Other work has identified that people do discuss different factors related to barriers and facilitators to use across early initiation and long-term use [ ], and studies that further explore the contributions of satisfaction and benefits could be useful here. For example, people who experience early symptom change might be more likely to stop using an MHapp (ie, “happy abandonment”) but might be more likely to return to it if symptoms reoccur or to recommend it to others struggling with similar issues.
Reflections on the Apps That the Youth Designed in the Last Session
In our final focus group session, we had the participants design their own apps using a web-based collaboration platform. Participants received a series of prompts that covered different common features they identified (ie, symptom tracking, distraction tools, games, information, and links to resources). The participants consistently identified simplicity as a major design feature, but other common features included information, meditation or mindfulness tools, and links to other resources. This is extremely similar to popular features identified by unemployed individuals and essential workers for MHapps intended to support distress during the COVID-19 pandemic . A few of our participants also noted distraction tools, which were also identified in the study [ ], especially among unemployed workers. However, these interests are somewhat inconsistent with common features within MHapps, with the finding that the most common features both alone and in combination are journaling and mood tracking [ ]. In fact, MHapps containing only journaling and mood tracking combined accounted for 16.5% of the reviewed 278 MHapps [ ]. Many MHapps tend to be complex and multifeatured; however, our participants came up with suggestions that were simple and targeted. The IntelliCare app suite was designed to include simple, single-featured apps [ ], and an analysis of users of this suite found that participants tended to use focused subsets of the various apps available to them [ ]. Especially for the youth, who tend to be tech-savvy and technologically engaged, if a given MHapp does not meet their needs or is overly complex, they might use other technologies, perhaps not specifically designed for mental health, to meet their needs. Efforts should be made to better understand how youth use MHapps as part of their digital ecologies, and what needs and opportunities are lacking from how they are currently using technology to support their mental health and well-being.
Lessons on Methodology to Engage With Youth
Recruitment and Community Partnership
We engaged with youth who had already participated in an educational community. Our research group had also engaged with the community before. These youth were receptive to participating in the research, which might be because of the norms of the organization and partnership with the research team. They were also insightful and engaged in providing feedback, which might not have been true for youth recruited through other means. Nevertheless, working with youth from this organization helped build trust and collaboration more quickly. It also provided opportunities to present our findings back to the leadership in the organization to consider how these lessons could shape other efforts to involve their youth in activities or consider opportunities for technology to support their wellness.
Familiarity of the Participant Group and Research Team
The participants knew each other before the focus groups started. However, the research team did not and was not aware of any of their relationships. This can create imbalances in the group because the research team is an outsider to the group. For future work, we recommend considering a group that has less familiarity so that everyone, including the research team, gets to know each other in a balanced way. Alternatively, our research team could have spent more time getting to know the youth and building a rapport before shifting to data collection. Another approach would be switching to remote activities only after a synchronous focus group where participants met each other . Other work engaging children and adolescents in participatory design [ , ], especially using remote activities made necessary by the COVID-19 pandemic, has similarly demonstrated the need to build rapport with children and adolescents both at the beginning and repeatedly throughout extended design activities as we conducted.
Engagement With the Youth and Data Collection
We found it beneficial to conduct icebreaker activities as part of the focus group sessions because it got the participants started with sharing information about themselves with the rest of the participants and with the research team. We recommend keeping the icebreaker activities short so that they provide a warm-up to the activities but do not take a significant amount of time. One of the primary facilitators of the focus group was an undergraduate student (KC), which we found beneficial because this student was closer to the age of the participants than our other facilitators (EA, SMS, and MN) and could relate to them more in the discussion. We also found it useful to use informal language, including emojis, in communication with the youth. Although we initially attempted to send updates from an account that simulated a bot, we found that youth had slightly more updates when one of our undergraduate team members (Ashley McKinnon) messaged the Discord channel. We attribute this to participants relating more to a human being than to a generic bot.
Using a Social Network for Data Collection
We used the Discord channel to communicate with youth between sessions. This was beneficial because they were already using Discord for other activities, and the Discord channel we created for this study was part of the wider Discord of the community. However, the research team was not familiar with the norms of the community and how it was used by the group. Our interactions and questions did not result in responses from participants on the web. We recommend that in the future, researchers use a social platform that is familiar to the participants but is separate from their existing communities (eg, a separate Discord server, a separate Slack channel, or a private group on a social network). This allows researchers to define the norms of the community and participation. Similar to guidelines in prior work , researchers could design activities that encourage using the platform early on, or during focus groups, to familiarize participants with how they are expected to use the platform.
This work was limited by its formative nature; that is, we worked with a small sample of youth who were recruited through 1 specific community partnership. However, we would also like to note other limitations in this work that might be helpful to other researchers who conduct work in similar areas. As previously noted, the youth population is not homogenous. The participants we worked with were in the age range of 15 to 21 years, which captures mid- to late adolescence. Researchers should be mindful about what age ranges of youth populations they are most interested in working with and also appreciate that in this period of rapid development, a developmental science lens to understand the interplay of youth and technologies is necessary. We also worked with only youth who identified as Latinx, and our results may not be representative of the views of MHapps in other populations. We might also have gained different insights if we used different categories of MHapps. Our last category was a miscellaneous category, and we could have either replaced the categories we used (ie, CBT or mindfulness) or extended the number of focus group sessions to focus on different categories of apps, such as tracking or peer support apps. Not every aspect of our project went as planned, and we had to adjust methodologies along the way; for example, removing between-session activities. We expect that other projects likely change their initial plans and encourage other researchers to share these deviations as an opportunity to help others conducting similar work. Our team did not include members with a Latinx personal background. It is possible that participants would be less likely to connect and open up with our team members if they shared a Latinx personal background or that our analyses did not capture some elements of the Latinx experience because of the positionality of our research team. Finally, it is worth calling out some specific things that we did not hear from the participants we worked with. For example, recent work has called out several aspects related to the digital divide that might be barriers to use among marginalized groups, such as cost and internet and device access . It is hard to disentangle whether these concerns are not present among the youth generally, perhaps because they are the most tech-savvy, these youth specifically, perhaps because we recruited them from a nonprofit team focused on Science, Technology, Engineering, Arts, and Math, or related to our selection of apps, which focused on free products. Nevertheless, it is important to note that assumptions related to digital literacy and technology access are worth checking and rechecking as knowledge and access continue to evolve.
Future Directions and Work
We identified a range of needs for personalization of MHapps for the needs of youth. The content of the app should have some amount of topical relevance to youth’s interests and situations. Future work is needed to understand the topics that are most relevant to the youth and in what way they should be presented. For example, how specific content can be so that it is relatable to many but still feels meaningful to those for whom it is intended. The youth valued playful and lightweight interactions as part of the apps. More work is needed to understand how to implement such designs as part of existing MHapps; for example, MHapp designs could include lightweight audio content, activities designed in a way that feels less serious, or pairing activities that are perceived as demanding with activities that make the user feel good at the moment.
Building on this work, we see various potential avenues for future studies. One would be to conduct similar workshops with more youth, additionally drawing in youth from different racial and ethnic backgrounds or for different age ranges. We also see multiple ways in which this work could go deeper—individual interviews with the youth, refined between-session activities, and additional quantitative measures to look at the impact of MHapps both while using the apps (ie, process measures) and at the end of participation (ie, outcome measures). We believe that interviews and between-session activities could help better understand youth’s early experiences with MHapps, or app activation as we described earlier, as well as continued and sustained use (ie, app engagement). Additional quantitative measures could help better understand the impact of MHapps on youth. Finally, although we noted that app discovery, activation, and engagement were all challenges, given that we provided a select list of MHapps for participants, we did not learn much about discovery beyond how they navigated the choices we gave them. Future work could also work with the youth who elect to use MHapps on their own to better understand why and how they use them. It is likely that not all mental health and wellness needs that the youth are attempting to address with technologies are being approached using MHapps. The youth might be using other technologies, such as social media and games, and further work could help better understand how the youth approach those technologies in addition to MHapps.
We conducted a co-design activity with the participants in the last focus group. Future work could use structured design activities with the youth to understand specific preferences related to MHapps, that is, the types of functionalities, content, and visual look and feel of MHapps as well as that interactions align with their technology use practices. Our work provides some high-level insights—it should be youth centric, simple, and focused; translating these insights into design guidelines and app features would be a useful next step.
Given the growing number of MHapps and youth’s tendency to use these resources, understanding their perspectives on MHapps can help guide the development, evaluation, and implementation of these tools for the youth. Our study developed and used a multisession focus group design to introduce MHapps to youth participants, elicit feedback on those apps, and learn more about what youth might want in MHapps. This work adds to a growing body of research focused on understanding youth’s needs and interests with regard to mental health technologies while also attempting to identify affordances that such technologies might offer, especially through a developmental science lens. We highlighted emerging themes, such as the need for simple and tailored content and the intersection of MHapps with youth’s offline lives. Overall, the participants we worked with expressed enthusiasm for this space, but more work needs to support building MHapps that are effective for the youth and determining how best to make these tools available to youth and integrate them into their lives.
The authors thank Listo America and specifically Gustavo Madrigal for help with recruiting youth and the support of their members. Ashley McKinnon assisted with the study procedures. Funding for this study was supported by Pivotal Ventures and One Mind with funding supporting One Mind PsyberGuide.
Conflicts of Interest
SMS is a member of Scientific Advisory Board for Headspace, for which he receives compensation, and has received consulting payments from Otsuka Pharmaceuticals and K Health (Trusst) for unrelated work.
- Institute Report. Digital Health Trends 2021: Innovation, Evidence, Regulation, and Adoption. IQVIA Institute for Human Data Sciences. 2021 Jul 22. URL: https://www.iqvia.com/insights/the-iqvia-institute/reports/digital-health-trends-2021 [accessed 2022-06-03]
- Wasil AR, Gillespie S, Shingleton R, Wilks CR, Weisz JR. Examining the reach of smartphone apps for depression and anxiety. Am J Psychiatry 2020 May 01;177(5):464-465. [CrossRef] [Medline]
- Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [CrossRef] [Medline]
- Schueller SM, Stiles-Shields C, Yarosh L. Online treatment and virtual therapists in child and adolescent psychiatry. Child Adolesc Psychiatr Clin N Am 2017 Jan;26(1):1-12 [FREE Full text] [CrossRef] [Medline]
- Uhlhaas P, Torous J. Digital tools for youth mental health. NPJ Digit Med 2019 Oct 18;2:104 [FREE Full text] [CrossRef] [Medline]
- Baghaei N, Naslund JA, Hach S, Liang HN. Editorial: designing technologies for youth mental health. Front Public Health 2020 Feb 25;8:45 [FREE Full text] [CrossRef] [Medline]
- Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun;62(6):593-602. [CrossRef] [Medline]
- Owens PL, Hoagwood K, Horwitz SM, Leaf PJ, Poduska JM, Kellam SG, et al. Barriers to children's mental health services. J Am Acad Child Adolesc Psychiatry 2002 Jun;41(6):731-738. [CrossRef] [Medline]
- Cummings JR, Allen L, Clennon J, Ji X, Druss BG. Geographic access to specialty mental health care across high- and low-income US communities. JAMA Psychiatry 2017 May 01;74(5):476-484 [FREE Full text] [CrossRef] [Medline]
- Kataoka SH, Zhang L, Wells KB. Unmet need for mental health care among U.S. children: variation by ethnicity and insurance status. Am J Psychiatry 2002 Sep;159(9):1548-1555. [CrossRef] [Medline]
- Marrast L, Himmelstein DU, Woolhandler S. Racial and ethnic disparities in mental health care for children and young adults: a national study. Int J Health Serv 2016 Oct;46(4):810-824. [CrossRef] [Medline]
- Lokko HN, Chen JA, Parekh RI, Stern TA. Racial and ethnic diversity in the US psychiatric workforce: a perspective and recommendations. Acad Psychiatry 2016 Dec;40(6):898-904. [CrossRef] [Medline]
- McGuire TG, Miranda J. New evidence regarding racial and ethnic disparities in mental health: policy implications. Health Aff (Millwood) 2008;27(2):393-403 [FREE Full text] [CrossRef] [Medline]
- Rickwood DJ, Deane FP, Wilson CJ. When and how do young people seek professional help for mental health problems? Med J Aust 2007 Oct 01;187(S7):S35-S39. [CrossRef] [Medline]
- McMiller WP, Weisz JR. Help-seeking preceding mental health clinic intake among African-American, Latino, and Caucasian youths. J Am Acad Child Adolesc Psychiatry 1996 Aug;35(8):1086-1094. [CrossRef] [Medline]
- Rideout V, Robb MB. The Common Sense census: media use by teens and teens. Common Sense Media. 2019. URL: https://www.commonsensemedia.org/sites/default/files/research/report/2019-census-8-to-18-full-report-updated.pdf [accessed 2022-06-03]
- Teens, social media and technology 2018. Pew Research Center. 2018 May 31. URL: https://www.pewresearch.org/internet/wp-content/uploads/sites/9/2018/05/PI_2018.05.31_TeensTech_FINAL.pdf [accessed 2022-05-30]
- Rideout V, Fox S, Peebles A, Robb MB. Coping with COVID-19: how young people use digital media to manage their mental health. Common Sense Media. 2021. URL: https://amerispeak.norc.org/content/dam/amerispeak/research/pdf/Coping%20with%20COVID-19%20-%20How%20Young%20People%20Use%20Digital%20Media%20to%20Manage%20Their %20Mental%20Health.pdf [accessed 2022-06-02]
- Schueller SM, Hunter JF, Figueroa C, Aguilera A. Use of digital mental health for marginalized and underserved populations. Curr Treat Options Psych 2019 Jul 5;6(3):243-255. [CrossRef]
- Ramos G, Chavira DA. Use of technology to provide mental health care for racial and ethnic minorities: evidence, promise, and challenges. Cogn Behavi Pract 2022 Feb;29(1):15-40. [CrossRef]
- Coffman J, Bates T, Geyn I, Spetz J. California’s current and future behavioral health workforce. Healthforce Center at UCSF. 2018 Feb 12. URL: https://healthforce.ucsf.edu/sites/healthforce.ucsf.edu/files/publication-pdf/California%E2%80%99s%20Current%20and%20Future%20Behavioral%20Health%20Workforce.pdf [accessed 2022-06-02]
- Ku BS, Li J, Lally C, Compton MT, Druss BG. Associations between mental health shortage areas and county-level suicide rates among adults aged 25 and older in the USA, 2010 to 2018. Gen Hosp Psychiatry 2021;70:44-50 [FREE Full text] [CrossRef] [Medline]
- Alexandre PK, Martins SS, Richard P. Disparities in adequate mental health care for past-year major depressive episodes among Caucasian and Hispanic youths. Psychiatr Serv 2009 Oct;60(10):1365-1371. [CrossRef] [Medline]
- Linardon J, Cuijpers P, Carlbring P, Messer M, Fuller-Tyszkiewicz M. The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry 2019 Oct;18(3):325-336 [FREE Full text] [CrossRef] [Medline]
- Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, et al. Mobile apps for mental health issues: meta-review of meta-analyses. JMIR Mhealth Uhealth 2020 May 29;8(5):e17458 [FREE Full text] [CrossRef] [Medline]
- Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual Research Review: digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry 2017 Apr;58(4):474-503. [CrossRef] [Medline]
- Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res 2017 May 25;19(5):e176 [FREE Full text] [CrossRef] [Medline]
- Punukollu M, Marques M. Use of mobile apps and technologies in child and adolescent mental health: a systematic review. Evid Based Ment Health 2019 Nov;22(4):161-166. [CrossRef] [Medline]
- Bergin AD, Vallejos EP, Davies EB, Daley D, Ford T, Harold G, et al. Preventive digital mental health interventions for children and young people: a review of the design and reporting of research. NPJ Digit Med 2020 Oct 15;3:133 [FREE Full text] [CrossRef] [Medline]
- Stiles-Shields C, Cummings C, Montague E, Plevinsky JM, Psihogios AM, Williams KD. A call to action: using and extending human-centered design methodologies to improve mental and behavioral health equity. Front Digit Health 2022 Apr 25;4:848052 [FREE Full text] [CrossRef] [Medline]
- Michel T, Tachtler F, Slovak P, Fitzpatrick G. A review of youth mental health promotion apps towards their fit with youth media preferences. EAI Endorsed Trans Pervasive Health Technol 2019 Feb 12;19(17):e2 [FREE Full text] [CrossRef]
- Yarosh S, Schueller SM. “Happiness inventors”: informing positive computing technologies through participatory design with children. J Med Internet Res 2017 Jan 17;19(1):e14 [FREE Full text] [CrossRef] [Medline]
- Thabrew H, Fleming T, Hetrick S, Merry S. Co-design of eHealth interventions with children and young people. Front Psychiatry 2018 Oct 18;9:481 [FREE Full text] [CrossRef] [Medline]
- Bevan Jones R, Stallard P, Agha SS, Rice S, Werner-Seidler A, Stasiak K, et al. Practitioner review: co-design of digital mental health technologies with children and young people. J Child Psychol Psychiatry 2020 Aug;61(8):928-940 [FREE Full text] [CrossRef] [Medline]
- Stoyanov SR, Zelenko O, Staneva A, Kavanagh DJ, Smith C, Sade G, et al. Development of the niggle app for supporting young people on their dynamic journey to well-being: co-design and qualitative research study. JMIR Mhealth Uhealth 2021 Apr 20;9(4):e21085 [FREE Full text] [CrossRef] [Medline]
- Alqahtani F, Orji R. Insights from user reviews to improve mental health apps. Health Informatics J 2020 Sep;26(3):2042-2066 [FREE Full text] [CrossRef] [Medline]
- Oyebode O, Alqahtani F, Orji R. Using machine learning and thematic analysis methods to evaluate mental health apps based on user reviews. IEEE Access 2020 Jun 12;8:111141-111158. [CrossRef]
- MacLeod H, Jelen B, Prabhakar A, Oehlberg L, Siek K, Connelly K. A guide to using asynchronous remote communities (ARC) for researching distributed populations. EAI Endorsed Trans Pervasive Health Technol 2017 Jul 18;17(11):e4. [CrossRef]
- MacLeod H, Jelen B, Prabhakar A, Oehlberg L, Siek K, Connelly K. Asynchronous remote communities (ARC) for researching distributed populations. In: Proceedings of the 10th EAI International Conference on Pervasive Computing Technologies for Healthcare. 2016 May Presented at: PervasiveHealth '16; May 16-19, 2016; Cancun, Mexico p. 1-8. [CrossRef]
- Maestre JF, MacLeod H, Connelly CL, Dunbar JC, Beck J, Siek KA, et al. Defining through expansion: conducting asynchronous remote communities (arc) research with stigmatized groups. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018 Presented at: CHI '18; April 21-26, 2018; Montreal, Canada p. 1-13. [CrossRef]
- Bhattacharya A, Liang C, Zeng EY, Shukla K, Wong ME, Munson SA, et al. Engaging teenagers in asynchronous online groups to design for stress management. In: Proceedings of the 18th ACM International Conference on Interaction Design and Children. 2019 Jun Presented at: IDC '19; June 12-15, 2019; Boise, ID, USA p. 26-37. [CrossRef]
- Ito M, Odgers C, Schueller S, Cabrera J, Conaway E, Cross R, et al. Social Media and Youth Wellbeing: What We Know and Where We Could Go. Connected Learning Alliance. 2020 Jun. URL: https://clalliance.org/wp-content/uploads/2020/06/Social-Media-and-Youth-Wellbeing-Report.pdf [accessed 2022-06-02]
- Poole ES, Peyton T. Interaction design research with adolescents: methodological challenges and best practices. In: Proceedings of the 12th International Conference on Interaction Design and Children. 2013 Presented at: IDC '13; June 24-27, 2013; New York, NY, USA p. 211-217. [CrossRef]
- Dow S, Fortuna J, Schwartz D, Altringer B, Schwartz DL, Klemmer SR. Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. In: Proceedings of the 2011 SIGCHI Conference on Human Factors in Computing Systems. 2011 Presented at: CHI '11; May 7–12, 2011; Vancouver, Canada p. 2807-2816. [CrossRef]
- Haldar S, Mishra SR, Khelifi M, Pollack AH, Pratt W. Beyond the patient portal: supporting needs of hospitalized patients. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019 Presented at: CHI '19; May 4-9, 2019; Glasgow, UK p. 1-14. [CrossRef]
- The Visual Collaboration Platform for Every Team. miro. URL: https://miro.com/ [accessed 2022-06-02]
- Discord. URL: https://discord.com/ [accessed 2022-06-02]
- Hudson G, Negbenose E, Neary M, Jansli SM, Schueller SM, Wykes T, et al. Comparing professional and consumer ratings of mental health apps: mixed methods study. JMIR Form Res 2022 Sep 23;6(9):e39813 [FREE Full text] [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
- Charmaz K. Constructing Grounded Theory. 2nd edition. Thousand Oaks, CA, USA: Sage Publications; 2006.
- Saldaña J. The Coding Manual for Qualitative Researchers. 4th edition. Thousand Oaks, CA, USA: Sage Publications; 2021.
- McDonald N, Schoenebeck S, Forte A. Reliability and inter-rater reliability in qualitative research: norms and guidelines for CSCW and HCI practice. Proc ACM Hum Comput Interact 2019 Nov 07;3(CSCW):1-23. [CrossRef]
- Odgers CL, Schueller SM, Ito M. Screen time, social media use, and adolescent development. Annu Rev Dev Psychol 2020 Dec 15;2(1):485-502. [CrossRef]
- Aguilera A, Berridge C. Qualitative feedback from a text messaging intervention for depression: benefits, drawbacks, and cultural differences. JMIR Mhealth Uhealth 2014 Nov 05;2(4):e46 [FREE Full text] [CrossRef] [Medline]
- Moreno MA, Uhls YT. Applying an affordances approach and a developmental lens to approach adolescent social media use. Digit Health 2019 Jan 31;5:2055207619826678 [FREE Full text] [CrossRef] [Medline]
- Psihogios AM, Lane-Fall MB, Graham AK. Adolescents are still waiting on a digital health revolution: accelerating research-to-practice translation through design for implementation. JAMA Pediatr 2022 Jun 01;176(6):545-546. [CrossRef] [Medline]
- Torous JB, Chan SR, Gipson SY, Kim JW, Nguyen TQ, Luo J, et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv 2018 May 01;69(5):498-500. [CrossRef] [Medline]
- Agarwal S, Jalan M, Wilcox HC, Sharma R, Hill R, Pantalone E, et al. Evaluation of Mental Health Mobile Applications. Technical Brief No. 41. Rockville, MD, USA: Agency for Healthcare Research and Quality; 2022.
- Lagan S, Emerson MR, King D, Matwin S, Chan SR, Proctor S, et al. Mental health app evaluation: updating the American Psychiatric Association's framework through a stakeholder-engaged workshop. Psychiatr Serv 2021 Sep 01;72(9):1095-1098. [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
- Lewis JR. The system usability scale: past, present, and future. Int J Human Comput Interact 2018 Mar 30;34(7):577-590. [CrossRef]
- Lau AS, Kim JJ, Nguyen DJ, Nguyen HT, Kodish T, Weiss B. Effects of preference on outcomes of preventive interventions among ethnically diverse adolescents at-risk of depression. J Clin Child Adolesc Psychol 2020;49(6):820-836 [FREE Full text] [CrossRef] [Medline]
- Balaskas A, Schueller SM, Cox AL, Doherty G. Understanding users' perspectives on mobile apps for anxiety management. Front Digit Health 2022 Sep 1;4:854263 [FREE Full text] [CrossRef] [Medline]
- Mata-Greve F, Johnson M, Pullmann MD, Friedman EC, Griffith Fillipo I, Comtois KA, et al. Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: cross-sectional survey study. JMIR Ment Health 2021 Aug 05;8(8):e28360 [FREE Full text] [CrossRef] [Medline]
- Lagan S, D'Mello R, Vaidyam A, Bilden R, Torous J. Assessing mental health apps marketplaces with objective metrics from 29,190 data points from 278 apps. Acta Psychiatr Scand 2021 Aug;144(2):201-210. [CrossRef] [Medline]
- Lattie EG, Schueller SM, Sargent E, Stiles-Shields C, Tomasino KN, Corden ME, et al. Uptake and usage of IntelliCare: a publicly available suite of mental health and well-being apps. Internet Interv 2016 May;4(2):152-158 [FREE Full text] [CrossRef] [Medline]
- Kwasny MJ, Schueller SM, Lattie E, Gray EL, Mohr DC. Exploring the use of multiple mental health apps within a platform: secondary analysis of the IntelliCare field trial. JMIR Ment Health 2019 Mar 21;6(3):e11572 [FREE Full text] [CrossRef] [Medline]
- Liang CA, Albertson K, Williams F, Inwards-Breland D, Munson SA, Kientz JA, et al. Designing an online sex education resource for gender-diverse youth. In: Proceedings of the Interaction Design and Children Conference. 2020 Presented at: IDC '20; June 21-24, 2020; London, UK p. 108-120. [CrossRef]
- Fails JA, Ratakonda DK, Koren N, Elsayed-Ali S, Bonsignore E, Yip J. Pushing boundaries of co-design by going online: lessons learned and reflections from three perspectives. Int J Child Comput Interact 2022 Sep;33:100476 [FREE Full text] [CrossRef] [Medline]
- Lee KJ, Roldan W, Zhu TQ, Saluja HK, Na S, Chin B, et al. The show must go on: a conceptual model of conducting synchronous participatory design with children online. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021 May Presented at: CHI '21; May 8-13, 2021; Yokohama, Japan p. 1-16. [CrossRef]
- Ramos G, Ponting C, Labao JP, Sobowale K. Considerations of diversity, equity, and inclusion in mental health apps: a scoping review of evaluation frameworks. Behav Res Ther 2021 Dec;147:103990 [FREE Full text] [CrossRef] [Medline]
|ARC: asynchronous remote community|
|CBT: cognitive behavioral therapy|
|MHapp: mental health app|
Edited by A Mavragani; submitted 02.07.22; peer-reviewed by KT Win, G Ramos; comments to author 24.07.22; revised version received 12.09.22; accepted 13.09.22; published 18.10.22Copyright
©Elena Agapie, Katherine Chang, Sneha Patrachari, Martha Neary, Stephen M Schueller. Originally published in JMIR Formative Research (https://formative.jmir.org), 18.10.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.