Adherence of the #Here4U App – Military Version to Criteria for the Development of Rigorous Mental Health Apps

Introduction

Background

Recent advances in and increased access to technology have created opportunities to shift mental health support services away from traditional person-delivered models toward those that are more technologically based, including those that may be provided through smartphone apps. The ease of use and availability of these apps are ideal for populations who may face logistical or social barriers to traditional service delivery, including members of the Canadian Armed Forces (CAF).

Military life is characterized by frequent relocations and family absences, and increased risk for illness, injury, and death [1]. Research has also identified how military life can affect health and well-being, including increasing risk for mental health difficulties and challenges with accessing health care services [2-4]. These events have been shown to impact all members of the community, including personnel [3,5,6], veterans [7], and their family members [8-10]. Despite the prevalence of mental health challenges faced by members of the CAF community, traditional mental health care continues to be underused. Commonly cited barriers include a preference for self-management [11], fear of impact on one’s military career [11], perceived stigma [12], and structural barriers such as living in remote locations and being frequently relocated [5]. As a result, a technology-based solution may better serve members of the military community.

The #Here4U App – Military Version (hereafter, #Here4U) is an app designed to provide evidence-informed mental health support to members of the CAF community, including serving personnel, veterans, and their adult family members. #Here4U uses artificial intelligence (AI) in the form of IBM’s Watson Assistant to carry on unique text-based conversations with users, identify presenting mental health concerns, and refer users to self-help resources or recommend professional health care where appropriate. Users are invited to converse with the chatbot and select their end point (ie, self-help solution) from several options, facilitating a user-directed experience. Options include (1) tip sheets, which provide the user with brief mental health education on the selected topic; (2) use of validated screening tools, which provide the user with a self-assessment of the severity of psychological distress they may be experiencing; (3) information on how to contact local supports, including counsellors, psychologists, and psychiatrists; (4) frequently asked questions (FAQs) about common mental disorders, offered as downloadable PDF; and (5) self-help resources, ranging from cognitive behavioral therapy (CBT) education and activities to simple at-home exercises such as tactical breathing, meditation, and progressive muscle relaxation. The beta version of #Here4U has been designed to serve members of the CAF, veterans, and adult military family members, with plans to expand the app’s applicability to additional target groups in the future. Figure 1 demonstrates the functionality of the app, using a flowchart design.

The #Here4U app was developed as an interdisciplinary project, in partnership with IBM Canada, the Queen’s University Centre for Advanced Computing, Kingston, ON, Canada, and the Queen’s University Health Services and Policy Research Institute. In addition to the research team, subject matter experts were engaged throughout the development process to share their experiences regarding common presenting mental health concerns among members of the CAF community, review resources provided by the app, and provide professional input and guidance on conversational flows. Subject matter experts included social workers, physicians (specializing in family medicine and psychiatry), psychologists, first responders, military family health researchers, child and youth workers, and members of the CAF community (active members, military spouses, family of adult military members, and veterans).

‎View this figure

Objective

As the availability and use of mental health apps has increased, so too has the list of recommendations for efficacious development. In 2016, Bakker and colleagues published a set of 16 evidence-based recommendations to support the development of rigorous apps [13]. Our objective is to demonstrate the rigorous development and testing processes undertaken between 2018 and 2020 to produce the beta version of the #Here4U app and, in doing so, evaluate the degree to which the app complies with each of the criteria laid out by Bakker and colleagues.

Methods

Development of the App

The first year of development included background research to inform the content of the app, in addition to the initial development of conversational flows. First, we undertook a preliminary study to determine the most common presenting mental health challenges within the CAF community, as well as to derive particular language and slang used in the community to discuss mental health–related challenges. This was necessary to train IBM’s Watson Assistant (IBM Corporation) to be able to recognize and interpret specific vernacular used within the community. We conducted a combination of online surveys (n=12), focus groups (n=17), and individual interviews (n=14) to achieve the goals of this phase of development, the results of which are described in detail elsewhere (available from the authors upon request). Ultimately, we identified 3 presenting concerns as the primary areas of focus: generalized anxiety, depression, and posttraumatic stress disorder (PTSD). We also identified several secondary concerns as priorities for later development (eg, substance use and addictive behaviors, general stress, difficulties related to sleep).

Next, we drew up initial logic trees to map likely conversational flows. These were then coded into the back end of the AI by members of the computing team. We initially designed conversational flows for secondary concerns to be more streamlined (ie, to program the AI to recognize the concern expressed and immediately offer potential resources), whereas flows for primary concerns were more intricate. From here, we selected useful, efficacious end points for these conversations with input from subject matter experts (eg, validated screening tools, evidence-based self-help resources and activity recommendations, tip sheets to facilitate improved mental health education).

Beta Testing

During the second year of development, we used a snowball sampling method to gather participants to test the beta version of the app. Participants were a mix of subject matter experts involved in app development, attendees at the Canadian Institute for Military and Veteran Health Research Forum in 2019, members of the CAF community (eg, active members, veterans, military family members, staff at Military Family Resource Centres, mental health professionals working within the military community), and personal contacts of members of the research team who were connected to military research in some way (n=93). Users were asked to engage in conversations with the app, submit their comments regarding usability and suggestions for improvement, and identify any problems with the conversation dialogue. For example, testers identified instances in which the AI misunderstood a term or comment, or drew incorrect conclusions, such as placing the user into a conversational flow for anxiety when they were expressing concerns related to depression. This allowed us to refine and expand the AI’s vocabulary, as well as to observe areas in need of improvement or expansion in the app. We made changes to the app in response to tester feedback using a waterfall approach, whereby we corrected conversational flows after receiving feedback from each tester.

Assessment of Rigorous Development

Finally, prior to conducting a formal evaluation of the app, we sought to assess whether we had met expectations regarding the recommendations for rigorous app development. We referred to Bakker and colleagues’ 16 criteria, outlining the degree to which the #Here4U app met (or did not meet) each of these criteria, emphasizing relevant features and functions of the app in doing so.

Ethics

All stages of research conducted to support the development of the #Here4U app were approved by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board. The research project was also approved by the Canadian Department of National Defence/CAF Social Science Research Review Board in accordance with Defense Administrative Orders & Directives 5062-0 and 5062-1 (#1172/18F).

Results

In this section, we address each of the 16 criteria put forward by Bakker and colleagues, highlighting where the #Here4U app met expectations and where selected criteria were not applicable.

Recommendation 1: Cognitive Behavioral Therapy Based

CBT is a common therapeutic approach used to treat several mental illnesses, such as anxiety and depression, designed to help individuals develop more adaptive methods of thinking and processing emotions and behavior [14]. The #Here4U app met this recommended criterion by providing users with CBT-based self-help resources, in the form of exercises and self-help workbooks. Users are also invited to access psychoeducational resources regarding the purpose and principles of CBT, including what the cognitive behavioral model looks like, and how to challenge problematic thinking by using tools such as recognizing cognitive distortions, decatastrophizing, and reframing.

Recommendation 2: Address Both Anxiety and Low Mood

The #Here4U app met this criterion by addressing 3 major mental health concerns common among the CAF community: anxiety, depression, and PTSD. We selected these concerns after primary research conducted among members of the CAF community revealed these to be the most common mental health concerns. User-led conversational flows and self-help resources are provided for each of these areas of concern, in addition to validated screening measures for assessing degree of distress [15-17]. The app is also equipped to manage multiple presenting concerns at a given time, using a “loop-back” function. For example, anxiety, depression, and PTSD are often comorbid conditions, among both the general population [18] and the military population [19]. If a user were to express concerns associated with more than one mental health concern in a single conversation, the Here4U app would initially direct the conversation toward the first concern, later returning—or looping back—to discuss additional challenges. In the event that a user expresses both primary and secondary concerns, the app was designed to prioritize primary concerns (eg, depression, anxiety, and PTSD), prior to returning to address secondary concerns. Figure 2 depicts the loop-back function. The ability to address multiple concerns during a single conversation ensures that users receive the most complete mental health information, in addition to eliminating the need to juggle multiple apps or multiple sessions to address different concerns.

‎View this figure

Recommendation 3: Designed for Use by Nonclinical Populations

The #Here4U app was specifically designed for use among nonclinical populations (ie, individuals who do not have a clinical diagnosis for a mental health disorder), with the app intended to be broadly applicable to a variety of individuals within the CAF community. While some of the screening tools contained in the app can be used to determine a clinical diagnosis when administered by a mental health professional (eg, the Patient Health Questionnaire-9), they are explicitly used here as measures of severity [16,17]. Apps that provide users with a likelihood of a clinical diagnosis have the potential to be harmful where users may be tempted to self-diagnose a mental illness when they may simply be experiencing low levels of distress [20]. Rather than providing a diagnosis or treatment, the #Here4U app was intended as an upstream measure (eg, with focus laid on mental health promotion and mental illness prevention). While the app can assist with directing users who may be experiencing more serious levels of psychological distress to a mental health professional for additional enquiry, the resources offered within the app were designed to provide users with the education to self-manage nonclinical, low levels of distress.

Recommendation 4: Automated Tailoring

The #Here4U app achieved automated tailoring through the use of IBM’s Watson Assistant AI agent, which is capable of interpreting user concerns (eg, referred to technically as entities, or key words, and intents, or key phrases) and connecting users to the most appropriate conversational flow. With this function, users do not have to sort through menu items to locate the information that is most relevant or useful to them.

As previously noted, the #Here4U app was designed to respond to 3 primary mental health concerns (anxiety, depression, and PTSD), each with its own end point solutions offered, rather than using a one-size-fits-all approach. Secondary conversational flows within the app allow for even more detailed tailoring. For example, changes in appetite and sleep are commonly linked to depression and anxiety among members of the CAF community [21,22], while experiencing chronic stress or pain can similarly lead to mental health difficulties [23,24]. Additionally, substance use and addictive behaviors are often adopted as negative coping mechanisms [25-27]. Many members of the CAF community have reported experiencing family-related challenges, such as frequent absences, child rearing, supporting an injured family member, and frequent relocations. In addition to primary concerns, the #Here4U app also offers brief conversational flows that provide resources for users experiencing any of these secondary concerns.

Recommendation 5: Reporting of Thoughts, Feelings, or Behaviors

While the #Here4U app does not track or store user information (see recommendation 12), this criterion was met by providing users with self-help resources that encourage the reporting of thoughts, emotions, and behaviors to improve awareness of changes in one’s mental health. For example, one option for users expressing concerns related to low mood is to begin using a daily gratitude journal, where users are invited to record three “good things” that happen every day. Users expressing challenges associated with stress management are provided with step-by-step guidelines on how to use progressive muscle relaxation as a stress reduction technique. All self-help resources provided through the app have been extracted or adapted from established programs (eg, Anxiety Canada) and have been reviewed by mental health professionals.

Recommendation 6: Recommend Activities

Many activities offered through the #Here4U app focus on mood improvement, such as encouraging users to engage in hobbies, leisure time, and physical activity and to spend time in nature. Behavioral activation is encouraged through CBT-based activities, including goal setting, cognitive distortion reframing, and decatastrophizing. Finally, users are invited to develop coping skills through improving their understanding of the importance of having a strong social support system and the adoption of coping activities such as progressive muscle relaxation, meditation, and tactical breathing.

Recommendation 7: Mental Health Information

The #Here4U app met this criterion by providing users with psychoeducation by offering tip sheets for anxiety, depression, and PTSD, as well as more detailed FAQ documents that cover areas such as common symptoms, treatments, and how to help someone else who may be struggling. General education on CBT techniques for improving and managing mental health is also provided as an option for users. All materials were developed using evidence-based, reputable resources and have been reviewed by mental health professionals to ensure accuracy of information.

Recommendation 8: Real-Time Engagement

The #Here4U app capitalizes on the real-time capabilities of mobile devices by encouraging users to engage with the recommended self-help activities and resources immediately upon accessing them. Within the app, users can complete screening tests, which provides them with an idea of the degree to which they may be experiencing psychological distress, using an interactive, question-and-answer format (Figure 3). While these screening tests do not provide users with a diagnosis due to the risks outlined in recommendation 3, the app can also link users to the Psychology Today search engine to help them find a psychologist or counsellor in their local area [28] or to the Calian Military Family Health Portal [29], where users can get help finding a family physician. Finally, the 24/7 availability of the app encourages users to reach out and access resources in real time whenever they experience a change in their mental health.

‎View this figure

Recommendation 9: Activities Explicitly Linked to Specific Reported Mood Problems

With its tailoring capabilities, combined with the ability to address multiple presenting issues in a single session, the #Here4U app links reported mood challenges to relevant activities, rather than using a one-size-fits-all approach. While there is some overlap between concerns (eg, CBT educational materials are provided for all users), a user experiencing symptoms of depression would be directed toward activities and self-help resources specifically designed to address low mood. Bakker and colleagues argued that linking activities in this way can improve engagement and encourage habit-forming use of the app [13].

Recommendation 10: Encourage Nontechnology-Based Activities

The #Here4U app met this criterion by encouraging users to engage in a selection of nontechnology-based activities to improve their mental health (eg, engaging in regular physical activity, healthy nutrition, or self-care hobbies or spending time in nature). All of these recommended activities have been linked to reducing stress and improving low mood and anxiety [30-33].

Recommendation 11: Gamification and Intrinsic Motivation to Engage

Gamification, or the use of game-based mechanics to improve users’ engagement, learning, and problem solving, can help to counteract problems related to lack of motivation and improve goal setting [13]. Currently, the #Here4U app does not contain any elements of gamification, but this may be an area for future development. Based on data derived from focus groups and interviews, gamification is a feature that potential users may find beneficial with respect to encouraging ongoing engagement with the app.

Recommendation 12: Log of Past App Use

During the first phase of research for the project, we consulted members of the CAF community and subject matter experts (eg, psychologists, counsellors, social workers) to determine, among other things, common barriers to using a mental health app. The most commonly cited barriers were the need for privacy and confidentiality, most often driven by concerns regarding the potential for mental illness stigma resulting in damage to one’s military reputation or career. For military families using the app, there was the concern that any kind of mental health challenge or family difficulty would reflect poorly on the military member. As a result of these findings, we made an informed decision to not implement any feature that would track app history. Bakker and colleagues mainly referred to this recommendation as a component of gamification, where users can track their progress week over week [13]. As the #Here4U app does not contain elements of gamification, logging past app use in this manner was not applicable.

Recommendation 13: Reminders to Engage

Though evidence suggests that providing users with reminders to engage (also known as push notifications) can help to improve and maintain engagement, too many reminders can also have the opposite effect, resulting in disengagement [13]. Given that users of the #Here4U app will be discussing sensitive topics related to mental health and emotional well-being, combined with the associated stigma within the military community [12] and a preexisting reluctance to seek help [3,5], we made an informed decision to not implement any push notifications.

Recommendation 14: Simple and Intuitive Interface and Interactions

The #Here4U app was designed to provide users with a sleek and simple interface. The app itself is driven by IBM’s Watson Assistant AI, which carries on text-based conversations with users to determine areas of concern and suggest potential self-care resources and other solutions. All components of the app, including tip sheets and FAQs to drive mental health education, self-help resources, and other activities, are offered within a single interface. Conversations and access to self-help resources are user led, meaning that users choose how in-depth to go with content.

Recommendation 15: Links to Crisis Support Services

Because the #Here4U app was not intended for use as a crisis service, it was imperative to ensure that the AI had the ability to accurately recognize expressions related to self-harm and suicidality. In this case, users are reminded that the app is not equipped to assist with acute mental health crises and are directed to contact Crisis Services Canada, their primary health care provider, or local hospital.

Recommendation 16: Experimental Trials to Establish Efficacy

To date, no evaluation work has been conducted on the efficacy of the #Here4U app, and this is therefore a major priority moving forward.

Discussion

Principal Findings

To our knowledge, the #Here4U app met more of Bakker and colleagues’ recommended criteria than any other available app targeting the mental health of military communities. Only 1 relevant criterion was not met: recommendation 16, to conduct experimental trials to establish efficacy of the app. While an evaluation study was out of scope for the development phase of the project, it is a recognizable gap for the #Here4U app and is therefore a major priority moving forward. Two other criteria were unmet, as they were not applicable to this project: recommendations 12 and 13, to maintain a log of past app use and to remind users to engage, respectively. Due to the sensitivity of the subject matter to be discussed within the app, combined with existing research regarding mental health–related stigma within the military community [12] and a preexisting reluctance to seek help [3,5], we made an informed decision early on to not implement any push notifications or information tracking within the #Here4U app.

Conclusion

Although not all criteria were met by the #Here4U app, Bakker and colleagues rightly noted that it may not be possible to design mental health apps that meet all suggested criteria. It is important to note, however, that those criteria that fall within the strongest ranks of evidence as determined by Bakker and colleagues (eg, recommendations 1-8) were all met by the #Here4U app [13]. Moving forward, an experimental evaluation study should be conducted to determine the efficacy of the app and user experiences more formally. The evaluation should incorporate assessments for effectiveness of the app in connecting users with mental health education and useful self-help resources, user interface performance and usability, and general user experience. Overall, the #Here4U app may be a promising new mental health e-solution for members of the CAF community, filling many of the gaps left by traditional service delivery.