Background: Young people experiencing low mood, thoughts related to self-harm, and suicidal ideation often struggle to communicate their emotions and receive timely support from family and friends. Technologically delivered support interventions may be useful in addressing this need.
Objective: This paper aimed to evaluate the acceptability and feasibility of “Village,” a communication app co-designed with young people and their family and friends from New Zealand.
Methods: A mixed methods pilot open trial design was adopted. Participants were primarily recruited via social media advertisements and clinicians in specialist mental health services over an 8-month period. The primary outcomes were acceptability of the app (via thematically analyzed qualitative feedback and retention rates) and feasibility of conducting a larger randomized controlled trial gauged via effectiveness of recruitment methods, completion of chosen outcome measures, and occurrence of unanticipated operational issues. Secondary outcomes were app usability, safety, and changes in symptoms of depression (via the Patient Health Questionnaire–9 modified for adolescents), suicidal ideation (on the Suicidal Ideation Questionnaire), and functioning (using the World Health Organization Disability Assessment Schedule 2.0 or Child and Youth version).
Results: A total of 26 young people (“users”) were enrolled in the trial, of which 21 recruited friends and family members (“buddies”) and completed quantitative outcome measures at baseline, 4 weeks, and 3 months. Furthermore, 13 users and 12 buddies also provided qualitative feedback about the app, identifying the key themes of appeal of app features and layout, usefulness of its content, and technological challenges (primarily with onboarding and notifications). Users gave Village a mean rating of 3.8 (range 2.7-4.6) out of 5 on a 5-point scale for app quality and an overall star rating of 3.4 out of 5 for subjective quality. Within this limited sample, users reported a clinically significant reduction in depressive symptoms (P=.007), but nonsignificant changes in suicidal ideation and functioning. The embedded risk detection software was activated on 3 occasions, and no additional support was required for users.
Conclusions: During this open trial, Village was found to be acceptable, usable, and safe. The feasibility of a larger randomized controlled trial was also confirmed after some modifications to the recruitment strategy and app.
Trial Registration: Australian New Zealand Clinical Trials Network Registry ACTRN12620000241932p; https://tinyurl.com/ya6t4fx2
Internationally, rates of mental distress and depression among young people have steadily increased over the past few decades [- ]. In the wake of COVID-19, they are predicted to rise further as part of a “long psychosocial tail” of pandemic-related consequences [ , ]. Young people experiencing these issues often struggle to verbally express strong and fluctuating emotions and reach out to their families or peers for timely assistance [ , ]. Instead, they may resort to the use of maladaptive coping strategies, including impulsive or planned self-harm [ ]. Self-harm is defined as intentional self-injury or self-poisoning, irrespective of the extent of suicidal intent [ ]. In New Zealand, approximately 24% of high school students were reported to be engaging in self-harm (17.9% of males and 29.1% of females aged 13-19 years) [ ]. The consequences of self-harm include hospitalization (80.8 per 100,000 males and 212 per 100,000 females aged 15-19 years) [ ] and suicide [ ]. Currently, owing to a likely combination of social and family-related factors [ ], New Zealand holds the dubious honor of having the highest rate of suicide among young people in the developed world (19.3 per 100,000 young people and even higher—36.4 per 100,000 among Indigenous Māori young people) [ ].
Over the past 20 years, rapidly evolving smart technology has led to the development of a range of digital health (eHealth) interventions, including those specifically designed to improve mental health . These include information-oriented websites for learning about stress management, game-based therapies on the web for common mental health problems, and mobile health self-help apps to improve well-being [ ]. Such interventions offer particular appeal to young people who have grown up as “digital natives” [ ]. Some have been shown to be as effective as face-to-face therapies [ ], and as such, are now recommended by the National Institute for Clinical Excellence in the United Kingdom as a first-line treatment for adolescent anxiety and depression [ ] and by international groups such as the Lancet Global Mental Health Group [ ] as a scalable means of addressing common mental health issues. Purported advantages of eHealth interventions include cost-effectiveness, flexibility of use, and potential for increasing equity of treatment access and reducing stigma [ ]. These are reflected in people’s keenness to use eHealth interventions [ ]. Recent reviews have confirmed the benefits of mobile health interventions in improving well-being and suicide-related psychosocial outcomes [ , ].
Alongside these substantial technological developments, there has been a growing awareness that self-empowerment of well-being , family-inclusive approaches to health [ ], and peer support [ ] can improve the quality of care and outcomes for people experiencing mental health issues. Empowering people to manage their own well-being in the first instance is encouraged by consumer organizations and the World Health Organization 2013-2020 Mental Health Action Plan [ ]. Family-inclusive approaches have also been shown to improve resilience among young people [ ]. Recent evidence from King et al [ ] in the United Kingdom suggests that, despite the lackluster performance of widely used suicide risk management and scoring systems, youth-nominated support teams (consisting of peers and family members) can actually be effective in reducing rates of suicidal ideation in the short term and rates of suicide among young people for over a decade [ ]. The key ingredients identified from this relatively novel approach are social support, suicide prevention literacy, and adolescent skill development (learning to communicate and accept help from others) [ ]. To date, such support has not been tested using eHealth approaches.
With these issues in mind, in 2020, our team undertook a co-design process to develop a working prototype for a digital, youth-nominated support system that might benefit young people experiencing low mood, self-harm, and suicidal ideation. Approximately 40 New Zealand youth (including many who had experienced these issues or were affiliated with a national telephone-based support organization [Youthline]), 20 family members, 3 mental health clinicians, and a team of 6 IT specialists from Datacom, one of New Zealand’s largest IT companies, were involved in the app’s co-design. After a 6-month, agile, iterative process that included sprints of app development and “scrums” of product review , “Village,” an innovative communication app (ie, an app primarily focused on improving communication between sets of users) was developed for pilot evaluation. Mental health–related content was initially drafted by the primary investigator (a child and adolescent psychiatrist) and refined after consultation with other mental health clinicians and user feedback. The purpose of Village is to help users (young people) obtain regular and timely support from a network of buddies (family members, whanau, or peers), who are in turn educated and supported to respond appropriately to their messages. During the co-design process, younger adolescents voiced a preference for receiving support from family members, whereas older adolescents and adults stated that they would prefer support via peers. As such, the app was consciously developed to meet the needs of adolescent and adult users as well as those of adolescent and adult supporters. The design of Village is underpinned by four theoretical constructs:
- The nonviolent communication model helps people recognize their specific emotions and needs before they engage in self-empathy or any kind of conflict or problem-solving involving other people. Once self-awareness is gained, it teaches people to make requests of others that are specific and free of demand [ ].
- Systems theory is a philosophy that focuses on the interdependence of individuals in a group to help understand and optimize the achievements of the system [ ]. It also aligns with New Zealand Māori concept of “Whanau Ora,” which identifies connection with family (“whanau”) as an integral part of good health [ ].
- Supportive therapy is designed to reduce psychological conflict and strengthen a person’s defenses through the use of various techniques such as reassurance, suggestion, counseling, and education [ ].
- Strength-based therapy seeks to focus a person’s attention on their own strengths, resourcefulness, and resilience to help them recognize positive aspects of their identity and to use these qualities to move forward [ ].
The primary aim of this open trial was to evaluate the acceptability and usability of the prototype app. Second, we sought to undertake a preliminary evaluation of the app’s usability, safety, and efficacy in altering mood, suicidal ideation, and functioning, as well as the feasibility of undertaking a more definitive randomized controlled trial (RCT). The key hypotheses were that the Village app would be acceptable, usable, and safe for users and buddies; that its use would lead to an improvement in mood and functioning and a reduction in suicidal ideation; and that an RCT using chosen or modified outcome measures and parameters would be feasible.
The single-arm, open trial of Village was conceptualized by 2 authors (HT and ES), and a mixed methods design was used. Participants were initially recruited by 2 authors (HK and HT) via specialist child and adolescent mental health services and primary health services in Auckland, New Zealand. Owing to low enrollment rates between November 2020 and February 2021 (most likely because of COVID-19–related service disruption), after the amendment of ethics approval, recruitment was expanded to social media advertisements (Facebook and Instagram linked to REDCap [Research Electronic Data Capture; Vanderbilt University], which recorded the number of individuals who signed up) between March 2021 and July 2021. Participants were selected by convenience and were supported to download the app via written and verbal instructions at the time of recruitment. They were encouraged to use the app as they wished to (with no set frequency) for 4 weeks. They were prompted (via automated REDCap messages and text if there was no response within 48 hours) to complete web-based quantitative outcome measures via REDCap at 3 time points: baseline, 4 weeks, and 3 months. After the use of Village for 4 weeks, more in-depth qualitative information regarding user experiences and feedback from buddies was collected by 1 author (HK, a female research assistant with prior experience of qualitative research and no prior relationship with participants before the study) on a single occasion, in person (one-on-one), at university premises via individual semistructured interviews lasting for 30-60 minutes. The interview schedule was developed by the research team and was road-tested with a couple of young people and adults. Interviewees were informed that questions were being asked to evaluate and improve the app before it became publicly available. The interviews were audio-recorded and transcribed by a certified transcriber who signed a confidentiality agreement. Data analysis and interpretation were undertaken by 2 authors (HT and HK). The paper was drafted by the first author (HT) and reviewed by all the authors (HT, HK, and ES) before submission.
Young people, aged between 16 and 25 years and of any gender, were invited to participate in the study. Eligible participants were experiencing low mood (self-reported, no quantified cutoff score), self-harm, or suicidal ideation; receiving psychotherapy or medication treatment; spoke adequate English to use the app; had access to a smartphone (iPhone or Android mobile); were able to provide electronic or written informed consent; and were able to nominate at least one buddy aged >16 years (as agreed with the ethics committee). Individuals who did not meet these criteria or who were currently receiving dialectical behavioral therapy (DBT), which includes active family and therapist support, were excluded. Each young person identified one of their friends to be invited to participate in follow-up interviews. These individuals provided separate written consent and demographic information before being interviewed.
Village is a communication app that helps young people experiencing low mood, self-harm, or suicidal ideation connect with a self-nominated support network of peers or family members. Young people (“users”) nominate and, via a text link, invite between 1 and 5 support people (“buddies”), to whom they can either send (the same or different) messages as often as they want and from whom they can receive daily check-ins and support. Users and buddies receive instructions on how to use the app through a series of onboarding messages. Users are guided to create messages letting buddies know how they are feeling and why, as well as what support they would ideally like them to provide. Buddies are coached via the app to respond sensitively to user messages. They were also provided with information about communication, common mental health issues (such as anxiety, depression, and self-harm), and what to do if they were worried about users’ safety. No additional or external support was provided to buddies during the trial. Software built into Village detects “risky messages” based on previously proven keywords for detecting high risk adolescents’ behavior on the web such as “suicide,” “kill myself,” “end my life,” “useless,” “want to die,” “dead,” and “worthless” . This software alerts users and gives them the option of seeking immediate help via a national telephone helpline or during the trial, being contacted by a member of the research team (HT) within 24 hours. All data were stored on user (and buddy) devices and erased when the app was deleted. For privacy reasons, no data were sent to external servers. Screenshots of Village are presented in - .
Acceptability of Village was qualitatively evaluated via thematically analyzed feedback from user and buddy interviews and retention rates after enrollment. The Feasibility of conducting an RCT was evaluated based on the proportion and speed of recruitment via different means, completion of chosen outcome measures, and occurrence of unanticipated operational issues. Usability was quantitatively evaluated via the user version of the Mobile Application Rating Scale (uMARS)  and qualitatively evaluated via feedback from user and buddy interviews (see for semistructured interview questions). The uMARS is a validated and reliable self-reported scale that consists of 6 sections (app engagement, functionality, esthetics, information, subjective impact, and perceived impact) and 3 subscales (overall app quality, subjective quality, and perceived impact). The uMARS questionnaire has a high internal consistency (Cronbach α=.90) and good test-retest reliability for the total scale. Safety was evaluated based on the frequency and outcome of risk detection software activation and self-reported episodes of self-harm or hospitalization during the follow-up interviews. Efficacy was evaluated by measuring changes over time in symptoms of major depressive disorder using the Patient Health Questionnaire–9 modified for adolescents [ ], suicidal ideation using the Suicidal Ideation Questionnaire (SIQ) [ ], and functioning using the World Health Organization Disability Assessment Schedule (WHODAS) 2.0 for participants aged >18 years and WHODAS–Children and Youth (WHODAS-CY) for participants aged <18 year [ , ]. Further details of the outcome measurement schedule are presented in . The Patient Health Questionnaire–9 is a 9-item questionnaire that assesses the degree of depression severity and has moderate internal consistency (Cronbach α=.7−.93) and good internal reliability (Cronbach α=.89). The level of severity score included ranges from 0-4 (minimal), 5-9 (mild), 10-14 (moderate), 15-19 (moderately severe), and 20-27 (severe). The SIQ is a 30-item questionnaire that measures the frequency of suicidal thoughts among adolescents. It has high internal consistency (Cronbach α=.97) and test-retest reliability (Cronbach α=.86). A score of ≥41 indicates that the individual requires further evaluation of the psychopathology and suicide risk. The WHODAS 2.0 is a 36-item scale that consists of six domains that measure the level of functioning of an adult in the last 30 days within the following areas: (1) cognition, (2) mobility, (3) self-care, (4) getting along, (5) life activities, and (6) participation. The scale has high internal consistency (Cronbach α=.98) and test-retest reliability (Cronbach α=.92). The WHODAS-CY has also been shown to have a high internal reliability (Cronbach α=.84). It measures all areas of functioning as per the WHODAS 2.0 and includes a section regarding school. Both WHODAS measures contain the same score ranging from 0 to 100 (0=no disability and 100=full disability).
|Measures||Baseline||4-week follow-up||3-month follow-up|
aPHQ-A: Patient Health Questionnaire–9 modified for adolescents.
b✓: denotes completion of measures.
cSIQ: Suicidal Ideation Questionnaire.
dWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.
eWHODAS-CY: World Health Organization Disability Assessment Schedule–Children and Youth.
fuMARS: user version of the Mobile Application Rating Scale.
Data Analysis and Statistical Methodology
Quantitative data were analyzed by 2 authors (HK and HT) using Microsoft Excel (version 16.52) and SPSS (version 27.0, IBM Corp). Analyses included basic descriptive statistics for demographic characteristics and changes in depression, suicidal ideation, and functioning. Linear mixed models were used to assess changes in depression, suicidal ideation, and functioning over time (between baseline, 4-week, and 12-week follow-up), accounting for repeated within-person measures over the study period. This analysis enabled the inclusion of information from individuals who did not complete or did not have data at all the 3 time points. P values of <.05 were considered statistically significant and 95% CIs were presented. Qualitative data from semistructured interviews of users and buddies were analyzed by 2 authors (HT and HK) using NVivo (QSR International) software and the 6-stage method of thematic analysis by Braun and Clarke  (familiarization, coding, generating themes from the data, reviewing themes, defining and naming themes, and writing up). The transcripts were not returned to participants for review. Key themes, subthemes, and supporting quotes were identified and coding discrepancies, if any, were addressed by consensus.
This study was approved by the New Zealand Health and Disability Ethics Committee (20/NTB/116). Users recruited via clinical service recruits and interviewed buddies provided written consent, and users recruited through the web provided electronic consent via REDCap. Written data were stored in secure filing cabinets, and electronic data were stored on a secure server, as per the University of Auckland regulations. A privacy impact assessment conducted at the request of the ethics committee rated the study as “low risk.” Although users had the option of being put through a telephone helpline if risky messages were detected by the app, none of their information was forwarded to any external agencies. All participants were encouraged to disclose any potential adverse events, including self-harm, suicide attempts, and hospitalization, to the research team at any stage of the study, and they were specifically asked about these events during the follow-up interviews. A log of these events and the number, dates, and outcomes of risky messages were maintained by the research team. Owing to the nature of the study, no external data safety monitoring committee was used. App users were informed that they were free to withdraw from the study at any stage. Regardless of the duration of the participants’ involvement, all enrolled users and buddies received a NZ $50 (US $30) gift voucher on exit from the study.
A total of 321 young people were made aware of the study, 23 via child and adolescent mental health services, 5 via general practice services, and 293 via social media advertisements. A total of 14 clinicians were involved in referring young people from the services mentioned above. Of those who knew about the study, 156 young people did not meet the eligibility criteria; 78 did not respond to contact from the research team about receiving more information regarding the trial; 58 declined to participate because they were not interested, their mental health was stable, or they did not have a suitable buddy; and 8 agreed to participate, but did not complete baseline measures and did not download the app. This resulted in 26 users, 21 of whom managed to enroll ≥1 buddies and complete the trial. The demographic characteristics of both groups are presented in. Unfortunately, because the REDCap forms are not mandatory, not all outcome measures were completed at all time points. At 12 weeks, all users of Village and their nominated buddies were invited to be interviewed about their experiences with the app. Of these, 13 users and 12 buddies agreed to participate.
|Participantsa, n (%)|
|Young people or users (n=26)|
|Registered and completed baseline outcome measures||26 (100)|
|Completed 4 and 12-week outcome measures||21 (81)|
|Completed follow-up interviews||13 (50)|
|Completed follow-up interviews||12 (57)|
|Age (years), mean (range)|
|Young people or users||17.7 (16-25)|
|Gender, n (%)|
|Young people or users (n=26)|
|Preferred not to answer||1 (5)|
|Ethnicity, n (%)|
|Young people or users (n=26)|
|New Zealand European||17 (65)|
|New Zealand European||12 (57)|
|Reason for participation and relationship to young persond, n (%)|
|Young people or users (n=26)|
|Low mood||23 (88)|
|Suicidal ideation||13 (50)|
|Place of recruitment, n (%)|
|Young people or users (n=26)|
|Child and adolescent mental health service||8 (31)|
|General practice||4 (15)|
|Social media||14 (54)|
|Invited by a young person||—e|
aOne buddy did not complete the demographic form.
bMELAA: Middle Eastern, Latin American, and African.
cOther: Fijian, Russian, and American.
dReason for participation: participants had the opportunity to select multiple reasons for participating in the trial. Other—hallucinations and dissociation identity disorder (n=1), anxiety and panic attacks (n=3), family issues (n=1), attention-deficit or hyperactivity disorder (n=1), and interest in marketing research (n=1).
Three key themes emerged from the analysis of user and buddy feedback: (1) appeal of app features and layout, (2) usefulness of content, and (3) technological challenges. Although most users and buddies found Village appealing to use and easy to navigate, a few experienced difficulties downloading or using the app for the first time. This was particularly an issue for buddies who needed to be invited by users and did not have access to support from one of the research team (HK). Both users and buddies found Village helpful in constructing useful messages. However, some who had prior experience with mental health services found guided responses “robotic” and said they would prefer greater freedom to compose more personally relevant messages. The themes, subthemes, and supporting examples are summarized in. Most of the recruited participants (21/26, 81%) were retained in the study and completed the follow-up outcome measures.
|Theme and subthemes||Examples|
|Appeal of app features and layout|
|Ease of use|
|Usefulness of content|
|Support with messaging|
|Improved mental health knowledge|
|Rote nature of responses|
|Positive impact on relationships|
|Suggestions for improvement|
aDBT: dialectical behavioral therapy.
Overall, recruitment via social media advertising (n=14) was substantially more effective than recruitment via specialist mental health services (n=8) or primary health services (n=4). However, recruitment rates were reversed between these methods, with 80% (4/5) of those from primary care, 67% (8/12) of those from specialist services, and 5% (14/280) of those who heard about the study via social media actually being enrolled. This suggests that more effective collaboration with clinicians may be necessary for successful recruitment via clinical services. The relatively low uptake via all means among males might indicate that this group needs to be specifically targeted during the next phase of research. There were no reported issues with comprehension or completion of outcome measures using the REDCap software. Furthermore, no other unanticipated operational issues were observed.
Usability of the Village app was evaluated using all 3 subscales of the uMARS questionnaire. Users gave Village a mean rating of 3.8 (range 2.7-4.6) out of 5 on a 5-point scale for app quality and an overall star rating of 3.4 out of 5 for subjective quality. Although knowledge, attitudes, and behavior change were rated lower (2.7-2.9 out of 5), the majority said it would increase the likelihood of future help-seeking (3.9 out of 5). Further details are provided in. Users reported varied engagement with Village during the trial. Some used the app multiple times a week, others occasionally, and a few only during COVID-19–related lockdowns. Examples of the feedback are presented in .
|Rated item||Score, mean (SD)|
|App quality rating|
|Engagement (fun, interesting, customizable, interactive, and has prompts)||3.6 (0.6)|
|Functionality (app functioning, easy to learn, navigation, flow logic, and gestural design of the app)||4.0 (0.6)|
|Esthetics (graphic design, overall visual appeal, color scheme and stylistic consistency)||3.7 (0.7)|
|Information (contains high-quality information from a credible source)||3.8 (0.8)|
|Total app quality mean score||3.8 (0.8)|
|App subjective quality|
|Would you recommend this app to people who might benefit from it? (1=not at all, 5=definitely)||3.0 (1.1)|
|How many times do you think you would use this app in the next 12 month if it was relevant to you? (1=none, 2=1−2, 3=3−10, 4=10−50, and 5≥50)||3.6 (0.9)|
|Would you pay for this app? (1=definitely not, 5=definitely yes)||1.6 (0.7)|
|What is your overall (star) rating of the app? (1=one of the worst apps I have used, 3=average, and 5=one of the best apps I have used)||3.4 (0.9)|
|App perceived impact|
|Awareness (this app has increased my awareness of the importance of addressing the health behavior)||3.5 (1.4)|
|Knowledge (this app has increased my knowledge or understanding of the health behavior)||2.9 (1.1)|
|Attitude (the app has changed my attitudes toward improving this health behavior)||2.7 (1.1)|
|Intention to change (the app has increased my intentions or motivation to address this health behavior)||3.4 (1.2)|
|Help-seeking (this app would encourage me to seek further help to address the health behavior)||3.9 (1.1)|
|Behavior change (use of this app will increase or decrease the health behavior)||2.9 (1.0)|
aAll rating scales ranged from 1 to 5 in the user version of the Mobile Application Rating Scale questionnaire.
Examples of user feedback regarding pattern of app use.
- “I used it when I was feeling bad and when I was feeling like I wanted to help my friends and stuff. So, I would go on the app to check if they needed any help or anything.” (User 16)
- “Most of the time I let people know when I wasn’t feeling good but like when I was overwhelmed more and when I wasn’t seeing my friends as much, when I wasn’t being able to check in on them in person. One of my friends I was messaging him probably like more, maybe like five times a week, especially like during the holidays and stuff when I wasn’t seeing him. And then other one µm like less often, maybe two times a week. And the other one maybe once a week.” (User 21)
- “If I’m being honest, at first, I didn’t really use it, but then, um, towards like the middle of like the trial, me and my friend would use it more because like lockdown and stuff happened and then there was another way to contact each other and let us know how we feel.” (User 11)
The risk detection software was activated on 3 occasions during the study. Two followed users entering the words “burden” or “suicidal ideation” and one followed a buddy entering of the words “useless” and “not good enough” within the same message. On all 3 occasions, participants elected to be contacted by a member of the research team rather than being put through to a national telephone helpline, and they reported that they were fine. None of the participants interviewed toward the end of the study reported any episodes of self-harm or hospitalization during the period of enrollment.
Preliminary evaluation of the efficacy of Village was undertaken by measuring changes in symptoms of depression, level of suicidal ideation, and level of functioning following app use; the results are presented in. In a repeated measures analysis, there was a statistically significant reduction in depression symptoms using the Patient Health Questionnaire–9 modified for adolescents between baseline and the 3-month follow-up (P=.002), but there were no statistically significant changes over time in suicidal ideation using the SIQ (P=.61) or in functioning using the WHODAS 2.0 or WHODAS-CY (P=.13). The severity scores at all time points for the 3 measures are displayed as tables in .
|Depression (PHQ-Aa)||Suicidal ideation (SIQb)||Functioning (WHODAS 2.0c or WHODAS-CYd)|
|Baseline||4-week follow-up||3-month follow-up||Baseline||4-week follow-up||3-month follow-up||Baseline||4-week follow-up||3-month follow-up|
|Mean (SD)||18.1 (5.6)||15.7 (6.6)||12.7 (5.7)||77.7 (48.4)||68.9 (45.8)||57.3 (44.2)||46.2 (16.5)||45.2 (16.3)||36.7 (19.4)|
|Mean difference (95% CI)e||N/Af||−2.4 (−6.3 to 0.9)||−5.4 (−9.2 to −2.2)||N/A||−8.8 (−42.7 to 13.2)||−20.4 (−53.9 to 1.2)||N/A||−1.0 (−13.2 to 8.9)||−9.5 (−21.7 to 0.2)|
|Effect size (Cohen d)g||N/A||N/A||0.9||N/A||N/A||0.4||N/A||N/A||0.5|
|Overall P value||N/A||N/A||.001||N/A||N/A||.16||N/A||N/A||.13|
aPHQ-A: Patient Health Questionnaire–9 modified for adolescents.
bSIQ: Suicidal Ideation Questionnaire.
cWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.
dWHODAS-CY: World Health Organization Disability Assessment Schedule–Children and Youth.
eEstimated marginal mean differences from a repeated measures analysis with pairwise comparisons between time points.
fN/A: not applicable.
gEffect sizes were calculated using baseline and 3-month follow-up means and SDs.
During this open trial, we found that Village was acceptable and usable to both users and buddies, and that a larger RCT appeared feasible, providing the following changes were made to the current version of the app and study protocol: (1) improved app onboarding instruction and notifications, (2) recruitment focused either directly on potential participants via social media or via collaborative clinicians at clinical services, and (3) all outcome measures being made mandatory on REDCap. Young people who experienced preexisting difficulty communicating with available support in person or via social media and buddies (families or friends) with limited mental health knowledge found the app most useful. The feedback suggested that some improvements to onboarding and notifications would further increase the appeal of the app. Short-term, statistically significant improvements in mood and nonsignificant changes in functioning or suicidal ideation need to be interpreted with caution given the small number of participants and the preliminary nature of the trial. Embedded risk detection software was appropriately activated on a few occasions and there were no reported episodes of self-harm or hospitalization among participants, suggesting the Village was safe to use with a clinically “high risk” cohort.
Although there is a plethora of digital interventions for supporting young people’s well-being and mental health, only a handful of other digital interventions have been specifically developed for young people experiencing self-harm and suicidal ideation. These are As Safe As Possible, during which young people complete a 3-hour safety-planning, emotion regulation session and then use an app to review their mood, safety plan, and learned skills ; Reframe-IT, a web-based cognitive behavioral therapy (CBT) program with 8 modules, video diaries, fact sheets, and a message board to interact with trained therapists [ ]; and a Crisis Care mobile app that includes personalized coping skills accessible to both young people and parents [ ]. Similar to Village, all 3 interventions have been piloted and shown to be acceptable to users, but not to improve mood, reduce self-harm, or reduce suicidal ideation. Better-powered trials of these interventions have been recommended [ ], and until then, uncertainty regarding their efficacy remains. A larger number of nondigital family and peer support interventions have also been developed over the past decade. These include DBT, a manualized, multisession face-to-face therapy that includes young people and parents [ ]; Safe Alternatives for Teens and Youth, a 12-week family-inclusive intervention based on principles of CBT and DBT [ ]; the Family Intervention for Suicide Prevention, a single-session family intervention with telephone follow-up [ ]; Attachment-Based Family Therapy, a therapy that aims to improve relationships between young people and caregivers and improve mental health symptoms [ ]; Self-Harm Intervention Family Therapy, a 6-month family therapy-based treatment [ ]; and integrated CBT, a combination of individual- and family-based CBT and parent training [ ]. Of these, DBT had the strongest evidence (via 2 RCTs) of reducing self-harm and suicidal ideation in the short term, and youth-nominated support teams–version 2 [ ] was the only intervention with evidence of long-term reduction of suicide. Others have demonstrated reasonable acceptability and variable short-term efficacy. Given the fluctuating nature of mood disorders and suicidal ideation, the timing and nature of outcome measurements are likely to be important in accurately evaluating the short-term efficacy of these interventions. Adequately powered studies and the collection of long-term follow-up data, such as those undertaken by King et al [ ], are also essential and should be supported by funders.
Given that most young people these days regularly use texting and multiple social media apps to keep in touch with family and friends, it is reasonable to wonder about the necessity for a new vehicle with which to communicate distress. Previous research has shown that although social media can make it easier to maintain contact with others, young people are very careful about how to use such networks for support [, ]. Concerns about public perception coupled with the need for trust and genuine understanding often lead those experiencing distress to be less open about their issues and perpetuate “false selves” [ ]. Negative social media experiences make young people even less likely to use them when distressed [ ]. When young people reach out via social media or in person, families and friends may not feel confident in responding because of their own feelings of discomfort, historically poor communication, or concerns about accountability [ , ]. Alternatively, they may respond in a manner that fails to address key emotions, thereby causing young people to feel no better or worse. Village affords a private forum for users to connect with chosen individuals in a genuine manner and for supporters to be educated about communication skills and mental health issues in a manner that leads them to sensitively respond to user needs. These factors are likely to increase both the depth of disclosure and satisfaction after disclosure [ ].
Most young people in this trial chose friends, rather than family members, as buddies. This is consistent with previously identified age-related preferences for support among young people with depression . It is also in keeping with developmentally related preferences identified during the co-design process for Village—younger adolescents (aged 13-16 years) who spent more time with their parents said they would prefer family members as buddies, whereas more independent and older adolescents (aged ≥16 years) said they would prefer to be supported by friends. Unfortunately, owing to ethical committee restrictions, younger adolescents were not included in this trial. However, we plan to include them in the upcoming RCT and conduct a subgroup analysis of acceptability to parents and friends to ensure that the app is useful to both groups.
Although conceptualized by an individual of Māori descent (ES), consciously designed to enhance “Whanau Ora,” and co-designed with a number of Māori young people and families (“whanau”), we did not conduct any specific evaluation of the cultural appeal or safety of the app during this open trial. Given the higher rates of depression, self-harm, and suicide among New Zealand Māori young people (consistent with rates among Indigenous groups in other countries ) and the concern that less-than-ideally designed digital interventions may increase health inequity for Indigenous populations [ ], it is important for these issues to be explored before it can be assumed that the app is effective for those with the greatest need. Despite additional concerns regarding the potential for digital interventions to increase inequity between audiences with and without access to technology, we believe this is less of an issue in New Zealand where 95% of young people have internet access and 72% own a smartphone [ ].
Despite the functionality of Village being rated above average, uMARS ratings were comparable with other recently developed mental health apps for young people , and despite users reporting improved awareness and greater likelihood of help-seeking following app use, it did not appeal to everyone. Onboarding instructions were not clear enough for everyone and were updated in light of participant feedback. Users and buddies with greater communication skills and mental health knowledge found the scaffolded communication restrictive. The pattern of app use varied considerably between participants, and most did not wish to use the app long term. It is possible that the ideal audience for the Village consists of young people who are relatively new to clinical services and family or friends with limited experience in providing mental health support. It is also possible that both audiences may learn what they need via the app and generalize communication or support skills to real life or other internet-based environments within a limited period. Village may play a vital role in “bridging the gap” between those in need and those who are able to access saturated mental health services [ ]. Further research is required to investigate these possibilities.
The strengths of this trial include the exploration of both user and buddy perspectives on the app and the combined use of quantitative and qualitative analyses to provide a richer understanding of its appeal and function. Weaknesses include the limited sample size, lack of a control group, and lack of complete follow-up measures for 20% of participants, which might have led to biased results. In addition, not all users and buddies agreed to be interviewed, and interviews were conducted by a member of the research team who had been involved in participant recruitment and data analysis, and objective data regarding app use were not collected for privacy reasons. The exclusion of participants aged <16 years (owing to ethics committee constraints) and those from outside New Zealand also means that our results may not be generalizable to younger users or those from other countries.
On the basis of the preliminary results from this open trial, Village appears to be an acceptable, usable, and safe communication app with which young people experiencing low mood, self-harm, and suicidal ideation can receive support from their family and friends. A larger RCT to confirm the current findings and evaluate the efficacy of the app appears to be feasible with minor modifications to the app and study protocol.
The authors wish to thank all the young people, families, and friends who participated in the co-design and open trial of Village. The authors also wish to thank Dr Alana Cavadino, a biostatistician at the University of Auckland, for her assistance with the study design and data analysis.
This work was supported by the New Zealand Starship Foundation (grant number SF 1960).
The study was conceptualized by all authors (HT and ES). The study results were analyzed by 2 authors (HT and HK). All the authors contributed to the drafting and review of the paper.
Conflicts of Interest
HT and ES are developers of the Village app. HK does not have any conflict of interest to declare.
Higher resolution version of. Supported messages.PNG File , 1929 KB
Semistructured interview questions.DOCX File , 16 KB
Severity scores for depression, suicidal ideation, and functioning.DOCX File , 19 KB
- Cybulski L, Ashcroft DM, Carr MJ, Garg S, Chew-Graham CA, Kapur N, et al. Temporal trends in annual incidence rates for psychiatric disorders and self-harm among children and adolescents in the UK, 2003-2018. BMC Psychiatry 2021 May 03;21(1):229 [FREE Full text] [CrossRef] [Medline]
- Wiens K, Bhattarai A, Pedram P, Dores A, Williams J, Bulloch A, et al. A growing need for youth mental health services in Canada: examining trends in youth mental health from 2011 to 2018. Epidemiol Psychiatr Sci 2020 Apr 17;29:e115. [CrossRef]
- Zheng H, Echave P. Are recent cohorts getting worse? Trends in US adult physiological status, mental health, and health behaviors across a century of birth cohorts. Am J Epidemiol 2021 Nov 02;190(11):2242-2255 [FREE Full text] [CrossRef] [Medline]
- Pierce M, Hope H, Ford T, Hatch S, Hotopf M, John A, et al. Mental health before and during the COVID-19 pandemic: a longitudinal probability sample survey of the UK population. Lancet Psychiatry 2020 Oct;7(10):883-892. [CrossRef]
- Serlachius A, Badawy SM, Thabrew H. Psychosocial challenges and opportunities for youth with chronic health conditions during the COVID-19 pandemic. JMIR Pediatr Parent 2020 Oct 12;3(2):e23057 [FREE Full text] [CrossRef] [Medline]
- Makri-Botsari E. Risk/protective effects on adolescent depression: role of individual, family and peer factors. Psychol Stud 2005;50(1):50-61.
- Zisk A, Abbott CH, Bounoua N, Diamond GS, Kobak R. Parent-teen communication predicts treatment benefit for depressed and suicidal adolescents. J Consult Clin Psychol 2019 Dec;87(12):1137-1148 [FREE Full text] [CrossRef] [Medline]
- Thabrew H, Gandeza E, Bahr G, Bettany D, Bampton C, Cooney E, et al. The management of young people who self-harm by New Zealand Infant, Child and Adolescent Mental Health Services: cutting-edge or cutting corners? Australas Psychiatry 2018 Apr 22;26(2):152-159. [CrossRef] [Medline]
- National Collaborating Centre for Mental Health (UK). Self-Harm: Longer-Term Management. Leicester (UK): British Psychological Society; 2011.
- Youth '12 Prevalence Tables The Health and Wellbeing of New Zealand Secondary School Students in 2012. New Zealand: University of Auckland - Faculty of Medical & Health Sciences; 2013.
- Mars B, Heron J, Crane C, Hawton K, Lewis G, Macleod J, et al. Clinical and social outcomes of adolescent self harm: population based birth cohort study. BMJ 2014 Oct 21;349(oct20 5):g5954 [FREE Full text] [CrossRef] [Medline]
- Beautrais AL. Risk factors for suicide and attempted suicide among young people. Aust N Z J Psychiatry 2000 Jun 26;34(3):420-436. [CrossRef] [Medline]
- Suicide Facts Deaths and Intentional Self-harm Hospitalisations, 2007. New Zealand: Ministry of Health; 2009.
- Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual research review: digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry 2017 Apr 10;58(4):474-503. [CrossRef] [Medline]
- Thabrew H, Stasiak K, Hetrick S, Wong S, Huss J, Merry S. E-Health interventions for anxiety and depression in children and adolescents with long-term physical conditions. Cochrane Database Syst Rev 2018 Aug 15;8(8):CD012489 [FREE Full text] [CrossRef] [Medline]
- Prensky M. Digital natives, digital immigrants part 2: do they really think differently? On the Horizon 2001 Nov;9(6):1-6. [CrossRef]
- Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. BMJ 2012 Apr 18;344(apr18 3):e2598 [FREE Full text] [CrossRef] [Medline]
- Hopkins K, Crosland P, Elliott N, Bewley S, Clinical Guidelines Update Committee B. Diagnosis and management of depression in children and young people: summary of updated NICE guidance. BMJ 2015 Mar 04;350(mar04 9):h824. [CrossRef] [Medline]
- Lancet Global Mental Health Group, Chisholm D, Flisher AJ, Lund C, Patel V, Saxena S, et al. Scale up services for mental disorders: a call for action. Lancet 2007 Oct 06;370(9594):1241-1252. [CrossRef] [Medline]
- Schröder J, Berger T, Westermann S, Klein JP, Moritz S. Internet interventions for depression: new developments. Dialogues Clin Neurosci 2022 Apr 01;18(2):203-212 [FREE Full text] [CrossRef]
- Torous J, Chan SR, Yee-Marie Tan S, Behrens J, Mathew I, Conrad EJ, et al. Patient smartphone ownership and interest in mobile apps to monitor symptoms of mental health conditions: a survey in four geographically distinct psychiatric clinics. JMIR Ment Health 2014;1(1):e5 [FREE Full text] [CrossRef] [Medline]
- Conley CS, Raposa EB, Bartolotta K, Broner SE, Hareli M, Forbes N, et al. The impact of mobile technology-delivered interventions on youth well-being: systematic review and 3-level meta-analysis. JMIR Ment Health 2022 Jul 29;9(7):e34254 [FREE Full text] [CrossRef] [Medline]
- Melia R, Francis K, Hickey E, Bogue J, Duggan J, O'Sullivan M, et al. Mobile health technology interventions for suicide prevention: systematic review. JMIR Mhealth Uhealth 2020 Jan 15;8(1):e12516 [FREE Full text] [CrossRef] [Medline]
- What works: going digital to deliver wellbeing services to young people? Insights from e-tools supporting youth mental health and parenting. Social Wellbeing Agency. 2016. URL: https://tinyurl.com/yscwk68n [accessed 2019-08-01]
- Kidd J, Gibbons V, Lawrenson R, Johnstone W. Pounamu: a whanau ora approach to health care for Maori. J Prim Health Care 2010;2(2):163. [CrossRef]
- Davidson L, Chinman M, Kloos B, Weingarten R, Stayner D, Tebes JK. Peer support among individuals with severe mental illness: a review of the evidence. Clin Psychol Sci Pract 1999;6(2):165-187. [CrossRef]
- Saxena S, Funk M, Chisholm D. World health assembly adopts comprehensive mental health action plan 2013–2020. Lancet 2013 Jun;381(9882):1970-1971. [CrossRef]
- Sanders J, Munford R. The interaction between culture, resilience, risks and outcomes: a New Zealand study. In: Youth Resilience and Culture. Cross-Cultural Advancements in Positive Psychology, vol 11. Dordrecht: Springer; 2015.
- Shaffer D, Vieland V, Garland A. Adolescent suicide attempters response to suicide-prevention programs. JAMA 1990 Dec 26;264(24):3151-3155. [CrossRef]
- King CA, Arango A, Kramer A, Busby D, Czyz E, Foster CE, et al. Association of the youth-nominated support team intervention for suicidal adolescents with 11- to 14-year mortality outcomes: secondary analysis of a randomized clinical trial. JAMA Psychiatry 2019 May 01;76(5):492-498 [FREE Full text] [CrossRef] [Medline]
- Thabrew H, Fleming T, Hetrick S, Merry S. Co-design of eHealth interventions with children and young people. Front Psychiatry 2018 Oct 18;9:481 [FREE Full text] [CrossRef] [Medline]
- Dickinson T. Non‐violent communication, compassionate communication∗. Med Conflict Survival 1998 Jan;14(1):56-62. [CrossRef]
- Beckett JA. General systems theory, psychiatry and psychotherapy. Int J Group Psychother 1973 Jul 29;23(3):292-305. [CrossRef] [Medline]
- Durie M. Whaiora Maori Health Development Second Edition. Oxford, England: Oxford University Press; 1999.
- Rockland L. Supportive therapy. New York, NY: Basic Books; Apr 17, 2003.
- Jones-Smith E. Strengths-Based Therapy Connecting Theory, Practice and Skills. Thousand Oaks, California, United States: SAGE Publications; Jan 09, 2013.
- Picardo J, McKenzie SK, Collings S, Jenkin G. Suicide and self-harm content on Instagram: a systematic scoping review. PLoS One 2020;15(9):e0238603 [FREE Full text] [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the mobile application rating scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
- Johnson JG, Harris ES, Spitzer RL, Williams JB. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health 2002 Mar;30(3):196-204. [CrossRef] [Medline]
- Davis JM. Suicidal ideation questionnaire. J Psychoeducational Assess 2016 Aug 18;10(3):298-301. [CrossRef]
- Federici S, Bracalenti M, Meloni F, Luciano JV. World Health Organization disability assessment schedule 2.0: an international systematic review. Disabil Rehabil 2017 Nov;39(23):2347-2380. [CrossRef] [Medline]
- Saltychev M, Katajapuu N, Bärlund E, Laimi K. Psychometric properties of 12-item self-administered World Health Organization disability assessment schedule 2.0 (WHODAS 2.0) among general population and people with non-acute physical causes of disability - systematic review. Disabil Rehabil 2021 Mar 23;43(6):789-794. [CrossRef] [Medline]
- Braun V, Clarke V, Terry G, Hayfield N. Thematic analysis. In: Handbook of Research Methods in Health Social Sciences. Singapore: Springer; Jan 13, 2019.
- Kennard BD, Goldstein T, Foxwell AA, McMakin DL, Wolfe K, Biernesser C, et al. As safe as possible (ASAP): a brief app-supported inpatient intervention to prevent postdischarge suicidal behavior in hospitalized, suicidal adolescents. Am J Psychiatry 2018 Sep 01;175(9):864-872 [FREE Full text] [CrossRef] [Medline]
- Hetrick SE, Yuen HP, Bailey E, Cox GR, Templer K, Rice SM, et al. Internet-based cognitive behavioural therapy for young people with suicide-related behaviour (Reframe-IT): a randomised controlled trial. Evid Based Ment Health 2017 Aug 12;20(3):76-82. [CrossRef] [Medline]
- McManama O'Brien KH, LeCloux M, Ross A, Gironda C, Wharff EA. A pilot study of the acceptability and usability of a smartphone application intervention for suicidal adolescents and their parents. Arch Suicide Res 2017 May 02;21(2):254-264. [CrossRef] [Medline]
- Busby DR, Hatkevich C, McGuire TC, King CA. Evidence-based interventions for youth suicide risk. Curr Psychiatry Rep 2020 Jan 18;22(2):5. [CrossRef] [Medline]
- McCauley E, Berk MS, Asarnow JR, Adrian M, Cohen J, Korslund K, et al. Efficacy of dialectical behavior therapy for adolescents at high risk for suicide: a randomized clinical trial. JAMA Psychiatry 2018 Aug 01;75(8):777-785 [FREE Full text] [CrossRef] [Medline]
- Asarnow JR, Hughes JL, Babeva KN, Sugar CA. Cognitive-behavioral family treatment for suicide attempt prevention: a randomized controlled trial. J Am Acad Child Adolesc Psychiatry 2017 Jun;56(6):506-514 [FREE Full text] [CrossRef] [Medline]
- Asarnow JR, Berk MS, Baraff LJ. Family intervention for suicide prevention: a specialized emergency department intervention for suicidal youths. Professional Psychology: Research and Practice 2009 Apr;40(2):118-125. [CrossRef]
- Diamond GS, Reis BF, Diamond GM, Siqueland L, Isaacs L. Attachment-based family therapy for depressed adolescents: a treatment development study. J Am Acad Child Adolesc Psychiatry 2002 Oct;41(10):1190-1196. [CrossRef] [Medline]
- Cottrell DJ, Wright-Hughes A, Collinson M, Boston P, Eisler I, Fortune S, et al. Effectiveness of systemic family therapy versus treatment as usual for young people after self-harm: a pragmatic, phase 3, multicentre, randomised controlled trial. Lancet Psychiatry 2018 Mar;5(3):203-216. [CrossRef]
- Esposito-Smythers C, Spirito A, Kahler CW, Hunt J, Monti P. Treatment of co-occurring substance abuse and suicidality among adolescents: a randomized trial. J Consult Clin Psychol 2011 Dec;79(6):728-739 [FREE Full text] [CrossRef] [Medline]
- Gibson K, Trnka S. Young people's priorities for support on social media: “It takes trust to talk about these issues”. Comput Human Behav 2020 Jan;102:238-247. [CrossRef]
- Nasier B, Gibson K, Trnka S. “PM me” or “LOL”: Young peoples’ observations of supportive and unsympathetic responses to distress on social media. Comput Human Behav 2021 Nov;124:106933. [CrossRef]
- Harter S, Marold DB, Whitesell NR, Cobbs G. A model of the effects of perceived parent and peer support on adolescent false self behavior. Child Development 1996 Apr;67(2):360-374. [CrossRef]
- Ahmed O, Nayeem Siddiqua SJ, Alam N, Griffiths MD. The mediating role of problematic social media use in the relationship between social avoidance/distress and self-esteem. Technol Soc 2021 Feb;64:101485. [CrossRef]
- Meschke LL, Juang LP. Obstacles to parent-adolescent communication in Hmong American families: exploring pathways to adolescent mental health promotion. Ethn Health 2014 Jul 11;19(2):144-159. [CrossRef] [Medline]
- Repper J, Carter T. A review of the literature on peer support in mental health services. J Ment Health 2011 Aug;20(4):392-411. [CrossRef] [Medline]
- Stice E, Ragan J, Randall P. Prospective relations between social support and depression: differential direction of effects for parent and peer support? J Abnormal Psychol 2004 Feb;113(1):155-159. [CrossRef]
- Stefanac N, Hetrick S, Hulbert C, Spittal MJ, Witt K, Robinson J. Are young female suicides increasing? A comparison of sex-specific rates and characteristics of youth suicides in Australia over 2004-2014. BMC Public Health 2019 Oct 28;19(1):1389 [FREE Full text] [CrossRef] [Medline]
- Strudwick G, Sockalingam S, Kassam I, Sequeira L, Bonato S, Youssef A, et al. Digital interventions to support population mental health in Canada during the COVID-19 pandemic: rapid review. JMIR Ment Health 2021 Mar 02;8(3):e26550 [FREE Full text] [CrossRef] [Medline]
- Pacheco E, Melhuish N. Exploring New Zealand children’s technology access, use, skills and opportunities. Evidence from Ngā taiohi matihiko o Aotearoa - New Zealand Kids Online. SocArXiv 2019 Dec 22 [FREE Full text] [CrossRef]
- Serlachius A, Boggiss A, Lim D, Schache K, Wallace-Boyd K, Brenton-Peters J, et al. Pilot study of a well-being app to support New Zealand young people during the COVID-19 pandemic. Internet Interv 2021 Dec;26:100464 [FREE Full text] [CrossRef] [Medline]
- Rowe SL, French RS, Henderson C, Ougrin D, Slade M, Moran P. Help-seeking behaviour and adolescent self-harm: a systematic review. Aust N Z J Psychiatry 2014 Dec;48(12):1083-1095. [CrossRef] [Medline]
|CBT: cognitive behavioral therapy|
|DBT: dialectical behavioral therapy|
|RCT: randomized controlled trial|
|REDCap: Research Electronic Data Capture|
|SIQ: Suicidal Ideation Questionnaire|
|uMARS: user version of the Mobile Application Rating Scale|
|WHODAS: World Health Organization Disability Assessment Schedule|
|WHODAS-CY: World Health Organization Disability Assessment Schedule–Children and Youth|
Edited by A Mavragani; submitted 21.07.22; peer-reviewed by K Champion, C Biernesser; comments to author 21.12.22; revised version received 12.01.23; accepted 24.01.23; published 13.03.23Copyright
©Hiran Thabrew, Harshali Kumar, Evandah Steadman. Originally published in JMIR Formative Research (https://formative.jmir.org), 13.03.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.