This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
The importance of mental health promotion is irrevocable and is especially important at a young age. More mental health-promoting mobile apps have been developed in the last few years. However, their usability and quality have been rarely assessed.
The aim of this study is to investigate how adolescents assess the usability, quality, and potential goal achievement of Opp and NettOpp. Opp is a universal mental health–promoting mobile app aimed at 13- to 19-year-olds, and NettOpp is a mobile app for children and adolescents between 11 to 16 years of age that have experienced negative incidents online.
A total of 45 adolescents tested either Opp (n=30) or NettOpp (n=15) for a period of 3 weeks and answered a questionnaire. The System Usability Scale (SUS) was used to measure the usability of the apps. A SUS score above 70 indicates acceptable usability. Items from the Mobile Application Rating Scale were adapted for study purposes and used to measure the quality and perceived goal achievement that Opp and NettOpp might have on adolescents’ knowledge, attitudes, and intention to change behavior. Furthermore, adolescents could answer an open comment question.
Opp had a mean SUS score of 80.37 (SD 9.27), and NettOpp’s mean SUS score was 80.33 (SD 10.30). In the overall evaluation, Opp and NettOpp were given a mean score of 3.78 (SD 0.42) and 4.20 (SD 0.56), respectively, on a 5-point scale, where 5 was best. Most adolescents who evaluated Opp rated that the app would increase knowledge about mental health and help young people deal with stress and difficult emotions or situations. Most adolescents who evaluated NettOpp agreed that the app would increase awareness and knowledge about cyberbullying, change attitudes toward cyberbullying, and motivate them to address cyberbullying. Some adolescents stated that Opp was difficult to navigate and consisted of too much text. Some of the adolescents that tested NettOpp stated that the app had crashed and that the design was a bit childish.
All in all, this study indicates that Opp and NettOpp have good usability and that adolescents are satisfied with both apps. It also indicates that the potential goal achievement of the apps, for example, increasing knowledge about mental health (Opp) or cyberbullying (NettOpp) is promising. While there are some comments from the users that are more difficult to solve (eg, Opp is too text-based), some comments helped improve the apps (eg, that the app crashed). Overall, the user evaluation provided valuable knowledge about how adolescents assess Opp and NettOpp. However, more extensive effectiveness studies are necessary to measure their actual goal achievement.
Adolescence is a crucial time in life to develop healthy habits that can improve both physical and mental health in later life [
In general, mental health promotion aims at increasing resilience, self-esteem, and coping skills [
Even though young people’s extensive use of social media and online services enables opportunities for socialization, entertainment, and rapid information retrieval, adolescents may also experience negative online events, such as mean comments, exclusion from events and groups, or threats [
A quick search on Google Play or the App Store shows that new apps that aim to improve young people’s mental health are constantly being developed. However, the apps that are on the market are of varying quality [
Opp (
Start page of Opp.
NettOpp (
Start page of NettOpp.
This study examines how adolescents evaluate the usability and quality of Opp and NettOpp, as well as their potential to achieve their given objectives. The study aims to respond to the following research questions:
RQ1: Are young people satisfied with the system usability of Opp and NettOpp?
RQ2: How do young people assess the quality of Opp and NettOpp?
RQ3: How do young people assess Opp and NettOpp’s potential to achieve their goals?
A total of 45 adolescents within the age range of 11 to 19 years were recruited in 2021 to participate in 2 cross-sectional user surveys. The participants were recruited through the authors’ colleagues and acquaintances. A total of 30 adolescents (n=20 girls and n=10 boys) evaluated Opp, and 15 adolescents (n=6 girls and n=9 boys) evaluated NettOpp. For Opp, 3 (10%) adolescents stated that they had not used the app at all and were therefore excluded from the analyses. Hence, the results presented are based on 27 adolescents (n=8 boys and n=19 girls) evaluating Opp. After a trial period of approximately 3 weeks, the adolescents who evaluated Opp answered an online questionnaire in Nettskjema [
The usability of the apps was measured using the System Usability Scale (SUS) [
The quality of the apps was assessed with items from the quality rating section of the Mobile Application Rating Scale (MARS) [
To measure how the adolescents evaluated the apps’ potential to achieve their given objectives, the app-specific rating section from the MARS [
The evaluation studies were approved by the Norwegian Centre for Research Data (reference for Opp: 450518; reference for NettOpp: 545417). Active parental consent from 1 guardian was required for the adolescents to participate in the study. The consent was retrieved in a digital consent form using Nettskjema [
All statistical analyses were performed in SPSS version 27 (IBM Corp). Descriptive statistics were calculated and included means, standard deviations, and frequency distributions. An independent sample
The overall mean SUS scores of Opp and NettOpp were 80.37 (SD 9.27) and 80.33 (SD 10.30), respectively. An overview of adolescents who agreed with statements in the SUS is presented in
Percentage of adolescents who answered that they strongly agree or agree with claims about NettOpp and Opp’s system usability.
|
Opp (n=27), n (%) | NettOpp (n=15), n (%) |
I think I would like to use this product frequently | 9 (33) | 5 (33) |
I found the product unnecessarily complex | 1 (4) | 0 (0) |
I thought the product was easy to use | 25 (93) | 13 (87) |
I think that I would need the support of a technical person to be able to use this product | 0 (0) | 0 (0) |
I found the various functions in the product were well integrated | 22 (82) | 14 (93) |
I thought there was too much inconsistency in this product | 0 (0) | 0 (0) |
I imagine that most people would learn to use this product very quickly | 26 (96) | 15 (100) |
I found the product very awkward to use | 2 (7) | 0 (0) |
I felt very confident using the product | 19 (70) | 11 (73) |
I needed to learn a lot of things before I could get going with this product | 1 (4) | 1 (7) |
Descriptive results from the adolescents’ evaluation of the quality of the apps are presented in
Descriptive statistics of the quality ratings for the mobile apps NettOpp and Opp.
|
Opp (n=27) | NettOpp (n=15) | ||
|
Min-max | Mean (SD) | Min-max | Mean (SD) |
Is the app interesting to use? | 3-5 | 3.78 (0.51) | 3-5 | 4.00 (0.65) |
Is the app content appropriate for the target audience? | 3-5 | 4.37 (0.68) | 3-5 | 4.47 (0.64) |
How accurately or fast do the app features and components work? | 3-5 | 4.67 (0.55) | 1-5 | 4.13 (1.25) |
How easy is it to learn how to use the app? | 3-5 | 4.48 (0.65) | 3-5 | 4.53 (0.64) |
Is arrangement and size of buttons or icons (…) on the screen appropriate or zoomable? | 2-5 | 4.37 (0.79) | 4-5 | 4.53 (0.52) |
How good does the app look? | 2-5 | 4.15 (0.66) | 3-5 | 4.13 (0.74) |
Does the app contain what is described? | 3-5 | 4.26 (0.17) | 4-5 | 4.80 (0.41) |
Would you recommend this app to people who might benefit from it? | 2-5 | 3.68 (0.84) | 1-5 | 4.33 (0.74) |
How many times do you think you would use this app in the next 12 months? | 1-5 | 3.33 (1.04) | 1-5 | 3.47 (1.25) |
What is your overall star rating of the app? | 3-4 | 3.78 (0.42) | 3-5 | 4.20 (0.56) |
Proportion of adolescents answering that they strongly agree or agree with statements about Opp’s potential to achieve its goals.
Opp will probably... | Participants (n=27), n (%) |
Increase well-being and mental health | 14 (52) |
Increase knowledge about mental health | 25 (92) |
Help young people deal with stress, difficult emotions, or situations in a good way | 18 (67) |
Motivate young people to speak out or ask for help when they need it | 9 (33) |
Make young people feel better about themselves | 14 (52) |
Decrease mental health problems | 10 (37) |
Proportion of adolescents who answer that they strongly agree or agree with statements about NettOpp’s potential to achieve its goals.
NettOpp will probably... | Participants (n=15), n (%) |
Increase awareness of the importance of addressing cyberbullying | 15 (100) |
Increase knowledge and understanding of cyberbullying | 14 (93) |
Change attitudes toward improving reactions of cyberbullying | 11 (73) |
Increase intentions or motivation to address cyberbullying | 13 (87) |
Encourage further help seeking after being exposed to cyberbullying | 14 (93) |
Decrease cyberbullying | 5 (33) |
In total, 7 adolescents left a comment about Opp and 5 left a comment about NettOpp in the open comment field (all comments are translated and presented in
I will not use the app again because I currently have nothing that bothers me and I bet it will stay that way, but I will probably use it if I start to struggle mentally.
I really like that the little figure praises you when you finish an exercise.
The app is probably best suited for children who are 10-14 years old, for example, if they struggle to talk to their parents. If the app is intended for all adolescents, the app must be adapted to older adolescents as well.
Maybe a little too much text but that is fine.
Very nice app! But I think it was difficult to get an overview of the main menu and what the app contained. It seemed more like you pressed something, and you also moved on, but it was difficult to come back.
All the exercises with audio files are very well formulated and informative, but difficult to use. The biggest problem is that you cannot listen to the audio file while the phone is in sleep mode (the screen is off). It was disturbing for the experience of the exercises when there was an interruption since the mobile phone automatically switches off and the sound from Opp is cut off. Functions such as listening to sounds such as music or podcasts in Spotify, etc is desired – so you can listen to the audio file or videos even if the phone automatically shuts off the screen.
There could have been more exercises, for example, relaxation exercises.
Feeling empty should have been an alternative, missed it when I was tired and did not really feel anything.
The app was easy to use. I never needed help from my parents to use the app. The quiz was good. Breathing technique exercise was good. I have tested them several times.
The app was very user-friendly. Very good to point out that experiencing cyberbullying is not your fault. The app’s main menu may possibly feel a bit childish, but I liked the layout.
I think it is a very good app with many features that are easy to use. Even though I do not need it, I think those who use it will be happy.
A bit too much text and will think that for some it will take away the motivation, and they cannot bear to read everything. Still feels that everything that is written is relevant. Technical problem: The app automatically closes several times while in use. I can go into it right afterward and it works as it should, but after a while is “thrown out” again. I think there is a nice common thread throughout the app, and I like the suggestions that come up along the way.
The menu button in the left corner can be a little more visible, it was a little difficult to see. Once you have taken the quiz, you will not see results, you can only choose to return to previous questions or close the page. Had been nice to see results and get a “prize.” The test app crashes all the time. The lock icon on thought clearing was confusing and made me think that the function was locked and that I could not use it. Very good design and nice app. Very good idea.
Some words were difficult, the app crashes 3 times, PLZ fix it!
Try to fix the piece where you are thrown out.
Several health-promoting apps have been developed in recent years, but only a few of them have been evaluated. Furthermore, studies that evaluate mobile apps are of varying quality [
Overall, the adolescents were satisfied with the system usability of both Opp and NettOpp. SUS scores for Opp and NettOpp corresponded to a B grade on the grade scale and to “excellent” for the adjective ratings [
There was also overall agreement among both the adolescents who evaluated Opp and those who evaluated NettOpp that the apps were of good quality. In this study, only some of the items from MARS were included in the questionnaire, so it will not be possible to compare the results of this evaluation study with previous studies using the complete MARS quality ratings [
As both apps aim to promote the mental health of adolescents, we sought to assess the adolescents’ opinion of whether the apps would have the ability to achieve their given objectives. Among the adolescents who evaluated Opp, there was broad agreement that the app would increase adolescents’ knowledge of mental health. Previous research has shown that teaching young people about mental health may be an effective preventive measure, which can also contribute to reducing the stigma related to mental disorders [
The sample that tested NettOpp thought that the app would increase awareness and knowledge of negative online incidents. Previous research has illustrated the importance of increasing knowledge and awareness about negative online incidents to ensure that both adolescents and people who work closely with the group (eg, teachers) are capable of handling the issue in an effective way [
Important limitations to address are related to the sample sizes and the sample itself that evaluated the apps, as well as to the cross-sectional design of the study. The 2 sample sizes were relatively small and not representative, so the findings of this study cannot be generalized to all adolescents in Norway. However, for the purpose of this study, we think that the sample sizes were sufficient. Some of the adolescents who evaluated the apps were also involved in developing the apps. This might have influenced their assessment in that they might, for example, have been more positively tuned when answering the questions. However, since study participation was anonymous, we cannot examine if it actually influenced the participants. The participants in the evaluation study were also compensated with a cinema gift card with a value of NOK 150 (US $15.03), which may have influenced the motivation of the study participants. Further, the age range of the sample who evaluated the apps was between 11 and 19 years. NettOpp’s target group is children and adolescents between the ages of 11 and 16 years, and Opp’s target group is adolescents aged between 13 and 19 years. One of the adolescents left a comment saying that Opp was best suited for adolescents aged between 10 and 14 years, and another adolescent left a comment saying that NettOpp was a bit childish. These comments may be due to the fact that the evaluators were older than the intended target group. However, the adolescents’ exact ages were not collected, which is a limitation for both this study and further work with the app. Another possible explanation is that some young people consider the design of the apps to be childish because they are more sophisticated than the designers and developers had anticipated.
As the aim of the study was to examine how adolescents assess the usability and quality of 2 mental health mobile apps, a cross-sectional design was justified as it allowed for an impression of a given topic at a given time point. However, a cross-sectional design does not allow for drawing conclusions about the long-term use of the apps or about their actual effectiveness.
Another limitation of this evaluation study is related to the apps’ functionality during the trial period. Comments were received for both Opp and NettOpp. Some adolescents found it difficult to navigate Opp. Several of the adolescents who evaluated NettOpp reported that the app crashed while they were using it. This is a frequent problem reported among app users [
A small proportion of adolescents who tested Opp thought the app could reduce mental health problems among adolescents. Only a few of the adolescents who evaluated NettOpp thought the app could reduce negative online incidents. It was not controlled in either the Opp sample or the NettOpp sample whether the adolescents had experienced mental health issues or negative online incidents. This was because we were mainly interested in mapping app usability and quality before they were evaluated in wider effectiveness studies.
This user evaluation study has provided helpful and important knowledge about how adolescents assess the apps’ usability, quality, and potential to achieve their given objectives. Overall, the findings are positive and promising. They suggest that mental health–promoting mobile apps offer a practical, relatively cheap, and appealing approach to reaching adolescents and improving their mental health. However, the cross-sectional design of the study does not allow for drawing any conclusions about the long-term use of such apps among adolescents. It is therefore necessary to conduct further studies that examine the use of mental health–promoting mobile apps in the long run. This study does not provide answers about the effectiveness of the 2 apps. It is therefore also necessary to conduct wider effectiveness studies for both apps in order to draw any conclusions on their actual goal achievement.
Mobile Application Rating Scale
System Usability Scale
We would like to thank the participants in this study. We would also like to thank our dear colleagues: Monica Martinussen, Frode Adolfsen, Marte Rye, Kyrre Breivik, and Reidar Jakobsen, who contributed to the development of the mobile apps. The development of the mobile app Opp was funded by a grant from the Dam Foundation through the Norwegian Council of Mental Health (Ref. 2020HE2-328433). The study was funded by RKBU North at UiT The Arctic University of Norway, with an additional grant from the Norwegian Directorate of Health. The publication charges for this article have been funded by a grant from the publication fund of UiT The Arctic University of Norway.
The data sets generated during or analyzed during this study are available from the corresponding author on reasonable request.
All authors provided substantial contributions to the evaluation study. HH wrote the first draft of the manuscript. SK and HK read, edited, and approved the final manuscript.
None declared.