This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2491) Attention Deficit Disorder (ADD/ADHD) (64) Human Factors and Usability Case Studies (809) User-centred Design Case Studies (602) Web-based and Mobile Health Interventions (3357)

Published on in Vol 6, No 10 (2022): October

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37137, first published .
Internet-Delivered Self-help for Adults With ADHD (MyADHD): Usability Study

Internet-Delivered Self-help for Adults With ADHD (MyADHD): Usability Study

Internet-Delivered Self-help for Adults With ADHD (MyADHD): Usability Study

Authors of this article:

Robin Maria Francisca Kenter1, 2 Author Orcid Image ;   Adrian Schønning1 Author Orcid Image ;   Yavuz Inal3 Author Orcid Image
Article Authors Cited by (4) Tweetations (3) Metrics

    Original Paper

    1Department of Clinical Psychology, Faculty of Psychology, University of Bergen, Bergen, Norway

    2Section for Research-Driven Innovation, Division of Psychiatry, Haukeland University Hospital, Bergen, Norway

    3Department of Design, Faculty of Architecture and Design, Norwegian University of Science and Technology, Gjøvik, Norway

    Corresponding Author:

    Robin Maria Francisca Kenter, PhD

    Department of Clinical Psychology

    Faculty of Psychology

    University of Bergen

    Christies gate 12

    Bergen, 5015

    Norway

    Phone: 47 48464314

    Email: robken@helse-bergen.no


    Background: Although effective pharmacological treatment exists, many adults with attention-deficit/hyperactivity disorder (ADHD) prefer a nonpharmacological option for managing their symptoms. Internet-delivered self-help interventions have the potential to address this unmet supportive care need reported by adults with ADHD, at relatively low costs. However, if the intervention does not offer optimal functions, content, and layout, it could decrease adherence and engagement and potentially compromise the effectiveness of such interventions. Thus, there is a need for examining the usability and factors that enhance and impair the usability of internet-delivered self-help interventions.

    Objective: This study evaluates the usability of an internet-delivered self-help intervention for adults with ADHD (MyADHD). The main goals were to (1) collect qualitative and quantitative data on usability and (2) identify usability problems.

    Methods: Individual think-aloud interviews and staged usability testing (N=5) were conducted to evaluate the usability of the MyADHD intervention in terms of function, content, and design. MyADHD end users provided iterative feedback to maximize engagement and usability. They performed tasks involved in operating the intervention and provided “think-aloud” commentary and postsession usability ratings. The interviews were recorded, transcribed verbatim, and analyzed.

    Results: Participants were satisfied with the overall usability of the program. The average perceived usability score out of 100 was 70 for the first round of testing and improved to 77.5 after applying modifications, with a mean score of 75.5 (SD 5.9) for all rounds of usability testing. The analysis of the interviews revealed 3 central themes: functionality, content, and layout.

    Conclusions: Optimizing the usability of internet-delivered self-guided interventions is a critical step in the design and development process. The usability testing in this study provided valuable information from users’ perspectives on the content and platform of the intervention. Analysis revealed the need for intervention enhancement with regard to design, functionality, and content from the perspective of potential end users. Overall, participants saw value in the MyADHD intervention and were confident that they could use it for the self-management of symptoms and expressed the desire to use the entire intervention when it becomes available. Through this development process, we produced an intervention that is likely to be used successfully and is ready for deployment in a randomized controlled trial.

    Trial Registration: ClinicalTrials.gov NCT04511169; https://clinicaltrials.gov/ct2/show/NCT04511169

    JMIR Form Res 2022;6(10):e37137

    doi:10.2196/37137

    Keywords

    usability testing; user evaluation; attention-deficit/hyperactivity disorder; self-guided intervention; internet-delivered; self-help; adults with attention-deficit/hyperactivity disorder; intervention; usability study; care needs; usability 


    Background

    Digital technology for the treatment and management of mental health issues has become increasingly available in recent years [1]. For common mental health disorders like depression and anxiety, internet-delivered interventions are part of official treatment guidelines [2]. Internet-delivered interventions either can be delivered with some form of guidance or can be entirely self-guided [3]. Effective internet-delivered self-guided interventions can dramatically increase treatment accessibility at relatively low costs, as they can be distributed over the internet directly to the user’s computer, without the need for therapist guidance [4]. However, in self-guided internet-delivered interventions, a commonly described issue is low adherence and engagement [5].

    Internet-Delivered Interventions for Adults With ADHD

    For people with attention-deficit/hyperactivity disorder (ADHD), few evidence-based internet-delivered interventions for adults exist today. ADHD in adults is characterized by pervasive symptoms of inattention, hyperactivity/impulsiveness, or both that persist across different situations [6]. With an estimated prevalence of 2%-3%, ADHD can be counted as a common mental disorder in adulthood [7-9]. Although effective pharmacological treatment exists [10], many adults with ADHD prefer a nonpharmacological option for managing their symptoms [11]. In this case, self-guided psychological interventions that can be delivered over the internet might have the potential to meet the treatment preferences of adults with ADHD. These types of interventions can help manage symptoms of ADHD by providing general information, tailored advice, support, and skill training via technology (eg, smartphones, tablets, or websites). The few studies that examine internet-delivered psychological treatments for adults with ADHD show good clinical outcomes [12-14]. Self-guided internet-delivered interventions for adults with ADHD have the potential to offer evidence-based interventions to many individuals, with 24/7 access to information, in the comfort of their own homes, where they can read and reread materials.

    However, self-guided interventions are also associated with higher dropout rates than guided interventions. Since adherence is related to outcome, it is important to focus on all factors that could improve adherence to self-guided internet-delivered interventions. This starts with designing good systems that address the needs of the end users and have good usability. The primary reasons for low engagement with internet-delivered interventions has been a lack of user-centered design and poor usability [15]. Interventions with poor usability may make it difficult to interact with the intervention, which could lower the engagement, which then can lead to suboptimal clinical outcomes or rejection of the intervention by the user. This might be especially a problem for adults with ADHD, where the cardinal symptoms often are inattention and impulsiveness. It is, therefore, important to test the usability of a new internet-delivered intervention for ADHD during the development process, so that usability flaws are known at an early stage and can be fixed before the intervention goes to full trial.

    Usability

    Usability is defined by the International Organization for Standardization [16] as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” It also refers to the quality of a user's experience when interacting with systems, including websites and mobile apps. The usability of an internet-delivered intervention has a direct influence on user satisfaction. When an internet-delivered intervention is not well designed, it can prevent users from effectively and efficiently using the program. Usability evaluation is a very valuable resource to understand how easy it is to use a developed system for new users. It can be summative (collecting evidence that the intervention is usable) or formative (informing the redesign and improvement of the intervention) [17].

    This study describes the results from a formative usability evaluation of the MyADHD intervention [18] that gathered feedback from end users and improved the design as part of an iterative design process.

    Formative Usability Evaluation of MyADHD

    In this study, we evaluated the usability of a first prototype of an internet-based intervention for adults with ADHD. The main goals were to (1) collect qualitative and quantitative data on usability and (2) identify usability problems. An iterative development process was used to promote the further development of the content, visual design, and interaction design. Once the first round of usability testing was completed, we evaluated, determined improvements, implemented the improvements, and retested the updated prototype. We included 5 rounds with usability testing with 5 potential end users, as 5 users are often enough to identify 80% of all usability problems [19]. The goal was to identify understandability, ease of learning the intervention’s platform, and appropriateness of the intervention. Through the testing, we were provided with feedback about what does and does not work in the intervention for end users and determined whether the features of the intervention were acceptable and feasible for users.

    Objectives

    This study aims to investigate the usability of a self-guided internet-delivered intervention (MyADHD), which targets symptoms of ADHD among adults.


    Procedures

    The usability evaluation was performed in 2 parts: (1) the expert evaluation and (2) the user evaluation. This study reports on findings from the first stage of the formative research process: usability testing. The aim is to investigate the usability of a self-guided internet-delivered intervention in terms of function, content, and design.

    Part 1: Expert Evaluation

    The first and last author of the paper (RMFK and YI) conducted an expert evaluation prior to the user evaluation. Through this evaluation method, we detected usability problems with the interface early in the process [20]. The expert evaluation involved an evaluation on how well a design supported its target audience in achieving their specified goals with effectiveness, efficiency, and satisfaction. The usability expert (YI) conducted a review according to a set of generally accepted usability guidelines, as well as their personal knowledge of the design’s domain. The evaluation was based on Nielsen’s heuristics. Problems detected by the evaluators were addressed immediately, so they did not influence the second part: the user evaluation.

    Part 2: User Evaluation

    Five individuals diagnosed with ADHD were recruited via the Norwegian patient association (ADHD Norge) to participate in a 1-hour laboratory-based usability test. The inclusion criteria for the usability tests were (1) a self-reported ADHD diagnosis, (2) age >18 years, and (3) willing to participate and able to meet at the laboratory. In the laboratory, participants used a laptop and were asked to use the prototype of MyADHD intervention. The test entailed the completion of 10 goal-oriented tasks (see Textbox 1).

    Tasks.
    1. Go to the program (adhd.youwell.no) and log in
    2. Start module 1, read the material, and say out loud what you think
    3. Start module 2 and try one of the breathing exercises
    4. Log your experience of the breathing exercise in the exercise log
    5. Complete module 2
    6. Go to the home page
    7. Fill out the questionnaires of module 3
    8. Go to my diary and fill out an entry
    9. Go to my calendar page and see this week’s homework assignments
    10. Try another breathing exercise and log your experience again
    Textbox 1. Tasks.

    First, the participants provided demographic information and reported on their experience with personal computers and the use of internet. Next, they were asked to perform a series of tasks in the platform and intervention. Two facilitators were present in each usability session. One led the session and 1 observed. Participants were asked to “think aloud” (ie, provide continuous commentary) in accordance with Concurrent Think-aloud Method (CTA) while operating the system [21]. The goal with CTA was to encourage participants to keep a running stream of consciousness as they worked with the tasks. This helped to understand participants’ thoughts as they tried to work through the intervention and elicit real-time feedback and emotional responses to the intervention. This is a common approach to usability testing that enables evaluation of the ease with which a system is learned [22]. Participants provided feedback on the MyADHD intervention (ie, layout, color, text, readability, and videos), and the content and provided information on their preferences regarding the use of MyADHD. If the participant asked a question, the facilitator remained neutral and replied with “What do you think?” or “What would you do?”

    During the CTA, the observer took notes and used an audio recorder to capture all that the participant did and said. At the end of the session, the System Usability Scale [23] was administered as an online survey in the platform. The information retrieved from the think-aloud interviews was used to further optimize the MyADHD intervention.

    Ethics Approval

    This study was reviewed and approved by the Norwegian Regional Committee for Medical and Health Research Ethics, REC South East #203804. The participants provided written informed consent to participate in this study. As a reimbursement for their time, participants received gift vouchers worth NOK 200 (US $19).

    Questionnaires

    The System Usability Scale

    The System Usability Scale (SUS) [23] is a widely used source for the assessment of the perceived usability of an evaluated system [24]. The SUS has sufficient reliability (coefficient α .9). It contains 10 items with a response from 1 (strongly disagree) to 5 (strongly agree), with 5 positive statements and 5 negative statements. The items are scored on a 5-point Likert scale. The sum of the items leads to a general measure of perceived usability with the total score varying between 0 and 100. Based on different studies in the literature, an overall usability score of below 70 shows poor usability of the evaluated system [25,26]. Higher overall scores represent a high quality of usability. The scale allows item-based analysis by calculating the mean score and standard deviation of each item. Minor wording changes do not appear to affect SUS scores [25,27]. In our research, we tailored the scale by replacing the term “system” with “the intervention.”

    Additionally, the participants were asked questions on sociodemographic and internet use characteristics.

    The MyADHD Intervention

    MyADHD development was theory based and person based [28]. The planning phase consisted of qualitative focus groups, the development of guiding principles, and a literature review. The first prototype of the intervention was delivered via an online secure portal, which is accessible on laptops and personal computers. The intervention is a structured self-guided intervention with modified elements from cognitive behavioral therapy, dialectical behavioral therapy, and goal management training to target specific challenges experienced by adults with ADHD. Kenter et al [18] described the development of the content of the intervention. The intervention consists of 7 training modules. In this study, we used the first 3 modules of the intervention (see Multimedia Appendix 1).

    The main goals of the intervention are to help participants obtain improved functioning in daily life activities and offer strategies that aim to reduce stress, reduce inattention, and improve quality of life. Each module includes psychoeducation alongside the text, audio, and video material instructing participants in the use of specific techniques. Further, modules include case vignettes and lived-experience videos that clarify the written content and help participants make connections between the material and their own experiences. Since all modules have the same structure and ingredients, we only used the first 3 modules in our test. Table 1 shows an overview of the intervention modules that were used in this study.

    Table 1. Overview of the first 3 modules of MyADHD.
    ModuleRationale and contentExercises and videos
    1. Start moduleGoal setting and practical information about how to use the internet-delivered intervention
    • One goal setting exercise
      • ‎
    • Describe how ADHDa affects your life exercise
      • ‎
    • One lived-experience video
      • ‎
    2. Mindful awarenessInattention is a core symptom of ADHD. In this module, participants are given information about different aspects of attention and concentration and how to cope with impairment. In this module, the participants start training on mindful awareness (“being here and now”) by focusing on their breathing (based on dialectical behavioral therapy).
    • Three different types of breathing exercises
      • ‎
    • Two lived-experience videos
      • ‎
    3. Inhibition trainingImpulsivity and loss of impulse control are common among adults with ADHD. This module consists of exercises focusing on impulse control and goal-oriented/goal-directed behavior (stop, observe, proceed, and check; based on goal management training).
    • Two STOP exercises
      • ‎
    • Two lived-experience videos
      • ‎

    aADHD: attention-deficit/hyperactivity disorder.

    Data Collection and Analysis

    Tasks and Questionnaires

    In each individual usability testing session, following informed consent, the participants were first asked to think aloud while using the program and doing the tasks (see Textbox 1). The procedure was pretested with an individual who was not involved in the study. A facilitator guided the participant through the usability sessions by presenting the tasks and reminding the participants to think aloud by questions like “Tell me what you’re thinking,” “what are you looking for?,” and “What do you think now?” The sessions were audiotaped and transcribed verbatim. We coded and analyzed the transcripts using NVIVO [29].

    Data Analyses

    The usability tests were recorded and transcribed verbatim by the second author. Transcripts were analyzed thematically using an iterative coding procedure. The focus of the analysis was on the features of the intervention that needed to be redesigned or improved. The categories were identified using an iterative process of reading and rereading the transcripts. Usability issues were coded into categories.


    Participants

    The participants had a mean age of 38.4 (SD 16.3) years. Three were men and 4 were highly educated (high school or higher). One participant reported difficulty using computers and the internet, while the others had reported good computer skills, see Table 2. All participants had ADHD diagnoses, and none of them had previously used an internet-delivered intervention before.

    Table 2. Demographics.

    		Value
    Demographics

    		Age (years)


    		Mean (SD)38.4 (16.3)


    		Range25-62

    		Gender, n


    		Male3


    		Female2

    		Educational level, n


    		Middle1


    		High4
    Internet and computer skills, n

    		Self-reported good computer skills4

    		Self-reported good internet skills4

    Usability Issues

    The transcripts revealed 3 main categories of barriers that limited usability: (1) functionality, (2) content, and (3) layout.

    Functionality

    This category referred to the need for the intervention to be easy to use, navigate, and have a logical flow. Analysis showed that usability was limited when navigation was difficult. The participants experienced problems related to, for example, unclear navigation (“I can’t move back to the homepage,” “where do I go to see my past entries?”), unclear functionality (“Does the calendar synchronize with my own phone?”), and it was unclear whether filled-out questionnaires were saved and submitted (“Did I save this now? I am not sure, can I go back? How do I check this?”). Furthermore, textboxes were too small to type in larger amounts of text.

    In preparation of new rounds of usability testing, back and forward buttons were added, more explanation was offered with the different elements such as calendar, automatic feedback that answers on questionnaires were sent in, and textboxes were made larger. At the end of usability testing, the participants found the calendar still confusing to use. Due to this, the calendar was removed from the intervention, and the participants were encouraged to use their own private calendar, for example, their mobile phone calendar or paper calendar. After this adjustment, think-aloud comments to the functionality were more neutral (“OK, now I click here to go to the next page”), positive (“Nice! I did not remember what I was supposed to do, but I can go back to read the previous page”), or focused on the specific intervention content.

    Content

    Overall, the participants had a general positive impression of the content in the program. The program felt “positive,” “light,” and “useful.” The exercises were perceived as helpful and useful (“This really speaks to me,” “this makes perfect sense”). Usability testing revealed that several participants (n=4) had difficulty understanding the text when it was too lengthy, with large blocks of text, making the information difficult to process, which limited the usability.

    A point of improvement involved the wording in the content, whereby language was perceived as too negative (“I do not recognize myself in having problems, I might have challenges, not problems”) and too complicated (“I have to read the most sentences twice”). There were also inconsistencies in wording that made it difficult to use the intervention (“first you call it dashboard, and now I have to go back to the homepage,” “it is called a program or an intervention? It seems now that these are two different things”).

    This resulted in changes in the wording; more positive wording was used, which resulted in more positive comments about the wording (“It is nice with empathy about that it can be difficult to be part of this ADHD intervention”). Wording was checked for inconsistencies, spelling errors were removed, and text was shortened or chunked up in smaller sections so that it became easier to read. We also added a short summary at the end of each module. All participants liked the videos, exercises, and psychoeducational text and viewed them as helpful.

    Layout

    All participants commented that they saw value in the images, visual aids, and videos. They liked the layout and described it as “calm,” “beautiful, yeah it is esthetic,” and “friendly” with “nice colors and pretty images.” Regarding the questionnaires, the participants found it hard to fill out the questionnaires because of the large amount of text and answering options.

    Satisfaction With the Intervention

    All participants felt confident they would be able to use the platform (n=3 strongly agreed, n=2 agreed). The participants were satisfied with how easy it was to use and viewed the intervention as helpful. All participants were able to perform the tasks and learn to use the intervention and the platform on their own.

    Perceived Usability Outcomes

    The results for perceived usability (as measured by the SUS) are presented in Table 3. The analysis of the data identified that the intervention was rated with an average score of 75.5 (SD 5.9). Of the 5 participants, 1 rated the MyADHD intervention as “excellent,” while 4 others rated it as “good.”

    Table 3. Result of the System Usability Scale.
    ItemsP1aP2P3P4P5Mean
    1. I think that I would like to use this intervention433433.4
    2. I found the intervention unnecessarily complex222111.6
    3. I found the intervention easy to use444554.4
    4. I think I would need support from a technical person to use this intervention121111.2
    5. I found the various functions in this intervention were well integrated344333.4
    6. I thought there was too much inconsistency in this intervention322332.6
    7. I would imagine that most people would learn to use this intervention quickly334543.8
    8. I found the intervention very cumbersome to use222121.8
    9. I felt very confident using the intervention353454
    10. I needed to learn a lot of things before I could start using this intervention122121.6
    SUSb,c score (0-100 scale)7072.572.58577.575.5

    aP1-5: participant 1-5.

    bSUS: System Usability Scale.

    cTotal score reversed to a 0-100 scale.


    Principal Results

    Usability testing demonstrated that all individuals could perform the desired tasks, and that they learned to use the intervention quickly. Results from the SUS revealed that the program obtained high scores, indicating that the intervention was considered very useful. Staged iterative usability testing was essential for discovering intervention enhancement needs (eg, more visual aids, more buttons) and for resolving design, functionalities (eg, more feedback on actions and explanation to functionalities), and content inadequacy (eg, inconsistencies, difficult wording, lengthy text). Overall, the participants saw value in the MyADHD intervention and were confident they could use it for self-management of symptoms and expressed the desire to use the entire intervention when it becomes available.

    Optimizing usability early in the process is a critical step in the development process of self-guided internet-delivered interventions [28], especially with self-guided interventions, where uptake and adherence are often low [4]. While there is a large number of studies examining the feasibility and efficacy of internet-delivered interventions for common mental health disorders like depression and anxiety [1], there are only a few studies that examine internet-delivered interventions for adults with ADHD [12,13,30]. These studies reported low adherence to the interventions. This is problematic as adherence is related to outcome [31,32]. Additionally, low usability could have an impact on the effectiveness of such interventions [33]. A study on usability is therefore important and relevant, as it may help to ensure that the interventions are well designed and therefore increase the interest and number of people who can benefit from it. End users have high expectations of digital products, and because of difficulties with inattention, adults with ADHD specifically might drop out if the intervention does not have optimal usability. We did not find other studies examining the usability of internet-delivered interventions for adults with ADHD.

    The overall SUS scores found in this study are comparable to other usability studies of internet-delivered interventions. A study of transdiagnostic internet-delivered treatment [34] found a mean SUS score of 81.89; a study on blended care [35] found a mean SUS score of 76.3 for the internet-delivered treatment, while a study on web-based support for people with mild intellectual disabilities [36] found a low mean SUS score of 56.4 and 51.1.

    Limitations and Strengths

    For usability tests, 5 users are often enough to identify 80% of all usability problems [19]. The optimal number of participants required is not clear. This depends on the problems raised by the participants and whether the participants give an adequate reflection of potential end users. Despite the small sample, participants with different profiles and backgrounds were included, and within this sample, theme saturation was achieved. We believe that our sample reflects the population of potential end users of MyADHD.

    The staged iterative usability tests provided knowledge regarding whether specific tasks could be performed in the intervention and gave direct input on how potential end users would use the intervention. The study was not conducted to identify every single usability problem but rather to show how usability testing with a small sample could identify usability problems that experts did not recognize prior, which allowed us to make significant improvements to the intervention before going full trial. The methods used were effective in identifying elements that needed modification.

    Conclusions

    Innovative technologies can play an important role in helping adults with ADHD manage their symptoms better. For such interventions, delivered over the internet without clinician support, to be viable, they need to be developed with the needs, characteristics, and preferences of their intended end users in mind. At the end of a user-centered development process, with usability evaluations, the MyADHD intervention was deemed ready for testing in real-world conditions.

    Acknowledgments

    This publication is part of the Introducing Mental Health Through Adaptive Technology (INTROMAT) project. The Research Council of Norway (NFR# 259293) funded the study. The funding body played no role in the design of the study, the collection, analysis, and interpretation of data; or in writing the manuscript.

    Authors' Contributions

    RMFK is the chief investigator in this study, led the proposal and study protocol development, contributed to the analysis, and drafted the manuscript. AS contributed to the execution of the study, the analysis, and the manuscript. YI contributed to the study design, the development of the platform interface, and the drafting of the manuscript. All authors critically reviewed and approved the final version of the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Screenshot of the intervention.

    DOCX File , 359 KB
    1. Andersson G, Titov N, Dear BF, Rozental A, Carlbring P. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry 2019;18(1):20-28 [FREE Full text] [CrossRef] [Medline]
    2. Folker AP, Mathiasen K, Lauridsen SM, Stenderup E, Dozeman E, Folker MP. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv 2018 Mar;11:60-70. [CrossRef]
    3. Andersson G, Titov N. Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry 2014 Feb;13(1):4-11. [CrossRef]
    4. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry 2017 Apr;74(4):351-359. [CrossRef] [Medline]
    5. Karyotaki E, Kleiboer A, Smit F, Turner DT, Pastor AM, Andersson G, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med 2015 Apr;45(13):2717-2726. [CrossRef]
    6. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. Washington, DC: American Psychiatric Association; 2013:591-643.
    7. Faraone SV, Biederman J, Mick E. The age-dependent decline of attention deficit hyperactivity disorder: a meta-analysis of follow-up studies. Psychol Med 2006 Feb;36(2):159-165. [CrossRef] [Medline]
    8. Franke B, Michelini G, Asherson P, Banaschewski T, Bilbow A, Buitelaar JK, et al. Live fast, die young? A review on the developmental trajectories of ADHD across the lifespan. Eur Neuropsychopharmacol 2018 Oct;28(10):1059-1088. [CrossRef]
    9. Kessler RC, Adler LA, Barkley R, Biederman J, Conners CK, Faraone SV, et al. Patterns and predictors of attention-deficit/hyperactivity disorder persistence into adulthood: results from the National Comorbidity Survey Replication. Biol Psychiatry 2005 Jun;57(11):1442-1451. [CrossRef]
    10. Spencer T, Biederman J, Wilens T. Pharmacotherapy of attention deficit hyperactivity disorder. Child Adolesc Psychiatr Clin N Am 2000 Jan;9(1):77-97. [Medline]
    11. Solberg BS, Haavik J, Halmøy A. Health care services for adults with ADHD: patient satisfaction and the role of psycho-education. J Atten Disord 2015 Jun;23(1):99-108. [CrossRef]
    12. Moëll B, Kollberg L, Nasri B, Lindefors N, Kaldo V. Living SMART — a randomized controlled trial of a guided online course teaching adults with ADHD or sub-clinical ADHD to use smartphones to structure their everyday life. Internet Interv 2015 Mar;2(1):24-31. [CrossRef]
    13. Nasri B. New Approach to the Treatment of ADHD Through Internet and Smartphones: From Impairment to Improvement. Sweden: Karolinska Institutet; 2017.
    14. Nordby ES, Kenter RM, Lundervold AJ, Nordgreen T. A self-guided internet-delivered intervention for adults with ADHD: a feasibility study. Internet Interv 2021 Sep;25:100416. [CrossRef]
    15. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018 Aug;21(3):116-119. [CrossRef] [Medline]
    16. ISO 9241-11:2018: Ergonomics of Human-System Interaction - Part 11: Usability: Definitions and Concepts. International Organization for Standardization. 2018.   URL: https://www.iso.org/standard/63500.html [accessed 2022-09-12]
    17. Inal Y, Wake JD, Guribye F, Nordgreen T. Usability evaluations of mobile mental health technologies: systematic review. J Med Internet Res 2020 Jan;22(1):e15337 [FREE Full text] [CrossRef] [Medline]
    18. Kenter RMF, Lundervold AJ, Nordgreen T. A self-guided internet-delivered intervention for adults with ADHD: a protocol for a randomized controlled trial. Internet Interv 2021 Dec;26:100485 [FREE Full text] [CrossRef] [Medline]
    19. Nielsen J, Landauer KT. A mathematical model of the finding of usability problems. In: Proceedings of the INTERACT'93 and CHI'93 Conference on Human Factors in Computing Systems. 1993 Presented at: CHI ’93; April 24-29, 1993; Amsterdam, The Netherlands. [CrossRef]
    20. Nielsen J. Finding usability problems through heuristic evaluation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1992 Presented at: CHI’92; May 3-7, 1992; Monterey, CA. [CrossRef]
    21. Fan M, Lin J, Chung C, Truong KN. Concurrent think-aloud verbalizations and usability problems. ACM Trans Comput-Hum Interact 2019;26(5):1-35. [CrossRef]
    22. Brinck T, Gergle D, Wood SD. Usability for the Web: Designing Web Sites That Work. San Francisco, CA: Elsevier; 2001.
    23. Brooke J. Sus: a quick and dirty usability. Usability Eval Ind 1996;189(3):4-7. [CrossRef]
    24. Lewis JR. The System Usability Scale: past, present, and future. Int J Human–Comput Interact 2018 Mar;34(7):577-590. [CrossRef]
    25. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009;4(3):114-123 [FREE Full text]
    26. McLellan S, Muddimer A, Camille Peres S. The effect of experience on system usability scale ratings. J Usability Stud 2012;7(2):56-67 [FREE Full text]
    27. Lewis JR, Sauro J. The factor structure of the system usability scale. In: International Conference on Human Centered Design.: Springer; 2009 Presented at: HCI International 2009; July 19-24, 2009; San Diego, CA. [CrossRef]
    28. Yardley L, Ainsworth B, Arden-Close E, Muller I. The person-based approach to enhancing the acceptability and feasibility of interventions. Pilot Feasibility Stud 2015;1:37 [FREE Full text] [CrossRef] [Medline]
    29. Castleberry A. NVivo 10 [software program]. Version 10. QSR International; 2012. Am J Pharmaceutical Education 2014 Feb;78(1):25-27. [CrossRef]
    30. Pettersson R, Söderström S, Edlund-Söderström K, Nilsson KW. Internet-based cognitive behavioral therapy for adults With ADHD in outpatient psychiatric care. J Atten Disord 2016 Jul;21(6):508-521. [CrossRef]
    31. Titov N, Dear B, Nielssen O, Staples L, Hadjistavropoulos H, Nugent M, et al. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv 2018 Sep;13:108-115 [FREE Full text] [CrossRef] [Medline]
    32. Simpson HB, Marcus SM, Zuckoff A, Franklin M, Foa EB. Patient adherence to cognitive-behavioral therapy predicts long-term outcome in obsessive-compulsive disorder. J Clin Psychiatry 2012 Sep;73(09):1265-1266. [CrossRef]
    33. Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015 Jan;39(2):1. [CrossRef]
    34. Díaz-García A, González-Robles A, Fernández-Álvarez J, Castilla D, Mira A, Bretón J. Usability of a transdiagnostic internet-delivered protocol for anxiety and depression in community patients. In: Cipresso P, Serino S, Villani D, editors. Pervasive Computing Paradigms for Mental Health. MindCare 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Cham: Springer; 2019.
    35. Etzelmueller A, Radkovsky A, Hannig W, Berking M, Ebert DD. Patient's experience with blended video- and internet based cognitive behavioural therapy service in routine care. Internet Interv 2018 Jun;12:165-175. [CrossRef]
    36. de Wit J, Dozeman E, Ruwaard J, Alblas J, Riper H. Web-based support for daily functioning of people with mild intellectual disabilities or chronic psychiatric disorders: A feasibility study in routine practice. Internet Interv 2015 May;2(2):161-168. [CrossRef]

    ‎
    ADHD: attention-deficit/hyperactivity disorder
    SUS: System Usability Scale

    Edited by A Mavragani; submitted 08.02.22; peer-reviewed by H Mehdizadeh, V Prasad; comments to author 04.07.22; revised version received 30.08.22; accepted 30.08.22; published 21.10.22

    Copyright

    ©Robin Maria Francisca Kenter, Adrian Schønning, Yavuz Inal. Originally published in JMIR Formative Research (https://formative.jmir.org), 21.10.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.