Original Paper
Abstract
Background: Shared decision-making between clinicians and service users is crucial in mental health care. One significant barrier to achieving this goal is the lack of user-centered services. Integrating digital tools into mental health services holds promise for addressing some of these challenges. However, the implementation of digital tools, such as mobile apps, remains limited, and attrition rates for mental health apps are typically high. Design thinking can support the development of tools tailored to the needs of service users and clinicians.
Objective: This study aims to develop and beta test a digital tool designed for individuals with severe mental disorders or substance use disorders to facilitate shared decision-making on treatment goals and strategies within mental health services.
Methods: We used a user-centered design approach to develop iTandem, an app facilitating collaborative treatment between service users and clinicians. Through qualitative interviews and workshops, we engaged 6 service users with severe mental disorders or substance use disorders, 6 clinicians, and 1 relative to identify and design relevant app modules. A beta test of iTandem was conducted to refine the app and plan for a pilot trial in a clinical setting. After 6 weeks of app use, 5 clinicians and 4 service users were interviewed to provide feedback on the concept, implementation, and technical issues. Safety and ethical considerations were thoroughly discussed and addressed.
Results: To avoid overload for the service users, we applied a pragmatic take on module content and size. Thus, iTandem includes the following 8 modules, primarily based on the needs of service users and clinicians: Sleep (sleep diary), Medication (intake and side effects), Recovery (measures, including well-being and personal recovery, and exercises, including good things and personal strengths), Mood (mood diary and report of daily feelings), Psychosis (level of positive symptoms and their consequences and level of negative symptoms), Activity (goal setting and progress), Substance use (weekly use, potential triggers or strategies used to abstain), and Feedback on therapy (of individual sessions and overall rating of the past week). For the beta testing, service users and clinicians collaborated in choosing 2-3 modules in iTandem to work with during treatment sessions. The testing showed that the app was well received by service users, and that facilitation for implementation is crucial.
Conclusions: iTandem and similar apps have the potential to enhance treatment outcomes by facilitating shared decision-making and tailoring treatment to the needs of service users. However, successful implementation requires thorough testing, iterative development, and evaluations of both utility and treatment effects. There is a critical need to focus on how technology integrates into clinical settings—from development to implementation—and to conduct further research on early health technology assessments to guide these processes.
doi:10.2196/57614
Keywords
Introduction
...in what way has it (iTandem) been helpful?
Like right now, I just went into the “substance use” module to write that I managed to say ‘no’ to buying heroin. Or if I see needles and stuff, I choose to try and get away – move away from where there are needles and that kind of stuff available.
[Beta tester of iTandem]
The ability to communicate effectively with service users is essential in health care. Therapeutic communication can lead to several benefits, including a more accurate description of symptoms, improved detection of emotional states, more appropriate therapeutic measures, better identification of the service user’s needs and understanding of their disorder and treatment, and an increased likelihood of shared decision-making between clinicians and service users [-]. Engaging individuals with severe mental disorders, substance use disorders, or both in their recovery process can be challenging. Various factors contribute to this difficulty, including a lack of trust in health care services [], poor therapeutic alliance [], reduced insight into their illness [], diminished cognitive functioning [], and issues with motivation and apathy [,].
Priebe and colleagues [] outlined 5 guiding principles for effective communication in mental health services: focusing on the service users’ concerns, demonstrating positive regard and personal respect, appropriately involving service users in decision-making, showing genuineness with a personal touch, and using a psychological treatment model. Emphasizing service users’ concerns aligns closely with the concept of “patient-centeredness” in health care. A recent review [] defined centeredness as encompassing attributes such as being treated as unique, being heard, and sharing responsibility. However, Feldthusen and colleagues [] emphasize that centeredness is a complex concept, with its essence rooted in the interactions and communication between clinicians and service users. For individuals with severe mental illness, achieving a sense of positive regard and personal respect can be challenging. Therefore, adopting a nonjudgmental approach has been shown to be particularly valuable []. Shared decision-making is a widely recognized concept in medicine, but it remains challenging to implement effectively in mental health care. For individuals with severe mental disorders, barriers include clinicians’ paternalistic approaches to care [] and service users’ internalized stigma []. Increasing service user involvement in activities such as goal setting, addressing values and preferences, and working with motivational factors is essential []. Additionally, exploring shared risk-taking can further enhance communication []. Genuineness and a personal touch involve openness and sharing but also reflect the therapist’s behavior and characteristics, such as warmth and flexibility []. Finally, adopting a coherent client-centered therapy approach, such as solution-focused therapy, problem-solving therapy, or cognitive behavioral therapy, is often preferable [].
Digitalization offers a potential solution to some of these challenges by making treatment more focused and efficient. There has been growing interest in the use of patient-reported outcomes and patient-reported outcome measures (PROMs) in health care [], which are frequently digitalized. The inclusion of validated assessment tools in digital formats provides clear advantages. However, a significant barrier for health professionals is the time and expertise required to interpret and integrate the extensive findings into their workflow []. Additionally, Nguyen and colleagues [] identified perceived irrelevance and lack of value for the service user as barriers to completing PROMs. As a result, both service users and clinicians emphasize the need for tools that are functional, straightforward, and concise, with as few questions as possible.
There has been a surge in smartphone apps designed to support individuals with mental health challenges, readily available on platforms such as the App Store (Apple Inc.) or Google Play (Alphabet Inc.). Most of these apps are standalone offerings in the free market, focusing on symptoms such as depression, anxiety, or stress. However, they are rarely developed to meet the safety standards typically required in health care settings, making integration into clinical practice difficult. Additionally, while many apps appear acceptable and feasible initially, attrition rates over time are often high []. Individuals with mental disorders may face unique challenges that demand flexible solutions, as therapeutic targets can shift throughout the course of illness. Thus, assessing attrition alone is often insufficient without a deeper understanding of the context. Proposed solutions include placing greater emphasis on external motivating factors and user experience (UX) design, as well as integrating digital mental health solutions into existing care pathways [,].
KISS—“keep it simple, stupid”—is a design principle [] frequently used in design processes and should be applied to service design as well []. This principle emphasizes that most systems function optimally when kept simple rather than overly complex []. Simpler innovations are more likely to be successfully deployed. Aligning with this concept, human-centered design has gained significant attention in recent years. It involves creative approaches that enable end users to influence the design of technology, tailoring it to better meet their needs []. One such approach is participatory design. A fundamental principle in this approach is that people affected by a decision should have the opportunity to influence it, with the goal of designing alternatives to improve quality of life []. Despite extensive research on the importance of applying design principles to technology to make it effective, only a limited number of studies in mental health care report using co-design and expert designers in their developmental process [,].
A previous service design project focused on the establishment of a low-threshold service for people with psychosis in health care services []. Based on insights from service users, caregivers, and clinicians, easy access to a first assessment of psychosis was established. During this project, it became evident that individuals with lived experience were interested in new ways of collaborating with health care services. They sought tools that could enhance the effects of treatment, make better use of the time between sessions, and help them stay focused on their goals. As one service user remarked, “I live my life between the appointments.” This highlighted the need to explore how technology could strengthen the focus on treatment between appointments.
The main aim of the iTandem development was to investigate the needs described by service users and clinicians and to integrate them into an app designed to enhance shared decision-making for people with psychotic, bipolar, or substance use disorders in treatment. We applied design thinking principles to incorporate user needs and feedback throughout the process [,]. Our long-term goal is to explore the barriers and enablers for integrating technology into shared decision-making in mental health care and to examine how the integration of digital tools affects the mode of service delivery. In this paper, we will present (1) the design process behind the development of iTandem and (2) the results of beta testing. Finally, we will discuss the importance of future research on the digital enhancement of service delivery.
Methods
Project Setting
iTandem was developed at the Early Intervention in Psychosis Advisory Unit for South East Norway (TIPS SE) at Oslo University Hospital (OUH). TIPS SE works to support mental health services in detecting and treating patients with early psychosis in accordance with national guidelines, with a particular emphasis on how these services are organized.
Design Process
Overview
External funding from the South-Eastern Norway Regional Health Authority enabled the hiring of experts in service design and interaction design to lead the process. The process progressed through 3 phases, involving multiple stakeholders including patients, clinicians, developers, data protection officers, and clinicians, in addition to the original project team consisting of researchers and clinicians from OUH (KLR, EAB, KK, EM, EBR, CS, TGV, and SRA). The phases involved are discussed in the following subsections.
Preparation Phase
Qualitative interviews were conducted to inform the design process by understanding the needs and preferences regarding app content. Six patients diagnosed with psychotic or bipolar disorder, as well as 1 relative, were interviewed individually. In addition, 2 focus groups were held with 6 clinicians: 1 nurse practitioner, 1 clinical psychologist, 1 psychiatrist, and 1 medical doctor (all from an early intervention in a psychosis team); as well as 1 psychologist from a general psychiatric outpatient service and 1 from the child and adolescent services for early psychosis.
Insight Phase
A private design company managed the design process and developed a clickable app prototype to inform the developers. The design company, in collaboration with researchers, agreed upon a project plan that included several meetings with clinicians (the same clinicians who had been involved in phase 1) and 4 patients (the same as in phase 1). The designers utilized the insights collected during phase 1 and knowledge generated through workshops to focus on themes that appeared to be both significant and feasible to work with, given the project’s budget and time constraints.
Developmental Phase
This phase was carried out in collaboration with the Mobile App team at the Center for Information Technology (USIT) at the University of Oslo (UiO), which developed a working app capable of transferring encrypted data to secure storage within Services for Sensitive Data (Tjenester for Sensitive Data or TSD). This was necessary to meet the safety requirements for the storage of sensitive data, as mandated by the data protection officer at OUH. A risk and vulnerability analysis was conducted at OUH. During this phase, 4 patients and 2 clinicians were invited to test the iTandem app during development to ensure it could be adjusted based on short iterations.
Project Organization
Overview of the Project Group
The project group included researchers and clinicians working with early psychosis treatment. TIPS SE also has a board composed of people with lived experience of bipolar and psychotic disorders. These individuals were either currently in treatment or had previously been in treatment at mental health services in OUH during the development project. The roles of the different groups are presented below.
Researchers
During phase 1, researchers conducted individual qualitative interviews with 6 service users and 1 relative and led 1 focus group with 6 clinicians. The purpose was to explore the challenges participants believed digital tools could address. During phase 2, the researchers served as a bridge between the clinic and the design team, facilitating workshops, collaborating with user representatives, and ensuring ethical issues were handled appropriately. They also suggested improvements to the app content to ensure it aligned with clinical guidelines and recommendations regarding treatment. During phase 3, the researchers facilitated and oversaw the developmental process of the app at USIT.
Designers
During phase 2, designers planned and conducted workshops with clinicians and researchers. Throughout the project, they interviewed clinicians and user representatives, and user tested different options in the studio with both clinicians and user representatives in an iterative process. The process was based on design thinking, working with divergent and convergent thinking according to the Double Diamond framework []. During phase 3, UX designers, engaged in making products or services usable, enjoyable, and accessible, were involved at USIT to further develop the interface and add functionality based on tests from the prototyping phase.
Service Users
Service users with severe mental disorders or substance use disorders were involved throughout all phases of the project. Service users (aged 18-30 years) and 1 relative (parent) provided input on functionality and were invited to user test different prototypes. This involvement had an impact on the final app interface and functionality. Some of these service users were among those interviewed during phase 1. Additionally, we met with the board of user representatives in the Department of Substance Use and Addiction at OUH to discuss app content relevant to them. Service users not directly involved in development were invited to beta test iTandem.
Clinicians
During the process, involved clinicians highlighted important aspects essential for implementation, and shared their ideas on how iTandem could be valuable in their daily clinical work. Additionally, they user tested the app during iterative cycles and provided feedback on how they believed the app would be received by their patients.
Software
The iTandem app was developed using the React Native Open-Source UI Software Framework (Meta Platforms, Inc.) and is available for both iOS and Android platforms. It can be downloaded from the App Store or Google Play. Data are transferred via HTTPS as JSON files through a secure web survey provider (Nettskjema), hosted by USIT at the UiO. The “Activity,” “List of Medication,” and “Diary of Good Things” modules are stored on the user’s phone for easy access, while aggregated data from all modules are encrypted and transferred to a secure server (TSD) at UiO once a module is completed. These data generate a report that can be accessed by clinicians during consultations and provide structured data available for researchers.
Ethical Considerations
The study was approved by the Data Protection Officer at OUH (approval numbers 2016/14408 and 20/18434). The regional ethical committee found that the development and testing fell outside their mandate (reference number 105851). All clinicians and service users who were interviewed signed informed consent forms. Service users who were engaged during the early phases of development, in collaboration with the service designers, also signed informed consent for their involvement in this work. All data were handled securely at TSD and by a secure research server at OUH. Service users were compensated with approximately US $23 per hour. All data used in this paper have been anonymized.
Results
The Development Process
Designing iTandem for Clinical Integration and Shared Decision-Making
We aimed to make the use of iTandem as true to a clinical setting as possible, keeping both the service user’s and clinicians’ experiences in mind when framing the app content. Our primary goal was to develop a tool to enhance shared decision-making. Besides supporting collaboration, we believe this is crucial for compliance. iTandem is not a standalone tool, and both its usage and the reasons for any attrition should be explored during treatment. We used phrases that closely mirror those used by clinicians. Furthermore, we chose to incorporate only a minimum of validated self-report measures, as the rationale behind the app was to serve as a communicative tool rather than a strict research assessment instrument.
The described needs were in line with current research, and based on the current literature, the rationale behind each module is outlined below (
Table 1).
| Module | Frequency of notifications | Aim | Contents | Basis of measure/intervention |
| Sleep |
|
|
|
|
| Medication |
|
|
|
|
| Recovery |
|
|
|
|
| Mood |
|
|
|
|
| Psychosis |
|
|
|
|
| Activity |
|
|
|
|
| Substance use |
|
|
|
|
| Feedback on treatment |
|
|
|
|
Sleep
Sleep disturbances are common in psychotic and bipolar disorders []. In bipolar disorder, sleep disturbances are frequently observed during and between episodes and are associated with more severe symptoms []. Therefore, being able to detect and monitor disturbed sleep patterns and implement treatments that focus on improving sleep may be crucial. Both service users and clinicians considered it valuable to have a tool that collects daily data on sleep to avoid recall bias. Additionally, clinicians expressed a need for a tool to assist in diagnosing sleep disorders. The Sleep module was based on the consensus sleep diary, which enables the monitoring of subjective sleep measures and the diagnosing of common sleep disorders []. The diary consists of a questionnaire in the morning about how well you slept and a questionnaire in the evening where users rate how their day was. The intended purpose of the Sleep module is to provide an overview of service users’ sleep patterns and, based on this, initiate relevant sleep treatment if necessary. Additionally, the module includes a link to a website [] with information about common sleep problems in severe mental disorders and sleep hygiene advice.
Medication
Both service users and clinicians wanted a tool that could assist with reminders (notifications) for taking medication and monitoring medication side effects. Medication is a key component in the treatment of both psychotic and bipolar disorders. Antipsychotics are effective, especially during the early and acute phase of psychosis [], and are also commonly used in people with bipolar disorder [], along with mood stabilizers []. However, these medications may have side effects that are not always properly addressed []. Recent studies have highlighted the importance of considering the user’s experience level, autonomy processes, values and risk preferences, knowledge, and needs to accurately understand the service users’ experience with medication []. The module was designed to enable service users to create a (revisable) list of their medications, set reminders for doses, and report side effects. Previous research has demonstrated that the language used to describe side effects can influence users’ expectations and may contribute to experiencing nocebo-induced side effects []. Webster and colleagues [] found this to be particularly significant for people with negative beliefs about side effects. To mitigate this, we used UX design to integrate the patient-rated UKU Side Effect Scale [], which consists of 48 single symptom items, into the Medication module. Instead of requiring the service user to rate 48 individual items, we chose to present the scale as a visualization of the human body. Service users can click on the body part where they experience side effects and select the relevant side effect(s).
Recovery
Clinicians were primarily interested in how treatment impacted clinical recovery in terms of symptom remission and adequate functioning. However, they also emphasized the importance of considering the impact on quality of life. Service users, by contrast, highlighted the need to focus more on their positive outcomes in line with personal recovery. This is commonly defined as the subjective experience of connectedness, hope, identity, meaning in life, and empowerment []. Personal recovery has received increased attention in mental health research in recent years, particularly in relation to psychotic disorders [,]. This highlights the importance of monitoring treatment outcomes in terms of well-being, personal recovery, and clinical recovery. Positive psychology has been applied to the treatment of mental disorders and, more recently, to psychotic disorders through positive psychotherapy for psychosis []. The Recovery module has 2 parts: First, it monitors personal recovery using a validated assessment measure [], focusing on good feelings and thoughts with a validated measure of mental well-being (Warrick-Edinburgh Mental Well-Being Scale) [], and life satisfaction with The Life Satisfaction Scale []. Second, the module includes 2 positive psychology exercises, namely, (1) Good Things, which is a positive diary of good experiences. The diary allows for daily notes to be stored on the phone, which can be revisited by the service user. It is based on positive psychology, suggesting that gratitude interventions can positively affect well-being, happiness, and life satisfaction []. (2) Personal Strength Exercise, which emphasizes the need for strength-based interventions for people with severe mental disorders []. The service user is presented with 24 strengths visualized by icons. They are encouraged to select 4 of their personal strengths and plan activities that utilize these strengths. This approach is adapted from the Positive Psychotherapy for Psychosis group intervention developed by Slade et al [], in accordance with the WELLFOCUS study [].
Mood
Detecting fluctuations in mood symptoms, such as elevated and depressed moods, is crucial in the treatment and relapse prevention of bipolar disorder, as emphasized by the service users. However, mood symptoms are not limited to bipolar disorder; they also play a significant role in other mental disorders, as highlighted by all the service users and clinicians who were interviewed [,]. Monitoring mood and affective symptoms is also relevant for personal recovery []. The service users wanted the opportunity to log affective states or emotions as they emerge in daily life, with the option of linking such experiences to external triggers. Service users with bipolar disorder may benefit from using daily mood charts to monitor their symptoms []. This feature was incorporated into the Mood module based on a paper version adapted from the Systematic Treatment Enhancement Program for Bipolar Disorder []. Additionally, we developed a module to monitor daily emotions, potential triggers, and an evaluation of fluctuating or stable emotions.
Psychosis
The clinicians were interested in measures to detect how psychotic symptoms fluctuate and also the ability to detect triggers. This functionality was less important to the service users. This might seem paradoxical but aligns with the research literature on recovery []. However, most service users agreed that it could be relevant to log their symptoms to register the effects of medication, although they argued that the questions needed to be brief and relevant. As we defined iTandem as a tool aimed at facilitating dialogue in treatment sessions and not for research assessment, we chose to include questions about psychosis as typically phrased by clinicians in clinical practice. Thus, the Psychosis module includes a list of positive psychosis symptoms to choose from and brief follow-up questions regarding how symptoms affect daily function, triggers, etc. This may serve as a means to monitor treatment progression regarding positive psychotic symptoms. Additionally, the clinicians wanted to monitor negative symptoms. Negative symptoms are challenging to assess due to their overlapping features with, for example, depression and medication side effects []. However, the Self-Evaluation of Negative Symptoms Scale has been shown to be a valid self-report scale, and we chose to incorporate this into the Psychosis module [].
Activity
Increasing goal-directed activity is important in the recovery process for severe mental disorders. However, achieving this in treatment can be challenging. The service users and clinicians wanted functionality that could be incorporated as part of treatment and motivate for change. This aligns with previous research, which states that goal planning is an important element in mental health service delivery, but we lack standardized approaches to support it []. Furthermore, both working alliance and goal achievement support the individual recovery process, and a strong working alliance seems to make people more capable of reaching their goals []. Potentially, digital tools could supplement and enhance this alliance. This module aims to support motivation for activity by allowing the service user to set long- and short-term goals. When an activity that supports the goal has been executed, the service user selects an icon representing this goal. Each completed activity creates a streak.
Substance Use
Substance use and abuse are of major concern in both psychotic [] and bipolar disorders []. In addition, people with substance use disorder have a higher prevalence of mental disorders than the general population []. However, even though both service users and clinicians agree that this is a major problem, it became apparent that the service users did not want to focus on detailed intake of substances. They stated that thorough monitoring of their weekly substance use did not motivate change and argued for brief questioning on drug intake. However, they found it useful to be able to convey which strategies they had used to abstain from taking drugs during the week. Clinicians were more prone to ask for detailed accounts of substance use to get an overview but were also concerned that this could lead to demotivation and disengagement. In addition, we were guided by principles from motivational interviewing and aimed to engage the service user in strategies to abstain from using drugs when feeling a craving []. Hence, we chose the same strategy as with the Psychosis module and included only brief questions about use and abstaining strategies, along with a link to a recommended user-friendly web page about substance use.
Treatment
The final module of iTandem was included to adhere to the Norwegian guidelines for psychosis treatment, which underline the need for better tools to evaluate the experience of treatment. PROMs and patient-reported experience measures (PREMs) are widely used []. PROMs are primarily used for the assessment of treatment outcomes, while PREMs are used to evaluate patients’ experiences with the service they receive []. They allow for more direct feedback to the provider. In a tool designed for therapeutic collaboration and alliance building, the service users’ experience and ability to provide feedback on the sessions were considered important for iTandem. Feedback on treatment is also a fundamental principle in cognitive behavioral therapy, which is one of the most frequently used treatment approaches in mental health care. Therefore, including this module would also support cognitive behavioral therapy. The module consisted of questions about how the last week had been, and questions about the individual treatment sessions, to be answered before and after each session (
Figure 1).
Report
Both clinicians and service users wanted a report to be able to log patterns. However, due to data security issues, only a limited amount of data could be stored on the phone. Except for the positive diary, activity, and medication list, data from all the modules were presented in the report. The positive diary was excluded from the report to allow service users to record positive events without feeling compelled to share them with clinicians. The Activity module incorporated elements of gamification, such as weekly streaks, which were believed to be a sufficient motivator. The other data entered were transferred to TSD. A report was available for clinicians, with the purpose of being shared with service users during sessions. Design principles were applied to make the report as easy to read as possible. Patients were not able to log into the report themselves. This decision was influenced by both technical and regulatory challenges, as well as how iTandem was intended to function as a tool for collaboration and shared decision-making. By integrating the report reading into the sessions, we believe it better supports the main goal (
Figure 2).
Beta Testing
Overview
Six clinicians (2 psychologists and 4 psychiatrists) were invited to participate in beta testing and attended a 3-hour training session on how to use iTandem. During this session, we demonstrated the different modules and their intended purposes. In addition, we provided an online protocol where participating clinicians and service users could access written information about the content and how to use the app. In collaboration with clinicians, service users were encouraged to choose 2-3 modules relevant to their treatment. It was not emphasized that this was set as an upper limit, but both clinicians and designers were concerned about overwhelming the patients with too many notifications. This could also undermine the overall goal of helping to focus sessions on issues considered particularly relevant to the patient. The patients used the app for 6 weeks (
Figure 3).
After beta testing, 4 service users and 5 clinicians agreed to be interviewed about their experiences with iTandem. This served as a preliminary proof of concept to determine whether the concept appeared to have potential. We also wanted feedback to adjust both iTandem and the educational implementation program before proceeding with a larger test in a clinical setting. Several issues were addressed, and the responses were sorted based on what appeared to be the most relevant attitudes and sentiments following the test.
Feedback on Concept by Service Users
The service users were mainly positive toward iTandem and the possibilities it offered. They appreciated the ability to select different modules that captured issues beyond the symptoms of their primary diagnosis, working alongside their clinician. This approach made it easier for them to focus on subjects and issues that were important to them and to convey this effectively to their clinician, thus supporting collaboration on goal setting and shared decision-making. They had concerns about how the report was used during sessions and emphasized the importance of clinicians utilizing the logged data; otherwise, the aspect of collaboration would be undermined. Some service users felt that the questions in the app were a bit repetitive (mainly the validated forms), and called for more opportunities to use free text to convey important information to their clinicians. It should be noted that there were service users who signed the informed consent to participate but did not follow through. According to clinicians, this was related to either paranoid ideas, high general symptom load, or being transferred to other departments.
Feedback on the Concept by Clinicians
Clinicians found the data from iTandem in the report to be useful; it provided important insights for their therapeutic work. Several were surprised by how the focus of treatment shifted when they received new information through the app, aligning with the goal of shared decision-making and increased collaboration. However, they found it was not always easy to remember to access and read the report, and they called for the report to be made accessible in the regular hospital records. By contrast, when this was not an option, they found the final solution to be manageable.
Implementation Issues
Many service users were eager to try several modules at once. However, notifications from multiple modules were perceived as overwhelming, highlighting the importance of working with 2-3 modules at the same time. The time spent on each module was deemed appropriate. Clinicians also felt overwhelmed by the challenge of integrating a digital tool into treatment sessions while finding space for it in their clinical routine. They found it stressful not to be fully familiar with the tool, making it difficult for them to effectively explain its use to the service user.
Technical Issues
Several technical issues arose regarding both the data sent from the app to TSD and the presentation in the report. This was partly due to errors in the app, but also because the report functionality was newly developed by the web app group at USIT and had not been properly tested beforehand. This led to several periods of downtime for the report. Even though this was seen as a major problem that persisted for several weeks, participants continued to enter data and found it meaningful. Furthermore, there were issues regarding the user-friendliness of the clock in the Sleep module. It was too easy to enter the wrong date, which obviously resulted in incorrect sleep patterns in the report. We also discovered missing data in the report.
Revision After Beta Testing and Plans for Pilot Testing
Based on the beta testing, we managed to resolve the technical issues. We revised the information material and teaching manual, omitting text and focusing on video films to show and tell how the app is intended to be used. In addition, we only recommend using a limited number of modules at a time to avoid overwhelming notifications. The new information material, manual, and the revised iTandem have since been further tested in a pilot trial conducted during winter 2023. In this trial, we tested iTandem as part of an early intervention service in a rural area in Norway. Data from this trial are currently being analyzed and will be part of a PhD thesis on how digitalization impacts service delivery in mental health care.
Furthermore, several studies are planned, incorporating iTandem as part of studies that use single modules and investigate implementation research connected to digitalization in mental health care.
Discussion
We have developed and beta tested a digital tool, iTandem, based on principles from design thinking. The main aim has been to utilize technology to strengthen and focus shared decision-making between service users and clinicians in mental health care and to enhance goal setting. The demand for simple functionality was underlined by both service users and clinicians.
We believe the design process was helpful in designing a tool that accommodates shared decision-making. We found the tool to follow many of the principles recommended by Priebe and colleagues [], such as a heightened focus on the service users’ concerns, positive regard and personal respect, appropriate involvement of service users in decision-making, genuineness with personal touch, and the use of a psychological treatment model []. By allowing service users to choose modules to aid focus in treatment, we aim to empower them during their recovery process.
What is meaningful for the user should be the priority in treatment. This does not mean that clinical evaluation is less important, but to build an alliance and secure a therapeutic relationship, the service users’ interests must remain in focus throughout therapy. This aligns with the demand for more patient-centered care. It is also in line with the shift from “patient-centered” to “person-centered” care, as discussed by Boardman and Dave []. There is a change in how we view the therapeutic relationship, where the therapeutic alliance is more about co-production as a key element in the clinician-service user collaboration.
Furthermore, we believe that the ability to choose themes to work on and the possibility to give feedback on treatment sessions, including questions about being met with respect, will underpin service users’ experience of being respected and treated in a nonjudgmental way. We hope this tool will enhance clinicians’ motivation to make changes in accordance with the service users’ experienced needs. By using design both as a creative process and applied to the visual elements, we have strived to make the tool engaging and less “clinical.” This may increase the feeling of a more personal touch. To accommodate a more generic therapeutic use, the elements of iTandem are well suited to fit cognitive behavioral therapy, focusing on agenda setting, goal setting, and the execution of activities. Unsurprisingly, a design process led to solutions that align well with this therapy approach, as it is collaborative, pragmatic, structured, and focused on current problems.
We have previously worked with Boundary Objects Theory, where using a digital object can be seen as a way to transfer knowledge between different knowledge domains []. Based on this work, for an object to function effectively, it needs to inherit features such as dynamism, flexibility, standardization, and a shared structure []. We believe iTandem has the potential to work as a boundary object. It accommodates the needs of the service user, and it is simple to engage in, hence adhering to the KISS principle. iTandem allows for the transfer of knowledge between service user and clinician with an inherent flexibility yet standardized approach. The ability to collaborate via this object allows for a deeper understanding of the problem, leading to more focused problem-solving.
iTandem differs from the numerous assessment tools offered as PROMs in Norwegian mental health care services at the moment. The positive aspects of PROMs lie in their basis on validated measures that aim to track progress in therapy. However, the extensive number of questions to be answered can be overwhelming, and the amount of data may be challenging for clinicians to manage and oversee effectively. PROMs often cover broad categories, and their questions can sometimes seem less relevant to the service users’ specific problems. Therefore, it may be necessary to scale down the assessments to focus on a minimum that allows for qualified evaluations of how treatment progresses. This approach can alleviate the burden on clinicians and service users by combining different tools, such as apps like iTandem and more focused classic PROMs. Utilizing language and questions that are closely aligned with a clinical setting might be sufficient to establish a rapport and gather relevant information about progress.
Another important aspect is how we provide flexibility in services. While we offer options such as video consultations, PROMs, apps, and similar tools, the solutions presented to service users often rely on traditional approaches, as if one size fits all. Alternating between different tools and formats during treatment sessions could, in our view, make the treatment more effective. Insights from user reviews of mental health apps indicate that highly rated apps tend to offer a variety of options, functionalities, and content for users to choose from []. This is particularly relevant to attrition rates, as different tools may be needed during various phases of the recovery process. A lack of use could even be a positive sign, indicating that the problem has been resolved and the user has moved on. Additionally, Alqahtani et al [] identified several common weaknesses of mental health apps, including poor usability, lack of personalization, credibility issues, security concerns, and inadequate customer service. We believe these findings align with the need for greater flexibility in services.
To address the future needs of health care services, we believe it is essential to examine how digital tools can be integrated into regular workflows. Greater insight is needed into the barriers and facilitators of implementation, as well as how health technology assessment methods can be used to evaluate which technologies benefit service users, clinicians, and organizations while also being economically effective.
Acknowledgments
The development of iTandem was funded by the South-Eastern Norway Regional Health Authority (reference number 17/00264-14). We want to thank all the service users and clinicians who willingly have offered their time and interest in this development process. We also want to thank the agile team at USIT, which is not represented as co-authors in this paper, for good collaboration. We especially thank the senior developer, Magnus Alderslyst Nygaard, and the highly skilled, creative designers. Finally, we will thank Jonathan Romm for the illustrations and Trine Vik Lagerberg for valuable support along the way.
Data Availability
Because of the small sample size and limitations in the written informed consent, the data are not available to other researchers.
Conflicts of Interest
None declared.
References
- Brédart A, Bouleuc C, Dolbeault S. Doctor-patient communication and satisfaction with care in oncology. Curr Opin Oncol. Jul 2005;17(4):351-354. [CrossRef] [Medline]
- Arora NK. Interacting with cancer patients: the significance of physicians' communication behavior. Soc Sci Med. Sep 2003;57(5):791-806. [CrossRef] [Medline]
- Lee SJ, Back AL, Block SD, Stewart SK. Enhancing physician-patient communication. Hematology Am Soc Hematol Educ Program. 2002:464-483. [CrossRef] [Medline]
- Aftab A, LaGrotta C, Zyzanski SJ, Mishra P, Mehdi SMA, Brown K, et al. Impact of psychiatric hospitalization on trust, disclosure and working alliance with the outpatient psychiatric provider: a pilot survey study. Cureus. Apr 22, 2019;11(4):e4515. [FREE Full text] [CrossRef] [Medline]
- Rübig LL, Fuchshuber J, Köldorfer P, Rinner A, Fink A, Unterrainer H. Attachment and therapeutic alliance in substance use disorders: initial findings for treatment in the therapeutic community. Front Psychiatry. 2021;12:730876. [FREE Full text] [CrossRef] [Medline]
- Reddy MS. Insight and psychosis. Indian J Psychol Med. 2015;37(3):257-260. [FREE Full text] [CrossRef] [Medline]
- Harvey PD, Bosia M, Cavallaro R, Howes OD, Kahn RS, Leucht S, et al. Cognitive dysfunction in schizophrenia: an expert group paper on the current state of the art. Schizophr Res Cogn. Sep 2022;29:100249. [FREE Full text] [CrossRef] [Medline]
- Lyngstad SH, Gardsjord ES, Engen MJ, Haatveit B, Ihler HM, Wedervang-Resell K, et al. Trajectory and early predictors of apathy development in first-episode psychosis and healthy controls: a 10-year follow-up study. Eur Arch Psychiatry Clin Neurosci. Sep 2020;270(6):709-722. [FREE Full text] [CrossRef] [Medline]
- Ihler HM, Lyngstad SH, Mørch-Johnsen LE, Lagerberg TV, Melle I, Romm KL. A transdiagnostic approach to negative symptoms: exploring factor structure and negative symptoms in bipolar disorders. Front Psychiatry. 2023;14:1136097. [FREE Full text] [CrossRef] [Medline]
- Priebe S, Dimic S, Wildgrube C, Jankovic J, Cushing A, McCabe R. Good communication in psychiatry--a conceptual review. Eur Psychiatry. Oct 2011;26(7):403-407. [CrossRef] [Medline]
- Feldthusen C, Forsgren E, Wallström S, Andersson V, Löfqvist N, Sawatzky R, et al. Centredness in health care: a systematic overview of reviews. Health Expect. Jun 2022;25(3):885-901. [FREE Full text] [CrossRef] [Medline]
- Brown K, Parry S. How do people with first episode psychosis experience therapeutic relationships with mental health practitioners? A narrative review. Psychosis. Dec 22, 2022;16(2):155-166. [CrossRef]
- Hamann J, Heres S. Adapting shared decision making for individuals with severe mental illness. Psychiatr Serv. Dec 01, 2014;65(12):1483-1486. [CrossRef] [Medline]
- Simmons M, Hetrick S, Jorm A. Shared decision-making: benefits, barriers and current opportunities for application. Australas Psychiatry. Oct 2010;18(5):394-397. [CrossRef] [Medline]
- Zisman-Ilani Y, Barnett E, Harik J, Pavlo A, O’Connell M. Expanding the concept of shared decision making for mental health: systematic search and scoping review of interventions. MHRJ. Sep 11, 2017;22(3):191-213. [CrossRef]
- Zisman-Ilani Y, Lysaker PH, Hasson-Ohayon I. Shared risk taking: shared decision making in serious mental illness. Psychiatr Serv. Apr 01, 2021;72(4):461-463. [CrossRef] [Medline]
- Castonguay LG, Constantino MJ, Holtforth MG. The working alliance: Where are we and where should we go? Psychotherapy (Chic). 2006;43(3):271-279. [CrossRef] [Medline]
- Weldring T, Smith SMS. Patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs). Health Serv Insights. 2013;6:61-68. [FREE Full text] [CrossRef] [Medline]
- Nguyen H, Butow P, Dhillon H, Sundaresan P. A review of the barriers to using patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs) in routine cancer care. J Med Radiat Sci. Jun 2021;68(2):186-195. [FREE Full text] [CrossRef] [Medline]
- Nwosu A, Boardman S, Husain MM, Doraiswamy PM. Digital therapeutics for mental health: is attrition the Achilles heel? Front Psychiatry. 2022;13:900615. [FREE Full text] [CrossRef] [Medline]
- Szinay D, Jones A, Chadborn T, Brown J, Naughton F. Influences on the uptake of and engagement with health and well-being smartphone apps: systematic review. J Med Internet Res. May 29, 2020;22(5):e17572. [FREE Full text] [CrossRef] [Medline]
- Mordecai D, Histon T, Neuwirth E, Heisler WS, Kraft A, Bang Y, et al. How Kaiser permanente created a mental health and wellness digital ecosystem. NEJM Catalyst. Jan 2021;2(1):1. [FREE Full text] [CrossRef]
- Dalzell T. The Routledge Dictionary of Modern American Slang and Unconventional English. London, UK. Routledge; Jul 25, 2008:595.
- Alwin DF, Beattie BA. The KISS principle in survey design. Sociological Methodology. Jun 21, 2016;46(1):121-152. [CrossRef]
- Rich BR. Clarence Leonard (Kelly) Johnson 1910-1990: A Biographical Memoir. Washington, DC. The National Academies Press; 1995:221-241.
- Vial S, Boudhraâ S, Dumont M. Human-centered design approaches in digital mental health interventions: exploratory mapping review. JMIR Ment Health. Jun 07, 2022;9(6):e35591. [FREE Full text] [CrossRef] [Medline]
- Halskov K, Hansen N. The diversity of participatory design research practice at PDC 2002–2012. International Journal of Human-Computer Studies. Feb 2015;74:81-92. [FREE Full text] [CrossRef]
- Bergin AD, Vallejos EP, Davies EB, Daley D, Ford T, Harold G, et al. Preventive digital mental health interventions for children and young people: a review of the design and reporting of research. NPJ Digit Med. 2020;3:133. [FREE Full text] [CrossRef] [Medline]
- Romm KL, Gardsjord ES, Gjermundsen K, Aguirre Ulloa M, Berentzen L, Melle I. Designing easy access to care for first-episode psychosis in complex organizations. Early Interv Psychiatry. Oct 27, 2019;13(5):1276-1282. [CrossRef] [Medline]
- Roberts JP, Fisher TR, Trowbridge MJ, Bent C. A design thinking framework for healthcare management and innovation. Healthc (Amst). Mar 2016;4(1):11-14. [CrossRef] [Medline]
- Altman M, Huang TT, Breland JY. Design thinking in health care. Prev Chronic Dis. Sep 27, 2018;15:E117. [FREE Full text] [CrossRef] [Medline]
- The Design Council. The Double Diamond. The Design Council. URL: https://www.designcouncil.org.uk/our-resources/the-double-diamond/ [accessed 2024-12-13]
- Laskemoen JF, Simonsen C, Büchmann C, Barrett EA, Bjella T, Lagerberg TV, et al. Sleep disturbances in schizophrenia spectrum and bipolar disorders - a transdiagnostic perspective. Compr Psychiatry. May 2019;91:6-12. [FREE Full text] [CrossRef] [Medline]
- Sylvia LG, Chang WC, Kamali M, Tohen M, Kinrys G, Deckersbach T, et al. Sleep disturbance may impact treatment outcome in bipolar disorder: a preliminary investigation in the context of a large comparative effectiveness trial. J Affect Disord. Jan 01, 2018;225:563-568. [CrossRef] [Medline]
- Carney CE, Buysse DJ, Ancoli-Israel S, Edinger JD, Krystal AD, Lichstein KL, et al. The consensus sleep diary: standardizing prospective sleep self-monitoring. Sleep. Feb 01, 2012;35(2):287-302. [FREE Full text] [CrossRef] [Medline]
- www.psykose-bipolar.no. Psykose/Bipolar. Oslo, Norway. Oslo University Hospital URL: https://www.psykose-bipolar.no/bipolar-lidelse/tidlige-tegn-pa-bipolar-lidelse/ [accessed 2024-12-17]
- Zhu Y, Li C, Huhn M, Rothe P, Krause M, Bighelli I, et al. How well do patients with a first episode of schizophrenia respond to antipsychotics: a systematic review and meta-analysis. Eur Neuropsychopharmacol. Sep 2017;27(9):835-844. [CrossRef] [Medline]
- Keramatian K, Chakrabarty T, Saraf G, Yatham LN. New developments in the use of atypical antipsychotics in the treatment of bipolar disorder: a systematic review of recent randomized controlled trials. Curr Psychiatry Rep. May 08, 2021;23(7):39. [CrossRef] [Medline]
- Kishi T, Ikuta T, Matsuda Y, Sakuma K, Okuya M, Mishima K, et al. Mood stabilizers and/or antipsychotics for bipolar disorder in the maintenance phase: a systematic review and network meta-analysis of randomized controlled trials. Mol Psychiatry. Aug 2021;26(8):4146-4157. [FREE Full text] [CrossRef] [Medline]
- Haddad PM, Correll CU. The acute efficacy of antipsychotics in schizophrenia: a review of recent meta-analyses. Ther Adv Psychopharmacol. Nov 2018;8(11):303-318. [FREE Full text] [CrossRef] [Medline]
- Bjornestad J, Lavik KO, Davidson L, Hjeltnes A, Moltu C, Veseth M. Antipsychotic treatment - a systematic literature review and meta-analysis of qualitative studies. J Ment Health. Oct 2020;29(5):513-523. [CrossRef] [Medline]
- Webster RK, Weinman J, Rubin GJ. How does the side-effect information in patient information leaflets influence peoples' side-effect expectations? A cross-sectional national survey of 18- to 65-year-olds in England. Health Expect. Dec 2017;20(6):1411-1420. [FREE Full text] [CrossRef] [Medline]
- Lindström E, Lewander T, Malm U, Malt UF, Lublin H, Ahlfors UG. Patient-rated versus clinician-rated side effects of drug treatment in schizophrenia. Clinical validation of a self-rating version of the UKU Side Effect Rating Scale (UKU-SERS-Pat). Nord J Psychiatry. 2001;55 Suppl 44:5-69. [CrossRef] [Medline]
- Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Br J Psychiatry. Dec 2011;199(6):445-452. [CrossRef] [Medline]
- Slade M, Amering M, Farkas M, Hamilton B, O'Hagan M, Panther G, et al. Uses and abuses of recovery: implementing recovery-oriented practices in mental health systems. World Psychiatry. Feb 2014;13(1):12-20. [FREE Full text] [CrossRef] [Medline]
- Åsbø G, Ueland T, Haatveit B, Bjella T, Flaaten CB, Wold KF, et al. The time is ripe for a consensus definition of clinical recovery in first-episode psychosis: suggestions based on a 10-year follow-up study. Schizophr Bull. Jun 21, 2022;48(4):839-849. [CrossRef] [Medline]
- Slade M, Brownell T, Rashid T, Schrank B. Positive Psychotherapy for Psychosis: A Clinician's Guide and Manual. London, UK. Routledge; Dec 19, 2016.
- Neil ST, Kilbride M, Pitt L, Nothard S, Welford M, Sellwood W, et al. The questionnaire about the process of recovery (QPR): a measurement tool developed in collaboration with service users. Psychosis. Aug 2009;1(2):145-155. [CrossRef]
- Tennant R, Hiller L, Fishwick R, Platt S, Joseph S, Weich S, et al. The Warwick-Edinburgh Mental Well-Being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes. Nov 27, 2007;5:63. [FREE Full text] [CrossRef] [Medline]
- Lehman A. A quality of life interview for the chronically mentally ill. Evaluation and Program Planning. Jul 03, 1988;11(1):51-62. [FREE Full text] [CrossRef]
- Dickens LR. Using gratitude to promote positive change: a series of meta-analyses investigating the effectiveness of gratitude interventions. Basic and Applied Social Psychology. May 30, 2017;39(4):193-208. [CrossRef]
- Tse S, Tsoi EW, Hamilton B, O'Hagan M, Shepherd G, Slade M, et al. Uses of strength-based interventions for people with serious mental illness: a critical review. Int J Soc Psychiatry. May 2016;62(3):281-291. [CrossRef] [Medline]
- Riches S, Schrank B, Rashid T, Slade M. WELLFOCUS PPT: modifying positive psychotherapy for psychosis. Psychotherapy (Chic). Mar 2016;53(1):68-77. [CrossRef] [Medline]
- Upthegrove R, Marwaha S, Birchwood M. Depression and schizophrenia: cause, consequence, or trans-diagnostic issue? Schizophr Bull. Mar 01, 2017;43(2):240-244. [FREE Full text] [CrossRef] [Medline]
- Alsheikh AM, Elemam MO, El-Bahnasawi M. Treatment of depression with alcohol and substance dependence: a systematic review. Cureus. Oct 26, 2020;12(10):e11168. [FREE Full text] [CrossRef] [Medline]
- Skar-Fröding R, Clausen H, Šaltytė Benth J, Ruud T, Slade M, S Heiervang K. Associations between personal recovery and service user-rated versus clinician-rated clinical recovery, a cross-sectional study. BMC Psychiatry. Jan 18, 2022;22(1):42. [FREE Full text] [CrossRef] [Medline]
- Yasui-Furukori N, Nakamura K. Bipolar disorder recurrence prevention using self-monitoring daily mood charts: case reports from a 5 year period. NDT. Mar 2017;Volume 13:733-736. [CrossRef]
- Sachs GS, Guille C, McMurrich SL. A clinical monitoring form for mood disorders. Bipolar Disord. Oct 2002;4(5):323-327. [CrossRef] [Medline]
- Slade M. Implementing shared decision making in routine mental health care. World Psychiatry. Jun 2017;16(2):146-153. [FREE Full text] [CrossRef] [Medline]
- Galderisi S, Mucci A, Dollfus S, Nordentoft M, Falkai P, Kaiser S, et al. EPA guidance on assessment of negative symptoms in schizophrenia. Eur Psychiatry. Feb 18, 2021;64(1):e23. [FREE Full text] [CrossRef] [Medline]
- Dollfus S, Mach C, Morello R. Self-evaluation of negative symptoms: a novel tool to assess negative symptoms. Schizophr Bull. May 2016;42(3):571-578. [FREE Full text] [CrossRef] [Medline]
- Stewart V, McMillan SS, Hu J, Ng R, El-Den S, O'Reilly C, et al. Goal planning in mental health service delivery: a systematic integrative review. Front Psychiatry. 2022;13:1057915. [FREE Full text] [CrossRef] [Medline]
- Rose G, Smith L. Mental health recovery, goal setting and working alliance in an Australian community-managed organisation. Health Psychol Open. 2018;5(1):2055102918774674. [FREE Full text] [CrossRef] [Medline]
- Hartz SM, Pato CN, Medeiros H, Cavazos-Rehg P, Sobell JL, Knowles JA, et al. Genomic Psychiatry Cohort Consortium. Comorbidity of severe psychotic disorders with measures of substance use. JAMA Psychiatry. Mar 2014;71(3):248-254. [FREE Full text] [CrossRef] [Medline]
- Hunt GE, Malhi GS, Cleary M, Lai HMX, Sitharthan T. Prevalence of comorbid bipolar and substance use disorders in clinical settings, 1990-2015: systematic review and meta-analysis. J Affect Disord. Dec 2016;206:331-349. [CrossRef] [Medline]
- Richert T, Anderberg M, Dahlberg M. Mental health problems among young people in substance abuse treatment in Sweden. Subst Abuse Treat Prev Policy. Jun 24, 2020;15(1):43. [FREE Full text] [CrossRef] [Medline]
- Galvani S. Motivational Interviewing: Helping People Change (3rd edition). Social Work Education. Mar 04, 2014;33(6):854-855. [CrossRef]
- Friedel AL, Siegel S, Kirstein CF, Gerigk M, Bingel U, Diehl A, et al. Measuring patient experience and patient satisfaction-how are we doing it and why does it matter? A comparison of European and U.S. American approaches. Healthcare (Basel). Mar 08, 2023;11(6):797. [FREE Full text] [CrossRef] [Medline]
- Kingsley C, Patel S. Patient-reported outcome measures and patient-reported experience measures. BJA Education. Apr 2017;17(4):137-144. [CrossRef]
- Boardman J, Dave S. Person-centred care and psychiatry: some key perspectives. BJPsych Int. Aug 2020;17(3):65-68. [FREE Full text] [CrossRef] [Medline]
- Saidi T, Mork E, Aminoff S, Lobban F, Romm KL. Crossing boundaries in the delivery of healthcare - a qualitative study of an eHealth intervention in relation to boundary object theory. Digit Health. 2023;9:20552076231196970. [FREE Full text] [CrossRef] [Medline]
- Star SL, Griesemer JR. Institutional Ecology, `Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Soc Stud Sci. Aug 01, 1989;19(3):387-420. [CrossRef]
- Alqahtani F, Orji R. Insights from user reviews to improve mental health apps. Health Informatics J. Sep 2020;26(3):2042-2066. [FREE Full text] [CrossRef] [Medline]
Abbreviations
| KISS: keep it simple, stupid |
| OUH: Oslo University Hospital |
| PREM: patient-reported experience measure |
| PROM: patient-reported outcome measure |
| TIPS SE: Early Intervention in Psychosis Advisory Unit for South East Norway |
| TSD: Services for Sensitive Data Tjenester for Sensitive Data |
| UiO: University of Oslo |
| USIT: University of Oslo’s Center for Information Technology |
| UX: user experience |
Edited by A Mavragani; submitted 22.02.24; peer-reviewed by A Oostdyk, A Kurilla; comments to author 14.10.24; revised version received 28.10.24; accepted 06.11.24; published 17.01.25.
Copyright©Kristin Lie Romm, Mari Skoge, Elizabeth Ann Barrett, Lars-Christian Berentzen, Dagfinn Bergsager, Pål Fugelli, Thomas Bjella, Erlend Strand Gardsjord, Kristine Kling, Sindre Hembre Kruse, Kari Jorunn Kværner, Ingrid Melle, Erlend Mork, Henrik Myhre Ihler, Eline Borger Rognli, Carmen Simonsen, Tor Gunnar Værnes, Sofie Ragnhild Aminoff. Originally published in JMIR Formative Research (https://formative.jmir.org), 17.01.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.