Using a mixed-methods approach, the feasibility of the mDiary app was assessed with qualitative interviews in 9 focus groups, as well as evaluations through a questionnaire measuring perceived usability. A total of 5 focus groups were dedicated to therapists only, whereas 4 were dedicated to patients only. Interviews were conducted on-site. Data from the interviews were recorded on an MP3 recorder during the interviews and transcribed verbatim afterwards.

Patients and therapists participating in the focus group interviews were concurrently given the System Usability Scale (SUS) [23] to evaluate system usability. SUS is widely used and is a valid and reliable assessment tool for usability of digital interventions. The total SUS scores are ranked from 0 to 100, where 100 represents the highest usability and a score of 68 reflects an average level of usability [24].

The quantitative data were used to formulate a theory of barriers and facilitators. The grounded theory (GT) approaches [25] of open, axial, and selective coding as well as theoretical sampling, ongoing development, and internally relating of concepts were used in the analysis.

The trustworthiness of the findings, in the sense that a credible and true picture of the phenomenon under scrutiny was presented [26], was addressed by basing conclusions on the verbatim transcripts of the focus groups [26], by discussing the derived concepts among the participating researchers, and by applying a form of triangulation of data by interviewing both therapists and patients and doing this separately, as well as using the SUS scores in the initial formation of a theory of barriers and facilitators. Trustworthiness was increased further by making comparisons of the findings with other broader theories of implementation of technology in the field.

All participating patients were enrolled in active DBT treatment in Danish public outpatient psychiatric care from January 2016 to December 2016. Before entering the study, all patients were assessed by a psychiatrist with the International statistical classification of diseases and related health problems. - 10th revision, Fifth edition, 2016 diagnostic manual.

. Patients were eligible for inclusion if they met the criteria for emotionally unstable personality disorder (F60.3), were admitted for psychiatric outpatient treatment, had at least 1 suicide attempt or at least 1 episode of self-harm within the last year, and active problems with suicidal and self-harm urges. Patients were excluded if they had no access to or ability to use a mobile phone or had a comorbid disorder, such as substance abuse, bipolar disorder, or a schizophrenia spectrum disorder. The majority of patients had comorbid disorders, such as depression, anxiety, posttraumatic stress disorder, substance abuse, and obsessive-compulsive Disorder. Alcohol and substance abuse were allowed if it was not the primary diagnosis. After admittance to DBT treatment, BPD symptoms and diagnoses were rechecked again in a separate individual session with an experienced therapist.

The patients’ mean age was 28.0 years (SD 6.2). Half of the sample had completed primary school or less and the other half had secondary education or more. All patients were enrolled in a 12-month DBT program [27] but were at different stages in their treatment program: 10 patients were in the beginning (months 1-3) of their treatment, whereas 6 patients had attended treatment between 4 and 12 months. All study subjects had previously tried paper registration and had switched to app registration for at least 4 weeks.

A total of 23 DBT therapists participated in the study. Half of the therapists were psychiatrists or psychologists and the other half were nurses, occupational therapists, or psychotherapists.

They had DBT therapy experience in the range of 1 to 14 years (mean 6.8 years). Their mean age was 44 years (SD 12). The therapists had different levels of experience with using the mDiary app solution, ranging from 1 month to 1 year.

The mDiary app was customized for dialectical behavior therapy treating borderline personality disorder. This platform consists of 2 modes of data handling: mobile phone-based and Web-based. Patient data were entered by the patient on a mobile app.

The study followed a GT approach [28]. The duration of all focus group interviews was 1 hour. During the interviews, a whiteboard was divided into a matrix of 2 rows by 3 columns: 2 horizontal rows held pros and cons; 3 vertical columns held the themes paper, app, and future scenarios. The participants were given sticky notes and encouraged to place them on the whiteboard and discuss the reason(s) for placing the input as either a pro or a con. The discussion was moderated by the principal investigator (SHJ) and 1 of the cowriters (TS). Themes from earlier interviews were offered to the participants when statements differed from what was discussed in the focus group (theoretical sampling). The qualitative analysis of the focus groups was performed in different steps as discussed further.

Patients signed an informed consent form before study participation. The study was approved by the Danish Ethics Committee in Region of Southern Denmark (S-20160085). The study protocol was approved by the Danish Data protection Agency (2008-58-0035). The approved database system used was Odense Patient Data Exploratory Network (OPEN) [29].

The SUS ratings showed that overall, patients were very satisfied with the solution (Table 2). Therapist SUS ratings were generally significantly lower compared with those of the patients (P<.001). Older users tended to rate lower usability than younger users (Pearson correlation coefficient: −0.60). At Site 1, the primary development site, therapists were significantly less satisfied than at the later development sites (P=.01), see Table 2 and Figure 3.

As shown in Table 2 and Figure 3, the first development site reported significantly lower usability than the rest of the sites, with scores falling below an SUS score of 60.

The therapists with the most influence on the development were the most critical of the solution. This is probably because they spent the most time dealing with bugs and problems. Our therapist SUS scores suggest that usability improved with every iteration of the development cycle, as more recent adopters had higher SUS scores and were generally more positive in the interviews than early adopters.

Implementation seen from the outside or from the inside is a matter of perspective. We define inside-out innovation as the instability introduced into the treatment context owing to introductions of new work tasks accompanying development and implementation. For instance: learning to use the app, adaptations needed in the context, getting the solution to work technically, and fitting the solution to the needs of the most important work tasks.

This can be seen as a complementary process to the outside-in innovation, which means new technology brought to the hospital from an external source. In the mDiary study, the outside-in innovation was the technical solution delivered from Monsenso. They delivered a starting point, a solution that had a general applicability, but were influenced by an outsider’s perspective regarding the specific mental health context it was implemented in.

A central barrier in the development process was found to be the inside-out innovation part.

A suggestion for reconfiguration would typically come from the involved users’ perspective. Figure 4 shows what is seen as valuable from either an outside-in perspective or an inside-out perspective. The smaller squares in the figure show the concrete behavioral actions associated with the 2 perspectives.

When comparing in-clinic innovation with simply implementing a solution already developed and finalized, the effort needed is very different: Reporting hassles and giving suggestions for improvement back to the app developers, upgrading to new versions, reporting technically succinct accounts of bugs back to the app developers (eg, “The bug was on which version of the operating system? What type of phones had the problem? Under precisely what conditions?”), and maybe the clinician encounters trouble accessing data or meet unexpected needs for technological upgrades. All this is time consuming and requires focus and energy, leading to multiple small reductions in time for other tasks. Even if the mDiary study was a time-limited endeavor, this led to frustrations among the clinicians involved in the development. The same demands of productivity during the development phase was expected as within normal operation of the clinic. Mental health workers view delivery of psychosocial treatment as their primary goal. Without clear alignment of expectations and extra resource allocation, this generated frustrations:

In the end, this came down to differences in values: In Figure 4, different views on finding faults in the mDiary app can be traced. The app developers’ perspective finds exploration of faults in the system as a very worthwhile endeavor in accordance with the AGILE project management approach that finds value in fail early, fail fast, fail forward [30], whereas therapists typically adhere to a value of minimizing faults at any cost owing to increased patient risk factors [31]. From one perspective, value might mean ability to fail often and fast thereby creating stability and profitability on the long term, whereas from another point of view, value might mean treating a specific patient, here and now, effectively without the delay of reporting bugs back.

The concept of balancing acceptance and change is borrowed from individual DBT psychotherapy [27]. We found this concept helpful to describe barriers and facilitators in the inside-out process. It seems that finding a balance between the status quo and changing old routines is essential and a culprit of many frustrations encountered during implementation.

When improvements were made, new procedures needed to be invented in the clinic, and uncertainty was added to known procedures. It was found that the introduction and development of new technology was a balancing act between acceptance and change. It could in other words be described as a search for the optimal level of frustration of the users. The central task was generating the largest amount of long-term positive change possible while keeping time use optimally focused on delivering therapy to the patients here and now. We have condensed the main acceptance-change dilemmas into 4 main sources of hassle, as shown in Figure 5.

The first 2 dilemmas were related to human factors.

User level of technical skills: The first dilemma was out-of-the-box intuitive usability versus training needed before the app was usable to the users. Note that here the term user relates to both staff and patients:

Ecological fit: The second dilemma was high versus low need for reorganizing known procedures and daily habits. The slight modification of therapy rules and procedures demanded new procedures at the organizational level:

This could imply new difficult-to-solve problems involving both developers and other layers of the hospital administration.

Another aspect of ecological fit could be seen from a patient perspective:

The final two dilemmas were related to technological issues.

Level of finalization: The third dilemma was the paradox between having a stable technological product, that is, where the coding is consolidated, versus a more flexible solution where coding is a work in progress. Users want both specific tailored solutions suited to the context and at the same time they want the system to be stable. During the development of a new system, it is difficult to have both:

Addressing the right need?: To what extent did the technological solution come across as relevant and sustainable here and now? To what extent does it need modification to effectively address the specific mental health problem the clinician is supposed to solve?:

Monitoring activity by GPS coordinates and counting time spent on talking on the phone were quite easy to do in the solution. Sensors were present in the mobile phones and the coding was already in place as it was used in the bipolar solution. It was not possible to switch this off, and since it was collected passively, it seemed like it did not take extra effort from the patients. The specific task of monitoring mood and skill use, however, did not require this, as it was not a therapeutic necessity. The BPD patients saw this kind of activity monitoring as an unnecessary invasion of their privacy, so even if it was easily accessible and possibly interesting—it was not enough a part of the central task at hand, and thus did not have enough direct relevance to patients. Monsenso, on the other hand, was quite reluctant to let go of this feature. Here, different needs seen from the outside-in and inside-out perspectives stood out clearly. The inside-out perspective seemed to favor utility here and now. The outside-in perspective tended to favor the long-term potential creating big data from the same variables across different diagnosis.

The acceptance and usability of the mDiary app was generally high among patients. The SUS scores showed sufficient acceptability among most of the test site therapists. But the primary development site had significantly lower acceptance of the solution. The therapists most involved in the development process were surprisingly the most critical. We have attempted to explain this with a hypothetical concept of inside-out innovation. We found that the most important dilemmas within inside-out implementation were related to user levels of technology skills, ecological fit, finalization of the app, and addressing the right need.

Resistance to implementing new technology in the health care setting is a well-known problem [32]. Since Davis’ seminal paper in 1989, the Technology Acceptance Models has focused on perceived usability and ease of use as central variables for the successful implementation of new technology [33,34]. This theory targets implementability by reducing complexity to the individual adopter’s viewpoint. The model has been replicated many times and specific data on health care found ease of use being less important than usability in health care settings [34-36]. Realizing that a more multifaceted approach was needed, Venkatesh et al [37] tested several variables, such as gender, age, experience with performance expectancy, and effort expectancy that could impact the user’s acceptability and satisfaction with technology. They also included the influence of attitude and social factors on behavioral intention to explain resistance more broadly using both background variables of users as well as attitudes. This extended model led to the formulation of the Unified theory of acceptance and utilization of technology [37]. In mDiary, usability scores followed this logic as it was found that user age influenced the SUS scores.

Van Gemert-Pijnen et al [38] have pointed out that innovation in mental health is a “collaboratory participatory process of constantly changing cycles.” Values in this cooperation are considered central. Values refer to what is considered meaningful in the context, not only economic value. Van Gemert-Pijnen described innovative change using 4 axes: business model, value drivers, user requirements, and prototyping. Value specification is considered important in obtaining desirable cooperation: Value specification implies the recognition and quantification of the economic, medical, social, or behavioral values of the key stakeholders [38]. Van Gemert-Pijnen’s approach can be thought of as a broader theory looking at implementation from a systems perspective, focusing on different or shared values in the systems’ interaction. Value specification was found to be important in our data. This is most clearly seen in the addressing the right need part of Figure 5 and the value sections in Figure 4.

The hassles captured in the concepts inside-out innovation and ecological fit have been described in a study by Heeks when he was exploring what he called the design-reality gap [32]. Perspectives differ whether you are a programmer or a user. This is an analog to the user level of technological skills dilemma found in Figure 5. The gap in technology between design and reality in Heeks’ theory can also be thought of as a case of poor ecological fit from our model.

Greenhalgh et al [39] have formulated a theory of “Nonadaptation, abandonment, scale-up, spread, and sustainability of new technology (NASSS).” The values described in Figure 4 fit very well as a theoretical clarification of Greenhalgh’s description of different types of value propositions. One value is described by her as “knowledge needed to use the technology.” She suggests that it is helpful to range this from simple to complex needs for knowledge. This is a similar concept to our user level of technological skills described in Figure 5. A concept similar to level of finalization in the same figure is also addressed by Greenhalgh, who found dependability to be a key value [39]. The dilemma of addressing the right need is covered in her theory as differences in value propositions.

This phenomenon of differences in inside-in and outside-in perspectives has also been addressed by Van Gemert-Pijnen et al [38], who also view explication of values of stakeholders as central to successful implementation. The Figure 5 elements of what we call ecological fit and skills needed are here conceptualized as differences in specification of what has value to whom. The following quote from Van Gemert-Pijnen (p. 10) sums up what we have encountered in developing and testing the mDiary app:

This quote illustrates that the needs related to implementation depend on which perspective you take. The value of implementation is to some extent negotiable and different depending on which stakeholder perspective you take. Designing, implementation, and enhancing usability are all part of the same circular process. In the collaboration between Monsenso and public psychiatry wards, the needs of the patients, the needs of Monsenso, and the needs of therapists were all part of a continuous negotiation: a negotiation of whether to use the severely burdened patients’ time on the long-term implementation of an—as yet untested—system; a negotiation of company resource utilization when improving the technical side of an app with an—as yet unknown ability to generate provenue; and finally, a negotiation with the therapists in getting them to allow for new procedures instead of established procedures of standard DBT treatment. The innovation in the mDiary app was this balancing act, eventually creating an app to the mutual benefit of all involved parties.

The study should be interpreted with the following limitations in mind. The number of participants and focus groups was small albeit adequate for a feasibility study. The sample was not randomly selected, but a select sample where patients were given a choice whether they wanted to test the mDiary app as a potential replacement for using paper and pen. Owing to the research method and sample size, the results can only be considered hypothesis-generating. In terms of trustworthiness, it must be noted that group polarization and mutual avoidance of discomfort could influence the results. The patient group is known to avoid difficult emotions [27] as well as being unstable in their baseline emotions [40], which might influence answers toward a more emotional direction. A large part of the sample of therapists were older and very well consolidated in the DBT procedures, which could lead to negative bias regarding approaches aimed at changing well-known procedures [41]. The study also has several strengths, including the involvement of the end users (both patients and therapists) in the development of the platform/app and a thorough and iterative process to optimize the platform/app.

The mDiary app is a useful and acceptable way of registering DBT diary cards, tracking emotion regulation, and skill acquisition and is now ready for implementation. Our data suggest adequate usability and feasibility in clinical departments with higher perceived usability from patients compared with therapists. At the present stage, the app is sufficiently ready to be used in further studies evaluating effectiveness. This will be done in the mDiary RCT study.