This study used a mixed methods design to evaluate whether our sham app could serve as a believable control condition in a randomized controlled trial as well as to evaluate participant perspectives of usability and satisfaction with the app.

This fully web-based study was approved by the University of Texas Health Science Center at San Antonio Institutional Review Board (protocol 20230232HU). Participants provided written informed consent prior to participation. During the consent process that was conducted remotely, participants were informed that the purpose of the study was to evaluate engagement, usability, and user satisfaction with a mobile app that offered information about OUD and other substance use disorders. Participants received US $200 for their participation in the form of a ClinCard MasterCard. Compensation was automatically credited after completion of the study. All data were deidentified prior to being analyzed.

Participants were all diagnosed with OUD, currently receiving MOUD, and in stable outpatient treatment for at least 4 weeks at the time of enrollment. They were recruited from outpatient MOUD treatment facilities in Texas. Recruitment strategies included placing flyers in clinic waiting rooms, announcements during web-based trainings sponsored by our staff, sending emails about the study to our network of MOUD providers, and outreach personnel speaking with people in line at methadone clinics. Individuals interested in participating in the study were directed to contact study staff via phone. Inclusion criteria for the study included (1) male or female outpatients 18 years of age or older; (2) diagnosis of OUD; (3) currently stable in OUD outpatient treatment, defined as medication adherence and attending outpatient appointments for 4 weeks or longer; and (4) ability to access the KIOS sham app via a smartphone or tablet. Participants were excluded if they (1) were unwilling or unable to comply with study requirements, (2) had a psychiatric or medical disorder interfering with the ability to use the app, (3) were incarcerated, or (4) were pregnant.

The Patient Health Questionnaire-9 (PHQ-9) is a 9-item validated self-report assessment that was used to measure depression severity over the past 2 weeks [18]. The World Health Organization Quality of Life-BREF (WHOQOL-BREF) assessment was used to assess self-reported quality of life across 4 domains: physical health, psychological health, social relationships, and environment over the past 2 weeks [19]. A User Satisfaction Survey was administered to assess the degree of satisfaction with the sham app [20]. The WHOQOL-BREF and PHQ-9 were chosen as outcomes because they assess complementary facets of recovery (eg, depression has a high comorbidity with alcohol and drug use) [21-23]. Improvements in depression and quality of life are some potential benefits that may arise when using an app-based intervention; thus, treatment targets for the app are not limited to opioid use but rather to all aspects of recovery that may impact opioid use and retention in MOUD treatment.

As this was a web-based study, communication between research assistants and participants occurred via telephone and Zoom (Zoom Video Communications). No changes were made to the app after the trial began.

This was a 1-sample study with no control group using a pre- and posttest design. Prior to engaging with the sham app, participants completed the PHQ-9 and WHOQOL-BREF delivered via an electronic data capture system. Surveys were completed while on the phone with research assistants in the event that participants had any questions. They were then given access to the app for 4 weeks, during which time they were instructed to engage with the app at a minimum of 3 times per week and as often as once daily. Participants were encouraged to explore all the functions within the app. At the end of the 4 weeks, they completed the PHQ-9 and WHOQOL-BREF as well as the User Satisfaction Survey.

Upon completion of the study, semistructured interviews were conducted to gather detailed information on believability, usability, and satisfaction with the sham app. Six questions were asked assessing app believability and satisfaction (eg, “Tell us what you thought of KIOS?” and “If you weren’t in the study anymore, would you use the app? Why or why not?”). Following these questions, a debriefing statement was delivered informing participants of the sham nature of the app. Once informed, participants were asked to reshare their perception of the sham app and indicate whether they suspected it was anything other than what it was presented as.

The sham app was developed for use in a randomized controlled trial testing the KIOS app, and it was created at the request of the FDA. The proposed design of the sham app was presented to the FDA for review prior to proceeding with development.

The sham app was meticulously designed to convince the user that it was an active intervention despite being designed to have no therapeutic value. To create a believable sham app, the active KIOS design was replicated to create a visually similar version that lacked key therapeutic components. The nonspecific factor of the sham app mirrored the active condition by replicating the layout and functionality of account log-in, color palette, button shapes, emergency contact functions, and landing pages. Features that incorporated elements of an active intervention were removed from the sham and were replaced with general drug and alcohol educational content taken from the National Institute on Drug Abuse (NIDA).

We identified, selected, and analyzed information about drugs using specific criteria. Drug-related facts were explored through key questions addressing the drug’s definition, use, short- and long-term effects, physical and chemical properties, and health impacts. Questions using a true or false format were included in the app, and they were designed to generate generic feedback that was presented to participants. These questions were developed by selecting topics from the NIDA website, identifying relevant passages, and rephrasing or editing them for clarity and conciseness. Correct “true” answers were based on accurate passages, while “false” answers included explanations derived from the content available from NIDA. The process also involved assessing the grade level of each question to ensure readability at or below a sixth-grade reading level, excluding drug names, and documenting both the grade level and source for each question. Participants were presented with the true or false questions 3 times a week followed by questions assessing their level of interest in the educational content being provided. Because education alone is unlikely to change long-term behavior [24], the final product was a fully functional, education-based learning tool that had no active components capable of creating meaningful improvements in clinical outcomes.

The second fundamental difference between the active and sham versions of KIOS is the reminder push notification. The push notification frequency was customizable to the user’s preference, ranging from daily, every other day, to not at all. The active KIOS app that was developed is a tailored, patient-centered app, thus necessitating a similar approach to the reminder notifications. Participants using the active version receive personalized reminders (eg, “Savor this one moment with a slow, deep breath. Then, connect with yourself further in KIOS”). In contrast, the sham app was designed to be standardized across users, including the reminders that used generic language (eg, “This is the KIOS reminder you set for yourself”). This difference is essential because the personalized reminders are designed to simulate the therapeutic intent of an active intervention while generic reminders do not.

Quantitative analyses included paired 2-tailed t tests to assess differences in PHQ-9 and WHOQOL-BREF scores after using the sham app. To determine overall satisfaction with the app, a total percentage of the aggregate responses of the User Satisfaction Survey was calculated.

Rapid qualitative analysis was used to systematically review the semistructured interviews that were conducted following completion of the study. Transcripts were thoroughly reviewed to identify recurring themes. Each question was separated with participant responses to the question being condensed into bullet points. After grouping the questions into overarching domains (eg, use patterns and engagement and perceptions of authenticity), recurrent themes were cataloged within the domains to capture larger research focus areas. The final summary template that was used to aggregate participant responses was standardized across participants to ensure consistency.

A matrix was then developed to display the condensed summaries. The columns were organized by domain with subsections for each question in the domain. Each row in the matrix was dedicated to a participant interview. The matrix was filled using the data from the summaries and analyzed across rows and down columns to identify central themes, allowing for a comprehensive understanding of key findings from the interviews. The qualitative analysis was completed manually, and the quantitative analyses were completed in Microsoft Excel.

Our findings indicated no significant change in self-reported depression severity or quality of life after using the sham app, and all but 1 participant believed that the app was an active intervention. Together, these findings meet our a priori criteria that the sham app appears to be a suitable control condition for a randomized trial.

Prescription digital therapeutics should be held to the highest standard of research. This makes blinded research designs crucial for determining true efficacy before prescribing these apps to patients. Because a randomized controlled trial is the optimal method for testing the evidence base of a treatment, a sham app should be expected. The creation of a sham app for testing a prescription digital therapeutic is a relatively new and modernizing advancement in research designs [25]. Recognized by the FDA as a uniquely difficult undertaking [17], this novel approach addresses the need for a more scientifically robust methodology in the study of prescription digital therapeutics.

With an appealing user interface, the design and execution of the sham software were pivotal in presenting the app to be nearly indistinguishable from an active intervention. The sham app was reported to be engaging, yet nontherapeutic. The latter signifies the possibility to surmount the potential benefits inherent in engaging with an interactive digital platform. The identical user interface, testing prior to using it in a randomized controlled trial, and positive engagement results we demonstrated here overcome previously cited limitations in the study and use of sham apps [25]. This sham app holds significant potential for OUD treatment and recovery, paving the way for creating studies to identify effective prescription digital therapeutics. Without it, the gap between OUD and a good prognosis will remain.

The success of the sham app extends beyond the immediate study, and it represents an opportunity to elevate the standards of study designs for all digital health research. Using the framework we have demonstrated here can further the development of future sham apps. We have shown some components of a sham app that make it believable without producing clinical improvements (eg, generic push reminders and using education in place of the active intervention). These components can be modified and adapted for future studies testing app interventions, making our framework appropriate for creating a sham app that will be suitable as a control condition. This innovation holds significant implications not only for KIOS but also for the broader research field, advancing the science, testing, and deployment of prescription digital therapeutics.

There are limitations that should be mentioned. This study included a small sample size, and the study duration was short. Because our initial pilot study demonstrated the feasibility and acceptability of the KIOS app and this study was designed to create a comparison, sham app for future testing of the KIOS app, we used a pre- and posttest design with no control group [9]. Participants were not required to see a medical provider to be included, and we do not have information regarding their opioid use or adherence to their medication regimen. There may have been variations in interview techniques among researchers. It is possible that a sham app may have some therapeutic benefits among those early in recovery. Finally, all participants were in Texas, making additional research in other geographic regions necessary. Despite these caveats, the insight gained here is invaluable.

With the rapid integration of technology in health care, there has been a call for more methodologically rigorous studies. Developing this sham app marks a historic accomplishment in the emerging field of prescription digital therapeutics. This advancement will drive high-quality research, providing the foundation for a new placebo that will revolutionize the future of clinical trials.