This project was an iterative content creation and evaluation study conducted as part of an internal program development effort focused on producing therapeutic materials used in an mHealth app–based intervention for SUD treatment [24-26]. The purpose was to describe and evaluate a careful, step-by-step process for how existing clinical content could be adapted for smartphone delivery and whether the end product met measurable standards for understandability and actionability.

Licensed alcohol and drug counselors (LADCs) employed by Mayo Clinic within the Division of Addiction Medicine (Department of Psychiatry and Psychology) were recruited to participate. They worked with a doctoral-level therapeutic specialist, employed within the same department. The a priori recruitment goal was 4 LADCs, which was felt to be a reasonable number to balance group discussion with the doctoral-level therapeutic specialist.

Approximately 15 LADCs are employed within the division at any given time. Consistent with their field broadly, a significant number of these providers are themselves openly in recovery from a SUD [27]. This background provides lived experience and a point of shared connection with their patients, a perspective sought out for this study. Our division has a strong history of patient- and staff-oriented addiction research. As such, a group of approximately 8 LADCs exists who have actively participated in previous projects. Four of these counselors, known to be openly in recovery from a SUD, were purposively recruited by email. Participation was entirely voluntary. The LADCs could freely decline to participate, as they had done previously when they were not interested in a project. Declining to participate would have no impact on their job.

The first 4 LADCs approached all agreed to participate. These specific LADCs were asked due to their experience working in multiple levels of addiction treatment, including level of care assessments, individual therapy, group-based outpatient groups, and residential programming, and their previous involvement in research within the division. Collectively, the participating LADCs had decades of experience working with individuals with SUDs and co-occurring mental health diagnoses and were themselves openly in recovery from a SUD. Their typical work roles include providing individual and group-based SUD treatment, performing care coordination on their patient’s behalf, and helping complete patient-related paperwork. Module review occurred in blocked times during their standard work hours. The doctoral-level therapeutic specialist performed this role as part of their typical duties within the department.

Fourteen therapeutic modules were developed from existing Mayo Clinic text-based materials originally created by a board-certified addiction psychiatrist and a doctoral-level education specialist for use in clinical practice. These materials required reformatting for a smartphone interface and needed to be easy to read, visually clear, and to use everyday language.

The home institution considered this project to be quality improvement in nature and therefore determined it exempt from review by the Mayo Clinic Institutional Review Board. Staff participated voluntarily and were given dedicated time during their regular workday to complete the study tasks. No patients were involved, and no patient information was used.

This early report described the iterative development and evaluation of 14 therapeutic modules for an mHealth app–based SUD treatment platform and examined whether they met recommended PEMAT-P benchmarks (>70%) for understandability and actionability. On aggregate, the modules exceeded both understandability (87.2%) and actionability (75.1%) targets, supporting the use of an interdisciplinary and iterative approach. Despite this resource- and time-intensive process, several modules fell below the actionability benchmark, which may indicate that our methodology was more effective for understandability or that understandability itself is easier to achieve than actionability. Additionally, these findings highlight the challenges of achieving consistent quality even when a structured process is implemented and suggest that process improvement is needed to maximize understandability and actionability for the end user.

Descriptions of how existing therapeutic content is adapted for digital delivery remain limited, particularly in SUD interventions, which also lack systematic evaluations of understandability and actionability [23]. This gap exists for chronic health conditions more broadly, where digital interventions (n=44) have been found to include few actionable targets for behavior change and to fall below accepted thresholds for understandability and actionability (medians of 42% and 0%, respectively) [22].

Our modules were found to have better understandability than actionability. This distinction has also been seen in past literature. In bipolar disorder, a review of online resources (n=52) found that most scored poorly on the PEMAT-P, with mean understandability of 56.0% (SD 14.0%) and mean actionability of 44.0% (SD 24.0%) [29]. Similarly, an evaluation of psychiatric hospital brochures found a mean PEMAT-P score of 48.4% (SD 15.4%) and noted that nearly all materials were written at a grade 12 or higher reading level [30]. Reported deficiencies included insufficient emphasis on key information, unexplained medical terminology, limited guidance on recommended behaviors, and visuals that were either irrelevant or too complex to enhance understanding [30]. Finally, problematic PEMAT-P scores have also been observed outside of mental health treatment. A review of online materials for rotator cuff repair reported mean understandability and actionability scores of 64.6% and 29.5%, respectively, with no identified material surpassing the actionability threshold [31].

Across these studies, a common thread is the influence of health literacy on whether patients can actually use the material in a meaningful way. Defined as an individual’s ability to obtain, process, and understand basic health information needed to make appropriate health decisions, health literacy is a critical determinant of equitable health care delivery [32]. Deficits in health literacy are pervasive, contributing to poorer outcomes and higher health care use [32]. Therapeutic materials that fail to meet health literacy standards can widen existing disparities by limiting access to information, reducing patient engagement, and undermining informed decision-making. Our process involved a doctoral-level therapeutic specialist to maintain the material at a fifth-grade reading level. Ensuring that educational and therapeutic content are understandable and actionable is essential for supporting patient engagement and helping individuals achieve their treatment goals [28,32].

In this context, using the above-described strategy, our tested modules compare favorably with mean understandability and actionability scores exceeding those reported in prior studies. The SDs observed for modules were variable, and at times, quite large. Looking at Table 1 and Figure 1, it is apparent that module-level differences exist. Differences may have also occurred at the reviewer-level; however, this study was not designed to compare the relative contribution of scale item, rater, and module effects. Information on this is also missing from the existing literature [23]. Prior work with the PEMAT-P suggests that expanding the number of independent reviewers in future evaluations may provide more robust estimates of content quality [33].

A structured, interdisciplinary approach can raise the quality of digital therapeutic materials, but content quality alone does not fully explain how users experience or benefit from them. Importantly, our study assessed content quality rather than perceived patient benefit. High PEMAT-P scores indicate that users should be able to grasp and act on the material. Conversely, modules scoring lower may still be perceived as useful by some users for reasons that lie outside the PEMAT-P’s scope. This interpretive gap is not unique to our project but reflects a broader challenge in digital health evaluation [34]. Known as the digital placebo effect, this phenomenon refers to improvements in outcomes or perceived benefit that arise from assuming an intervention will be useful because the content is medical in nature [34]. For example, in a study of healthy participants using an inert mHealth app, being told prospectively or retrospectively that the intervention would improve mood and stress was associated with higher credibility and expectancy ratings compared with a control condition [35].

While the therapeutic value of mHealth apps depends primarily on the quality of their content and how they are delivered [7,14], the digital placebo effect suggests that users’ expectations can be shaped by the way an intervention is framed [34,36]. As such, future research should prioritize rigorous development and evaluation of both content and delivery strategies.

Several steps are essential to advancing mHealth app–based research that directly assesses the efficacy of therapeutic content (Figure 2). First, incorporating parallel expectancy scoring alongside PEMAT-P ratings may help identify gaps between content quality and perceived credibility. For example, counselors could provide a brief expectancy rating (eg, “How likely is this module to feel credible or motivating to patients?”). Second, introducing brief patient expectancy probes in patient-facing pilots, such as a single premodule item (eg, “How helpful do you expect this module will be?”), could quantify baseline expectations and help separate content effects from potential placebo effects in follow-up data. Third, incorporating automated follow-up action checks may provide an observable behavioral end point to compare with PEMAT-P’s actionability scores. For instance, modules that conclude with behavioral activation activities could be paired with an mHealth app–delivered push notification 1 hour later asking whether the assigned task was accomplished. Finally, iterative A/B feature testing could evaluate how variations in framing, design cues, or reminders impact user follow-through. Collectively, these steps may help provide a more accurate representation of the mHealth app’s effect.

This project has several limitations. The number of LADCs involved was small (n=4) and purposively recruited, which may have introduced selection bias and limited the range of perspectives represented. Additionally, this group of LADCs helped both create and review the material. This level of investment may have added bias, such as contributing to score inflation. A group of independent LADC evaluators may have added a layer of objectivity. While these counselors were in recovery from SUDs and had extensive knowledge and training in SUD treatment, their ratings may not reflect how patients with varying literacy levels, lived experiences, referral sources, or differing trajectories of addiction severity and treatment responsiveness may interpret the material [37,38]. Including participants with SUDs who were not health care professionals would have strengthened the human-centered design aspect of module evaluation. Additionally, the PEMAT-P captures important aspects of quality, but it does not measure perceived credibility, motivation, or expectancy, which may influence how users engage with therapeutic content. The modules were adapted from existing clinical materials rather than created de novo, which may have influenced how easily some sections could be reshaped for mobile delivery. While the original source material was from the Mayo Clinic, which has been found broadly to have a rating of good for the ability of its patient-facing information to assist health care consumers on treatment information quality and resource validity [39], we have no validated measures that assess whether the original source material was effective. Finally, as this study focused solely on content characteristics and not on patient outcomes, no conclusions can be drawn about clinical effectiveness.

This early report highlights the value of a human-centered, interdisciplinary process for adapting therapeutic materials for digital delivery within an mHealth app–based SUD treatment intervention. By combining clinical SUD expertise, lived experience in recovery, and health care communication knowledge through iterative drafting and review, the team produced content that, in aggregate, exceeded recommended PEMAT-P thresholds. Understandability was superior to actionability; at the module level, 35.7% (5/14) of modules fell below the acceptable threshold. This fits within the larger landscape of patient-facing materials, where actionability is often lacking. As digital SUD interventions continue to expand, leaving patients with clear actions that can be taken based on the content presented remains an important challenge. These findings also suggest that a robust adaptation process is not enough to guarantee strong content. Including active patients in this content design and iteration process may be beneficial to this end. Evaluating material at the unit level (eg, modules and chapters) instead of broadly as one corpus, may help ensure isolated weaknesses are not missed.