Published on in Vol 7 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37270, first published .
Assessment and Disruption of Ruminative Episodes to Enhance Mobile Cognitive Behavioral Therapy Just-in-Time Adaptive Interventions in Clinical Depression: Pilot Randomized Controlled Trial

Assessment and Disruption of Ruminative Episodes to Enhance Mobile Cognitive Behavioral Therapy Just-in-Time Adaptive Interventions in Clinical Depression: Pilot Randomized Controlled Trial

Assessment and Disruption of Ruminative Episodes to Enhance Mobile Cognitive Behavioral Therapy Just-in-Time Adaptive Interventions in Clinical Depression: Pilot Randomized Controlled Trial

Authors of this article:

Liyuan Wang 1 Author Orcid Image ;   Lynn Miller 2 Author Orcid Image

Original Paper

1Children's Hospital, Los Angeles, Department of Adolescent and Young Adult, Los Angeles, CA, United States

2University of Southern California, Annenberg School for Communication and Journalism, Los Angeles, CA, United States

Corresponding Author:

Liyuan Wang, PhD

Children's Hospital, Los Angeles

Department of Adolescent and Young Adult

3250 Wilshire Blvd

Suite 1202

Los Angeles, CA, 90027-0000

United States

Phone: 1 7654911022

Email: liwang@chla.usc.edu


Background: A just-in-time adaptive intervention (JITAI) is “designed to address the dynamically changing needs of individuals via the provision of the type or amount of support needed, at the right time when needed.” If and how rumination-focused cognitive behavioral therapy (RFCBT), the gold standard, blocks emotional cascades underlying rumination is unclear. Furthermore, cognitive behavioral therapy has been successfully used as a mobile variant, but RFCBT has not been adapted for a mobile variant (mobile RFCBT [MRFCBT]) or for a JITAI variant.

Objective: This study aimed to pilot-test a fully automated JITAI leveraging RFCBT and ways to identify and block cascading depressive rumination.

Methods: Patients in therapy for clinical depression were recruited for a randomized controlled trial (RCT). After consenting to be part of the RCT, they were randomly assigned to either of the 2 mobile versions of the RFCBT conditions personalized to the individual’s rumination timing patterns (JITAI-MRFCBT) or a no-treatment control condition through a double-blind procedure. Although the initial design was to have a 3-armed trial with 2 JITAI conditions (a JITAI and a narrative JITAI condition), we later opted to collapse those 2 conditions into 1 JITAI condition because of the low number of participants. All participants were recruited and participated through their smartphones, receiving 5 SMS text message reminders on each of the 35 days to self-report their rumination-related symptoms (eg, rumination episodes and duration). In the JITAI-MRFCBT condition, they also received treatment materials. The first 7 days provided a rumination baseline, and the last 7 days provided a postintervention rumination value. In total, 42% (25/59) of volunteers were eligible and provided their phone numbers, 20% (5/25) of whom never replied to the SMS text message reminding them to start the RCT. A total of 90% (18/20) of volunteers completed it (ie, finishing, as prespecified, 80% of the questionnaires and training tasks) and, therefore, were included in the analysis.

Results: Using independent 2-tailed t tests with bootstrapping, results showed that participants in the JITAI-MRFCBT condition, compared with those in the control condition, reported a greater reduction in counts of rumination episodes (mean −25.28, SD 14.50 vs mean 1.44, SD 4.12, P<.001) and greater reduced average time (minutes) spent in rumination (mean −21.53, SD 17.6 vs mean 1.47, SD 1.5; P=.04). Results also suggest that, compared with those in the control group, those in treatment reduced ruminative carryover from one episode to the next.

Conclusions: The results suggest that JITAI-MRFCBT may reduce negative rumination by providing RFCBT just in time following rumination, thereby blocking the next rumination episode using the same trigger. This study supports a subsequent, full-scale JITAI and the importance of leveraging mobile smartphone technology with MRFCBT to curb depressive symptoms.

Trial Registration: ClinicalTrials.gov NCT04554706; https://clinicaltrials.gov/ct2/show/NCT04554706

JMIR Form Res 2023;7():e37270

doi:10.2196/37270

Keywords



Background

Major depressive disorder (MDD) [1], or clinical depression, is a mood disorder associated with prevalent sadness, hopelessness, and anhedonia and often manifests symptoms associated with cognitive impairment and social functioning disability. A challenge underlying MDD is dysregulation of the emotional cascades underlying depressive rumination. Rumination, or depressive rumination response, is a thinking style defined as “the tendency to passively and repetitively focus on the experience of negative moods, as well as their causes and consequences” [2]. When rumination is driven by cascades of negative emotions, one can become more depressed and use more depressive rumination to cope with their depression [3]. Specifically, depressive rumination and the underlying emotional cascades usually stem from problematic interpersonal communication [3,4]. Negative verbal and nonverbal communicative patterns of depressed individuals can adversely affect the nature of social interactions and strain social networks [5,6]. Social impairments such as inhibited communication with family members and friends and the use of negative self-focused verbalizations have long been associated with negative psychological states, including depression [7]. In interpersonal relationships, depression-related patterns of rumination usually precede a subsequent problematic interpersonal communication episode, exacerbating levels of depression [7].

Cognitive behavioral therapy (CBT) has long been one of the chief promising face-to-face interventions in this domain [8]. Patients in CBT interventions learn ways to monitor and improve their psychological symptoms through a series of tasks such as learning how to use diaries, emotion regulation techniques, and mindfulness tools [9]. CBT offers a wide range of strategies that help patients maintain, trigger, and mitigate their depressive symptoms [9,10]. A number of meta-analytical reviews have shown that CBT has strong efficacy in mitigating depression outcomes and recommend it as the gold standard frontline treatment for depression [10-13].

Just-in-Time Adaptive Interventions Using Rumination-Focused CBT to Reduce Depression

A number of meta-analytical reviews have suggested that people’s everyday technologies (eg, smartphones and wearable devices) can provide not only scalable but efficacious mental health treatment [14-16]. Specifically, those reviews point to the efficacy of CBT delivered via mobile phone (mobile CBT [MCBT]) through apps. MCBTs have the capacity to integrate “gold standard treatment (CBT)” to bolster and maintain psychological resilience with scalable mobile alternatives [14].

However, MCBTs are not without their limitations. Although MCBT apps can provide some treatment information, none of those apps currently have the functionality to further diagnose and provide customized in-the-moment feedback based on a given client’s specific symptoms [5]. This is due at least in part to a failure to typically use a customized within-person approach affording intensive measurements (ie, ≥100 measures per variable per person), which can be leveraged to track a given participant’s progress within a given context and across different situational contexts [17,18]. Therefore, there is a need for rumination-focused MCBTs (or mobile rumination-focused CBTs [MRFCBTs]) that are more specifically designed to address rumination via a mobile device and effectively use a within-person approach for personalized feedback to changes in client state.

There are many reasons why an intensive within-subject MRFCBT using a just-in-time adaptive intervention (JITAI) approach—affording cues for personalized delivery of in-the-moment diagnosis and feedback—matters. First, a given client with depression does not respond with rumination across all contexts. Therefore, MCBTs without intensive within-subject assessment offered by JITAIs could be perceived as nonresponsive to the changing state of the client with depression, which could undermine treatment effectiveness [19-22]. Second, an intensive within-person design could allow for the collection of multimodal information using noninvasive sensors and active self-reports. These rich data can enable computational possibilities to predict and prevent risks when participants are the most vulnerable. For example, ecological momentary assessments (EMAs), which actively measure people’s in-the-moment emotional reactivity to different stressful events at various time points [23-25], have also been shown to be reliable predictors of subsequent depressive outcomes. Third, it is unclear how rumination in one context for an individual with depression carries over into other contexts, exacerbating those effects as rumination within subjects is not even assessed—let alone used for potential markers for personalized feedback or interventions. Thus, current MCBT apps function more like a pool of knowledge for active information seekers rather than professional health care providers from which one can seek personalized and responsive help. Fourth, MRFCBT—with rumination treatment in mind [26]—has not been successfully tested, to our knowledge, in mobile form, let alone with JITAI features.

Clearly, there is a strong need for current MRFCBT apps to integrate JITAI features (ie, algorithms), taking advantage of multi-time assessments (both active and passive) to detect vulnerabilities and provide feedback at the exact moment of need [27]. To advance this science, this project used EMA, where such moment-to-moment assessments in real time have been successfully applied in numerous health science domains [28]. That work suggests that JITAI designs could be translated into MCBTs, including an MRFCBT. JITAIs tested by researchers without “real-time” human clinicians—that have given participants feedback based on their self-reported problems—have been found to be effective [5]. Therefore, this project was designed to test the efficacy of an MRFCBT with JITAI features in reducing ruminative thoughts—a major symptom of depression. To date, this has not been assessed. Specifically, we hypothesized (hypothesis 1) that a JITAI-delivered MRFCBT intervention would be more effective than a no-treatment control group (assessing rumination but not providing treatment) in reducing rumination over time.

The Link Between Rumination and Depression

JITAI design can target highly intrapersonal symptoms that are otherwise hard to track. In particular, one’s ruminative responses (ie, trait rumination) to their immediate social interactions can contribute to one’s ruminative patterns and depression outcomes [29-31]. People with strong ruminative tendencies may find themselves subconsciously revisiting and recurrently thinking about some unpleasant interpersonal events. For example, after experiencing an interpersonal disagreement, those who are high in ruminative response may find themselves more likely to engage in thoughts concerning “Why did this happen to me,” “What was that person thinking of me,” and “Why couldn’t I handle this better.” Such ruminative processes usually lead individuals to recurrently revisit negative experiences, exacerbating negative affective states without producing strategies to improve future communication patterns [32].

Theories of depression [30,31,33] from both the cognitive and interpersonal or stress perspectives [4,34] have found that rumination over unpleasant experiences is a major reason for the increase in interpersonal conflict and the lack of relationship satisfaction, which subsequently leads to elevated depressive outcomes. In various studies on depression, the aftermath of depressive rumination is usually documented from an intrapersonal perspective. For example, using interviews, Lyubomirsky et al [29] found that, when engaging in more ruminative thoughts, people are more likely to generate biased interpretations of unpleasant events [29], and this usually elicits negative autobiographical memories [29], which exacerbate and prolong depression [35-37]. Without intensive assessments of one’s daily thought patterns, it would be impossible to track this highly intrapersonal process in terms of when and how it occurs, not to mention providing responsive treatment feedback for those who are in great need. With JITAIs and formative research on baseline rumination per participant, the occurrence of a ruminative episode could more easily be targeted via intensive measurements at multiple intervals.


Intervention System

The intervention system was created for a pilot study that, if successful, might lead to a subsequent full-scale randomized controlled clinical trial (RCT) nationally (ClinicalTrials.gov identifier NCT04554706). As a pilot RCT that precedes a large-scale trial, this project has some differences in sample size and measurement described in the preregistered trial. To be specific, this pilot is a 35-day–long, small-scale RCT with 2 between-subject conditions: a JITAI-MRFCBT condition (with personalized timing for participants’ own rumination patterns) and a no-treatment control condition with data collection to assess participant rumination but with no treatment. As the goal of this project is to focus on a major symptom of depression, which is excessive negative rumination, we adapted materials from rumination-focused CBT (RFCBT; materials on RFCBT were obtained through subscription to Psychology Tools [38]. The efficacy of RFCBT has already been tested in previous research [26]. MRFCBT includes activities, training exercises, and diaries that are designed to reduce ruminative episodes in patients with clinical depression [26]. Therapists can take advantage of the diaries provided by patients to offer tailored behavioral training (eg, reframing emotions, attribution perceptions, goal setting, and problem-solving) for patients. The RFCBT program is ideal for adaptation to MRFCBT with JITAI features: (1) the diary and rating systems with which therapists deliver tailored training sessions could be conveniently designed as questionnaires for participants to self-rate and document their conditions (providing also a 1-week baseline for personalized delivery, an adaptive feedback system sending treatment messages as participants progress into treatment, and a 1-week postintervention assessment), (2) the JITAI system can have the capacity to deliver automatic training sessions tailored to the results of individual self-reports, and (3) participants can have access to training materials and messages at the exact time of their requests. Therefore, 3 training material sets (ie, problem-solving, conflict attribution, and emotion regulation) from RFCBT were adapted into this JITAI condition. The JITAI conditions asked participants to document their ruminative episodes for 7 days. After that, they received intervention materials and training prompts for 3 weeks (21 days). They were asked to document their ruminative episodes for another week after completing the 3-week training materials (to compare the baseline experience with the one immediately after the intervention). It is worth pointing out that, as originally planned for the full registered RCT, we designed 2 of the JITAI conditions, one using the original adapted MRFCBT materials directly and the other taking a narrative approach to delivery of the MRFCBT. However, as we could not secure enough participants for the 2 JITAI conditions and there were no apparent differences between them in the pattern of findings (n=4 in the JITAI-MRFCBT condition and n=5 in the JITAI narrative RFCBT condition), we combined the 2 conditions as the JITAI condition for this pilot study. The no-treatment control condition did not have any training materials from the MRFCBT. Participants in this condition were only asked to document ruminative episodes (5 times a day) for 35 days.

Figure 1 is an illustration of the design of the JITAI system in this intervention. We used EMA as the main assessment tool to make decisions regarding which messages to deliver and when. EMA involved asking participants if they had experienced stressful events in the past 3 hours and, if so, what the events were (dotted double-headed arrows). If the participant identified the immediate presence of triggers of rumination episodes, the system prompted an inquiry regarding the nature of that trigger. At such trigger points, participants were quite vulnerable in terms of the probability of experiencing rumination episode events (dotted single-headed arrows). In addition, EMA measures, along with the previous baseline daily activity surveys, can determine whether participants are receptive to treatment at the moment of the rumination event (dotted single-headed arrows). If participants were not engaged in other activities (eg, driving and walking), they received JITAI feedback tailored to the type of trigger in the form of support, necessary problem-solving skills, motivation, and supportive messages (solid lines in Figure 1). In the meantime, we provided several sessions of RFCBT, adapted to (1) the screen size of a mobile phone and (2) readable length, when participants were receptive (dashed single-headed arrow).

Figure 1. Just-in-time adaptive intervention (JITAI) design process.
View this figure

Participants signed up and consented to take part before they were shown a link to the intervention website that asked participants to enter their phone numbers to start the RCT. This procedure was enabled by a programmable (through R; R Foundation for Statistical Computing) survey platform, Formr [39]. Using this platform, participants’ responses at each time were documented with a unique, anonymous identifier that linked to their phone number. The Formr platform was programmed to send reminder SMS text messages 5 times a day. The SMS text messages contained the link for them to document their symptoms. This pilot intervention did not have cointerventions.

The design of this RCT followed the necessary procedures that are required for RFCBT. That is, to complete the RCT, participants were asked to self-report and document their ruminative episodes every 3 hours (4-5 times a day). Each time, if a participant documented their recent ruminative episode in the past 3 hours, they were prompted to document the trigger, immediate environments, duration, and emotional experiences related to the current ruminative episode. The intervention system was designed to be tailored to participants’ ruminative patterns in three ways: (1) based on participants’ ruminative episodes (triggers and immediate environments), participants would receive a set of training materials that best targeted the trigger of their ruminative episode (eg, problem-solving and interpersonal conflicts); (2) participants could choose to review their training materials at any time they wished as a means to provide in-the-moment feedback; and (3) participants could skip and withdraw from answering some of their documented prompts at a time when they were otherwise engaged in another activity. Therefore, each participant could have up to 175 times within the 35 days of the intervention to report on their ruminative episodes.

To enhance our understanding of rumination and potential interventions to reduce it, participants were encouraged to document their ruminative episodes as many times as possible. Figure 2 shows the RCT platforms on a mobile phone screen.

Figure 2. Randomized controlled trial platforms for participants to self-report and to receive information and feedbacks like the level of negative emotions one has experienced, and cognitive behavioral therapy training materials tailored to their current rumination episode. No-treatment control platforms for participants to self-report information (participants only report and document their rumination episodes.
View this figure

Ethical Considerations

This study was approved by the institutional review board at the University of Southern California (IRB; UP-20-01073-AM002). The IRB that reviewed this work required that, for this trial involving diagnosed patients with depression, researchers had to exclude patients who had not taken medication or had not visited a therapist in the past 3 months—we did so. Therefore, all participants in the treatment and control groups in this study were in active treatment for clinical depression. We explained to participants, both in the recruitment message and the consent form, that (1) the study was a double-blinded RCT to help reduce ruminative episodes related to depression, (2) there was an equal chance of being assigned to the no-treatment control and the JITAI condition, (3) this RCT was designed to test the effectiveness of mobile phones as a potential mode of delivery for rumination-based CBTs (MRFCBT) rather than to provide treatment for clinical depression, and (4) this RCT would only be delivered via mobile phones. They were also informed of the name of the research institution whose IRB approved the research. Participants were also entered into a lottery with 1 in 3 odds of receiving a US $50 gift card. A total of 59 volunteers agreed to be enrolled in this RCT and participated in the screener survey on the web.

Recruitment

Participants in this study were recruited via a volunteering website originally developed using National Institutes of Health funds—ResearchMatch, a national web-based registry of >150,000 volunteers seeking to volunteer for new, potentially disorder-relevant studies. Upon approval from the IRB, ResearchMatch allows researchers to access and contact volunteers who are considering participating in research studies or clinical trials [40,41]. ResearchMatch offers filters for researchers to screen participants who may be eligible. It also has a message system for researchers to contact potential participants. We contacted potential participants who (1) were aged ≥18 years; (2) had self-reported recently (within the past 12 months) having been given a diagnosis of clinical depression; (3) had concurrently self-reported that they had no diagnosis of any other mental health disorder; (4) self-reported that they could understand, read, and write in English; and (5) reported that they had a smartphone with a data plan. Participants were also asked to indicate their medication and specify their most recent visit to a therapist. These were our inclusion criteria.

Of the 59 volunteers, 25 (42%) were eligible and provided their phone numbers. A total of 20% (5/25) of the volunteers never replied to the SMS text message reminding them to start the RCT. Of the 20 participants who logged on to the RCT website and initiated the RCT, 2 (10%) requested to quit the RCT during the first week, and 18 (90%) completed the RCT (ie, finishing 80% of the questionnaires and training tasks). On average, those 18 participants answered 145.76 (SD 13.35) out of 175 survey prompts, affording enough data points to conduct analysis on intensive within-person changes. Data for the 10% (2/20) of participants who left the RCT before the actual treatment (second week) were not included in the analysis. Table 1 shows the demographic backgrounds and other information of that 90% (18/20) of participants. A total of 50% (9/18) of the participants were assigned to the JITAI condition, and the other 50% (9/18) were assigned to the no-treatment control condition.

Table 1. Participants’ demographics and diagnosis information (N=18).
CharacteristicsParticipants, n (%)
Gender

Man6 (33)

Woman12 (67)

Nonbinary0
Income (US $)

<30,0003 (17)

30,000-59,9996 (33)

60,000-89,9997 (39)

≥90,0002 (11)
Ethnicity

Non-Hispanic White11 (61)

Hispanic3 (17)

Black2 (11)

Pacific Islander0 (0)

Native American1 (6)

Middle Eastern1 (6)

Asian American0 (0)
Depression diagnosis

Major depressive disorder12 (67)

Persistent depressive disorder1 (6)

Psychotic depression2 (11)

“Situational” depression1 (6)

Mild depressive diagnosis2 (11)

Double-blind assignment of participants was achieved through 2 survey platforms: participants first filled in a consent form and a screening survey on REDCap (Research Electronic Data Capture; Vanderbilt University), and REDCap automatically directed those who were eligible to the RCT survey platform, which randomly assigned participants to one of the 2 RCT conditions. There was no way to link the screener and contact information of REDCap to the RCT website. Therefore, neither researchers nor participants had access to RCT assignment information during the RCT. After participants completed the RCT, the researchers could retrieve the no-treatment control and treatment information per participant through website logs. There were no significant differences in demographics and diagnosis information by condition, as would be expected given randomization to condition.

Might participants have ascertained the condition to which they were assigned? This is possible. Unfortunately, we did not ask participants at the conclusion of the study what condition they thought they had been assigned to, and this should be added in future research. However, participants were unlikely to ascertain with any certainty the condition information. For example, participants in one of the experimental conditions might have thought they were in an experimental condition (vs control). However, it would have been hard to know if that experimental condition was not actually a control condition against some other experimental intervention group. Participants in the control condition also received message prompts every 3 hours to document their rumination experiences and emotional states. Therefore, participants who were assigned to the control group might have thought that they were in an experimental group, that is, an experimental group in which there was frequent monitoring of their rumination (eg, vs a no-monitoring control). Indeed, before the beginning of participation, although participants were told that there was an “experimental” and a “control group,” the nature of these conditions was not specified.

Measurements

Overview

In addition to a set of demographic questions, we asked participants to document daily their level of COVID-19 stress and the total amount of standard drinks that they had consumed. To evaluate the ruminative experiences, participants also indicated their responses on a battery of depression measurements (see the following sections). After the intervention, participants were also asked to indicate their user experiences (see the following sections).

Ruminative Response Scale

Rumination patterns for each individual (and the differences therein) toward their depressive feelings were measured using the Ruminative Response Scale [42]. The Ruminative Response Scale consists of 22 items for assessing ruminative patterns and their dimensions (ie, depression, brooding, and reflection). This is a trait-like assessment of persistent rumination responses. In this study, participants rated each of these items on a scale of 1 to 7 (1=not at all like me and 7=extremely like me), such as “I think about how alone I feel” or “what am I doing to deserve this?” This measure exhibited excellent reliability in this study (Cronbach α=.95).

Ruminative Episode and Duration

To evaluate participants’ experiences of rumination, we asked 2 questions. First, in the past 3 hours, did you experience a ruminative episode (by ruminative episode, we mean experiencing a battery of negative thoughts that distracted from your current situation)? Second, if the participants answered “yes,” they were asked, “how long did this ruminative episode last?”

State Depression Scale

The State Depression Scale (SDS) is a 3-item scale for participants to rate their in-the-moment level of depressed mood, anhedonia, and irritability on 7-point Likert scales ranging from 1 (not at all) to 7 (very much) at the time of the alert. The SDS has already been tested and used in previous EMA studies on rumination and depression [43]. This scale was presented as a slider bar to fit the participants’ mobile phone screen size. The Cronbach α for this scale has consistently been high (ie, .8 to .9 range) each time it is used in research.

Emotional Experiences

Participants’ in-the-moment emotions at the time of sampling were measured using a pictorial self-report by the name of Pick-A-Mood. This is essentially a 7-point Likert scale measuring a range of feelings such as tense, annoyed, calm, and relaxed. Pick-A-Mood was designed specifically for mobile use [44], which serves as a better alternative than other emotion measurements. The 8 emotional items were later combined into a set of positive, neutral, and negative feelings. As in the case with the SDS, the Cronbach α for this scale was consistently high (ie, .8 to .9).

Statistical Analysis

Power

As the goal of this RCT involved evaluating changes in ruminative episodes during the intervention, it follows the recommended design of an intensive within-person, microrandomized trial. There are 2 issues pertaining to power that are important to differentiate. First, normally when considering power for interventions, the authors calculate the number of participants needed for sufficient power (eg, 0.80) to detect an effect given one’s design. Unfortunately, there is currently no concrete standard to calculate this type of sample size for adequate power (or power given sample size) in such trials to formally test the effectiveness of the intervention versus the control group per se [45]. There is a second type of power consideration that is specific to Group Iterative Multiple Model Estimation (GIMME). The issue is as follows: are there enough data points (eg, EMA events) within participants for the researchers conducting such analyses to trust the rigor of the results? That is, with GIMME, to have sufficient power to detect a small effect size when researchers are tracking individual changes, each participant is required to document changes in their behaviors of interest with sufficient time points (approximately n=60-75) [45]. Moreover, GIMME package developers note that it usually takes 7 to 10 participants within each comparison group to detect between-group differences [46,47]. In this study, of the 18 participants who completed the study, 9 (50%) were in the control condition and 9 (50%) were in the JITAI condition.

The GIMME Approach

Today’s technologies (eg, smartphone just-in-time interventions) can provide an enormous amount of intensive data over time for a given person. At any given point in time when given events occur (or not), an individual may have co-occurring affective reactions, cognitions, and other states. In addition, each or none of these phenomena in the moment (and their relationships with one another) may predict reactions at the next moment in time (and the relationship among elements at that later point in time). Often, these complex data are merely averaged over time or over chunks of time within a person, with considerable loss of information as a given individual moves from moment to moment and context to context, where variability in behavior at the within-person level needs to be understood. In our work, for example, we would argue that a therapeutic client may benefit more from a rumination intervention if given within a few moments of a potentially causal event, cognition, or emotional reaction that precipitated rumination than from an intervention that happens the next day or a week later outside the current context. Such detailed co-occurrent and cross–time-lagged networks of information are critical for therapists who would desire to know when to provide what therapeutic message to optimize their effectiveness. To be able to understand this intensive within-person idiographic level of response and respond to it, we need newer statistical approaches.

The need for intensive within-person measurement has already become apparent in other areas of social science using intensive (every 3 hours) EMAs, such as in personality psychology. Over the last 10 years—recognizing the importance of within-person variability that cannot be accounted for based on between-person measures (eg, Big Five trait measures)—the field has embraced the importance of GIMME, which affords an intensive within-person assessment over time. Beck and Jackson [48], publishing in the flagship journal of the field, the Journal of Personality and Social Psychology, delineated this new approach to idiographic personality models and how it identifies varying patterns of behavior and behavior change by person by context and lagging over time over 2 years. The method affords insight compared with nomothetic approaches but also compared with other idiographic approaches that only look at phenomena at one or relatively few points in time. GIMME affords an incredible tool for examining these sophisticated within-person dynamics over context and time.

Although GIMME is a relatively new approach, it leverages many existing statistical approaches that have been widely used for many decades across science. For example, GIMME leverages network science (and computer science) approaches to implementing network theory. Although a full description of this approach is beyond the scope of this paper, the statistical rationale for network analysis has been delineated in leading journals in science, including in Nature (eg, see the work by Whitfield [49]). In brief, network theory is a part of graph theory [50]—a network can be defined as a graph consisting of attributes such as nodes (eg, in this case, rumination episodes and affective states) or edges (associations between nodes). Therefore, the edges in a network (as in GIMME) are essentially correlation coefficients. GIMME allows us to examine the relationships (symmetrical or asymmetrical) among these attributes in a network cotemporaneously as well as lagged over time. GIMME takes advantage of the recent advances in idiographic models and network science. Rather than using a nomothetic multivariate time series analysis, GIMME uses a network approach that highlights relationships among variables both visually and quantitatively, representing both direct and indirect associations among them. Beck and Jackson [48] delineated how idiographic networks can answer questions involving intraindividual change that are not easily addressed using nomothetic models [51].

At the within level, GIMME uses an estimation analogous to autoregression. Using GIMME, one can model the subsequent scores (or occurrences) of an event using the previous scores (or occurrences). For example, in this study, we measured every 3 to 5 hours whether a participant experienced rumination in each of the 35 days of the study period. We examined contemporaneous and lagged relationships. Contemporaneous relationships estimate probabilistic within-person relationships at the same time point. This is the tendency for ruminative symptoms to occur at the same time. Lagged relationships examine probabilistic within-person, cross–time-point (or cross-lagged) relationships. This is the tendency for ruminative symptoms to follow one another across measurement conditions. For example, in general, for ruminative behavior, if a rumination episode happens at time 1, then there is a high possibility for a similar rumination episode to happen later. Using those experienced rumination episodes, one can use GIMME to construct a time series model of subsequent occurrences of rumination episodes in the same way that one can construct a time series multilevel model, but one can construct a model that is truly ideographic as there is no shrinkage of individual-level estimates [52]. That is, GIMME can model the associations between co-occurring scores (eg, symptoms or occurrences) both concurrently (0 lag) and at cross–time lags. In our analysis, we also measured every 3 to 5 hours scores of affective reactions of participants. Putting scores of affective reactions as co-occurring variables with rumination, GIMME can generate a model to estimate both (1) the association between affective reactions and rumination at the same time when they occur and (2) whether previous affective reactions could be predictive of other variables in the model. In this study, we used GIMME to examine whether previous affective states could be predictive of the occurrence of subsequent rumination episodes. Thus, GIMME [52,53] can reliably obtain the presence and direction of effects among variables collected at multiple (intensive) time points, and it uses the full information maximum likelihood that rigorously estimates the missing data points.

Furthermore, GIMME estimates both between- and within-person effects similar to a multilevel model, where there is dependence between individual- and group-level estimates. Specifically, GIMME can estimate both associations among variables measured simultaneously and how variables measured at a particular time point predict the effects and directions of relationships with variables collected at subsequent time points. To do so, GIMME estimates those associations and directions of effects through both unified structural equation modeling (uSEM) and network analysis. That is, in estimating a GIMME model, the algorithm first uses uSEM, a structural vector autoregressive model, to fit both the contemporaneous and lagged parameters. However, it does so in a way that is different from a multilevel model that simply estimates “average” relationships—GIMME keeps group-level paths that are significant for at least 75% of individuals and gets rid of paths that are not significant for at least 75% of the sample. GIMME uses the selected path from uSEM to retain those group-level paths [46,54].

GIMME models work well for considering the effects of an intervention—participants who reported experiencing a rumination episode were immediately sent a JITAI message tailored to the specific individual trigger of their rumination (eg, problem-solving skills or interpersonal skills). The GIMME model, with its capacity to evaluate the significant changes in rumination episodes since their previous occurrences, can evaluate the extent to which tailored message feedback was successful in helping participants take control of their condition by showing whether there was a significant reduction in rumination episodes from their previous rumination occurrences. After fitting each individual model, GIMME also fits a group- or subgroup-level model that looks at between-group effects. GIMME analysis also produces network connections for each participant and group (if there is a group-level effect). Network graphs are produced by treating each variable of interest as a node and the associations (both contemporaneous and lagged) between each variable as edges [46,54]. Analyses were conducted using the GIMME package in R [46,54].


Overview

Hypothesis 1 predicted stronger effects of the JITAI-MRFCBT condition versus the control condition in reducing depressive rumination. Table 2 shows each participant’s reported statistics (ie, total count and average duration) regarding ruminative episodes (“in the past three hours, have you experienced a negative thinking episode?”). Table 2 is a summary of rumination episodes and average minutes spent per rumination episode per week throughout the 5-week period. Owing to the limited sample size (each group had 9 participants), we used bootstrapped independent 2-tailed t tests to compare differences between the treatment and control conditions for the baseline and postintervention weeks. In the baseline week (Table 2), there were no significant differences in the counts of rumination episodes (t16=−0.94; P=.93) and the average time spent ruminating between the treatment and control groups (t16=−0.73; P=.94). We also calculated reduction values. Reduction of rumination episode counts and reduction in time spent on rumination were created by subtracting rumination episode counts in week 5 from those in week 1 per participant. Between groups, using a series of independent t tests with bootstrapping, results showed that participants in treatment (mean −25.28, SD 14.50) reported a significantly greater reduction in rumination episode counts than those in the control condition (mean 1.44, SD 4.12, 95% CI 14.25-33.67; P<.001; Cohen d=2.5). Similarly, those in the treatment condition (mean −21.53, SD 17.6) showed a significant reduction in average minutes spent in rumination compared with those in the control condition (mean 1.47, SD 1.5, 95% CI 10.58-32.00; P=.04; Cohen d=1.84). Figure 3 also shows participants’ counts of ruminative episodes and the average time they spent on each rumination episode per week.

Table 2. The count of rumination episodes and average duration (in minutes) of each ruminative episode each week.
Condition and participant IDWeek 1, NaWeek 2, NWeek 3, NWeek 4, NWeek 5, NWeek 1, mean (SD)bWeek 2, mean (SD)Week 3, mean (SD)Week 4, mean (SD)Week 5, mean (SD)
Treatment

i_120197119.518.7517.96416.5313.2414.2308

i_2510001515.610.0150

i_3257432113.571107.56100.360.756.5

i_4978969535455.5550.545.540.535.4

i_55038342720110110.697.689.579

i_6151574320.4520.515.899.769.4

i_72912115055.950.8945.337.821.4

i_840444327136968.3367.8965.356.2

i_9787060553079.9978.6375.873.4672.3
Control

i_109888788099757078.878.975.2

i_1130222528295555.779.356.3553.5

i_12454443424456.565.556.354.553.5

i_13424143424162.562.563.5245.560.25

i_1419181516218985.386.585.587

i_1523222420224346.350.3643.540.47

i_16323335122031.2333.7332.333.2328

i_17565955605569.6867.568.669.770

i_18252925272665.365.1269.765.366.04

aCount of ruminative episodes.

bAverage minutes spent on rumination.

Figure 3. Changes in count of ruminative episodes and average minutes per each ruminative episode 5 weeks.
View this figure

Principal GIMME Results

GIMME provides a detailed look at the rumination reduction found in this study. This is because traditional t tests could not provide a rigorous estimation for RCT effects because of the sample size at the between-group level [51]. A GIMME analysis was conducted to evaluate within-person changes, which provided robust results in explaining the effects of this RCT. We specified the analysis as a multiple group estimation that compares the differences between the JITAI MRFCBT and no-treatment control conditions. GIMME modeling first estimates the association between each variable as β coefficients at each individual level (if any). Table 3 shows a set of fit indexes that indicate the model fit for each within-person change model (18 individuals) using the uSEM approach. The root mean square error of approximation (RMSEA) was evaluated with chi-square tests. In this case, an RMSEA of 0.01 is considered an excellent fit [55]. A measure of comparative fit is the comparative fit index (CFI); in this case, CFI=0.98. Hu and Bentler [56] suggested that RMSEA values <0.06 and CFI values >0.95 are indicative of a good fit. A more recent measure of fit to avoid model misspecification is the standardized root mean square residual (SRMR) [57]. In this case, an SRMR of 0.06 indicates a good fit as it is <0.08. Our results show that a great majority (15/18, 83%) of these models achieved excellent fit. The estimation of group-level effects is set at 75%, meaning that group-level analysis will not be conducted unless 75% of the participants have shown an excellent fit [46,54].

Table 3. Within-individual fit indexes in Group Iterative Multiple Model Estimation (GIMME; waitlist control and just-in-time adaptive intervention [JITAI] comparison).a
Condition and participant IDChi-square (df)P valueRMSEAbSRMRcNFIdCFIe
JITAI condition

individual_1109.9389 (78).010.070.070.960.97

individual_2101.2896 (77).030.060.070.960.97

individual_397.9413 (79).070.070.090.970.98

individual_494.4551 (79).110.050.060.980.99

individual_5123.9945 (77)<.0010.070.050.950.97

individual_6111.9069 (80).020.060.070.950.97

individual_7114.0346 (79).360.020.061.001.00

individual_8116.1622 (76).020.070.060.960.97

individual_9118.2899 (74)<.0010.090.050.940.96
Control condition

individual_10120.4176 (79)<.0010.400.050.950.97

individual_11122.5452 (76).040.300.050.940.97

individual_12124.6729 (76).080.350.050.940.96

individual_13126.8006 (76)<.0010.450.050.940.96

individual_14128.9283 (75).020.520.050.940.96

individual_15131.0559 (75).020.590.040.940.96

individual_1695.5461 (77).010.050.060.980.99

individual_1785.0285 (76).220.050.080.960.98

individual_18146.7335 (77)<.0010.110.050.940.96

aFit indexes provided by GIMME are subject to the same rule of thumb in structural equation modeling [46].

bRMSEA: root mean square error of approximation.

cSRMR: standardized root mean square residual.

dNFI: normed fit index.

eCFI: comparative fit index.

A sample of β correlations per participant (individual 16 in the control condition and individual 7 in the JITAI condition) showing predictive models based on a participant’s self-reported rumination relationships can be found in Tables 4 and 5, respectively (we did not provide the full estimation for all 18 participants to avoid repetition). The results (Table 4 for individual 16) showed that the rumination episode significantly predicted the occurrence of a subsequent rumination episode (β=.022; P=.02). Table 5 shows a model that describes another participant’s (JITAI condition, individual 7) self-reported changes during the intervention. This model also shows an acceptable model fit (Table 3), with RMSEA=0.02, SRMR=0.06, and CFI=0.98. Interestingly, we see that, for this participant, the rumination episode (episode) was not a significant predictor of the occurrence of a subsequent rumination episode (β=−.03; P=.75).

Table 4. Sample path analysis within individual 16 (control condition).a
PredictorDVbβSEz scoreP value
EpisodecEpisodelagd.0220.092.43.02
DurationminseDurationminslag.020.060.37.71
StateRMfStateRMlag−.060.04−1.41.16
NeutralgNeutrallag−.110.10−1.11.27
NegativehNegativelag.240.092.52.01
PositivePositivelag.120.081.40.16
CovidstiCovidstlag.910.0252.88<.001
DrinksjDrinkslag.860.0333.58<.001
StateRMEpisode1.170.0620.82<.001
DurationminsEpisode.780.0419.50<.001
EpisodeNeutral−.400.08−4.73<.001
NegativeStateRM.300.093.23<.001
StateRMDurationmins−.350.07−4.85<.001
PositiveNegative−.440.07−5.87<.001

aEpisode is the occurrence of a ruminative episode at each time; stateRM is the state depressive rumination level; and negative, positive, and neutral indicate participants’ emotional state.

bDV: dependent variable.

cEpisode: the number of rumination episodes.

dThe affix -lag that comes after each variable (eg, episodelag or positivelag) indicates the subsequent scores and occurrences of the variable.

eDurationmins: minutes spent on a rumination episode.

fStateRM: level of rumination at the rumination episode.

gNeutral: level of neutral affect in the past 3 to 5 hours.

hNegative: level of negative affect in the past hour.

iCovidst: stress related to COVID-19.

jDrinks: drinking related to rumination.

Table 5. Sample path analysis within individual 7 (just-in-time adaptive intervention mobile rumination-focused cognitive behavioral therapy condition).a
PredictorDVbβSEz scoreP value
EpisodecEpisodelagd−.030.09−0.32.75
DurationminseDurationminslag.000.090.00>.99
StateRMfStateRMlag.010.040.16.87
NeutralgNeutrallag−.080.09−0.94.35
NegativehNegativelag.180.082.23.03
PositivePositivelag.030.110.23.82
CovidstiCovidstlag.960.0198.47<.001
DrinksjDrinkslag.990.00625.20<.001
StateRMEpisode.730.0514.79<.001
DurationminsEpisode.590.077.99<.001
EpisodeNeutral−.640.07−9.78<.001
NegativeStaterm.030.120.26.79
NeutralPositive.600.078.55<.001
StateRMNeutral−.260.06−4.61<.001
NegativeNeutral−.640.11−5.77<.001

aEpisode is the occurrence of a ruminative episode at each time; stateRM is the state depressive rumination level; and negative, positive, and neutral indicate participants’ emotional state.

bDV: dependent variable.

cEpisode: the number of rumination episodes.

dThe affix -lag that comes after each variable (eg, episodelag or positivelag) indicates the subsequent scores or occurrences of the variable.

eDurationmins: minutes spent on a rumination episode.

fStateRM: level of rumination at the rumination episode.

gNeutral: level of neutral affect in the past 3 to 5 hours.

hNegative: level of negative affect in the past hour.

iCovidst: stress related to COVID-19.

jDrinks: drinking related to rumination.

The GIMME results of comparisons both at the group (Figure 4) and individual levels provide a detailed look at the network of associations and effect carryover in time. In this analysis, shown in Figure 4, our model showed that, at the between-group level, there are significant predictors that are responsible for the occurrence of a subsequent ruminative episode. For example, in the control condition, we found that factors such as rumination episode, negative emotions, and duration of the rumination episode were also estimated as significant predictors of a subsequent rumination episode. In Figure 4, for the JITAI-MRFCBT overall effects, it is noteworthy that the dashed lines found in the no-treatment control are absent, suggesting little carryover from earlier rumination episodes and their duration. The other noteworthy difference between these conditions is the presence of a green line from positive to neutral. This can be interpreted as the presence in the moment of more positive affect that may be bolstering the user’s ability to respond more productively in the face of the rumination episode.

Figure 4. Between-group effects (no-treatment control vs just-in-time adaptive intervention [JITAI] mobile rumination-focused cognitive behavioral therapy [MRFCBT]). Note: the graph on the left shows the associations of different variables within the no-treatment control, while the one on the right shows the overall path models within the JITAI MRFCBT condition. Solid lines indicate a contemporaneous effect; dashed lines indicate a lagged effect. Black lines indicate a group-level effect; green paths indicate an effect at a subgroup; gray paths indicate an individual effect.
View this figure

The lack of dashed lines for participants in the treatment group indicates that these participants did not seem to generate as many subsequent cascading rumination episodes compared with the control participants. This difference at the group level and the individual level—and the detail with which we can “see” these network processes over time—suggests that participants in the JITAI condition did not bring their negative thoughts and previous ruminative experiences into their next ruminative episode. Indeed, the main goal of the JITAI-MRFCBT condition was to train participants to make different attributions and focus them on problem-solving when they experienced a particular ruminative episode.


Principal Findings

This pilot study compared a JITAI-MRFCBT condition with a no-treatment control condition in reducing rumination. To assess rumination reduction, we compared the baseline (week 1) and the postintervention week (week 5)—we found that both ruminative episodes and the average time participants spent on those ruminative episodes decreased significantly in both groups, but the reduction was significantly greater in the JITAI-MRFCBT condition. Our results showed that participants in the control condition demonstrated some time-lagged effects such that experiencing ruminative episodes or negative emotions at a time point predicted an increase in such experiences subsequently. However, such lagged effects were not present in the JITAI condition. That is, no dotted lines in the treatment condition indicate lagged effects. This difference at the group level shows that participants in the JITAI condition, compared with those in the control condition, did not bring their negative thoughts and previous ruminative experiences into their next ruminative episode. The treatment appeared to stop the “echo chamber” of the previous negative experiences.

This study, in addition to affording a comparison between treatment and control groups at both the group and the individual level, involved four innovations: (1) using MRFCBT as the “what” of the intervention and JITAI as the “when,” which has promise for optimizing responsive and effective interventions per client; (2) devising a “how” (in this case using participant baseline measures of rumination and GIMME) to identify typical baseline patterns per participant when rumination (and change in affect and cognitions) is likely; (3) using GIMME within persons per client to better understand “why,” leveraging smartphone technologies to deeply understand and predict, for the first time, networks of cognitions and affect and patterns of rumination and cascading rumination (and possible interpersonal predictors) from one moment to the next; and (4) ability (via combining these technologies) to afford identification, understanding, and change in disruptive psychological patterns (eg, rumination) personalized to the user leveraging smartphone technologies. In the following sections, we discuss each of these innovations along with their implications for the literature on rumination, depression, interpersonal communication, and techniques and technologies (eg, GIMME, EMA, and JITAI). We then discuss the limitations of this work and our overall conclusions.

Comparison With Prior Work

The “What” and “When” of CBT Using JITAIs

CBT has been a “gold standard” for a diverse array of disorders. The emergence of mobile phones as an alternative means to deliver face-to-face CBTs also speaks to an increasing need for mental health services. Researchers have been working on phone apps [58,59] to provide CBT (ie, MCBT) to those who are depressed. Despite reports of positive user experiences [60], there is still a lack of symptom-focused MCBTs (eg, MCBTs for depressive rumination). MCBTs to date, unfortunately, still lack the level of interactive features needed to provide more tailored feedback based on individual characteristics. This is despite calls to address this need [60].

However, for many disorders, CBT delivered face-to-face is apt to be less than optimally effective as clients cannot receive the treatment at the optimal time. Even when included in a mobile device, CBT is typically not responsive to the dynamics of the client’s experience. Rumination provides a very good example of these difficulties. If negative cognitions and affective states are allowed to cascade, as mentioned earlier, this exacerbates depression. Simply providing CBT materials people can access even on a mobile device is likely not enough. A scaffolding tool for responding at just the right moment with the right therapeutic tools and messages for mitigating rumination threats should be the intervention goal to which we aspire. Although no previous study has tested mobile phone–delivered JITAIs in reducing depressive rumination [26], the results of this study suggest that JITAIs could afford an essential means to deliver more responsive CBT for ruminating clients diagnosed with clinical levels of depression and that, in doing so, especially with intensive rumination measures and treatments responsive to them, rumination could be reduced.

Interesting transdiagnostic CBT approaches are promising treatments for a broad range of emotional disorders but, in reviewing this work, only rumination and negative metacognition significantly mediated client functioning [61]. These findings highlight the importance of mitigating the effects of rumination for the effectiveness of CBT interventions, thus highlighting the importance of this method and approach for “seeing” and reducing rumination.

The “How” of JITAIs for MCBT

JITAIs focus on the “when” of intervention delivery [27]. However, knowing “when” to deliver a given intervention—the “what” (eg, MCBT component)—depends on what cues and indicators are available and should be used (eg, with one’s smartphone) to make that determination. For example, JITAIs are used in trying to inhibit an individual’s alcohol [62] or illicit drug use [63,64], which need interventions to note a client’s typical source of these substances (eg, local bars and location of dealers). Knowing what environmental or personal cues might initiate a problematic behavior (that the intervention is designed to deter) provides the basis for the “how” (how JITAIs know “when” to intervene). Some possibilities are through EMA and sensor technologies in the smartphone—these can potentially provide the basis (ie, possible algorithms based on formative research) for a JITAI to “know” “when” to deliver “what” to optimize intervention goals. In this study, rumination had long been identified as problematic in exacerbating depression [31,33,37]. The “how” for JITAI in this study became how can we determine when a client (who ruminates and is diagnosed with depression) is most likely to be ruminating? Our interim step in ascertaining a relatively crude “how” was to first determine how often individuals who were clinically depressed (and ruminating) ruminated, for example, how many times a day did this occur? Across individuals in this population, there was variability in these patterns. Therefore, “how” required a more personalized assessment (ie, a baseline) of the times of day. The goal of RFCBT is to train participants to make different attributions and focus on problem-solving when they experience a particular ruminative episode. The presumption is that, when the next ruminative episode occurs, participants can immediately become aware of their ruminative thoughts and adopt one of the strategies offered to them by RFCBT. In this study, we achieved this goal through the week 1 baseline assessment. That is, we used the individualized information, which tells us the frequency, duration, types, and high time of each client’s rumination patterns, to create messages tailored to each participant. For example, if the week 1 baseline information told us that a participant had been experiencing more rumination episodes elicited by interpersonal stress rather than personal setbacks, this participant received more messages on interpersonal problem-solving.

Our process of using baseline EMA reports for the “how” for estimating probable rumination times provides a model not only for “rumination” estimates at the individual level but also for any behavior of interest we seek to change. That is, can we identify in time and space when this behavior of interest is more probable per person of interest? Can we identify cues or indicators (and possible algorithms using smartphones and EMA) that anticipate the behavior of interest? Can we collect that information at baseline to then use it in the intervention of a personalized JITAI?

Sufficiently Fine-Grained Measurement for the “Why”: Blocking Rumination Cascades

The reported findings using GIMME within persons per client provide insight into the “why” of ruminative cascades. Identifying rumination, rumination cascades, and rumination blocking is a known challenge [3,65]. The emotional cascade model suggests that emotional dysregulation turns into subsequent behavioral dysregulation through a process called emotional cascades [66]. Emotional cascades are the feedback loops between intense rumination and negative affect. Emotional cascades occur mostly after a negative trigger such as unpleasant interpersonal experiences, personal losses, and setbacks. As one becomes more focused on the negative experiences, one can find it hard to divert attention away from reliving the negative event. This cycle is repeated as rumination and negative affect interact, resulting in more dysregulated behaviors. However, “seeing” these networks of cognition and affect, especially in anything close to “real time,” has not been possible. This makes it hard to understand what exactly is triggering in the situation for this individual and how it is being framed by them. Such insights could provide targets for personalized interventions. GIMME provides a method for examining at the individual level the associations between cognition and affect for the individual at each time point and how these associations carry over and predict subsequent ruminative episodes. GIMME methods, and sufficient time sampling using EMA, afford tools to not only understand the client’s cognitive and affective construals (eg, activating hurtful perceptions) of interpersonal interactions but also “see” those with detail in a more fine-grained network than has been previously available. This makes it possible for researchers, clients, and their therapists to identify how these perceptions change and trigger rumination events for specific individuals with depression from moment to moment over time (and whether interventions designed to address them are proving effective).

Numerous empirical studies have highlighted the link between ruminative process and behavioral dysregulation and called for effective means of rumination disruption. However, research suggesting ways to effectively break this vicious cycle is scarce. Blocking rumination is historically difficult to achieve [3,65]. An approach is to use “normal methods of distraction,” which unfortunately “becomes ineffective” [65]. Part of the difficulty may be that, historically, rumination has been assessed through self-reports, often summarizing over days, weeks, or months. As far as we know, there are no current effective measures of rumination closer to “in the moment” in real time. The results of this study suggest that GIMME tools and personalized baselines can help us better “see” rumination in a more fine-grained way in each episode. This means that we can better “see” whether our interventions are effectively reducing the ruminative links from one ruminative episode to the next. “Seeing” in a fine-grained way when disruption of rumination is occurring and why is a major advance.

Combining Intervention Design, Techniques, and Technologies: Advancing Rumination Science

A major innovation in this pilot work is (1) putting these 3 innovations together, leveraging MRFCBT materials to afford identification, understanding, and change in rumination patterns, all while personalizing the intervention to the user (when to intervene with what), leveraging smartphone technologies, and (2) showing the promise of this JITAI-MRFCBT compared with the no-treatment control group. JITAIs, as used in this study, are an effective method to disrupt emotional cascades rather than simply creating distraction—instead of using traditional means of shifting attention (eg, physical sensation seeking), participants can be provided with a scaffolding frame that helps them reappraise their current emotions and alter attributions at the exact moment of the emotional cascade.

In addition, this pilot trial evaluated the promise of a mobile phone–delivered JITAI (with MRFCBT) as a delivery means of reaching participants with an in-the-moment intervention [27,67]. The results of this project have shown that, in JITAIs, participants ruminating can receive in-the-moment feedback on their mobile phones. Third, this study provides a data-driven explanation regarding the effects of JITAIs. That is, our GIMME results suggest that, with messages timely tailored and sent to participants in the time of greatest need, participants in the treatment condition showed a significant reduction in “next-time” rumination episodes. Results from the JITAI conditions are promising for reducing the number of ruminative episodes and lowering average rumination times. Therefore, the results of this pilot RCT study indicate the viability of a full-scale RCT for reducing rumination in a population diagnosed with clinical depression. Thus, this pilot serves an overarching goal of identifying a viable approach that can deliver JITAIs focusing on depression outcomes and provides the groundwork for subsequent clinical trials.

Rumination plays a significant role in a wide array of mental health disorders and problematic behaviors. The focus in this study was on MDD, but rumination also plays a significant role in other mental health disorders such as bipolar disorder [68], anxiety, and mood disorders [66,69]. Nolen-Hoeksema [30], noting the high comorbidity among many mental health disorders, conceptualized rumination as a transdiagnostic process associated with not only depression and anxiety but also a variety of deleterious outcomes (ie, substance abuse, binge eating, and self-injuring behaviors; see McLaughlin et al [66]). Indeed, emerging literature provides evidence to support such claims [66,70,71], with McLaughlin and Nolen-Hoeksema [70] arguing for “the importance of targeting rumination in transdiagnostic treatment approaches for emotional disorders.” As recently argued, “there is consistent evidence showing that a reduced use of cognitive reappraisal and an increased use of negative rumination are present across a number of disorders, whereas increased levels of positive rumination appear to be confined to bipolar disorder” [72]. Given the transdiagnostic role of rumination across many disorders, the importance of tools to measure and understand it (and intervene with respect to it) in a detailed way for individual clients across contexts and over time is critical and highly generalizable.

Limitations

However, the limitations of this RCT are apparent. The major limitation is the self-reported nature of participants’ depression diagnoses. Although we designed a series of questions such as asking participants to report their recent diagnosis and provide names of their depression medications, we may still not be able to fully generalize the efficacy of these interventions to all participants with clinical depression. Therefore, the results of this study are only limited to participants with self-reported depression diagnoses. There are other less major limitations. First, despite the small sample of participants, this analysis could offer a robust pre- and postestimation of the RCT following suggestions of a micro-RCT. However, this suggests a need for a full-scale RCT with power that can speak to between-group comparisons. Therefore, a subsequent RCT is needed. Second, because of delays in the IRB process of >1 year, some exploratory research questions on how COVID-19 and drinking patterns at the time of COVID-19 (and the impact on ruminative patterns) could not be examined in the timely way originally planned. Among participants who provided daily entries of their COVID-19–related stress, most (18/25, 72%) indicated that they experienced zero or very low COVID-19 stress. Therefore, we did not detect any significant role of COVID-19 and related drinking behaviors (shown in the GIMME models). That is, the results of this study could only be evaluated as a pilot for subsequent RCTs during future widespread events. Third, although this RCT showed some effects of treatment in reducing ruminative episodes among participants with clinical depression who were in therapy, it did not examine the effect of the RCT on participants who were depressed but who were not in therapy. Therefore, we do not know if the RCT would be effective for patients who could not secure a therapeutic relationship. Fourth, we do not know whether the treatment would be promising for patients who are not clinically depressed. Therefore, we cannot generalize the efficacy of the RCT to a sample of participants who are not clinically depressed and in therapy. Finally, as part of this RCT, we specifically designed materials for participants to reanalyze their hurtful interpersonal feelings and the support they received; these factors were designed as self-report diaries for participants in line with previous CBT for rumination. However, as we could not recruit enough participants to code these variables, we could not evaluate perceived hurtfulness and social support as potential mediators in the statistical analysis. Therefore, in subsequent work, there is a need to design better methods and measures of these potential mediators.

Conclusions

This study is the first of its kind to focus on the episode-to-episode measurement, both at the individual client and group levels, of one specific outcome (depressive rumination) underlying depression. This pilot shows how promising this JITAI-MRFCBT approach is for identifying and blocking cascades of rumination in a timely way, providing appropriate CBT at the critical time of its need. “Seeing” the individual patient networks affords opportunities to better understand and mitigate rumination—a critical psychological process that exacerbates not only depression but also a range of mental health disorders and adverse health outcomes. Indeed, as others have noted, rumination appears to be a transdiagnostic process that provides insight across a large swath of disorders, making it particularly important to use tools such as those developed in this study to better understand these processes and mitigate their adverse effects—as doing so would appear to have broad generalizability. It is worth pointing out that, in our study, none of the users reported experiencing technical difficulties using their smartphones to access, navigate, and interact with the RCT website. Therefore, it speaks to the feasibility and scalability of delivering JITAI-MRFCBT without technical difficulties. As the initial pilot intervention was designed to reduce the complexity of installing apps and complicated interactive features, users were able to adhere to the intervention with minimal technical support. Most importantly, this design minimizes the requirements for specific types of smartphones. That is, a patient can have access to the RCT if their phone has access to the internet and a browser. As such, it is a promising approach in an era of increasingly personalized interventions that provide feedback and scaffolding for the client and their therapist and for researchers trying to better understand and mitigate adverse patterns of cognition, affect, and behavior across contexts and over time to advance wellness and this emerging field of rumination science.

Acknowledgments

This project was funded by the Annenberg School of Communication and Journalism Dissertation funds and summer research funds.

Authors' Contributions

LW is the corresponding author who worked on design, data collection, analysis, and writing of the manuscript. LCM also worked on design, data collection, and writing of the manuscript.

Conflicts of Interest

None declared.

  1. American Psychiatric Association. Desk Reference to the Diagnostic Criteria from DSM-5. Washington, DC, USA: American Psychiatric Association Publishing; 2014.
  2. Flynn M, Kecmanovic J, Alloy LB. An examination of integrated cognitive-interpersonal vulnerability to depression: the role of rumination, perceived social support, and interpersonal stress generation. Cognit Ther Res 2010 Oct;34(5):456-466 [FREE Full text] [CrossRef] [Medline]
  3. Selby EA, Joiner Jr TE. Cascades of emotion: the emergence of borderline personality disorder from emotional and behavioral dysregulation. Rev Gen Psychol 2009 Sep 01;13(3):219 [FREE Full text] [CrossRef] [Medline]
  4. Joiner Jr TE. A test of the hopelessness theory of depression in youth psychiatric inpatients. J Clin Child Psychol 2000 Jun;29(2):167-176. [CrossRef] [Medline]
  5. Wang L, Miller LC. Just-in-the-Moment adaptive interventions (JITAI): a meta-analytical review. Health Commun 2020 Nov;35(12):1531-1544. [CrossRef] [Medline]
  6. Wang YG, Wang YQ, Chen SL, Zhu CY, Wang K. Theory of mind disability in major depression with or without psychotic symptoms: a componential view. Psychiatry Res 2008 Nov 30;161(2):153-161. [CrossRef] [Medline]
  7. Nalabandian T, Ireland M. Depressed individuals use negative self-focused language when recalling recent interactions with close romantic partners but not family or friends. In: Proceedings of the 6th Workshop on Computational Linguistics and Clinical Psychology. 2019 Presented at: CLPsych '19; June 6, 2019; Minneapolis, MN, USA p. 62-73. [CrossRef]
  8. Anxiety disorders | Quality standards. National Institutes for Health and Care Excellence. 2014 Feb 6.   URL: https://www.nice.org.uk/guidance/qs53 [accessed 2021-08-29]
  9. Reavell J, Hopkinson M, Clarkesmith D, Lane DA. Effectiveness of cognitive behavioral therapy for depression and anxiety in patients with cardiovascular disease: a systematic review and meta-analysis. Psychosom Med 2018 Oct;80(8):742-753. [CrossRef] [Medline]
  10. Kuhn E, Greene C, Hoffman J, Nguyen T, Wald L, Schmidt J, et al. Preliminary evaluation of PTSD Coach, a smartphone app for post-traumatic stress symptoms. Mil Med 2014 Jan;179(1):12-18. [CrossRef] [Medline]
  11. Horsch CH, Lancee J, Griffioen-Both F, Spruit S, Fitrianie S, Neerincx MA, et al. Mobile phone-delivered cognitive behavioral therapy for insomnia: a randomized waitlist controlled trial. J Med Internet Res 2017 Apr 11;19(4):e70 [FREE Full text] [CrossRef] [Medline]
  12. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided Internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry 2017 Apr 01;74(4):351-359. [CrossRef] [Medline]
  13. Sijbrandij M, Kunovski I, Cuijpers P. Effectiveness of Internet-delivered cognitive behavioral therapy for posttraumatic stress disorder: a systematic review and meta-analysis. Depress Anxiety 2016 Sep;33(9):783-791. [CrossRef] [Medline]
  14. Chandrashekar P. Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth 2018;4:6 [FREE Full text] [CrossRef] [Medline]
  15. Chmitorz A, Kunzler A, Helmreich I, Tüscher O, Kalisch R, Kubiak T, et al. Intervention studies to foster resilience - a systematic review and proposal for a resilience framework in future intervention studies. Clin Psychol Rev 2018 Feb;59:78-100 [FREE Full text] [CrossRef] [Medline]
  16. Joyce S, Shand F, Tighe J, Laurent SJ, Bryant RA, Harvey SB. Road to resilience: a systematic review and meta-analysis of resilience training programmes and interventions. BMJ Open 2018 Jun 14;8(6):e017858 [FREE Full text] [CrossRef] [Medline]
  17. Stone A. Ecological momentary assessment and experience sampling. In: Vannette DL, Krosnick JA, editors. The Palgrave Handbook of Survey Research. Cham, Switzerland: Palgrave Macmillan; 2018:563-571.
  18. Stone AA, Shiffman S. Ecological momentary assessment (EMA) in behavioral medicine. Ann Behav Med 1994;16(3):199-202. [CrossRef]
  19. Nahum-Shani I, Smith SN, Spring BJ, Collins LM, Witkiewitz K, Tewari A, et al. Just-in-Time Adaptive Interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann Behav Med 2018 May 18;52(6):446-462 [FREE Full text] [CrossRef] [Medline]
  20. Pham Q, Wiljer D, Cafazzo JA. Beyond the randomized controlled trial: a review of alternatives in mHealth clinical trial methods. JMIR Mhealth Uhealth 2016 Sep 09;4(3):e107 [FREE Full text] [CrossRef] [Medline]
  21. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
  22. Klasnja P, Hekler EB, Shiffman S, Boruvka A, Almirall D, Tewari A, et al. Microrandomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol 2015 Dec;34S:1220-1228 [FREE Full text] [CrossRef] [Medline]
  23. Connolly SL, Alloy LB. Rumination interacts with life stress to predict depressive symptoms: an ecological momentary assessment study. Behav Res Ther 2017 Oct;97:86-95 [FREE Full text] [CrossRef] [Medline]
  24. Hallensleben N, Glaesmer H, Forkmann T, Rath D, Strauss M, Kersting A, et al. Predicting suicidal ideation by interpersonal variables, hopelessness and depression in real-time. An ecological momentary assessment study in psychiatric inpatients with depression. Eur Psychiatry 2019 Feb;56:43-50. [CrossRef] [Medline]
  25. Kim H, Lee S, Lee S, Hong S, Kang H, Kim N. Depression prediction by using ecological momentary assessment, Actiwatch data, and machine learning: observational study on older adults living alone. JMIR Mhealth Uhealth 2019 Oct 16;7(10):e14149 [FREE Full text] [CrossRef] [Medline]
  26. Watkins ER, Mullan E, Wingrove J, Rimes K, Steiner H, Bathurst N, et al. Rumination-focused cognitive-behavioural therapy for residual depression: phase II randomised controlled trial. Br J Psychiatry 2011 Oct;199(4):317-322. [CrossRef] [Medline]
  27. Sharmin M, Raij S, Epstien D, Nahum-Shani I, Beck JG, Vhaduri S, et al. Visualization of time-series sensor data to inform the design of just-in-time adaptive stress interventions. Proc ACM Int Conf Ubiquitous Comput 2015 Sep;2015:505-516 [FREE Full text] [CrossRef] [Medline]
  28. Heron KE, Smyth JM. Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. Br J Health Psychol 2010 Feb;15(Pt 1):1-39 [FREE Full text] [CrossRef] [Medline]
  29. Lyubomirsky S, Nolen-Hoeksema S. Effects of self-focused rumination on negative thinking and interpersonal problem solving. J Pers Soc Psychol 1995 Jul;69(1):176-190. [CrossRef] [Medline]
  30. Nolen-Hoeksema S. The role of rumination in depressive disorders and mixed anxiety/depressive symptoms. J Abnorm Psychol 2000 Aug;109(3):504-511. [Medline]
  31. Nolen-Hoeksema S, Wisco BE, Lyubomirsky S. Rethinking rumination. Perspect Psychol Sci 2008 Sep;3(5):400-424. [CrossRef] [Medline]
  32. Camper PM, Jacobson NS, Holtzworth-Munroe A, Schmaling KB. Causal attributions for interactional behaviors in married couples. Cogn Ther Res 1988 Apr;12(2):195-209. [CrossRef]
  33. Michl LC, McLaughlin KA, Shepherd K, Nolen-Hoeksema S. Rumination as a mechanism linking stressful life events to symptoms of depression and anxiety: longitudinal evidence in early adolescents and adults. J Abnorm Psychol 2013 May;122(2):339-352 [FREE Full text] [CrossRef] [Medline]
  34. Joiner Jr TE, Metalsky GI. A prospective test of an integrative interpersonal theory of depression: a naturalistic study of college roommates. J Pers Soc Psychol 1995 Oct;69(4):778-788. [CrossRef] [Medline]
  35. Johnson DP, Whisman MA. Gender differences in rumination: a meta-analysis. Pers Individ Dif 2013 Aug;55(4):367-374 [FREE Full text] [CrossRef] [Medline]
  36. Olatunji BO, Naragon-Gainey K, Wolitzky-Taylor KB. Specificity of rumination in anxiety and depression: a multimodal meta-analysis. Clin Psychol Sci Pract 2013 Sep 16;20(3):225-257. [CrossRef]
  37. Rood L, Roelofs J, Bögels SM, Nolen-Hoeksema S, Schouten E. The influence of emotion-focused rumination and distraction on depressive symptoms in non-clinical youth: a meta-analytic review. Clin Psychol Rev 2009 Nov;29(7):607-616. [CrossRef] [Medline]
  38. Psychologytools.   URL: https://www.psychologytools.com/ [accessed 2022-12-15]
  39. Arslan RC, Walther MP, Tata CS. formr: a study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R. Behav Res Methods 2020 Feb;52(1):376-387 [FREE Full text] [CrossRef] [Medline]
  40. Harris PA, Scott KW, Lebo L, Hassan N, Lightner C, Pulley J. ResearchMatch: a national registry to recruit volunteers for clinical research. Acad Med 2012 Jan;87(1):66-73 [FREE Full text] [CrossRef] [Medline]
  41. Pulley JM, Jerome RN, Bernard GR, Olson EJ, Tan J, Wilkins CH, et al. Connecting the public with clinical trial options: the ResearchMatch Trials Today tool. J Clin Transl Sci 2018 Aug;2(4):253-257 [FREE Full text] [CrossRef] [Medline]
  42. Nolen-Hoeksema S. Responses to depression and their effects on the duration of depressive episodes. J Abnorm Psychol 1991 Nov;100(4):569-582. [CrossRef] [Medline]
  43. Moberly NJ, Dickson JM. Goal conflict, ambivalence and psychological distress: concurrent and longitudinal relationships. Pers Individ Dif 2018 Jul;129:38-42. [CrossRef]
  44. Desmet PM, Vastenburg MH, Romero N. Mood measurement with Pick-A-Mood: review of current methods and design of a pictorial self-report scale. J Des Res 2016;14(3):241-279. [CrossRef]
  45. Ruwaard J, Kooistra L, Thong M. Ecological Momentary Assessment in Mental Health Research. EMA Research Manual. 2018 Nov 26.   URL: https://jruwaard.github.io/aph_ema_handbook/ [accessed 2022-08-27]
  46. Beltz AM, Gates KM. Network mapping with GIMME. Multivariate Behav Res 2017;52(6):789-804 [FREE Full text] [CrossRef] [Medline]
  47. Lane ST, Gates KM. Automated selection of robust individual-level structural equation models for time series data. Struct Equ Modeling 2017 May 01;24(5):768-782. [CrossRef]
  48. Beck ED, Jackson JJ. Consistency and change in idiographic personality: a longitudinal ESM network study. J Pers Soc Psychol 2020 May;118(5):1080-1100. [CrossRef] [Medline]
  49. Whitfield J. Group theory; What makes a successful team? John Whitfield looks at research that uses massive online databases and network analysis to come up with some rules of thumb for productive collaborations. Nature 2008 Oct 9;455(7214):720-724.
  50. Scott J, Carrington PJ. The SAGE Handbook of Social Network Analysis. Thousand Oaks, CA, USA: Sage Publications; 2011.
  51. Piccirillo ML, Beck ED, Rodebaugh TL. A clinician’s primer for idiographic research: considerations and recommendations. Behav Ther 2019 Sep;50(5):938-951. [CrossRef] [Medline]
  52. Gates KM, Molenaar PC. Group search algorithm recovers effective connectivity maps for individuals in homogeneous and heterogeneous samples. Neuroimage 2012 Oct 15;63(1):310-319. [CrossRef] [Medline]
  53. Gates KM, Molenaar PC, Iyer SP, Nigg JT, Fair DA. Organizing heterogeneous samples using community detection of GIMME-derived resting state functional networks. PLoS One 2014 Mar 18;9(3):e91322 [FREE Full text] [CrossRef] [Medline]
  54. Beltz AM, Moser JS, Zhu DC, Burt SA, Klump KL. Using person-specific neural networks to characterize heterogeneity in eating disorders: illustrative links between emotional eating and ovarian hormones. Int J Eat Disord 2018 Jul;51(7):730-740 [FREE Full text] [CrossRef] [Medline]
  55. MacCallum RC, Browne MW, Sugawara HM. Power analysis and determination of sample size for covariance structure modeling. Psychol Methods 1996 Jun;1(2):130-149. [CrossRef]
  56. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model 1999 Jan;6(1):1-55. [CrossRef]
  57. Henseler J, Dijkstra TK, Sarstedt M, Ringle CM, Diamantopoulos A, Straub DW, et al. Common beliefs and reality about PLS: comments on Rönkkö and Evermann (2013). Organ Res Methods 2014 Apr 10;17(2):182-209. [CrossRef]
  58. Wright JH, Mishkind M, Eells TD, Chan SR. Computer-assisted cognitive-behavior therapy and mobile apps for depression and anxiety. Curr Psychiatry Rep 2019 Jun 27;21(7):62. [CrossRef] [Medline]
  59. Josephine K, Josefine L, Philipp D, David E, Harald B. Internet- and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord 2017 Dec 01;223:28-40. [CrossRef] [Medline]
  60. Stawarz K, Preist C, Tallon D, Wiles N, Coyle D. User experience of cognitive behavioral therapy apps for depression: an analysis of app functionality and user reviews. J Med Internet Res 2018 Jun 06;20(6):e10120 [FREE Full text] [CrossRef] [Medline]
  61. Muñoz-Navarro R, Medrano LA, Limonero JT, González-Blanch C, Moriana JA, Ruiz-Rodríguez P, et al. The mediating role of emotion regulation in transdiagnostic cognitive behavioural therapy for emotional disorders in primary care: secondary analyses of the PsicAP randomized controlled trial. J Affect Disord 2022 Apr 15;303:206-215 [FREE Full text] [CrossRef] [Medline]
  62. Gustafson DH, McTavish FM, Chih M, Atwood AK, Johnson RA, Boyle MG, et al. A smartphone application to support recovery from alcoholism: a randomized clinical trial. JAMA Psychiatry 2014 May;71(5):566-572 [FREE Full text] [CrossRef] [Medline]
  63. Singh NB, Björling EA. A review of EMA assessment period reporting for mood variables in substance use research: expanding existing EMA guidelines. Addict Behav 2019 Jul;94:133-146. [CrossRef] [Medline]
  64. Linas BS, Genz A, Westergaard RP, Chang LW, Bollinger RC, Latkin C, et al. Ecological momentary assessment of illicit drug use compared to biological and self-reported methods. JMIR Mhealth Uhealth 2016 Mar 15;4(1):e27 [FREE Full text] [CrossRef] [Medline]
  65. Tuna E, Bozo Ö. Exploring the link between emotional and behavioral dysregulation: a test of the emotional cascade model. J Gen Psychol 2014;141(1):1-17. [CrossRef] [Medline]
  66. McLaughlin KA, Aldao A, Wisco BE, Hilt LM. Rumination as a transdiagnostic factor underlying transitions between internalizing symptoms and aggressive behavior in early adolescents. J Abnorm Psychol 2014 Feb;123(1):13-23 [FREE Full text] [CrossRef] [Medline]
  67. Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychol 2015 Dec;34S:1209-1219 [FREE Full text] [CrossRef] [Medline]
  68. Kovács LN, Takacs ZK, Tóth Z, Simon E, Schmelowszky Á, Kökönyei G. Rumination in major depressive and bipolar disorder - a meta-analysis. J Affect Disord 2020 Nov 01;276:1131-1141 [FREE Full text] [CrossRef] [Medline]
  69. Starr LR, Davila J. Excessive reassurance seeking, depression, and interpersonal rejection: a meta-analytic review. J Abnorm Psychol 2008 Nov;117(4):762-775. [CrossRef] [Medline]
  70. McLaughlin KA, Nolen-Hoeksema S. Rumination as a transdiagnostic factor in depression and anxiety. Behav Res Ther 2011 Mar;49(3):186-193 [FREE Full text] [CrossRef] [Medline]
  71. Hatzenbuehler ML, McLaughlin KA, Nolen-Hoeksema S. Emotion regulation and internalizing symptoms in a longitudinal study of sexual minority and heterosexual adolescents. J Child Psychol Psychiatry 2008 Dec;49(12):1270-1278 [FREE Full text] [CrossRef] [Medline]
  72. Cludius B, Mennin D, Ehring T. Emotion regulation as a transdiagnostic process. Emotion 2020 Feb;20(1):37-42. [CrossRef] [Medline]


CBT: cognitive behavioral therapy
CFI: comparative fit index
EMA: ecological momentary assessment
GIMME: Group Iterative Multiple Model Estimation
IRB: institutional review board
JITAI: just-in-time adaptive intervention
MCBT: mobile cognitive behavioral therapy
MDD: major depressive disorder
MRFCBT: mobile rumination-focused cognitive behavioral therapy
RCT: randomized controlled trial
REDCap: Research Electronic Data Capture
RFCBT: rumination-focused cognitive behavioral therapy
RMSEA: root mean square error of approximation
SDS: State Depression Scale
SRMR: standardized root mean square residual
uSEM: unified structural equation modeling


Edited by A Mavragani; submitted 13.02.22; peer-reviewed by H Kim, S Goldberg; comments to author 14.04.22; revised version received 20.06.22; accepted 24.10.22; published 05.01.23

Copyright

©Liyuan Wang, Lynn Miller. Originally published in JMIR Formative Research (https://formative.jmir.org), 05.01.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.