Smartphone-Based Survey and Message Compliance in Adults Initially Unready to Quit Smoking: Secondary Analysis of a Randomized Controlled Trial

Introduction

Overview

The phase-based model (PBM) argues that smoking cessation intervention messages should be customized to each phase of readiness to quit smoking (ie, motivation, precessation, cessation, and maintenance), thereby meeting the individual where they are and providing targeted and relevant intervention programming [1]. Application of dynamic models, like the PBM, to mobile health (mHealth) interventions for smoking cessation may be especially relevant since 80% of smokers are not ready to quit smoking within the next 30 days [2]. Yet, people who smoke may rapidly cycle through decisions to initiate, continue, and end smoking cessation attempts [2]. Smartphone-based surveys are increasingly being used to identify predictors of health behaviors (eg, smoking and alcohol use) [3-5] and inform mHealth intervention messaging via applied theoretical frameworks like PBM [6-9].

The efficacy of smartphone-based interventions depends not only on adequate intervention content but also on the level and duration of exposure to that content. For instance, SMS text messaging–based smoking cessation interventions have demonstrated efficacy in increasing quit rates, with one meta-analysis showing a 36% higher quit rate compared with control groups [10]. In addition, a systematic review found that SMS text messaging–based support significantly increased biochemically verified continuous abstinence and 7-day point prevalence abstinence rates [11]. Duration of exposure to intervention content matters. For example, Bricker et al [12] found that the longer individuals used a smoking cessation app, the higher the odds of 12-month abstinence. Notably, compared to those who used the app for only 1 week, those who used the app for at least 4 weeks had 50% higher odds of cessation, and those who used the app for 26 weeks had 397% higher odds of cessation [12]. Despite the demonstrated efficacy of these interventions, there is limited research directly examining predictors of compliance with survey completion and viewing of app-based intervention messages in longer-term smoking cessation interventions.

Predictors of Survey Compliance

Prior research has identified a general trend of declining mHealth intervention engagement over time. For instance, as study durations surpass 7 days, smartphone-based survey completion rates tend to decline [13-15]. Ottenstein and Werner’s [15] meta-analysis of 488 smartphone survey studies demonstrated lower overall survey compliance with longer study periods and higher total surveys prompted. However, the average duration of these studies was only 16.5 days with the median being only 8 days [15]. Less is known about the extent to which longer study periods (≥1 month) relate to declines in interactions with mHealth apps, especially for interventions that last 6 months or longer. Some studies conducted over shorter periods have demonstrated increased compliance with smartphone-based surveys among women versus men [14,16], older adults versus younger adults [13,14,17], and those without versus those with mental health diagnoses [14,18,19]. Yet, longer studies (≥1 month) have identified fewer differences in survey compliance across participant characteristics [18,20,21]. Other potential predictors of smartphone-based survey compliance include race or ethnicity [22-24], education [23-25], and employment status [26]. Municipality (ie, urban or rural), to our knowledge, has not been used to predict smartphone-based survey compliance. Additional research is needed to identify factors that predict smartphone-based survey completion and message viewing for extended study periods (eg, 26 weeks) and how participant characteristics impact the number of interactions with mHealth interventions over time.

Purpose of This Study

The purpose of this study was to identify correlates of weekly smartphone-based survey completion and viewing of twice daily prompted messages. This aim is relevant since messages, which were tailored to current readiness to quit smoking, were given to a sample of people who were initially not ready to quit smoking in the next 30 days. Based on previous research, we hypothesized that aggregate survey completion and message viewing would be higher in women, nonminoritized adults, those without a history of a mental health diagnosis, and older adults. We also hypothesized that message compliance would be higher in the intervention versus control groups. Finally, we explored whether participant characteristics were related to the rate of reductions in weekly survey completion and message viewing over the 26-week study period. Ultimately, these findings may be used to improve the overall efficacy of smartphone-based interventions.

Methods

Participants and Procedure

This study is a secondary analysis of data collected for a randomized controlled trial of 152 adults who were initially not ready to quit smoking cigarettes in the next 30 days. Eligibility requirements included (1) being at 18 years or older, (2) demonstrating >6th grade English literacy level [27], (3) owning an active Android smartphone with an operating system version of 5.2 or higher and a data plan, (4) agreeing to download the Insight study smartphone app (University of Oklahoma Health Sciences Center) onto their phone, (5) agreeing to complete 26 weekly assessments using the study smartphone app, (6) scoring >1 and <7 on the Readiness to Quit Ladder (ie, no plans to stop smoking within 30 days) [28], (7) proving smoking status by either having an expired carbon monoxide level >7 parts per million (in-person baselines) or texting a picture of their pack of cigarettes to study staff immediately when requested during the enrollment phone call, (8) smoking ≥5 cigarettes every day, and (9) having no contraindications to over the counter nicotine replacement therapy (NRT).

Participants were randomized into one of 3 groups: Phoenix only, Phoenix+NRT, or Factoid. They were prompted to complete 1 weekly smartphone-based survey on Saturday mornings that included questions focused on smoking, other health behaviors, and readiness to quit smoking. The app had 4 distinct study phases and participants could move between phases at will. Participants started the study in the motivation phase and remained in that phase until they used the app to set a smoking quit date. When or if a quit date was set, the app moved to the precessation phase and all participants gained access to smoking cessation resources. Participants automatically moved to the cessation phase when the selected quit date arrived and moved to the maintenance phase when desired. Intervention content and resources were specific based on group assignments. All participants received daily messages. Examples of messages are provided in Table S1 in Multimedia Appendix 1. To alert participants of a new message, smartphones rang and vibrated. In addition, a pop-up box appeared on the smartphone home screen asking participants if they would like to review the study message. If participants clicked “yes,” the app automatically opened and displayed the message. If participants ignored the message or clicked “no,” the message was recorded as unread. A total of 2 daily messages (ie, one 30 minutes after waking and one 60 minutes before bedtime) were prompted during the motivation, precessation, and maintenance phases, and up to 5 messages were prompted per day during the cessation phase (ie, messages were presented at the end of cessation phase surveys). Participants could remain in the intensive cessation phase as long as desired. Those assigned to the Phoenix or Phoenix+NRT groups received messages that were tailored to their current cessation phase. Factoid participants received factual or trivia messages that were not related to smoking cessation across all study phases. Days spent in the cessation phase were excluded from the current analyses.

Ethical Considerations

All study procedures were approved by the University of Oklahoma Health Sciences Center’s institutional review board (IRB#8814). Written informed consent was obtained either in-person or electronically via REDCap (Vanderbilt University) before study enrollment. Each participant was assigned a unique ID number that was linked to their data. Personally identifying information was stored on encrypted servers and accessible only to research personnel. Written signatures were kept in locked and secure filing cabinets behind a locked door. Participants consented to secondary analysis of data collected in this study. Participants were paid US $30 for completing the baseline survey, US $25 for completing the 26-week follow-up, and those who were abstinent at 25 weeks could earn an additional US $20 for providing a carbon monoxide sample. Study participants were compensated with US $5 for each weekly survey they completed throughout the 26-week study. Up to US $185-$205 could be earned by each participant. All payments were made via Greenphire gift cards.

Measures

Baseline Measures

The baseline survey assessed demographic characteristics including age (years), biological sex (male=1, female=0), education (no formal schooling=0, postgraduate degree=20), employment status (employed=1, unemployed=0), race or ethnicity (non-Hispanic White=1, minoritized race=0), and zip codes used to indicate municipality (rural-urban commuting area codes 4-10=rural, rural-urban commuting area codes 1-3=urban) [29]. All participants were asked if they were ever diagnosed with major depressive disorder, bipolar disorder, schizophrenia or schizoaffective disorder, posttraumatic stress disorder, or any other anxiety disorder. A mental health disorder history variable was created whereby those that endorsed a history of any previous mental health diagnosis were coded as 1 and those without such diagnoses were coded as 0.

Weekly Survey Measures

The weekly survey consisted of 32 questions, which covered topics such as daily cigarette consumption, quit attempts, readiness to quit smoking, use of smoking cessation resources, lifestyle factors like exercise, diet, and sleep, and personal impacts of the COVID-19 pandemic.

Aggregating Survey Completion and Message Viewing

Aggregated person-level survey completion rates were calculated using binary variables for each survey or week (range 1-26 surveys). Incomplete surveys were coded as 0. Second, we calculated aggregated person-level message viewing rates across all study days for the morning (range 1-182), evening (range 1-182), and total messages viewed (range 1-364). Messages that were not opened were coded as 0.

Analytic Plan

Descriptive statistics, including means and standard deviations, were calculated for continuous variables, and frequency and percentages were calculated for categorical variables. For each predictor, we fit 4 models. First, we used linear regression to examine whether each predictor was associated with person-level aggregate weekly survey compliance. Second, we used linear regression to examine whether each predictor was associated with aggregate morning, evening, and total message viewing compliance. Third, we used generalized linear mixed-effects models to examine whether any of the demographic, mental health, or study group variables moderated the association between weeks in the study (ie, up to 26) and survey completion (binary variable). Fourth, we used linear mixed-effects models to examine whether any of the demographic, mental health, or study group variables moderated the association between weeks in the study (ie, up to 26) and weekly aggregated message viewing (ie, up to 14 messages per week). For both mixed-effects models (survey completion and message viewing), if significant interactions arose, we modeled those associations separately for each predictor level (eg, male, female). Survey and message compliance data collected during the cessation phase were excluded from analyses because participants who reached the intensive cessation phase (ie, 5 messages per day) could end it at any time. All analyses were performed using SAS (version 9.4; SAS Institute Inc).

Results

Participant Characteristics

On average, participants were aged 50 (SD 12.5, range 19-75) years and had 13.3 (SD 1.6, range 10-20) years of education. Most participants were female (103/152, 67.8%), identified as White (113/152, 74.3%), and lived in urban areas (117/152, 77%). Over half of the participants were unemployed (80/152, 52.6%) and reported having a history of mental illness (93/152, 61.2%). Participants completed 71.8% (2782/3872) of the prompted weekly surveys and viewed 60.6% of the prompted daily messages (31,503/52,460 total). Table 1 provides descriptive statistics for compliance rates and the demographic and mental health diagnoses relevant to this study. On average, 25.5 (SD 1.8) out of the 26 scheduled weekly surveys during the study period were included in the analyses since they were successfully prompted through the Insight app. Of these included weekly surveys, an average of 18.30 (SD 6.8) surveys were completed by participants. When morning and evening message compliance were aggregated, an average of 345.1 (SD 57.5) messages were presented across the 26 study weeks, of which an average total of 207.3 (SD 96.2) messages were viewed per participant.

Weekly survey completion and daily message viewing decreased across the 26-week study period (all P values<.001). For the first week of the study, participants completed 77% (114/148) of prompted surveys and viewed 72.3% (1533/2120) of total messages. In the 26th week of the study, survey completion declined to 56% (84/150), and total messages viewed declined to 44.6% (868/1946). The rate of this decline was 1.1% for morning, evening, and total messages viewed per week, which equates to a 27.7% decrease from the start to the end of the 26-week study. In the following sections, we present associations between each predictor variable and (1) person-level aggregate survey compliance; (2) change in weekly survey completion across the 26-week study period; (3) person-level aggregate morning, evening, and total message compliance; and (4) change in weekly compliance for morning, evening, and total messages viewed across the study period.

Predictors and Moderators of Aggregated Survey Compliance

Age was significantly associated with the aggregated weekly survey completion rate (P=.003), with an expected 0.5% greater survey compliance for each year increase in age. Notably, age, sex, ethnicity, employment status, and municipality moderated the association between weeks in the study and weekly survey completion. Slower declines were observed in older adults, women, Hispanic/Latino participants, those with rural residence, and those unemployed. Predictors of weekly survey compliance across time are presented in Table 2.

Predictors and Moderators of Aggregate Message Viewing

Age was positively associated with aggregate viewing of morning (β=0.005, P=.007) and total (β=0.004, P=.02), but not evening, messages. Age, sex, municipality, employment status, and education significantly moderated the association between weeks in the study and morning and total messages viewed. Slower declines in morning and total message viewing were observed for older adults, female participants, rural residents, and among the unemployed and higher educated. For evening message viewing compliance, only age, sex, and education moderated this relationship. Significant moderators of the decline in message viewing are shown in Table 3.

Discussion

Principal Findings

This study contributes to a growing body of literature that aims to identify predictors of smartphone-based survey completion and interaction with or viewing of mHealth intervention messages over long study periods. Study findings indicated that 71.8% (2782/3872) of weekly surveys were completed and 60.6% (31,503/52,460) of daily messages were viewed by participants over 26 weeks. Results showed that weekly survey compliance and message viewing declined over time, and several variables moderated the rate of decline in weekly survey compliance (ie, age, sex, ethnicity, municipality, employment status) and message viewing (ie, age, sex, municipality, employment status, and years of education). Numerous prior studies have examined predictors of compliance to smartphone-based surveys [3,13,14,18,20,21], and some have identified variables related to compliance over time [30,31]. However, this study is the first to simultaneously assess predictors of both weekly survey completion and viewing of prompted daily messages, and the first to do so over such a long study period (ie, 26 weeks). Findings from this study provide insights into demographic subgroups that should be monitored and addressed to maintain high engagement with mHealth interventions focused on prompting and supporting health behavior change.

Implications of Variables Related to Weekly Survey Completion

Age was the only measured variable that was significantly related to person-level aggregate survey completion. Our findings are consistent with Comulada et al [17], which examined adherence to self-monitoring of healthy lifestyle behaviors through smartphone-based surveys and photographic food records over 26 weeks. Comulada et al [17] found that age, but not other demographic characteristics, was significantly related to overall survey completion. Furthermore, this study is consistent with other smartphone-based studies with durations ≥1 month that failed to detect differences in compliance across non–age-related demographic variables [18,20,21]. Although aggregate survey completion did not vary across predictor variables (except for age), several variables moderated the rate of decline in weekly survey completion over the 6-month study period. Specifically, slower declines in survey compliance were observed for female versus male participants, non-Hispanic versus Hispanic participants, older versus younger participants, unemployed versus employed participants, and rural versus urban residents. These findings indicate that multiple demographic factors may play important roles in the reduction of engagement with weekly surveys.

Implications of Variables Related to the Decline in Message Viewing

Similar to survey completion, findings indicated that age was the sole predictor of person-level aggregate message viewing over the 26-week study period. Furthermore, the rate of decline in daily message viewing varied across several demographic factors. Specifically, the reduction in message viewing over 26 weeks was less rapid for female versus male participants (ie, morning and evening messages), those with rural versus urban residency (ie, morning messages), unemployed versus employed individuals (ie, morning messages), and those with more years of education (ie, morning and evening messages). Engagement with intervention content may be tied to intervention efficacy [12,32]. Therefore, targeted support and strategies that aim to boost engagement with smartphone-based interventions may be needed for individuals in subgroups that displayed a greater rate of decline in message viewing, especially for morning messages.

Strengths and Weaknesses

This study has multiple strengths. While the majority of previous smartphone-based survey studies have examined relatively brief study periods (eg, 7-30 days [13,14,18,20,21]), our findings are among the first to assess smartphone-based survey completion rates and message viewing over a 26-week study period. Despite the extended duration of this study, participants completed 71.8% (2782/3872) of prompted weekly surveys and viewed 60.6% (31,503/52,460) of prompted twice daily messages. Notably, message viewing was not incentivized, which may suggest that participants found the message content engaging. These findings suggest that people who smoke and are not initially ready to quit may engage with a low-burden smartphone app for relatively long periods.

This study also has some notable weaknesses. First, participants were only prompted to complete 1 survey each week. This study design decision was purposefully made to determine the feasibility of this type of low-burden, long-duration mHealth intervention that aimed to motivate and support behavior change in adults who were not initially ready to quit smoking. Thus, this study does not necessarily offer insights about whether more intensive smartphone survey protocols (ie, multiple surveys per day) are acceptable for long-duration mHealth interventions. Second, weekly survey completion was incentivized (ie, US $5 per weekly survey completed). Future research should investigate if observed patterns emerge in real-world nonincentivized settings. Third, Phoenix participants who never set a quit date could have received duplicate messages after 76 days, whereas those who set quit dates were less likely to receive duplicate messages. Factoid participants did not receive duplicate messages. Receiving duplicate messages could have reduced message viewing as participants may not have liked repeated messages. Fourth, the sample was relatively small and was mostly middle-aged, White female individuals from urban areas in Oklahoma and Texas. These findings may not generalize to other groups of people who smoke and those who are ready to quit smoking.

Conclusions and Future Directions

Overall, this 26-week smoking cessation induction trial showed robust participant engagement with the intervention and weekly prompted surveys and revealed that only age directly influenced overall survey and message engagement over the extended study period. Further, we identified several demographic variables that significantly moderated the decline in weekly survey completion and message viewing, offering insights that could be useful for future personalized intervention development. Future research should determine if these findings replicate in larger trials and whether efforts to increase engagement with smartphone-based health behavior change interventions are fruitful over time. Future interventions could include additional prompts and reminders to complete weekly surveys and read intervention messages for certain subgroups. Further, researchers should investigate if those who are more compliant are more likely to quit smoking by examining how survey completion and message viewing are associated with intervention effectiveness.