This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Social media–delivered lifestyle interventions have shown promising outcomes, often generating modest but significant weight loss. Participant engagement appears to be an important predictor of weight loss outcomes; however, engagement generally declines over time and is highly variable both within and across studies. Research on factors that influence participant engagement remains scant in the context of social media–delivered lifestyle interventions.
This study aimed to identify predictors of participant engagement from the content generated during a social media–delivered lifestyle intervention, including characteristics of the posts, the conversation that followed the post, and participants’ previous engagement patterns.
We performed secondary analyses using data from a pilot randomized trial that delivered 2 lifestyle interventions via Facebook. We analyzed 80 participants’ engagement data over a 16-week intervention period and linked them to predictors, including characteristics of the posts, conversations that followed the post, and participants’ previous engagement, using a mixed-effects model. We also performed machine learning–based classification to confirm the importance of the significant predictors previously identified and explore how well these measures can predict whether participants will engage with a specific post.
The probability of participants’ engagement with each post decreased by 0.28% each week (
Findings revealed several predictors of engagement derived from the content generated by interventionists and other participants. Results have implications for increasing engagement in asynchronous, remotely delivered lifestyle interventions, which could improve outcomes. Our results also point to the potential of data science and natural language processing to analyze microlevel conversational data and identify factors influencing participant engagement. Future studies should validate these results in larger trials.
ClinicalTrials.gov NCT02656680; https://clinicaltrials.gov/ct2/show/NCT02656680
Obesity is prevalent in the United States and is a known risk factor for cardiovascular disease, type 2 diabetes [
Systematic reviews and meta-analyses show support for the efficacy of social media–delivered lifestyle interventions [
A promising approach to increase our understanding of the factors influencing participants’ engagement in social media–based behavioral interventions is to study the content and interactions generated by the interventionists and participants during the intervention using natural language processing (NLP). Data collected directly from web-based platforms (eg, Facebook) can provide detailed, real-time behavioral information over the course of intervention programs. NLP can handle a large quantity of text, generate reliable qualitative coding [
Using data from a 16-week pilot feasibility randomized weight loss trial that delivered lifestyle interventions via Facebook, drawing on multilevel factors influencing participant engagement identified by previous web-based communication literature, we derived various factors from the content generated by participants and interventionists over the course of the intervention, including characteristics of the posts (eg, poster, time, and topic), conversations that followed the post (eg, sentiment and receiving replies), and participants’ previous engagement behaviors, and assessed how well these factors predict participant engagement individually and all together in the context of a social media–delivered lifestyle intervention.
In a pilot feasibility randomized trial, we randomized 80 participants who were overweight or obese into 1 of 2 remotely delivered lifestyle interventions. We recruited people interested in losing weight via web-based advertisements at the University of Connecticut on ResearchMatch and in yard sale or neighborhood Facebook groups in 37 states across the United States between June and October of 2019. Inclusion criteria included BMI between 27 and 45 kg/m², smartphone ownership, active Facebook user (ie, comments or posts more than once a week), aged 18 to 65 years, and having daily internet access. Exclusion criteria included pregnancy or planning to become pregnant during the study, bariatric surgery or plans for bariatric surgery during the study, ≥5% weight loss in the past 3 months, pre-existing conditions that preclude physical activity or dietary changes, taking medications affecting weight, incapable of walking one-fourth of a mile unaided without stopping, type 1 or type 2 diabetes, and participation in prior weight loss studies under the principal investigator.
Participants completed an orientation webinar before randomization to learn more about the study, and those still interested in participating were mailed a Wi-Fi scale (FitBit Aria, FitBit Inc) and asked to provide the staff with their log-in information for the scale so that weights could be recorded for the assessments. We randomized 80 participants to the 2 conditions.
Participants were randomized to either a Facebook group in which new participants were continually enrolled during weeks 1 to 8 (open enrollment) or a Facebook group that included only the original 40 randomized participants (closed enrollment). In the open enrollment condition, 54 additional participants were enrolled between weeks 1 to 8 for a final group size of 94. However, we only included the original 80 randomized participants in this study to ensure all participants had an equal amount of time to engage in all 16 weeks of the intervention.
Both conditions received the identical 16-week lifestyle intervention based on the Diabetes Prevention Program (DPP) but modified to be delivered in a private Facebook group where twice-daily posts guided participants through the program, which was led by a dietitian (counselor) who was assisted by a student counselor. We adapted the DPP content to be appropriate for a web-based setting, as described elsewhere [
This pilot feasibility randomized trial was approved by the University of Connecticut Institutional Review Board (H17-215) in October 2017.
We included all posts and comments or replies within posts from interventionists and randomized and nonrandomized participants to construct the measures. Posts without text (approximately 6% of the posts were excluded) and polls were excluded, which resulted in 761 posts and 9396 comments or replies across the 2 intervention arms.
The outcome of interest was on the postparticipant pair level; that is, whether each participant had engaged with (ie, commented or replied to) a post in the Facebook group (1 if yes and 0 if no). Comments are in response to the original post, whereas replies are responses to comments made by others on a post. We focused on comments and replies as these activities are active forms of engagement rather than passive types of engagement such as views and reactions (eg, “likes”) and have been shown to positively predict weight loss [
The post characteristics described in
We used a binary variable to indicate whether the focal post was created by the interventionist (1) or participant (0).
We measured the number of words in each post.
We measured the average sentiment (text polarity) of each post’s content. Text polarity measures the valence and emotion in the text and ranges on a continuous spectrum from negative (lower value) to positive (higher value). We standardized the measure of sentiment for the analysis. Sentiment analysis was performed using the
We used natural language processing to identify the topics that appeared in each post, comment, and reply. The content was preprocessed to remove emojis and non-English characters. Topics were detected using
We collected the time (number of days from day 1 of the intervention) and day of the week when the post was created.
We constructed a series of variables representing the characteristics of replies or comments on each post. To reflect the content of conversations before each participant’s engagement, if the participant engaged with the post, we calculated these variables based on all previous comments or replies under the post before their engagement for each unique postparticipant pair; if the participant did not engage with the post, we calculated these measures based on all the comments or replies under the post. The characteristics are described in
We created two binary variables to represent whether each participant had been tagged or mentioned by (1) interventionists or (2) other participants in the previous replies or comments within the same post. It is worth mentioning that most tags or mentions in our data were generated automatically by Facebook (eg, when participant A comments or replies to participant B’s content, Facebook automatically generates a tag on B in A’s reply or comment). Thus, most tags or mentions in our data represent reply or comment relationships. In very few instances, interventionists deliberately tagged previously disengaged participants; however, the sample size was too small to test their effects separately.
We measured the average sentiment of all replies or comments for each postparticipant pair. The measure was standardized for the analysis.
The included participant characteristics were as follows:
Percentage of previous posts commented or replied: For each post, we calculated the percentage of previous posts each participant has commented or replied to.
Baseline and sociodemographic characteristics: Although these variables were not the focus of our analysis, we included baseline characteristics for each participant, including treatment condition (open vs closed), baseline weight, BMI, age, race, sex, education, marital status, number of people in the household, and employment status, as covariates in the analyses.
We focused our analysis on whether each randomized participant (N=80) had engaged with each post, as randomized participants had access to the Facebook group the entire length of the intervention (it should be noted that each post was only available in a particular treatment arm and, thus, can only be seen by 40 randomized participants). To examine what predicts participant engagement with each post, analyses were performed on the postparticipant pair level (ie, whether each participant engaged with each post). This allowed us to construct measures that accurately reflect the content (ie, posts and conversations) before each participant’s engagement. We included all possible engagements (ie, instances where participants engaged and instances where they did not engage) from the 80 randomized participants with each of the 761 posts, which resulted in a final sample of 31,968 observations (participants engaged in 4462 instances and did not engage in 27,506 instances) for our analysis.
The overall analysis framework is depicted in
Analysis framework to identify important predictors of participant engagement. Left panel: an example of the intervention post and the comments or replies following it. Right panel: flow chart of the analysis. NLP: natural language processing.
Although regression analyses are useful to identify the statistical significance of linear relationships, some of the relationships might be much more complex (eg, nonlinear or moderated by other variables). To confirm the importance of significant predictors that we previously identified and to investigate how well these variables as a whole can predict participants’ engagement with a particular post, we included all aforementioned predictors in machine learning algorithms, including gradient boosting machines, deep learning models, and an ensemble of them [
Participant characteristics (N=80).
Participant characteristics | Closed enrollment (n=40) | Open enrollment (n=40) | |
Age (years), mean (SD) | 40.4 (11.8) | 40.0 (10.6) | |
Female, n (%) | 34 (85) | 34 (85) | |
Baseline BMI (kg/m2), mean (SD) | 34.8 (5.4) | 34.0 (4.6) | |
Hispanic or Latino, n (%) | 3 (8) | 1 (3) | |
|
|||
|
White | 36 (90) | 36 (90) |
|
Black or African American | 3 (8) | 3 (8) |
|
Asian | 0 (0) | 0 (0) |
|
Native Hawaiian or other Pacific Islander | 0 (0) | 0 (0) |
|
American Indian or Alaska Native | 0 (0) | 0 (0) |
|
Multiethnic | 0 (0) | 1 (3) |
|
Unknown | 1 (3) | 0 (0) |
|
|||
|
Married or living with partner but not married | 29 (73) | 30 (75) |
|
Single | 8 (20) | 6 (15) |
|
Widowed, divorced, or separated | 3 (8) | 4 (10) |
|
|||
|
Less than high school, high school degree, GEDa, equivalent | 1 (3) | 2 (5) |
|
Trade, technical, some college, associates | 8 (20) | 11 (28) |
|
Bachelor’s degree or some graduate school | 21 (53) | 17 (43) |
|
Graduate degree | 10 (25) | 10 (25) |
|
|||
|
Employed full-time | 28 (70) | 27 (68) |
|
Employed part-time | 7 (18) | 4 (10) |
|
Student | 2 (5) | 2 (5) |
|
Unemployed, retired, disabled, or homemaker | 3 (8) | 6 (15) |
aGED: General Educational Development.
Post and reply or comment characteristics over the 16-week intervention.
Post characteristics | Closed enrollment (n=374) | Open enrollment (n=387) | |
Content sentiment, mean (SD) | 0.134 (0.197) | 0.133 (0.195) | |
Number of words, mean (SD) | 33.78 (24.63) | 33.28 (23.12) | |
Created by interventionists, n (%) | 225 (60.2) | 211 (54.5) | |
|
|||
|
Exercise | 80 (21.4) | 83 (21.4) |
|
Diet | 158 (42.2) | 152 (39.3) |
|
Weight | 64 (17.1) | 74 (19.1) |
|
MyFitnessPal app | 61 (16.3) | 67 (17.3) |
|
Expressing emotion | 28 (7.5) | 28 (7.2) |
|
Sleep | 6 (1.6) | 5 (1.3) |
|
Goals or plans | 80 (21.4) | 72 (18.6) |
|
|||
|
Content sentiment, mean (SD) | 0.171 (0.255) | 0.156 (0.234) |
|
Participant’s reply to other participants, n (%) | 750 (23.8) | 803 (12.9) |
|
Interventionist reply to a participant, n (%) | 1018 (32.3) | 1195 (19.1) |
aClosed enrollment n=3152 and open enrollment n=6244.
To confirm the importance of previously identified predictors and test how well the aforementioned variables can predict the probability of a participant engaging with a post, we performed a variety of machine learning–based classification algorithms with all the aforementioned predictors as the input and participant engagement as the outcome. Of the 32 models we tested, the ensemble approach of gradient boosting machine learning–based and deep learning–based classification algorithms performed the best, with an average area under the curve of 0.963 using 5-fold cross-validation (see more results in
Mixed-effects regression results predicting participants’ engagement (N=31,968)a.
Mixed-effects regression | Values, mean (SD; range) | Coefficient (95% CI) | |||
Outcome: participants’ engagement | 0.140 (0.347; 0 to 1) | —b | — | ||
|
|||||
|
Created by interventionists | 0.583 (0.493; 0 to 1) | 0.0627 (0.0507 to 0.0746) | <.001 | |
|
Number of words | 33.44 (23.80; 1 to 107) | 0.0005 (0.0003 to 0.0008) | <.001 | |
|
Content sentiment (standardized) | 0 (1; −3.67 to 4.74) | −.0042 (−0.097 to 0.0012) | .13 | |
|
|
||||
|
|
Exercise | 0.212 (0.409; 0 to 1) | 0.0096 (−0.0075 to 0.0266) | .27 |
|
|
Diet | 0.389 (0.487; 0 to 1) | −0.0085 (−0.0249 to 0.0078) | .31 |
|
|
Weight | 0.191 (0.392; 0 to 1) | 0.0654 (0.0494 to 0.0814) | <.001 |
|
|
MyFitnessPal app | 0.163 (0.369; 0 to 1) | −0.0377 (−0.0534 to −0.0219) | <.001 |
|
|
Expressing emotion | 0.071 (0.257; 0 to 1) | 0.0083 (−0.01558 to 0.0321) | .50 |
|
|
Sleep | 0.0141 (0.117; 0 to 1) | −0.0587 (−0.1070 to −0.0103) | .02 |
|
|
Goals or plans | 0.209 (0.407; 0 to 1) | 0.0612 (0.0414 to 0.0811) | <.001 |
|
Time of the post | 46.61 (32.94; 1 to 112) | −0.0004 (−0.0006 to −0.0002) | <.001 | |
|
|||||
|
Content sentiment (standardized) | 0 (1; −7.84 to 5.91) | −0.0539 (−0.0589 to −0.0488) | <.001 | |
|
Replied by other participants | 0.026 (0.161; 0 to 1) | 0.4484 (0.4279 to 0.4690) | <.001 | |
|
Replied by interventionists | 0.029 (0.167; 0 to 1) | 0.4604 (0.4409 to 0.4798) | <.001 | |
|
|||||
|
Percentage previous posts commented or replied | 13.16 (13.68; 0 to 100) | 0.0076 (0.0074 to 0.0079) | <.001 |
aThe model included postlevel random effects and also controlled for day of the week when the post was created and other baseline and sociodemographic characteristics of the participants, including treatment assignment, race, marital status, education, employment, number of people in the household, age, gender, baseline BMI, and weight.
bNot available.
Variable importance of predicting participant engagement across 20 machine learning models. The x-axis shows different model names, and variables from top to bottom on the y-axis are baseline weight, number of people in the household, baseline BMI, age, post topic weight, post topic goal or plan, education, treatment assignment, post topic diet, post topic MyFitnessPal app, post topic exercise, employment status, marital status, post topic drink, post topic sleep, gender, race, post topic expressing emotion, post sentiment, whether the post is created by interventionists, day of the intervention when the post is created, word count of the post, replies or comments sentiment, day of the week when the post is created, percentage of previous posts engaged, and whether replied by other participants or by interventionists.
In this study, we conducted secondary analyses using data from a 2-arm pilot feasibility randomized controlled trial that delivered lifestyle interventions via Facebook. We analyzed commenting or replying behavior from 80 participants in response to each of the 761 posts generated by counselors and participants over the 16-week intervention period and linked them to predictors, including post characteristics (eg, time, post length, and post topic), conversation characteristics (eg, sentiment of the conversation and participants being replied to), and participant characteristics (eg, sociodemographics and previous commenting or replying behavior). Our findings suggest that although participants’ comments or replies decreased over time, important characteristics of the post, the conversation attached to that post, and the participants’ engagement patterns predicted whether a participant engaged with a specific post. For example, we found that participants who engaged more with prior posts were more likely to engage with future posts. Posts that were longer (with the maximum number of words not exceeding 107), were created by interventionists, or had content related to weight (eg, weigh-in posts) and goal setting are more likely to attract engagement. The latter is consistent with the design of the intervention—participants were asked to set diet and exercise goals for the week each Monday, report their progress on their goals on Sunday, and report their weight change for the week each Friday. This is also encouraging because goal setting [
In this study, we demonstrate the potential of using NLP tools to analyze microlevel conversational data and identify factors influencing participants’ commenting or replying behavior in a social media–delivered weight loss intervention. Our findings shed light on some important microlevel characteristics of the participants, posts, and conversations, which can shape participants’ experiences during the intervention and predict their future engagement. These results have implications for the design and implementation of social media–delivered behavioral interventions in ways that maximize participant engagement. We previously reported a strong association between participant engagement and weight loss [
Although many studies have tested social media–delivered weight loss interventions or emphasized the importance of participant engagement in web-based communities [
Similar to previous studies, we found that participant engagement is highly variable [
This study has several limitations that point to avenues for future research. First, our sample size was small (80 participants; 10,157 total posts, comments, and replies) and our participants were predominantly White (72/80, 90%) and female (68/80, 85%). This limits the generalizability of our results, following a long-standing pattern in weight loss studies of difficulty recruiting male participants [
In this study, we performed secondary analyses using data from a pilot feasibility randomized weight loss trial that delivered a lifestyle intervention via Facebook and linked participants’ engagement with several important predictors, including characteristics of the posts, replies or comments, and participants. Our results point to the potential of using data science and NLP tools to analyze microlevel behavioral or conversational data and identify factors influencing participants’ engagement during the social media weight loss intervention, which have implications for the design and implementation of future interventions that could lead to more favorable weight loss outcomes. Future studies are warranted to validate our results and further explore these relationships in similar and larger trials.
Supplementary analyses and results.
Diabetes Prevention Program
natural language processing
This project was supported by the National Institutes of Health grant K24HL124366 (principal investigator: SP).
SP has been a paid advisor for WW (formerly Weight Watchers) and FitBit.