Obesity is prevalent in the United States and is a known risk factor for cardiovascular disease, type 2 diabetes [1,2], and cancer. Although lifestyle interventions are effective for weight loss and diabetes prevention [3], they require numerous clinic visits for up to a year, which is burdensome for many people. Technology-delivered lifestyle interventions, by not requiring visits, are less burdensome for participants and have shown promising weight loss outcomes [4]. Some technology-based interventions use popular commercial social media platforms such as Facebook in an effort to meet people where they are [5,6]. Many social media users already use these platforms to discuss their health experiences [7,8]. Community-building features on social media platforms, such as private groups [9,10], make them particularly amenable to delivering group-based lifestyle interventions.

Systematic reviews and meta-analyses show support for the efficacy of social media–delivered lifestyle interventions [4,11]; however, this area of research is still nascent. Participant engagement, defined as posts in the group, replies to posts, “likes,” and votes in polls, appears to be an important predictor of outcomes [12-15]. For example, a study found that every 10 posts by participants corresponded to −0.5% weight loss [16]. Another study found only certain types of engagement predicted weight loss [17]. Interestingly, the degree of participant engagement reported in studies of social media–delivered interventions is highly variable, ranging from an average of once during the entire intervention period to 11 times per week [17-19]. Engagement also generally declines over time in these programs [16]. Our understanding of the factors that influence participant engagement in these interventions is limited. Emerging evidence in the web-based communication literature shows the importance of multilevel factors influencing participant engagement, such as the characteristics of posts (eg, post length and topic and popularity of the poster), characteristics of the conversation thread in response to the post (eg, sentiment and reciprocity behavior), and participant characteristics (eg, motivation and habits) [14,20-27]. However, research is scant in the context of social media–delivered behavioral interventions [28]. Furthermore, characteristics of the conversation thread (ie, other peoples’ replies and comments) preceding each participant’s engagement is often ignored, which could be valuable in terms of providing microlevel contextual data that influences each participant’s decision to engage.

A promising approach to increase our understanding of the factors influencing participants’ engagement in social media–based behavioral interventions is to study the content and interactions generated by the interventionists and participants during the intervention using natural language processing (NLP). Data collected directly from web-based platforms (eg, Facebook) can provide detailed, real-time behavioral information over the course of intervention programs. NLP can handle a large quantity of text, generate reliable qualitative coding [29], be leveraged to derive various real-time microlevel insights concerning the characteristics of the posts (eg, topics) and conversations that followed the posts (eg, sentiment), and understand how they affect participants’ decisions of engagement independently and aggregately. This has potential implications for the design and implementation of future interventions to increase participant engagement, which could lead to more favorable weight loss outcomes.

Using data from a 16-week pilot feasibility randomized weight loss trial that delivered lifestyle interventions via Facebook, drawing on multilevel factors influencing participant engagement identified by previous web-based communication literature, we derived various factors from the content generated by participants and interventionists over the course of the intervention, including characteristics of the posts (eg, poster, time, and topic), conversations that followed the post (eg, sentiment and receiving replies), and participants’ previous engagement behaviors, and assessed how well these factors predict participant engagement individually and all together in the context of a social media–delivered lifestyle intervention.

In a pilot feasibility randomized trial, we randomized 80 participants who were overweight or obese into 1 of 2 remotely delivered lifestyle interventions. We recruited people interested in losing weight via web-based advertisements at the University of Connecticut on ResearchMatch and in yard sale or neighborhood Facebook groups in 37 states across the United States between June and October of 2019. Inclusion criteria included BMI between 27 and 45 kg/m², smartphone ownership, active Facebook user (ie, comments or posts more than once a week), aged 18 to 65 years, and having daily internet access. Exclusion criteria included pregnancy or planning to become pregnant during the study, bariatric surgery or plans for bariatric surgery during the study, ≥5% weight loss in the past 3 months, pre-existing conditions that preclude physical activity or dietary changes, taking medications affecting weight, incapable of walking one-fourth of a mile unaided without stopping, type 1 or type 2 diabetes, and participation in prior weight loss studies under the principal investigator.

Participants completed an orientation webinar before randomization to learn more about the study, and those still interested in participating were mailed a Wi-Fi scale (FitBit Aria, FitBit Inc) and asked to provide the staff with their log-in information for the scale so that weights could be recorded for the assessments. We randomized 80 participants to the 2 conditions.

This pilot feasibility randomized trial was approved by the University of Connecticut Institutional Review Board (H17-215) in October 2017.

We focused our analysis on whether each randomized participant (N=80) had engaged with each post, as randomized participants had access to the Facebook group the entire length of the intervention (it should be noted that each post was only available in a particular treatment arm and, thus, can only be seen by 40 randomized participants). To examine what predicts participant engagement with each post, analyses were performed on the postparticipant pair level (ie, whether each participant engaged with each post). This allowed us to construct measures that accurately reflect the content (ie, posts and conversations) before each participant’s engagement. We included all possible engagements (ie, instances where participants engaged and instances where they did not engage) from the 80 randomized participants with each of the 761 posts, which resulted in a final sample of 31,968 observations (participants engaged in 4462 instances and did not engage in 27,506 instances) for our analysis.

The overall analysis framework is depicted in Figure 1. Data were screened for deviations from assumptions required for the used statistical analyses. We calculated descriptive statistics for the outcome and key independent variables for each treatment condition. To account for the fact that engagement was nested within each post, we performed a mixed-effects regression with postlevel random effects using participants’ engagement as the primary outcome, along with all aforementioned key predictors (ie, characteristics of the post, reply or comment, and the participants’ previous engagement behavior) as independent variables, with participants’ baseline and sociodemographic characteristics as covariates. We also included participant-level fixed effects as an alternative specification to account for possible omitted variable bias. As a robustness check, we also conducted a mixed-effects logistic regression with the same variables (Multimedia Appendix 1). To identify the important predictors of participant engagement, we reported the coefficient, 95% CI, and the associated P value for each predictor. All analyses were performed in STATA standard edition (version 16).

Although regression analyses are useful to identify the statistical significance of linear relationships, some of the relationships might be much more complex (eg, nonlinear or moderated by other variables). To confirm the importance of significant predictors that we previously identified and to investigate how well these variables as a whole can predict participants’ engagement with a particular post, we included all aforementioned predictors in machine learning algorithms, including gradient boosting machines, deep learning models, and an ensemble of them [35,36], and examined the performance of these models by calculating key metrics, including (1) area under the curve (ranging from 0.5 to 1, with 1 being the best) using a 5-fold cross-validation, (2) variable importance across different models, and (3) out-of-sample prediction accuracies on a balanced sample with 1600 observations (800 random observations with engagement and 800 observations without). In the machine learning model, variable importance was determined by calculating the relative influence of each variable: in the tree-based model, it was calculated based on whether that variable was selected or included during the tree-building process and how much it improves the model fit. In other non–tree-based models, it was calculated as the magnitude of the weight or coefficient connecting a specific input or variable to the output [37]. We also evaluated the variable importance using an alternative approach called Shapley Additive Explanations contribution in one of the top performing models, which measures how much the average model prediction would change with and without a specific feature or variable [38], as shown in Multimedia Appendix 1. All the analyses were performed using the h2o package [39] in R (R Foundation for Statistical Computing; version 3.6.1).

In this study, we conducted secondary analyses using data from a 2-arm pilot feasibility randomized controlled trial that delivered lifestyle interventions via Facebook. We analyzed commenting or replying behavior from 80 participants in response to each of the 761 posts generated by counselors and participants over the 16-week intervention period and linked them to predictors, including post characteristics (eg, time, post length, and post topic), conversation characteristics (eg, sentiment of the conversation and participants being replied to), and participant characteristics (eg, sociodemographics and previous commenting or replying behavior). Our findings suggest that although participants’ comments or replies decreased over time, important characteristics of the post, the conversation attached to that post, and the participants’ engagement patterns predicted whether a participant engaged with a specific post. For example, we found that participants who engaged more with prior posts were more likely to engage with future posts. Posts that were longer (with the maximum number of words not exceeding 107), were created by interventionists, or had content related to weight (eg, weigh-in posts) and goal setting are more likely to attract engagement. The latter is consistent with the design of the intervention—participants were asked to set diet and exercise goals for the week each Monday, report their progress on their goals on Sunday, and report their weight change for the week each Friday. This is also encouraging because goal setting [40] and frequent self-weighing [41] are key behavioral weight loss strategies. Furthermore, posts with replies or comments that directly mentioned the focal participant were much more likely to attract subsequent replies from that participant. Moreover, posts with replies or comments that contained negative sentiments were more likely to attract subsequent comments. This is likely because participants who share struggles, problems, and challenges are naturally more negative in sentiment (see Multimedia Appendix 1 for examples of replies or comments with negative sentiments), and such content often attracts support and brainstorming from other participants who may feel called to help when others are struggling. These results were robust to multiple alternative specifications. Machine learning results also show that together, these characteristics can predict participant engagement with a high accuracy of 90.9%.

In this study, we demonstrate the potential of using NLP tools to analyze microlevel conversational data and identify factors influencing participants’ commenting or replying behavior in a social media–delivered weight loss intervention. Our findings shed light on some important microlevel characteristics of the participants, posts, and conversations, which can shape participants’ experiences during the intervention and predict their future engagement. These results have implications for the design and implementation of social media–delivered behavioral interventions in ways that maximize participant engagement. We previously reported a strong association between participant engagement and weight loss [31], which suggests that engagement-enhancing strategies could lead to more favorable outcomes. For example, enhancing engagement early on may help with sustained engagement. Furthermore, we found that receiving replies appears to stimulate further engagement from the participant. This may also be a function of whether a participant shares something substantive about themselves in a comment. For example, if a participant’s comment is “thanks!” the interventionist may just hit the “like” button; however, if a participant’s comment is a question or sharing of a goal, the interventionist and other participants are more likely to reply to continue that conversation. Further research should explore the type of comments (eg, questions, sharing a problem, and setting a goal) that are most likely to elicit replies from others. Program content should be designed in ways that nudge participants to post more often and interact more with each other (eg, use open-ended questions and encourage peer-to-peer support). Although in this study we found that participants were more likely to respond to interventionists, greater peer-to-peer engagement could also facilitate stronger group cohesion, thereby further enhancing participant engagement. Posts in which participants share struggles, problems, and challenges they have encountered during the weight loss process may draw more participants into the conversation, which may generate richer brainstorming and social support, both of which could also enhance group cohesion. These implications could be applicable not only to social media–delivered weight loss interventions but also to other digital health interventions more generally.

Although many studies have tested social media–delivered weight loss interventions or emphasized the importance of participant engagement in web-based communities [12-15,42], only a handful of studies have identified the factors that can influence participant engagement in digital health interventions [43,44], and most of them focused on macrolevel factors such as post type and participants’ characteristics (eg, age and gender) [9,14]. Few studies have examined microlevel factors such as conversation dynamics. Several studies recognized this limitation and called for more research to identify all relevant factors that can predict participant engagement [28,43,45,46]. This study contributes to this line of research in two ways: (1) we demonstrate how content generated by interventionists and participants during the course of a digital intervention can be leveraged and combined with data science and NLP tools to identify microlevel predictors of participant engagement, and (2) we have identified many microlevel factors that influence participant engagement, which, to the best of our knowledge, have not been studied in previous social media–based behavioral interventions. This has practical implications for future intervention designs that can maximize participant engagement.

Similar to previous studies, we found that participant engagement is highly variable [17-19], and it generally declines over time [16,47]. Although many factors identified in this study have not been studied in the social media–based behavioral intervention context, our findings are consistent with psychological and sociological theories, as well as several empirical research on web-based communication. For example, our finding that posts created by interventionists are more popular is consistent with other studies on web-based communities, showing that important users or those with designated roles are more likely to draw responses from other users [26], possibly because of preferential attachment [48]. Our finding that replies or comments with negative sentiments draw more engagement implies that participants are more likely to reply or comment when they see others sharing their struggles and challenges. This could be possibly explained by social support processes, which have substantial empirical support across various web-based settings [22-25,49]. Finally, the importance of being replied to by interventionists and other participants can be explained by the preference for reciprocity [50], which has been found to be an important driver for communication in many other web-based settings [20,21].

This study has several limitations that point to avenues for future research. First, our sample size was small (80 participants; 10,157 total posts, comments, and replies) and our participants were predominantly White (72/80, 90%) and female (68/80, 85%). This limits the generalizability of our results, following a long-standing pattern in weight loss studies of difficulty recruiting male participants [51]. Similarly, given that 96% (77/80) of our participants reported attaining a college degree or advanced degree, we cannot generalize our results to individuals with lower levels of education. Future studies should devise recruitment strategies that attract more male participants and participants with low levels of education to further explore the individual heterogeneity across people from different backgrounds. Second, this study did not fully tease out all possible confounding factors and thus cannot establish causality. For instance, participants who are more successful in losing weight might also be more likely to comment simply because they are paying more attention to the group and have more to say as they are applying the knowledge and strategies they are learning. Future studies should include larger trials, surveys with more longitudinal measures (eg, physical activity and diet tracking, mental health, and other behaviors), and qualitative studies to establish the possible bidirectional and causal relationships between engagement and these factors. Third, we focused on replies and comments and did not explore other types of engagement such as reactions and views of posts and comments. Although comments and replies have been considered more substantive than other engagements (eg, likes and “lurking”), other engagements potentially comprise a substantial proportion of social media use and thus warrant careful consideration in future studies [33]. Similarly, although we included participants’ posts in our analysis, we did not include posts only with images or videos or investigate the factors influencing participants’ decisions to create posts, which is another important form of engagement. In addition, although we included tags or mentions relationships in our study, most tags were automatically generated by Facebook during replies or comments. Future studies should consider whether deliberate tagging can nudge disengaged participants to re-engage with the program. Fourth, we assumed all participants had an opportunity to engage with every post in the group, and we considered all replies or comments when constructing the predictors for participants who did not engage with a certain post, which might not necessarily be the case if the participant did not see the post or the previous comments or replies. Future studies could take additional information into account, such as what participants viewed and the time a participant spends on Facebook. This will allow researchers to construct more refined measures to reflect the condition under which participants make a decision of whether to engage. Finally, although we observed that longer posts are more likely to draw engagement from participants, it should be noted the posts in this intervention were generally short by design (mean 33.4, maximum 107 words). Social media marketing reports reveal that Facebook posts that have <50 characters receive the highest level of engagement relative to longer posts [52]. A/B testing of a wide range of post lengths and different types of posts (eg, goal setting vs problem sharing) is needed to determine the ideal length of posts to maximize engagement in behavioral interventions.

In this study, we performed secondary analyses using data from a pilot feasibility randomized weight loss trial that delivered a lifestyle intervention via Facebook and linked participants’ engagement with several important predictors, including characteristics of the posts, replies or comments, and participants. Our results point to the potential of using data science and NLP tools to analyze microlevel behavioral or conversational data and identify factors influencing participants’ engagement during the social media weight loss intervention, which have implications for the design and implementation of future interventions that could lead to more favorable weight loss outcomes. Future studies are warranted to validate our results and further explore these relationships in similar and larger trials.