It is well established that sufficient physical activity is important for good health. Despite this, the World Health Organization still reports that 1 in 4 adults and 2 in 4 adolescents are inactive, with a global direct cost of $54 billion and $14 billion attributed to lost productivity [1]. Over 4 billion people currently use smartphones worldwide, with this number projected to reach 6.1 billion by 2029 [2]. With such widespread use, mobile apps show promise as delivery channels for incentives that promote physical activity [3].

Systematic reviews of randomized controlled trials and observational real-world studies suggest that financial incentives are effective in encouraging physical activity [4-8]. Despite their apparent effectiveness, the question remains as to whether financial incentives can offer a cost-effective intervention that should be funded at a population level [9]. A Canadian incentive program appeared to be cost-effective as the reward levels were kept at a minimum, but it was closed due to funding constraints [10]. A longitudinal study that reduced reward size over time in the same Canadian intervention found that the number of points (or size of the reward) offered was not as important as the type, timing, and content [4]. Vitality UK is a health and life insurance company that offers Vitality Active Rewards, a reward program for staying active. Vitality’s Active Rewards program was the subject of an observational real-world study in 2019. The researchers found that larger long-term incentives alone did not increase physical activity or prolong engagement as effectively as when combined with smaller immediate incentives. Another UK study using the Sweatcoin app, which converts steps into a digital currency that can be exchanged for vouchers, looked at the mean daily step count 3 months after registering on the app compared with the mean daily step count in each of the 3 months before registration. They found that the step count increased significantly over the 6-month period [7].

While real-world incentive-driven interventions show promise, there remains a question about the feasibility of financial incentives in the long term [11,12]. This has led to calls for research that systematically manipulates reward size and win probabilities to explore methods for reducing costs [13]. Lottery-based incentives, rather than assured rewards, may offer a way to control costs by fixing the payout [14-22].

In Singapore, a 2021 study of more than 1 million people employed small-scale, immediate, or lottery incentives and observed increased daily steps in users [8]. The researchers used real-time data analytics to inform their incentive strategy and other design features of the intervention, which varied over 3 waves. However, there is little evidence of the relative effectiveness of lottery-based incentives compared with assured rewards. To our knowledge, Patel and colleagues [17,18] are the only researchers who have directly compared lottery and assured incentives in an app-based study. They compared daily, assured, gain-based incentives, lottery incentives, and loss incentives, ultimately finding that only loss-framed incentives were effective compared with control [18]. They also compared different lottery conditions—higher frequency smaller reward, jackpot, and combined—and found only the combined lottery incentive to be significantly greater than the control [17]. In another study where participants were given the choice to receive rewards as guaranteed cash payments or a lottery ticket with a 1-in-10 chance of receiving the same expected value, not surprisingly, most opted for the guaranteed payout; therefore, the lottery-group sample size was inadequate to make a meaningful comparison [20].

In commercial app-based interventions, the incentives are often subject to change, usually because of a business or financial decision rather than being theory-based or driven by research aims [23,24]. Changes in app incentive features offer the opportunity to observe patterns in data before and after the change, giving insights into which ones may be worthy of further analysis or study [25]. Observational studies generate a lot of ecologically valid data, but the lack of randomization between treatment conditions means that it is not possible to draw conclusions about causation because other confounding variables may be driving the outcomes [26].

This study aimed to explore patterns of tracked physical activity by users of a commercial app before and after a change in incentive strategy. This is a quasi-experimental study that took advantage of a business decision to alter the incentives in the BetterPoints app from certain points to lottery-focused to control costs. A secondary aim was to better understand how to analyze app-derived data sets to lay the groundwork for future trials that may help to establish the causal effects of different incentive strategies.

A total of 1666 app users were registered on or before the start of the study and had tracked an activity during the study period. These users were 53% (n=881) female, with a mean age of 42 (SD 12.13) years. Among them, 29.6% (n=493) were from the most deprived deciles. Table 2 summarizes demographic data of participants.

The mean weekly minutes of physical activity were 137 (SD 203) at T1, 138 (SD 209) at T2, 156 (SD 219) at T3, and 153 (SD 206) at T4.

Table 3 shows the mean, SD, and median for each time point.

Overall, there were significant differences in the weekly minutes of activity before and after the change in incentive strategy as assessed by a Friedman test (χ²₂=42, P<.001).

The results of post hoc Wilcoxon signed-rank tests indicated a significant difference in the paired observations between the last week of the assured points incentive condition (T2: mean 138, SD 203) and the first week of the lottery incentive condition (T3: mean 156, SD 219) with a z score of –4.14 (P<.001). Table 4 shows the results of all the pairwise comparisons.

Secondary analyses explored physical activity over time. Plots are presented in Figures 1 and 2 show a more complex picture than before/after comparisons. They suggest app users were waning in their activity levels during the middle part of the incentive schedule, before the change, and increased their physical activity in the lead-up to the change in incentives. There was a peak in activity at the time of the change in incentives strategy, and then physical activity levels dropped quite dramatically. The median weekly activity reached 0 in the week commencing October 27, 2018, and remained there.

CIs overlapped during the period before and after the change in incentives. Note that the CIs were derived from bootstrapped simulations within each week and are not as powerful as the within-subject statistical tests.

The plot in Figure 3 shows differences in the variation of tracked physical activity around the time of the change in incentives. This suggests that age may moderate response to different incentive designs. The plots in Figures 4 and 5 show that men and women and those in different IMD deciles tracked similar patterns of physical activity over time, which may indicate that gender and socioeconomic status are less important moderators.

This study aimed to explore patterns of tracked physical activity by users of a commercial app before and after a change in incentive strategy. The last 2 weeks before the change and the first 2 weeks after the change were statistically significantly different, suggesting that the change in incentives to a greater focus on lottery awards led to more physical activity among users. However, time-series analysis with CIs shows that important fluctuations occurred over time in the tracked physical activity data which require further study to explain. This suggests the significant before/after difference may not represent the effect of a change in incentive strategies. We note that had we not carried out the longitudinal analysis, we may have been misled by the statistical tests to draw premature conclusions about the relative effectiveness of assured versus lottery-based incentive strategies.

It may be possible that individual differences are moderating the effectiveness of incentive strategies over time. We conducted some exploratory analysis, but further research is needed on demographic and psychological factors that may be moderating responses to incentives. Age may be a factor, as the plots showed that those in different age groups had different patterns in their tracked weekly minutes of physical activity around the time of the change in incentives. Men and women and those in different IMD deciles showed similar variation in physical activity over time, suggesting that they may be responding in a similar way to the change in incentives. The potential for demographics, personality, and prior behavior to moderate the effectiveness of incentives on physical activity outcomes was beyond the scope of this paper and requires further study.

This study highlights the difficulties and limitations of using real-world data in a retrospective study. When the study was originally conceived, the change in incentive strategy seemed straightforward: more emphasis on the lottery and less on assured points. However, upon closer inspection, the complexity of the 2 strategies became apparent, as shown in Table 1. Multiple changes to different features were made simultaneously, including the goals, certainty, and incentive amount. It was impossible to ascertain retrospectively if the actual value of the reward in monetary terms equalized due to the unknown total sample for whom the rewards were available (ie, those in sponsored programs) and the multiple available prizes that would have made calculating the expected value difficult. Future work should simplify incentive strategies and equalize value across assured points and lottery-based strategies.

The continuous nature of real-world data, with individuals regularly joining and leaving the app, required decisions to be made over what data to include in the analysis. We chose to remove users who joined after the study start date and/or tracked nothing during the study, but these approaches could be reconsidered in future studies. Patterns of engagement with an app are also important.

The quasi-experimental nature of this study does not allow for conclusions regarding causality and the relative effectiveness of different incentive strategies [26]. Before and after tests appear, at best, inadequate for assessing the effectiveness of incentive strategies and, at worst, misleading. The method of statistical process control that has recently moved from manufacturing to health informatics may be informative here. Health data are often not in control/stable, which means they do not lend themselves well to before-and-after studies because we do not know if deviations in observed health behaviors are correlated with changes introduced, just part of their natural fluctuation, or the result of some other confounding or moderating factors. The challenge for this area of research is how to transfer more robust methods to real-world settings.

Fluctuations in tracked physical activity over time have typically been given less attention than before and after results derived from static tests of difference at set time points. App-based studies using the Carrot rewards app in Canada and the Vitality app in the United Kingdom both suggested short-term smaller rewards were more effective; however, both studies also showed large ongoing fluctuations in longitudinal plots [4,6]. Brief explanations pertaining to dips during the December holiday period or seasonality relating to weather were offered but without further analysis. Chew and colleagues [8] in Singapore presented times-series plots of daily step counts in each of the 3 waves, showing that the data were unstable. The researchers were aware of these fluctuations and offered possible explanations relating to external factors such as festival times. They also used these observed changes in daily step counts to prompt changes in the intervention, such as spreading out rewards and the introduction of thematic challenges [8]. However, they did not highlight these variations as possibly undermining conclusions drawn from the analysis of a static baseline before the condition with static follow-up time points. The mean daily step count in the after or postintervention condition could have obscured important weekly variations in the data.

It is not recommended that the before-and-after approach be generalized to other studies that seek to understand the effectiveness of incentives. Future work may look in more detail at fluctuations in physical activity over time. This may employ visualizations, growth modeling, and cluster analysis to understand individual trajectories of physical activity and similarities among app users. Individual psychological, sociodemographic, and behavioral factors that correlate with or predict physical activity trajectories or group memberships should also be examined. Additionally, incentive strategies should be further tested in randomized trials to ascertain causality and confidently determine which strategies are working well and for whom.