The global incidence of diabetes has increased considerably in recent years, accompanied by escalating disease severity. This progression has detrimental ramifications, including compromised quality of life, multifarious complications, and expensive surgical treatment. The medical expenses for diabetes are expected to increase to approximately US $2.5 trillion worldwide by 2030 [1]. This has facilitated the urgent need to reduce the severity of diabetes. The recent national clinical practice guidelines [2] state that the basic principle of this preventive treatment is to maintain glucose levels within the normal range. The central strategy to manage glucose levels is to maintain an appropriate lifestyle of eating and exercising [3], in addition to insulin or oral medication. In recent years, technological advances in continuous glucose monitoring devices have helped patients manage their glucose levels with mobile apps at any given time and place, thus promoting self-management [4,5]. However, it is difficult for patients to learn the optimal behavioral action to maintain glucose within a normal range [6]. Therefore, a personalized system that recommends individualized optimal behavioral actions is required to ensure that glucose levels do not deviate from the normal range in the future.

Over the past decade, it has become possible for patients with prediabetes to self-monitor their glucose levels, along with their lifestyle habits, such as diet and exercise, by visualizing them through an integrated mobile app [4,7]. While this has made it easier for patients to manage their own glucose levels daily, and maintaining high user adherence to such monitoring is achievable, as demonstrated by recent studies reporting over 90% daily adherence [8], learning the best behaviors to avoid abnormal glucose levels remains a significant hurdle [4]. For instance, the total grams of carbohydrates to be consumed or the total minutes of walking exercise to be performed vary from person to person. Therefore, the behavioral recommendations should be automatically generated for each patient by analyzing personal datasets collected daily.

Thus far, there has been significant research on personalized behavioral recommendations through mobile apps for the prevention of chronic diseases, including diabetes [9], and a framework called just-in-time adaptive intervention has been established. In just-in-time adaptive intervention, to improve both the short-term and long-term outcomes, decision rules regarding intervention conditions, such as content, frequency, and timing of recommendations, are personally optimized [10]. For instance, in [11], to promote physical activity in patients with hypertension, many interventions with different conditions were allocated and performed on each patient’s timeline, and the best-fitting intervention conditions were identified based on the results of the interventional experiments. While this approach can robustly identify personally appropriate intervention conditions, its practical application for practitioners is constrained by the significant burden associated with prolonged experimental periods and the necessity for patients to undergo multiple interventions that may prove to be suboptimal. This underscores the need to develop more efficient and effective personalization methods.

With regard to the aforementioned factors, recent studies have focused on sequentially optimizing personal intervention conditions using reinforcement learning [12]. Yom-Tov et al [13] proposed a contextual bandit that optimized the message content of exercise interventions for patients with diabetes. In this study, an action is the message content, and a reward is the amount of activity after the recommendation. The individual optimization of message content can be adaptively performed by selecting an action based on a reward using Boltzmann sampling. Liao et al [14] proposed a framework that applies a multistep Markov decision process by introducing a user state, which represents how much intervention to increase daily walking has been received so far. This helps adjust both the intervention content and the intervention frequency to avoid user dropout caused by frequent interventions. However, while the focus of these existing methods is on patient behavior change, it is still inconclusive whether these methods can offer optimal interventions for improving clinical outcomes. For instance, in the case of diabetes, the extent to which the intervention lowers glucose levels is highly significant. In other words, further research is needed to identify the intervention conditions that can lower glucose levels most effectively to the target range.

Some studies have proposed optimizing clinical insulin interventions for people with more severe diabetes using reinforcement learning with an artificial pancreas to directly control their glucose levels [15,16]. When insulin is injected, the glucose level drops immediately. Therefore, the primary goal of such research is to determine the appropriate insulin dose and timing for each patient to ensure that the glucose level remains within the normal range. The action is the insulin dosage, the state is the user’s glucose level, and the reward is the normality of the glucose levels after insulin injection. The integration of reinforcement learning helped obtain the optimal insulin dosage for each patient [17]. However, in this type of study, the variations in action were limited to clinical and direct interventions, such as insulin injection, which are not directly controlled by the patient. Incorporating behavioral interventions, such as diet and exercise recommendations, in addition to clinical interventions, is currently regarded as future work [15,16].

Altogether, there is a lack of approaches that individually optimize behavioral recommendations such as diet and exercise to control glucose levels. In this paper, we propose a multiarmed bandit algorithm that can individually plan mobile behavioral interventions, making it easy for users to perform and reducing glucose levels after eating (ie, postprandial glucose, which is most likely to become abnormal in the daily lives of people with diabetes). The most novel aspect of the proposed method is the introduction of a two-stage reward prediction mechanism for each intervention: (1) prediction of the actual user behaviors that occur after the intervention, and (2) prediction of the postprandial glucose trajectory from these predicted behaviors.

An overview of the proposed method is provided in Figure 1. The aim is to lower postprandial glucose levels with the intervention (Figure 1A). Here, the action (or arm) is a mobile behavioral intervention that combines recommended amounts of carbohydrate intake and postprandial walking exercise, which have been proven to have specific effects on glucose levels [18,19]. Multiple alternative candidates for these recommended amounts are prepared in advance, and the optimal action for each user is selected based on the reward predicted in a two-stage manner, as provided in Figure 1B. In this reward prediction, the actual amounts of both carbohydrate intake and postprandial walking are predicted for each action candidate. For example, for carbohydrate intake, when a user receives a recommendation to take 60 g of carbohydrates, it is predicted that the participant will take 70 g because the recommended amount is small according to their dietary preference. Second, based on the predicted dietary and exercise behaviors, the temporal trajectory of postprandial glucose levels is predicted. Third, according to the predicted glucose trajectory, batched actions are selected using the ε-greedy method, based on the assumption that the greater the suppression of the increase in postprandial glucose, the higher the reward obtained by a user. Finally, after performing the selected actions using a real intervention, the behavioral adherence and glycemic response models are retrained using the buffer dataset of actual behaviors and glucose, and the parameters are updated accordingly. By repeating this experiential learning cycle, which is composed of action planning and real experiments, it is possible to fill the gap between each user’s real response and the virtually predicted response, which yields a more optimal action selection for lowering postprandial glucose levels. Furthermore, to accelerate this cycle, we suggest incorporating prior knowledge into the prediction models for retraining.

We demonstrate the effectiveness of the proposed method through simulation experiments and clarify the conditions under which the proposed method works best. Then, with a simplified algorithm based on these conditions, we conducted a small-scale mobile intervention experiment with healthy participants. The results provide preliminary evidence that the proposed method improves both behavior and glucose levels. To the best of our knowledge, this is the first attempt to (1) clarify how to optimize behavioral interventions for improving glucose levels and (2) demonstrate the effectiveness of the method through simulation experiments and real-world intervention experiments.

In the following subsections, we describe the problem setting, the details of the proposed method, the simulation experiment setup, and the real-world feasibility study setup. The proposed method is formulated as a model-based multiarmed bandit framework with a two-stage reward prediction model that integrates behavioral adherence modeling and glycemic response modeling to enable data-efficient personalization of dietary and exercise recommendations. Its effectiveness is evaluated through both simulation experiments and a real-world feasibility study.

The purpose of our method is to select and recommend the best action candidate for each user that is easy to perform and reduces postprandial glucose levels the most through interaction with the user. For this purpose, a multiarmed bandit algorithm is often used, as in previous literature [13,20,21]. The multiarmed bandit is one of the reinforcement learning methods. It continuously addresses the dilemma of selecting actions to gather more information (exploration) vs selecting actions to maximize immediate reward based on current knowledge (exploitation). Thus, a bandit algorithm can adjust this exploration-exploitation trade-off [22] by estimating the rewards associated with different actions.

In our method, the reward r can be defined as the function of the total increase in postprandial glucose levels after the target diet, which occurs after recommending an action A to a certain user. We mainly focus on postprandial glucose, given that glucose levels always rise after a meal and that deviations from the normal range of glucose levels occur most frequently after a meal [23,24].

An action in our method A(∈A) can be represented as a combination of A={ad,ae}, where ad means a recommendation for “how much carbohydrates in grams a user should take at maximum in target diet,” and ae means a recommendation for “the minimum duration a user should walk just after target diet.” For clarity, users do not directly estimate carbohydrate content; instead, it is automatically calculated by a mobile app such as MyFitnessPal (MyFitnessPal, Inc) based on inputted food items. The accuracy of MyFitnessPal’s carbohydrate calculations has been rigorously validated, showing strong correlations (r=0.90) with established food composition databases [25]. Based on the assumption that each user has their own appropriate amounts for both actions, multiple action candidates are prepared in advance within a clinically plausible range. Regarding the amount of carbohydrate intake (ad), we prepared 10 candidates in the range of 10‐100 g, referring to the low-carbohydrate diet recommendations [26]. In addition, regarding the amount of postprandial walking exercise (ae), we prepared 6 candidates between 5 and 30 minutes based on the experimental results of the effect of walking exercise on glucose levels [19,27-29]. Therefore, there are 60 action candidates in the action set A based on previous research [19,26-29], as shown in Figure 2.

Here, there are 2 approaches for action selection in a multiarmed bandit, that is, the experience- and the model-based approaches. The former (eg, Upper Confidence Bound) evaluates each action based on the actual observed rewards, which enables robust evaluation but requires extensive exploration for all actions. In contrast, the latter (eg, Thompson Sampling) can reduce exploratory trials because it can estimate the reward (or value) for each action by learning the parameters of the reward distribution. Returning to our setup (Figure 1A), each observation of the reward (glucose level change) requires users to wear a glucose monitoring device, imposing a recurring physical burden. Therefore, it is important to reduce the burden of exploration trials on users for real applications, and we adopted this model-based approach to predict the reward for untried actions efficiently from limited observations.

Now, the reward prediction model is represented as follows:

The input is an action A, and Θ is a model parameter set of a target user. Because Θ is unknown at the initial trial, it is sequentially updated when observing user responses to previously recommended actions. Here, the user response is composed of 3 variables, (x,y,z) as shown in Figure 3; x is the actual amount vector of each carbohydrate intake at the target diet, z is the actual walking duration vector of each walking event occurring after starting the target diet, and y is the time series of postprandial glucose levels after the start time of the target diet as τ∗. Because the glucose level within 1 hour after a diet is of clinical importance, we focused on user responses within 90 minutes of starting the diet. As a result, for each action, the observed data are represented as D={A,x,y,z} and are added to the buffered dataset B after observation.

Here, glucose levels and behaviors in 1 observation “D” would involve high uncertainty because they may be affected by many factors. Thus, a parameter set “Θ” is updated in a Bayesian fashion when obtaining batched observation data containing “M” actions and “M” user responses. That is, as shown on the right side of Figure 1B, parameter learning is executed after recommending “M” actions to the users. This can be considered as a batched multiarmed bandits setting [30]. Hereafter, the method of selecting “M“ actions is referred to as the policy.

In this subsection, we describe the proposed multiarmed bandit algorithm based on two-stage reward prediction. The pseudocode for the proposed algorithm is shown in Textbox 1. In this algorithm, the following processes are executed in the given order: (1) actions are recommended to users according to the predetermined policy, and batch observation data are obtained; (2) the parameters of the behavioral adherence and glycemic response models are retrained using these observation data; (3) the reward for each action candidate is predicted using a two-stage reward model with retrained parameters; and (4) finally, the next batch actions are selected based on the reward prediction result using the ε-greedy method.

These processes are repeated to improve the rewards and model parameters. This predictive model–based approach allows the algorithm to estimate potential outcomes of all possible actions based on learned models, rather than requiring direct, burdensome observation for every action candidate. In the first iteration, default actions are recommended for pure parameter exploration, aiming to gather diverse data for initial model parameter estimation. Afterward, as the number of iterations increases, more robust parameter learning can be performed with more observational data, resulting in accurate reward prediction and action selection. However, if the convergence of this parameter learning requires long-term (eg, several months of) observation data owing to the uncertainty contained in the observation, the user would drop out due to the burden of such time-consuming experiments for collecting data. To prevent this, we propose introducing prior knowledge for each parameter and performing Bayesian learning.

Here, we first focus on describing a two-stage reward prediction model that integrates behavioral adherence models into a glycemic response model. The starting point for modeling is based on previous clinical findings, which state that the changes in postprandial glucose are highly dependent on the amount of carbohydrates an individual takes [31] and the amount of exercise performed after a meal [19,27-29]. Furthermore, these dietary and exercise behaviors are assumed to depend on the kind of actions recommended immediately beforehand. Based on these premises, the reward prediction model is described as follows:

where R(A|Θ) is the reward, y∗(A|Θ) is the glucose time series, x∗ is the amount of carbohydrate intake, and z∗ is the walking duration. Further, Θ={θg,θd,θe} and θg,θd,θe represent parameters in a glycemic response model, a carbohydrate intake model, and a postprandial exercise model, respectively. These parameters are defined for each user. In actual reward prediction, the first stage is to predict the dietary and exercise behaviors using Equation 3 with behavior adherence models hd(⋅) and he(⋅), and the second stage predicts the trajectory of the postprandial glucose increase y∗ using Equation 2 with the predicted behaviors (x∗,z∗). Finally, the glucose increase trajectory is converted into a reward scale using Equation 1.

The first stage is behavior prediction. Because each user receives a specific action immediately beforehand, we can assume that they act according to this action. However, certain actions are difficult to execute in real-world scenarios. For example, most users would find it easy to keep their carbohydrate intake less than 100 g, but it is difficult to consume less than 10 g. In addition, the actual reaction to reducing carbohydrate intake to 50 g may vary for each user. This is consistent with the claim in the theory of planned behavior [32], which states that behavioral intentions may be influenced by “perceived behavioral control.” Inspired by this, in the behavior adherence model, we hypothesize that there is a perceptual critical point specific to each user for each action candidate, as indicated in Figure 2, and that the actual behavioral tendency changes depending on this critical point. Specifically, as shown in Figure S1 (left) in Multimedia Appendix 1, we assume that if the recommended amount of carbohydrate ad is higher than the critical point Xth, the user is likely to try to behave in accordance with the action. In contrast, if ad is lower than Xth, the user is likely to ignore the action and behave normally.

Based on the aforementioned assumptions, the behavior adherence model of carbohydrate intake is represented by the following linear model:

where θd={Xth,Xnorm} and Xth is the perceptual critical point for a target user, Xnorm is a parameter that represents the normal carbohydrate intake at the target diet. We also introduce Gaussian noises ex1∼N(0,σx1) and ex2∼N(0,σx2) to represent the uncertainty of carbohydrate intake from other factors. Here, different variance parameters (σx1,σx2) are set because the level of behavioral uncertainty may depend on whether the critical point is exceeded.

Subsequently, the adherence model for postprandial exercise is based on the same concept, as described for carbohydrate intake above. As shown in Figure S1 (right) in Multimedia Appendix 1, if the recommended walking duration ae included in the action is shorter than the perceptual critical point of each user, it is expected to be adhered to. Otherwise, the recommended action is not adhered to, that is, the actual walking duration tends to be zero. Specifically, the exercise adherence model is represented using the following linear model:

where θe={Zth,Zmax} and Zth is the perceptual critical point and Zmax is the parameter that represents the upper bound of walking duration in terms of the user’s physical strength. In this model, it was assumed that the walking time is zero at the upper bound. Further, we introduce Gaussian noises ez1∼N(0,σz1) and ez2∼N(0,σz2) to represent the uncertainty of exercise behavior from other factors.

Based on these behavior adherence models, each user’s dietary behavior parameters θd and exercise behavior parameters θe are inferred according to the buffered observation data Bi accumulated for each user at an iterative time point i. Then, by deploying the inferred parameters θd(i),θe(i) into each corresponding model hd(⋅),he(⋅), the behaviors x∗(i), z∗(i) in response to each action candidate can be predicted.

Next, the postprandial glucose trajectory y∗ is predicted based on each behavior x∗,z∗ predicted for action A. Our approach models this trajectory based on the clinical findings. First, it is well known that glucose levels increase with carbohydrate intake. For example, Ashrafi et al [33] showed that the actual postprandial glucose trajectory could be fitted by a response curve to carbohydrate intake. Second, it has been proven that exercises, such as walking, immediately lower glucose levels, as previously mentioned in this “two-stage reward prediction model” subheading [19,27-29]. The mechanism is that energy expenditure in muscle tissues due to exercise causes glucose absorption for energy supply [34]. For example, Jankovic et al [35] and Xie and Wang [36] adopted a linear additive model to predict the glucose trajectory and showed that predictive performance was improved by adding the exercise effect on glucose as a factor, in addition to the carbohydrate intake effect. Based on this glucose modeling, we hypothesized that the observed postprandial glucose trajectory y can be represented based on the summation of the response curve Rd to carbohydrate intake and the response curve Re to exercise to the baseline preprandial glucose level ybase, as follows [37]:

Because the time range of the glucose trajectory that we focus on here is short-term, we assume that the baseline value ybase is constant and substitute the average of the glucose history from 15 minutes before starting the target diet. Here, ey is the vector of Gaussian noise following N(0,σy). The response curves Rd,Re are represented by the following equations:

where τx,i,τz,j represents the time of event occurrence of the ith carbohydrate intake or the jth walking event, respectively. Further, P represents the number of carbohydrates consumed within the target diet, usually P=1. Also, Q represents the number of walking events that occurred within 90 minutes of starting the target diet. In these equations, we adopted a bell-shaped function as the response curve, similar to that in previous works [33,38], owing to its interpretability and the small number of parameters. Some specific examples of each response curve, Rd,Re are shown in Figure S2 in Multimedia Appendix 1. This response curve starts at the timing of event occurrence τx,i,τz,j. The response curve is then amplified based on the amount of carbohydrate intake xi or the duration of each walk zj. Here βd,βe is a parameter that represents the degree of amplification with respect to the volume of each behavior. Further, αd,αe is a parameter that represents the degree of time delay of the response. A hyperparameter of the prior is introduced for each parameter βd,αd,βe,αe, and we describe how this can be obtained in the subsequent “Parameter Learning” subheading.

Based on the above setting, specifically defined by Equations 6-8, it can be seen that the parameters of our glycemic response model θg={βd,αd,βe,αe} for each user are estimated from the observed behavior sequence (x,z) and glucose trajectory (y), at each iteration point (i). In this parameter learning, we use the observed behaviors (x,z) rather than the predicted behaviors (x∗,z∗) as training data to identify the parameters more accurately, reflecting the real relationship. In contrast, the predicted behaviors (x∗,z∗) are used as inputs for reward prediction for the action candidate A. Accordingly, the predicted postprandial trajectory is represented as follows:

where the baseline term is removed compared to Equation 6. This is because the baseline term does not contribute to the glucose increase and, hence, the reward. Then, y∗(A|Θ)=g(x∗,z∗|θg) represents the series of pure increase of postprandial glucose affected by behaviors. It should be noted that for each iteration (i), the estimated parameter values θg(i) are used as θg in these equations.

Finally, the glucose increase trajectory, y∗, is converted into a reward scale. The reward should be designed such that the smaller the degree of glucose increase, the higher the reward. We use the incremental area under the curve (iAUC), a traditional glucose metric [39], as the degree of glucose increase. Based on these factors, we perform reward conversion using the sigmoid function, as shown below:

where iAUC is the total value of the trajectory of glucose increase. As described in the “Problem Setting” subheading, τ∗ denotes the start time of the diet, and T=90 minutes. a,b are parameters for the conversion from the iAUC to the reward scale. We set a=−3 and b=1 empirically, based on the distribution of the iAUC. Consequently, the reward value ranges between 0 and 1.

Thus, for each action candidate A, a reward calculated from postprandial glucose increase R(A|Θ)=f(y∗(A|Θ)) is predicted based on our original two-stage prediction.

After the reward is predicted for each action candidate, the policy for selecting batched actions for the real intervention is selected. Specifically, the action that maximizes the predicted reward is selected. However, the predicted reward is not always accurate due to overfitting caused by scarce training data at the initial stage. Moreover, if the action is always selected to maximize that reward, the real action for the user becomes fixed, and the observed user response subsequently exhibits a small variation, which also leads to overfitting. Therefore, we adopt the ε-greedy method for balancing exploration and exploitation. For exploitation, assuming that the prediction model is accurate, actions that maximize the reward (π) should be selected. For exploration, assuming that the prediction model needs to be trained more, actions are selected randomly (πrandom). Such policies π∗ and πrandom are taken with probabilities of 1−ε and ε, respectively. Because parameter learning is performed in a batched manner, M actions are selected simultaneously for each policy. Therefore, π∗ and πrandom are represented as follows:

where, Random(A) represents an operation for extracting an element randomly from a set A. Specifically, in πrandom, this operation is repeated M times independently.

The learning of the parameter set Θ={θg,θd,θe} in the reward model is performed at each iteration time just after M-intervention experiments are finished, and the batched observation data are accumulated in the buffered dataset. In this learning, all the buffered datasets accumulated up to the ith iteration point are used. If the ith batch of data Di are observed, the training dataset at that time becomes Bi=⋃j=1iDj. The batched data includes a history of 4 types of data (action, postprandial glucose level, amount of carbohydrate intake, and postprandial walking duration) for M target meals. This is represented as Dj=⋃m=1M{Aj,m,yj,m,xj,m,zj,m}. In the observation data, carbohydrate intake and postprandial walking can occur multiple times per target diet. Therefore, they are represented as vectors (recall Figure 3B). In practice, θg,θd,θe are separately trained from the following subsets of the buffer data Bi(g),Bi(d),Bi(e) for each parameter.

Here, Bi(g),Bi(d),Bi(e) are the training datasets for the glycemic response, dietary adherence, and exercise adherence models, respectively.

To improve the efficiency of this learning process, we introduce prior distributions for each parameter. Specifically, for obtaining prior information for the behavioral parameter sets θd,θe, we ask each user to answer questions corresponding to each parameter beforehand and use the responses to define the prior distribution. For example, the question for the critical point Xth is “If you were to reduce carbohydrates in your dinner, what would you say is the minimum amount of carbohydrates?” Then, the user’s answer is set to the mean μXth of the Gaussian prior distribution N(μXth,σXth). For the other parameters Xnorm,Zth,Zmax, each prior distribution is set similarly.

On the other hand, the glycemic parameter set θg cannot be clarified through user interviews because each parameter is related to a complex individual biological response. Therefore, we adopt a data-driven approach wherein the prior distribution is pretrained through the original upstream task. Before starting the first intervention experiment, we recruited additional participants belonging to the same disease group as the target users. Second, we collect a glucose dataset, D(g)=⋃l=1L{yl,xl,zl} for L diet events from each participant. Third, this dataset is used to infer glycemic parameters θg for each participant. Finally, the distribution of these parameters θg among all participants is used as the Gaussian prior distribution of θg. Specifically, the mean μ∼g and the variance σ∼g of these parameters θg are set as N(μ∼g,σ∼g). Here, each parameter θg is assumed to be generated independently.

Through this process, each prior distribution of the parameter set θg,θd,θe is obtained before the first intervention experiment, and this prior is used for Bayesian parameter learning at every iteration time point. This parameter learning is performed by executing a Markov Chain Monte Carlo simulation with the No U-Turn Sampler implemented in RStan (Stan Development Team) [40].

We studied in a simulation experiment whether our online algorithm can improve postprandial glucose levels with personalized recommendations. In addition, to carry out real experiments with real personalized recommendations, we investigated the best setting for the proposed method to maximize performance. Specifically, we aim to clarify the following questions through simulation experiments:

To answer these questions, we set up several virtual users who exhibited different behavioral and glycemic responses, performed repeated action planning and pseudo-intervention experiments using the proposed method for each user, and evaluated these experimental results to address the above questions. At the same time, we also show that by introducing the proposed method, the reward performance can be improved compared to selecting default actions uniformly.

In the real world, behavioral tendencies and glycemic responses differ from user to user. Therefore, in this simulation, a parameter set θg,θd,θe for each user was set to ensure that virtual users exhibit response patterns as diverse as possible. Table 1 shows the actual settings of parameter values for each user. The total number of virtual users was 10.

As an example of user diversity, it is difficult for user ID4 to reduce carbohydrate intake because of Xth=90, whereas user ID9, with Xth=10 finds it easy to reduce carbohydrate intake. User ID4 also has βd=0.05, indicating a greater glycemic response to carbohydrate intake, while user ID5 has a lower glycemic response due to βd=0.04. Thus, each parameter value was set differently for different users. The value ranges of the behavior parameters θd,θe were within the range of the action set shown in Figure 2. These ranges were defined following established dietary and exercise guidelines. Specifically, the minimum carbohydrate intake (Xth) for virtual users was set with reference to “very low-carbohydrate” diets (20‐50 g/d) and “low-carbohydrate” diets (<130 g/d) as defined by Oh et al [41], while the normal carbohydrate intake (Xnorm) was guided by typical daily carbohydrate intake levels (200‐325 g/d) as described in the literature [26]. For the parameters of postprandial walking durations (Zth,Zmax), typical ranges in prior experimental studies (5‐30 minutes) [19,27,29] and recommendations for at least 10 minutes of walking [27] informed the setting of Zth, and cases of walking up to 90 minutes [28] guided Zmax. To introduce a diverse range of realistic individual differences among virtual users, slight deviations from these foundational literature values were sometimes permitted. Nevertheless, the chosen parameter ranges for virtual users in Table 1 do not drastically diverge from the prior information collected from real participants (in Table 2), suggesting that the simulated settings are not unrealistic. In addition, the value ranges of the glycemic parameters θg were set based on the prior knowledge obtained in the upstream task.

Specifically, the cumulative reward of the virtual user (n) at iteration of the experiential learning cycle (i) can be represented as follows:

where rj,d,n represents the actual reward transformed by Equation 10 from the glucose trajectory y˘ of the virtual user n after the dth target diet in the jth intervention experiment. Further, D is the number of target diets included in each experiment; and D=6 was used in the simulation. The average cumulative reward of N virtual users, which is used as an evaluation metric, can be represented as follows:

By increasing i to the maximum iteration time I, we can track how the metric J(i) changes depending on each condition of ε. Considering the burden on users in real experiments, I was set to 10. This means that each user must continue wearing the necessary sensors to collect user responses for a maximum of 60 meals when D=6.

In this simulation, the true value of each parameter for each user was known, as listed in Table 1. This enables the investigation of the estimation error of each parameter for any setting of our method. To answer the question (Q2), we investigated how the estimation error changes as the iteration time increases (ie, the amount of training data), with and without prior distribution. The estimation error was defined as follows:

This corresponds to the standard error (%). p is the parameter number, θp is the true value of the parameter in Table 1, and θ^p(i) is the estimated value of the parameter in the ith iteration time.

Additionally, we conducted a preliminary set of short intervention experiments with real users using the proposed method, based on the simulation results. This experiment was positioned as a pilot study before conducting a full-scale study. The purpose of this experiment was to explore the feasibility of whether postprandial glucose levels can be improved in real users after recommending actions updated by the proposed method compared to default actions. Accordingly, in this study, we compared the performance when default actions were selected in the first experiment (referred to as the “default policy”) with the performance when actions were selected by the proposed method in the second experiment (referred to as the “optimized policy”) using a within-participant study design, to evaluate the feasibility of the proposed method. The optimized policy was updated from the observed participant responses in the first experiment, following the proposed method.

First, we recruited 6 healthy participants (4 men and 2 women) as real users from April 2023 to July 2023. Because this experiment was the first feasibility study, the target group was healthy rather than diabetic. All participants were researchers at Aalto University and the University of Helsinki.

Next, each participant was assigned the goal of reducing postprandial glucose levels to prevent diabetes. To achieve this goal, the participants received daily recommendations for the target diet, including total carbohydrate intake and postprandial walking duration, via a dedicated smartphone app. The target diet was either dinner or lunch each day, and the participants chose according to their lifestyle preferences. While the recommendations in the first experiment were preset to default actions, those in the subsequent experiment were preset to actions selected by the proposed method. Participants were free to choose whether to follow the recommendations and to act accordingly. To collect a participant’s behavioral and glycemic responses to the recommendation actions, they wore a continuous glucose monitoring device and an activity tracker.

This process of mobile intervention and data collection for each participant was carried out for 6 days in the first experiment and for 3 days in the second experiment. Because the first experiment corresponded primarily to the exploration phase and the second experiment corresponded to the exploitation phase, the number of days in the second experiment was set to be short.

An action was delivered as a recommendation in a notification app on a dedicated smartphone (iPhone 12), which was provided to each participant. In the notification app, a dedicated message including the action was delivered along with a notification sound, a few hours before the target diet. The frequency and duration of this recommendation were predetermined according to participants’ preferences.

An example of this message appearing on the smartphone screen is “Eat less than 60 g of carbohydrates at dinner, and walk for more than 20 minutes just after dinner.” Depending on the selected action, the amount of carbohydrate and postprandial walking duration in the message changed. For default actions in the first experiment, carbohydrate intake (ad) was within the range of 20‐80 g, and the postprandial walking duration (ae) was within the range of 5‐20 minutes, considering the participants’ preferences and previous literature [19,26-29]. For example, in the second experiment, actions were selected based on the two-stage reward prediction results of each action candidate; therefore, they were not always within this range. This reward prediction was attained by setting the parameters learned from the user response data obtained in the first experiment.

For user response data, we collected data related to the participant’s continuous glucose levels, physical activity, and food records. Figure 4 illustrates how the data were collected. Continuous glucose measurements were collected every 5 minutes using a Guardian Connect medical device (Medtronic Inc), which was inserted on the participant’s skin. At the same time, physical activity data were collected using the activity tracker Vivosmart 3 (Garmin Ltd), which was worn on the participant’s arm. To mitigate erroneous detection of walking from various daily movements, the device identifies a walking event when characteristic arm movements indicative of continuous walking are detected for a duration exceeding a certain minimum threshold. The physical activity data included the start time and duration of each walking event that was automatically detected by the device. A dedicated fitness tracker was chosen over participants’ personal smartphones to ensure data consistency and reliability across participants, as smartphone usage and carrying patterns (eg, in a pocket or bag) can vary significantly and lead to inconsistent activity detection. This approach also minimizes issues related to battery drain on personal devices and addresses potential privacy concerns by avoiding continuous access to personal smartphone data. The glucose and activity data were continuously sent to a dedicated iPhone through a Bluetooth connection with each device. In addition, a food diary, including the amount of carbohydrates and the start time of each target diet, was recorded manually by the participant. The carbohydrate content was automatically calculated using the mobile app MyFitnessPal by inputting all food contents and amounts.

Finally, because we focus on the improvement of postprandial glucose levels, we extracted postprandial segment data from the longitudinal time series as user response data for each action. Specifically, the segment data were defined from the start of each target diet until 90 minutes later, as shown by the blue frames in Figure 4. Each data segment was added to a buffered dataset.

The evaluation was based on whether the reward related to postprandial glucose increased in the second experiment compared to the first. Therefore, we set the following metric as the actual reward average J′(i) for all diets of all N participants in the ith experiment, and this metric was compared between the first and second experiments.

It should be noted that the actual reward ri,d,k is transformed by Equation 10 from the glucose increase trajectory, which is obtained by subtracting its baseline derived from preprandial glucose levels from the observed glucose trajectory y.

To introduce the prior distribution of the behavioral parameters, each participant was asked to answer a questionnaire before starting the experiment. As described in the “Parameter Learning” subheading, the result was set as the mean value of the prior distribution of the corresponding parameter. Table 2 shows the actual answers for each parameter together with the corresponding question. It can be observed that the values vary according to the participant. As for glycemic parameters, we estimated the prior distribution N(μ∼g,σ∼g), which is common to target participants, using the glucose dataset from other participants. To obtain this, we recruited 4 additional healthy participants and collected continuous glucose levels, physical activity, and food records around lunchtime over 6 days for each participant in the same manner, as described previously. Consequently, a total of 6 segment data were obtained for each participant, as shown in Figure 4. Then, after estimating the glycemic parameters θg for each participant using his segment data, the parameters μ∼g,σ∼g were calculated among 4 participants, as described in the “Parameter Learning” subheading in this section. As a result of this calculation, we obtained μ∼g={μ∼βd,μ∼βe,μ∼αd,μ∼αe}={0.047,−0.148,19.07,6.22} and σ∼g={σ∼βd,σ∼βe,σ∼αd,σ∼αe}={0.028,0.035,1.91,0.39}.

The study was approved by the Aalto University Research Ethics Committee (Treatment Planning Project/08.06/2023). Written informed consent was obtained from all participants before the start of the experiment. Direct identifiers were removed, and data were pseudonymized and stored securely. No personal data were shared with third parties. Additionally, to ensure participant safety, participants were informed orally and via the consent form that they could withdraw at any time if they experienced discomfort or unease.

We present the simulated results of the average cumulative rewards (J(i)) with and without prior distribution in Figure 5. First, in any case, when an action selection by the proposed method is adopted (ie, when ε is other than 1.0), we can confirm that the cumulative reward gradually increases from the second iteration (i=2) after executing default actions at the first iteration (i=1). This demonstrates the effectiveness of action selection by the proposed method in the second and subsequent iterations (i≥2). Regarding question Q1, by focusing on the variation of ε, we can also confirm that the smaller the value of ε, the greater the accumulated reward, and the maximum cumulative reward is obtained when ε=0.01.

The theoretical boundary of the cumulative reward is shown by the black dotted line in Figure 5, and it is calculated as an ideal condition, wherein the true parameter values are known and used for action selection from the second iteration. The cumulative reward for ε=0.01 with prior knowledge shows almost the same value as that in this theoretical bound. In other words, when prior knowledge is introduced, exploration with the policy πrandom is unnecessary, and the best performance can be attained by full exploitation with the policy π that maximizes the predicted reward from the beginning. On the other hand, in the case without prior knowledge in Figure 5B, the cumulative reward decreases significantly when ε=0.01 and 0.2 are compared with the case with prior knowledge in Figure 5A. In particular, when ε=0.01, the decrease in the second iteration is remarkable.

Next, regarding question Q2, we show the actual changes in the parameter estimation error (ϵp(i)) of the behavioral adherence and glycemic response models in Figure 6. First, it can be seen that the case without prior knowledge (▲) has a larger estimation error than the case with prior knowledge (■) for the first iteration. This large error leads to a prediction error of the reward for each action candidate and therefore makes it easier to select worse actions with the policy π∗. This may be the reason why the cumulative reward decreases immediately in case ε=0.01 and 0.2 due to the erroneous action selection at i=2 in Figure 5B. Furthermore, in the case without prior knowledge, it can be seen that the estimation errors of some parameters (such as Zmax,αe) are not improved as iteration time of experiential learning cycle increases. This indicates the potential risk in real experiments, showing that it is impossible to collect the necessary training data for stable parameter learning, even when imposing heavy burdens of repeated interventions on users. However, when introducing prior knowledge, we can confirm that the parameter errors are minimized from the first iteration for most parameters. This suggests that introducing prior knowledge can eliminate the burden on users and maximize their cumulative rewards.

Based on the simulation results, we conducted a real-world experiment with a simplified algorithm involving a single update of the policy. According to the setting derived in the simulation experiment, we introduced prior knowledge for parameter learning, and we took the greedy policy with ε=0 for action selection.

The obtained metric values for each participant from the real experiments conducted on the within-participant design are shown in Figure 7A. For all participants except ID5, the simple reward average increased in the second experiment using the proposed method. On average, for all participants, this metric increased from 0.65 in the first experiment to 0.80 in the second experiment, representing a 23% increase in reward. To statistically evaluate this improvement, we performed a paired t test comparing the average rewards between the first (default policy, R1) and second (optimized policy, R2) experiments across the 6 participants. The analysis revealed a statistically significant increase in reward following the optimized policy (P=.04). This result indicates that action selection using the proposed method is effective for real-world applications, consistent with the simulation results.

Moreover, we show how carbohydrate intake and postprandial walking duration changed in the first and second experiments in Figures 7B and 7C. All participants showed decreased carbohydrate intake and increased postprandial walking duration. For all participants, the average carbohydrate intake decreased by 35.3% from 60.5 to 39.1 g, and postprandial walking duration increased by 34.3% from 14.0 to 18.8 minutes. To further quantify these behavioral improvements, we conducted paired t tests for carbohydrate intake and postprandial walking duration. For carbohydrate intake, the mean difference (R2 – R1) was −21.425 g, and the paired t test showed a statistically significant reduction (P=.003). Similarly, for postprandial walking duration, the mean difference (R2 – R1) was 4.780 minutes, and the paired t test indicated a statistically significant increase (P=.03). These statistically significant behavioral changes underscore the effectiveness of the proposed method in guiding users toward healthier actions.

Furthermore, to investigate the impact on behavioral adherence, we show the actual behaviors of all participants in response to the recommended actions in Figure 8. The adherence to carbohydrate intake improved from the first experiment to the second experiment. In fact, as a result of calculating the adherence rate for each experiment, the adherence rate for carbohydrate intake increased from 45.7% to 83.3% and that for postprandial walking duration increased from 40% to 55.6%. These behavioral improvements support the hypothesis that action selection using the proposed method is appropriate for participants from the viewpoint of ease of behavior change.

In this paper, we propose a multiarmed bandit algorithm based on the two-stage reward prediction mechanism, which can individually plan mobile behavioral interventions and make it easier for users to perform actions and reduce glucose levels after eating.

From the simulation experiment results, first, we confirm that a larger cumulative reward can be obtained by adopting the proposed method than by selecting the default action. This demonstrates the effectiveness of our online algorithm for personalizing recommendations. Second, it is also demonstrated that the parameter learning converged immediately at the first iteration (i=1) by introducing prior knowledge. This indicates it is not always necessary to repeat the intervention experiment many times for obtaining accurate parameters, emphasizing the importance of the initial single update we conducted in the real-world feasibility study.

From the short-term real-world feasibility study, with a simplified proposed method involving a single update of the policy into a personalized one, it is found that actual postprandial glucose levels can be improved using the proposed method through behavioral improvements in both carbohydrate intake and postprandial walking. Specifically, we observed an average 23% improvement in actual glucose responses, which was statistically significant (P=.04), along with statistically significant improvements in behavioral adherence to the recommendations concerning carbohydrate intake (P=.003) and postprandial walking (P=.03). The adherence rates also increased notably (from 45.7% to 83.3% for carbohydrate intake and 40% to 55.6% for postprandial walking). While promising, these results are based on a small-scale pilot study with healthy participants and a simplified intervention protocol. Therefore, they should be viewed as preliminary evidence rather than definitive conclusions, primarily serving as guidance for the design of future, more comprehensive clinical trials. This demonstrates the potential effectiveness of the proposed method in a real-world setting. Crucially, this pilot demonstrated that even a single update, informed by our novel two-stage reward prediction model and based on initial user-specific data, can yield statistically significant improvements in clinical outcomes (glucose levels) and behavioral adherence in a real-world setting. This offers compelling initial evidence for the practical viability and immediate impact of our personalized approach, serving as a vital proof-of-concept for its core mechanism in a complex human context.

While simulation experiments elucidated the algorithm’s behavior across diverse iteration numbers (i), the real-world feasibility study, designed as a pilot, adopted a simplified approach: a single policy update (from default to optimized after the initial experiment), prioritizing feasibility and logistical tractability. This design inherently meant that the full extent of continuous learning and adaptive capabilities, central to multiarmed bandit frameworks over extended iterations, was not fully explored. Consequently, this study primarily demonstrated the potential of personalized recommendations derived from an initial learning phase, rather than evaluating continuous adaptive optimization over a prolonged period. This outcome underscores the critical need for future investigations to deploy the full online algorithm, addressing the challenges and benefits of continuous policy updates, particularly concerning user engagement and dynamic model accuracy.

To the best of our knowledge, this is the first attempt to (1) clarify how to optimize behavioral interventions for improving glucose levels and (2) demonstrate the effectiveness of the method through simulation experiments and real interventional experiments.

While this preliminary study offers promising insights, it is important to acknowledge several limitations that warrant future investigation to ensure comprehensive validation and generalization. First, the real experiment was short-term and small-scale. The findings from this preliminary study, which involved only 6 healthy participants over a short duration, should therefore be interpreted with caution and are not readily generalizable to a broader population, particularly including patients with diabetes who represent the primary target group for this intervention. The exclusion of patients with diabetes in this initial feasibility study naturally limits the immediate clinical applicability of these specific real-world results. While this real-world feasibility study offers encouraging initial evidence that effective personalized policy updates could be established within a practical, short-term data collection period (eg, 6 days), future research will include a long-term and massive intervention study. This is needed to fully evaluate the sustained effects and long-term stability of parameter convergence and policy updates beyond the initial phase (ie, after the third iteration [i≥3]) in diverse real-world settings and patient populations. Also, as our target in this study was limited to healthy participants, the interventional experiments in patients with diabetes should be performed for a comprehensive evaluation, adhering to robust statistical principles, including multiple comparison adjustments.

Second, this study primarily compared the proposed online algorithm against a randomized (default) policy. While this comparison effectively demonstrated the benefit of personalization over a nonpersonalized baseline, it did not include a direct comparison with other existing reinforcement learning-based or contextual bandit algorithms from the literature. Such comparisons would provide a more comprehensive understanding of the relative performance advantages and disadvantages of our proposed method against alternative state-of-the-art approaches. Future work could aim to include these comparisons, potentially through simulation studies based on established benchmarks or by incorporating other algorithms into real-world trials. Furthermore, a limitation exists in the parameterization of our simulation experiments. While parameter ranges for virtual users (Table 1) were based on literature, specific values were partially informed by our small-scale feasibility study (Table 2). This limited sample of 6 healthy participants may not be fully representative, potentially affecting the generalizability of simulation results. Consequently, these simulations primarily served to qualitatively explore the algorithm’s behavior and identify optimal operational settings (eg, epsilon values and importance of prior knowledge) for future large-scale trials, acknowledging they do not provide definitive quantitative predictions. Future research should prioritize parameter validation and derivation from larger, more diverse datasets, possibly through crowdsourcing or extensive clinical trials, to enhance simulation robustness.

Third, the predictive models, key to our two-stage reward prediction mechanism, possess inherent limitations, particularly regarding data requirements and accuracy. Furthermore, the two-stage prediction itself introduces a compounding effect of prediction errors: inaccuracies in behavioral adherence predictions can propagate and amplify errors in subsequent glycemic response prediction. This “multiplication” of noise impacts reward estimation, especially with limited data. While our preliminary study showed improvements in both glucose and behavior, the potential impact of such compounded prediction errors on long-term reward estimation and intervention effectiveness needs to be thoroughly evaluated in future large-scale studies. Finally, our behavior adherence model is simple, and user responses are also affected by external contextual factors such as weather or daily schedules [14], and adherence levels may decrease with engagement time [42,43]. Therefore, future development should focus on robustly handling limited initial data, efficient data collection strategies, precise quantification or mitigation of propagated uncertainty, and incorporating psychological and contextual factors into sophisticated behavioral models. Crucially, improving behavioral adherence through this future development is also key to minimizing the compounding effects of prediction errors within our two-stage model.

Fourth, the behavioral parameters’ prior knowledge, derived from self-reported questionnaires (Table 2), may not perfectly reflect true user tendencies due to cognitive biases. Inaccuracies in these initial priors could affect the model’s early parameter estimation accuracy and the speed of convergence to optimal recommendations. While our multiarmed bandit naturally adapts over time through experiential learning, more robust methods for acquiring these priors, such as integrating objective behavioral data, could further enhance initial performance and reduce the burden of exploration.

Finally, while our current action candidates are tailored around carbohydrate intake and postprandial walking, expanding the scope to include other types of exercise or dietary components would enhance personalization. Also, redesigning the reward function to align more closely with clinical guidelines for diabetes would improve its clinical relevance. Incorporating advanced behavioral change techniques [44], such as gamification, could further boost user engagement and usability.

Beyond the immediate limitations, our study highlights several important directions for future research and practical implementation. Foremost among these, addressing the ethical considerations and practical challenges associated with long-term behavior modification interventions is crucial for successful clinical deployment. Factors such as user fatigue from continuous data collection and potential disengagement with recommendations over extended periods need careful consideration. Moreover, the current framework does not explicitly incorporate mechanisms to prevent the recommendation of potentially clinically damaging actions. While action candidates are prepared within a clinically plausible range, individual-specific contraindications or extreme physiological states could render even a seemingly “plausible” recommendation detrimental (eg, recommending a very low carbohydrate intake to someone with specific metabolic conditions). Future work will also include empirically establishing practical timelines for robust parameter convergence and policy updates, considering user engagement and burden.

Another critical area concerns the data requirements and accuracy for personalized recommendations. A significant challenge, especially for initial deployment, is the need for sufficient user-specific data to effectively learn and personalize recommendations. While our method uses prior knowledge to accelerate initial learning, acquiring this initial information (eg, via questionnaires or pretraining data) can be difficult and prone to cognitive biases, as self-reported behavioral priors (Table 2) may not perfectly reflect true user tendencies. However, recent advancements in mobile health apps, such as eMOM (Helsinki University Hospital) [8], are enabling the comprehensive collection and visualization of vast amounts of user behavioral and physiological data, including glucose levels. This digital self-tracking, by visualizing relevant data, has shown a positive impact on user understanding of their own health characteristics and offers a promising avenue for acquiring more robust and objective priors. For future clinical real-world applications, directly providing recommendations without proper context carries medical risks. Therefore, a clinician-in-the-loop approach is crucial. This involves an initial phase focused on data collection and visualization (eg, through mobile apps such as eMOM [8]), followed by a collaborative process where clinicians work with users to understand individual characteristics based on the visualized data. This systematic approach ensures the quantity and quality of prior information for each user before recommendations are made. Future work will integrate such an eMOM-like mobile app within a clinician-in-the-loop framework, enabling large-scale validation of the entire process from data visualization to personalized recommendation.

To achieve full clinical deployment, it is paramount to integrate safety constraints into the action selection process, developing methods to filter out adverse clinical outcomes (eg, through real-time physiological data integration or clinician-in-the-loop systems). Future work should proactively investigate strategies to mitigate these issues, such as dynamic adjustment of intervention intensity, gamification elements to maintain sustained engagement, and robust ethical frameworks ensuring user well-being, privacy, and safety in longitudinal deployments.

In this study, we proposed a method for optimizing the planning of dietary and exercise recommendations to improve postprandial glucose levels through behavioral changes. The proposed method is a multiarmed bandit based on a two-stage reward prediction model, where an action is a combination of the amount of carbohydrate intake and postprandial walking duration, and the reward is the postprandial glucose level. We specifically realized the reward prediction for each action by predicting the behavioral responses to an action and then predicting the postprandial glycemic response using the predicted responses. From the simulation experiment, it was demonstrated that action selection using the proposed method significantly increased the cumulative reward compared to the default action selection. Simultaneously, we found that the most beneficial setting of the proposed method is adopting a policy to maximize the predicted rewards from the beginning, together with robust parameter learning using prior distributions for the prediction models. Finally, based on this finding, an intervention experiment was conducted on 6 healthy participants, and it was shown that the application of the proposed method provided preliminary evidence that it could improve postprandial glucose levels along with the behaviors of carbohydrate intake and postprandial walking. This initial real-world investigation, while valuable for understanding the practical considerations and informing the design of larger trials, underscores the need for extensive long-term validation in diverse patient populations, ensuring user safety and promoting sustained engagement.