This study consisted of 60 digital, personalized N-of-1 crossover trials comparing the effects of 3.0 mg and 0.5 mg of melatonin versus placebo for poor sleep. The trial was comprised of a 2-week baseline assessment period and a 12-week intervention period. Before beginning the study, participants were provided with the necessary supplements, 3 Nomi by SMRxT smart pill bottles to contain the supplements, and a Fitbit Charge 5 (Google) activity tracker. The Nomi smart pill bottle is a medication adherence device that uses a weight sensor and cellular signals to measure and transmit medication adherence data in real-time.

During the 2-week baseline assessment period, participants did not take any melatonin or placebo pills and were asked to manage their sleep as they usually would. Participants were also encouraged to wear their Fitbit activity tracker and complete ecological momentary assessment (EMA) measures of fatigue, pain, concentration, confidence, mood, and stress delivered by text message 3 times daily, at randomized times. Each morning, participants completed an additional text-message survey assessing their sleep quality from the previous night.

Following baseline, participants were randomized into 1 of 2 treatment orders comprised of six 2-week intervention blocks of 3.0 mg melatonin, 0.5 mg melatonin, or placebo pills. Following the intervention, each participant was provided with a report summarizing the results of their personalized N-of-1 trial. At the end of the study, participants completed a survey containing satisfaction measures and the System Usability Scale (SUS). An example of the participant timeline can be found in Figure S1 in Multimedia Appendix 1. Study recruitment began in April 2022 and was completed in May 2023. Additional details about the study design can be found in the trial protocol [34].

This was a digital, remote clinical trial. Participants were recruited digitally from several states within the United States. Recruitment methods included internet and social media posting such as Facebook (Meta) and Google advertisements. In addition, online posts to communities on platforms such as Reddit, Craigslist, and LinkedIn were used to appeal to individuals with poor sleep quality. With 93% of the US adults reporting using the internet, 69% reporting ever using Facebook, and 18% reporting using Reddit, usage of online recruitment methods significantly broadened the potential span of our outreach [35]. Emails and advertisements were also distributed through the Northwell Health system, consisting of approximately 85,000 employees [36].

All potential participants identified as having poor sleep for at least the past month using the Insomnia Symptom Questionnaire [37]. Poor sleep for this study was defined by self-reported difficulty falling asleep, staying asleep, and/or feeling refreshed by sleep 3 to 7 times per week, with 1 or more of those symptoms lasting for a period of at least 4 weeks. Participants were also required to report that 1 or more poor sleep quality symptoms bother them “quite a bit” or “always” to meet eligibility criteria.

Participants each received 3 Nomi pill bottles labeled A, B, and C. Bottles labeled with “A” contained 28 pills of 3.0 mg melatonin, bottles with “B” contained 28 pills of 0.5 mg melatonin, and bottles with “C” contained 28 pills of a placebo. All melatonin and placebo doses were provided for this study by Pure Encapsulations LLC, a National Sanitation Foundation– and Good Manufacturing Practices–registered laboratory with a long history of contributions to clinical research [38-40]. The placebo was 100% cellulose powder contained in a capsule, and the capsules of each placebo or supplement were color-matched and designed to be identical. Participants were blind to which capsule was stored in each bottle. The authors selected 3.0 mg and 0.5 mg doses to compare higher and lower doses of melatonin. Previous research has identified 0.3 to 1.0 mg as a low dose of melatonin capable of improving sleep [41]. Doses ranging between 2.0 and 6.0 mg of melatonin have also been found to be effective for improving sleep [42,43]. The doses of 3.0 mg, 0.5 mg, and placebo were selected in this study as we were able to obtain identical capsules for all 3 pills.

Participants were assigned 1 of 2 treatment orders: ABCCBA and CBAABC. Participants received an intervention schedule indicating which bottle labeled “A,” “B,” or “C” they should use for which weeks in the trial. During the intervention, participants received nightly reminder text messages instructing them to take a pill from the appropriate bottle (eg, “Take a pill from Bottle A”). All participants were discouraged from taking other melatonin products while enrolled in the trial. In total, 30 participants were assigned to each treatment order with study randomization conducted by the statistical team using Python to help randomize a list of each treatment order of 30. Randomization allocation was concealed from the study coordinators and investigators, and treatment orders were assigned by the statistical team only after minimum protocol adherence thresholds were achieved by each individual participant during baseline. The participant timeline in Figure S1 in Multimedia Appendix 1 shows these 2 treatment orders. During all study periods, participants were encouraged to wear the Fitbit device 24 hours a day and answer survey measures.

During the baseline period, the study team assessed participant adherence to the protocol. Adherence was evaluated using the Fitbit tracker and survey completion. Participants who wore the Fitbit for 12 or more hours per day and for at least 180 minutes while sleeping were defined as adherent to Fitbit wear [44,45]. Submission of a survey before the end of the 2-week baseline period was defined as adherent to survey completion. Participants who did not demonstrate adherence to 80% of study measures during baseline did not progress to the intervention.

Participants who completed the trial received a 15-page personalized report summarizing their trial results by showing the impact of 3.0 mg melatonin, 0.5 mg melatonin, and placebo on their Fitbit-recorded sleep duration. The participant report also presented summaries and results for other sleep- and wellness-related outcomes including fatigue, stress, pain, and physical activity that may help the participant identify whether melatonin is useful to help their sleep or other well-being goals. Participants were also provided with summaries of their adherence to study measures and to the intervention. All reports were generated automatically but were reviewed by multiple study team members before being sent to participants. A similar version of the participant report in a series of personalized trials for chronic lower back pain is described and displayed in a study by D’Angelo et al [46].

The primary outcome of this study was feasibility measured using the 10-item SUS [47] and acceptability as measured by an 18-item satisfaction survey. The SUS is a validated questionnaire that assesses the usability of a system with 10 items on a 5-point Likert scale rated from “strongly disagree” (1) to “strongly agree” (5) [5,48,49]. To obtain a total score, item scores are multiplied by 2.5 and summed to generate a total score ranging from 0 to 100, with higher scores indicating a greater level of usability. The SUS has been validated in multiple contexts [48,49] and has also been used for previous series of personalized N-of-1 trials [45,50]. As participant usability of a personalized trial is essential for a personalized N-of-1 design to be successful, the SUS is used as a feasibility measure. Analyses of the SUS include mean total scores and individual item scores with SDs. Mean SUS scores of ≥68 have been identified as “acceptable” levels of usability while scores ≥80 have been identified as “excellent” levels of usability [51]. We compared participant scores from this study to these established standards as well as SUS scores for other comparable trials to identify the feasibility of this series of personalized trials.

Participant satisfaction with the trial was measured using a 6-item survey rated on a scale of 1 (“not very satisfied”) to 5 (“very satisfied”). This survey measures participant satisfaction with specific elements of the trial (eg, the Fitbit activity tracker, SMS text messaging for reminders and surveys, and the personalized trial design). Participants also received a 7-item survey assessing their experience with the trial overall (eg, trial onboarding and informational videos) rated on a scale of 1 (“strongly disagree”) to 5 (“strongly agree”). Means and SDs for each item were represented, and frequencies of each participant response were depicted graphically. All satisfaction survey measures were structured similarly to previous series of personalized N-of-1 trials [44,45] but had the specific text altered to fit this trial design.

The secondary outcomes included sleep duration measured using the Fitbit wearable activity tracker and sleep quality measured using a modified version of the consensus sleep diary (CSD). Nightly sleep duration was calculated using Fitbit’s proprietary algorithm with participant information from the Fitbit Charge 5 device. Participant Fitbit activity tracker data were retrieved using the online portals N1thrive and Fitabase. These portals allow for extraction of minute-to-minute activity data as well as daily summaries of outcomes including sleep duration and steps count. Although previous meta-analyses indicate that Fitbit activity tracking devices may overestimate sleep duration [52], the personalized N-of-1 design allows for within-participant comparisons of sleep durations between intervention periods. The CSD is a validated survey instrument used to assess and track intervention effects on sleep symptoms, quality, and satisfaction [53]. This study used a modified version of the CSD to assess perceived sleep quality using a 5-point Likert scale rated from “very poor” to “very good,” with higher scores indicating better sleep. The CSD was delivered daily via text message each morning within an hour of the participant’s self-reported typical wake time. Means and SDs of participant sleep duration and sleep quality were reported for baseline and each of the intervention periods (3.0 mg melatonin, 0.5 mg melatonin, and placebo). Paired samples t tests were used to compare mean participant sleep duration and sleep quality during the intervention periods to baseline. For analyses of self-reported sleep quality, participant responses were change log-transformed to comply with the assumptions for a normal distribution in a paired samples t test.

Additional secondary outcomes measured included self-reported EMA fatigue ratings and self-reported EMA stress ratings. EMA fatigue and stress were assessed using single-item measures adapted from the numeric pain rating scale [54]. These EMA measures were administered 3 times daily via SMS text message asking participants to rate their fatigue and stress in the current moment on a scale of 0-10. The EMA assessments were delivered randomly throughout the participant’s self-reported sleep and wake times. Paired samples t tests were used to compare ratings of EMA fatigue and EMA stress between baseline and each of the intervention periods (3.0 mg melatonin, 0.5 mg melatonin, and placebo).

The effect of each intervention (3.0 mg melatonin, 0.5 mg melatonin, and placebo) on sleep duration, EMA fatigue, and EMA stress was examined using generalized linear mixed models with an autoregressive model, which was used to account for possible linear trends and autocorrelation between outcome ratings over time. These models provided estimates of the effects of each treatment on each secondary outcome across all participants. We conducted additional analyses at the individual participant level to identify the effects of each intervention for each participant. These regressions also used generalized least squares analyses with autoregressive models. Using these regressions, we were able to generate estimates of the effect of each intervention on each outcome for every participant individually. Due to low levels of variation in self-reported sleep quality, participant ratings were averaged over each week, and a linear mixed model was conducted using these weekly averages for the overall sample and for each individual participant.

To support the utility of conducting a series of N-of-1 trials of melatonin for poor sleep, we examined the HTE for both 3.0 mg and 0.5 mg of melatonin. The HTE for both doses of melatonin was examined for 4 outcomes (ie, sleep duration, sleep quality, EMA fatigue, and EMA stress) using methodology previously developed for analyses of N-of-1 trials [27]. Briefly, we modeled the effects of 3.0 mg and 0.5 mg of melatonin relative to placebo on outcomes using linear mixed models with first-order autocorrelation autoregressive model. We then compared 2 models of the effects for 3.0 mg and 0.5 mg of melatonin versus placebo (1 with a random intercept only and 1 with random slopes) using a likelihood ratio test. The random slope model accounts for HTE, whereas the random intercept model does not. The results of this likelihood ratio test were then used to determine whether HTE exists for a particular treatment on a particular outcome. We also calculated an index of heterogeneity to quantify the HTE for each treatment on each outcome [27,55]. This index is used to interpret the utility of N-of-1 trials; higher values indicate that an N-of-1 trial may be more useful for examining the effects of a particular treatment on a particular outcome.

The sample size of 60 participants was chosen to ensure a sufficient number to obtain a preliminary assessment of the feasibility of this series of personalized trials of melatonin therapy for sleep. With 60 participants receiving the intervention and use of a single sample binomial test with α set to 2.5% significance, we will have approximately 90% power if 70% (42/60) of participants complete the trial. With 57 participants completing the outcome, we have greater than 90% power.

In total, 654 participants were screened for the study, yielding an enrolled sample of 60 participants. Characteristics of the enrolled sample can be found in Table 1. Full details regarding study screening and enrollment can be found in the CONSORT (Consolidated Standards of Reporting Trials) diagram (Figure 1). Of this sample, 57 participants completed the primary outcome of the SUS. In this sample of 57 individuals, the mean age was 41.6 (SD 12.8) years, sex was 53% (n=30) female, 58% (n=33) White, and 18% (n=10) were Hispanic. Participant characteristics did not differ between the 2 treatment orders. Full characteristics of this sample can be found in Table S1 in Multimedia Appendix 1.

A total of 57 participants completed the primary outcome of the SUS. The mean SUS score across all participants was 76.27 (SD 17.09, Table 2), indicating that this trial had an acceptable level of usability (SUS total score≥70) [51]. Scores on the SUS ranged from 30 to 100 (Figure 2), indicating that several participants (13/57, 22.8%) found the usability and learnability of this series of personalized trials to be less than acceptable, while others (22/57, 38.6%) rated the usability of this series of personalized trials to be excellent. The most positively rated items from the SUS were “I do not think that I would need the support of a technical person to be able to use this system” (mean 3.28, SD 0.94) and “I felt very confident using the system” (mean 3.25, SD 0.81, Table 2). The least positively rated items were “I found the various functions in this system were well integrated” (mean 2.88, SD 0.76) and “I did not think there was too much inconsistency in this system” (mean 2.81, SD 0.97). Full distributions of responses for each item on the SUS can be found in Figure 3.

Participant satisfaction ratings of the trial were generally high on a 5-point scale (Figures 4 and 5). The most positively rated items on the satisfaction survey were “use of the Fitbit to track your activity and sleep” (mean 4.58, SD 0.94) and “presentation of your results” (mean 4.51, 0.89). The least positively rated items on the satisfaction survey were “I enjoyed receiving daily text message prompts and surveys on my cell phone” (mean 3.12, SD 1.23) and “text messaging for survey questions” (mean 3.47, SD 1.48).

In addition, the majority of participants (36/57, 63%) stated they would “strongly recommend” the trial to others ( Multimedia Appendix 2). Only 3.5% (2/57) of participants stated they “would not recommend” the personalized trial to others. Participants mostly felt the trial was helpful; 12% (7/57) found the trial “extremely” helpful, 40% (23/57) rated the trial “very much” helpful, and 39% (22/57) found the trial “somewhat” helpful (Multimedia Appendix 3).

At baseline, participants had 399 (SD 53) minutes of sleep per night, indicating participants started with roughly 6-and-a-half hours of sleep per night. Participant sleep durations were comparable during placebo (mean 398, SD 59 min), 0.5 mg melatonin (mean 398, SD 54 min), and 3.0 mg melatonin intervention periods (mean 402, SD 61 minutes) (Table 3). Paired samples t tests also showed no significant difference in mean sleep duration from baseline between placebo (P=.80), 0.5 mg melatonin (P=.90), and 3.0 mg melatonin (P=.70) intervention periods. Regression models in the pooled sample of participants showed no significant overall effects of 3.0 mg melatonin versus placebo (B=4.73, 95% CI −6.85 to 16.30), 0.5 mg melatonin versus placebo (B=1.23, 95% CI −8.28 to 10.75), and 3.0 mg melatonin versus 0.5 mg melatonin (B=3.40, 95% CI −6.24 to 13.03) (Table 4). Regression estimates for these treatment effects with 95% CIs can be found for overall sample and for each individual participant in Figure 6.

During baseline, participants reported that their sleep quality was average (mean 3.13, SD 0.64 on a scale of 1 to 5, Table 3). Paired samples t tests also showed significant increases in mean sleep quality from baseline between placebo (P<.001), 0.5 mg melatonin (P<.001), and 3.0 mg melatonin (P=.005) intervention periods. However, regression models in the pooled sample of participants showed no significant overall effects for 3.0 mg melatonin versus placebo (B=−0.04, 95% CI −0.13 to 0.06) and 0.5 mg melatonin versus placebo (B=0.04, 95% CI −0.05 to 0.13). There was a small but statistically significant difference between 3.0 mg melatonin versus 0.5 mg melatonin (B=−0.09, 95% CI −0.18 to −0.01; Table 4). In Figure 7, regression estimates for these treatment effects with 95% CIs can be found for the overall sample and for each individual participant.

During baseline (mean 3.85, SD 1.95), participants also reported relatively low levels of fatigue (Table 3). Mean participant levels of fatigue decreased during placebo (mean 3.44, SD 2.00), 3.0 mg melatonin (mean 3.58, SD 1.98), and 0.5 mg melatonin periods (mean 3.49, SD 2.07). In addition, paired sample t tests showed small but statistically significant reductions in EMA fatigue between baseline and both 0.5 mg melatonin (mean difference −0.36, 95% CI −0.72 to −0.01; P=.045) and placebo periods (mean difference −0.41, 95% CI −0.77 to −0.04; P=.03). No significant differences were found between baseline and 3.0 mg melatonin periods in paired samples t tests (P=.08; Table 3). Regression estimates comparing EMA fatigue values between 3.0 mg melatonin (B=0.08, 95% CI −0.11 to 0.26) and 0.5 mg melatonin (B=0.04, 95% CI −0.12 to 0.20) versus placebo periods were also not statistically significant (Table 4).

Participants reported relatively low levels of stress during baseline (mean 3.19, SD 1.86), placebo (mean 3.33, SD 1.96), 0.5 mg melatonin (mean 3.29, SD 1.86), and 3.0 mg melatonin periods (mean 3.34, SD 1.78). Results from paired samples t tests did not show any significant difference between baseline and intervention periods (Table 3). Furthermore, there was no difference between treatment periods in the autoregression models conducted (Table 4). Figures 8 and 9 present regression estimates for these fatigue and stress, respectively, with 95% CIs for the overall sample and for each individual participant.

Analyses of the HTE for placebo, 0.5 mg melatonin, and 3.0 mg melatonin showed that a random slope model was a better fit for the data than a random intercept for modeling the outcomes of sleep duration, EMA fatigue, and EMA stress (Table 5; likelihood ratio P value<.001). The SD of the treatment effect for 3.0 mg versus placebo and 0.5 mg melatonin versus placebo was also larger than the treatment effects. However, the indices of HTE for 3.0 mg versus placebo and 0.5 mg versus placebo were low for sleep duration, EMA fatigue, and EMA stress. These indices reflect the potential power for future personalized N-of-1 trials of each intervention for each outcome [27]. Based on these indices, future series of personalized N-of-1 trials will have statistical power ranging between 7% and 23% depending on the intervention and outcome (Table 5). This suggests that although a random slope model is more useful than a random intercept model for examining the effects of each dose of melatonin on these 3 outcomes, participants did not have a heterogeneous treatment response. Based on these preliminary findings, further N-of-1 trials of melatonin for sleep duration, EMA fatigue, and EMA stress are unlikely to be necessary in this population.

Analyses of the HTE for self-reported sleep quality were unable to be completed due to a lack of variance in participant responses. While a formal analysis of the HTE was not conducted for sleep quality, this lack of variance in participant responses also suggests that treatment effects of melatonin on sleep quality were not heterogeneous.

The results from this trial show that digital, remote personalized N-of-1 trials of melatonin for poor sleep are both feasible and acceptable to participants. However, the melatonin interventions were not significantly associated with changes in measures of poor sleep (including device-measured sleep duration and self-reported sleep quality). This suggests that, although this system of personalized N-of-1 trials can be successfully implemented among individuals with poor sleep, melatonin may not be an ideal intervention for a personalized N-of-1 trial design.

Participants rated the feasibility (defined using the SUS) as being acceptable. Although the mean SUS scores were lower than the previous series of personalized trials conducted by the authors [45,50], participant ratings were still higher than many other interventions with sleep as the primary outcome. In fact, usability ratings were comparable or better than previous interventions to improve sleep, including blue light therapy for sleep among patients with Parkinson disease (SUS median70, IQR 57.5-80.0) [56], a digital sleep diary intervention (SUS mean 74.6, SD 9.4) [57], a mobile health (mHealth) intervention to improve sleep among shift workers (SUS mean 62.7, SD 12.7) [58], an mHealth app to support the use of continuous positive airway pressure (CPAP) among patients with obstructive sleep apnea (SUS mean 79.4, SD 19.3) [59], an mHealth sleep monitoring device for management of sleep symptoms among older individuals (SUS mean 75.5, SD 10.2) [60], and a web-based decision aid for identifying treatment for obstructive sleep apnea (SUS mean 78.2, SD 15.1) [61]. Interestingly, Fitbit wearable activity trackers tend to have acceptable levels of usability on the SUS in previous research [62,63].

What is perhaps more interesting is the fact that both 3.0 mg and 0.5 mg doses of melatonin did not differ significantly from placebo in terms of Fitbit device-recorded sleep duration, self-reported sleep quality, and EMA measures of fatigue and stress. Furthermore, we did not find heterogeneity in the treatment response among participants. Notably, however, there were small but statistically significant differences in sleep quality and fatigue between the baseline and intervention periods. In addition, meta-analyses of the effect of melatonin on sleep duration have found an average increase of 12 minutes [14] and a significant improvement in sleep quality [20]. Both of these meta-analytic studies found heterogeneity in the treatment effects of melatonin [14,20]. Although this heterogeneity was identified in both meta-analyses using Cochran Q statistic (a method that has been called into question in recent literature) [64], our study found homogeneous, nonsignificant effects of melatonin versus placebo for many of our outcomes. This is especially surprising since the authors have previously found interventions used in series of N-of-1 trials to treat chronic lower back pain and fatigue had small overall effects but high levels of heterogeneity in treatment response for some interventions and outcomes [65,66].

While this finding is surprising, it is not unprecedented. Previous research has found that melatonin is most effective among individuals dealing with sleep restriction or altered sleep schedules (eg, individuals with jet lag and shift workers) [67]. This is logical given melatonin’s role in the regulation of circadian rhythm. However, the causes of poor sleep can be variable and complex. Lifestyle factors (eg, smoking, diet, and physical activity) [4,13], individual characteristics (eg, sex, BMI, socioeconomic status, and medical comorbidity) [4,13,68,69], and environment factors (eg, having children, marital status, irregular work hours, and noise) [4,70-72] can all influence sleep quality and duration. While melatonin may address how some of these factors impact circadian rhythm, melatonin supplements’ ability to address how environmental factors, such as noise, influence sleep will be difficult. However, we would hope that some participants in this trial would have poor sleep that was linked to a disruption of circadian rhythm. The lack of heterogeneity in our effects suggests that melatonin may not be a useful intervention for personalized N-of-1 trials that broadly recruit individuals with self-reported poor sleep.

This study has several notable strengths. First, this series of personalized trials was implemented by a research team with extensive experience in the design and conduct of personalized N-of-1 research. The statistical team supporting the work of this trial has also developed methods for specifically analyzing personalized N-of-1 data and calculating HTE [27,73]. We believe this expertise has contributed to a well-designed and well-conducted series of personalized N-of-1 trials. Second, the use of digital, remote interventions, data collection, and dissemination of findings also means that the design of this study may be easily replicable. Third, feasibility and participant satisfaction with the trial were assessed using multiple metrics, including the SUS, a validated measure of usability used to evaluate other digital interventions. This measure was supplemented by satisfaction surveys that were tailored specifically to the design of this trial. These measures have allowed us to compare our series of personalized N-of-1 trials with other digital interventions for sleep while also collecting feedback specific to the design of this study.

Although this trial does have many strengths, there are also some limitations to consider. First, the outcomes used in this trial are all self-reported measures or measurements from a commercially available device. Both sleep duration [52] and sleep quality [74] can be measured using more accurate and precise measures to better capture dimensions of poor sleep. However, this trial uses readily available measures and devices designed to reduce participant burden. By using accessible and lower-cost tools to capture dimensions of poor sleep, the goal was to increase the feasibility of personalized N-of-1 trials for sleep. Regardless, more accurate measures of sleep duration and sleep quality may capture nuances of poor sleep that the measures in this trial do not. Second, participants entered the trial with generally high levels of self-reported sleep duration and sleep quality. The mean baseline sleep duration for the sample was 399 minutes. Although this duration is lower than the 7‐10 hours of sleep duration recommended for adults [75], sleep durations of ≤5 hours per night are mostly associated with adverse outcomes [76]. However, this average sleep duration falls within the definition of “short sleep” [76,77]. In addition, participants reported average levels of sleep quality at baseline (mean 3.13, SD 0.64). Individuals with poorer levels of sleep during baseline may have benefitted more from melatonin during the intervention period. Third, this trial only examined 2 doses of commercially available melatonin (3.0 mg and 0.5 mg). As discussed previously, doses of melatonin ranging from 0.3 mg to 6.0 mg have been used to successfully improve poor sleep [41-43]. The doses used in this trial may not have been optimal melatonin levels to influence poor sleep. Another limitation with commercially available melatonin is that some evidence suggests that the actual dose in melatonin supplements may vary significantly between lots [78]. We aimed to minimize this limitation by purchasing supplements from a single manufacturer in a single lot. A fourth and final limitation is that this trial was not designed to enroll a representative sample of individuals with poor sleep. Although there was a fairly diverse sample, to definitively say that the results from this series of personalized trials will generalize to all populations is impossible.

The findings of this study show that personalized N-of-1 trials for poor sleep are highly feasible and acceptable. However, our results show that melatonin supplements did not significantly improve sleep duration or sleep quality. Furthermore, these treatment effects were not heterogeneous between participants. This suggests that, despite the increased prevalence of melatonin use in the United States [79], melatonin may not be broadly applicable as a treatment for poor sleep, especially in personalized N-of-1 trials.