This preplanned process evaluation is part of an RCT for which the methodological details are presented in Multimedia Appendix 1 [32-38]. In short, it assessed the effectiveness of a purposefully designed mobile gaming app (Textbox 1). It was developed to decrease the loneliness of older adults (aged 65 years or older) by facilitating valuable playful interaction moments. The app conducted monthly questionnaires for 12 months on, for example, loneliness, mobility, and well-being. Sign-up, onboarding, and the intervention were done entirely digitally. Using 3 different app versions, we compared three conditions: (1) games eliciting interaction about participants’ personal lives; (2) games eliciting sharing generic stories; and (3) no games at all. With this design, we simultaneously aimed to assess the effectiveness of a gaming app and the effectiveness of the personalized aspect of the games. The study focused on adults aged 65 years or older (the main group). To allow the older adults a network of people to play with, all younger adults (aged between 18 and 64 years; the side group) were allowed to sign up, play, and invite others as well. The side group received a brief questionnaire to obtain information on whom the main group played with.

In the mixed methods process evaluation, we used quantitative data collected during the RCT. Furthermore, we conducted postintervention interviews and focus groups with RCT participants and welfare organization representatives to collect their experiences with the RCT and the app.

The RCT with process evaluation was reviewed by the research ethics committee of the Radboud university medical center (file 2020-6884). It did not fall within the remit of the Dutch Medical Research Involving Human Subjects Act (WMO). The ethics committee approved the study based on the Dutch Code of Conduct for health research, the Dutch Code of Conduct for responsible use, the Dutch Personal Data Protection Act, and the Medical Treatment Agreement Act. The RCT is registered on ClinicalTrials.gov (NCT04733898).

For the RCT, participants gave informed consent in-app before data collection. For the process evaluation, participants received the information letter containing study details by email before the interview. They gave verbal informed consent at the start of the interview or focus group.

This study described the mixed methods process evaluation of an entirely remotely conducted gerontological RCT assessing a newly developed digital gaming app for strengthening the social network of older adults. The evaluation showed that it is difficult to keep participants engaged for longer periods when conducting a digital trial with older adults without in-person contact. Remote recruitment and only informed consent acquisition started by social media, newsletters, and mailing lists of welfare organizations were insufficient to reach the desired study population. It resulted in low sample sizes, low questionnaire adherence, and little gameplay, consistent with recruitment and attrition challenges in previous studies [3,9,39].

The results reveal several difficulties encountered in this study and digital interventions for older adults in general [40,41]. Less than half of the people signing informed consent were aged 65 years or older and eligible for our main sample, and we only managed to reach 1.5% (33/2220) of our sample size goal. Moreover, we lost more than half of the sample in the app download step and another half during chatbot interaction and signing informed consent. It indicates that a completely digital onboarding process with relatively many steps results in a very low inclusion rate. A previous feasibility study of this app [31] already showed the enormous loss of people during digital recruitment, which, combined with this study, indicates that a completely digital onboarding process for older adults might not be feasible.

The literature stresses the importance of cultivating a relationship with local organizations and face-to-face contact with participants [40,42,43], all of which, in this study, were complicated by social restrictions and web-based recruitment in general. Although many local and national aging, welfare, and volunteer organizations shared study information through their newsletters, mailing lists, or websites, building a substantive relationship with organizations and potential participants was challenging. Social media recruitment seems promising [44], yet it still requires a precise target group definition and an extensive initial reach to have sufficient participants. Retention also poses a challenge due to the need to build a relationship with participants [23,24].

In our RCT, 96% (73/76) of the participants answered the baseline questionnaire, declining to 24% (18/76) after just 1 month. This reveals a difficulty in keeping participants engaged in a remote intervention, which is in line with previous research [9,31,45]. A recent qualitative evaluation of adherence in a depression intervention showed that intrinsic motivation, time availability, and the relative value of the intervention were predictive of greater adherence [46]. Furthermore, a web-based friendship course against loneliness in older adults revealed that intervention dropout decreased with engagement [9], which the authors defined as using the intervention as instructed. These findings suggest that engagement and intrinsic motivation are important predictors of adherence and effectiveness in socializing interventions [47]. It might have been the case that participants in the RCT did not feel intrinsically motivated, possibly due to the low availability of people to play games within the app or not feeling the games were “for them,” as they did not feel lonely [30]. A possible disconnection between the app’s goal and the players’ sentiments might have influenced the game and questionnaire adherence.

Another important aspect in improving game adherence is self-efficacy. Enjoying easy-to-use games positively affects self-efficacy and social interactions [48,49]. Providing older adults with learning opportunities that show them the added benefit of technology to their social connections is an important predictor of motivation and skill [49,50], which in turn is associated with health improvement benefits [51]. Furthermore, social media self-efficacy is associated with lower levels of loneliness [52,53]. Following this, Chen and Gao [53] suggest providing older adults with offline information and communication technology learning to strengthen their perception of connectedness and digital skills. This aligns with the findings of our process evaluation and emphasizes the importance of repeated practice sessions for effective digital gaming adaptation. Primarily due to COVID-19 lockdown restrictions, this study did not include such practice sessions. However, we recognize the crucial role they may play in ensuring the long-term engagement of older adults in digital interventions.

In terms of the games, we found that participants enjoyed the games for a while but felt that they lacked challenge in the long term. Older adults increasingly participate in digital technology and games [54]. They play games for various reasons, for example, social interaction, competition, challenge, arousal, relaxation, and passing the time [55-58]. Qualitative literature suggests that social games for older adults might be most effective when players are matched with the right, balanced team of partners, share high scores, and incorporate vicarious play [59]. Competition aspects were deliberately not added to our app, as they might have shifted attention away from social interaction. However, this might have created a discrepancy between the players’ expectations and the app’s goal, decreasing motivation to play. Furthermore, the games needed more challenge for more digitally skilled individuals, as indicated in the interviews, while simultaneously being too difficult for others, equating to an unfitting amount of challenge for most participants.

The app’s initial goal was to encourage intergenerational contact. Game design for different generations is possible but has specific co-design requirements [60-62]. One can best achieve intergenerational contact by building on the shared ideas of both generations, using the skills of both younger and older adults (eg, by letting children assist older adults), and allowing both generations to spend time together [60]. Our app incorporated games known to older adults, an interface resembling apps familiar to older and younger generations, and used language believed to appeal to both. These design decisions might have worked if older adults had played with younger children. However, in this study, older people played by themselves in a child-friendly environment that might have felt childish.

This process evaluation gives suggestions to improve gameplay, remote recruitment, and data acquisition. Given the difficulty in reaching older adults and the limited effectiveness of loneliness interventions, process evaluation is vital to inform future interventions. We minimize the use of valuable resources by allowing future interventions to apply our lessons learned.

The low response rate for the process evaluation may be a limitation. It proved difficult to recruit respondents, even after multiple contact attempts. The responses were homogeneous, indicating data saturation. However, we cannot confidently conclude this, as we failed to represent the complete sample of RCT participants and organizations in our qualitative evaluation. Participants who declined indicated they quit RCT participation long ago, were uninterested, or felt they were not of added value due to limited playtime. Furthermore, our sample’s relatively high number of organization representatives could have slightly biased our results. These representatives are generally more knowledgeable about recruiting older adults than their actual game experience.

Game design and evaluation is a multistage iterative process in which small-scale feedback rounds continuously enhance the product. On the other hand, a large trial demands that the interventional product remains the same throughout the study period. This means that we could not update the app once inclusion started, apart from necessary bug fixes. Furthermore, using the app as a scientific intervention allowed for rich and systematic data collection to evaluate gaming behavior but also required aspects not usually included in a game, like questionnaires and informed consent procedures. Therefore, it asks for a pretrial design process to obtain good feedback on design choices while thoroughly assessing whether the trial intervention is suitable for large-scale research through feasibility and pilot testing.

This study showed that both older adults and organizations are optimistic about the concept of a gaming app for social connectedness. However, merely relying on web-based recruitment is insufficient to reach lonely older adults. Furthermore, a digital intervention and onboarding process create challenges in participants’ understanding, engagement, and motivation.

The app should be accessible, easy to use, understandable, have various difficulty levels, and have minimal functionality to make it accessible for a digitally low-skilled population. Future study designs should start on a small scale, with few questionnaires and no follow-up measurements, and subsequently build up to more complex designs, thereby avoiding wasting time and resources. Process analysis should always be preplanned, as it is crucial to improving eHealth applications and evaluations. In conclusion, social health games may help to strengthen the social connectedness of older adults, facing natural age-related declines in their social networks. However, a scientifically sound evaluation is still needed, and the most effective set-up of such interventions remains to be developed in collaboration with the stakeholders.