We used the concept mapping methodology, originally introduced by Trochim [37], to identify (1) possible reasons that patients have according to therapists for not complying (fully) with the CBT homework and (2) game design elements that could help patients to comply more with CBT homework. According to Trochim and Kane [38], concept mapping is a methodology to organize information in a visual way. It combines specific analysis and data interpretation methods to produce maps so that interrelations in the information can be seen.

The concept mapping methodology according to Kane and Trochim [39] consists of six phases: (1) preparation, (2) generation, (3) structuring, (4) representation, (5) interpretation, and (6) utilization. In the (1) preparation phase, focal areas and criteria for participant selection are determined. In the (2) generation phase, selected participants brainstorm on the focal question and generate a list of items that will be used in the next phase. During the (3) structuring phase, participants individually organize the generated items in groups of perceived similarity. Thereafter, each item is rated on 2 or more domains. The representation phase (4) consists of entering the sorted and rated data into concept mapping software to make visual representations (concept maps). In the (5) interpretation phase, participants analyze the data qualitatively and discuss the results collectively. In the final phase, (6) utilization, participants discuss how the results relate to the focal question. 

The concept mapping meeting was held on March 31, 2017. Before the generation phase started, the facilitator explained the purpose and guidelines of the brainstorm session. Next, the researcher (HML) gave a general introduction to gamification, after which the generation process began.

First, the participants were asked to individually generate items based on focal question 1. The items were written down by the participants on their own sheet of paper. After this, the participants brainstormed on the same matter as a group. To maximize participation, therapists named one individually generated item, taking turns. This continued till all the items on the sheets of the therapists’ papers were named. When triggered by others’ items, all participants were allowed to name new items. When no new items were generated, the group brainstorming ended. All items were entered into the computer immediately and projected on a large screen so that every participant could see all the generated items in order to prevent duplicate items. The same procedure was followed for focal question 2: What game design elements could be important to help clients do their homework better or more frequently?

In the weeks that followed the concept mapping meeting, the data that were obtained from the previous phases were analyzed by the researchers (HML and Jeroen Ruwaard) using The Concept System Global Max online application [43]. The application assigns a unique number to the generated items. Several maps were generated, based on multidimensional scaling [44] with a 2D approach [45]. The software also generated the initial labels for the clusters of items. The researchers examined these cluster solutions that were considered to be logically grouped. Each potential cluster solution was discussed among the researchers. Eventually, the researchers picked 1 cluster solution that was seen as the most logical way of categorizing the generated items.

In the interpretation phase, the generated maps were mailed to the participating therapists. They were asked whether they concurred with the proposed clusters and cluster names. When not, participants were asked to make suggestions.

After collecting the feedback from the participants in the previous phase, the researchers discussed how the data could be used to answer the research questions and made recommendations for how to use this information in future research.

This human participants study is compliant with the ethical rules and guidelines of the affiliated institution and has received a retrospective institutional review board exemption from the Medical Research Ethics Committee of Amsterdam University Medical Centers (review or reference 2025.0507). All participants consented to participation in the study and gave permission to use the collected data for both current and future analyses. The data collected were anonymized before analysis to protect the participants’ privacy. Participants received a gift card with a value of 15 euros (US $15.996 as per the exchange rate on March 31, 2017) as compensation for their participation.

In this study, 7 therapists (6 female and 1 male) participated in the concept mapping procedures. The concept mapping session, consisting of the generation and structuring phases, took 3 and a half hours in total.

The generation phase resulted in 29 items for focal question 1 and 38 items for focal question 2.

The results of the sorting part of the structuring phase consisted of 41 piles of items for focal question 1 and 40 piles of items for focal question 2. The rating part of the structuring phase resulted in 2 forms with ratings on severity for the items generated for focal question 1 and importance for items generated for focal question 2.

In concept mapping, several different maps are generated to visualize the data so that interrelationships in the information can be seen. The representation phase resulted in 15 data visualizations.

The generated maps are based on a common multidimensional scaling arranged structure named point map (Figures 1 and 2). Each point represents a generated item. Note that the items’ exact locations on the map have no meaning. The distance between the items illustrates the degree of similarity between them. Items that are closer to each other on the map were sorted together more often by participants. For example, in Figure 1 the items 6 (boring or no fun), 14 (time consuming), and 23 (little immediate reward) were considered related by the participants as can be seen by their relative proximity on the map, while items 11 (therapist does not emphasize importance) and 20 (cognitive problems) were considered less similar by the participants and thus further away from each other. In Figure 2, items 2 (reminder), 14 (visualize pitfalls), and 24 (visualize goals) are relatively close to each other on the map and thus can be considered more similar by the participants. The distance between items 1 (rewards of different degrees), 25 (pop-ups), and 29 (invite friends) on the maps shows that the participants considered them less similar.

Tables 3 and 4 show all of the generated items on focal questions 1 and 2, respectively, and their average ratings on homework and game element outcomes. Each item has a unique identification number. The generated items are grouped into clusters.

In the point-cluster maps, it can be seen that a 6-cluster solution has been chosen for the reasons not doing homework, and a 5-cluster solution for the game design elements (Figures 3 and 4, respectively). These cluster solutions were chosen by the researchers and participants based on what fitted the data best. Each cluster has a label that is the best-fit. The cluster labels were chosen by the software first. Then the researchers and participants examined the cluster labels in relation to the items in each cluster. Cluster labels that were regarded as mismatch, were replaced by new labels that the researchers and participants agreed upon.

Cluster rating maps can be used to display these ratings as well (Figures 5-9). The average ratings of all the items of the same cluster for all participants together are indicated by layers. The map of severity (Figure 5) shows that symptoms unrelated internal factors (such as, physically not capable), external factors (such as, no time due to crisis situations), and motivation (such as, not aspiring therapy goal) were rated highest and thus perceived as the most important reasons why patients do not do their homework. The map of difficulty (Figure 6) shows that symptoms unrelated and related internal factors (such as, no self-confidence) were highest in rating meaning that those factors were perceived as most difficult to change for the patient. The map of importance (Figure 7) shows that facilitation (such as, giving reminder to do homework) and rewards (such as, achievements) were perceived as most important factors that could help patients to do their homework. As an example of giving reminder, one can think of games that will give you a reminder through a notification pop-up that some tasks need to be done in order to gain points or other profits, even when the game is not opened on a smartphone. In order to use achievements to help patients do their homework, one can think of a token or digital emblem to reward the patient for doing homework. The map of acceptance therapist (Figure 8) shows that facilitation and rewards were rated highest and were thus most accepted by the therapists to implement. The maps of acceptance patient (Figure 9) shows that therapists think that facilitation and rewards will be most acceptable by patients to be used to help patients do their homework.

In pattern matching, patterns of variables are compared, as illustrated in Figures 10-12. The relationship between severity and difficulty is presented in Figure 10. Again, symptoms unrelated to internal factors (eg, physically not capable) were seen as most important for patients not to do homework. At the same time, these “symptoms unrelated to internal factors” (Figure 10) were regarded as hardest for patients to change. Homework-related reasons (eg, homework necessities not at hand) were perceived as relatively easy to change by patients. In Figure 11, the relationship between importance and acceptability of the therapist is shown. Rewards are seen as both important and acceptable by therapists. Social elements, such as inviting friends to join or an in-game social network, were perceived as not important and not acceptable by therapists due to privacy and confidentiality issues. Although facilitation and fun were acceptable to therapists, their importance was comparatively low. Figure 12 shows the relationship between importance and acceptability of the patient. The results are similar to the previous figure.

The variables severity and difficulty were plotted against one another to explore their relationship (Figure 13). The analysis showed that item number 24 (troubles with Dutch language) was seen as most severe and most difficult to change. The variables importance and acceptance by the therapist were plotted against one another to examine their relationship, as illustrated in Figure 14. The plot shows that game design element 1 (rewards of different degrees) was seen as the most important and one of the most accepted by therapists to implement. The variables importance and acceptance by the patient (according to the therapists) were plotted against one another to examine their relationship, as can be seen in Figure 15. The analysis showed that game design element 1 (rewards of different degrees) was seen as the most important and one of the most accepted by patients (from the therapist’s perception) to implement.

The utilization phase resulted in a summary of results, limitations, and conclusions.

This study aimed to gain a better understanding of the potential of using game design elements in CBT homework to boost adherence. A total of 7 behavioral therapists participated in the concept mapping session. Symptoms unrelated to internal factors (such as physical incapability) were perceived as the most important reasons for patients not to do homework. These factors were perceived as difficult to change and might not be the responsibility of the therapist. However, next to these, external factors (no time due to crisis situations, do not want immediate environment to know, home-situation does not provide enough space or structure, and therapy out of picture due to daily activities) and motivation (such as too little intrinsic motivation, forgot to do the assignment, and being lazy) were perceived as relatively important reasons that cause patients not to do assignments. These barriers are similar to the findings of Bunnell and colleagues [46]. Homework-related characteristics, although not the most important, were seen as fairly easy to change. Facilitation (such as reminders, clear end goal, daily life integration, visualize goals, pop-ups, short, and easy and clear instructions) and reward (rewards of different degrees, feedback, achievements, daily bonus that increases in value, time limit, unlock new features, levels, repetition of assignment till mastery, buy upgrades, and overview of progress) were perceived as the most important gamification clusters that could help patients do their homework. Therapists think that facilitation and rewards will be most acceptable to be used in CBT homework and also to be most accepted by patients as well.

The result of this study, the perception of therapists that the game design element facilitation, such as reminders, can help patients make CBT homework, fits nicely with earlier findings that homework adherence is higher when less time passes after the assignment of homework [10]. Giving reminders soon after the assignment of homework can thus contribute to a higher homework adherence. Also similar to earlier research, facilitation such as giving clear instructions to patients is an important component of homework adherence [47,48]. In gamified form, it could mean a tutorial or explanation of the homework with immediate rewards. In many mobile games (such as FarmVille 3 [Zynga Inc] or TopHeroes [River Game HK Limited]), the onboarding tutorial often gives immediate rewards. In the same way, using rewards, such as achievements, seems compatible with existing literature suggesting that patients’ motivation is strongly linked to homework adherence [9,12]. Earlier studies also show that using rewards in internet interventions can boost adherence, although not specifically focusing on CBT homework [35]. Although gamification seems to have potential to tackle some of the obstacles that patients could have to make CBT homework, it should be acknowledged that technology like gamification cannot solve all of these obstacles.

One of our limitations is the small sample size of the concept mapping. One of our limitations is the small sample size of the concept mapping, although there are no fixed rules for the sample size to be applied. Adequate sample sizes are influenced by the complexity and interactions between concepts, among others. Another limitation is the heterogeneity of the group as most of the therapists in our study were female (6 out of 7) even though it is a good representation of the field. Furthermore, no patients participated in this study; thus, the perception of what is acceptable or seen as important to implement in order to help patients do homework better cannot be confirmed.

For future research, it would be interesting to study the effects of (combinations of) specific game design elements, not just the overall category or cluster. Also, there are combinations of game design elements that could cancel each other’s effects. Furthermore, the minimum number of game design elements of the same category to be used in order to be effective should also be investigated. The distinction between serving game design elements online or traditionally (nondigitally) is another intriguing aspect. Furthermore, research on patients’ preferences on to be implemented game design elements is needed. All these could contribute to the future of using gamification to help patients do their CBT homework better and thus improve therapy outcomes.