The study consisted of scenario-based usability tests with additional interview questions and the Dutch User Experience Questionnaire (UEQ-Dutch) [17]. The usability tests were conducted using a scenario-based, think-aloud method [18], captured by researcher observations and voice recordings, and took place from June 29 to July 3, 2020. A beta version of the app was evaluated using test iOS (version 0.1, build 172) and Android (version 0.3.1, build 107) phones. Mock-ups (Figma, version 0.7.1) were used for the scenarios that could not be tested in the beta version due to the current stage of development (ie, being able to download the app from the App Store). The study was conducted at the University of Twente’s DesignLab and in the Experivan mobile lab [19], which was used to visit participants with an intellectual disability. The BMS Lab protocol for coronavirus-safe research on humans has been approved by the Executive Board of the University of Twente. Hands and equipment were disinfected before and after the tests, and national measures were followed (per the National Institute for Public Health and the Environment [RIVM] [20]).

Participants were recruited using convenience sampling in Twente.

To test whether the app is inclusive, participants from the following target groups were included:

The study was approved by the university’s Ethical Committee (BCE200953).

Participants were informed of the voluntary nature of their participation and confidentiality was guaranteed. All participants signed an informed consent, and a parent or guardian signed the informed consent for underaged participants.

Adults (aged 40-64 years), older adults (aged >65 years), and migrants were contacted by the researchers by telephone. In this call, an explanation was provided about the nature and purpose of the study and an appointment was scheduled for the test. Recruitment of the younger participants was arranged through an intermediary (eg, school principal, mentor). The parents or guardians of minors were contacted by the intermediary via telephone and asked for their permission to let their children participate in the study. After the call with the researcher, all participants (or intermediaries) received email confirmation of the appointment, which also included additional information on participation. Figure 1 provides an overview of the procedure for participants.

The usability tests were conducted individually and in pairs in the case of youth in the lower-education category since research shows that minors are more capable of identifying a larger number of problems and details while working in pairs [21]. Before the test, the nature and purpose of the study were explained again, and permission for participation and audio recordings was given by signing the informed consent form. Parents or guardians had to sign the informed consent for minor youth. After providing additional consent, 6 participants additionally wore Tobii eye-tracker glasses (Tobii Gaming) for gaze analysis. An eye tracker was used due to people’s strong urge to move their eyes, so that the fovea is pointed at whatever stimulus they are currently thinking about or processing. This is referred to as the so-called eye-mind link [22-24]. This theory makes eye tracking a reliable tool for exploring visual attention. These results are discussed elsewhere [25] and are outside of the scope of this paper.

The test protocol (Multimedia Appendix 1) started with general questions about the impact of the pandemic on the participants’ lives and about what they had already heard about contact tracing apps for COVID-19. Thereafter, each participant actioned four scenarios on the app, which represented actual use of the app (1-hour test) while simultaneously thinking aloud. Before the usability test started, participants could choose between an iOS and Android test smartphone, per their preference. The four scenarios were:

After completing the scenarios, closing interview questions were asked about the participants’ attitude toward the app and their willingness to use it. Additionally, a general questionnaire was administered (Multimedia Appendix 3), collecting data on gender, age, highest completed level of education, physical limitations in using apps, and a self-reported digital skills assessment, combined with the UEQ-Dutch. Within the UEQ-Dutch, participants had to assess different characteristics of the app (eg, whether the app is easy to use, visually attractive, supportive, and reliable) on a 7-point Likert scale. Researchers focused on whether the steps to be completed on the app were clear and easy to follow, and whether participants understood the utility and consequences of sharing their key. Differences between target groups were explored to investigate whether the app is inclusive. Whether the conversation with the PHA worker matched the steps that must be completed in the app was also examined.

The recordings of the usability tests were pseudonymized (by BB) and stored on a data server at the University of Twente and were only accessible to the researchers involved. These storage and associated processes were certified according to ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) 27001 and NEN 7510 standards. Recordings of 3 participants (1 youth with a higher educational level and 2 youths with an intellectual disability) were not stored correctly due to technical errors, so these participants were excluded from the study. Recordings were transcribed verbatim (by MS, LB, JG), and all transcripts were analyzed (by BB and MS) to identify fragments on user-friendliness, understandability, reliability and credibility, and inclusiveness. Relevant fragments were labeled with the main codes “user-friendliness,” “understandability,” “reliability and credibility,” and “inclusiveness” in Microsoft Excel. Next, the fragments within the main codes were analyzed axially to link fragments to each other and create new subcodes within each main code. Two researchers (BB and MS) coded 6 transcripts together to determine coding agreements. BB and MS each coded half of the other transcripts while considering the coding agreements. The coding scheme was revised several times by both researchers, and fragments were reread and recoded if necessary.

Descriptive statistics were generated for the general questionnaire using SPSS, version 20 (IBM Corp) [26]. The outcomes of the UEQ-Dutch were analyzed within the data analysis tool [27] offered by UEQ. This tool is benchmarked every year by UEQ to keep the validity of the tools as high as possible. The data gathered from the UEQ-Dutch questionnaire were input into the tool, which resulted in descriptive statistics and graphical representations of the gathered UEQ-Dutch data.

The transcribed data are not publicly available due to privacy restrictions but are available from the corresponding author upon reasonable request. All screenshots of the CoronaMelder app are online, available via Figma.

In total, data from 44 participants were included in this study. The sample comprised 31 males (70.5%). The mean age was 40 years, ranging from 13 to 79 years, and 26 (59%) participants reported having completed a higher education level. The majority indicated not having physical limitations to using apps in general. Nearly all rated their digital skills to be between “not handy, not clumsy” and “very handy.” All participant characteristics are listed in detail in Table 1.

The following sections will focus on the research question: is the CoronaMelder user-friendly, understandable, reliable and credible, and inclusive? Table 2 presents an overview of how many arguments were mentioned by participants per theme, indicating their positive or negative opinion, or whether they understood the items related to the theme. The majority of the participants were positive about the user-friendliness, reliability, and credibility of the app. Participants from all target groups indicated more negative comments than positive ones regarding the understandability of the working mechanism of the CoronaMelder.

Multimedia Appendix 4 provides an overview of the number of positive and negative comments per topic, per target group.

User-friendliness was assessed with the UEQ-Dutch and during the interview. In Figure 2, the outcomes of the UEQ-Dutch for the entire population are displayed for each of the six assessed scales. The scales include attractiveness (an overall impression of how much users like or dislike the app), perspicuity (how easy it is for participants to become familiar with the app), efficiency (the ability of users to use the app as intended), dependability (whether the user feels in control of the interaction with the app), stimulation (whether the app is exciting and motivating to use), and novelty (whether the app catches the interest of the user).

Participants rated the CoronaMelder app between 0.9 and 1.7 for all scales, which represents a positive evaluation (values >0.8) [28]. For perspicuity and efficiency, the app was rated relatively high, indicating the CoronaMelder is easy to become familiar with and participants were able to use the app as intended. The CoronaMelder scored relatively low on novelty, indicating that it was not proficient in capturing user interest.

Participants from all target groups provided more negative comments than positive ones on the understandability of several topics. Understandability problems occurred due to inconsistency in the terms used and not reading information on how the app works. Most problems concerned receiving a notification from the app and sharing the key when tested positive to support contact tracing. The app was inconsistent in the ways it referred to the coronavirus (eg, “coronavirus,” “COVID-19,” and “corona”) and to the key that should be shared after testing positive (eg, “ID,” “code,” “control code”). Furthermore, a clear definition or explanation was lacking within the app about what it means to be exposed (“exposure”) or at “increased risk” of a COVID-19 infection. The texts also include technical English vocabulary, such as “ID,” “share,” “enable,” and “upload.” Older participants in particular did not know what those words meant and were confused by them:

The app was assumed to be reliable because it was presented as a government app, and participants trust the VWS:

The explanation about data storage and anonymity (Figure 7) earns trust, and adult participants in particular applauded the fact that the app does not require personal data.

Although adults assessed the app as trustworthy, youth in the lower-education group did not understand how the app guarantees privacy. Participants (from different target groups) believed that they would be tracked and that others would know their name and address when they reported a positive diagnosis. These misconceptions are not only caused by a lack of explanation or by participants not reading the information provided, but also because participants mistrust the use of Bluetooth—some participants may think it will still track their location and that the app will connect with people who are not directly in close proximity to them (eg, separated by a wall):

Other participants were less doubtful about the use of Bluetooth, reporting that they thought the acquisition and storage of data was safe. Additionally, if privacy is guaranteed, multiple participants (mostly youth in the higher-education category and adults) mentioned their willingness to use the app:

Differences in inclusiveness were found across age groups. In general, participants from all target groups with the exception of older adults did not have any problems using the app. Among older adults, there was a dichotomy between those who had digital skills and those who had trouble finding specific information within the app, opening and closing specific screens, and attending a telephone call while simultaneously opening and using the app:

Older adults were able to perform the steps to share the key under the guidance of the PHA worker but had difficulty performing these steps while talking on their mobile phone. For example, older adults were not aware of how they could turn on the speaker or close the call screen and open the app. However, they signaled willingness to learn how to use the app:

Additionally, youth with an intellectual disability appeared to lean on the researcher while conducting the test. They were in doubt and asked for confirmation each time before clicking on a screen or button. They appeared to not know what they were doing or why they must do it. For example, while sharing the key, they followed the steps the PHA worker provided them, without seeming to understand what they were doing or what would happen next.

Regarding the language used in the app, participants reacted differently according to their educational or cultural background. For example, youth in the higher-education category and both adult and older adults thought the language was clear and easy to understand and that appropriate words were used to express the purpose of the app:

Youth with a lower educational level, youth with an intellectual disability, and migrants reported that the words used were too difficult to understand and texts were too long. The latter indicated that the app should work in other languages, such as English and Arabic.

This study aims to answer the research question: is the CoronaMelder user-friendly, understandable, reliable and credible, and inclusive? Based on the findings, we can conclude that the CoronaMelder is easy to use. The app was seen by most as a good initiative because it warns them about possible infections, protects them, and could help avoid a second viral wave. The app was considered reliable because it is an initiative from the government (VWS). After participants read the information in the App Store, they indicated understanding how the app operates, and many were curious to become familiar with the app and expressed their intention to download it. Several general reasons why participants were willing to use the app were indicated, such as protecting themselves and their loved ones, creating sufficient support for the app, and helping to get COVID-19 under control and ease nationwide measures. However, it appeared that essential parts of the app were not understood by the participants, such as the notification system, the sharing of the authorization key via PHA, or how the app guarantees privacy.

Doubts and fears were expressed regarding privacy, usefulness, and consequences of the CoronaMelder. Among the reasons for these negative attitudes were fewer positive arguments in the media and the number of false positives. Reasons not to use the CoronaMelder were expressed, such as perceiving the app as useless, thinking the coronavirus and corresponding measures were overrated, not wanting to be in quarantine (without confirmed risk), and limited phone memory or battery capacity. In terms of inclusiveness, it appears the CoronaMelder is not accessible to various target groups. Youth at a lower educational level or those with disabilities had difficulties using the app due to low literacy and language problems, and the older adults experienced difficulties related to limited digital skills.

Whether the app will be effective in supporting traditional contact tracing is a concern since the majority of participants did not understand how the app operates or why there is a delay between exposure/increased risk and receiving a notification. The app provides complex information and lacks explanations; therefore, users find it unclear what actions the app expects from them. This lack of clarity has led to misconceptions about the app in terms of operation, privacy, and usefulness, thus affecting participants’ willingness to use it. This also affects the adoption of the app and adherence. Additionally, the protocol of PHA workers lacks guidance, explanation, and empathy, revealing that PHA staff are unprepared to support users with the app to the fullest extent during the pandemic (eg, with key-sharing procedures) in addition to their other responsibilities.

To the best of our knowledge, this is the first study to pretest the usability of the Dutch CoronaMelder app. Studies of other countries’ contact tracing apps that operate similarly to the CoronaMelder reported comparable findings on participants’ attitudes toward these kinds of apps. In a study by Horstmann et al [29] on the German Corona-Warn-App, participants indicated that there were no reasons not to use the app, that the benefits would outweigh the risks, and that they believed the app would contribute to slowing down the pandemic [29]. On the other hand, in studies of apps in Germany, Switzerland, and France, the most frequently mentioned reasons not to use the app were privacy concerns [29-35], doubts about usefulness [29,34,36], and lack of technical equipment (eg, not all smartphone operating systems can access the apps) [29,34,37]. In both of the studies on the Corona-Warn-App (Germany) [29] and the SwissCovid app (Switzerland) [34], as well as in a longitudinal survey about the Dutch CoronaMelder [38], privacy concerns appeared to be associated with a lack of trust in the national government (or in PHA).

In addition, the StopCovid app (France) showed low uptake [36,39] because the app appeared uninteresting and ineffective [36]. The same study also reported that 71% of their participants suggested that better communication strategies would increase uptake of the app. Our study stated that the reasons mentioned for not using the app were already expressed before contact tracing apps were launched. However, it was reported [29,40] that these concerns were still raised after app implementation, which indicates that public health campaigns promoting contact tracing apps were not able to eliminate these concerns.

The first strength of this study is its focus on participants of different backgrounds (age, education, etc) to test whether the CoronaMelder is accessible to all residents of the Netherlands. The second strength is the real-time pretesting of the key parts of the CoronaMelder to enable revisions before the definitive launch. The findings of the study were communicated with the software development team to exchange feedback on adjustments to the app and to revise the app during the test days. During development, minor adjustments in the app’s design were made, which means that participants who tested the app later in the study may have viewed some screens that were different to those tested by participants earlier, even though the essential parts of the app were the same. Based on the findings of this study, the VWS decided to launch the app. However, the definitive launch (October 10, 2020) was postponed due to changes in testing policy. The premise “test without symptoms,” which is an important driver for using the app, was changed due to a lack of testing capacity.

To fulfil the requirements of the CoronaMelder [11], the design of the app can be improved. The accessibility and understandability of the app should be customized to differences in literacy and digital skills. Think-aloud, real-world–based scenarios and eye tracking should be designed to involve end users with different literacy levels and digital skills to test use of the app in real time.

Evaluation of the CoronaMelder app should focus on the key essentials of the app to support early and better contact tracing. Data should therefore be collected on use and adherence regarding follow-up actions after a notification, sharing a key to inform PHA and other users (via the app), and going into isolation when testing positive. The privacy-by-design policy could complicate gaining insight into the added value of the CoronaMelder app since it hinders data collection. A critical view is needed on how to find a balance between user-centered design and the privacy-by-design policy.

Future studies should also focus on how communication campaigns can be best targeted and customized to reduce uncertainties and misconceptions, thereby improving the understanding of digital contact tracing apps as well as adoption and adherence. Overall, an adequate infrastructure (resources, personnel, capacities, etc) and powerful management tools are needed to implement the CoronaMelder and other digital tools to facilitate and optimize contact tracing to fight a pandemic. The COVID-19 pandemic has had an adverse global impact and requires an interdisciplinary approach. Future studies of the CoronaMelder app should consider the app not as a stand-alone device but as part of a coherent package of anti–COVID-19 measures to fight the pandemic, considering the impact on users, stakeholders, and testing and tracing procedures.