We conducted a simulation by varying the model parameters of the 5 statistical models while keeping the input data fixed. The outcomes obtained using each new parameter set were compared with those obtained using the default values. For the incubation time (used for the infection period and infection spread use case) and the serial interval (used for the illness period use case), the literature review described in the study by Kühne et al [14] identified more than one study reporting suitable parameters. Our selection approach to the default parameters for our web application is described in the first chapter of Multimedia Appendix 1 [15-74] and visualized in Figures S1 and S2 in Multimedia Appendix 1. Table 1 summarizes the main aspects of the simulation. In the following section, we briefly explain each use case, where each subsection corresponds to one column in Table 1. Afterward, we describe the dissimilarity metrics used to compare the outcomes and how we visualized the results. For more details, see also the second chapter of Multimedia Appendix 1. The code and extracted study data can be found on GitHub [75].

For the infection period, illness period, and infectious period use cases, the web application provides an answer sentence containing the 80% HDRs as a range of dates accompanied by a plot of the distributions highlighting the 80% HDRs. Thus, the goal was to test how these outputs change with different model parameters. We decided to compare the 80% HDRs of 1 parameter set (HDR₁) with the 80% HDRs resulting from the default values (HDR₂) using an inverted version of the intersection-over-union (IoU) metric [77]:

By inverting the IoU metric, a metric value of 0 indicates a perfect equality of 2 outcomes, and a metric value of 1 corresponds to the situation in which the 80% HDRs do not overlap at all.

These distributions were compared using the Wasserstein metric with p=1 [78], which is defined as where F and G denote the cumulative distribution functions of their respective distributions. As the discretized versions of the distributions were used in our calculations, we also estimated the integral as a standardized sum of differences. With W₁, the changes in the distribution tails were also considered. Although the minimum of 0 of the Wasserstein metric indicates equal distributions, there is no upper bound for the metric as the masses of the 2 distributions can be arbitrarily far away from each other.

For the infection spread, the difference Δpred between the number of total symptomatic infections predicted using simulation parameters and predicted using default parameters was determined. In the last use case, the change in the probability of no further infections was calculated as the difference Δprob between the probability calculated using simulated and default parameters. Both differences allow for easy and direct interpretation of the changes in the respective outcomes.

For all simulations—except the test sensitivity evaluation of the last statistical model—a contour plot was created showing the resulting values; the currently used parameters as red squares; and contour lines with step size 0.2 for the 1-IoU plots, 1 for the W₁ plots, and 0.05 for the Δprob plot. In the infection period plot, no contour lines were shown as the Δpred values were integers. Furthermore, where available, the published parameter combinations were visualized as different marker shapes according to their period of data collection. For each statistical model, we included the most relevant plot in this paper. All the plots as well as a more detailed description can be found in the third chapter of Multimedia Appendix 1.

For the usability evaluation, we conducted cognitive walk-throughs based on the theory by Lewis and Wharton [79]. The process of a cognitive walk-through includes the definition of a sample task that potential users try to solve. This action sequence is then analyzed to provide feedback to the developers about the users’ ability to perform tasks using the web application and which actions likely result in operating errors. The focus was on understanding the learnability of the system for new or occasional users.

For the evaluation, we performed cognitive walk-throughs with potential end users of the web application. The tests included example tasks on contact tracing, one test for each of the 5 use cases (Textbox 1).

The testers were encouraged to speak their thoughts out loud while using the web application. In addition to the example tasks, we asked questions about the usability and applicability of the EsteR toolkit (eg, about the orientation on the starting page and within each tool), as well as the amount and level of detail of the inputs and results. The comprehensiveness of the visualizations was of special interest as these results are the basis for deciding on further handling of the case or contact person. The users were asked how difficult it was to complete the tasks. A 5-point Likert scale was presented indicating the users’ impressions of the web application. The describing attributes included disturbing or supportive, complicated or simple, inefficient or efficient, confusing or clear, and uninteresting or interesting, with the possibility to rate on the scale between these opposites. Moreover, the users were encouraged to provide general feedback. The test was audio recorded, transcribed, and analyzed to provide improvements for the user interface (UI) and suggest a guideline for use.

Focus group interviews are often conducted in public health research as they allow for an exploration of the attitudes, perceptions, and ideas of the target group regarding a topic. The focus group consists of a small group of people with common characteristics, in our case, containment scouts at a local health authority in Germany. The interview is designed to allow for a reflection on the questions asked by the interviewer and a discussion of different answers in the group [80].

We conducted 2 independent focus group interviews with representative containment scouts of the cooperating local health authorities in Berlin-Reinickendorf and Bremerhaven. The containment scouts were asked to share their perceptions in more detail after they conducted the cognitive walk-through and, thus, gained firsthand experience with the web application. The focus was on identifying potential implementation barriers in the local health authorities as well as technical and ergonomic requirements. The containment scouts were asked to rate the effort and benefits of using the web application in their daily work and for additional tasks possibly coming up in the course of the pandemic. The interviews were audio recorded and transcribed. The results were obtained through qualitative content analysis in addition to the impression of the interviewers.

The included studies for the incubation time gathered around a line with μ* ≈ E(X) – 1, recognizable in Figure 1A (corresponding to Figure S3 in Multimedia Appendix 1), and only 7% (2/28) of studies below this line could be considered outliers. All 3 (11%) studies with data from the second half of 2020 and the first half of 2021 were concentrated close to our default model parameters.

For the infection period, the 1-IoU of the predicted 80% HDRs showed small values along 3 directions starting from the default values: first, toward the lower left corner along the included studies; second, almost horizontally through the 2 outlier studies to the right; and, finally, upward to the right but staying below the upper included studies. The highest 1-IoU values were found along the diagonal, where μ* ≈ E(X). In the other areas of the plot, the values remained mainly <0.6, and approximately half (15/28, 54%) of the studies lay inside the first contour line of 0.2.

In general, the Wasserstein metric exhibited a similar behavior regarding areas of high and low values (Figure S4 in Multimedia Appendix 1).

The distributions of the 3 parameter sets with different 1-IoU values, together with the distribution from our default parameters, are plotted in Figure 1B. For the lognormal distribution, the density had a narrow, high peak when the mean and median were close together, as can be observed for point 3. In contrast, when the mean and median were farther apart, the SD increased and the peak became smaller. When the mean and median were shifted by the same amount, the distribution width remained intact but was rather shifted. For the parameter set of point 1, the skew to the right of the lognormal distribution increased, whereas for point 2, it decreased. Hence, the Wasserstein metric of point 1 was relatively high compared with that of the 1-IoU, whereas both metrics behaved similarly for point 2.

For the infection spread model, the higher the median or the lower the mean incubation time, the higher the predicted total infections were. However, the change in prediction was far more distinct for the median, which is visible in Figure 2 (Figure S5 in Multimedia Appendix 1). No change in prediction could be observed along the line through the default parameters, with a slope of approximately 0.4. The maximum value of 9 could be found close to the diagonal with μ* ≈ E(X) fanning out toward the top, whereas the minimum of –6 was located in the lower left corner. All prediction differences were ≤9 because, in the considered setting, the prediction from the default values was 11, and all predictions were limited upward by the total group size of 20. As only mean and median incubation times of at least 3.9 and 3, respectively, were tested and a scenario with 3 observed illnesses after 4 days was considered, the minimal predicted number of total infections was 5, and the minimal difference from the default prediction was –6. The predictions were rounded upward, which resulted in the discrete differences and stepwise changing colors.

As shown in Figure 3A, the included studies for the serial interval spread more than the studies for the incubation time, but they mainly remained within the range of SD(X) ∈ [E(X) – 3, E(X) – 0.5]. Only a small number of studies (5/35, 14%) were outside this range and, thus, far away from the default parameters.

In all 3 generations, the 1-IoU of the 80% HDRs of the symptom onset followed a pyramidal course (Figures S6-S8 in Multimedia Appendix 1).

For the Wasserstein metric, Figure 3A (corresponding to Figure S9 in Multimedia Appendix 1) shows that the course of the values has an oval shape stretched vertically. Along the mean, the increase in the Wasserstein metric was not as high as that of the SD. The highest values could be found on the left and right sides of the plot, with a maximum of approximately ≥3. A total of 54% (19/35) of the included studies were within the contour line with a value of 1. For the second and third generations, the shape of the plot remained the same, whereas the horizontal incline increased up to double its original value, as can be seen in Figures S10 and S11 in Multimedia Appendix 1.

The distributions of 1 parameter set with a low Wasserstein metric and 2 with high Wasserstein metrics for the first generation, together with the distribution resulting from our default parameters, are plotted in Figure 3B. Increasing the SD while maintaining the mean shifted the mode and made the distribution more skewed, whereas most of the mass remained in place. When the mean increased, the main mass of the distribution shifted to the right and the skew slightly decreased. For a low mean and SD, the distribution was very narrow and had a high peak around the mean. Hence, both the 1-IoU and the Wasserstein metric were relatively low for point 1 and high for points 2 and 3. In a joint alteration of E(X) and SD(X), the effects may have partly canceled each other out, and the main changes happened in the tails of the distribution. This explains why the 1-IoU followed a pyramidal course, whereas the Wasserstein metric had a vertically stretched oval shape. This effect was strengthened over different generations as, with the convolution we used in our model, the first parameter of the gamma distribution was multiplied by the generation number.

Both the 1-IoU of the 80% HDRs and the Wasserstein metric plots showed a similar behavior as that of the illness period model as it also used a gamma distribution. The pyramidal shape of the 1-IoU is shown in Figure 4 (Figure S12 in Multimedia Appendix 1), where the highest values of ≥0.8 are in the lower left and right corners. With changes of up to 0.6 in both directions in the mean and SD, the 1-IoU remained <0.2. For the Wasserstein metric plot, see Figure S13 in Multimedia Appendix 1.

Figure 5A (corresponding to Figure S14 in Multimedia Appendix 1) shows the results of the childcare scenario. In the part of the plot with E(K|K>0) ∈ [1.5, 4.7], the predicted probability of no further infections increased with a decreasing transmission probability P(K>0) and an increasing conditional mean E(K|K>0). A white line can be detected, which represents the parameter sets with the same probability as the default set. This part was mirrored at the horizontal lines with conditional mean≈1.5 and 4.7, below and above which the predicted probability decreased with increasing E(K|K>0). Similar behavior could be observed in the school scenario; however, the plot was mirrored at a different conditional mean, and the white line with Δprob=0 had a higher slope (Figure S15 in Multimedia Appendix 1).

In Figure 5B, the likelihood and priors with 2 different parameter sets that result in the same predicted probability are plotted. The likelihood had a maximum value of 1 for no further infections and decreased monotonically for higher numbers of newly infected persons. The priors had their maximum value also for no infection but with different behavior. The first prior had higher values for a smaller number of infected persons, whereas the second prior has higher values for >3 infected persons as the higher E(K|K>0), the more mass was shifted from the lower values toward the upper values.

The influence of the test sensitivity on the predicted probability was almost linear, with a slope of approximately 0.2 (Figure S16 in Multimedia Appendix 1).

We conducted the cognitive walk-through in the local health authorities of Berlin-Reinickendorf and Bremerhaven with 3 containment scouts each. The users tested the web application version as of May 2022.

Most of the testers (4/6, 67%) were able to complete tasks 1 to 3, apart from testers 2 and 6, who did not manage to apply the task to the tools. Even for the testers who were able to complete the task, the results of task 2 were confusing as the graphic was not intuitive and was described using misleading terms. Tasks 4 and 5 were completed only by testers 4 and 5. For the other testers, the input was not intuitive; therefore, they were not able to complete tasks 4 and 5.

In general, the users rated the web application as supporting and efficient but rather complicated and confusing when using it for the first time (Figure 6). Assuming a Likert scale from 1=disturbing to 5=supporting, the median of 4 refers to most users (4/6, 67%) rating the EsteR toolkit as supporting. Accordingly, the users found the web application to be more efficient than inefficient (median 4.25) and more interesting than uninteresting (median 4.5). Rating the web application as 1=simple to 5=complicated revealed heterogeneous results, with a tendency to perceive it as simple (median 3.5). On the Likert scale of 1=clear to 5=confusing, the users tended to find the front end clear (median 3.5). At least 33% (2/6) of the users each agreed in describing the EsteR toolkit as simple, efficient, clear, and interesting. None of the users stated that the web application was fully disturbing, complicated, inefficient, confusing, or uninteresting.

All users (6/6, 100%) stated that the visualization of the results was generally helpful, but some (4/6, 67%) found it hard to interpret. In particular, the interpretation of the 80% and 95% time intervals was unclear to the testers. Some testers (3/6, 50%) expected more detailed information on the event in question, especially for the last 2 tasks. For the testers who tried, it was not possible to generate a PDF document of the results shown in the web application. Nearly all users (5/6, 83%) stated that more guidance and experience would be helpful to use the toolkit efficiently.

The testers indicated that they would adapt the tools for the risk assessment of diffuse events under (partly) unknown circumstances. Moreover, the containment scouts stated that the use cases might help manage groups of >3 people. The detailed results of the 6 cognitive walk-throughs are presented in Multimedia Appendix 2.

The interviews were conducted with focus groups in each of the local health authorities of Berlin-Reinickendorf and Bremerhaven. The 2 focus groups consisted of the 3 containment scouts from each local health authority who had previously tested the web application via the cognitive walk-through.

In general, the containment scouts were highly experienced in assessing transmission risks when applying the official guidelines of the federal government agency RKI. The scouts emphasized that they assessed individual cases and, therefore, valued the personal contact with the index case and contact persons via telephone. In this way, they normally gathered detailed information about the event in question. This workflow is time-consuming and could not be maintained during the entire course of the pandemic depending on local transmission clusters, current policies, and guidelines. Therefore, the containment scouts appreciated digital decision tools as they might help in handling the varying workload and human resources. They saw the potential of the web application to offer support, especially in justifying their decision on group quarantine as the toolkit provides a concrete specification of transmission risk. A total of 33% (2/6) of the testers saw the potential of using the web application to train new staff. Unexperienced containment scouts could insert known data and view time frames when transmission may have occurred.

The containment scouts raised concerns that using the web application during common workflows might be too time-consuming. They agreed on preferring short and intuitive features accompanied by less text and explanations. As a trade-off, the functionalities and results needed to be understood. Therefore, more training with the web application would be needed. In one focus group, the idea of a manual with basic explanations of functionalities and their interpretation was discussed.

All the containment scouts (6/6, 100%) were experienced in assessing transmission risk and periods. Most of them (4/6, 67%) expressed that the toolkit’s results were in agreement with their expectations based on experience. In total, 33% (2/6) of the containment scouts raised the concern that inexperienced colleagues could tend to rely on the web application’s results without questioning their plausibility.

A major concern was the dependence of local health authorities on policies and guidance from the RKI. The local health authorities needed to continuously adapt their daily work to the course of the pandemic. The containment scouts indicated that the web application might not fit their workflows at different stages. Nevertheless, they agreed that contact tracing might be more relevant in the future when the epidemiologic situation and policies change. Further potential was remarked for the first risk assessment for staff in childcare facilities and schools, as well as an adaptation to other infectious diseases.

For each use case, we created a single scenario defined by the input data for which we conducted our simulations. However, many more combinations of different input data could be chosen for the use cases of the infection period for more than one person, the infection spread, and the risk assessment for group quarantine. For the sensitivity analysis, considering all possible combinations would increase the complexity of the simulation and make the interpretation of the results more challenging. Thus, we decided to analyze the infection period only for one person, and we created group situations we typically expect as user input for the infection spread and the risk assessment of group quarantine. Nevertheless, some sensitivity effects might be missed.

Furthermore, the literature review was from March 2022 and resulted in papers that mainly analyzed data from early 2020. Thus, work published since March 2022 is not included in the review or in our analysis. For the infectious period and the risk assessment for group quarantine, we even identified no further studies and, thus, had to define plausible model parameter ranges for the sensitivity analysis on our own.

Sensitivity analysis is a widely used tool, and in the literature, it has been used for various COVID-19–related research questions (eg, it has been used to analyze epidemiological models [5,6,8] or understand the transmission of COVID-19 [7]). Weng and Yi [9] conducted a sensitivity analysis to estimate the COVID-19 incubation time based on a systematic literature review. The results of the sensitivity analysis showed that the incubation time varied when excluding or including certain studies. Our statistical model for the infection period and spread was also based on the incubation period. Our data obtained from a literature review [14] also showed a high heterogeneity.

We expected more differential feedback for the use of the web application as each local health authority in Germany works slightly differently. Only 6 containment scouts from 2 local health authorities participated in the cognitive walk-through. Unfortunately, it was not possible to reach out to another local health authority for the usability evaluation.

Furthermore, user feedback was limited to their first impressions and whether they could solve the tasks on the first or second try. Related to the feedback that they would prefer to try out some of the web application functionalities intuitively and get an impression of how to use it, we will provide the web application to the users before conducting the cognitive walk-throughs next time. We would expect more specific feedback and further thoughts on implementation after users have already tried it out. However, users should not already be familiar with the web application to gather their perceptions for intuitive use.

Furthermore, the usability evaluation was limited to only 6 participants from 2 local health authorities.

Our results might be biased as the containment scouts we approached were highly experienced with contact tracing. Their perceptions of efficient decision support coincided with feedback on guidance and explanations. Nevertheless, the participants had different backgrounds using digital tools. Therefore, we assessed different levels of expertise in working with the UI. However, a major limitation is that the containment scouts were not mainly working on contact tracing at the time of the interview owing to the currently valid regulations. By that time, the RKI policy did not focus on contact tracing, and persons suspected of being infected were quarantined. Therefore, the containment scouts had to think back to the time when they had to justify their decision on whether to order quarantine. Nevertheless, they expected the regulations to be adapted to the course of the pandemic, so contact tracing might be important again during times of higher infection numbers or higher hospitalization rates.

Although the cognitive walk-through method has been applied for UI evaluation of digital applications in general [79], it has recently been applied to digital applications targeting health outcomes. For example, user-centered cognitive walk-throughs have been applied to evaluate a diabetes self-management tool. This method has been shown to be sensitive in identifying critical usability problems and generating recommendations to improve the effectiveness, efficiency, and user acceptance of the tool [10]. Moreover, a decision support tool in the clinical context of self-medication has recently been evaluated through cognitive walk-throughs. Dargan [11] found that their tool was generally understandable and helpful for users. The participants provided recommendations to improve content and layout that the developers could adapt to improve the usability and satisfaction of the decision support tool.

Focus group interviews have been applied for usability evaluation (eg, of clinical decision support tools [12,13]). Kastner et al [12] assessed physicians’ attitudes toward an osteoporosis disease management tool and consequently designed a functional prototype of the tool. The interviews revealed the importance of the use of the tool being as time-saving and handy as possible, as well as specific recommendations for features to be implemented. Ahearn and Kerr [13] conducted focus group interviews to explore the intended users’ attitudes and expectations for integrating such tools into their workflows. The participants stated that the output should be as narrow as possible to not overwhelm the user with less important information.

Accordingly, in our study, we identified functionalities that were not working properly and features that led to misuse and misunderstanding, obtained recommendations for layout improvements, and obtained feedback that the web application would help familiarize the user with the context and support the decision-making process. The containment scouts highlighted the importance of efficient contact tracing to contain COVID-19 outbreaks. We assessed similar concerns as in the focus group interviews conducted by Kastner et al [12] and Ahearn and Kerr [13], namely, that the features need to be easy and fast to use to implement the tool in the workflow.