The FluCARE pilot program ran from September 2019 to December 2020 across 62 ACFs in the SLHD [22]. Any ACF in the district was able to voluntarily register for a FluCARE account. Accounts were validated by the PHU to ensure appropriate access to the app. Users were typically nurse managers or infection prevention and control managers. All users were required to complete FluCARE training via a face-to-face workshop or online modules prior to using FluCARE.

The FluCARE app and pilot had been widely promoted to ACFs through face-to-face workshops, teleconference updates, email communications, and telephone calls. Dedicated email and telephone support was provided to ACFs for registration, user training, app utilization, and troubleshooting. A detailed description of the package of activities to support app implementation is provided in an earlier article [21]. Evaluation tools, training modules, and user manuals for ACFs were also updated from March 2020 to support use and evaluation of the additional COVID-19 functionalities.

Ethical approval was granted by the Ethics Review Committee of the SLHD (protocol number: X19-0157). All registered ACF users in the SLHD were invited to participate in the study and were emailed a “participant information statement” outlining the details of the study and a consent form to sign and return to the research team.

One or more staff members from ACFs in the SLHD were eligible for the study based on completion of voluntary registration to FluCARE. The eligible staff members were generally the person(s) responsible for submitting line lists to the PHU, for example, a care manager, registered nurse, infection control officer, or facility manager. Participation in the study was not a requirement for using FluCARE. Those who consented to participate but did not subsequently complete the required training modules or workshop to use FluCARE were excluded from the study. Study participants were invited to complete the training evaluation survey immediately following their completion of training and the FluCARE evaluation survey at the end of the pilot period in December 2020.

The logic frame model for the FluCARE pilot program and evaluation is shown in Figure 1. The inputs and activities conducted in the design, development, and implementation of FluCARE have been published elsewhere [21]. In the following sections, we describe the evaluation tools used to measure app acceptability, utility, and safety, as well as the barriers and facilitators to implementation of the app.

As shown in Figure 2, 176 ACF staff were invited to participate in this evaluation study of the FluCARE pilot; 52 (29.5%) users from 34 facilities consented and were eligible (ie, completed training and app registration) to participate in this study (Figure 2). Of these users and facilities, 31 (62%) users from 27 (79%) facilities completed the postpilot app evaluation survey.

Of the 46 users completing face-to-face workshop training for FluCARE use in influenza outbreak management, 40 (87%) participated in the study (Figure 2). Of the 60 and 46 users completing online training for FluCARE use in influenza and COVID-19 outbreak management, respectively, 24 (40%) and 18 (39%; for a total of 42 respondents) participated (Figure 2). Of all workshop respondents, all (40/40, 100%) agreed that the content of the workshop was helpful. The vast majority (38/40, 95%) agreed that the workshop was engaging, all (40/40, 100%) felt more confident in using FluCARE, and 92% (35/38) said that the content was interesting. A vast majority of online training respondents agreed that the online training module content was easy to follow (40/42, 95%), useful (40/42, 95%), and of sufficient depth (39/42, 93%). Overall, 95% (40/42) of respondents felt confident in using FluCARE at the end of the training. Users rated the online training courses 7.5 of 10. The survey responses from app users who attended or completed training are shown in Table 1.

Of the 50 users (from 34 facilities) who consented to participate in our study, 31 (62%) users from 27 (79%) facilities completed the postpilot app evaluation survey (Figure 2); 6 (19%) respondents reported not using FluCARE for outbreak reporting during the study period. The reasons given included no outbreak to report, access issues with the app, or issues with delineation of users to complete line listing. Therefore, 25 (81%) respondents had used FluCARE for outbreak surveillance and reporting. Their key characteristics are described in Table 2.

Respondents reported having a median of 1 (range 0-4) influenza outbreak at their facility in the past 5 years. The majority of respondents reported using the FluCARE app to record a single respiratory virus case (23/25, 92%) or a suspected case of COVID-19 on the line list (19/25, 76%) during the pilot period (multichoice answer), whereas 68% (17/25) reported using FluCARE to record multiple respiratory virus cases (including cases of influenza-like illness [ILI] or confirmed influenza). The survey results found that 32% (8/25) of respondents used the app to access resources and information. As shown in Table 3, FluCARE was helpful in identifying the first few cases of ILI (15/25, 60% agreement) or in determining whether the facility had a COVID-19 situation to monitor (16/25, 64% agreement). Other survey results revealed that 84% (21/25) of respondents reported that it was easy or very easy to submit the daily line listings via the app.

User ratings of FluCARE on the uMARS scale are summarized in Table 4. Of the 4 domains in the uMARS, the highest scores were for the quality of information (mean 4.53, SD 0.33), followed by the aesthetics of the app (mean 4.25, SD 0.43). FluCARE was rated 3.91 of 5 overall on the uMARS.

Of the survey respondents, 68% (17/25) reported that they would “definitely” recommend the app to other people, and 80% (20/25) reported that they would “definitely” use the app for future influenza and COVID-19 outbreaks. However, only 28% (7/25) of respondents would definitely pay for the app. There was a high level of self-reported agreement (agree and strongly agree) that FluCARE would (1) improve the ability of the facility to recognize both influenza (24/25, 96%) and COVID-19 outbreaks (24/25, 96%), (2) assist the facility in knowing which actions to take once an outbreak has been recognized (19/25, 76%), (3) reduce the time to notify the PHU (23/25, 92%), (4) reduce time in talking to stakeholders in outbreak management (22/25, 88%), (5) make the submission of line listing easier (25/25, 100%), and (6) be useful and beneficial for influenza (25/25, 100%) and COVID-19 (25/25, 100%) outbreak management.

Regarding technical barriers or facilitators to use of the app, there was a high level of agreement (21/25, 84%) from respondents that the app made it easy to (1) complete outbreak detection, notification, and management tasks and (2) access it within the local IT network and that respondents were (3) confident of the information and (4) security within the app. However, respondents were less agreeable (13/25, 52%) about how easy the app was to access outside their organization (ie, from home).

There was a high level of agreement from respondents that (1) the education and training provided sufficient support to users to learn how to use the app (25/25,100%), (2) the technical support provided by the PHU to users was adequate and supported use (25/25, 100%), and (3) they had sufficient time to practice using the app in routine practice (21/25, 84%). However, respondents were less likely to agree (7/25, 28%) that they felt involved in the design of the app.

At least two-thirds of respondents agreed that the use of FluCARE was prioritized by their team and organization for use in outbreak management (22/25, 88%) and they had sufficient organizational support to use the app (18/25, 72%). However, there was less agreement (10/25, 40%) from respondents that there was accountability within their organization if FluCARE was not used and that there were incentives from their organization to use the app.

From September 2019 to December 2020, there were 27 entries in the issue log. Of these, 21 were reported by PHU staff, and 6 were reported by ACF staff. The majority of recorded issues were related to access (18/27, 67%). Access issues were primarily technical, with incidents of generic page not found or internal server errors on attempting to access FluCARE. Most of these were resolved on the same day, either spontaneously or with ICT assistance. However, on one occasion, restoration of access took 20 days, which required implementation of our Business Continuity Plan with manual procedures and provision of downtime packs and communications to ACFs. Three access issues were related to delays in receiving the 2-factor authentication code. Other issues on the log related to the line list (7/27, 26%), notifications or alerts (1/27, 4%), registration (1/27, 4%), and reports (1/27, 4%).

This pilot evaluation study of the novel FluCARE app found the app to be acceptable, useful, and safe for use by ACF staff, the primary target users of the app. Thus, the app has demonstrated feasibility for ongoing implementation and further scale-up to other public health districts. User acceptance and satisfaction are fundamental to the long-term success of an app, particularly for adoption and sustained use [28]. FluCARE was highly acceptable to ACF staff, with a majority of users indicating that they would definitely use the app for future outbreaks and that they would definitely recommend the app to others. This is consistent with the high user rating of FluCARE, with an above average overall score (3.91 out of 5) and above average to excellent ratings in functionality, aesthetics, and information quality. There was unanimous agreement in the overall utility of FluCARE in assisting ACFs in outbreak management. Specifically, ACF staff agreed that the app improved outbreak recognition and reduced time to notification of the PHU and response stakeholders. Although these were not objectively measured with outbreak data, we have the ability to collect and report on these data (as shown in Table 2) in future efficacy and effectiveness studies. Finally, there were no adverse events reported in the use of FluCARE, particularly with regards to missed outbreaks, indicating that the app algorithms were working in accordance with national guidelines. Rather than any intrinsic app issues, the predominant reported issues were related to site access, having been addressed with improvements in the hosting environment.

Since the pilot period for FluCARE, there have been at least two instances of specific mobile health (mHealth) apps designed to help detect, manage, and report on respiratory outbreaks in institutions here in Australia [29] and overseas [30]. The article from Ahn et al [29] describes the development of their app, noting that an mHealth app with dashboard functionality should help improve collection and reporting of outbreak data. However, although the article from Echeverria et al [30] describes the implementation and use of their mHealth app, including the detailed collection of line listing data from over 196 institutions and 10,000 sick residents (albeit only for 30 days), their article does not mention any evaluation focused on the implementation barriers or enablers from an organizational or workforce perspective.

This study also provided findings on barriers and facilitators for ACF staff in using FluCARE, which informs a further roll-out of FluCARE. It is well-recognized that an effective and sustainable digital health intervention is dependent not only on the app or digital tool itself but also on various implementation support to ensure an enabling environment for the adoption and use of the digital tool [31]. Our study found that the implemented training and education for FluCARE and the technical support from the PHU were effective and useful in facilitating FluCARE use by ACF staff. This is consistent with previous reviews of eHealth implementation, which found that adequate staff training (including allocation of time and resources for training) and dedicated technical support are critical to increased user acceptance [32,33]. These elements also help to overcome a number of common barriers, such as negative attitudes to a new technology, skill deficits, and disruptions to existing workflow [32,33]. Similarly, as shown in our study, evidence has demonstrated that training of ACF staff and provision of 24/7 support are essential for any implementation of new health technology in nursing homes. Support can be delivered in various formats, such as troubleshooting guides and telephone help lines [34]. In our study, identified barriers for ACF staff to use FluCARE were (1) access to FluCARE outside the organization, (2) lack of involvement in the design of the app, and (3) accountability and lack of incentive within their organization to use the app. Addressing these barriers will require a combination of technical, social, and organizational approaches. For example, in addition to the prior use of consultative processes and ACF engagements [21], establishing direct feedback mechanisms (eg, via the app), including ACF staff in the multidisciplinary FluCARE working group, and training staff champions to lead implementation in each facility are potential strategies. Involving end users in design and development and empowering nursing staff to plan and lead implementation in their facility reduce barriers related to user-friendliness; compatibility (to work environment or processes); and acceptance, confidence, and ownership of the new technology [31,33]. Further exploration of whether FluCARE can integrate with clinical applications to streamline notifiable disease management in collaboration with local geriatric specialist colleagues is needed.

Last, to our knowledge, this is the first time a web-based app for outbreak management in ACFs in Australia has been evaluated and reported in a feasibility pilot program. This study contributes a structured, replicable approach to a feasibility assessment of a novel app in providing the necessary evidence base to guide further implementation and scale-up.

A key limitation of this study is the fundamental alteration in the public health context and respiratory disease transmission dynamics due to the COVID-19 pandemic, the onset of which coincided with the latter half of the study period. This was a period of extraordinary public health measures against COVID-19, including closure of Australia’s border, which also saw an all-time low in circulating influenza and other respiratory viruses [35]. Consequently, none of the participating ACFs experienced an influenza outbreak in which FluCARE could be used, although ACFs still actively used FluCARE for influenza and COVID-19 case monitoring. It is unclear the direction of impact the concurrent pandemic may have had on ACF staff perceptions of FluCARE. Heightened concern for timely detection of respiratory infections may have promoted FluCARE utility. Simultaneously, increased PHU oversight and support of ACFs for COVID-19 may have superseded some of FluCARE’s functions, thus reducing its perceived utility over this period. Another limitation of this study was that ACF participation in the FluCARE pilot and staff participation in the evaluation study were self-selected. This may have resulted in selection bias toward facilities and staff who have pre-existing openness or confidence for new technologies, thus bolstering the acceptance of FluCARE.

This study took place in a single district within metropolitan Sydney. Therefore, findings may not be generalizable to all other public health jurisdictions, such as regional or rural areas with lower densities of ACFs and different levels of digital infrastructure. Further implementation research is necessary to inform and adapt strategies for roll-out into other districts.

Our study was limited to using an evaluation instrument designed to measure the acceptability and safety of mobile-based apps at the time, as no other validated tools were available to evaluate web-based apps specifically. In addition, piloting our ACF user survey enabled the instrument to reach face and content validity; further work is required to ensure the survey has adequate internal reliability over time. Additionally, this study only reports on the perceptions of the ACF staff. Although they are the primary target users of the app, there are also PHU users and other stakeholders whose perceptions are relevant for app adoption and utilization. Further evaluation surveys should also include perceptions of PHU users and collect and analyze in-depth qualitative data to uncover any insights around adoption and use not already reported.

FluCARE demonstrates high acceptance, utility, and safety for ACF staff in the management of influenza and COVID-19 outbreaks. FluCARE was piloted with a package of implementation activities including mandatory user training and technical support for ACFs. FluCARE has demonstrated feasibility for efficacy studies including further scale-up to other districts. Further evaluation, guided by implementation science frameworks and utilization of outbreak metrics, will be needed.