This study applied a multimethod design including qualitative and quantitative data collection. To obtain lessons learned from the continued implementation of Caren, a focus group was conducted with the development team of Caren. To conduct a pre-post COVID-19 comparison of usage behavior, a cross-sectional web-based survey was conducted among ICs and CRs who were users of the platform Caren pre-COVID (2019) and post onset (2022).

The ethics committee of the University of Twente (Faculty of Behavioural, Management and Social Sciences) provided ethics approval for this study according to European regulations (request numbers: 220250, 221180, and 190190). The surveys were designed to ensure complete anonymity, with no personal data being collected from participants. For the focus groups, participant identities were de-identified to maintain confidentiality and protect their privacy.All participants provided informed consent prior to participation in the study.

Participants eligible for the focus group were required to be at least 18 years of age and hold a position in the development team of the Caren platform. Recruitment took place via corporate email, supplying information about the purpose and content of the focus group on the basis of which participants could indicate their interest to take part. Each of the approached potential participants agreed to participate in the focus group which took place in-person at the developers’ company.

Participants for the surveys were recruited through purposive sampling. In 2019 and 2022, a cross-sectional web-based survey was offered to users of the platform Caren via a pop-up message on the platform. Participants were included in the study if, at the time of data collection, they used the platform either as an IC or as a CR. Responses from subjects younger than 18 years and incomplete responses, deemed as where the respondent did not provide answers to all mandatory questions, were excluded from the analysis.

In the following section, we present lessons learned from the continued implementation of Caren throughout the COVID-19 pandemic from the technology developer’s perspective. The lessons are grouped according to the 4 CFIR domains and are also available in tabular form in Multimedia Appendix 4.

Within the domain intervention characteristics, 3 lessons learned related to the changes in certain (technical) characteristics of the platform or services surrounding the platform were identified.

Within the domain characteristics of individuals, we identified 2 lessons learned related to the end users of Caren throughout the COVID-19 pandemic.

Within the domain outer setting, we identified 2 lessons learned related to the broader implementation context of Caren.

The study identified several lessons that were learned about the continued implementation of the informal care platform Caren throughout the COVID-19 pandemic from the technology developers’ perspective. Furthermore, the study revealed pre-post COVID-19 changes in usage behavior and support functionality use among ICs and CRs.

Lessons learned relating to the continued implementation of Caren were categorized on the basis of the CFIR. The first domain is the inner setting, which refers to the environment of an organization or system where a particular health care intervention or program is being implemented [19]. The increasing use of digital technology during COVID-19 has impacted the work cultures in a multitude of ways. This includes organization of the work within teams, performance or measurement of work done, effective communication, and collaboration within or across teams [21]. The findings of this study are along similar lines. The technology developers and designers had to adjust to and accommodate new ways of working (eg, working from home). Broadly, these changes rely heavily on the usage of video- or audioconferencing tools, thus demanding organizations to upgrade their technology infrastructure [21]. However, in Caren’s case, the company had the infrastructure to manage IT overloads, which streamlined the process. Furthermore, as Caren employees have adapted to remote work, there is a growing tendency for organizations to maintain this practice.

The outer setting is the broader context in which an organization operates, encompassing the broader social, economic, and political factors that can impact the implementation of a health care intervention or program [19]. A direct impact of the used business model by organizations responsible for developing and facilitating care infrastructure (outer setting) became apparent in this study. Legislations were passed to take digital initiatives during the start of the pandemic, but these were not sustainable for the long term [22]. In the study by Bokolo [23], the impact of the shortage of government funding and reimbursement model was found to be significantly impacting the implementation of digital health tools during the COVID-19 pandemic. In that regard, Caren not depending on government funding facilitated its continued implementation. Finally, due to the increased use of technological interventions in care, enormous amounts of data on CRs are available. While artificial intelligence can help in improving the quality of in-general health care (by correct prediction, classification, and useful recommendations) [24,25], it is also mandatory to invest and implement new data security ways to ensure safety and privacy of such personal datasets [26]. In this study, technology developers and designers were also concerned regarding the transparency of the medical data.

Regarding the domain intervention characteristics, developers recommended relying on automated response functions. This is in line with the work of Zand and colleagues [27]. In fact, their analyses of electronic dialog data showed that patient-provider communication through automatic messaging can dramatically optimize workflows. In fact, allocating simple tasks to the chatbot leaves more time available for better patient care. Furthermore, chatbots are continuously available and have no waiting time. Aside from simple tasks, such as managing appointments or delivering bulk messages such as in the case of Caren, the authors suggest that chatbots can also be used for more sophisticated, although instrumental, parts of care.

Finally, regarding the domain of characteristics of individuals, developers indicated that the COVID-19 pandemic, and especially the social isolation measures that came from it, was a catalyst for the adoption of technology for health. In particular, developers thought that the users might have been initially “externally motivated” to adopt Caren and have kept using it because they found it helpful. In fact, numerous telehealth solutions gained popularity during the COVID-19 pandemic because of their perceived usefulness [28,29]. However, the feasibility and application of telehealth in resource-limited settings and low- and middle-income countries are still debatable [30].

As indicated by our focus group participants, the Caren platform has seen a rise in platform users during COVID-19. However, the 2022 survey had a lower response rate than 2019, possibly due to being open for a shorter time. The quantitative results of our study showed that throughout the COVID-19 pandemic there was an increase of ICs and CRs who used the Caren platform several times per day. This seems understandable as due to COVID-19, face-to-face interaction was possible only to a limited extent. For both groups (ICs and CRs) we found an increase in the use of a shared digital agenda to make and view appointments (calendar), a messaging function to receive updates and communicate with formal and informal caregivers (messages), and digital notes to store important information (notes), suggesting that these support functionalities gained relevance for users throughout the COVID-19 pandemic and should therefore stay core platform elements. It stood out that the dossier functionality use (a connection to the electronic client record of the involved home care provider) decreased among ICs between 2019 and 2022. One would expect that being able to read along in the electronic client record of their CR would become more important for ICs in times where face-to-face contact is limited, especially for ICs who care on a distance of whom there were significantly more in 2022 as compared with 2019. However, the 2022 sample was significantly older than the 2019 sample, which might have hindered some ICs to use this more complex functionality (requiring additional steps such as 2-factor web-based authentication to access the records). Another explanation might stem from previous research which found that, before COVID-19, ICs who used Caren often criticized incomplete or delayed information in the electronic client record [31]. Due to the burden caused by COVID-19, care staff might have found it even more difficult to adequately fill in the electronic client record, reducing its usefulness for ICs.

We believe that using an implementation framework such as the CFIR, which has frequently been used to retrospectively evaluate implementation or to prospectively design implementation strategies [32,33], provided a valuable tool for interpreting the results. Furthermore, to the best of our knowledge, this is the first study that compares the usage behavior and support functionality use of a well-established digital care tool before and after the COVID-19 pandemic. In addition, the comparison was made by using preexisting (as opposed to retrospective) data from before the pandemic.

The study also has some limitations that should be taken into consideration. First, the study was conducted within the context of the Dutch health care system, which may restrict its generalizability to other health care contexts. Second, the lack of input from significant stakeholders, such as formal caregivers or employees within the business department of Caren, is another notable limitation. The inclusion of these relevant stakeholders, as well as similar evaluations of non-Dutch digital health care platforms, should be considered for future research. Third, although the survey conducted in 2019 and 2022 contained similar questions, recoding response options was required to enable a comparison of the results. Finally, ICs and CRs from 2019 and 2022 differed regarding several demographic variables which might limit their comparability. A follow-up analysis (eg, multivariate regression) could identify variables that might have driven some of the observed differences between 2019 and 2022 use patterns. Furthermore, conducting subgroup analyses could provide insights into whether usage patterns differ across specific groups, such as younger versus older ICs.

Our study identified several lessons learned from the continued implementation of the informal care platform Caren throughout the COVID-19 pandemic, such as finding innovative solutions to engage the target audience, considering automated user support, taking into account the fluctuation of user groups, and addressing concerns about data transparency in health care. Furthermore, our study shows which platform support functionalities gained relevance throughout the COVID-19 pandemic among both ICs and CRs. The findings can be used to improve the design and implementation of current and future digital platforms to support informal care toward the “new digital normal.”