The FallAkte Plus (FA+) system emerged as a possible candidate due to its potential to comprehensively address the network’s technological requirements. The FA+ system is an implementation of the specification of the Elektronische FallAkte 2.0 (EFA 2.0), developed by the Fraunhofer Institute for Software and Systems Engineering. The EFA 2.0 specification is a blueprint for the implementation of medical data storage, taking international standards (Health Level Seven and Integrating the Healthcare Enterprise) as well as data privacy regulations into account (Figure 1).

It emphasizes decentralized storage of patient-related data within Germany and incorporates 2-factor authentication to enhance access security. The EFA 2.0 specification is publicly available and can be used under the condition of acknowledging the copyright [27]. FA+ is available on the market for service providers in the German health care system, including hospitals, practices, and physician networks [28,29].

Data protection in the FA+ system is ensured through various measures, including the use of a virtual private network, an association of statutory health insurance physicians (KV-Connect), and the telematics infrastructure. KV-Connect is the secure, privacy-compliant communication service provided by the Associations of Statutory Health Insurance Physicians (Kassenärztliche Bundesvereinigung) and the National Association of Statutory Health Insurance Physicians (Kassenärztliche Bundesvereinigung) in Germany [30]. Telematics infrastructure digitally connects all stakeholders in the statutory health insurance system, enabling secure cross-sectoral exchange of patient data among health care professionals [31]. The servers at the data center of Aachen University Hospital, where the FA+ system is hosted, hold technical inspection association level 3 certification, signifying rigorous testing for a high level of cybersecurity assurance and compliance with strict international standards. Additionally, they adhere to ISO (International Organization for Standardization) 27001-certified processes to ensure data safety. Compliance with the general data protection regulation in the European Union ensures that it meets the requirements for data security.

To assess the usability and acceptance of the FA+ system in its current stage of development, we combined qualitative and quantitative methods to gather data. The qualitative analysis aimed to achieve an in-depth understanding and captured nuanced feedback, while the quantitative analysis measured usability and acceptance.

The think-aloud technique was used to explore the user experiences of the FA+ interface among researchers and practice personnel [32]. The researchers from 7 institutes and practice personnel from 10 practices were invited by mail and in person. Participants received emails with login credentials for registering in the FA+ system. They were further provided with a link to an on-demand tutorial video demonstrating the step-by-step process of installing and logging into the FA+ system. Digital sessions were subsequently arranged for groups of participants (n=1-3). Participants were instructed to verbalize each step they took while using the interface, including tasks such as installation, login, and use of the FA+ system. This test included a brief questionnaire on vaccines, serving as an example to assess the system’s usability and user experience. The interviews were transcribed and coded according to the grounded theory approach [33].

A web-based survey was developed using the German SoSci Survey platform (SoSci Survey Co). The survey was conducted among researchers from 7 institutes, and GPs and PrAs from 10 general practices. The survey consisted of three different instruments to assess the users’ experiences with the FA+ user interface: (1) the system usability scale (SUS) is a widely used and validated tool for assessing system usability [34]. Using a 5-point Likert scale, it consists of 10 questions and results in a score of 0 to 100, with higher scores indicating better perceived usability [34]. Systems with scores above 85 are considered “excellent,” those with scores 71-84 “good,” and those with SUS scores of 51-70 “Ok” [34]. (2) The German school-grade scoring system (1=excellent to 5=insufficient) was used to evaluate users’ experiences with the FA+ system as well as the training sessions using 17 different self-developed items. (3) The users’ technical affinity was measured with the 9-item affinity for technology interaction (ATI) scale [35]. This instrument was included to distinguish problems caused by the user interface from problems arising from users’ limited technical capabilities. A higher median technical affinity score suggests better ATI, while lower values suggest poorer ATI.

A total of 36 respondents participated: 24 researchers from the NRW-GPRN-associated university institutes and 12 participants from general practices, including 8 family physicians and 4 PrAs. The think-aloud interviews had an average duration of 60 (range 30-75) minutes. The results of the think-aloud observation protocols are presented in Table 1.

All participants, including both researchers and practice personnel, found several aspects of the FA+ system to be highly favorable, including the overview of patients and studies, the ease and speed of data entry, and the search function. PrAs found the summary overview before creating a patient record particularly helpful, allowing for smoother patient management.

Despite the overall positive assessments, the participants identified some challenges and issues. Researchers highlighted specific concerns related to data protection (password requirements too lax and patient confidentiality). Additionally, the system erroneously marked emails as “read” after accessing the inbox, which could lead to communication errors. Participants, including researchers and practice personnel, raised critical concerns regarding data security and protection. They reported that patient data, including records not created by the user, were visible to unauthorized individuals. This included sensitive information such as the patient’s surname, first name, and date of birth. This raised significant concerns about the confidentiality and privacy of patient data within the system, necessitating urgent attention and improvements in data security measures to safeguard sensitive patient information.

Practice personnel from various practices experienced difficulties with the user interface when managing multiple studies within the FA+ system, indicating room for improvement in this aspect. However, several concerns were raised by participants within the scope of the user training. A common issue mentioned by all participants was the speed of the video explanation, which was considered too fast and at times confusing. Users requested a navigation function enabling them to quickly jump to the “relevant” sections of the video. Practice personnel also expressed concerns about the training materials, noting that they were not tailored to their specific roles. They found the instructions to be overly complex, hindering their ability to effectively use the training resources.

The questionnaire was voluntary; hence, a total of 21 of 36 participants completed the survey (response rate=60%). The median system usability score was 60/100 (IQR 33.0-85.0) points. There were notable differences in the SUS score among the groups: the GPs or PrAs rated FA+ as user-friendly with a median SUS score of 82 (IQR 58.0-94.0) points, indicating nearly good usability. With a median score of 44 (IQR 14.0-61.5) points, NRW-GPRN researchers gave the system a poor rating. The detailed results can be found in Table 2.

The results from users’ experiences with the FA+ system using the German school grading system showed a similar discrepancy between GPs or PrAs and researchers. Among all participants, the functionalities of the FA+ system (GPs or PrAs: median 1.5, IQR 1.0-2.8 and researchers: median 3.0, IQR 2.8-4.0) and the preceding training (GPs or PrAs: median 1.0, IQR 1.0-2.0 and researchers: median 2.5, IQR 1.0-3.0) received excellent to satisfactory ratings. The researchers rated the following items as good: creating a folder for new study participants (median 2.0, IQR 2.0-2.0), computer settings (median 2.0, IQR 2.0-3.0), and filling out the questionnaire (median 2.0, IQR 2.0-2.8). The researchers were more critical and rated 3 items (data protection, data security, and speed of the system) with scores of 4.5, 4.0, and 4.0, respectively. However, compared to GPs and PrAs, the researchers rated these aspects lower by 2.5, 2, and 2 grades, respectively (Table 3).

The users’ median technical affinity score was 3.0 (IQR 3.0-4.0) for the GPs or PrAs and 4.0 (IQR 3.0-4.0) for the researchers. In the total sample, the median score was 3.5 (IQR 3.0-4.0). Categorizing the results, approximately 4 (17%) of both GPs or PrAs and researchers showed a high technical affinity, while the majority of all participants (n=17, 71%) fell into the medium technical affinity group. For more details, refer to Table 4.