Rising costs, fragmented care, aging populations, and chronic disease burden challenge health care sustainability [1]. Evaluating health outcomes is essential to assess system performance. Value-based health care (VBHC) focuses on health outcomes relative to the costs of achieving them [2]. Health care systems have begun implementing these principles [3], but patient-centered outcome data are not consistently collected in routine care. Performance assessment should include outcomes relevant to patients and the populations served [4].

Patient-reported outcomes (PROs) are one such crucial metric. PROs are direct reports from a patient about their health condition without influence by a clinician or anyone else [5]. One method of measuring these health aspects involves using validated patient-reported outcome measures (PROMs), which patients complete to assess the effects of symptoms, treatments, or both on their functional status, psychological well-being, social functioning [6], and overall health-related quality of life [7,8]. By capturing outcomes from the patient’s perspective, PRO data can influence decision-making across multiple health care stakeholders, improve care pathways and health policies, inform health technology assessments, support medicine development and regulatory approval, and facilitate better communication between patients and health care professionals (HCPs) [5,8,9]. Evidence suggests that monitoring self-reported symptoms and ensuring timely responses from health care teams could lead to treatment adjustments and even improved patient survival [10].

Despite established use in clinical trials and medicine development [5,11], PROMs remain incompletely integrated into routine care. Paper collection and manual data entry introduce risks of transcription errors and delays, as documented in clinical data workflows [12-15]. PROM data can support both provider improvement and individual care, but these uses require appropriate collection and feedback arrangements [8,9].

Systematic and electronic patient-reported outcome measures (ePROMs) could enable real-time symptom monitoring, detect changes in disease progression, and facilitate timely interventions [16]. PROs should form a central component of the broader health outcome measurement framework [17-19], as their assessment is critical in promoting a patient-centered approach to health care. This approach empowers patients through self-management, engagement, and active participation in their care.

Clinical data, PROs, patient characteristics, and care-process data need to be considered together. Core outcome sets (COSs), developed through stakeholder consensus, standardize which outcomes should be collected for specific conditions [20,21]. Their practical value also depends on how those outcomes are collected and used.

The Health Outcomes Observatory (H2O) promotes VBHC through systematic collection and use of patient-relevant outcomes [22]. National observatories in Austria, Germany, the Netherlands, and Spain initially focused on diabetes, breast and lung cancer, and inflammatory bowel disease. This feasibility study evaluated patient and HCP experiences with the local technologies and workflows used to collect and review ePROMs in routine care.

The primary objective was to evaluate the usability and perceived clinical utility of the national H2O ePROM infrastructure for routine care delivery from the perspectives of patients and HCPs. The secondary objectives were to identify the implementation factors most critical to successful ePROM integration into the diverse clinical practice settings and to understand current stakeholder acceptance, facilitators, and perceived barriers that could hinder implementation and scalability.

This multicenter, prospective, pre-post study used a convergent mixed methods design with embedded qualitative components. Quantitative and qualitative data were collected in parallel with equal priority and integrated during interpretation. This feasibility design was selected as survey measures could quantify usability and satisfaction, while interviews could better explain the technical, workflow, and organizational conditions underlying quantitative results. One site in each of Austria, Germany, the Netherlands, and Spain participated, covering 1-4 indications per site. Local ethical approval was obtained at each site, and COS data collection was classified as noninterventional under local regulations. Additional protocol details are available elsewhere [23]. The mixed methods reporting of this study follows the Good Reporting of a Mixed Methods Study (GRAMMS) framework (refer to Multimedia Appendix 1 for the completed checklist).

H2O is a federated digital infrastructure. It combines disease-specific COS, locally selected patient-facing mobile or web apps, hospital dashboards or data-capture systems, and national governance arrangements. Site personnel enrolled and registered patients, provided training, and prompted completion of PROMs at baseline and scheduled follow-ups (electronic follow-up completion reminders were also provided). Patients completed questionnaires through the locally implemented app using a smartphone, tablet, or computer. Clinicians reviewed data through locally available dashboards or clinical tools; integration with electronic health records (EHRs) differed by site, and incomplete interoperability was itself an implementation limitation.

“H2O digital infrastructure” denotes the federated ecosystem, “local app” the patient-facing tool, and “study database” Clincase, which held feasibility evaluation data rather than supporting routine clinical review.

Collecting sites did not provide complete app names and versions, hosting and storage locations, EHR registration pathways, or PROM-to-EHR interfaces. These technical details could not be specified consistently across centers.

Feasibility was defined and operationalized as the ability to deploy and sustain digital ePROM collection under routine-care conditions across participating sites. Usability referred to perceived ease of use and interaction quality, assessed using the System Usability Scale (SUS) and process indicators. Acceptability referred to willingness to recommend and favorable or unfavorable experiences described in interviews. Implementation referred to the technical, workflow, staffing, and organizational conditions affecting deployment. Perceived clinical utility referred to whether patients and HCPs considered the collected PRO data useful for monitoring, communication, and clinical decision-making.

Each site used locally selected technology compatible with smartphones, tablets, or web browsers to capture and display PROs. Data architecture and access arrangements followed the relevant national H2O framework and local privacy and cybersecurity requirements. The degree of EHR integration differed by site, and some clinicians relied on separate dashboards or tools to review PROM data. Participation did not alter routine care; treatment decisions and disease monitoring remained at the treating physician’s discretion, and no additional study-specific clinic visits were required.

The study lasted 22 months, from first consent in September 2023 to final completion in July 2025 (Figure 1). Follow-up was planned for 12 months per participant, with study initiation and assessment timing varying according to site-level implementation of the COS and technology. Quantitative data assessed system usability, patient satisfaction with care, perceived utility, and process indicators. In parallel, interviews with patients and HCPs explored experiences, acceptability, and implementation context.

Implementation and follow-up timing were not uniform across centers. One technology provider supporting an observatory became insolvent during data collection, delaying data retrieval. Other disruptions included prolonged technology implementation and troubleshooting, PROM licensing and translation barriers, and site-level modifications to planned procedures. Although the Patient Satisfaction Questionnaire Short Form (PSQ-18) follow-up was planned for 12 months, actual pre-post intervals ranged from 6 to 16 months. These deviations were treated as implementation constraints when interpreting feasibility and comparative results.

The collecting sites did not provide complete event-level records of vendor changes, software failures, protocol modifications, or dates of disruption and recovery. Consequently, the reported events cannot be assigned reliably to individual centers or linked quantitatively to outcomes. Site-level noncollection, including the absence of German PSQ-18 data, was distinguished from participant nonresponse when interpreting data availability.

Participating observatories followed national legal and ethical governance arrangements. Consent materials and measures were adapted to local systems and languages while retaining common study procedures. All data transferred for analyses were received as pseudonymized and deidentified. Use of the data was limited to the scope of this study (ie, no secondary use). The master version of the informed consent, which was adapted for use by each site, is provided in Multimedia Appendix 2. Ethical approval was obtained from local institutional review boards of the Medical University of Vienna (EK1191/2023), Vall d’Hebron University Hospital (PR(AG)216/2023), Erasmus Medical Center (EC-2023-0735), and Charité University Hospital (EA4_034_22).

Baseline included the PSQ-18; SUS was scheduled at 6 months, the Perceived Utility Questionnaire at 6 and 12 months, PSQ-18 follow-up at 12 months, and interviews at 12-16 months. Actual paired PSQ-18 intervals ranged from 6 to 16 months; the schematic does not represent uniform achieved follow-up. Six patients per disease group and 3 HCPs specializing in each disease area were invited to interviews conducted 12-16 months after enrollment.

Patients diagnosed with metastatic breast cancer (MBC), lung cancer (LC), Crohn disease (CD), ulcerative colitis (UC), or type 1 or type 2 diabetes mellitus (DM) who were receiving treatment at the Vienna General Hospital, Erasmus Medical Center, Vall d’Hebron University Hospital, and Charité University Hospital, along with their HCPs, were included in the study. Patients had to be at least 18 years old and able to read and understand the relevant local languages. Finally, all participants had to be willing and able to complete data collection using an electronic device. Each participant signed an informed consent form (or provided electronic confirmation) indicating understanding of agreement to participate in the feasibility study.

The study aimed to enroll a minimum of 200 patients, with 50 participants per site and at least 25 per disease group. Eligible participants had a diagnosis of type 1 or type 2 DM, CD, UC, MBC, or LC. No formal power calculation was performed for this exploratory study. Targets reflected historical clinic recruitment rates and the feasibility of obtaining descriptive and qualitative data.

HCPs and clinical staff at participating hospital centers recruited consecutive eligible patients during on-site clinical care, according to the usual clinical practice at each center. Informational materials supported the recruitment process and were reviewed with the patients. The HCP inviting the patient explained the purpose of the study and the role of PROMs and answered any questions before the patient decided whether to participate. Following this, interested patients provided written or electronic informed consent before participation. After consent, HCPs, with potential support from technology providers, guided patients on how to use the app and access the questionnaires. A purposive, maximum-variation sampling stratified by clinical condition was chosen. A subset of participants, stratified by age, gender, and education, was invited to participate in semistructured qualitative interviews. Patients could withdraw from the study at any time without explanation.

The purposive interview strategy described the intended sampling approach. The achieved sample was uneven across countries. Complete participant-level information on age, gender, education, and professional background was not provided by participants or collecting sites, preventing verification that the intended variation was achieved. The country-level demographic summaries available for the interviews are reported in the “Results” section.

The study collected disease-specific COS data, including PROs, clinical measures, and demographics. Additional feasibility study data included patient satisfaction with care and utility questionnaires, and qualitative interview responses. Data were collected from patients and HCPs through qualitative interviews and quantitative questionnaires (Multimedia Appendix 3).

The SUS assessed perceived usability. This widely used 10-item instrument covers ease of use, complexity, and confidence [24]. Items are rated on a 5-point scale from strongly disagree to strongly agree and converted using the standard scoring procedure to a total score from 0 to 100, with higher scores indicating better usability. SUS scores are scale scores, not percentages. For the descriptive interpretation used here, scores below 50 indicated poor usability and scores above 80 indicated very good usability [24]. These benchmarks describe perceived usability rather than clinical effectiveness or implementation readiness. Published evidence supports use of the SUS across technologies; however, internal consistency was not estimated for the present sample.

The PSQ-18 assessed satisfaction with health care services. Its 18 items form 7 subscales covering general satisfaction, technical quality, interpersonal manner, communication, financial aspects, time spent with the doctor, and accessibility and convenience [25]. Items are rated on a 5-point scale and scored according to the published guidance, including reverse coding, so that higher scores indicate greater satisfaction. The PSQ-18 was planned at baseline and 12-month follow-up, although actual paired assessment intervals varied from 6 to 16 months. Internal consistency was not estimated for the present sample.

The PSQ-18 technical-quality subscale measures satisfaction with the technical quality of medical care; it does not directly measure the technical performance of software. Published measurement evidence does not establish validity in every clinical or language context. Complete item-level data needed to estimate internal consistency of SUS and PSQ-18 scores in this sample were not provided by the collecting sites; study-specific reliability and cross-language measurement equivalence could therefore not be evaluated.

The Perceived Utility Questionnaire (labeled by this name in Figure 1) was developed for the study to assess participants’ experience of completing PROMs through the local H2O app. It collected gender, age group, and education and asked about willingness to recommend participation, ease of answering questions, time to completion, and ease of correcting responses using 0-10 numeric rating scales. It also recorded completion setting and device type. These items were used as indicators of practical usability, acceptability, and perceived utility; the questionnaire was not treated as a previously validated measurement instrument.

Age groups were collected in the Perceived Utility Questionnaire, whereas the demographic summaries report age in years as means and SDs.

Patient interview guides explored daily life, self-care, communication, use of PRO data during consultations, access barriers, and suggested improvements to training and app design. HCP guides covered clinical usefulness, patient-provider relationships, workflow integration, and implementation barriers. Both used open-ended questions on usability and perceived value.

Of 241 participants at the first assessment, 115 (47.7%) contributed any follow-up data, representing attrition of 126 (52.3%). Denominators differed across measures because of participant nonresponse and site-level noncollection. Completion setting, onboarding staff role, and device data were available for 120/241 (49.8%), 116/241 (48.1%), and 144/241 (59.8%) participants, respectively (Table 1). The SUS was completed by 205 participants, and 50 paired PSQ-18 records were available for the pre-post comparison. Missing data were analyzed using available cases without imputation. Relative to the 241 enrolled participants, completion-setting data were unavailable for 121 (50.2%), onboarding-role data for 125 (51.9%), device data for 97 (40.2%), and SUS data for 36 (14.9%). Fifty paired PSQ-18 records represented 20.7% of the enrolled cohort; this is an analysis-availability proportion, not a PSQ-18–specific dropout rate. Detailed enrollment, withdrawal, and noncompletion counts by site, disease, and scheduled time point were not provided by the collecting sites; so, a complete attrition matrix and reasons for individual noncompletion could not be reconstructed.

Across participants, the mean age was 46 (SD 14.4) years, and 130/241 (54%) were female (Table 2).

SUS data were available for 205 participants, and 195 unique PSQ-18 assessments were reported for the PSQ-18. Table 3 presents the supplied assessment counts by country. The interview column combines 26 patient and 8 HCP interviews.

Among records with process data, patients were most often recruited or onboarded by a physician or other HCP and completed PROMs on a smartphone at home or work (Table 1). Mean completion time was 231.34 (261.1) seconds (approximately 4 minutes; Table 4). Across 205 SUS respondents, the mean score was 44.65/100 (SD 9.78), indicating poor overall usability; site-level scores ranged from 26 to 86. No association with disease was reported. The German site did not collect PSQ-18 data.

Overall, 44/53 (83%) respondents indicated that they would recommend H2O. No association with disease, age, or sex was reported. Willingness to recommend differed across device groups in an omnibus comparison (n=49; F₃,₄₅=3.51, P=.02; η²=0.190, 95% CI 0.002-0.343); this result alone does not establish a smartphone-specific difference. Of respondents willing to recommend H2O, 62% were smartphone users and 63% completed PROMs at home or work; these are distributions among recommenders, not within-group recommendation rates. Reported correlations with the device-use variable were r=0.80 for readability (P<.001), r=0.70 for response revision (P<.001), and r=−0.38 for initiation time (P=.005). Device coding and analysis-specific denominators were not supplied, so directional interpretations were not assigned. An association between SUS total score and willingness to recommend was reported (R²=0.49, P=.009), but the model specification and sample size were unavailable for independent verification.

The PSQ-18 comparison was limited to 50 participants with baseline and follow-up data. Follow-up was planned for 12 months, but actual intervals varied across sites from 6 to 16 months. Of the 7 PSQ-18 subscales, technical quality declined at follow-up (t₄₉=2.20, P=.03). No statistically significant changes were reported for general satisfaction, interpersonal manner, communication, financial aspects, time spent with the doctor, or accessibility and convenience, and no center-level difference was detected. Usability and onboarding staff role were not associated with patient satisfaction. Because the analysis used a small completer-only sample, these estimates are vulnerable to selection bias and should be interpreted cautiously.

Baseline and follow-up subscale means and SDs, paired mean changes, and their CIs were not supplied with the aggregate test results. The magnitude and clinical importance of the reported decline therefore cannot be determined from the P value alone. Nonsignificant comparisons should not be interpreted as evidence of equivalence or absence of an effect.

Thirty-four interviews were conducted during the second half of the study: 26 with patients and 8 with HCPs. Of these, 23/34 (mean age 42, SD 13 years; 82% female) were conducted in Spain, 7/34 (mean age 66, SD 15 years; 57% female) in the Netherlands, 4/34 (mean age 46, SD 21 years; 50% female) in Austria, and none in Germany. The themes therefore primarily reflect Spanish implementation and should not be interpreted as representative of all 4 national infrastructures.

The available age and sex summaries were not accompanied by complete demographic denominators or separate patient and HCP profiles. Education, HCP professional roles, and a country-by-participant-type interview roster were not provided by participants or collecting sites. These omissions limit assessment of the achieved maximum-variation sample.

Seven patient themes were identified (Multimedia Appendix 4). Ease of navigation concerned participants’ descriptions of simple navigation. Technical access and login problems concerned difficulty entering or using the local app. Reflection on physical and psychological health concerned the opportunity to consider symptoms and well-being when completing PROMs. Communication with the care team included frustration when responses were not discussed during appointments.

Perceived monitoring and research value concerned the potential usefulness of reporting health experiences and contributing data to research. Questionnaire design and personalization concerned restricted response options and requests for more personalized and engaging questionnaires. Unmet mental health support concerned participants’ requests for greater attention to psychological needs. These findings describe reported experiences and preferences, rather than demonstrated improvements in clinical outcomes.

Five HCP themes were identified (Multimedia Appendix 4). Clinical value of PROs concerned the additional view of patients’ health experiences offered by PROM collection. Identification of psychosocial and adherence concerns concerned issues that might otherwise remain unrecognized in routine consultations.

Workflow and EHR integration concerned poor synchronization with clinical tools, limited review of responses, and missed opportunities to discuss findings. Technical and digital-inclusion barriers concerned usability difficulties, particularly for older or less digitally experienced patients. Staff training and organizational support concerned the need for staff preparation and explicit routines for reviewing and acting on PROM data. HCPs supported PROM integration in principle while identifying practical conditions needed for routine use.

The findings from this H2O feasibility study present a mixed picture. There is positive evidence for the perceived utility and acceptability of the H2O system from both patients and HCPs, while technical usability of the system was unsatisfactory. Willingness to recommend was retained as one global indicator of acceptability [27,28], but it should not be interpreted as a proxy for usability. Although 44/53 (83%) respondents indicated willingness to recommend H2O, the mean SUS score of 44.65 (SD 9.78) indicated poor usability; technical-quality satisfaction declined among the 50 paired PSQ-18 respondents. While the collection of PROMs via smartphone was predictive of willingness to recommend, further testing is required to accurately interpret this relationship. These results, together with workflow disruption and uneven site participation, limit conclusions about sustained use and require cautious interpretation beyond the participating settings and completer-only sample.

Patients valued reflection on health and research participation, consistent with earlier PROM studies [29-31]. Login difficulties, repetitive questions, and limited response options echoed reported barriers involving questionnaire fatigue and digital literacy [29,32]. Secure authentication remains necessary, but access should be usable. These findings primarily reflect that the Spanish interview sample of HCPs recognizes the value of collecting PROMs to better understand patients’ health experiences and improve care quality. However, challenges remain, including difficulty reviewing responses in a timely manner, limited usability, and poor integration with existing clinical tools. Integrating PROM data into EHR was emphasized as crucial for facilitating consultations and informed discussions. Patient engagement, especially among older or less tech-savvy individuals, highlights the need for staff training to effectively support PROM use. These findings align with previous qualitative studies and systematic reviews reporting similar barriers and the importance of training and system integration for successful PROM implementation [33-35].

The quantitative and qualitative strands converged in showing perceived value alongside substantial technical friction. Interviews helped explain how willingness to recommend could coexist with low SUS scores: participants valued remote reporting, reflection, communication, and the purpose of PROM collection, while describing login problems, unstable technology, limited clinical follow-up, and weak integration with routine tools. HCP accounts similarly contextualized site-level variability through workflow constraints, staffing, training needs, and limited interoperability. The strands therefore support conditional, rather than unqualified, feasibility; that is, perceived value depended on reliable technology and meaningful use of PROM data in care.

The mean SUS score of 44.65 (SD 9.78) was in the poor range. Its association with willingness to recommend does not offset the low absolute usability rating; its direction requires confirmation of recommendation coding and model specification. Variation across sites may reflect differences in local apps and implementation, but causal attribution is not possible without adjusted analyses accounting for site and missingness. Usability optimization remains a central requirement for future PROM systems.

Among the 50 participants with paired PSQ-18 data, satisfaction with the technical quality of medical care declined. This domain does not directly assess software reliability, so the finding cannot be equated with deterioration in platform performance. Interviews identified technology and workflow problems, but these accounts do not establish the cause of the PSQ-18 change. The small completer-only sample, variable 6- to 16-month intervals, multiple comparisons, and absence of adjustment for site or nonrandom missingness limit interpretation. Nonsignificant changes in other domains do not establish unchanged satisfaction.

Taken together, the findings identify usability and technical reliability as priorities for future implementation. Future studies should evaluate local apps across devices and settings using larger, more complete datasets and clearly documented assessment schedules. Whether such changes improve patient engagement or satisfaction with care requires further evaluation.

The multinational, cross-disease design enabled evaluation across heterogeneous clinical and technical contexts and helped identify recurring implementation challenges. However, transferability beyond the participating settings is limited by substantial attrition, uneven site and disease representation, incomplete assessment coverage, and the geographic imbalance of the qualitative sample.

Provider insolvency, implementation delays, and licensing and translation barriers produced uneven technology exposure and follow-up conditions.

Recruitment and data collection depended on HCP engagement and local staffing, technical expertise, and turnover. These constraints contributed to inconsistent sample sizes, missing data, and variable measurement timing. Attrition was 52.3%; only 50 paired PSQ-18 records were analyzed, the German site did not collect PSQ-18 data, and no missing-completely-at-random test, imputation, sensitivity analysis, or adjustment for site clustering was performed. The qualitative sample was also imbalanced: 23/34 interviews came from Spain, 7 from the Netherlands, 4 from Austria, and none from Germany. The themes therefore primarily reflect Spanish workflows and should not be generalized to all 4 national infrastructures.

The national observatories were established independently of this study, and detailed site-specific architecture, vendor-change logs, software-failure records, and protocol-deviation timelines were not provided by the collecting sites. The effect of individual disruptions could therefore not be quantified, which limits reproducibility. Study-specific internal consistency estimates, completer-noncompleter comparisons, and PSQ-18 effect estimates and CIs were also unavailable; the device ANOVA includes η² and its 95% CI.

Mixed methods integration was limited by unequal coverage of countries, time points, and participants across the 2 strands. Interview accounts cannot establish why noncompleters discontinued participation or explain quantitative results from Germany, where no interviews were collected. Translation and AI-assisted summarization may also have altered nuance, particularly after removal of interview questions and labels. Apparent convergence should therefore be interpreted as contextual support rather than independent validation of survey estimates.

The H2O digital infrastructure could be deployed across heterogeneous settings, and patients and HCPs described value in systematic digital PROM collection. However, willingness to recommend should be interpreted alongside poor overall usability, declining technical-quality satisfaction in a small paired sample, substantial attrition, uneven site participation, and limited interoperability. The findings support conditional feasibility rather than broad implementation readiness. Reliable technology, consistent follow-up, integration of PROM results into clinical workflows, and adequately powered analyses that address clustering and missingness are needed before effectiveness or generalizability can be established. The successful routine collection of PROMs, which depends substantially on the ease of use and usability of the underlying technology, is essential for effectively incorporating the patient perspective into clinical care. Furthermore, the collection of PROMs across diseases and national borders offers substantial potential for research, including comparative evaluations of treatment outcomes. However, realizing this potential requires robust, well-implemented, and adaptable systems that are practical for routine clinical use.