Published on in Vol 8 (2024)

Preprints (earlier versions) of this paper are available at, first published .
Remote Symptom Monitoring Using Patient-Reported Outcomes in Patients With Chronic Kidney Disease: Process Evaluation of a Randomized Controlled Trial

Remote Symptom Monitoring Using Patient-Reported Outcomes in Patients With Chronic Kidney Disease: Process Evaluation of a Randomized Controlled Trial

Remote Symptom Monitoring Using Patient-Reported Outcomes in Patients With Chronic Kidney Disease: Process Evaluation of a Randomized Controlled Trial

Original Paper

1AmbuFlex, Centre for Patient-Reported Outcomes, Gødstrup Hospital, Herning, Denmark

2Department of Clinical Medicine, Aarhus University, Aarhus, Denmark

3Department of Rheumatology, Aarhus University Hospital, Aarhus, Denmark

4Department of Renal Medicine, Aarhus University Hospital, Aarhus, Denmark

5Department of Clinical Epidemiology, Aarhus University, Aarhus, Denmark

Corresponding Author:

Birgith Engelst Grove, RN, MHSc, PhD

AmbuFlex, Centre for Patient-Reported Outcomes

Gødstrup Hospital

Møllegade 16

Herning, 7400


Phone: 45 28904835


Background: In Denmark, outpatient follow-up for patients with chronic kidney disease (CKD) is changing from in-hospital visits toward more remote health care delivery. The nonuse of remote patient-reported outcomes (PROs) is a well-known challenge, and it can be difficult to explain which mechanisms of interventions influence the outcome. Process evaluation may, therefore, be used to answer important questions on how and why interventions work, aiming to enhance the implications for clinical practice.

Objective: This study aimed to provide insight into the intervention process by evaluating (1) the representativity of the study population, (2) patient and physician use patterns, (3) patient adherence to the intervention, and (4) clinical engagement.

Methods: A process evaluation determining the reach, dose, fidelity, and clinical engagement was carried out, alongside a multicenter randomized controlled trial (RCT). We developed and implemented an intervention using PRO measures to monitor outpatients remotely. Data were collected for the PRO intervention arms in the RCT from 4 sources: (1) PRO data from the participants to determine personal factors, (2) the web-based PRO system to identify key usage intervention patterns, (3) medical records to identify clinical factors relating to the use of the intervention, and (4) semistructured interviews conducted with involved physicians.

Results: Of the 320 patients invited, 152 (47.5%) accepted to participate. The study population reflected the target population. The mean adherence rate to the PRO intervention arms was 82% (95% CI 76-87). The questionnaire response rate was 539/544 (99.1%). A minority of 13 (12.9%) of 101 patients needed assistance to complete study procedures. Physicians assessed 477/539 (88.5%) of the questionnaires. Contact was established in 417/539 (77.4%) of the cases, and 122/539 (22.6%) of the patients did not have contact. Physicians initiated 288/417 (69.1%) and patients requested 129/417 (30.9%) of all the contacts. The primary causes of contact were clinical data (242/417, 58%), PRO data (92/417, 22.1%), and medication concerns and precautionary reasons (83/417, 19.9%). Physicians found the use of PRO measures in remote follow-up beneficial for assessing the patient’s health. The inclusion of self-reported clinical data in the questionnaire motivated physicians to assess patient responses. However, some barriers were emphasized, such as loss of a personal relationship with the patient and the risk of missing important symptoms in the absence of a face-to-face assessment.

Conclusions: This study demonstrates the importance and practical use of remote monitoring among patients with CKD. Overall, the intervention was implemented as intended. We observed high patient adherence rates, and the physicians managed most questionnaires. Some physicians worried that distance from the patients made it unfeasible to use their “clinical glance,” posing a potential risk of overlooking crucial patients‘ symptoms. These findings underscore key considerations for the implementation of remote follow-up. Introducing a hybrid approach combining remote and face-to-face consultations may address these concerns.

Trial Registration: NCT03847766;

JMIR Form Res 2024;8:e48173



Lifestyle and a growing elderly population in Denmark have increased the number of patients with chronic diseases to approximately 1 million in a population of 6 million, with growing health care expenditure as a consequence [1]. To improve efficacy, new ways of delivering health care to people with chronic kidney disease (CKD) have been posited, such as remote monitoring, which is defined as using technology to monitor patients at a distance [2,3]. One way of remotely monitoring patients is to collect information about their symptoms by obtaining repetitive patient-reported outcomes (PROs) while the patients are at home. This allows for frequent capture of important disease-specific outcomes and also allows clinicians direct access to the patients’ health status [4,5]. PROs are measures of a patient’s health conveyed directly by the patient, without interpretation by a clinician or anyone else [6], often gathered from electronic questionnaires. Remote monitoring using PROs is increasingly applied among patients with chronic conditions in Denmark [4,7]. However, several personal and external factors may affect a person’s ability to engage with and use digital health interventions [8,9]. Prior studies have shown sociodemographic and economic inequality in patients who attend [10,11] and adhere to PRO-based remote follow-up [12].

A pragmatic randomized controlled trial (RCT) named PROKID (PRO measures in Kidney care) is currently investigating whether PRO-based remote follow-up is a safe and effective alternative to health care delivery to patients with CKD [7]. The primary outcome in the PROKID trial is renal function measured using the estimated glomerular filtration rate (eGFR) as this is the single-most accurate measurement for CKD progression [13] and thereby indicates the level of safety in remote monitoring. The effectiveness of remote monitoring is measured using clinical data, resource use, and PROs, such as the quality of life and illness perception [14]; also see Grove et al (unpublished data, 2024). PRO-based interventions in other patient populations have shown the ability to (1) reduce the number of outpatient visits [15,16], (2) improve communication between health care professionals and patients [17-20], and (3) offer patients a more comprehensive understanding and greater management of their condition [21-23]. However, several knowledge gaps regarding the use and acceptability of remote PRO interventions are still present. Trials are vulnerable to various biases that may undermine external validity [24]. Evidence suggests that often, this risk of bias is related to patients not using the technology as intended [25]. Thus, examining the quality (fidelity) and quantity (dose) of what was implemented in practice and the extent to which the intervention reached its intended users is essential when evaluating trials [26]. The degree to which patients engage with and use the intervention as intended is a crucial component in evaluations [26]. The nonuse of remote PRO is a well-known challenge [16,27] and may blur the overall interpretation of using PRO as the basis for remote follow-up. However, little is known about the components that might enhance patient and clinical engagement and thus might have the most significant impact in terms of use and adherence to the intervention. The PROKID trial [14] cannot explain which mechanisms of the intervention have an influence on the outcome. Process evaluation may therefore be used to answer important questions on how and why interventions work, aiming to enhance the implications for clinical practice [26].

Thus, we aimed to conduct a process evaluation to obtain insights into the PROKID intervention’s working mechanisms by evaluating (1) the representativity of the study population, (2) patient and physician use patterns, (3) patient adherence to the intervention, and (4) clinical engagement.

Study Setting

In 2019, the PROKID trial, a Danish multicenter RCT, was initiated to evaluate the noninferiority of PRO-based remote follow-up compared to usual outpatient clinic visits in managing the decline in renal function and maintaining patients’ quality of life [7]. The intervention was implemented in a real-life setting, and clinicians and patients were involved in the development process. We provided short oral seminars for the staff in the outpatient clinics, where the intervention was discussed and refined. We invited patients to provide input into the design and the patient information sheets of the intervention. Participants were eligible for the trial if they attended follow-up from January 2019 to August 2021 at a renal outpatient clinic in Central Denmark Region and had a renal function of eGFR≤40 mL/minute. In addition, participants had to be ≥18 years old and able to complete a questionnaire. Newly referred outpatients were randomized into the following 3 groups: (1) PRO-based follow-up, (2) PRO-based telephone follow-up, or (3) usual outpatient follow-up (Figure 1).

Figure 1. Recruitment of patients with CKD from Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital to the PROKID trial and process evaluation from January 2019 to August 2021. CKD: chronic kidney disease; PROKID: PRO measures in Kidney care.

In the intervention arms, each patient participated in 6 PRO consultations (when PROs were used in a consultation) during 18 months of follow-up. Prior to each PRO consultation, patients needed to undergo blood tests, measure their blood pressure and weight, and complete a disease-specific questionnaire. The questionnaires were sent to patients 7 days ahead of each consultation, and in the case of nonresponse, they were sent reminders on the fourth day and the day just prior to the consultation. The questionnaire included information about blood pressure, weight, self-rated health, renal-specific symptoms, and a free-text box [28]. The physicians accessed the patients’ PRO responses through a graphical overview embedded in the electronic health record system [5]. In the PRO-based follow-up intervention group, physicians had to approve each patient’s responses, manage them according to a color-coded algorithm, and determine whether the patient needed contact. A clinical expert group has assigned a color to each item response according to the severity of the symptom, as previously described: red, yellow, or green [14,28]. Each patient’s need for contact was evaluated based on their PRO responses and other clinical data, such as blood samples and blood pressure. Therefore, it was up to the physicians to decide whether contact with the patient was needed. This applied not only to instances where the color code was red or yellow but also to instances in which the questionnaire displayed a green color code. In the PRO-based telephone follow-up group, the physicians had to approve each patient’s response, call the patient, and document the conversation as conducted. Figure 2 outlines the groups, the content, and the purpose of using PRO.

Figure 2. Overview of the intervention groups in the PROKID trial including patients with CKD at Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital from January 2019 to August 2021. CKD: chronic kidney disease; PROKID: PRO measures in Kidney care.

Study Design

The process evaluation was an integral component of the PROKID trial and focused on adding information not provided in the trial and clarifying how the intervention was received in practice to increase accuracy in the trial results. The design and methods of the PROKID trial are described elsewhere [7]. The design of this evaluation was influenced by Steckler and Linnan’s [26] process evaluation framework, building on 4 themes: reach, dose, fidelity, and clinical engagement. This study used qualitative and quantitative methods to answer the research questions. The study was reported in accordance with Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth (CONSORT-EHEALTH) guidelines [29].


We obtained data from patients attending outpatient follow-up at the renal outpatient clinics at Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital in Central Denmark Region. In the quantitative phase of the process evaluation, aimed to identify the reach of the intervention, the sample consisted of 320 patients, all eligible for randomization. The PROKID trial recruited 152 (47.5%) patients, and the 105 (69.1%) patients allocated to the PRO intervention groups informed the dose and fidelity of the trial. The qualitative phase of the process evaluation consisted of semistructured interviews with physicians (N=10) involved in the PROKID trial (Figure 1). Physicians were purposefully sampled from all involved outpatient clinics. Of the 10 physicians, 5 (50%) were female, 6 (60%) were senior consultants, and 4 (40%) were consultants, and their ages ranged from 37 to 67 years. Most of them had under 2 years of experience in using PRO in clinical practice.

Process Evaluation Components

The research team identified core process questions for the evaluation stages. These related to describing the quantity and quality of what was delivered, covered by the following 4 components: reach, fidelity, dose, and clinical engagement [26]. An overview of the evaluation components and methods is outlined in Table 1.

Table 1. Research questions and key components in the process evaluation following the PROKIDa trial, Central Denmark Region.
DomainResearch questionsApplied methodData source
  • Who receives the intervention, and is the sample representative?
  • Why do patients disagree to participate?
  • Quantitative personal and clinical profiling of participants and nonparticipants
  • Patient-reported reasons for nonparticipation
  • Hospital Business Intelligence Register
  • AmbuFlex database
  • Questionnaire data/PROsb
  • Who adheres to the intervention?
  • To what extent have the patients received and engaged in the intervention?
  • Quantitative personal and clinical profiling in the level of adherence
  • Number of distributed questionnaires and patient responses
  • Number of item responses
  • Time spent completing the questionnaire
  • Color-coded algorithm
  • Paper/web distribution
  • Number of reminders
  • Hospital Business Intelligence Register
  • AmbuFlex database
  • REDCapc database
  • Is the intervention delivered as intended?
  • Do the physicians incorporate the patients’ PRO responses in the consultation?
  • Number of clinics and physicians using the system
  • Physician assessments of patient responses (contact/no contact)
  • Number of physicians involved in each patient pathway
  • AmbuFlex database
Clinical engagement
  • How do the physicians perceive the intervention in clinical practice?
  • Individual semistructured interviews
  • Involved physicians from all centers

aPROKID: PRO measures in Kidney care.

bPRO: patient-reported outcome.

cREDCap: Research Electronic Data Capture.


Reach was measured by the degree to which the intended target population participated in the intervention. Participation was defined as patients being eligible and willing to participate in the PROKID trial. Participants and nonparticipants were compared according to demographic information and PROs, measured using questionnaires and clinical data collected before entrance to the trial. All patients who declined to participate were encouraged to provide a reason for refusal. A complete overview of the data sources and the PROs and clinical outcomes is presented in Table S1 in Multimedia Appendix 1.


The assessment encompassed measuring the quantity of the intervention components given to the patients, as well as gauging the extent to which patients engaged in and adhered to the intervention. Intervention adherence to PRO-based remote follow-up was exclusively calculated among patients in the intervention groups (N=105). Adherence was defined as the patient being able to complete the questionnaires, measure their blood pressure, and have blood samples taken prior to each of the 6 PRO consultations without assistance from the clinicians or the researcher. The percentage of successfully completed PRO consultations was used as a proxy measure for adherence. An adherence rate of 100% indicated that the patients successfully attended each consultation without assistance from anyone. Engagement in the intervention was measured by calculating as number of questionnaires distributed to patients/number of questionnaire responses received from patients, including the need for sending reminders. The extent of nonresponders was calculated. Furthermore, the item response rate was measured as total number of items distributed to patients/number of item responses. The time spent completing the questionnaire and paper/electronic version distribution was automatically logged in the AmbuFlex database [30].


Fidelity measured the extent to which the intervention was delivered as planned. Fidelity was measured by counting the number of physicians using the system, including the extent to which the physicians assessed the patients’ responses. Furthermore, the mean number of physicians involved in each patient pathway was reported. The patients’ preferences for mode of contact with their physicians, as well as the extent and nature of the clinical assessment of their responses, was obtained. This information was automatically registered in the AmbuFlex database [30]. A study nurse or the project researcher obtained a quantification of patients who needed course support during the trial and registered in the Research Electronic Data Capture (REDCap) system [31]. Course support was defined as the patient asking for help from the researcher or the project nurse regarding completion of the study procedures. Evaluation components of dose and fidelity were described separately for each PRO intervention group.

Clinical Engagement

There is no universal definition of clinical engagement; it may be an attitude, a behavior, or an outcome [32]. We defined clinical engagement as the physicians’ perception, attitude, and satisfaction of using PRO actively in the decision-making processes in remote renal care. The clinical engagement toward using PRO-based remote follow-up in clinical practice was explored among the renal physicians who delivered the intervention. We used a qualitative approach inspired by Braun and Clarke’s [33] 6-phase thematic analysis. Individual semistructured interviews with purposively sampled physicians (N=10) most experienced with PRO-based follow-up were performed. The interviews were conducted by the project researcher (author BEG) with experience in qualitative research and occurred immediately after the physicians completed the patients’ final PRO consultation in one of the involved outpatient clinics. An interview guide was used to elicit clinical engagement, asking about the physicians’ views toward the PRO intervention and how they had experienced following the patients remotely (Table S2 in Multimedia Appendix 2). During each interview, notes were written down by BEG, and statements were summed up at the end of each interview and confirmed by the interviewee. These notes constituted the data material. For analysis of all the notes, an analytic coding scheme was developed based on the interview questions and an initial reading of the notes from the interviews (Table S3 in Multimedia Appendix 3) [34]. Citations used in this paper have been translated from Danish.

Data Analyses

Quantitative data were analyzed using descriptive statistics, such as frequencies, means (SDs), or median (IQRs), as appropriate. Descriptive quantitative information on reach, dose, and fidelity was provided. The presentation of variations between patients in the 2 intervention arms in terms of dose and fidelity were presented. The participation rate was calculated as number of patients consenting to participate/number of eligible patients. Reasons for nonparticipation were synthesized into categories by 2 researchers. We classified patients with low adherence by using the lower quartile. Thus, adherence<83% was the threshold defining low adherence to PRO-based remote follow-up. The adherence threshold was estimated from patients in both PRO intervention arms. Deceased and patients who withdrew due to ending follow-up at the hospital were excluded from the analyses. Participation and adherence data were linked to demographic data collected prebaseline of the trial. Personal and clinical characteristics of high or low adherence were described. Differences were determined using the χ2 test.

Notes from the semistructured interviews constituted the qualitative data. We performed a thematic analysis to extract important aspects that influenced the physicians’ engagement toward PRO-based remote follow-up. First, all notes were read and re-read several times for familiarization with the data. Second, for an overview of the complete data, statements for each patient were entered into a Microsoft Excel spreadsheet by interview question. Third, preliminary themes that were relevant to the aim of evaluation were identified. Fourth, a coding framework was developed based on the initial themes. Finally, themes were identified and reviewed against the original statements and context. The quantitative and qualitative analyses and reporting were conducted prior to knowing the trial outcomes to avoid biased interpretation of the results [26].

Ethical Considerations

Patients provided written consent to participate in the trial, including the use of information from their medical records, the AmbuFlex database, and registers. Additionally, verbal consent was obtained from the physicians participating in the interviews. The study was approved by the Danish Authorities for Health Research (number 1-45-70-8-22).


In total, 1060 patients with CKD were screened for participation in the PROKID trial (Figure 1). Of these, 320 (30.2%) patients were found eligible, and 152 (47.5%) agreed to participate in the PRO-based remote follow-up intervention. No statistically significant differences in patient and clinical factors were found between participants and nonparticipants (Table 2). A tendency toward lower participation by older age and lower health literacy was seen. The reasons for not participating in the PRO-based intervention are outlined in Table 3.

Table 2. Characteristics of patients with CKDa (N=320) among participants and nonparticipants from Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital in the PROKIDb trial from January 2019 to August 2021.
CharacteristicsTotal patients (N=320)Participants (n=152)Nonparticipants (n=168)
Age (years), median (IQR)74 (12)74 (11)75 (11)

≤69, n (%)91 (28.4)48 (31.6)44 (26.2)

70-79, n (%)147 (45.9)70 (46.1)76 (45.2)

≥80, n (%)82 (25.6)34 (22.4)48 (28.6)
Gender, n (%)

Male200 (62.5)98 (64.5)102 (60.7)
Renal function

eGFRc, mean (SD)29.2 (6.7)28.6 (6.2)29.8 (7.1)

CKD 3b, n (%) 151 (47.2)70 (46.1)81 (48.2)

CKD 4/5, n (%)155 (48.4)82 (53.9)73 (43.5)

Missing values, n (%)14 (4.4)N/Ad14 (8.3)
Educational level, n (%)

Low (<10 years): none/short <1 year93 (29.1)37 (24.3)56 (33.3)

Medium (10-12 years): skilled worker/short148 (46.3)71 (46.7)77 (45.8)

Long (>12 years): middle/long higher58 (18.1)29 (19.1)29 (17.3)

Missing values21 (6.6)15 (9.9)6 (3.6)
Labor market affiliation, n (%)

Employed43 (13.4)20 (13.2)26 (15.5)

Unemployed (retirement, early retirement)259 (80.9)119 (78.3)140 (83.3)

Missing values18 (5.6)13 (8.6)2 (1.2)
Comorbidity (Charlson index), n (%)

High (>2)110 (34.4)52 (34.2)58 (34.5)

Medium (1-2)195 (60.9)94 (61.8)101 (60.1)

Low (0)15 (4.7)6 (3.9)9 (5.4)
Health Literacy Questionnaire (HLQ) 4: social support for health, n (%)

Mean (SD)3.3 (0.55)3.2 (0.50)3.2 (0.59)

Median (IQR)3.2 (0.6)3.2 (0.6)3.2 (1)

Missing values, n (%)21 (6.6)15 (9.9)6 (3.6)
HLQ 6: ability to actively engage with health care providers

Mean (SD)3.9 (0.71)3.9(0.70)3.8 (0.71)

Median (IQR)4 (0.8)4 (0.8)4 (1)

Missing values, n (%)21 (6.6)14 (9.2)7 (4.2)
HLQ 9: understanding health information well enough to know what to do

Mean (SD)3.9 (0.67)3.9 (0.60)3.9 (0.72)

Median (IQR)4 (0.8)3.8 (0.8)4 (0.8)

Missing values, n (%)21 (6.6)14 (9.2)7 (4.2)
Self-efficacy (General Self-Efficacy [GSE] scale)

Mean (SD)29.8 (5.4)30 (5)29.6 (5.7)

Median (IQR)30 (8)30 (7)30 (8)

Missing values, n (%)23 (7.2)14 (9.2)9 (5.4)
General health, n (%)

Excellent/very good53 (16.6)29 (19.1)24 (14.3)

Good149 (46.6)63 (41.4)86 (51.2)

Fair/poor114 (35.6)60 (39.5)54 (32.1)

Missing values4 (1.3)N/A4 (2.4)

aCKD: chronic kidney disease.

bPROKID: PRO measures in Kidney care.

ceGFR: estimated glomerular filtration rate.

dN/A: not applicable.

Table 3. Reasons for nonparticipating in the PROa-based remote follow-up intervention among patients with CKDb (n=168) at Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital.
ReasonsNonparticipants (n=168), n (%)
Patient-reported reasons

Unknown (information unavailable)17 (10.1)

Did not want to participate (no further reasons reported)15 (8.9)

Could not cope with participating14 (8.3)

Preferred telephone consultations/no attendance13 (7.7)

Had enough of coping with comorbidity13 (7.7)

Visual or hearing disability11 (6.5)

Memory issues7 (4.2)

Preferred the standard follow-up program6 (3.6)

Attends another outpatient follow-up (requiring attendance)5 (3.0)

Did not wish to complete the questionnaire4 (2.4)
Clinical reported reasons

Health care professional reasonsc52 (31.0)

Otherd10 (6.0)

aPRO: patient-reported outcome.

bCKD: chronic kidney disease.

cEnded follow-up, rapid illness progression, compliance issues, comorbidity.

dDied before enrollment, departed, other study participation.


Of the 152 patients, 105 (69.1%) were allocated to the 2 PRO-based intervention arms. Of those, 8 (7.6%) patients died during follow-up and 4 (3.8%) left the study due to termination of outpatient follow-up, leaving 93 (88.6%) patients adhering to PRO-based remote follow-up (Figure 1). The mean adherence rate to the intervention was 82% (95% CI 76-87). In total, 70 (75.3%) of 93 patients had an adherence rate ≥83%, as indicated by the dotted vertical line in Figure 3. The distribution of patients according to the level of adherence rates is shown in Figure 3.

Figure 3. Distribution of patients with CKD in the PRO intervention arms according to adherence to the intervention. Threshold (dotted vertical line) indicating low adherence divided at the lower quartile (<83%) of the distribution (n=93, 88.6%). CKD: chronic kidney disease; PRO: patient-reported outcome; PROKID: PRO measures in Kidney care.

Overall, no statistical differences were found in patient and clinical factors between patients with high or low intervention adherence, albeit a tendency toward lower adherence by poor self-reported health status and lower patient activation was seen (Table S4 in Multimedia Appendix 4). Patients in the PRO-based follow-up group had a significantly lower adherence rate than patients in the PRO-based telephone follow-up group (Table 4).

The overall response rate was 99.1% (539/544). Accordingly, the item response rates were relatively high, with a mean response rate of 96.7% (14,314/14,788; Table 4). In total, 453 (84%) of the 539 patients responded electronically, and they spent a median of 9.22, IQR (6.87) minutes completing the questionnaire.

Table 4. Results of the process evaluation of the PROKIDa trial among patients with CKDb (N=101) from Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital.
Elements of process evaluationTotal patients (N=101)PROc-based follow-up (n=48)PRO-based telephone follow-up (n=53)

Adherence rate, mean % (95% CI)82 (76-87)73 (63-83)90 (84-95)

Questionnaire response rate response/total, n/N (%)539/544 (99.1)249/250 (99.6)290/294 (98.6)

Item response rate/ total, n/N (%)14,314/14,788 (96.7)6329/6696 (95.5)7985/8092 (98.7)

Time completing questionnaire (minutes), median (IQR)9.22 (6.87)9.37 (6.97)9.20 (6.78)

Reminders, n/N (%)367/367 (100)174/367 (47.4)193/367 (52.6)

Web responses, n/N (%)453/539 (84.0)199/249 (89.9)254/290 (87.6)

Assessed questionnaires, n/N (%)477/539 (88.5)215/249 (86.3)262/290 (90.3)

Course supportd, n/N (%)13/101 (12.9)9/48 (18.8)4/53 (7.5)

Involved physician per patient, median (range)3 (1-7)3 (1-6)3 (1-7)
Color coded algorithm, n/N (%)

RedeN/Af151/249 (60.6)N/A

YellowgN/A83/249 (33.3)N/A

GreenhN/A15/249 (6.0)N/A
Clinical assessment

Contact, n/N (%)417/539 (77.4)137/249 (55.0)280/290 (96.6)

No contact, n/N (%)122/539 (22.6)112/249 (45.0)10/290 (3.4)

aPROKID: PRO measures in Kidney care.

bPRO: patient-reported outcome.

cCKD: chronic kidney disease.

dThe patient asked for help from the researcher or the project nurse regarding completion of study procedures.

eHigh symptom burden or the patient wishes for contact.

fN/A: not applicable.

gSome symptom burden or the patient may need contact.

hNo symptom burden and the patient needs no contact.


The patients completed 539 questionnaires, and in 477 (88.5%) of the 539 cases, a physician assessed their responses (Table 4), leaving 11.5% (62/539) of the questionnaires unnoticed by a physician. The algorithm indicated that 234 (94%) of the 249 patients in the PRO-based follow-up group experienced some disease burden or needed clinical contact flagged by a red or yellow color code. When this was assessed, the physicians found that 137 (55%) of the 249 patients needed to be contacted and 112 (45%) did not need to be contacted. In 288 (69.1%) of all the 417 cases, the physician initiated the contact, and in 129 (30.9%) of the cases, the patient requested contact. The primary causes of contact (242/417, 58%) were attributed to clinical data, including blood samples and blood pressure. Contact based on PRO answers alone accounted for 22.1% (92/417) of the contacts, while concerns related to medication and a precautionary approach prompted the remaining contacts (83/417, 19.9%). In total, 10 (3.4%) of the 290 patients in the PRO-based telephone follow-up group were unavailable when the physician contacted them by phone. On average, 3 (range 1-7) physicians were involved in each patient pathway. A minority of 13 (12.9%) of 101 patients needed assistance to complete study procedures. Common reasons for needing assistance were measuring blood pressure and obtaining information about how to undergo blood tests or how to access the questionnaire. A study nurse or the researcher contacted the patient to address the reasons behind noncompliance and collaboratively devise a plan to ensure future adherence. The majority of those needing assistance were in the PRO-based follow-up group.

Clinical Engagement

The physicians emphasized that their perception and commitment to the use of PRO-based remote follow-up was influenced by the fact that the PROKID intervention enriched their consultations, as a physician explained:

I believe patients receive a more thorough and attentive consultation through this approach, where all symptoms are carefully examined.
[Physician 9]

However, PRO-based follow-up also seemed to limit their ability to assess the patients’ health status, as explained in the following by a physician:

I find it quite straightforward to assess individuals over the phone. However, what I miss is observing their movement from the waiting room to the consultation room; it reveals a great deal, such as being able to better evaluate characteristics for instance like skin color and psychical impairment.
[Physician 4]

On the one hand, all physicians felt that the PROKID intervention is beneficial for their ability to manage and identify patients’ symptoms and estimate their need for contact. The clinicians’ motivation to review the patients’ responses was heightened because the questionnaire included clinical data, such as information about blood pressure and weight.

The clear advantage lies in having everything compiled in this questionnaire, which significantly aids in retaining and managing all the information.
[Physician 3]

On the other hand, the physicians were also concerned that they might overlook something in the patients’ health status. This 2-sided notion was described in an interview in the following way:

You might lose some connection with the patients, the familiarity that comes with knowing them, and eventually find yourself relying more on questionnaires and blood tests. This shift can make the interaction feel less personal…Conversely, we receive something self-reported (PRO) that delves into deeply personal matters. The risk lies in potentially overlooking something, especially when the patient prefers minimal contact.
[Physician 10]

To remedy this risk of overlooking symptoms, the physicians wished to combine remote and face-to-face consultations in the future implementation of the PROKID intervention’s remote follow-up. Specifically, the assessment of patients in the PRO-based follow-up group troubled the physicians, as these patients were not automatically contacted by phone as opposed to patients in the PRO-based telephone follow-up group. This was described by a physician in the following way:

I sometimes feel concerned about patients in PRO-based follow-up, who does not visit the hospital. For instance, does this woman truly comprehend the extent of her illness? Is she receiving the necessary information? She may be unaware of what she does not know.
[Physician 5]

The thematic analysis revealed important aspects that influenced the physicians’ engagement in using PRO. These aspects represented facilitators of and barriers to the implementation of PRO-based remote follow-up in clinical practice. As shown in Textbox 1, the physicians’ engagement in the implementation of PRO-based remote follow-up was supported by several facilitating factors—for instance, PRO enabled an excellent overview of the patients’ health status and increased the possibility for visualization and clarity, as described in this quote:

I wouldn’t have inquired about this if the patient had not communicated it to me. It has completely altered the course of our conversation.
[Physician 5]
Textbox 1. Physicians’ perceptions on potential facilitators and barriers for establishing clinical engagement in the use of patient-reported outcome (PRO) in remote renal care at Aarhus University Hospital, Gødstrup Hospital, and Viborg Regional Hospital.

Facilitators of PRO use in remote care:

  • Better overview of the patient’s overall health condition
  • Ensured relevant data are available and lined up
  • A great basis for visualizing the patient provided by the combination of PRO and blood tests
  • Qualified phone consultations
  • Enhanced the patients’ ability to reflect on their symptoms
  • Enabled patient-centered focus
  • More convenient for the physician and the patient compared to usual follow-up
  • An excellent supportive tool for patients during a period of stable illness

Barriers to using PRO in remote care:

  • Decrease in physicians’ ability to sense the patients’ condition through clinical observation
  • Decrease in physicians’ knowledge of patients
  • Decrease in crucial knowledge on patient medical history
  • Induced risk of forgetting the responsibility of the patient
  • Provoked a lower personal relationship with the patient
  • Training needed to manage patients’ health status remotely
  • Specific patient flow needed to maintain clinical skills of using PRO
  • Incentive required by physicians to assess patients’ PRO responses

PRO was also perceived as enabling patient-focused care, to be convenient for both the patient and the clinic, as described by this physician:

It must be convenient for someone like him, who does not have to make the one-hour trip every third month just to let me pressure his angles a bit. We can just as easily handle that over the phone.
[Physician 2]

However, some barriers to the physicians’ engagement in implementing PRO-based follow-up worked against some of the facilitating factors. Specifically, the physicians faced challenges in accurately evaluating the patients’ health condition and determining their requirement for medical attention and intervention. This was mainly due to the lack of face-to-face interaction and the inability to use their clinical observation skills.

Principal Findings

In this process evaluation, the intervention processes of PRO-based remote follow-up (the PROKID trial) were evaluated [7]. In total, 47.5% of the eligible patients participated, and they reflected the target population. Overall, we found high percentages of the utility of the system. The dose ranged from 73% to 99%, with a mean adherence rate of 82%. Fidelity showed high usage by physicians, with 88.5% of the total questionnaires being assessed. In total, 12.9% of patients needed assistance to comply with study procedures; this need was predominantly observed in the PRO-based follow-up group. Ten physicians were interviewed about their engagement in the PROKID intervention; most of them rated PRO as being of additional value. However, some barriers, such as decreased knowledge of patients and a lack of using their clinical observational skills, were highlighted.

Overall, the percentages of utility were high in the PRO-based intervention. We believe that several reasons may explain this success. First, an internal pilot study was conducted in an outpatient clinic before trial onset. The pilot study showed that patients were willing to participate and that it was possible to deliver the intervention. We spent time observing the routines and activities in the outpatient clinic to inform our intervention. We had continuous dialogues and meetings with staff members and patients to target the intervention to existing clinical practice. Several studies have highlighted the importance of patients and health care providers knowing why PROs are used and the relevance of the questions asked [35,36]. The fact that the intervention was implemented in a real-life setting and that the users were involved in its development may have impacted patient and clinical engagement with PRO-based remote follow-up [4,37].

Another possible explanation for the high usage percentages may be the procedures used to approach and recruit patients. We included newly referred patients, and it became clear that most preferred no follow-up or remote follow-up, probably due to a feeling of being more ill when entering a hospital [38] or a general lack of experience attending outpatient follow-up. Patient-reported reasons for nonparticipation confirmed this finding, as 7% of the eligible patients declined participation due to preference for telephone consultation or no attendance. Thus, trial participation had the advantage that patients could attend remote follow-up, which was not part of the routine practice at the time. Finally, the degree of involvement of the project researcher and the clinical study nurse in the recruitment phase and during the trial may have motivated patients to participate and made them adhere to and use the intervention. In total, 12.9% of the patients needed course support, highlighting the benefit of having someone to call when problems occur. Prior findings from a parallel qualitative study, in which we interviewed 15 patients in the PROKID trial, support this finding [38].

A well-known concern in RCTs is the challenge of inducing selection bias caused by participation of a selected group failing to represent the target population [24]. We did not find any personal or clinical differences between participants and nonparticipants in our study, which adds value to the generalization of the results of the PROKID trial. Wiegel et al [12] found, in a recently published systematic review, an overall adherence to repetitive electronic PROs in populations with chronic diseases ranging between 61% and 96%, which corresponds well with the mean adherence of 82% found in our study. However, the definition of adherence is known to vary between studies [12]. We believe that having a predefined cutoff point for low adherence may help researchers and clinicians identify patients needing further assistance. Presumably, due to low power, we found no differences in personal or clinical factors between patients with low or high adherence. This might challenge clinicians seeking to identify patients most suitable for PRO-based remote follow-up. The definition and threshold for adherence correspond to the goal of using remote PRO in the outpatient clinic, as in the real-life setting, patients should be capable of completing the PRO questionnaire by themselves. We found a high response rate in our study, contrary to other previous findings [35]. We believe the 2 main reasons for this were the high use of reminders and the fact that a study nurse called the patient if questionnaire responses had not been submitted at the date of the PRO consultations. The use of reminders is known to increase the response rate [39]. Knowledge of the extent of this phenomenon is important to convey to the outpatient clinics when the intervention is widely implemented.

Another interesting finding is the number of physicians involved in each patient pathway. Preferably, patients should have 1 contact physician during follow-up, but we saw that each patient had contact with 3 (range 1-7) physicians during the 18 months’ trial. Evidence from a meta-analysis suggests that for the safety and continuity of care, it is essential to have as few different health care professionals as possible [40]. A qualitative study integral to the PROKID intervention concluded that barriers to patient engagement in PRO-based remote follow-up are unfamiliarity with the physician and remote follow-up challenging the patient-physician relationship [38]. Furthermore, the involvement of multiple physicians in reviewing each patient over the 18 months posed a challenge, leading to diminishing familiarity with the medical history of the patients. Remote monitoring is increasingly substituting traditional hospital visits [41]. A recent study argued that telephone follow-up is as safe as visits to the outpatient clinic, especially if the patient is acquainted with the physician [42]. Thus, it seems crucial to incorporate individual patient-physician consultations to the extent possible.

A prior study of field observations during the PRO consultations showed that the extent to which the physicians incorporated the patient responses into the consultations reached almost 100% [38]. However, data from this process evaluation study found that the physicians merely assessed 88.5% of the questionnaire responses received. This might indicate observer bias in the qualitative study [38], showing a falsely high result, although it might also indicate a false lower result in our study because of recall bias, as it seemed as if the physicians actually opened and assessed the patients’ PRO responses but forgot to approve and document the responses in the AmbuFlex database. However, it is a well-known challenge to get clinicians to assess and approve patients’ PRO responses [19,43]. The qualitative analyses showed that the physicians found PRO-based remote follow-up to enrich the consultations, but it also induced a risk of failing to notice important symptoms. Prior research has shown that engagement decreases when activities or tasks are perceived as wasteful of resources or creating more harm than benefit [44]. Therefore, enhancing the physicians’ satisfaction and perception of the usability of implementing PROs in remote care seems crucial. The physicians emphasized that they needed an incentive to open the patients’ PRO responses. During the interviews, it became clear that the patients’ blood pressure and weight were the physicians’ primary objects of interest. This result was supported in a qualitative study on patient perspectives [38] and has also been found to be an issue in other studies [22,45]. Hence, an important factor driving the assessment of patients’ PRO responses was the incorporation of the self-reported clinical outcomes (blood pressure and weight) in the questionnaire.

The color-coded algorithm showed that most of the patients needed contact with a physician or were burdened by disease-specific symptoms. Thus, the proportion of patients contacted by physicians was relatively high, which may indicate a high claim for clinical attention in this patient group. However, it may also indicate a sensitive algorithm [4]. The clinical experts aimed to develop an algorithm with high sensitivity since a low level of false-negative cases was more important than a high level of false-positive cases. Thus, the physicians’ ability to assess whether to contact patients was pivotal for the follow-up process. Further studies that consider the results of the green-yellow-red color-coded algorithm will need to be undertaken.

Strengths and Limitations

Process evaluation was considered integral to the PROKID trial, and the results of this process evaluation were analyzed prior to knowledge of the trial outcomes. Thereby, the risk of bias was reduced [46]. This study enabled us to perform a more detailed examination of the process of recruitment and intervention adherence to inform the interpretation of trial results and generalizability [47]. A main strength of this study is that it included patients from a well-defined population covering all centers following patients with CKD in Central Denmark Region. Quantitative and qualitative methods complemented one another, and a combination of self-reported, qualitative data and register-based data was used to link different data sources. Another key strength of this study was the use of Steckler and Linnan’s [26] process evaluation framework, which allowed us to structure the process evaluation appropriately. Observations and interviews had the potential to influence the involved physicians’ and patients’ behavior and commitment. However, we believe that the timing of the evaluation in the finishing part of the trial limited this effect. The evaluator responsible was also the principal investigator in the PROKID trial, which provided insight into and understanding of the context, as an outsider could not have added to the findings.

This study also has some limitations. Initial development and use of a logic model may have better illustrated the underpinning theory [26]. The absence of predefined core questions was also a notable limitation. In the qualitative data analyses, the researcher’s familiarity with the participating departments may have induced a certain risk of bias. This may have led to a more positive attitude toward PRO-based remote follow-up among the physicians. However, it may also have made them confident that critical comments would be received constructively.


The PRO-based PROKID intervention was generally well received by patients and physicians, as we found high percentages of utility across all process evaluation components. Our findings highlight that remote PRO-based follow-up is feasible in a clinical setting and may be a relevant and valuable tool when remotely monitoring patients with CKD. We found high adherence and response rates among the patients, and the physicians assessed the majority of patient responses. However, it seems important to have a key figure who helps patients in need of assistance to comply with PRO-based remote follow-up. The physicians emphasized both enabling factors, such as improved overview and patient-focused care, and barriers, particularly the complexities of evaluating health conditions without direct interaction. Suggestions for future implementations included a combination of remote and face-to-face consultations to address this concern.


BEG, AdT, and LMVS conceived the study. BEG collected data. BEG and AKK performed the analyses. All authors contributed to the data interpretation and critical revision of the manuscript. All authors have read and approved the final manuscript and stand by the integrity of the entire work. This paper was written without the use of artificial intelligence technology. The Karen Elise Jensen Foundation and the AmbuFlex – Centre for Patient-reported Outcomes kindly funded this study.

Data Availability

The data sets generated and analyzed during this study are available in anonymous form from the corresponding author on request or by contacting the AmbuFlex – Centre for Patient-reported Outcomes.

Conflicts of Interest

BEG, RG, NHH, AKK, and LMVS are employees of AmbuFlex – Centre for Patient-reported Outcomes. However, this is a public-funded department with no financial benefit.

Multimedia Appendix 1

Investigated patient and clinical factors.

DOCX File , 15 KB

Multimedia Appendix 2

Interview guide.

DOCX File , 24 KB

Multimedia Appendix 3

Preliminary coding scheme aiming to explore the physicians' perceptions and experiences using PRO-based remote follow-up. PRO: patient-reported outcome.

PDF File (Adobe PDF File), 565 KB

Multimedia Appendix 4

Characteristics of patients adhering to PRO-based follow-up according to demographic, disease-related, and individual determinants divided into high (≥83%) and low (<83%) adherence (n=93). PRO: patient-reported outcome.

PDF File (Adobe PDF File), 441 KB

Multimedia Appendix 5


PDF File (Adobe PDF File), 7484 KB

  1. Healthcare in Denmark - an Overview. Healthcare Denmark. 2017. URL: [accessed 2024-04-09]
  2. Ladin K, Porteny T, Perugini JM, Gonzales KM, Aufort KE, Levine SK, et al. Perceptions of Telehealth vs In-Person Visits Among Older Adults With Advanced Kidney Disease, Care Partners, and Clinicians. JAMA Netw Open. Dec 01, 2021;4(12):e2137193. [FREE Full text] [CrossRef] [Medline]
  3. World Health Organization. WHO-ITU global standard for accessibility of telehealth services. Geneva. World Health Organization; 2022.
  4. Hjollund NHI, Larsen LP, de Thurah AL, Grove BE, Skuladottir H, Linnet H, et al. Patient-reported outcome (PRO) measurements in chronic and malignant diseases: ten years' experience with PRO-algorithm-based patient-clinician interaction (telePRO) in AmbuFlex. Qual Life Res. Apr 2023;32(4):1053-1067. [FREE Full text] [CrossRef] [Medline]
  5. Schougaard LMV, Larsen LP, Jessen A, Sidenius P, Dorflinger L, de Thurah A, et al. AmbuFlex: tele-patient-reported outcomes (telePRO) as the basis for follow-up in chronic and malignant diseases. Qual Life Res. Mar 2016;25(3):525-534. [FREE Full text] [CrossRef] [Medline]
  6. Food and Drug Administration. Guidance for industry. Patient-reported oucome measures: use in medicinal product development to support labeling claims. U.S. Food and Drug Administration. Dec 2009. URL: [accessed 2024-04-09]
  7. Grove BE, Ivarsen P, de Thurah A, Schougaard LM, Kyte D, Hjøllund NH. Remote follow-up using patient-reported outcome measures in patients with chronic kidney disease: the PROKID study - study protocol for a non-inferiority pragmatic randomised controlled trial. BMC Health Serv Res. Sep 04, 2019;19(1):631. [FREE Full text] [CrossRef] [Medline]
  8. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions. Am J Prev Med. Nov 2016;51(5):833-842. [CrossRef] [Medline]
  9. O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. Sep 15, 2016;16(1):120. [FREE Full text] [CrossRef] [Medline]
  10. Schougaard LMV, de Thurah A, Christensen J, Lomborg K, Maindal HT, Mejdahl CT, et al. Sociodemographic, personal, and disease-related determinants of referral to patient-reported outcome-based follow-up of remote outpatients: a prospective cohort study. Qual Life Res. May 2020;29(5):1335-1347. [FREE Full text] [CrossRef] [Medline]
  11. Schougaard LMV, Knudsen LR, Grove BE, Vestergaard JM, Hjollund NH, Hauge E, et al. Socioeconomic, Disease-Related, and Personal Factors Associated With Participation in Remote Follow-Up in Rheumatoid Arthritis: A Cross-Sectional Study. Arthritis Care Res (Hoboken). Oct 2023;75(10):2044-2053. [CrossRef] [Medline]
  12. Wiegel J, Seppen B, van der Leeden M, van der Esch M, de Vries R, Bos W. Adherence to Telemonitoring by Electronic Patient-Reported Outcome Measures in Patients with Chronic Diseases: A Systematic Review. Int J Environ Res Public Health. Sep 27, 2021;18(19):10161. [FREE Full text] [CrossRef] [Medline]
  13. National Kidney Foundation. K/DOQI clinical practice guidelines for chronic kidney disease: evaluation, classification, and stratification. Am J Kidney Dis. Feb 2002;39(2 Suppl 1):S1-266. [Medline]
  14. Grove BE, Ivarsen P, de Thurah A, Schougaard LM, Kyte D, Hjøllund NH. Remote follow-up using patient-reported outcome measures in patients with chronic kidney disease: the PROKID study - study protocol for a non-inferiority pragmatic randomised controlled trial. BMC Health Serv Res. Sep 04, 2019;19(1):631. [FREE Full text] [CrossRef] [Medline]
  15. Appel CW, Pedersen SC, Nielsen AS, Larsen BF. Telemedicine based on patient-reported outcomes in management of patients with inflammatory bowel disease in a real-life setting - a before and after cohort study. Scand J Gastroenterol. Jul 2022;57(7):825-831. [CrossRef] [Medline]
  16. Schougaard LMV, Mejdahl CT, Christensen J, Lomborg K, Maindal HT, de Thurah A, et al. Patient-initiated versus fixed-interval patient-reported outcome-based follow-up in outpatients with epilepsy: a pragmatic randomized controlled trial. J Patient Rep Outcomes. Sep 13, 2019;3(1):61. [FREE Full text] [CrossRef] [Medline]
  17. Ishaque S, Karnon J, Chen G, Nair R, Salter AB. A systematic review of randomised controlled trials evaluating the use of patient-reported outcome measures (PROMs). Qual Life Res. Mar 2019;28(3):567-592. [CrossRef] [Medline]
  18. Kotronoulas G, Kearney N, Maguire R, Harrow A, Di Domenico D, Croy S, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. May 10, 2014;32(14):1480-1501. [FREE Full text] [CrossRef] [Medline]
  19. Boyce MB, Browne JP, Greenhalgh J. The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. BMJ Qual Saf. Jun 2014;23(6):508-518. [FREE Full text] [CrossRef] [Medline]
  20. Detmar SB, Muller MJ, Schornagel JH, Wever LDV, Aaronson NK. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA. Dec 18, 2002;288(23):3027-3034. [CrossRef] [Medline]
  21. Greenhalgh J, Dalkin S, Gooding K, Gibbons E, Wright J, Meads D, Black N, Valderas JM, Pawson R.. Functionality and feedback: a realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. NIHR Journals Library. 2017.:chapt 8. [CrossRef] [Medline]
  22. Mejdahl CT, Schougaard LMV, Hjollund NH, Riiskjær E, Thorne S, Lomborg K. PRO-based follow-up as a means of self-management support - an interpretive description of the patient perspective. J Patient Rep Outcomes. 2017;2:38. [FREE Full text] [CrossRef] [Medline]
  23. Girgis A, Durcinoska I, Arnold A, Descallar J, Kaadan N, Koh E, et al. Web-Based Patient-Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care): Multicenter Pragmatic Nonrandomized Trial. J Med Internet Res. Oct 29, 2020;22(10):e19685. [FREE Full text] [CrossRef] [Medline]
  24. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Cochrane Bias Methods Group, et al. Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. Oct 18, 2011;343:d5928. [FREE Full text] [CrossRef] [Medline]
  25. Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence. J Med Internet Res. Dec 06, 2017;19(12):e402. [FREE Full text] [CrossRef] [Medline]
  26. Steckler A, Linnan L. Process evaluation for public health interventions and research. San Francisco, CA. Jossey-Bass; 2002;1-23.
  27. Nielsen AS, Kidholm K, Kayser L. Patients' reasons for non-use of digital patient-reported outcome concepts: A scoping review. Health Informatics J. Dec 2020;26(4):2811-2833. [FREE Full text] [CrossRef] [Medline]
  28. Grove BE, Schougaard LMV, Ivarsen PR, Kyte D, Hjollund NH, de Thurah A. Patient-reported outcome measures for clinical decision-making in outpatient follow-up: validity and reliability of a renal disease questionnaire. J Patient Rep Outcomes. Oct 16, 2021;5(1):107. [FREE Full text] [CrossRef] [Medline]
  29. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. Dec 31, 2011;13(4):e126. [FREE Full text] [CrossRef] [Medline]
  30. Hjollund NHI, Larsen LP, Biering K, Johnsen SP, Riiskjær E, Schougaard LM. Use of Patient-Reported Outcome (PRO) Measures at Group and Patient Levels: Experiences From the Generic Integrated PRO System, WestChronic. Interact J Med Res. Feb 11, 2014;3(1):e5. [FREE Full text] [CrossRef] [Medline]
  31. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. Apr 2009;42(2):377-381. [FREE Full text] [CrossRef] [Medline]
  32. Pannick S, Sevdalis N, Athanasiou T. Beyond clinical engagement: a pragmatic model for quality improvement interventions, aligning clinical and managerial priorities. BMJ Qual Saf. Sep 2016;25(9):716-725. [FREE Full text] [CrossRef] [Medline]
  33. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. Jan 2006;3(2):77-101. [FREE Full text] [CrossRef]
  34. Kirk MD, Pires SM, Black RE, Caipo M, Crump JA, Devleesschauwer B, et al. Correction: World Health Organization Estimates of the Global and Regional Disease Burden of 22 Foodborne Bacterial, Protozoal, and Viral Diseases, 2010: A Data Synthesis. PLoS Med. Dec 2015;12(12):e1001940. [FREE Full text] [CrossRef] [Medline]
  35. Foster A, Croot L, Brazier J, Harris J, O'Cathain A. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes. Dec 2018;2:46. [FREE Full text] [CrossRef] [Medline]
  36. Stover AM, Haverman L, van Oers HA, Greenhalgh J, Potter CM, ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res. Nov 2021;30(11):3015-3033. [FREE Full text] [CrossRef] [Medline]
  37. Porter I, Gonçalves-Bradley D, Ricci-Cabello I, Gibbons C, Gangannagaripalli J, Fitzpatrick R, et al. Framework and guidance for implementing patient-reported outcomes in clinical practice: evidence, challenges and opportunities. J Comp Eff Res. Aug 2016;5(5):507-519. [FREE Full text] [CrossRef] [Medline]
  38. Grove BE, Valen Schougaard LM, Ivarsen P, Hjollund NH, de Thurah A, Mejdahl CT. Remote follow-up based on patient-reported outcomes in patients with chronic kidney disease: A qualitative study of patient perspectives. PLoS One. 2023;18(2):e0281393. [FREE Full text] [CrossRef] [Medline]
  39. Nielsen LK, King M, Möller S, Jarden M, Andersen CL, Frederiksen H, et al. Strategies to improve patient-reported outcome completion rates in longitudinal studies. Qual Life Res. Feb 2020;29(2):335-346. [FREE Full text] [CrossRef] [Medline]
  40. Waibel S, Henao D, Aller M, Vargas I, Vázquez ML. What do we know about patients' perceptions of continuity of care? A meta-synthesis of qualitative studies. Int J Qual Health Care. Feb 2012;24(1):39-48. [CrossRef] [Medline]
  41. Bokolo AJ. Application of telemedicine and eHealth technology for clinical services in response to COVID‑19 pandemic. Health Technol (Berl). 2021;11(2):359-366. [FREE Full text] [CrossRef] [Medline]
  42. Ackerman SL, Gleason N, Shipman SA. Comparing Patients' Experiences with Electronic and Traditional Consultation: Results from a Multisite Survey. J Gen Intern Med. Apr 2020;35(4):1135-1142. [FREE Full text] [CrossRef] [Medline]
  43. Mejdahl CT, Schougaard LMV, Hjollund NH, Riiskjær E, Lomborg K. Patient-reported outcome measures in the interaction between patient and clinician - a multi-perspective qualitative study. J Patient Rep Outcomes. Jan 09, 2020;4(1):3. [FREE Full text] [CrossRef] [Medline]
  44. Davies HTO, Powell AE, Rushmer R. Healthcare professionals' views on clinician engagement in quality improvement: a literature review. London, UK. The Health Foundation; 2007.
  45. Greenhalgh J, Abhyankar P, McCluskey S, Takeuchi E, Velikova G. How do doctors refer to patient-reported outcome measures (PROMS) in oncology consultations? Qual Life Res. Jun 2013;22(5):939-950. [CrossRef] [Medline]
  46. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ. Feb 18, 2006;332(7538):413-416. [FREE Full text] [CrossRef] [Medline]
  47. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. Jan 12, 2013;14:15. [FREE Full text] [CrossRef] [Medline]

CKD: chronic kidney disease
eGFR: estimated glomerular filtration rate
PRO: patient-reported outcome
PROKID: PRO measures in Kidney care
RCT: randomized controlled trial
REDCap: Research Electronic Data Capture

Edited by A Mavragani; submitted 22.04.23; peer-reviewed by S Avery, KC Wong; comments to author 08.12.23; revised version received 02.02.24; accepted 15.03.24; published 24.04.24.


©Birgith Engelst Grove, Annette de Thurah, Per Ivarsen, Ann Katrine Kvisgaard, Niels Henrik Hjollund, Regine Grytnes, Liv Marit Valen Schougaard. Originally published in JMIR Formative Research (, 24.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.