Published on in Vol 6, No 7 (2022): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37291, first published .
Optimization of the Chronic Kidney Disease–Peritoneal Dialysis App to Improve Care for Patients on Peritoneal Dialysis in Northeast Thailand: User-Centered Design Study

Optimization of the Chronic Kidney Disease–Peritoneal Dialysis App to Improve Care for Patients on Peritoneal Dialysis in Northeast Thailand: User-Centered Design Study

Optimization of the Chronic Kidney Disease–Peritoneal Dialysis App to Improve Care for Patients on Peritoneal Dialysis in Northeast Thailand: User-Centered Design Study

Original Paper

1Center of Excellence in Kidney Diseases, Srinagarind Hospital, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand

2Ophthalmic Consultants of Boston, Harvard Medical School, Boston, MA, United States

3Division of Nephrology, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand

4Department of Computer Engineering, Faculty of Engineering, Khon Kaen University, Khon Kaen, Thailand

5Data Management and Statistical Analysis Center, Faculty of Public Health, Khon Kaen University, Khon Kaen, Thailand

6Kidney Unit, Srinagarind Hospital, Faculty of Medicine, Khon Kaen University, Khon Kaen, Thailand

7Massachusetts General Hospital Center for Global Health, Harvard Medical School, Boston, MA, United States

Corresponding Author:

Katharine E Morley, MPH, MD

Massachusetts General Hospital Center for Global Health

Harvard Medical School

125 Nashua St. Suite 722

Boston, MA, 02114

United States

Phone: 1 617 726 2000

Email: kemorley@mgh.harvard.edu


Background: The prevalence of peritoneal dialysis (PD) in Thailand is increasing rapidly in part because of Thailand’s Peritoneal Dialysis First policy. PD is a home-based renal replacement therapy in which patients with chronic kidney disease perform up to 4 exchanges of dialysate fluid per day in the peritoneal cavity. Overhydration is one of the most common complications in patients on PD and is associated with increased morbidity and mortality. To monitor hydration status, patients collect hydration metrics, including body weight, blood pressure, urine output, and ultrafiltration volume, from each dialysis cycle and enter this information into a PD logbook. This information is reviewed bimonthly at PD clinic appointments. The chronic kidney disease-PD (CKD-PD) app with near-field communication (NFC) and optical character recognition (OCR) was developed to automate hydration metric collection. The information was displayed in the app for self-monitoring and uploaded to a database for real-time monitoring by the PD clinic staff. Early detection and treatment of overhydration could potentially reduce the morbidity and mortality related to overhydration.

Objective: This study aims to identify usability issues and technology adoption barriers for the CKD-PD app with NFC and OCR and a monitoring system and to use this information to make rapid cycle improvements.

Methods: A multidisciplinary team of nephrologists, PD clinic nurses, computer programmers, and engineers trained and observed 2 groups of 5 participants in the use of the CKD-PD app with NFC and OCR and a monitoring system. The participants were observed using technology in their homes in 3 phases. The data collected included the Unified Theory of Acceptance and Use of Technology questionnaire, think-aloud observation, user ratings, completion of hydration metrics, and upload of hydration metrics to the central database. These results were used by the team between phases to improve the functionality and usefulness of the app.

Results: The CKD-PD app with NFC and OCR and a monitoring system underwent 3 rapid improvement cycles. Issues were identified regarding the usability of the NFC and OCR data collection, app stability, user interface, hydration metric calculation, and display. NFC and OCR improved hydration metric capture; however, issues remained with their usability. App stability and user interface issues were corrected, and hydration metrics were successfully uploaded by the end of phase 3. Participants’ scores on technology adoption decreased but were still high, and there was enthusiasm for the self-monitoring and clinical communication features.

Conclusions: Our rapid cycle process improvement methodology identified and resolved key barriers and usability issues for the CKD-PD app with NFC and OCR and a monitoring system. We believe that this methodology can be accomplished with limited training in data collection, statistical analysis, and funding.

JMIR Form Res 2022;6(7):e37291

doi:10.2196/37291

Keywords



Chronic Kidney Disease and Peritoneal Dialysis in Northeast Thailand

Chronic kidney disease (CKD) is a major health problem in Thailand because of its high prevalence, cost of treatment, significant morbidity and mortality, and substantial impact on the quality of life of patients and their families. From 2017 to 2018, the Chronic Kidney Disease Prevention in the Northeast of Thailand (CKDNET) project found that the prevalence of CKD was 27% in the rural provinces of northeast Thailand [1], primarily because of rising rates of diabetes, hypertension, and other primary renal diseases [2]. Despite this increased need, there is only 1 nephrologist for every 593,000 population compared with every 44,000 population in Bangkok [3]. In 2008, the Thai government adopted the Peritoneal DialysisFirst policy for renal replacement therapy under its universal health care coverage scheme, increasing access in low-resource settings, with approximately 21% of patients on peritoneal dialysis (PD) in northeast Thailand [4]. Patients on PD manage their PD at home manually or by using a PD cycler to deliver dialysate fluid through a catheter placed in the peritoneal cavity, where the fluid remains for several hours. They enter hydration metric data, including body weight, blood pressure, urine output, and ultrafiltration volume for each cycle, in handwritten notebooks for review by a nephrologist at bimonthly clinic appointments.

User Design and Evaluation of Successful Adoption of the CKD-PD App

In 2018, the Data Management and Statistical Analysis Center, Faculty of Public Health, Khon Kaen University, developed the CKD-PD mobile app to help nephrologists and patients on PD manage fluid status to prevent overhydration. It is a common complication in patients on PD. Overhydration increases morbidity because of PD-related peritonitis, stroke, congestive heart failure, major adverse cardiac events, and mortality [4-9]. The CKD-PD app has been of interest not only to nephrologists but also to the Thai Health Security Office as an intervention for improving the care of patients on PD. Early treatment of overhydration can decrease these related complications, thereby reducing hospitalization and health care expenses [10].

Many apps fail because of a lack of evaluation and removal of barriers to user adoption, fidelity of the technology, and design of the health care delivery system in which the app will be deployed [11,12]. Achieving rapid design and deployment of digital health interventions are 2 challenges facing the successful adoption and implementation of mobile health (mHealth) technologies [13]. This is especially true in low- and middle-income countries where mHealth interventions have been touted as solutions to a wide variety of health care challenges [14-16]; however, little is known about how they perform [17-19]. Before studying the effectiveness of the CKD-PD in managing overhydration in a real-world setting, a user design study was conducted using rapid cycle process improvement methods.

The objectives of this study were to (1) optimize the design and usability of the CKD-PD app and test the app in the context in which it is deployed using rapid cycle process improvement methods; (2) evaluate automatic data entry features using near-field communication (NFC) and optical character recognition (OCR) technology; and (3) identify and address the practical challenges that influence the successful adoption of the CKD-PD app and remote monitoring system in a real-world, low-resource setting.


Study Population

The study was conducted at Srinagarind Hospital, Khon Kaen University, between November 1, 2020, and April 30, 2021. Patients on PD for >3 months without a change in their PD prescription were invited to participate if they met the following inclusion criteria: aged ≥18 years, having access to a smartphone capable of running the CKD-PD app, and willing to allow research staff to observe their use of the CKD-PD app with NFC and OCR features and monitoring equipment in their home. Vulnerable populations specified as pregnant women, children, prisoners, individuals who were institutionalized, or those unable to participate in home data collection were excluded. Informed consent was obtained by the trained research staff. A total of 10 participants were enrolled and divided into group 1 (participants 1-5) and group 2 (participants 6-10). Baseline demographic data, including age, sex, education level, time on PD, and whether they used continuous ambulatory PD (CAPD) or automated PD (APD), were collected at enrollment. Participants were given prepaid cards to cover the cost of study related to internet or cellular data expenses.

CKD-PD App and Its Features

The CKD-PD app is available for free download in Android and iOS formats and is designed for daily hydration metric collection for patients on PD (Multimedia Appendix 1). Users can enter hydration metrics using manual input or voice recognition. The CKD-PD app graphically displays a patient’s hydration metrics over time and uploads them to the CKDNET database stored in the Thai Care Cloud data repository [20], which is accessible to the PD clinic staff. Nephrologists can set individual hydration parameters for a patient so that they can self-monitor their hydration status. Alerts can be set to notify patients on PD and PD clinic staff when there are actionable hydration metric abnormalities, triggering a prompt review by a nephrologist and allowing the early detection and treatment of overhydration. Another key feature of the CKD-PD app is a direct link using LINE, a social messaging app widely used in Thailand, between the patient and PD clinic, facilitating communication about symptoms and changes in PD management.

A new prototype using NFC and OCR to automate the entry of hydration metric data into the CKD-PD app from measurement devices was developed. NFC uses radio frequency communication and wirelessly transmits data to an NFC-enabled device when it is placed within 4 cm. This offers a simple, low-cost solution that does not require an external power source or pairing, similar to Bluetooth. It is commonly used for mobile payments ad transit cards and, more recently, with subdermal glucose sensors [21-25]. NFC radio frequency tags with unique ID numbers can be added to medical devices such as scales. Users tap a card with an NFC receiver tag on the NFC-equipped scale and then tap it on the NFC card reader, and the data are transferred to the CKD-PD (Multimedia Appendix 2). The CKD-PD app is also equipped with OCR technology, which uses a smartphone camera to capture the digital output from the blood pressure machine and store it in the CKD-PD app.

Study Design

Overview

There were 3 user design phases separated by 2 improvement cycles. During the improvement cycle, the app improvement team analyzed the results from the previous phase and revised the CKD-PD app using the NFC, OCR, and monitoring system (Table 1). The app improvement team included study nephrologists, computer engineers and app developers, and the Data Management and Statistical Analysis Center Thai Care Cloud database team. The app improvement team met as needed to define technology and user design issues, develop solutions, and test modifications.

Table 1. Overview and timeline of research activities.
Research activityPhase 1 (group 1)ICa 1Phase 2 (group 2)IC 2Phase 3 (groups 1 and 2)

WeekWeekWeekWeekWeek

01-2345-889-10111212-1617-1819-20
In-clinic training and observation









UTAUTb survey





Home observation







Completion of hydration metrics






Contact with PDc clinic






Validation of hydration metrics








Observation logbook










aIC: improvement cycle.

bUTAUT: Unified Theory of Acceptance and Use of Technology.

cPD: peritoneal dialysis.

Phase 1
Overview

Group 1 participants received training on the use of the CKD-PD app with the NFC and OCR and monitoring system during week 0. In-home observations were conducted once during weeks 1 to 4, followed by research activities 2 to 3 (see the following sections) during weeks 1 and 2. During weeks 3 and 4, the participants used the CKD-PD app with the NFC and OCR system at their homes and completed research activities 4 to 6 (see in the following sections).

Improvement Cycle 1

After completion of phase 1, the results were summarized and presented with the completion of hydration metrics and clinical contact results to the app improvement team. The app improvement team analyzed the results and modified the components of the system, including measurement devices, NFC, OCR, app design, and programming. The user processes were also adjusted.

Phase 2
Overview

Group 2 participants received training in the use of the CKD-PD app with the NFC and OCR and monitoring system during week 8. In-home observations were conducted during weeks 9 to 12 using the modified CKD-PD app with the NFC and OCR and monitoring system. Research activities 2 to 3 were conducted during weeks 9 and 10, followed by research activities 4 to 6 in weeks 11 and 12.

Improvement Cycle 2

After the completion of phase 2, the app improvement team reviewed the new results from participant observations, hydration metric completion, and clinic contacts. Additional issues were identified, and new solutions were developed and tested. Phase 3 was launched after the completion of the improvement cycle 2 modifications.

Phase 3
Overview

Participants from groups 1 and 2 performed research activities 2 to 6 again during weeks 17 to 20 using the modified CKD-PD app and monitoring system modified without NFC and OCR. This allowed for the evaluation of the usability and functionality of the CKD-PD app without NFC and OCR. In phase 3, participants were asked to rate the entry of body weight and dialysate volumes with NFC again based on their earlier experience for comparison with phases 1 and 2. In addition, research activity 7 (see the following sections) was performed to compare using the usual practice of entering their hydration metrics by hand into a logbook with the CKD-PD app.

Final Improvements

After completion of phase 3, the app improvement team made final improvements based on the results of the phase 3 home observation and hydration metric collection using the CKD-PD app and monitoring system without NFC and OCR.

Description of Study Activities

The details of the study activities are described in the following sections.

In-Clinic Training

Participants were trained to use the CKD-PD app with NFC and OCR and home monitoring equipment in the PD clinic by research staff and PD nurses, starting with the CKD-PD app downloaded onto the participant’s mobile phone and account registration. Research assistants demonstrated the use of the CKD-PD app, followed by instructions on how to use the NFC and home monitoring equipment, how the uploaded hydration metrics can be monitored by the PD clinic, and how they can self-monitor at home on the app. A set of NFC hydration metric data collection equipment (body weight scale, NFC card, NFC card reader and connectors, and blood pressure machine) was prepared for each participant to use at home. They were trained to set up the equipment using a teaching video (Multimedia Appendix 2) followed by hands-on practice with a research assistant. During the study period, the participants were instructed to use the NFC data collection system and equipment, in addition to the standard method of recording their hydration metrics in a logbook.

Unified Theory of Acceptance and Use of Technology Survey

A structured interview questionnaire based on the Unified Theory of Acceptance and Use of Technology (UTAUT) model was completed at the beginning and end of each phase to collect perceptions of user technology acceptance and usability [26]. The questionnaire has 6 domains, each with 3 questions representing different factors affecting technology use and adoption. It was translated into the Thai language, and each question was scored on a 5-point Likert scale, with 1=strongly disagree and 5=strongly agree. The final score for each domain is the sum of the scores of the 3 questions for that domain and ranges from 3=strongly disagree to 15=strongly agree. The questionnaire was explained by the research staff and self-administered by the participants (Multimedia Appendix 3).

Home Observation

Research staff conducted the home observation of participants using the CKD-PD app with NFC and OCR using an observation guide (Multimedia Appendix 4), using the think-aloud method [27]. They asked the participants what they liked and disliked about each task and feature. Participants were asked to rate each feature as follows: 1=good, 2=neutral, and 3=not good. This information was recorded by research assistants using handwritten notes and summarized for use by the app improvement team.

Completion of Hydration Metrics

During the 2-week home use period, the number of times the participant successfully uploaded each of the required hydration metric values (body weight, blood pressure, and use of the CKD-PD app) was recorded. Participants using CAPD required 4 dialysate exchange volumes per day, whereas those using APD only required 1 each day; thus, the number of required hydration metrics varied among participants. Successful completion of hydration metrics included the entry of hydration metrics into the app and accurate upload to the CKDNET database.

Contact With PD Clinic

The number of times the participants contacted the PD clinic during all 3 phases was collected, along with the reason for contact.

Validation of Hydration Metrics

The hydration metrics collected by each participant were validated by comparing the values uploaded to the CKDNET database with the results recorded in their logbooks during the study period. Participants sent screenshots from the app, or the research staff reviewed the data on the participant’s smartphone to confirm the correct data entry in the CKD-PD app. Successful validation of hydration metrics was defined by the completion of data entry by the participant and agreement between values recorded in their logbook and the uploaded results.

Observation of Logbook Use

Participants were observed while entering hydration metrics in their logbooks. They were asked what they liked and disliked about each task and feature and rated each feature as 1=good, 2=neutral, and 3=not good.

Ethics Approval

This study was approved by the Ethics Committee for Human Research, Faculty of Medicine, Khon Kaen University, Thailand (project number HE621494), and the Mass General Brigham institutional review board (protocol number 2019P002648). All participants provided written informed consent in the Thai language.


Participant Characteristics

Phase 1 participants characteristics had a mean age of 46 years (SD 10.3) and a mean time on PD of 5.5 years (SD 3.8); were female (2/5 40%); used the PD method of CAPD (3/5, 60%) or APD (2/5, 40%); and had educational backgrounds of high school (2/5, 40%), bachelor’s degree (1/5, 20%), and postbachelor’s degree (2/5, 40%).

Phase 2 participant characteristics had a mean age of 48 years (SD 15.7) and a mean time on PD of 1.3 years (SD 2.5); were female (3/5, 60%); used the PD method of CAPD (1/5, 20%) or APD (4/5, 80%); and had educational backgrounds of high school (2/5, 40%), bachelor’s degree (2/5, 40%), and postbachelor’s degree (1/5, 20%). One of the group 2 participants did not complete phase 3 because of sudden death.

In comparison, the characteristics of patients on PD at Srinagarind Hospital were as follows: mean age 49 years (SD 13); mean time on PD 3 years (SD 29.7); female (33/74, 45%); PD method CAPD (42/74, 57%) or APD (32/74, 43%); and educational background of primary school (37/74, 50%), high school (12/74, 16%), bachelor’s degree (23/74, 31%), and postbachelor’s degree (2/74, 3%).

UTAUT Survey

The UTAUT survey scores for each participant are presented in Table 2 as the total score (sum of all 6 domains) and the scores for each domain. The difference in the total score between the beginning of phase 1 or phase 2 and the end of phase 3 is also presented. Detailed results, including the scores at the beginning and end of all phases, are available in Multimedia Appendix 5.

Table 2. Difference in Unified Theory of Acceptance and Use of Technology total scoresa by participant and domain between the beginning of phase 1 or 2 and end of phase 3 (detailed results in Multimedia Appendix 5).
Participant numberb12345678910Values, mean
Total score

Phase 1 or 28963868187796675717777

Phase 3747680747772N/Ac80797276

Difference−1513−6−7−10−7N/A58−5−2.7
Difference by domains

Performance expectancy−2311−1−1N/A1100.33

Effort expectancy−241−11−1N/A0010.33

Social influence−3310−2−1N/A−11−1−0.67

Voluntariness−3−3−5−2−3−3N/A23−4−2

Intention to use−23−4−4−60N/A230−0.9

Facilitating conditions−330−11−1N/A10−1−0.1

aScores for individual questions from 1 (strongly disagree) to 5 (strongly agree). The domain scores ranged from 5 to 15. The total score for all the domains ranged from 30 to 90.

bParticipants 1 to 5: phase 1 and phase 3; participants 6 to 10: phase 2 and phase 3.

cN/A: not applicable; participant 7 expired before phase 3.

Of the 9 participants who completed phase 1 or 2 and phase 3, 6 (67%) individuals had a decrease in the total UTAUT score, ranging from −5 to −15 points. There were 33% (3/9) of individuals who had an increase in the total UTAUT score, ranging from 5 to 13 points. The mean score at the beginning of phases 1 or 2 was 77 (SD 8.8), with a range of 63 to 89. The mean score at the end of phase 3 was 76 (SD 3.2), with a range of 72 to 80. In group 1, the domains of voluntariness (−3) and intention to use (−2.6) showed the largest decrease in mean difference between the beginning of phase 1 and the end of phase 3, with all participants reporting a negative score in voluntariness and 80% (4/5) of participants reporting a negative score in intention to use. In group 2, there were small decreases in voluntariness (−0.5), social influence (−0.5), and facilitating conditions (−0.25) and an increase in intention to use (1.25) between phases 2 and 3 (Multimedia Appendix 5).

Participant Observation

The home observation results for each participant using the CKD-PD app with the NFC and OCR and monitoring system during the home observations are summarized by phase for all participants in Table 3, with detailed responses in Multimedia Appendix 6. Feature ratings are provided for each phase. The participants rated almost all features as good, except for some data entry tasks using NFC, which were rated as neutral. Participants liked automatic data entry with NFC; however, technical issues such as slow transfer of time and difficulty tapping the card resulted in lower scores. Participants were asked to rate the entry of body weight and dialysate volume using NFC at the start of phase 3 based on their experience in phases 1 or 2 for comparison with manual data entry used in phase 3. The average rating for data entry with NFC was 2.3 for each task compared with 1.1 for manual data entry (Multimedia Appendix 6). Scores for all tasks using the handwritten logbook were nearly all higher than the CKD app with or without NFC and OCR, consistent with the preference for using the app.

Table 3. Feature and task ratingsa from participant observation in phases 1, 2, and 3 (detailed results in Multimedia Appendix 6).
FeaturePhase 1b, mean (range)Phase 2c, mean (range)Phase 3d, mean (range)Phase 3e, mean (range)
Open screens1.1 (1-2)1 (1)1.6 (1-3)N/Af
Enter body weight1.1 (1-2)1.2 (1-2)1.1 (1-2)1.4 (1-2)
Enter blood pressure1.1 (1-2)1.6 (1-3)1.1 (1-2)1.4 (1-3)
Enter dialysate1.1 (1-2)1.2 (1-2)1.1 (1-2)1.8 (1-3)
View metrics1.3 (1-2)1 (1)1.1 (1-2)1.7 (1-3)
Interpret metrics1.1 (1-2)1.2 (1-2)1.1 (1-2)N/A
Communications1 (1)1 (1)1 (1)1.2 (1-3)
User incentives1.1 (1-2)1.2 (1-2)1.2 (1-3)1.9 (1-3)

aRating of task or feature: 1=good, 2=neutral, and 3=not good.

bGroup 1 using the chronic kidney disease–peritoneal dialysis app with near-field communication or optical character recognition data entry; participants 1 to 5.

cGroup 2 using the chronic kidney disease–peritoneal dialysis app with near-field communication or optical character recognition data entry; participants 6 to 10.

dGroup 1 and 2 using the chronic kidney disease–peritoneal dialysis app with manual entry; participants 1 to 10, excluding 7.

eGroup 1 and 2 using logbook; participants 1 to 10, excluding 7.

fN/A: not applicable.

In general, participants found that manual data entry was easier to perform than using the NFC and OCR system. The use of NFC and OCR presented multiple challenges. Some of these issues were solved by providing tools such as an extension device for tapping the card to eliminate the need to bend over and touch the card to the scale on the floor and training participants to wait long enough for data transfer. Some issues were more difficult to troubleshoot within the time frame of this study, such as (1) lack of space in participants’ homes for NFC setup and access to power outlets; (2) internet instability resulting in slow data transfer; (3) size and design of the NFC-assembled weight scale, making it difficult for older or obese patients to stand on; and (4) lack of an alert to indicate that the NFC device was ready to place the NFC card for reading. Most patients also did not like to use the OCR function to enter the blood pressure readings because of poor image clarity and variations in ambient home lighting conditions.

Several important issues related to the functionality of the CKD-PD app were detected and corrected. The dialysate in and out volumes were reported by PD cycle, and the net daily ultrafiltration volume was not accurately calculated and displayed. There were issues with the display of hydration metrics, such as incorrect scale on the graph and a lack of previous results for comparison. During home observation of participants using the CKD-PD app with the NFC and OCR and monitoring system, participants expressed concerns about the usability of the app. An example was slowness when opening each icon, which was determined to be from opening multiple apps at once, and slow cellular or Wi-Fi network speeds were contributing factors. BMI was misinterpreted as overweight because of translation issues between the English and Thai languages. This was corrected by changing the wording so it would not be confused with weight. Participant height had to be entered daily to calculate the BMI. As this measurement did not change, the app was modified so that it automatically entered the participant height obtained daily. The original font was small and difficult to read. This was adjusted, and the readability of the screens was improved.

Completion and Validation of Hydration Metrics

The percentage completion of the hydration metrics for each participant was collected for all 3 phases (Table 4). The mean percentages of completion in phases 1 and 2 were 88% (SD 19) and 83% (SD 6.3), respectively, with a range of 57% to 100%. This was compared with the mean percentage of completion in phase 3 for all participants (68%, SD 18.6, range 18%-100%; Multimedia Appendix 7). The most common reasons for not collecting the hydration metrics in phases 1 and 2 were not knowing how to enter the data, forgetting to enter the data, problems with the system uploading the data to the CKDNET database, and switching from Android to iOS systems. The reasons for incomplete hydration metric collection in phase 3 (not using NFC) were forgetting to send the data to the CKDNET and switching from Android to iOS.

Table 4. Percentage completion of hydration metrics (detailed in Multimedia Appendix 7).
ParticipantPhase 1aPhase 2bPhase 3cDifference (%)d

Total, NDone, n (%)Total, NDone, n (%)Total, NDone, n (%)
18480 (95)e4219 (45)−50
26639 (59)8442 (50)−9
34242 (100)4226 (62)−38
48473 (87)8479 (94)7
54848 (100)5640 (71)−29
64238 (91)4242 (100)9
7f3935 (90)N/AgN/AN/A
84237 (88)4239 (93)5
93024 (57)8415 (18)−39
108027 (90)4233 (79)−11
Values, mean6556 (88)46.632 (83)57.637 (68)−15

aPhase 1: participants 1 to 5.

bPhase 2: participants 6 to 10.

cPhase 3: participants 1 to 10.

dDifference in percentage completion between phase 1 or 2 and phase 3.

eNot available.

fParticipant 7 expired before completion of the study.

gN/A: not applicable.

Of the 9 participants who entered hydration metric data in phase 1 or 2 and phase 3, there were 4 participants who had a greater than 29 percentage point decrease in their hydration metric completion. The other 5 participants had minimally decreased or increased hydration metric completion, ranging from −11% to 9%.

Validation of the hydration metrics improved from phases 1 or 2 and phase 3 (Table 5). In phases 1 and 2, the main issues were the upload of inaccurate body weight and ultrafiltration metric results to the CKDNET. This was because of problems with the NFC weight scale and app calculation of the ultrafiltration volume. In phase 3, these issues were resolved, the hydration metrics were fully validated with participants correctly entering the data in the CKD-PD app, and the results entered were accurately uploaded to the CKDNET.

Table 5. Validation of hydration metrics.
Participant12345678910Values, n (%)a
Phase 1b

Body weightNocNoNoNoNod0 (0)

Blood pressureYeseYesNoYesYes4 (80)

Dialysate inYesNoYesNoYes3 (60)

Dialysate outYesNoYesNoYes3 (60)

Ultrafiltration volumeYesNoYesNoNo2 (40)
Phase 2f

Body weightYesNoYesNoYes3 (60)

Blood pressureYesYesYesYesYes5 (100)

Dialysate inYesYesYesYesYes5 (100)

Dialysate outYesYesYesYesNo4 (80)

Ultrafiltration volumeYesNoYesYesNo3 (60)
Phase 3g

Body weightYesNDhNDYesYesYesN/AiYesYesYes7 (100)

Blood pressureYesYesYesYesYesYesN/AYesYesYes9 (100)

Dialysate inYesYesYesYesYesYesN/AYesYesYes9 (100)

Dialysate outYesYesYesYesYesYesN/AYesYesYes9 (100)

Ultrafiltration volumeYesYesYesYesYesYesN/AYesYesND9 (100)

aNumber and percentage of hydration metric values entered and uploaded with the correct results to Chronic Kidney Disease Prevention in the Northeast of Thailand.

bPhase 1: participants 1 to 5.

cNot uploaded accurately to Chronic Kidney Disease Prevention in the Northeast of Thailand.

dNot available.

eUploaded accurately to Chronic Kidney Disease Prevention in the Northeast of Thailand.

fPhase 2: participants 6 to 10.

gPhase 3: participants 1 to 10.

hND: not done by participant.

iN/A: not applicable; participant 7 expired before completion of the study.

Clinic Contacts

The number of times each participant contacted the PD clinic during each phase is presented in Table 6. All contacts were made using the Line messaging app by chat, audio, or video calls, except for1 in-person contact during phase 1. In phase 1, there were 104 clinic contacts compared with 13 and 22 in phases 2 and 3, respectively, and most were related to issues using the CKD-PD app. In phase 3, 59% (13/22) of the contacts were related to clinical concerns.

Table 6. Number and reason for contacting the peritoneal dialysis clinic by phase.
PhaseTotal number of contacts, NClinical issue, n (%)App issue, n (%)Other, n (%)
Phase 11040 (0)99 (95.2)5 (4.8)
Phase 2133 (23.1)10 (76.9)0 (0)
Phase 32213 (59.1)9 (40.9)0 (0)

Improvement Team Activities

After the completion of each phase, the research team summarized the findings for the app improvement team (Multimedia Appendix 8). In phase 1, troubleshooting issues with NFC and OCR were the focus of the improvement activities, along with app design issues. In phase 2, additional work was performed with the NFC data entry; however, more focus was given to resolving issues with how the hydration metric data were displayed and uploaded to the CKDNET database. In phase 3, the NFC prototype was not used as the participants preferred to input data manually into the CKD-PD app, and the upload of the hydration metric data to the CKDNET database was not reliable using NFC. The app improvement team made final improvements to improve the stability of the data transfer from the CKD-PD app to the CKDNET database, added the total daily ultrafiltration volume to the CKDNET, and set parameters for overhydration alerts.


Principal Findings

Overview

mHealth apps are touted as having great potential to transform the delivery of health care; however, the real-world development of mHealth apps in low- and middle-income countries reveals few success stories [12,17]. Often, a good app idea fails to succeed as the design does not meet the users’ needs and is difficult to use. Our research describes how a rapid cycle process improvement strategy helped researchers understand the benefits and challenges of using an mHealth app for use by patients on PD in a low-resource setting and optimize its features. The process revealed valuable insights into the factors influencing user attitudes, identifying technological and design flaws, and addressing barriers to user adoption of the CKD-PD app and monitoring system. Moreover, we believe that this methodology can be accomplished without requiring significant training in data collection, statistical analysis, or funding. The general principles of the user design process are applicable to a wide variety of locations, contexts, and subject domains.

User Adoption

The total scores for all the participants for most domains were in the agree and strongly agree categories, demonstrating a strong willingness to adopt the technology at the beginning of the study. Over the course of the study, 67% (6/9) of participants who completed both assigned phases demonstrated decreased interest in using the technology, although the total scores remained in the agree range or above. The decreases in the total score were primarily driven by decreases in the domains of voluntariness and intention to use. We believe this is likely because of issues with the NFC and OCR data entry features and app speed and functionality. We acknowledge that the UTAUT questionnaire is not typically used to evaluate technology adoption over time; however, we found that it provided insights into how our participants’ views changed over the course of using the CKD-PD app and monitoring system, although they had limited generalizability. The small sample size limited the statistical validity of our findings. However, we found the information useful as a sentiment analysis regarding user experience and provided insights into how different individuals may adopt the CKD-PD app.

CKD-PD App Issues

Design issues centered on simplicity and ease of use compared with a pencil and paper logbook. Simplifying the use of the app, for example, opening the app and the organization of the screens, readability of the fonts, input of the data, and graphical representation of trends, clearly needed to be refined based on user feedback. PD requires a substantial commitment of time to collect and record data from patients and their families throughout the day, whether in a logbook or an app. Our participants generally reacted positively to the self-monitoring features and quickly engaged with this new functionality but became frustrated when some features did not work well for them.

Technology Issues

Technology issues included problems with the NFC devices, unstable internet access, slow internet speed, app stability, uploading data to the CKD-NET cloud, and software issues causing it to freeze or not upload data properly. The participant observation revealed that NFC and OCR were good solutions for automatic data entry but that they need to be seamlessly integrated into the measurement devices; for example, the location of NFC on the scales on the floor was inconvenient, slow data transfer, and poor quality image capture with OCR. Despite these barriers, participants liked the concept, and with further development, we believe that NFC and OCR could be simple and effective methods of mitigating the barrier of data entry [28,29], as has been done with other devices such as glucose monitors.

Internet access, especially in low-income rural areas, is frequently slow or unstable, making the use of digital apps cumbersome. Some patients were concerned about the cost of sending or receiving data, although internet access was subsidized for the participants in this study. These will be challenging issues to remedy; however, if successful in reducing complications from overhydration, it may prove cost-effective in the long run to provide financial support for internet access for patients on PD, given the alternative costs of not using the app.

Programming issues resulting in app instability and data upload errors were critical problems that were identified. These were fixable by our software engineers and resulted in improved validation of the hydration metrics in phase 3. The completion of the hydration metrics dropped between phases 1 and 3. We believe this occurred as participants in phases 1 and 2 used NFC, which automatically uploaded the hydration metrics, and decreased after the team elected not to use NFC and OCR in phase 3 because of usability issues and inaccurate upload of the hydrate metrics to the CKDNET with NFC and OCR. The hydration metric accuracy improved with this change and resulted in 100% validation of the hydration metrics in phase 3. However, when manual entry was used in phase 3, participants had to remember to upload their data, resulting in incomplete hydration metrics. In the final improvements, a reminder alert addressed this issue.

Communication Issues

Contact with the PD clinic was much more frequent in phase 1 than in phase 2, although the participants in both phases were new users of the app. The study staff felt that this was because of differences in individual comfort when using the CKD-PD app. In phase 3, there were fewer contacts about the app, although there were 10 participants compared with 5 in phases 1 and 2. The contacts in phase 3 were about clinical concerns, suggesting increased confidence in self-monitoring features. In general, participants gave high ratings and positive comments about the ease of communication with the PD clinic using the CKD-PD app.

Impact on Future Deployment and Development of CKD-PD

Our research findings provided critically useful information for the optimization of the CKD-PD app and monitoring system. We plan to use it in a randomized controlled trial to further evaluate the efficacy of CKD-PD in the early detection and treatment of overhydration.

We have developed training materials for CKD-PD app users based on the insights gained from this study and plan to share them with future CKD-PD app users. We have improved awareness of user adoption issues and realize that this will be an iterative process as more patients on PD use the app. We believe that with additional engineering and design work, NFC is a potential solution for automating data entry, and we plan to pursue the further development of these features.

Limitations

Our study has several limitations. The NFC system and CKD-PD app were initially tested together; however, the NFC issues were too complicated to solve within the time frame and budget of the study period. Through targeted observations and data collection, we were able to tease out NFC and data entry issues from the app design and cloud-based monitoring processes.

We identified potential sources of bias. Observation bias could have influenced our results as participant observations were conducted by research staff from the PD clinic. Patients on PD, cared for by nephrologists and PD nurses from their clinic, may be inclined to report feedback they thought the PD staff wanted to hear. Confirmation bias may also be a concern as research staff may be looking for patterns or use issues that confirm opinions they already hold about the CKD-PD app, NFC and OCR data entry, and the monitoring system. Our sample size was small; however, this is consistent with recommendations for user design studies, indicating that a few participants can uncover most usability issues [30]. We also acknowledge the potential for selection bias in our participants, given their higher educational level than the average population in rural northeast Thailand, although the participants were representative of the Srinagarind Hospital PD patient population. We selected participants who had baseline comfort with smartphones for the user design study, although it limits generalizability as they were more interested and engaged in the design process.

An additional weakness is that we did not conduct an in-depth investigation into why our participants responded to the UTAUT questionnaire as much as they did. Although it has been validated and used in Thai populations [26,31], we did not consider the impact of age, gender, education, and cultural influences on UTAUT scores [32,33]. We acknowledge that UTAUT results have limited generalizability.

Comparison With Prior Work

Our research included several recommended strategies for mHealth app development and evaluation [34,35]. Our iterative rapid cycle process improvement approach used a multidisciplinary team, including patients, computer engineers and programmers, nephrologists, and PD nurses, to evaluate the CKD-PD app. We conducted our study activities in the participants’ homes and PD clinics where it would be used [36,37]. Our structured observation using the think-aloud method provided the app improvement team with a broad sense of what worked and what did not work, in addition to specific actionable feedback. Finally, our study participants were selected from the same study population as those who will use the CKD-PD app in northeast Thailand.

Our research team found several critical usability and functionality issues during the study. This experience reinforces the importance of these steps before an mHealth app can be successfully implemented [12,13,17]. Although our current NFC and OCR data entry features are not optimized, our results support the idea that app users want these features [27].

Conclusions

In conclusion, using rapid cycle progress improvement methodologies with a multidisciplinary team proved to be a useful strategy for optimizing the CKD-PD app and monitoring system. Our experience aligns with insights and recommendations regarding mHealth technology design and evaluation that using a multidisciplinary team in the context of the system in which it is used is essential for successful deployment. This process revealed critical issues that once addressed, position the CKD-PD app and monitoring system to achieve its potential to improve health outcomes.

Acknowledgments

The research reported in this publication was supported by the Fogarty International Center of the National Institutes of Health under award number R21TW010963. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors would like to acknowledge the Chronic Kidney Disease Northeast Thailand research project staff for their assistance in organizing and assisting with home observations. The authors would like to thank Ms Thankyalak Rattanasawad; the staff of the Center of Excellence in Kidney Diseases, Srinagarind Hospital, Faculty of Medicine at Khon Kaen University; Dr Jessica Haberer and the Massachusetts General Hospital Center for Global Health; Dr Nathaphop Chaichaya, Data Management and Statistical Analysis Center, Faculty of Public Health, Khon Kaen University Thailand; the other staff from the Data Management and Statistical Analysis Center and the Faculty of Public Health at Khon Kaen University; and Pobporn Danvirutai from T. Robotics, Co Ltd, Bangkok, Thailand.

Authors' Contributions

SA, EL, KEM, MGM, and BT designed the study, including working with the Data Management and Statistical Analysis Center app development team. CS developed and oversaw the use of near-field communication or optical character recognition and integration with the chronic kidney disease–peritoneal dialysis app. EL, SK, WC, TC, and PW collected and analyzed the data and drafted the manuscript. SA, KEM, and MGM assisted with the data analysis and critically revised the manuscript. All authors have read and approved the final manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Chronic Kidney Disease–Peritoneal Dialysis app screenshots.

PDF File (Adobe PDF File), 2613 KB

Multimedia Appendix 2

Using Chronic Kidney Disease–Peritoneal Dialysis app (near-field communication version).

MP4 File (MP4 Video), 15806 KB

Multimedia Appendix 3

Unified Theory of Acceptance and Use of Technology questionnaire on adoption and use of the Chronic Kidney Disease–Peritoneal Dialysis app.

PDF File (Adobe PDF File), 95 KB

Multimedia Appendix 4

Participant observation guide.

PDF File (Adobe PDF File), 42 KB

Multimedia Appendix 5

Unified Theory of Acceptance and Use of Technology scores by participant and domain between phases 1 and 3 or phases 2 and 3.

PDF File (Adobe PDF File), 174 KB

Multimedia Appendix 6

Summary of participant observation phases 1, 2, and 3.

PDF File (Adobe PDF File), 74 KB

Multimedia Appendix 7

Detailed hydration metric completion.

PDF File (Adobe PDF File), 78 KB

Multimedia Appendix 8

Recommendations from the app improvement team.

PDF File (Adobe PDF File), 55 KB

  1. Cha'on U, Wongtrangan K, Thinkhamrop B, Tatiyanupanwong S, Limwattananon C, Pongskul C, CKDNET group. CKDNET, a quality improvement project for prevention and reduction of chronic kidney disease in the Northeast Thailand. BMC Public Health 2020 Aug 27;20(1):1299 [FREE Full text] [CrossRef] [Medline]
  2. Anutrakulchai S, Mairiang P, Pongskul C, Thepsuthammarat K, Chan-On C, Thinkhamrop B. Mortality and treatment costs of hospitalized chronic kidney disease patients between the three major health insurance schemes in Thailand. BMC Health Serv Res 2016 Sep 29;16(1):528 [FREE Full text] [CrossRef] [Medline]
  3. Thailand Renal Replacement Therapy: Year 2020. Bangkok: The Nephrology Society of Thailand; 2020.
  4. Jotterand Drepper V, Kihm LP, Kälble F, Diekmann C, Seckinger J, Sommerer C, et al. Overhydration is a strong predictor of mortality in peritoneal dialysis patients - independently of cardiac failure. PLoS One 2016;11(7):e0158741 [FREE Full text] [CrossRef] [Medline]
  5. Guo Q, Lin J, Li J, Yi C, Mao H, Yang X, et al. The effect of fluid overload on clinical outcome in southern Chinese patients undergoing continuous ambulatory peritoneal dialysis. Perit Dial Int 2015 Dec;35(7):691-702 [FREE Full text] [CrossRef] [Medline]
  6. Dao Bui Quy Q, Pham Ngoc Huy T, Nguyen Duc L, Pham Van M, Nguyen Huu D, Nguyen Duy T, et al. Overhydration and low serum prealbumin predict peritoneal dialysis-related peritonitis in continuous ambulatory peritoneal dialysis patients. BMC Nephrol 2020 Nov 25;21(1):512 [FREE Full text] [CrossRef] [Medline]
  7. Shu Y, Liu J, Zeng X, Hong HG, Li Y, Zhong H, et al. The effect of overhydration on mortality and technique failure among peritoneal dialysis patients: a systematic review and meta-analysis. Blood Purif 2018;46(4):350-358. [CrossRef] [Medline]
  8. Ng JK, Kwan BC, Chow K, Pang W, Cheng PM, Leung C, et al. Asymptomatic fluid overload predicts survival and cardiovascular event in incident Chinese peritoneal dialysis patients. PLoS One 2018;13(8):e0202203 [FREE Full text] [CrossRef] [Medline]
  9. Hong YA, Yoon HE, Choi BS, Shin SJ, Kim Y, Lee SY, et al. The effect of strict volume control assessed by repeated bioimpedance spectroscopy on cardiac function in peritoneal dialysis patients. Sci Rep 2019 Nov 27;9(1):17679 [FREE Full text] [CrossRef] [Medline]
  10. Kim Y, Biesen WV. Fluid overload in peritoneal dialysis patients. Semin Nephrol 2017 Jan;37(1):43-53. [CrossRef] [Medline]
  11. Kuehn BM. Is there an app to solve app overload? JAMA 2015 Apr 14;313(14):1405-1407. [CrossRef] [Medline]
  12. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
  13. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res 2017 Jun 29;19(6):e232 [FREE Full text] [CrossRef] [Medline]
  14. L'Engle K, Raney L, D'Adamo M. mHealth resources to strengthen health programs. Glob Health Sci Pract 2014 Feb;2(1):130-131 [FREE Full text] [CrossRef] [Medline]
  15. Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G. mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob Health Sci Pract 2013 Aug;1(2):160-171 [FREE Full text] [CrossRef] [Medline]
  16. Hoque MR, Rahman MS, Nipa NJ, Hasan MR. Mobile health interventions in developing countries: a systematic review. Health Informatics J 2020 Dec;26(4):2792-2810 [FREE Full text] [CrossRef] [Medline]
  17. Roess A. The promise, growth, and reality of mobile health - another data-free zone. N Engl J Med 2017 Nov 23;377(21):2010-2011. [CrossRef] [Medline]
  18. Arie S. Can mobile phones transform healthcare in low and middle income countries? BMJ 2015 Apr 22;350:h1975. [CrossRef] [Medline]
  19. Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382 [FREE Full text] [CrossRef] [Medline]
  20. DAMASAC Data Management and Statistical Analysis Center. Faculty of Public Health, Khon Kaen University.   URL: http://www.thaicarecloud.org/ [accessed 2022-02-10]
  21. Coskun V, Ozdenizci B, Ok K. The survey on near field communication. Sensors (Basel) 2015 Jun 05;15(6):13348-13405 [FREE Full text] [CrossRef] [Medline]
  22. Morak J, Schreier G. Design and evaluation of Near Field Communication (NFC) technology based solutions for mHealth challenges. In: Mobile Health. Cham: Springer; 2015.
  23. Morak J, Kumpusch H, Hayn D, Modre-Osprian R, Schreier G. Design and evaluation of a telemonitoring concept based on NFC-enabled mobile phones and sensor devices. IEEE Trans Inf Technol Biomed 2012 Jan;16(1):17-23. [CrossRef] [Medline]
  24. UA-767NFC Blood Pressure Monitor with NFC devices. A&D Medical.   URL: https:/​/www.​andmedical.com.au/​product-service/​telemedicine-bluetooth-and-nfc-products/​ua-767nfc-blood-pressure-monitor-with-nfc-devices [accessed 2022-02-10]
  25. Anabtawi N, Freeman S, Ferzli R. A fully implantable, NFC enabled, continuous interstitial glucose monitor. IEEE EMBS Int Conf Biomed Health Inform 2016 Feb;2016:612-615 [FREE Full text] [CrossRef] [Medline]
  26. Kijsanayotin B, Pannarunothai S, Speedie SM. Factors influencing health information technology adoption in Thailand's community health centers: applying the UTAUT model. Int J Med Inform 2009 Jun;78(6):404-416. [CrossRef] [Medline]
  27. Jaspers M, Steen T, Bos C, Geenen M. The think aloud method: a guide to user interface design. Intl J Med Inform 2004 Nov;73(11-12):781-795. [CrossRef] [Medline]
  28. Rajan JV, Moura J, Gourley G, Kiso K, Sizilio A, Cortez AM, et al. Understanding the barriers to successful adoption and use of a mobile health information system in a community health center in São Paulo, Brazil: a cohort study. BMC Med Inform Decis Mak 2016 Nov 17;16(1):146 [FREE Full text] [CrossRef] [Medline]
  29. Cornet VP, Toscos T, Bolchini D, Rohani Ghahari R, Ahmed R, Daley C, et al. Untold stories in user-centered design of mobile health: practical challenges and strategies learned from the design and evaluation of an app for older adults with heart failure. JMIR Mhealth Uhealth 2020 Jul 21;8(7):e17703 [FREE Full text] [CrossRef] [Medline]
  30. Nielson J, Landauer TK. A mathematical model of the finding of usability problems. In: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems. 1993 Presented at: INTERCHI93: Conference on Human Factors in Computing; Apr 24 - 29, 1993; Amsterdam The Netherlands. [CrossRef]
  31. Boontarig W, Chutimaskul W, Chongsuphajaisiddhi V, Papasratorn B. Factors influencing the Thai elderly intention to use smartphone for e-Health services. In: Proceedings of the 2012 IEEE Symposium on Humanities, Science and Engineering Research. 2012 Presented at: 2012 IEEE Symposium on Humanities, Science and Engineering Research; Jun 24-27, 2012; Kuala Lumpur, Malaysia. [CrossRef]
  32. Nunes A, Limpo T, Castro SL. Acceptance of mobile health applications: examining key determinants and moderators. Front Psychol 2019;10:2791 [FREE Full text] [CrossRef] [Medline]
  33. Dwivedi Y, Shareef M, Simintiras A, Lal B, Weerakkody V. A generalised adoption model for services: a cross-country comparison of mobile health (m-health). Government Inf Q 2016 Jan;33(1):174-187 [FREE Full text] [CrossRef]
  34. Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med 2016 Nov;51(5):843-851 [FREE Full text] [CrossRef] [Medline]
  35. Wallis L, Blessing P, Dalwai M, Shin SD. Integrating mHealth at point of care in low- and middle-income settings: the system perspective. Glob Health Action 2017 Jun;10(sup3):1327686 [FREE Full text] [CrossRef] [Medline]
  36. Woulfe F, Fadahunsi KP, Smith S, Chirambo GB, Larsson E, Henn P, et al. Identification and evaluation of methodologies to assess the quality of mobile health apps in high-, low-, and middle-income countries: rapid review. JMIR Mhealth Uhealth 2021 Oct 12;9(10):e28384 [FREE Full text] [CrossRef] [Medline]
  37. Feroz A, Kadir MM, Saleem S. Health systems readiness for adopting mhealth interventions for addressing non-communicable diseases in low- and middle-income countries: a current debate. Glob Health Action 2018;11(1):1496887 [FREE Full text] [CrossRef] [Medline]


APD: automatic peritoneal dialysis
CAPD: continuous ambulatory peritoneal dialysis
CKD: chronic kidney disease
CKDNET: Chronic Kidney Disease Prevention in the Northeast of Thailand
CKD-PD: chronic kidney disease–peritoneal dialysis
mHealth: mobile health
NFC: near-field communication
OCR: optical character recognition
PD: peritoneal dialysis
UTAUT: Unified Theory of Acceptance and Use of Technology


Edited by A Mavragani; submitted 01.03.22; peer-reviewed by K Paik, R Marshall; comments to author 08.05.22; revised version received 26.05.22; accepted 29.05.22; published 06.07.22

Copyright

©Eakalak Lukkanalikitkul, Sawinee Kongpetch, Wijittra Chotmongkol, Michael G Morley, Sirirat Anutrakulchai, Chavis Srichan, Bandit Thinkhamrop, Theenatchar Chunghom, Pongsai Wiangnon, Wilaiphorn Thinkhamrop, Katharine E Morley. Originally published in JMIR Formative Research (https://formative.jmir.org), 06.07.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.