Designing Survey-Based Mobile Interfaces for Rural Patients With Cancer Using Apple’s ResearchKit and CareKit: Usability Study

Introduction

Background

Mobile health (mHealth) technologies have been around for over a decade, yet the percentage of adult patients actively using these mHealth technologies is lower than desired [1,2]. The demographics of adults not using mHealth solutions are consistent with patients from rural populations, racial and ethnic minority groups, and older individuals, which overlaps with persons categorized as medically underserved [3]. According to the Health Resources and Services Administration, medically underserved populations have been designated as having too few primary care providers, a high infant mortality rate, prevalent poverty, or a high older adult population [4,5]. Specifically, rural communities, such as those of the Southeastern United States or Appalachia, commonly have higher rates of chronic disease, reduced access to providers, and fewer medical resources than their urban counterparts [6-10]. The ubiquity of mobile devices makes mHealth particularly attractive for reaching populations that are disadvantaged [11-14]. A promising use of mHealth is remote patient monitoring, which can include objective data, such as biometrics via sensor devices, or subjective data, such as quality-of-life surveys via patient-reported outcomes, resulting in a better understanding of a patient’s overall health and symptom tracking between visits [15,16].

As of 2023, a total of 90% of people in the United States own a smartphone. In addition, it was reported that while 27% of people who lived in rural areas did not have broadband at home, 87% owned a smartphone [17,18]. In particular, adopting innovations in rural communities is essential because the disparities between advantaged communities and those that are disadvantaged continue to grow for digital literacy [16,19,20], also known as the digital divide [20-22]. Factors in the divide between advantaged groups and those that are disadvantaged are health literacy, knowledge of technology, and comfort of use [20,23,24]. Designers should ensure that system user interfaces (UIs) are universally acceptable, particularly concerning users with limited technical proficiency [20]. Ensuring the usability of a system is essential for accurate data collection and reducing attrition rates [25-27].

Simply digitizing a paper-based survey may present complexities that render the digitized counterpart unusable and discourage the required frequency and accuracy necessary to improve adherence [28-30]. For example, patients might accidentally submit their responses prematurely or fail to submit them at all. In addition, usability plays a crucial role in the adoption of innovative technologies, as explored through the technology acceptance model [31-33] and research on mHealth adoption [27,34-36]. A participant’s age has been shown to substantially affect the ease of navigation and learnability, especially as cognition and motor control decline [37]. However, proper interface design can minimize user error and allow a smooth user experience [38]. To address the aforementioned concerns, researchers and developers can co-design the UI to ensure digitization is tailored to the respective demographic [39,40].

Distress Screening

According to the National Cancer Institute, distress is an “emotional, social, spiritual, or physical pain or suffering that may cause a person to feel sad, afraid, depressed, anxious, or lonely” [41]. Distress is prevalent in patients with cancer regardless of disease stage or modality [8,42-45], and untreated distress has been shown to lead to greater pain, reduced physical function, increased medical costs, and longer stays in the hospital [8,42,46]. The National Comprehensive Cancer Network (NCCN) created the Distress Thermometer and Problem List, hereafter referred to as the NCCN assessment, for use as a screening tool for recognizing distress in patients with cancer (Figure 1) [47,48], and has since been shown to indicate distress accurately [42,49]. The NCCN assessment was designed to improve patient care and increase patient quality of life. Furthermore, studies have shown that distress screening can improve health outcomes, including reduced morbidity and mortality [8,42].

‎

Unfortunately, due to factors such as staff burnout or emotional fatigue, signs of distress in patients may go unnoticed [42,43]. In addition, there can also be variations across different cancer centers regarding when patients should be screened [42]. This raises the need for a more effective and efficient process related to distress screening [43]. The implementation of the NCCN assessment as a mobile app poses many advantages, such as real-time identification of distress factors and triage to the proper provider, generating insightful data around common issues during the cancer experience, and providing insight into potential resource allocation [14,15,50].

Conversely, there are barriers to the implementation of new tools in health care. For example, modifying any clinical practice can be challenging, and providers hesitate to make drastic changes without sufficient evidence of substantial benefit and patient-driven motivation [51-53]. In addition, digital implementations of distress screening that are considered complex or not user-friendly by target users can lead to reduced effectiveness. Effective distress screening requires patient adherence and accurate information input to enable providers to devise proper interventions and follow-ups [14]. Despite the challenges, technology poses a great solution to address the needs of patient distress monitoring when resources and access to care are limited [54,55]. In particular, the prevalence and ubiquity of mobile devices present opportunities for patients in remote and rural areas to use mHealth apps to enhance their care. By reducing the time between distress screenings, providers and researchers can improve their understanding of a patient’s overall distress and track symptoms between visits.

Open-Source Frameworks

In 2014, Apple launched HealthKit, a central repository for health and fitness data that is automatically available on all iOS devices, and subsequently launched an open-sourced ResearchKit in 2015. Three major modules make up ResearchKit: informed consent, surveys, and active tasks [56,57]. Institutions such as Duke and Stanford have launched research-based mobile apps using ResearchKit [58]. Mobile apps developed using ResearchKit have already begun to be integrated into standard hospital software systems such as Epic Systems [59]. The ResearchKit framework has been used in various mHealth apps, including those focused on asthma, autism, Parkinson disease, type 2 diabetes, cancer, cardiovascular issues, mental health, pregnancy, postpartum conditions, hepatitis, and epilepsy [56].

In 2016, Apple released and open sourced a complementary framework to HealthKit and ResearchKit, called CareKit (Apple Inc), which supports personalized health care with customized care plans, adherence tracking, and visualization of trends in user data [60]. CareKit consists of 3 independent modules: CareKitUI, CareKitStore, and CareKitFHIR. CareKitUI provides a set of health, fitness, and medical views that can be customized to create mobile apps. CareKitStore provides local storage of patient data on personal devices using CoreData, which is Apple’s implementation of an SQLite database. Data generated using the CareKitStore framework are securely stored and encrypted on the device [61]. CareKitFHIR enables seamless conversion between CareKitStore objects and Health Level 7 Fast Healthcare Interoperability Resources [62] resources to integrate with Fast Healthcare Interoperability Resources–based electronic health records and applications. Combining HealthKit, CareKit, and ResearchKit allows for the development of mHealth apps with many desired features for remote patient monitoring and self-management of health by users [26,63] with reduced effort from developers [60]. Together, these iOS and iPadOS frameworks enable the collection and sharing of user-generated health data and streamline the process of building survey-based mobile apps for research [64].

Survey Design Heuristics

The following heuristics from previous research can be followed to provide an optimal user experience for respondents in digital surveys. Surveys should be aesthetically pleasing, easy to navigate [28,30], and have an explicit visual flow [65]. Although some researchers [66] have found that scrolling layouts can sometimes have faster completion times, designers should still be strategic in deciding between paging versus scrolling along with the grouping and sequencing of questions. Furthermore, when considering answer choices, potential options should include some variation of “do not know” [67,68]. In addition, surveys should be succinct [30,65,69] and maintain a standardized format, as variations in format can lead to decreased usability [28]. Surveys should always be easy to understand, with clear directions for answering questions [28,65,69]. Moreover, survey language should mimic verbal dialogue whenever possible [28]. Additional features to consider implementing are showing participants their progress toward completion, a thank you page, and an overview of results at the end [28].

A set of usability heuristics often used as a baseline for designing systems is the 10 principles for interaction design postulated by Nielsen [70], that consist of the following guidelines: (1) visibility of system status; (2) match between system and real world; (3) user control and freedom; (4) consistency and standards; (5) error prevention; (6) recognition rather than recall; (7) flexibility and efficiency of use; (8) aesthetic and minimalist design; (9) recognize, diagnose, and recover from errors; and (10) help and documentation. Finally, incorporating the 10 general principles for interaction design by Nielsen [70,71] will make UIs more accessible, user-friendly, and intuitive.

This Study

Ensuring the interface usability of an mHealth system is essential to its effectiveness, which often requires patient adherence and accurate information input to enable providers to devise proper interventions and follow-ups and prevent attrition [27,72]. Previous research suggests that co-designing for users with limited digital literacy, such as older or rural users, may be required to build suitable usable interfaces, but it often requires considerable time and resources [73-75]. Designers often co-design the UI to address the concerns and ensure digitization is tailored to the respective demographic [40]. This pilot study assesses the usability of multiple UI implementations of the NCCN assessment (Figure 1), particularly for understudied populations such as Appalachian and rural patients with cancer who are underserved and vulnerable [76,77]. The different UIs were designed without co-design to assess whether usable UIs could be achieved for this demographic when resources for the design stage are limited.

Methods

Ethical Considerations

The University of Kentucky’s Institutional Review Board approved all research activities (approval number: 64149). Informed consent information was provided to participants with a cover letter, and participants confirmed their consent to participate in the study after an in-app onboarding and consent process. Participants could withdraw from the study at any time. Data were collected anonymously and stored on encrypted servers. Participants were not compensated for taking part in this study.

Recruitment

Patients with cancer were recruited in person from the University of Kentucky’s Markey Cancer Center to participate in this study between July and August 2021. Two medical oncologists at the cancer center permitted us to interact with willing patients at their clinics. The physicians asked if patients would be willing to speak to a researcher about the study during their visits. If patients agreed, we went to the respective waiting room; informed the patients about the purpose of the study; gauged interest; and, if applicable, proceeded with the study tasks. If patients were not interested in the study, we thanked them for their time, and they were not entered into the pilot study. We recruited 30 patients to participate in this study. Participants did not need to have a certain level of digital literacy, as we were interested in participants who were not very familiar with mobile devices and apps to assess whether Assuage would be usable for people with limited digital literacy. Participants were not offered payment to participate in this study.

Procedure

This pilot study used between-groups A/B testing to compare the usability of 4 different UI designs for completing a distress survey in a mobile app. A/B testing, or split testing, is a randomized experiment where users are shown ≥2 versions of a system, website, or app to determine which version performs better based on specific metrics [78]. A/B testing protocols are commonly used in industry, and different system versions are randomly assigned to users for comparative analysis [78,79]. All research procedures were conducted in person at the Markey Cancer Center. Assuage was preinstalled onto an iPad for participants to use. After we went over informed consent information with the patient, the procedure went as follows: we asked patients the following demographic questions: age range, sex, ethnicity and race, education, residence, familiarity with the paper form of the NCCN assessment, mobile app use frequency, and mobile apps for health and medical use frequency.

We then introduced Assuage to the patient, which reiterated consent via an in-app onboarding process and study information and reverified that the patient was still interested in participating. Assuage was programmed to randomly select one of the UIs to display to users following the in-app onboarding. This was done by assigning a number from 0 to 3 to the different UIs and randomly selecting an integer in that range. Therefore, we did not have direct control over which UI group participants were assigned. Participants were presented with the randomly selected UI and instructed to follow the app prompts to complete the distress assessment. If a participant went through the NCCN assessment with a companion, the participant did all the physical interaction with the interface. It was appropriate for 7% (2/30) of the patients to enlist the help of their accompanying caregiver, as this mimics assistance needed naturally in the clinical or at-home setting.

While participants were completing the NCCN assessment in the app, we observed and collected notes on any usability issues, software bugs, and other noteworthy information. If a participant got stuck or confused using the app, we gently nudged them on how to proceed and documented the usability issue. After participants finished using Assuage to complete the assessment, they completed a usability assessment. Afterward, participants were asked to provide additional comments regarding the study and their use of Assuage. We also inquired about each participant’s specific set of mobile devices. No identifiable participant information was collected through the Assuage app. No video or audio recordings took place. Notes about the participants’ actions, usability issues, and responses were also collected, and usability issues were organized into related themes.

Outcomes Measured

This study measured perceived usability by participants after completing the distress assessment with Assuage. Scores from the System Usability Scale (SUS) [80] were compared among the UI design variations within Assuage. The SUS is a validated tool with a reputation for providing swift and reliable results [81,82]. The SUS consists of 10 statements, or items, with a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). A negative response is considered a score <3 for positively worded statements and >3 for negatively worded statements. While the SUS is not a diagnostic tool, it can effectively determine whether the tested system would be generally usable even when used to evaluate small sample sizes with as few as 5 users [80-86]. The SUS has been used in industry and academic research and is sufficient for pilot studies of mHealth apps [25,27,40,83,87-89]. Individual SUS items are provided in Figure 2.

‎

The SUS scores from participants (N=30) were grouped by the respective UIs tested by the participants, and the results were analyzed using the SciPy Python (version 3; Python Software Foundation) package in iPython Notebooks [90]. A one-way ANOVA was performed to compare the effect of the different UIs on usability, represented by the SUS score. Lewis and Sauro [81] assessed data from 241 usability studies to create a curved grading scale where an SUS score of 68 is a “C” grade and considered acceptable usability. However, industry targets an SUS score of 80 to represent an above-average user experience [81]. We used a content analysis approach to analyze qualitative data, such as observed usability issues and participant comments. Content analysis is a method used to systematically classify data, usually written, into segments with codes (labels) to make inferences about the content and underlying themes [91]. Data were coded using Taguette, which is a free and open-source qualitative tool [92].

System Design and Development

Assuage is a Health Insurance Portability and Accountability Act (HIPAA)–compliant mobile iOS, iPadOS, and watchOS platform developed using Apple’s HealthKit [93], CareKit [60], and ResearchKit [56]. Assuage is a research test bed for assessing and improving patient care through health-related studies. Remote patient monitoring can be accomplished through Assuage by adding various quality-of-life surveys, such as the NCCN assessment in Figure 1. Additional frameworks such as ParseCareKit [94] synchronize ResearchKit and CareKit data with a HIPAA-compliant server [95]. Assuage offers multiple UIs for patient input of subjective information such as their distress symptoms. The decision to provide multiple UIs is based on the knowledge that some demographics, such as rural patients with cancer, have not heavily adopted mHealth but are also not completely removed from modern everyday technologies, such as mobile devices or smartphones [17,18,96]. Conversely, the number of rural-dwelling adults who own a smartphone continues to rise [18], creating avenues for mHealth to have a larger impact on this population. Therefore, we wanted to gauge if standard UI elements common in mobile interfaces provide acceptable usability for an mHealth use case, such as symptom reporting, without expending extra resources on co-design.

Four UIs were implemented in Assuage for the pilot. All the UIs were designed with Apple’s ResearchKit and 1 with CareKit, which leverages Apple’s Human Interface Guidelines [97]. ResearchKit and CareKit provide out-of-the-box UI views and elements for developers to build health and medical mobile apps, which have been used in various research studies [34,64,98-101]. Screenshots of the different UIs are shown (Figures 3-6). In particular, the UIs differ in how the NCCN assessment components are displayed and navigated. When gauging a patient’s distress using the NCCN assessment, the reported value of the distress thermometer component typically correlates to the immediate actions taken by the care team regarding the patient. With this in mind, the entirely digitized interfaces (UIs 2-4) present the distress thermometer first, but patients can still choose to skip any question in all UI versions. Descriptions of the different UIs are presented in Textbox 1.

‎

‎

‎

‎

To reiterate, the most significant change in design among the different UIs is the navigation style and how a user will traverse through the survey in the app. Regarding mHealth tracking apps for users with chronic illness, the design should be simple, self-explanatory, visually appealing, and intuitive to navigate [30].

Figure 7 highlights which of the usability principles by Nielsen were present in the different UI variations. Note that 3 out of the 4 UIs used out-of-the-box interface elements, while we customized the navigation elements of UI 3 to include a flexible survey navigation method. This customization aligns with the guideline by Nielsen for flexibility and efficiency of use (#7) and is supported by prior research on electronic survey navigation design [28,30]. Further description of how the different UIs in Assuage satisfied the usability heuristics can be found in Table S1 in Multimedia Appendix 1.

‎

Results

Overview

This section presents the findings of this pilot study regarding the 4 UIs. Descriptive statistics are reported in Table 1. A total of 30 usability surveys were completed across Assuage’s 4 UIs. This study was not designed or powered to detect the differences among the UIs; therefore, the comparative results reported should be considered preliminary evidence [40].

Participant Demographics

The demographics of participants are summarized in Table 1. Most participants were older than 50 years (24/30, 80%), had up to a high school education (19/30, 63%), lived in a rural area (25/30, 83%), and were unfamiliar with mHealth apps (21/30, 70%). Participant sex and general mobile app use were split, with slightly more female participants (16/30, 53%) and users of mobile apps with a frequency of at least sometimes or more (17/30, 57%). Approximately half of the participants (16/30, 53%) used Assuage in light mode, and the rest (14/30, 47%) used Assuage in dark mode. While we did not gather specific data on each participant’s cancer type and mode of treatment, the 2 oncologists who were a part of this study specialize in the following: (1) all forms of lung, head, and neck cancers; and (2) bone and soft tissue sarcomas, colorectal, pancreatic, and hepatobiliary cancers. In total, 83% (25/30) of the participants were patients of the first oncologist, and 17% (5/30) were patients of the second oncologist. Most participants (29/30, 97%) resided in Kentucky, and the remainder (1/30, 3%) resided in West Virginia.

System Usability Scores

The mean SUS score across the UIs was 75.8 (SD 22.2). Participants were randomly distributed across the 4 UIs. Among the 30 participants, 6 (20%) assessed UI 1 with a mean SUS score of 70.4 (SD 25.3), 8 (27%) assessed UI 2 with a mean SUS score of 67.2 (SD 31.2), 7 (23%) assessed UI 3 with a mean SUS score of 80.0 (SD 14.1), and 9 (30%) assessed UI 4 with a mean SUS score of 80.3 (SD 16.1). The SUS scores for each UI are reported in Table 2. Figure 8 shows the distribution of SUS scores for the different UI groups in relation to different target SUS scores. The dashed line represents an acceptable usability rating of ≥68 [81]. The dash-dotted line represents the industry target score of 80 to determine good usability [81]. Of the 4 UIs, 3 (UI 1, UI 3, and UI 4) had an average SUS score above the acceptable threshold of at least 68 and 2 (UI 3 and UI 4) met the industry threshold of at least 80. The average SUS score of UI 2 was just below what can be considered acceptable usability by 0.8 points. A one-way ANOVA was done to compare the effect of the UIs on the SUS scores. However, the results were not statistically significant (F_3,26=0.68; P=.57). Figure 9 depicts the SUS scores across the UIs grouped by participant age. Additional figures depicting the SUS scores across the different UIs grouped by participant mobile use and light mode versus dark mode are shown in Figures S1 and S2 in Multimedia Appendix 1. The averages of these groupings are shown in Table S2 in Multimedia Appendix 1. The average score for each item on the SUS (Figure 2) is also reported in Table S3 in Multimedia Appendix 1. Of the participants who rated the UIs in Assuage as having less-than-acceptable usability, all were aged >50 years and unfamiliar with health apps (10/30, 33%) and a couple did not regularly use mobile apps (2/30, 7%).

‎

‎

An overview of the usability for each UI group is presented in Table 2, including the number of participants per group, the mean and median SUS scores per group, how many participants rated a UI with a less-than-acceptable usability score, and how many usability issues occurred with each UI group.

Though our findings depict differences in scores between participant groups, they are not as significant. For example, UI 3 had a mean difference of approximately 14 points, and UI 4 had a mean difference of approximately 1.3 points when comparing mobile app users to nonusers (Table S2 in Multimedia Appendix 1). Similarly, when looking at participants aged >50 years, UI 3 and UI 4 have a smaller variance in usability score, and UI 3 had the tightest distribution with only 1 user rating below acceptable usability. Another interesting finding was the difference in usability scores among interfaces in dark mode and light mode. Aside from UI 1, the UIs in dark mode received significantly lower average usability scores, approximately 20 to 30 points, when compared to light mode.

Usability Issues

Although all UIs were considered usable for patients, there were quite a few usability issues that could correlate with users’ lack of digital literacy. Approximately half of the participants (16/30, 53%) encountered usability issues when using Assuage. Most participants who experienced issues were aged >50 years (13/16, 81%) and did not regularly use mobile apps (12/16, 75%). A total of 16 usability problems were identified during the study. The usability issues experienced were divided into the following themes: data input and collection (15 issues), navigation (12 issues), instructions (3 issues), NCCN (4 issues), and color and interaction (2 issues). Table 3 presents the usability issues and the frequency of occurrence. Data input and collection are issues that could affect the user’s accuracy and input of distress data. Navigation issues are related to how the user navigates the assessments within the app. Instructions are issues where clearer instruction is needed. NCCN are issues related to the NCCN questionnaire. Color and interaction are usability issues that did not fit well in the previous themes.

Participant Feedback

Patients had mixed perceptions of the different UIs’ learnability and usefulness. Positive responses from the participants described the UIs as easy, simple, intuitive, helpful, and good. Negative responses can be summarized as difficult, nonintuitive, inconsistent, and not for everyone. Regarding overall willingness to use an app for self-reporting symptoms, 7% (2/30) of the participants explicitly said they would want to use a symptom-reporting app more frequently (separate from the SUS item 1 [Figure 2], which states, “I think that I would be willing to use this system frequently”). Participants also expressed that if a physician told them to use the app, they would. Table 4 presents selected participants’ comments after using Assuage.

Desired features and improvements for reporting distress symptoms in an mHealth platform included distress data being sent directly to the physician, flagging the medical team if a patient reports high distress, prompts following completion of the distress assessment that can direct patients on who to contact depending on symptoms reported, proper feedback letting the patient know that their answers have been recorded, and an option to answer “none” if the patient has no symptoms instead of choosing to skip the question set.

Discussion

Principal Findings

This study assessed if state-of-the-art mobile app interface designs from Apple’s open-source ResearchKit and CareKit libraries would be usable for patients with cancer from rural areas. We leveraged the UI elements from Apple’s ResearchKit and CareKit frameworks to implement 4 different UI designs for patients to complete the NCCN distress assessment on the Assuage platform. The UIs varied by how the assessment questions were presented and navigated. This pilot study found that a survey-based app developed with Apple’s open-source libraries had a usable interface for patients with cancer within our target demographic. Specifically, using these frameworks, we achieved acceptable usability scores among nontraditional users, such as those who were older and did not regularly use mobile apps. The implication is that the frameworks are suitable for carrying out mHealth research with this demographic and can be used as a base for full-stack mHealth apps.

In addition, we evaluated if co-designing the interfaces was necessary to achieve acceptable usability with patients with cancer who were older or from rural areas. The results of this study show that it is possible to achieve good usability without co-design, which can reduce the time and resources spent in the design and development stages of a system or app for conducting mHealth research. Predictably, participants who were older than 50 years and did not use mobile devices regularly experienced the most usability issues. The most prominent usability issues were related to data input and navigation, with 15 and 12 occurrences, respectively. The most critical usability issues were participants needing to learn how to answer the distress scale and the UI assuming a participant knows when to scroll vertically. Not only did these 2 issues have the highest count of participants who experienced them, 23% (7/30) and 13% (4/30), respectively, but not addressing them can hinder participant completion of the survey, accurate reporting of symptoms and distress, and motivation to use the system.

Finally, we wanted to understand what caused a specific UI to have a higher usability rating than the others as a basis to move forward for future research studies with our target demographic. Our findings show that most participants (20/30, 67%) rated the UIs as having acceptable and above-average usability across the different interfaces, with UI 3 and UI 4 averaging approximately 10 points higher than UI 1 and UI 2 using the SUS. UI 3 and UI 4 also met the industry threshold for good usability with average SUS scores of at least 80. Despite navigation and input challenges, participants could still complete the in-app survey and expressed the willingness to use an mHealth system for self-reporting symptoms. Unsurprisingly, participants were more concerned about what happened after reporting symptoms, such as whether the physician would be notified or if the participant would receive feedback on how to continue based on the reported symptoms.

Comparison With Prior Work

Prior work suggests that mHealth systems should be co-designed with target users for optimal outcomes and usability [7,8,40]. Aronoff-Spencer et al [40] used participatory design to recreate an alternative design to the NCCN assessment. Digital and paper prototypes of the redesigned survey were compared to the original using the SUS, resulting in patients finding the digital prototypes more usable than their paper counterparts. The usability of Assuage’s different UIs was comparable to the co-designed prototypes without undergoing the resource-intensive process. Similarly, our usability results were comparable to other mHealth studies using the SUS to assess iterative designs [25,102-104].

While the usability issues encountered by participants could be attributed to digital literacy, developers can take extra steps to ensure universal design when using development frameworks. Formatting a survey for web and mobile delivery has been evaluated, but has had conflicting results [65-67,69]. For example, usability heuristics say that vertical and horizontal scrolling should be avoided when possible. Apple’s Human Interface Guidelines provide best practices for scroll views, including the use of scroll indicators. These indicators show users how much of the content they have scrolled through and how much is left [97]. Designing using paging instead of scrolling formats surveys in a clean and easy-to-read manner. Minimizing scrolling prevents users from missing questions or important interface elements, such as navigation buttons. Alternatively, studies have also found that scrolling layouts resulted in higher perceived usability and faster survey completion times [65,66]. Our usability results were slightly better with a paging design (UI 2 vs UI 3, with an average usability score of 67.2, SD 31.2 and 80.0, SD 14.1, respectively). UI 4 used Apple’s CareKit UI (a modular design combined with vertical scrolling) and received good usability scores (mean 80.3, SD 16.1), contradicting some of the best practices found in the literature. Notably, the modularized surveys are displayed in a manner similar to paging designs. In addition, it is interesting to note that the 2 UIs that provided more freedom in navigating the survey were the most highly rated. Reflecting on the usability heuristics by Nielsen (Figure 7), the navigation schemes implemented in UI 3 and UI 4 were the only interfaces that satisfied the heuristic (#7) of flexibility and efficiency of use. Considering the visual similarities between UI 2 and UI 3, we can infer that the flexible navigation, coupled with the grouping of questions on different pages, significantly improved usability scores.

Prior work suggests that respondents should be offered a “none” option or similar when presented with a list of other choices [68]. However, the placement of that option influences whether participants choose it. Placing an option, such as “none,” when other choices do not apply at the top of the page results in more respondents choosing it compared to when placed at the bottom of the survey [67], which can be important to consider for the thoroughness of data. In our case, we did not require participants to input an answer in every section and included a “skip” option at the bottom of the page, separate from the possible symptom choices. Nevertheless, some participants would have preferred an actual answer choice instead of skipping the page, as it made them feel like they were not fully completing the assessment. At times, the “skip” button did not stand out to participants as a tappable button compared to the “next” button, which had a visible background (eg, Figure 5, steps 2-3).

Participants encountered the most problems with the distress scale. The use of rating scales in surveys is fairly common [29,30]; however, for some participants, it was not intuitive to slide or tap to interact with the distress scale. All but one of the participants (6/7, 86%) who experienced this problem did not regularly use mobile apps. We attempted to keep the question format as similar to the original NCCN assessment as possible; however, an alternative format to a rating scale could be a number picker or text entry with specific number values. Similar to the symptoms, a list view could also be considered, although potentially less efficient if all numbers do not fit on the device screen. Alternatively, gestural signifiers can be used to demonstrate how to complete tasks. The findings of this usability study support prior research on electronic survey design, particularly with aging users, such as those older than 50 years, which should be considered when using frameworks that provide predetermined UI features. It should be noted that although important, prior work suggests that question wording does not affect usability as much as the layout [67].

Regarding the use of dark mode versus light mode in UI designs, studies have investigated how the trend of dark mode, or negative polarity, interfaces impact users [105-108]. A recent study found that light-mode interfaces are more advantageous to young and older users concerning cognitive load [106]. Considering most of our participants were aged >50 years, this could give insight into the drastic difference in usability scores between those who used Assuage in light mode and those who used Assuage in dark mode. Similarly, many patients with cancer and survivors of cancer experience cognitive effects due to cancer and its treatment [39]. Therefore, while developers of mHealth systems can implement a dark-mode interface, they must ensure that the different UI elements do not create unnecessary cognitive burden for users [107]. However, based on these preliminary results, not implementing dark mode should not have an adverse effect on our demographic of patients with cancer who are older than 50 years and from rural areas.

Limitations and Future Work

A sample size of 30 is typically considered small; however, previous research on system usability studies implies that small sample sizes, approximately 5, can capture most usability issues [85,86]. This study was also interrupted due to a spike in COVID-19 cases, which resulted in the hospital halting all nonessential and nonmedical activities, limiting our sample size. We attempted to use additional techniques during usability testing, such as a think-aloud approach; however, as patient participants were being seen between appointments, brain fog from chemotherapy treatments resulted in frustration from participants with this approach. Excluding cognitive impairment due to cancer-related treatments, the normal aging process can also cause a decline in cognitive function for older people in similar studies. Similarly, with respect to participant time, the study survey was kept as short as possible. This further supported our choice to use the SUS versus a more in-depth questionnaire, such as the Mobile Application Rating Scale [109], the Health Information Technology Usability Evaluation Scale [110], or the mHealth App Usability Questionnaire [111]. Finally, we invited health care professionals to assess Assuage; however, only 1 responded, and we did not include their SUS score in this paper.

Despite limitations, we identified areas of improvement for the interface design of surveys in mobile apps. We also determined which UIs in Assuage would be suitable for future deployment studies with our target demographic of patients with cancer who are from rural areas, older than 50 years, and may not regularly use mobile apps. Not all participants owned mobile devices, posing a potential wide-scale implementation problem. While reports show smartphone use to be consistently rising among members of the rural United States, this may not be consistent across all rural areas. Conversely, participants without mobile devices usually had other family members with mobile devices and smartphones. Most participants expressed a willingness to use an app to monitor their symptoms. However, deploying the app among rural patients in the southeastern and Appalachian regions is necessary to determine if apps are a viable solution for this demographic. In the future, we plan to conduct follow-up studies to assess adherence and reasons for engagement with Assuage to report distress symptoms of patients over time.

Conclusions

Digital implementations of validated paper-based surveys can have unexpected outcomes on the usability of the survey and an app. If a digital survey has low usability, patients could be deterred from entering information, or the data could be unreliable, limiting the tool’s effectiveness. This could also affect research findings from this method or how the clinic responds. The findings show that 67% (20/30) of the patients with cancer who participated in this pilot usability study rated the different interfaces of Assuage as above average (SUS score >68) [81]. This suggests that Apple’s health and research frameworks provide usable UIs with minimal alterations to the default interface for users older than 50 years and with limited digital literacy. The usability issues observed align with common usability problems for designing surveys. ResearchKit and CareKit can be used to reliably design a mobile app for collecting survey-based data. However, heuristics for both usability and electronic survey design should be considered when deciding how to best segment and navigate surveys and how to present important interface elements.

The main difference between the UIs was how users could navigate between the different survey sections. The interfaces that satisfied the heuristic by Nielsen regarding flexibility and efficiency of use (#7), allowed users to freely jump between survey sections nonsequentially and achieved the highest usability scores. Therefore, it can be inferred that flexible question navigation is a feature that should not be overlooked when digitizing surveys. Other ways to increase the usability of interface designs for self-reporting outcomes by patients who do not frequently use mobile apps include gestural signifiers; visual cues when scrolling is available, such as scroll indicators; minimizing scrolling per page; and a dedicated answer choice when none apply.

The findings from this paper do not aim to undermine the importance or benefits of co-design or participatory design for underserved and understudied populations but to demonstrate the possibility for successful digital implementations when resources cannot be heavily allocated to the design process. Although the UIs in the Assuage app had overall good usability, if resources and time permit, involving end users in the design process can improve the overall usability of the final product, increasing the chance for sustained use. However, for survey-based mHealth iOS apps, ResearchKit and CareKit are legitimate options for developers and researchers seeking open-source libraries with suitable interface designs to use with similar populations to this study. Participatory design is still suggested to understand key features to support users unfamiliar with smart devices and touch interfaces when assistance is not readily available. A follow-up longitudinal study deploying Assuage with end users is currently underway.