This study was conducted in three sequential phases guided by the Technology Acceptance Model and user-centered design principles. The objective of phase 1 was to understand current care gaps and needs in cancer care through interviews with health care providers, survivors, and care partners. In phase 2, we gathered design and content specifications for the planned app through semistructured interviews with care partners, survivors, and clinicians. Finally, in phase 3, we created the first version of the TOGETHERCare mobile app for informal care partners to remotely monitor cancer survivors’ health and their own health and for providing care partner resources. We collected feedback from a small beta testing group of care partners who used the app for 28 days in two geographically different academic cancer centers (Duke and Stanford). This study has been registered on clinicaltrials.gov (NCT04018677). This iterative information gathering process was conducted to address three main questions. Table 1 lists the questions and our methods for addressing the questions.

We talked to stakeholders who would have insights into the needs of cancer patients, including providers, pharmaceutical company scientists, advocates, social workers, medical directors, researchers, and care partners. Stakeholders were identified first through Project Team Advisory Group (PTAG) members (N=14) and Patient Advocacy Council (PAC) members (N=4). PTAG members included survivor advocates, clinicians, and researchers with direct experience working with cancer survivors. PAC members included cancer survivors and care partners. A snowball technique was used to identify additional stakeholders, so that each informant identified additional people to interview until saturation of concepts was achieved.

The purpose of the interviews was to identify gaps in or barriers to cancer care, care partners, workflows, and the potential value of possible app components to organizations and individuals. These interviews were open-ended and did not use interview guides. Responses to the interviews were categorized by the research team [33].

From a convenience sample from the Stanford Cancer Institute and Monterey-Salinas California health care systems, two interviewers conducted nine interviews using Institutional Review Board (IRB)-approved semistructured guides. The interviews were conducted with three physicians who work with cancer survivors, three cancer survivors, and three care partners currently caring for cancer survivors. Three interviews with Spanish-speaking survivors and care partners were conducted with a medical interpreter translating for the interviewer. Notes from the interviews were transcribed into a prepared template. The results were compiled, and response concepts were coded by two members of the core team.

We conducted a beta test with cancer care partners to assess feasibility. Informed consent was collected through the app. The app included the following tabs: “Profile” for care partner’s name and demographics; “Activities” for informed consent, Health Insurance Portability and Accountability Act authorization, PRO surveys related to care partners’ health, survivor demographics, and surveys related to survivors’ health, including a targeted symptom list and a preappointment concerns survey; and “Resources” with links to resources related to care partners’ health, caregiving tasks, and local referral information.

The aim of the beta test was to test the consent and enrollment process and gather feedback from real care partners. Beta testing was also used to catch software or interoperability bugs before extensive usability testing. Cancer survivors (n=6) were recruited by staff at Stanford and Duke Universities and asked to consent to the project and identify their care partners. After providing informed consent, the care partners (n=7) (one survivor named two care partners) were enrolled in the study and instructed on how to download the app and enter data. Stanford University recruited survivors from survivor support groups, and Duke University recruited survivors from the palliative care clinic.

We conducted semistructured interviews with most beta participants at day 7 and all participants at day 28 to see what they liked and did not like about using the app. Interviews were zoom calls, typically 30 to 45 min in length, and the same script was followed in each interview, except for the first question (“Tell us about your care partner situation”) that we did not ask at the day 28 interview. Interviews provided much more in-depth information about care partners’ experiences using the app than we would have obtained using generic app satisfaction scales such as the System Usability Scale (SUS) [34,35] and mHealth App Usability Questionnaire (MAUQ) [36]. Interviews were recorded and transcribed.

Transcripts of the interviews were coded to identify themes. The thematic coding was done based on the principles of content analysis, where textual data are identified, analyzed, and grouped to form meaningful categories [37]. The level of analysis was entire sentences, as the interviews were semistructured and coding by word would give undue weight to a single respondent who had longer responses. Sentences from the interviews were coded based on the main concept contained in them, and the most commonly occurring themes across interviews were compiled.

Statistical methods for analyzing the in-app surveys included frequency counts and percentages to determine the completion rate of each survey, the number of surveys that were completed during the 4-week beta test, and the time required to complete each survey.

All procedures and study materials were independently approved by IRBs at Stanford and Duke Universities (clinicaltrials.gov NCT04018677). Because this was an unrandomized trial, no CONSORT checklist was filed. All procedures were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all participants included in the study.

The responses from the 138 stakeholders resulted in 262 responses that fell into different categories as follows: burnout/stress (n=108), cancer types/comorbid conditions (n=65), revenue/business (n=40), office value (n=22), insurance (n=12), and media/events attended by respondents (n=15). The major categories that emerged across all areas included the following: (1) survivors who come in for crisis visits are older, have comorbidities, have care partners at home who are burned out, live geographically far away, or have difficulty accessing appropriate resources and services; (2) buyers of new applications are risk sensitive or want a tool supported by validation studies proving reduction of clinical burdens and crisis visits when care partners are supported and less stressed; and (3) fitting into the electronic health record (EHR) system is important.

Responses by the 108 stakeholders who commented on burnout/stress are detailed in Table 2. Scheduling, keeping upcoming visits organized, or feeling overwhelmed with too much information related to different doctors, medications, and appointments were mentioned 27 times by most stakeholders across job functions.

The 32 care partners who were interviewed wanted an app to help them (1) be better organized and feel prepared to take care of their loved ones, including organizational support for scheduling visits, keeping up on the treatment and following treatment protocols, or knowing what signs/symptoms to look out for in order to identify adverse events earlier; (2) be more knowledgeable about various side effects, insurance options, treatment options, and other resources available to the survivor and care partner; and (3) have a sense of support and understanding of challenges from the clinical team, friends, family, and coworkers.

The demographics for the semistructured interviewees were quite varied. Physicians (n=3) were between 39 and 66 years of age and had been providing health care for those aged 14 to 39 years. There were two medical oncologists and one surgeon. Survivors (n=3) ranged in age from 43 to 74 years and had been diagnosed between 8 months and 19 years prior. Cancer diagnoses included breast cancer, melanoma, ovarian cancer, colon cancer, and thyroid cancer (one survivor had multiple diagnoses). Care partners (n=3) ranged in age from 22 to 63 years and had been care partners for a range of less than 1 year to 6 years. We interviewed two white non-Hispanic females, one white non-Hispanic male, one Asian male, three Latino females, and two Latino males.

For many of the coded interview concepts, there was general agreement across the physicians, cancer survivors, and care partners. Although all three groups agreed that there is currently no systematic way for specialists to keep in touch with survivors once they have moved to community care, survivorship care plans (SCPs) would be useful. The SCP provides treatment history, management of side effects, and information on who to contact. However, they currently do not receive or prepare an SCP. All three groups concurred that the survivor had to initiate either a visit or call to the specialist. All three groups agreed that they have smartphones and that an app including the ability to communicate between the different groups, along with other modules, such as guidance on assisting with daily medical tasks and activities of daily living, would be useful.

There were also differences between the three groups of semistructured interviewees in concepts coded for responses to four of the interview questions. Care partners and survivors had different kinds of questions they would like to ask physicians, compared with questions clinical staff felt they heard frequently from care partners. Cancer survivors indicated no concerns about the app, and one care partner mentioned a concern about keeping medical information private, but clinical staff were concerned about the added workload and whether the app would prove useful unless all members of the care team participated in the effort. Clinical staff had specific ideas for smartphone app modules, including communication tips for survivors and care partners to make better use of their time with the clinical team; a dashboard or status bar that would track the survivor through the care process, from the initial treatment through all treatments provided by different specialties; and addition of PROs or ObsROs. Survivors and care partners were interested in modules on medication, diet, self-care, reminders, and a version in Spanish.

Evidence from the beta test semistructured qualitative surveys indicated that care partners (n=7) found the app to be useful, and would continue to use it, as well as recommend it to others. They also suggested that adding features to the app, such as the ability to search for specialized information and insert open-ended notes, would greatly enhance the functionality of the app. Care partners were focused on survivor health, and were not too interested in responding to health questions about themselves; instead, they felt that most of the questions should focus on the survivor. Moreover, they requested more feedback from the daily surveys, such as an explanation of what their survivors’ health measurements indicated in terms of the survivors’ current health statuses. The four main themes that emerged from the beta test qualitative interviews are shown in Table 3.

Seven care partners participated in the beta test. The completion rates below refer to the percentage of surveys for which care partners answered all the questions. During the beta test, care partners were told that the demographics section was optional, so that was the least frequently completed survey (Stanford, 25%; Duke, 33%). The Patient Health Questionnaire-4 [38] survey about the care partner, which measures depression and anxiety in four items, was completed only 31% of the time by both groups combined, with the care partners infrequently answering any of the questions. Demographics about the survivor were completed 90% of the time, and “My loved one’s health” (about the survivor) and the preappointment survey were completed 91% of the time. All other surveys were completed between 97% and 100% of the time by both groups combined.

Within the 4-week test period, care partners were expected to answer each survey a certain number of times. Table 4 lists the survey name, the frequency with which the survey came up on the app, the number of times it was available during the 28-day beta test time frame, and the median and mean numbers of care partner beta testers (n=7) who started the survey. Care partners started surveys close to the frequency at which they were available, but not all the survey questions were completed (participants could skip any questions they did not wish to answer). The “number started” refers to the number of surveys for which at least one question was answered, regardless of whether all questions in that survey were completed. One-time surveys took on average from 32 seconds (care partner eligibility) to 211 seconds (e-consent document and signature [consent previously explained and reviewed by the clinical team]) to complete. Repeated surveys were completed, on average, in 2 minutes or less. The Patient Health Questionnaire-4 (PHQ-4) had an average completion time of 8 seconds, and the “My loved one’s health” was completed on average in 129 seconds.