Millions of individuals depend on family caregivers to manage their care [1]. Although family caregivers are a central part of health care [2], they often are invisible in our health care system, so much so that they are sometimes referred to as “hidden patients” [3]. The economic value of unpaid hours of care by family caregivers was estimated at US $470 billion in 2013, and their contributions continue to intensify [4]. Indeed, with the aging population in the United States and the rising need for caregivers, efforts to foster caregiver health and well-being are essential for sustaining long-term care [5]. Caregivers assist patients with a wide range of activities, including managing complex medical tasks, organizing care plans, and advocating on their behalf [3]. These demands are of a time- and labor-intensive nature, and they place caregivers at high risk for injury and adverse events [3,6-8]. Addressing the needs of at-risk caregivers is an urgent public health priority [1].

Caregiver burden is defined as the “negative reaction to the impact of providing care on the caregiver’s social, occupational, and personal roles” [9]. Much focus has been placed on the wide range of negative implications associated with caregiving [10] (eg, depression and anxiety) [11]. Despite this, most caregivers have recognized the benefits of caregiving [12,13]. The imbalance of focusing primarily on negative aspects may limit our ability to develop new assessment and intervention methods [14]. Thus, a “corrective focus” is needed in caregiving research to expand our knowledge on the positive aspects of caregiving [15,16]. Research on self-management suggests that self-efficacy, a positive aspect, can promote caregiver health, well-being, and positive health behaviors (ie, improved sleep and physical activity) [17,18].

The positive aspects of caregiving may explain how caregivers can positively engage patients in self-care activities [19]. Caregivers with better self-efficacy and well-being (ie, health-related quality of life) may positively affect patients’ health outcomes [20-22]. Simple strategies aimed at enhancing positive thoughts, emotions, and behaviors have been shown to be effective and highly scalable [23-25]. Positive activity interventions, such as daily positive reflection, using gratitude journals, and conducting acts of kindness, have been used in the management of heart disease, cancer, diabetes, and chronic pain [26-29].

Blood and marrow transplant (BMT), commonly referred to as hematopoietic stem cell transplant (HSCT), is an intense but potentially curative therapy for a number of life-threatening blood diseases [30]. Given the high risk associated with BMT, a dedicated caregiver is necessary and expected for at least the first 100 days after the transplant [31]. However, HSCT caregivers are often unprepared for this role; it is not surprising that HSCT caregivers experience significant levels of anxiety and distress, especially during the peritransplant period [32,33]. Psychoeducational, skills training, and therapeutic counseling interventions have been shown to benefit caregiver health and well-being [34]. However, major barriers in translating successful interventions to clinical practice have included (1) limited understanding of the mechanism of action of an intervention and the (2) need for expert trainers, intensive training, and monitoring [3]. Interventions that are mechanism focused, low cost, and sustainable are needed [35].

We recently developed BMT Roadmap (Roadmap 1.0) as a mobile health (mHealth) app to provide patient-specific information, education, and skill-building exercises for caregivers to use during their inpatient stay. The modular components included patient-specific disease characteristics (eg, infectious disease markers, blood type, donor characteristics, and conditioning chemotherapy regimen), laboratory studies (ie, results shown in real time), medications (ie, lists of medications grouped according to indication, eg, antibiotic or antiemetic), clinical trials (ie, easy-to-read description of clinical trials and copies of informed consents), and a health care provider directory (ie, photographs of nurses, physicians, social workers, pharmacists, and nutritionists) in a yearbook style. To date, more than 100 HSCT caregivers have enrolled in institutional review board–approved studies to assess the feasibility of implementing Roadmap 1.0. Major themes that emerged from qualitative interviews conducted with users of Roadmap 1.0 included the following: (1) Roadmap 1.0’s usefulness, ease of use, and likeability; (2) positive aspects of caregiving (ie, benefits of providing care); and (3) desire to expand Roadmap 1.0 to the outpatient setting, specifically targeting “caregiver-specific resources” and “positive activities” components [36-40].

Thus, in addition to the qualitative research findings from our single institution, we sought to develop a national caregiver health survey that could be broadly distributed to a diverse sample of HSCT caregivers. The goal of the survey was to examine design considerations for an outpatient version of Roadmap 1.0 (will be referred to as Roadmap 2.0 henceforth). Specifically, our intention was to develop a useful and understandable survey aimed at HSCT caregivers as the target audience. Thus, the aim of this study was to create a methodologically rigorous, broadly national survey that would help further inform the development of the app, in addition to contributing to substantive empirical research on caregivers of patients who have undergone HSCT. To do this, we conducted cognitive interviews to assess each survey item and adjusted, iterated, and rewrote the survey thereafter, which we report herein.

This work is part of a multiphase project (Figure 1) that will develop and test Roadmap 2.0 in a randomized controlled trial.

Development of the Caregiver Health Survey was based on research derived from phases 1 to 6 [36-38,40-45]. The 6 sequential phases led to the development of a draft survey that included (1) demographics, (2) general caregiving duties and life experiences after transplant, (3) use of mobile technology (eg, mHealth apps and wearable sensors), (4) personal enrichment through positive psychology–based activities, (5) mood, (6) confidence in providing care for a loved one (patient) and self-care (self-efficacy), and (7) ability to handle stress and use coping strategies. For items 5 to 7, we incorporated the Patient Health Questionnaire [46], Caregiver Self-Efficacy Scale (CaSES) [47], and Brief Coping Orientation to Problems Experienced (COPE) inventory [48], with permission. The psychometric properties of these instruments are provided in Multimedia Appendix 1.

To ensure that survey items were clearly worded and provided contextualized understanding relevant to HSCT caregiving experiences, we conducted cognitive interviews. In short, the cognitive interview is based on the conceptual frameworks and methods in cognitive and social psychology [49]. As it is an important method used in survey development, particularly under the cognitive aspects of survey methodology (CASM), this approach was used in our survey design to ensure the quality and interpretability of question items [50]. We evaluated interviews using the 4 CASM steps that describe the question-answering process: (1) comprehension, (2) information retrieval, (3) judgment and decision making, and (4) responding [51].

During cognitive interviews, the interviewer read aloud each item and asked the participant to express any questions or concerns regarding the item. The interviewer used both think-aloud interview and verbal probing techniques. Think-aloud interviewing provided an opportunity for open-ended answers without interviewer direction. Verbal probing was used after the participant answered the think-aloud interview to provide further insight into the response [49-51]. In addition, the interviewer recorded field notes or observations from each interview session and also documented handwritten notes after reading each item out loud to the participant.

The study was approved by the institutional review board (HUM00115569). Cognitive interviews were conducted with family caregivers of patients of the BMT unit of a large academic medical center in the midwestern part of the United States. Eligibility conditions for study participation were that the subject should be (1) the primary family caregiver who had already experienced the transplant procedure with their loved one (patient) and was in the posttransplant phase of care, (2) aged ≥18 years, and (3) comfortable with reading and speaking English. Participants were recruited through referrals from the clinical team (eg, physician or advanced practitioner). The clinical team recruited caregivers who met eligibility criteria from the inpatient and outpatient settings. Only one caregiver declined participation; another caregiver signed the informed consent but was not available on the interview day. Thus, a total of 10 caregivers signed the informed consent and participated in the study.

The cognitive interviews took place between November 19, 2018, and February 7, 2019. Each interview session was approximately 30 to 50 mins in length and was audio-recorded, with permission, and subsequently professionally transcribed (Babbletype LLC). One caregiver participant refused audio-recording. Participants were compensated with a $10 gift card for their participation. A trained project manager with a background in survey methodology (Survey Research Operations, Survey Research Center, Institute of Social Research) moderated the cognitive interviews in a private hospital conference room. The interviewer was not affiliated with the BMT program. Recruitment ended once it was determined that no new data were being identified that informed the content of the survey items. Saturation was defined as a criterion for discontinuing data collection and/or analysis [52].

The analysis approach included 3 steps. First, two experts in public health and survey methodology (Survey Research Operations, Survey Research Center, Institute of Social Research), neither affiliated with the BMT program, read the audio-recorded transcripts and the observation and summary notes of each survey item independently. They generated their own notes of each survey item, met together to compare notes, and provided suggested edits (ie, changes to survey items) to the research team. Second, the research team reviewed the results, validated the interpretations and conclusions in a peer-debriefing session, and developed a revised draft survey. Third, a survey methodologist at an external survey research organization (Center for Survey Research) reviewed all of the observation and summary notes and draft survey and provided additional edits of the draft survey. All changes to survey items that led to the final survey were made in collaboration with the lead study investigator.

As shown in Table 1, the median age of the study participants was 57 years (range: 35-70 years). Most participants were recruited from the inpatient setting (7/10, 70%), female (9/10, 90%), white (9/10, 90%), married (9/10, 90%), caregivers of adult patients who had undergone HSCT (7/10, 70%), and employed at least part time (6/10, 60%) and had some college education (9/10, 90%) and an annual household income of $60,000 or higher (6/10, 60%). Detailed demographics are provided in Multimedia Appendix 2. Overall, participants were engaged in the cognitive interview process. The findings are described below per survey topic. A list of questionnaire items from which the quotes were derived is provided in Multimedia Appendix 3.

In this study, we report the findings of cognitive interviews conducted in caregivers of patients who had undergone HSCT that assessed each survey item. Following the 4 CASM steps that describe the question-answering process—(1) comprehension, (2) information retrieval, (3) judgment and decision making, and (4) responding and using the think-aloud and probing techniques—we found that this methodologically rigorous process informed revisions and improved our final questionnaire design. Indeed, evidence-based data have shown that pilot testing a survey is typically insufficient to ensure the quality and accuracy of the questionnaire [49].

Some of the participants identified confusion within certain sections of the survey that may have been missed with pilot testing alone. Interestingly, the sections that prompted the most concerns were the CaSES and Brief COPE items. For example, the wording of some of items created confusion for our participants, which may have been because of the context of care that is unique to HSCT (eg, pre-HSCT, during HSCT, post-HSCT, inpatient, outpatient, caregiver of an adult patient, or caregiver of a pediatric patient). Thus, this led to the following changes: (1) inclusion of more succinct and clear instructions in the introduction or preamble to each section; (2) incorporation of anchoring terms, such as “at the time of transplant”; and (3) inclusion of questionnaire items tailored to the HSCT population (ie, to better align the items to HSCT, we deleted some items that repeatedly raised concerns in the participants).

Importantly, the interviews revealed that the original set of items was not exhausted and highlighted areas wherein survey items could be further refined by (1) offering more response choices (eg, “NA”), (2) removing some negatively worded items, (3) including more items to make our points clear, (4) moving the order of some items so that it flowed more clearly within each topic, and (5) collapsing redundant items (eg, collapsing income brackets/categories). Thus, we found that during the course of this study, we were able to examine item interpretation and readability and adjust, refine, and rewrite items that would be understood correctly by future survey respondents. We also redesigned the formatting of certain sections by (1) creating a grid or matrix question format with x-axis that listed the item and y-axis that listed the response options, (2) developing a single-response or “radio-question” format, and (3) auto-populating or piping in the patient’s name from a previous response to further reduce the readability burden. Personal enrichment through positive activity exercises did not identify any potential problems that might lead to survey response error, and thus, no further changes were made to any of its items.

Overall, we found that participants were willing to contribute to this type of project, specifically to help future caregivers who would undergo this process, which was consistent with our prior research [53]. Participants of this study were caregivers of both adult and pediatric patients who had undergone BMT. The cognitive interviews identified areas that helped us refine language to allow for interpretability regardless of caregiver type (eg, adult or pediatric). Prior HSCT survey reports have examined either adult or pediatric HSCT [54,55]. Thus, our findings offer a unique contribution to the literature.

Conducting cognitive interviews and using techniques such as think-aloud can help us to learn how participants interpret questionnaire items in their own words, thereby facilitating the development of an instrument that is discriminating, reliable, and valid. de Leeuw et al conducted 2 recent studies [56,57] using a rigorous methodological process that included cognitive interviews in the development of an instructional design evaluation survey. Our findings herein support the use of such rigorous processes in developing surveys that verify how participants are interpreting survey items and whether the survey format and response sets are understandable.

Major strengths of the study include the development of a refined survey through rigorous methodology that involved a trained interviewer, experts in qualitative data analysis through the lens of survey methodology (and not affiliated with the BMT study population), a research team with extensive knowledge in the BMT study population, and an external survey methodology research investigator (not affiliated with our institution). Not involving survey methodology experts in our study population helped remove biases in the interpretation of our findings.

On the basis of the review of the transcript data and field notes captured by the moderator, we found that the think-aloud technique was successful in capturing constructive feedback, particularly related to the self-efficacy and coping-related items. Participants freely shared that some items were not pertinent to them, insensitively phrased, or required more response options. Verbal probing revealed items that caused confusion (ie, the participant was stuck or paused for a long time, and the interviewer posed a clarifying question or comment to identify the confusion). Overall, the probes were not directive, and participants were able to verbalize their thoughts freely and openly. In the instances that participants refused to answer an item (ie, because of the insensitive nature of the survey item or privacy concerns), the moderator did not probe further. For example, a caregiver declined to answer the 4 psychological distress items because of privacy concerns. It is possible that survey quality will improve in the future with an anonymous, self-administered survey (ie, removing the interviewer).

Despite our extensive work to develop a survey that would be reliable and interpretable, we recognize the limitations of our work. First, participants who were engaged in this research participated in our research. All but one caregiver agreed to participate when approached by the clinical care team, which may reflect social desirability to please the health care providers. In addition, this could mean that our data were skewed by selective input of those engaged in research. Most participants were white, female, married, and highly educated. We recognize that caregiving experiences could be different based on race, gender, and other identities. Our study was also conducted in a single institution in a midwestern location in the United States. The location could change the needs of a community, and a single-institution study could have reflected the interpretation of individuals attending our center. Although cognitive interviews were conducted to improve questionnaire design and to inform revisions, it is likely that some individuals may still have difficulty interpreting some items as intended, which could lead to inaccurate responses or missing data, if left unanswered.

Nonetheless, findings from our cognitive interviews were invaluable in the refinement of our final Caregiver Health Survey. In general, interventions to support caregivers longitudinally across the trajectory of care are limited. Our larger research agenda aims to contribute to the intervention literature. We hope that the findings from our study, which highlight the importance of cognitive interviews, will be useful for research investigators designing surveys with caregivers in mind, especially surveys in support of developing health interventions. Furthermore, in-depth explorations of survey items by asking caregivers about their perceptions provided them with an important opportunity to include them as active partners in the care of their loved ones (patients). Sharing these data about caregivers’ views of survey items and what their thought processes are when responding to items may also facilitate future caregiving work.

The main goal of this study was to create a survey that will inform the development of our future Roadmap 2.0 app and continue research on mHealth interventions. Data collected herein informed our national caregiver health survey, which was deployed nationally from May to June 2019 (data analyses are forthcoming). Although this investigator-initiated survey queried respondents on health behavior and use of mHealth apps generally to inform our Roadmap 2.0 app, once developed and tested, we anticipate using one of the many recent well-developed and well-validated surveys, such as the Health Information Technology Usability Evaluation Scale [58], mHealth App Usability Questionnaire [59], and/or the Mobile App Rating Scale [60], to assess the app’s usability.