Our interest in TTO emerges from research examining whether the general public can act as a proxy and estimate HRQoL in place of persons with Alzheimer’s disease (AD) [16]. We plan to ask members of the general public to read descriptions of the mild, moderate, and severe health states of AD. Subsequently, they will perform the TTO task for each health state and generate health utility scores for each state. The health utility scores from the general public will be compared to a set of similarly elicited scores from persons with AD. We have already established that the general public can use health state descriptions to discriminate between mild, moderate, and severe AD [16].

To enable the interviewing of participants in multiple locations, we decided the TTO task should be administered using CAPI rather than paper-and-pencil. We wanted interviewers to carry only a laptop between locations, instead of bulky sets of forms, props, and writing materials.

Prior to developing our own CAPI app, we reviewed the literature to assess whether existing software could serve our purpose. We found 13 TTO studies that reported using CAPI software during in-person interviews: 7 studies used the EQ-VT [11,12,17-21] and 6 did not describe the specific software [22-27]. Of these 6 studies, one [23] cited a previous study [28] as the source of the CAPI software. The previous study contained screenshots of the software but no technical details. Another 2 of the 6 studies reported using hard-copy visual aids to conduct the TTO task, with CAPI software reserved for data entry only [24,25]. A total of 22 other studies used Internet-based CAPI software, where participants logged onto websites and completed the TTO task on their own, without an interviewer present [22,29-49]. Five [30-34] of these 22 studies provided screenshots of the software, but none of the 22 studies described the technical details of the software.

The lack of information on existing CAPI software led us to develop our own Web-based app with a point-and-click interface to administer the TTO task. This paper describes the technical details and pretest of the app. At the outset, we specified two prerequisites for the app. First, the mild, moderate, and severe health states of AD would have to be presented to study participants in one of six random orders to help minimize biases due to ordering effects. This is because we will ask members of the general public and persons with AD to perform the TTO task for all three AD health states. Second, the app would start the lead-time TTO iterations at a value of 20 years in perfect health followed by death for Life A (x=10; l=10). The static Life B comparator would be 10 years in perfect health, followed by 10 years in the AD health state in question (t=10), and then death. Multimedia Appendix 1 shows the iterations in our software. Health utility scores would be calculated using the following formula: (x – l)/t, with x being the amount of time in perfect health (Life A) at the point of indifference between Life A and Life B. The formula permits the calculation of health utility scores less than 0, which represent health states worse than death (eg, persistent vegitative state).

Our TTO app is still in its formative stages. The content of the app is based on established lead-time TTO methods [5] that have been implemented in a large valuation study conducted by one of the co-authors (FX) [50]. We report on the development and initial pretest results for the app, which is Web-based and freely available to all potential users.

As shown in Figure 1, there are four data collection instruments in the TTO app:

In each of the three AD instruments (numbers 2-4 above), questions appear in an order based on the end-user responses to TTO iterations shown in Multimedia Appendix 1. In REDCap, each singular data entry is called a field. Various types of project fields were used to house questions and then combined in sequence one after the other to form the instruments. The following project fields were used to design the instruments: text box, multiple-choice radio buttons, multiple-choice drop-down list, yes-no, descriptive text, and calculated field. Hypertext markup language (HTML) tags were used to customize the appearance of the text in each of the instruments.

We employed two logic options in the TTO app:

The end-user can get access to the survey through a public uniform resource locator (URL) or quick response (QR) code. Researchers can export the collected data from all instruments at once without being affected by the randomized order of the TTO tasks. Researchers can also export the data from specific instruments or fields separately (Figure 5).

For building the TTO app, REDCap Version 6.12.0 was installed on the isysign webserver, which provides high security for hosted projects. We assigned different levels of data access rights to research team members, who had access to the project only through their individual usernames and passwords.

One of the issues in the design of the TTO app was the possibility of incorrect data entry in ordinary text fields by the end-user. This problem was avoided using the validation option in each field. This option ensures end-users can enter only free-form text in predefined formats. Otherwise, a pop-up alert will inform the end-user of the correct format (Figure 6). This option is not only available for the format of the data but can also be used to minimize the problem of outliers by defining a valid range for numeric free-text fields such as age.

Before publishing the app for pretesting, the research team tested the accuracy and reliability of the app’s iteration and randomization sequences. The team also reviewed the user interface from various perspectives, including consistency of appearance, clarity of cues, and simplicity of user engagement.

Following the development of the software, members of the research team recruited a purposive sample of the general public to provide quality assurance feedback on the survey. These pretest participants were recruited from the researchers’ social networks (eg, friends, family, landlords). None of the pretest participants had an academic background. Each participant met with an interviewer for a one-on-one interview, during which the interviewer explained the purpose of the pretest, that is, to get feedback on the comprehensibility of the survey questions and the TTO task, as well as to assess the appearance and functionality of the online survey. The interviewers gave participants control over a laptop, and the participants completed the survey at their own pace.

The pretests followed the method of cognitive interviewing [52]. As participants conducted the TTO task for the first AD health state, the interviewers asked them a series of scripted questions about whether they understood the description of the health state and the difference between Life A and Life B. After completing the first TTO task, the participants were asked to describe what they had just done using their own words. The interviewers encouraged participants to verbally express any thoughts that came to mind (ie, think aloud [53]) as they performed the TTO task for the remaining two health states. After participants finished the survey, they were asked to comment on the similarities and differences between the three health state descriptions. The interviewers also elicited feedback about the following components of the survey: the visual aids used to depict Life A and Life B, the appearance of the webpages and fonts, and the wording used to articulate the questions. After aggregating and discussing the pretest feedback, the researchers revised the survey.

We considered the pretest participants to be part of the research team. While these individuals had to complete the survey, they were told at the outset that their thoughts about the survey were of interest, not their responses to specific survey questions. The interviewers informed participants that their responses to specific survey questions would not be used in later research, nor would these responses be published anywhere.

Research ethics approval was obtained from the University of Waterloo’s Office of Research Ethics (study #21461) and the Hamilton Integrated Research Ethics Board.

We will use the TTO app in a research study that examines whether the general public can provide proxy HRQoL estimates in place of persons with AD. Due to cognitive impairment, many persons with AD are unable to estimate their own HRQoL using instruments such as the EQ-5D [54,55]. Family caregivers sometimes act as proxies, but they often integrate their own life experiences (eg, the burdens and stresses of caregiving) into the proxy assessments and underestimate the HRQoL of their loved ones with AD [56-62]. To the best of our knowledge, no one has examined whether the general public can provide a better set of proxy HRQoL estimates than caregivers in AD.

The TTO app is publicly available for other researchers to adapt to their own studies. As described above, existing CAPI software to conduct the TTO task is poorly documented in the literature and not readily available for researchers to access. Our app fills a gap for researchers who require a means of administering the TTO task in their studies.

The pretest interviewers found the app worked well with regard to presenting all of the information necessary to conduct the TTO task, that is, the scenarios describing the three AD health states, the diagrammatic representations of Life A and Life B, and the questions about participants’ preferences for Life A or Life B. Additionally, the app worked quickly on a variety of public (eg, university) and private (eg, personal residence) wireless Internet connections. We did not notice any difference in performance on laptops running Windows, OS X, or Linux operating systems.

The use of laptops provides flexibility in conducting interviews. Interviewers can turn the laptops over to participants and allow the TTO task to be performed in a completely self-administered fashion. Interviewers can also operate the point-and-click interface to record participants’ responses and help them navigate the task. This versatility is especially important when conducting research in populations that may be less familiar with technology (eg, seniors) or that may experience challenges in using technology (eg, persons with AD).

The use of REDCap is timely given other developments in the HRQoL field. Recently, the EuroQoL Group released REDCap versions of the EQ-5D in over 50 languages to facilitate the collection of HRQoL data [63].

The TTO app described in this manuscript will facilitate the conduct of high-quality cost-utility analyses, which are undertaken to assess the costs and benefits of new health technologies. The health utility scores generated from the TTO task are an essential ingredient of these analyses. Cost-utility analyses are increasing in frequency and can influence public sector treatment reimbursement decisions and health program priority setting [64].

The pretest sample was one of convenience and may therefore be unrepresentative of average members of the general public. However, the pretest specifically excluded people with academic backgrounds to enable us to collect feedback on the app from individuals who would reflect a variety of lay opinions. Also, cognitive interviewing methods do not require the recruitment of representative samples nor do they specifically call for the use of formal qualitative analytical approaches [65,66]. In this research, we did not focus on analyzing pretest participants’ specific health utilities because the objective of the work was to develop and test an online TTO app. We have already shown in previous research that members of the general public can differentiate between AD health states [16].

The TTO app was developed to meet the needs of a specific research protocol. Therefore, the app does not contain multiple means of eliciting health utilities (eg, visual analogue scale or standard gamble approach [67]) nor does it include multiple methods of iterating between Life A and Life B (eg, titration, bisection [3]). Some users may find REDCap’s graphic capabilities uncustomizable to their needs.

The TTO app is still preliminary and may undergo further refinement as we move forward with our study of general public proxy HRQoL estimates in AD. However, our work demonstrates that researchers can easily use REDCap to computerize the TTO task and create a useful online CAPI app.

We developed an online app to administer the TTO task. Other researchers may access and alter the app for their own research purposes. Our app fills an operational gap in eliciting health utility scores because existing TTO apps are poorly documented and inaccessible to researchers. Researchers may contact us for information on how to use the app in their projects.