RefCheck Maintenance Notice

On Monday, December 3, 2018, from 16:00-18:00 EST, RefCheck will be undergoing maintenance. RefCheck is the process where, during copyediting, all references are extracted from the manuscript file, parsed, matched against various databases (eg, PubMed and CrossRef), and automatically corrected. For more information on RefCheck, please visit our Knowledge Base.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 23.01.18 in Vol 2, No 1 (2018): Jan-Jun

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/8202, first published Jun 13, 2017.

This paper is in the following e-collection/theme issue:

    Original Paper

    A Computer-Assisted Personal Interview App in Research Electronic Data Capture for Administering Time Trade-off Surveys (REDCap): Development and Pretest

    1School of Public Health and Health Systems, University of Waterloo, Waterloo, ON, Canada

    2Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada

    3Program for Health Economics and Outcome Measures, Hamilton, ON, Canada

    4Centre for Evaluation of Medicines, St. Joseph’s Healthcare Hamilton, Hamilton, ON, Canada

    Corresponding Author:

    Mark Oremus, PhD

    School of Public Health and Health Systems

    University of Waterloo

    200 University Ave W

    Waterloo, ON,

    Canada

    Phone: 1 519 888 4567 ext 35129

    Email:


    Related Article:

    This is a corrected version. See correction statement: http://formative.jmir.org/2018/2/e11436/

    ABSTRACT

    Background: The time trade-off (TTO) task is a method of eliciting health utility scores, which range from 0 (equivalent to death) to 1 (equivalent to perfect health). These scores numerically represent a person’s health-related quality of life. Software apps exist to administer the TTO task; however, most of these apps are poorly documented and unavailable to researchers.

    Objective: To fill the void, we developed an online app to administer the TTO task for a research study that is examining general public proxy health-related quality of life estimates for persons with Alzheimer’s disease. This manuscript describes the development and pretest of the app.

    Methods: We used Research Electronic Data Capture (REDCap) to build the TTO app. The app’s modular structure and REDCap’s object-oriented environment facilitated development. After the TTO app was built, we recruited a purposive sample of 11 members of the general public to pretest its functionality and ease of use.

    Results: Feedback from the pretest group was positive. Minor modifications included clarity enhancements, such as rearranging some paragraph text into bullet points, labeling the app to delineate different question sections, and revising or deleting text. We also added a research question to enable the identification of respondents who know someone with Alzheimer’s disease.

    Conclusions: We developed an online app to administer the TTO task. Other researchers may access and customize the app for their own research purposes.

    JMIR Formativ Res 2018;2(1):e3

    doi:10.2196/formative.8202

    KEYWORDS



    Introduction

    Health-related quality of life (HRQoL) is an individual’s normative perception of how disease and treatment affect their physical, functional, psychological, and social well-being [1]. HRQoL can be measured using health utility scores, which lie on an interval scale ranging from 0 (equivalent to death) to 1 (equivalent to full health).

    Numerous methods exist to elicit health utility scores [2,3]. One method is the time trade-off (TTO) method [4], which has three variants: conventional [4], lead-time [5], and composite [6]. Regardless of the variant, the basic premise of TTO remains the same: individuals read a brief description of a health state and choose between two (usually hypothetical) outcomes (ie, Life A or Life B). For example, Life A involves living in a full health state for x years, followed by death. Life B involves living in the health state in question for a fixed number of years (t, usually 10 years), followed by death. The lead-time and composite TTO approaches involve adding additional years of life in full health (l) to both Life A and Life B [3].

    After being presented with Life A and Life B, individuals may choose Life A, Life B, or they can say that the two options are equivalent. When individuals choose Life A or B, they are presented with a second iteration where t in Life B is held constant, but x in Life A is varied to reflect a changing length of time in full health. Once again, individuals are asked to choose between Life A or B, or equivalence. Subsequent iterations involve further variations of x in Life A, while t in Life B is always held constant. The iterations continue until individuals indicate that Life A and Life B are equivalent, that is, they are indifferent between the two choices. Once the point of indifference is known, researchers use a formula to calculate the health utility score. The formula differs according to the TTO variant [4-6].

    Early protocols called for the TTO task to be administered using a paper-and-pencil format in face-to-face interviews [7]. Interviewers used props such as cards, colored envelopes, and custom-made boards with sliding scales to depict health states and the number of years spent in Life A and Life B. The sliding scales permitted interviewers to visually illustrate differences between the various iterations of Life A and Life B.

    To standardize the TTO task and ease the burden of using multiple props to administer the activity, researchers developed computer-assisted personal interview (CAPI) software such as U-titer [8] and iMPACT3 [9] to replace the paper-and-pencil format. The EuroQol Group currently uses CAPI software (EuroQol Valuation Technology [EQ-VT]) [10-12] to administer the TTO task in studies designed to value health states for the EQ-5D [13,14] (a 5-item questionnaire created to measure HRQoL). Despite the documented existence of these three software apps, U-titer and iIMPACT3 do not appear to be available any longer. The EQ-VT’s technical specifications have not been published, and the software does not appear to be available on the EuroQol Group’s website [15].

    The Need for a Computer-Assisted Personal Interview Time Trade-Off App

    Our interest in TTO emerges from research examining whether the general public can act as a proxy and estimate HRQoL in place of persons with Alzheimer’s disease (AD) [16]. We plan to ask members of the general public to read descriptions of the mild, moderate, and severe health states of AD. Subsequently, they will perform the TTO task for each health state and generate health utility scores for each state. The health utility scores from the general public will be compared to a set of similarly elicited scores from persons with AD. We have already established that the general public can use health state descriptions to discriminate between mild, moderate, and severe AD [16].

    To enable the interviewing of participants in multiple locations, we decided the TTO task should be administered using CAPI rather than paper-and-pencil. We wanted interviewers to carry only a laptop between locations, instead of bulky sets of forms, props, and writing materials.

    Prior to developing our own CAPI app, we reviewed the literature to assess whether existing software could serve our purpose. We found 13 TTO studies that reported using CAPI software during in-person interviews: 7 studies used the EQ-VT [11,12,17-21] and 6 did not describe the specific software [22-27]. Of these 6 studies, one [23] cited a previous study [28] as the source of the CAPI software. The previous study contained screenshots of the software but no technical details. Another 2 of the 6 studies reported using hard-copy visual aids to conduct the TTO task, with CAPI software reserved for data entry only [24,25]. A total of 22 other studies used Internet-based CAPI software, where participants logged onto websites and completed the TTO task on their own, without an interviewer present [22,29-49]. Five [30-34] of these 22 studies provided screenshots of the software, but none of the 22 studies described the technical details of the software.

    The lack of information on existing CAPI software led us to develop our own Web-based app with a point-and-click interface to administer the TTO task. This paper describes the technical details and pretest of the app. At the outset, we specified two prerequisites for the app. First, the mild, moderate, and severe health states of AD would have to be presented to study participants in one of six random orders to help minimize biases due to ordering effects. This is because we will ask members of the general public and persons with AD to perform the TTO task for all three AD health states. Second, the app would start the lead-time TTO iterations at a value of 20 years in perfect health followed by death for Life A (x=10; l=10). The static Life B comparator would be 10 years in perfect health, followed by 10 years in the AD health state in question (t=10), and then death. Multimedia Appendix 1 shows the iterations in our software. Health utility scores would be calculated using the following formula: (x – l)/t, with x being the amount of time in perfect health (Life A) at the point of indifference between Life A and Life B. The formula permits the calculation of health utility scores less than 0, which represent health states worse than death (eg, persistent vegitative state).

    Our TTO app is still in its formative stages. The content of the app is based on established lead-time TTO methods [5] that have been implemented in a large valuation study conducted by one of the co-authors (FX) [50]. We report on the development and initial pretest results for the app, which is Web-based and freely available to all potential users.


    Methods

    We used the Research Electronic Data Capture (REDCap) app [51] to build the TTO app. REDCap is a secured Web-based app that allows researchers to create CAPI apps for collecting, storing, and manipulating research data. REDCap is hosted on the server of the University of Waterloo, and we selected it as the preferred platform because of the following features: (1) minimal programming requirements, (2) easy accessibility to the TTO app across academic departments and institutions (at the time of writing, REDCap has 1911 active institutional partners), (3) flexibility in defining user privileges, such as limited/temporary privileges, (4) real-time data format/range validation, (5) data export options for common statistical software packages (eg, SPSS, SAS, R), (6) software and support for no charge, (7) modest hardware and software requirements, and (8) availability as a mobile app.

    The TTO app was designed in a modular format. Its overall structure is presented in Figure 1. The app’s modular structure and REDCap’s object-oriented environment facilitate the app’s ready customization to meet researchers’ needs (Figure 2). In addition to accessibility through the project home page, the TTO-CAPI eXtensible markup language (XML) file and installation manual are included as Multimedia Appendices 2 and 3 to guarantee the availability of the software in the future. The detailed methodology to construct the TTO app follows.

    Time Trade-Off Software

    As shown in Figure 1, there are four data collection instruments in the TTO app:

    1. Preliminary information: demographic data (ie, age, gender, education level, income level), knowing anyone with AD, and EQ-5D-5L (5L indicates 5 response options per question); this instrument assigns a unique identification number to each record (study participant) in the background
    2. Mild AD: Mild AD TTO data; EQ-5D-5L data (participants answer the EQ-5D-5L as if they had AD according to the health state description just read [all three health states])
    3. Moderate AD: Moderate AD TTO data; EQ-5D-5L data
    4. Severe AD: Severe AD TTO data; EQ-5D-5L data

    In each of the three AD instruments (numbers 2-4 above), questions appear in an order based on the end-user responses to TTO iterations shown in Multimedia Appendix 1. In REDCap, each singular data entry is called a field. Various types of project fields were used to house questions and then combined in sequence one after the other to form the instruments. The following project fields were used to design the instruments: text box, multiple-choice radio buttons, multiple-choice drop-down list, yes-no, descriptive text, and calculated field. Hypertext markup language (HTML) tags were used to customize the appearance of the text in each of the instruments.

    Figure 1. Modular structure of the time trade-off (TTO) app. AD: Alzheimer’s disease; EQ-5D: EuroQol 5-dimension.
    View this figure
    Figure 2. Workflow for accessing the time trade-off (TTO) app. REDCap: Research Electronic Data Capture.
    View this figure
    Figure 3. Six possible navigation options between Alzheimer's disease (AD) instruments based on mode.
    View this figure

    We employed two logic options in the TTO app:

    1. Survey queue: a set of navigation conditions was defined to ensure the three TTO tasks would appear in a uniformly random sequence (Figure 3). Table 1 shows the conditions fixed for each instrument. We used an “auto start” feature to enable smooth and immediate navigation between modules. Uniform randomization was done by defining a variable named “mod_6” in the Preliminary information instrument. This variable could take a value from 1-6 and represent one of the six possible orders of AD tasks shown in Figure 3. The completion of a task was monitored by a flag named “part i _complete”, where i is the task number that can take a value from 1-3 for mild, moderate, and severe AD, respectively. The following values were used for the flag status: Complete=2; Incomplete=0.
    2. Branching logic: used to (1) control the order of questions in the TTO task, (2) initiate logical error check pop-up texts (eg, in situations when users choose life with AD as being more desirable than a state of full health; see Figure 4), and (3) demonstrate users’ progression toward completing the survey.
    Table 1. Survey queue conditions for Alzheimer’s disease (AD) instruments.
    View this table

    The end-user can get access to the survey through a public uniform resource locator (URL) or quick response (QR) code. Researchers can export the collected data from all instruments at once without being affected by the randomized order of the TTO tasks. Researchers can also export the data from specific instruments or fields separately (Figure 5).

    For building the TTO app, REDCap Version 6.12.0 was installed on the isysign webserver, which provides high security for hosted projects. We assigned different levels of data access rights to research team members, who had access to the project only through their individual usernames and passwords.

    One of the issues in the design of the TTO app was the possibility of incorrect data entry in ordinary text fields by the end-user. This problem was avoided using the validation option in each field. This option ensures end-users can enter only free-form text in predefined formats. Otherwise, a pop-up alert will inform the end-user of the correct format (Figure 6). This option is not only available for the format of the data but can also be used to minimize the problem of outliers by defining a valid range for numeric free-text fields such as age.

    Before publishing the app for pretesting, the research team tested the accuracy and reliability of the app’s iteration and randomization sequences. The team also reviewed the user interface from various perspectives, including consistency of appearance, clarity of cues, and simplicity of user engagement.

    Pretest

    Following the development of the software, members of the research team recruited a purposive sample of the general public to provide quality assurance feedback on the survey. These pretest participants were recruited from the researchers’ social networks (eg, friends, family, landlords). None of the pretest participants had an academic background. Each participant met with an interviewer for a one-on-one interview, during which the interviewer explained the purpose of the pretest, that is, to get feedback on the comprehensibility of the survey questions and the TTO task, as well as to assess the appearance and functionality of the online survey. The interviewers gave participants control over a laptop, and the participants completed the survey at their own pace.

    Figure 4. Screenshot of a logical error check for the pop-up text.
    View this figure
    Figure 5. Record export dashboard.
    View this figure
    Figure 6. Screenshot of a pop-up alert for incorrect free-text entry by end-user for email address.
    View this figure

    The pretests followed the method of cognitive interviewing [52]. As participants conducted the TTO task for the first AD health state, the interviewers asked them a series of scripted questions about whether they understood the description of the health state and the difference between Life A and Life B. After completing the first TTO task, the participants were asked to describe what they had just done using their own words. The interviewers encouraged participants to verbally express any thoughts that came to mind (ie, think aloud [53]) as they performed the TTO task for the remaining two health states. After participants finished the survey, they were asked to comment on the similarities and differences between the three health state descriptions. The interviewers also elicited feedback about the following components of the survey: the visual aids used to depict Life A and Life B, the appearance of the webpages and fonts, and the wording used to articulate the questions. After aggregating and discussing the pretest feedback, the researchers revised the survey.

    We considered the pretest participants to be part of the research team. While these individuals had to complete the survey, they were told at the outset that their thoughts about the survey were of interest, not their responses to specific survey questions. The interviewers informed participants that their responses to specific survey questions would not be used in later research, nor would these responses be published anywhere.

    Ethics

    Research ethics approval was obtained from the University of Waterloo’s Office of Research Ethics (study #21461) and the Hamilton Integrated Research Ethics Board.


    Results

    We pretested the TTO app on 11 members of the general public who had not previously seen the app or responded to a TTO survey. Nine pretest participants were female, the median age of the 11 participants was 46 years (range 21-59 years), and none of the participants had a university degree. A member of the research team (GPM) and a student volunteer who was trained by the research team conducted the pretests. The pretest interviewers summarized their findings, which were discussed by the research team. The findings and discussion led to six modifications of the TTO app:

    1. To enhance readability, the scenarios describing the mild, moderate, and severe AD health states were rewritten in bullet-point form, rather than in paragraph form.
    2. To assess whether participants’ responses to the TTO task might be influenced by their experiences with AD, a question about knowing someone with AD was inserted into the Preliminary information instrument.
    3. To help participants identify different parts of the survey, the scenarios were labeled as Description 1, 2, or 3.
    4. To enhance the empathy of the TTO task, a phrase comparing life with AD to a state worse than death was deleted from the app.
    5. To enrich the descriptions of Life A and Life B, the text was modified to indicate that death will occur from natural causes after the specified number of years spent in full health or with AD.
    6. To enhance comprehensibility, a phrase in the TTO response options was changed from “Life A and Life B are the same” to “Life A and Life B are equivalent”.

    The pretest participants reported understanding the substantive differences between scenarios. One participant found the need to value Life A in comparison to Life B as cognitively challenging, though this was not an issue with the online app but with her comprehension of the TTO task itself. The iterations functioned as described in Multimedia Appendix 1.


    Discussion

    Principal Considerations

    We will use the TTO app in a research study that examines whether the general public can provide proxy HRQoL estimates in place of persons with AD. Due to cognitive impairment, many persons with AD are unable to estimate their own HRQoL using instruments such as the EQ-5D [54,55]. Family caregivers sometimes act as proxies, but they often integrate their own life experiences (eg, the burdens and stresses of caregiving) into the proxy assessments and underestimate the HRQoL of their loved ones with AD [56-62]. To the best of our knowledge, no one has examined whether the general public can provide a better set of proxy HRQoL estimates than caregivers in AD.

    The TTO app is publicly available for other researchers to adapt to their own studies. As described above, existing CAPI software to conduct the TTO task is poorly documented in the literature and not readily available for researchers to access. Our app fills a gap for researchers who require a means of administering the TTO task in their studies.

    The pretest interviewers found the app worked well with regard to presenting all of the information necessary to conduct the TTO task, that is, the scenarios describing the three AD health states, the diagrammatic representations of Life A and Life B, and the questions about participants’ preferences for Life A or Life B. Additionally, the app worked quickly on a variety of public (eg, university) and private (eg, personal residence) wireless Internet connections. We did not notice any difference in performance on laptops running Windows, OS X, or Linux operating systems.

    The use of laptops provides flexibility in conducting interviews. Interviewers can turn the laptops over to participants and allow the TTO task to be performed in a completely self-administered fashion. Interviewers can also operate the point-and-click interface to record participants’ responses and help them navigate the task. This versatility is especially important when conducting research in populations that may be less familiar with technology (eg, seniors) or that may experience challenges in using technology (eg, persons with AD).

    The use of REDCap is timely given other developments in the HRQoL field. Recently, the EuroQoL Group released REDCap versions of the EQ-5D in over 50 languages to facilitate the collection of HRQoL data [63].

    The TTO app described in this manuscript will facilitate the conduct of high-quality cost-utility analyses, which are undertaken to assess the costs and benefits of new health technologies. The health utility scores generated from the TTO task are an essential ingredient of these analyses. Cost-utility analyses are increasing in frequency and can influence public sector treatment reimbursement decisions and health program priority setting [64].

    Limitations

    The pretest sample was one of convenience and may therefore be unrepresentative of average members of the general public. However, the pretest specifically excluded people with academic backgrounds to enable us to collect feedback on the app from individuals who would reflect a variety of lay opinions. Also, cognitive interviewing methods do not require the recruitment of representative samples nor do they specifically call for the use of formal qualitative analytical approaches [65,66]. In this research, we did not focus on analyzing pretest participants’ specific health utilities because the objective of the work was to develop and test an online TTO app. We have already shown in previous research that members of the general public can differentiate between AD health states [16].

    The TTO app was developed to meet the needs of a specific research protocol. Therefore, the app does not contain multiple means of eliciting health utilities (eg, visual analogue scale or standard gamble approach [67]) nor does it include multiple methods of iterating between Life A and Life B (eg, titration, bisection [3]). Some users may find REDCap’s graphic capabilities uncustomizable to their needs.

    The TTO app is still preliminary and may undergo further refinement as we move forward with our study of general public proxy HRQoL estimates in AD. However, our work demonstrates that researchers can easily use REDCap to computerize the TTO task and create a useful online CAPI app.

    Conclusions

    We developed an online app to administer the TTO task. Other researchers may access and alter the app for their own research purposes. Our app fills an operational gap in eliciting health utility scores because existing TTO apps are poorly documented and inaccessible to researchers. Researchers may contact us for information on how to use the app in their projects.

    Acknowledgments

    We wish to thank the pretest participants for their helpful comments on the TTO survey and also thank Vanessa De Rubeis for administering pretest interviews.

    The Canadian Institutes of Health Research funded this study (IAO - 126567).

    Authors' Contributions

    MO and FX conceived the purpose of the TTO survey and supervised the CAPI software design. AS programmed the TTO survey and performed ongoing technical tests and bug fixing. GPM and AS assessed the initial suitability of the REDCap app. GPM and XJ reviewed the CAPI software for functionality (eg, randomization, skip patterns). GPM administered pretest interviews. MO and FX alpha tested the software. MO led the writing of the manuscript, and all authors contributed to the writing of the manuscript. All authors read and approved the final manuscript.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Time trade-off iteration.

    PDF File (Adobe PDF File), 25KB

    Multimedia Appendix 2

    TTO-CAPI XML file.

    PDF File (Adobe PDF File), 10MB

    Multimedia Appendix 3

    TTO-CAPI installation on local REDCapTM server (manual).

    PDF File (Adobe PDF File), 189KB

    References

    1. International Society for Quality of Life Research. Health-Related Quality of Life Research   URL: http://www.isoqol.org/about-isoqol/what-is-health-related-quality-of-life-research [accessed 2017-06-14] [WebCite Cache]
    2. Drummond M, Sculpher M, Claxton K. Methods for the Economic Evaluation of Health Care Programmes. Fourth Edition. Oxford, UK: Oxford University Press; 2015.
    3. Oppe M, Rand-Hendriksen K, Shah K, Ramos-Goñi JM, Luo N. EuroQol Protocols for Time Trade-Off Valuation of Health Outcomes. Pharmacoeconomics 2016 Oct;34(10):993-1004 [FREE Full text] [CrossRef] [Medline]
    4. Torrance GW, Thomas WH, Sackett DL. A utility maximization model for evaluation of health care programs. Health Serv Res 1972;7(2):118-133 [FREE Full text] [Medline]
    5. Robinson A, Spencer A. Exploring challenges to TTO utilities: valuing states worse than dead. Health Econ 2006 Apr;15(4):393-402. [CrossRef] [Medline]
    6. Janssen BMF, Oppe M, Versteegh MM, Stolk EA. Introducing the composite time trade-off: a test of feasibility and face validity. Eur J Health Econ 2013 Jul;14 Suppl 1:S5-13. [CrossRef] [Medline]
    7. Gudex C. Time Trade-off User Manual: Props and Self-Completion Methods.   URL: http://www.york.ac.uk/media/che/documents/papers/occasionalpapers/CHE [WebCite Cache]
    8. Sumner W, Nease R, Littenberg B. U-titer: a utility assessment tool. Proc Annu Symp Comput Appl Med Care 1991:701-705 [FREE Full text] [Medline]
    9. Lenert LA, Sturley A, Watson ME. iMPACT3: Internet-based development and administration of utility elicitation protocols. Med Decis Making 2002;22(6):464-474. [CrossRef] [Medline]
    10. Devlin NJ, Krabbe PFM. The development of new research methods for the valuation of EQ-5D-5L. Eur J Health Econ 2013 Jul;14 Suppl 1:S1-S3. [CrossRef] [Medline]
    11. Shah K, Mulhern B, Longworth L, Janssen MFB. An Empirical Study of Two Alternative Comparators for Use in Time Trade-Off Studies. Value Health 2016 Jan;19(1):53-59 [FREE Full text] [CrossRef] [Medline]
    12. Oppe M, Devlin NJ, van HB, Krabbe PFM, de CF. A program of methodological research to arrive at the new international EQ-5D-5L valuation protocol. Value Health 2014 Jun;17(4):445-453 [FREE Full text] [CrossRef] [Medline]
    13. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res 2011 Dec;20(10):1727-1736 [FREE Full text] [CrossRef] [Medline]
    14. EuroQol Group. EuroQol--a new facility for the measurement of health-related quality of life. Health Policy 1990 Dec;16(3):199-208. [Medline]
    15. EuroQol Group. What is EQ-5D?.   URL: http://www.euroqol.org/home.html [WebCite Cache]
    16. Oremus M, Xie F, Pullenayegum E, Gaebel K. Can the general public use vignettes to discriminate between Alzheimer's disease health states? BMC Geriatr 2016 Feb 03;16:36 [FREE Full text] [CrossRef] [Medline]
    17. Ramos-Goñi JM, Pinto-Prades J, Oppe M, Cabasés JM, Serrano-Aguilar P, Rivero-Arias O. Valuation and Modeling of EQ-5D-5L Health States Using a Hybrid Approach. Med Care 2017 Jul;55(7):e51-e58. [CrossRef] [Medline]
    18. Sayah FA, Bansback N, Bryan S, Ohinmaa A, Poissant L, Pullenayegum E, et al. Determinants of time trade-off valuations for EQ-5D-5L health states: data from the Canadian EQ-5D-5L valuation study. Qual Life Res 2016 Dec;25(7):1679-1685. [CrossRef] [Medline]
    19. Shah KK, Lloyd A, Oppe M, Devlin NJ. One-to-one versus group setting for conducting computer-assisted TTO studies: findings from pilot studies in England and the Netherlands. Eur J Health Econ 2013 Jul;14 Suppl 1:S65-S73 [FREE Full text] [CrossRef] [Medline]
    20. Wang P, Li MH, Liu GG, Thumboo J, Luo N. Do Chinese have similar health-state preferences? A comparison of mainland Chinese and Singaporean Chinese. Eur J Health Econ 2015 Nov;16(8):857-863. [CrossRef] [Medline]
    21. Luo N, Li M, Stolk EA, Devlin NJ. The effects of lead time and visual aids in TTO valuation: a study of the EQ-VT framework. Eur J Health Econ 2013 Jul;14 Suppl 1:S15-S24 [FREE Full text] [CrossRef] [Medline]
    22. Mulhern B, Bansback N, Brazier J, Buckingham K, Cairns J, Devlin N, et al. Preparatory study for the revaluation of the EQ-5D tariff: methodology report. Health Technol Assess 2014 Feb;18(12):1-191 [FREE Full text] [CrossRef] [Medline]
    23. Marcellusi A, Capone A, Favato G, Mennini FS, Baio G, Haeussler K, et al. Health utilities lost and risk factors associated with HPV-induced diseases in men and women: the HPV Italian collaborative study group. Clin Ther 2015 Jan 01;37(1):156-167. [CrossRef] [Medline]
    24. Scalone L, Cortesi PA, Ciampichini R, Belisari A, D'Angiolella LS, Cesana G, et al. Italian population-based values of EQ-5D health states. Value Health 2013;16(5):814-822 [FREE Full text] [CrossRef] [Medline]
    25. Wittrup-Jensen KU, Lauridsen J, Gudex C, Pedersen KM. Generation of a Danish TTO value set for EQ-5D health states. Scand J Public Health 2009 Jul;37(5):459-466. [CrossRef] [Medline]
    26. Mulhern B, Shah K, Janssen MFB, Longworth L, Ibbotson R. Valuing Health Using Time Trade-Off and Discrete Choice Experiment Methods: Does Dimension Order Impact on Health State Values? Value Health 2016;19(2):210-217 [FREE Full text] [CrossRef] [Medline]
    27. Schmitt J, Meurer M, Klon M, Frick KD. Assessment of health state utilities of controlled and uncontrolled psoriasis and atopic eczema: a population-based study. Br J Dermatol 2008 Feb;158(2):351-359. [CrossRef] [Medline]
    28. Mennini FS, Panatto D, Marcellusi A, Cristoforoni P, De VR, Di CE, et al. Time trade-off procedure for measuring health utilities loss with human papillomavirus-induced diseases: a multicenter, retrospective, observational pilot study in Italy. Clin Ther 2011 Aug;33(8):1084-1095.e4. [CrossRef] [Medline]
    29. Goodwin E, Green C, Spencer A. Estimating a Preference-Based Index for an Eight-Dimensional Health State Classification System for Multiple Sclerosis. Value Health 2015 Dec;18(8):1025-1036 [FREE Full text] [CrossRef] [Medline]
    30. Norman R, King MT, Clarke D, Viney R, Cronin P, Street D. Does mode of administration matter? Comparison of online and face-to-face administration of a time trade-off task. Qual Life Res 2010 May;19(4):499-508. [CrossRef] [Medline]
    31. Prosser LA, Payne K, Rusinak D, Shi P, Uyeki T, Messonnier M. Valuing health across the lifespan: health state preferences for seasonal influenza illnesses in patients of different ages. Value Health 2011 Jan;14(1):135-143 [FREE Full text] [CrossRef] [Medline]
    32. Robinson A, Gyrd-Hansen D, Bacon P, Baker R, Pennington M, Donaldson C, EuroVaQ Team. Estimating a WTP-based value of a QALY: the 'chained' approach. Soc Sci Med 2013 Sep;92:92-104. [CrossRef] [Medline]
    33. Szende A, Brazier J, Schaefer C, Deuson R, Isitt JJ, Vyas P. Measurement of utility values in the UK for health states related to immune thrombocytopenic purpura. Curr Med Res Opin 2010 Aug;26(8):1893-1903. [CrossRef] [Medline]
    34. Viney R, Norman R, Brazier J, Cronin P, King MT, Ratcliffe J, et al. An Australian discrete choice experiment to value eq-5d health states. Health Econ 2014 Jun;23(6):729-742. [CrossRef] [Medline]
    35. Evans M, Jensen HH, Bøgelund M, Gundgaard J, Chubb B, Khunti K. Flexible insulin dosing improves health-related quality-of-life (HRQoL): a time trade-off survey. J Med Econ 2013 Nov;16(11):1357-1365. [CrossRef] [Medline]
    36. Evans M, Khunti K, Mamdani M, Galbo-Jørgensen CB, Gundgaard J, Bøgelund M, et al. Health-related quality of life associated with daytime and nocturnal hypoglycaemic events: a time trade-off survey in five countries. Health Qual Life Outcomes 2013 Jun 03;11:90 [FREE Full text] [CrossRef] [Medline]
    37. Hutchins R, Viera AJ, Sheridan SL, Pignone MP. Quantifying the utility of taking pills for cardiovascular prevention. Circ Cardiovasc Qual Outcomes 2015 Mar;8(2):155-163 [FREE Full text] [CrossRef] [Medline]
    38. Lieu TA, Ortega-Sanchez I, Ray GT, Rusinak D, Yih WK, Choo PW, et al. Community and patient values for preventing herpes zoster. Pharmacoeconomics 2008;26(3):235-249. [Medline]
    39. Lieu TA, Ray GT, Ortega-Sanchez IR, Kleinman K, Rusinak D, Prosser LA. Willingness to pay for a QALY based on community member and patient preferences for temporary health states associated with herpes zoster. Pharmacoeconomics 2009;27(12):1005-1016. [CrossRef] [Medline]
    40. Lloyd JC, Yen T, Pietrobon R, Wiener JS, Ross SS, Kokorowski PJ, et al. Estimating utility values for vesicoureteral reflux in the general public using an online tool. J Pediatr Urol 2014 Dec;10(6):1026-1031 [FREE Full text] [CrossRef] [Medline]
    41. Olofsson S, Norrlid H, Persson U. Preferences for improvements in attributes associated with basal insulin: a time trade-off and willingness-to-pay survey of a diabetic and non-diabetic population in Sweden. J Med Econ 2016 Oct;19(10):945-958. [CrossRef] [Medline]
    42. Polster M, Zanutto E, McDonald S, Conner C, Hammer M. A comparison of preferences for two GLP-1 products--liraglutide and exenatide--for the treatment of type 2 diabetes. J Med Econ 2010;13(4):655-661. [CrossRef] [Medline]
    43. Prosser LA, Payne K, Rusinak D, Shi P, Messonnier M. Using a discrete choice experiment to elicit time trade-off and willingness-to-pay amounts for influenza health-related quality of life at different ages. Pharmacoeconomics 2013 Apr;31(4):305-315. [CrossRef] [Medline]
    44. Shiroiwa T, Fukuda T, Ikeda S, Shimozuma K. QALY and productivity loss: empirical evidence for “double counting”. Value Health 2013 Jun;16(4):581-587 [FREE Full text] [CrossRef] [Medline]
    45. Tilling C, Krol M, Attema AE, Tsuchiya A, Brazier J, van EJ, et al. Exploring a new method for deriving the monetary value of a QALY. Eur J Health Econ 2016 Sep;17(7):801-809. [CrossRef] [Medline]
    46. Tilling C, Krol M, Tsuchiya A, Brazier J, Exel JV, Brouwer W. Does the EQ-5D reflect lost earnings? Pharmacoeconomics 2012 Jan;30(1):47-61. [CrossRef] [Medline]
    47. Trent M, Lehmann HP, Qian Q, Thompson CB, Ellen JM, Frick KD. Adolescent and parental utilities for the health states associated with pelvic inflammatory disease. Sex Transm Infect 2011 Dec;87(7):583-587. [CrossRef] [Medline]
    48. van Nooten FE, Koolman X, Brouwer WBF. The influence of subjective life expectancy on health state valuations using a 10 year TTO. Health Econ 2009 May;18(5):549-558. [CrossRef] [Medline]
    49. Versteegh MM, Attema AE, Oppe M, Devlin NJ, Stolk EA. Time to tweak the TTO: results from a comparison of alternative specifications of the TTO. Eur J Health Econ 2013 Jul;14 Suppl 1:S43-S51 [FREE Full text] [CrossRef] [Medline]
    50. Xie F, Pullenayegum E, Gaebel K, Bansback N, Bryan S, Ohinmaa A, Canadian EVSG. A time trade-off-derived value set of the EQ-5D-5L for Canada. Med Care 2016 Jan;54(1):98-105 [FREE Full text] [CrossRef] [Medline]
    51. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
    52. Willis G. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications; 2005.
    53. Presser S, Couper M, Lessler J, Martin E, Martin J, Rothgeb JM, et al. Methods for Testing and Evaluating Survey Questions. Public Opinion Quarterly 2004 Apr 22;68(1):109-130. [CrossRef]
    54. Crespo M, Hornillos C, Gómez MM. Assessing quality of life of nursing home residents with dementia: feasibility and limitations in patients with severe cognitive impairment. Int Psychogeriatr 2013 Oct;25(10):1687-1695. [CrossRef] [Medline]
    55. Woods R, Nelis S, Martyr A, Roberts J, Whitaker CJ, Markova I, et al. What contributes to a good quality of life in early dementia? Awareness and the QoL-AD: a cross-sectional study. Health Qual Life Outcomes 2014 Jun 11;12:94 [FREE Full text] [CrossRef] [Medline]
    56. Naglie G, Hogan DB, Krahn M, Black SE, Beattie BL, Patterson C, et al. Predictors of family caregiver ratings of patient quality of life in Alzheimer disease: cross-sectional results from the Canadian Alzheimer's Disease Quality of Life Study. Am J Geriatr Psychiatry 2011 Oct;19(10):891-901 [FREE Full text] [CrossRef] [Medline]
    57. Arons AM, Krabbe PF, Schölzel-Dorenbos CJ, van der Wilt GJ, Rikkert MG. Quality of life in dementia: a study on proxy bias. BMC Med Res Methodol 2013 Sep 06;13:110 [FREE Full text] [CrossRef] [Medline]
    58. Bosboom PR, Alfonso H, Almeida OP. Determining the predictors of change in quality of life self-ratings and carer-ratings for community-dwelling people with Alzheimer disease. Alzheimer Dis Assoc Disord 2013;27(4):363-371. [CrossRef] [Medline]
    59. Sousa MF, Santos RL, Arcoverde C, Simões P, Belfort T, Adler I, et al. Quality of life in dementia: the role of non-cognitive factors in the ratings of people with dementia and family caregivers. Int Psychogeriatr 2013 Jul;25(7):1097-1105. [CrossRef] [Medline]
    60. Conde-Sala JL, Reñé-Ramírez R, Turró-Garriga O, Gascón-Bayarri J, Juncadella-Puig M, Moreno-Cordón L, et al. Factors associated with the variability in caregiver assessments of the capacities of patients with Alzheimer disease. J Geriatr Psychiatry Neurol 2013 Jun;26(2):86-94. [CrossRef] [Medline]
    61. Schulz R, Cook TB, Beach SR, Lingler JH, Martire LM, Monin JK, et al. Magnitude and causes of bias among family caregivers rating Alzheimer disease patients. Am J Geriatr Psychiatry 2013 Jan;21(1):14-25 [FREE Full text] [CrossRef] [Medline]
    62. Black BS, Johnston D, Morrison A, Rabins PV, Lyketsos CG, Samus QM. Quality of life of community-residing persons with dementia based on self-rated and caregiver-rated measures. Qual Life Res 2012 Oct;21(8):1379-1389 [FREE Full text] [CrossRef] [Medline]
    63. EuroQoL Group. Launch of REDCap EQ-5D   URL: http://www.euroqol.org/news-list/article/launch-of-redcap-eq-5d.html [accessed 2017-06-13] [WebCite Cache]
    64. Niessen LW, Bridges J, Lau BD, Wilson RF, Sharma R, Walker DG, et al. Assessing the Impact of Economic Evidence on Policymakers in Health Care - a Systematic Review. Rockville, MD: Agency for Healthcare Research and Quality; 2012.
    65. Willis G. Analysis of the Cognitive Interview in Questionnaire Design. Oxford, UK: Oxford University Press; 2015.
    66. Willis GB, Artino AR. What Do Our Respondents Think We're Asking? Using Cognitive Interviewing to Improve Medical Education Surveys. J Grad Med Educ 2013 Sep;5(3):353-356 [FREE Full text] [CrossRef] [Medline]
    67. Wee H, Li S, Xie F, Zhang X, Luo N, Feeny D, et al. Validity, feasibility and acceptability of time trade-off and standard gamble assessments in health valuation studies: a study in a multiethnic Asian population in Singapore. Value Health 2008 Mar;11 Suppl 1:S3-10 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    AD: Alzheimer’s disease
    CAPI: computer-assisted personal interview
    EQ-5D: EuroQol 5-dimension
    EQ-5D-5L: EuroQol 5-dimension 5-level
    EQ-VT: EuroQol Valuation Technology
    HRQoL: health-related quality of life
    REDCap: Research Electronic Data Capture
    TTO: time trade-off


    Edited by G Eysenbach; submitted 13.06.17; peer-reviewed by N Luo, L Lenert; comments to author 22.11.17; revised version received 01.12.17; accepted 06.12.17; published 23.01.18

    ©Mark Oremus, Anis Sharafoddini, Gian Paolo Morgano, Xuejing Jin, Feng Xie. Originally published in JMIR Formative Research (http://formative.jmir.org), 23.01.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.