Published on in Vol 2, No 2 (2018): Jul-Dec

Preprints (earlier versions) of this paper are available at, first published .
The iPrevent Online Breast Cancer Risk Assessment and Risk Management Tool: Usability and Acceptability Testing

The iPrevent Online Breast Cancer Risk Assessment and Risk Management Tool: Usability and Acceptability Testing

The iPrevent Online Breast Cancer Risk Assessment and Risk Management Tool: Usability and Acceptability Testing

Original Paper

1Department of Medical Oncology, Peter MacCallum Cancer Centre, Victoria, Australia

2School of Medicine, Deakin University, Geelong, Australia

3Centre for Biostatistics and Clinical Trials, Peter MacCallum Cancer Centre, Melbourne, Australia

4Centre for Medical Psychology & Evidence-Based Decision-Making, University of Sydney, Sydney, Australia

5Department of General Practice and the Centre for Cancer Research, The University of Melbourne, Melbourne, Australia

6School of Primary, Aboriginal and Rural Health Care, University of Western Australia, Perth, Australia

7Centre for Health Equity, Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, Australia

8Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, Australia

9Parkville Familial Cancer Centre, Peter MacCallum Cancer Centre, Melbourne, Australia

10The Sir Peter MacCallum Department of Oncology, The University of Melbourne, Parkville, Australia

11Department of Surgery, The University of Melbourne, Melbourne, Australia

12Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, United Kingdom

13Centre for Cancer Prevention, Wolfson Institute of Preventive Medicine, Queen Mary University of London, London, United Kingdom

Corresponding Author:

Kelly-Anne Phillips, MBBS (Hons), MD

Department of Medical Oncology

Peter MacCallum Cancer Centre

Locked Bag 1, A'Beckett Street

Victoria, 8006


Phone: 61 3 85597860

Fax:61 3 85597739


Background: iPrevent estimates breast cancer (BC) risk and provides tailored risk management information.

Objective: The objective of this study was to assess the usability and acceptability of the iPrevent prototype.

Methods: Clinicians were eligible for participation in the study if they worked in primary care, breast surgery, or genetics clinics. Female patients aged 18-70 years with no personal cancer history were eligible. Clinicians were first familiarized with iPrevent using hypothetical paper-based cases and then actor scenarios; subsequently, they used iPrevent with their patients. Clinicians and patients completed the System Usability Scale (SUS) and an Acceptability questionnaire 2 weeks after using iPrevent; patients also completed measures of BC worry, anxiety, risk perception, and knowledge pre- and 2 weeks post-iPrevent. Data were summarized using descriptive statistics.

Results: The SUS and Acceptability questionnaires were completed by 19 of 20 clinicians and 37 of 43 patients. Usability was above average (SUS score >68) for 68% (13/19) clinicians and 76% (28/37) patients. The amount of information provided by iPrevent was reported as “about right” by 89% (17/19) clinicians and 89% (33/37) patients and 95% (18/19) and 97% (36/37), respectively, would recommend iPrevent to others, although 53% (10/19) clinicians and 27% (10/37) patients found it too long. Exploratory analyses suggested that iPrevent could improve risk perception, decrease frequency of BC worry, and enhance BC prevention knowledge without changing state anxiety.

Conclusions: The iPrevent prototype demonstrated good usability and acceptability. Because concerns about length could be an implementation barrier, data entry has been abbreviated in the publicly available version of iPrevent.

JMIR Formativ Res 2018;2(2):e24



Breast cancer (BC) is a major public health problem, accounting for over 2 million cases worldwide each year [1]. In addition to population-based educational and public health policy interventions to minimize exposure to modifiable BC risk factors and optimize cancer screening, identifying women at increased risk and implementing risk-stratified, evidence-based prevention and intensified screening strategies for them is a priority [2]. Health care providers often have difficulty assessing and communicating BC risk as well as the absolute benefits and disadvantages of risk management interventions such as risk-reducing medication, surgery, and cancer screening [3,4].

Several tools exist to estimate BC risk based on personal risk factors, but none provides risk-adapted, individually-tailored, risk management information [5,6]. iPrevent was designed to help women and their health care providers, including primary care physicians (PCP), breast surgeons (BS), and genetics clinicians (GC), to assess and manage BC risk collaboratively [7]. It integrates BC risk estimation, using either the International Breast Cancer Intervention Study model or the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm model (as appropriate for the woman’s risk factors), with tailored risk management information [8-10].

iPrevent users are first given a qualitative risk estimate according to Cancer Australia definitions: average or slightly above average risk (<1.5 times population risk at that age), moderately increased risk (1.5-3 times population risk), or high risk (>3 times population risk) [11]. Women can then choose to see their risk information displayed as a percentage, a pictogram, and a graph. Women are also provided with a menu of risk management strategies appropriate to their risk category, based on Australian National Guidelines [11], with more detailed optional information about each strategy, including estimates of the absolute (rather than relative) risk reductions for each medical and surgical intervention and tailored lifestyle advice.

The aims of this pilot study of patients and their clinicians were to assess the iPrevent prototype with regard to its clinical usability and the acceptability of its content and layout and to identify potential barriers to its implementation. Exploratory aims included assessing its potential impact on patient risk perception, anxiety, BC worry, and BC prevention knowledge.

Study Setting

Stage 1 piloting was undertaken by the researchers with women who had previously received risk assessment and risk management advice at the Peter MacCallum Cancer Centre (PMCC) Breast and Ovarian Cancer Risk Management Clinic [12]. Stage 2 piloting involved PCP, BS, and GC in public hospitals and private primary care and breast and genetics clinics as well as their patients. Patients and clinicians were not selected according to their level of BC risk or prior experience with BC risk assessment.

Eligibility Criteria

Eligible patients were women aged 18-70 years with no personal history of cancer who provided written informed consent. Patients with previous risk-reducing bilateral mastectomy or major medical comorbidities were excluded. Eligible clinicians were PCP, BS, or GC with a workplace computer with Web access. English proficiency was required for all participants. This study was approved by the Human Research and Ethics Committees of the University of Melbourne and the PMCC. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Stage 1: Piloting on Patients With Prior Risk Assessment

We enrolled 10 patients from the PMCC Breast and Ovarian Cancer Risk Management Clinic. Baseline information on age, education, computer literacy [13], and both the perceived BC risk category (average, somewhat increased, or substantially increased) [11] and perceived percentage lifetime BC risk were collected. Patients then used iPrevent under the supervision of a research assistant (PW or ES). The time for data input was recorded. Patients were emailed the report in a PDF format. Two weeks after using iPrevent, they completed a questionnaire assessing usability and acceptability of iPrevent, knowledge, and psychosocial outcomes. They could review the emailed iPrevent output while answering these questions.

System Usability Scale

This 10-item instrument [14] uses a 5-point Likert rating scale from “strongly agree” to “strongly disagree” to measure product usability. It is applicable to small samples [15] and correlates well with other subjective measures of usability [16,17]. Final scores range from 0-100, and a System Usability Scale (SUS) score >68 is considered above average.

iPrevent Acceptability Questionnaire

This 9-item measure, adapted from a previous evaluation of a decision aid [18], uses Likert scales to elicit perceptions of the length, clarity, balance, and usefulness of iPrevent.

Breast Cancer Risk Perception

This single item, adapted from a study measuring the impact of genetic counseling, asks patients about their BC risk category: “average,” “somewhat increased,” or “substantially increased” [19]. Women were classified as underestimators, accurate estimators, or overestimators based on comparison with the risk estimated by iPrevent.

Breast Cancer Worry Scale

The Lerman BC worry scale is a 3-item scale. Higher scores indicate increased frequency and impact of worry [20].

Spielberger State-Trait Anxiety Inventory

The short form State-Trait Anxiety Inventory (STAI; 6 items) measures state anxiety; higher scores indicate higher anxiety [21].

Breast Cancer Prevention Knowledge

We used 16 items assessing knowledge regarding BC (11 items), risk-reducing medication (3 items), and risk-reducing mastectomy (2 items), which were adapted from published knowledge measures (see Multimedia Appendix 1) [22,23]. Although every woman was asked to answer all questions, the number of responses scored for each participant was dependent on the iPrevent-determined risk category. All average-risk women and moderate-risk women who were aged <35 years were assessed only on BC knowledge questions. Older, moderate-risk women were also assessed on risk-reducing medication questions. High-risk women were assessed on all 16 questions. The proportion of correct responses was calculated.

Stage 2: Piloting With Clinicians and Their Patients

We recruited 20 clinicians from previous focus groups [3-4] (5 BS and 3 PCP), via email invitation from KAP (1 BS and 6 GC), and through the PMCC PCP liaison officer (5 PCP). Clinicians first underwent an iPrevent “familiarization” session. Supervised by a research assistant (PW or ES), clinicians first entered data into iPrevent on 3 hypothetical patients (high, moderate, and average risk) and reviewed the iPrevent output information. On the same day, clinicians then conducted 2 mock consultations with female actors: one at high risk and the other moderate risk. Patient (actor) information was pre-entered into iPrevent, and clinicians were asked to use the iPrevent output with the actors as they might in a clinical consultation.

Clinicians were then asked to invite 3 eligible patients from their practice (either during patient appointments or via telephone prior) during the following 3 months to participate by entering their information into iPrevent prior to a consultation and attending an appointment with the clinician to receive the “output.” Patients were provided a printout of their iPrevent output via email. Clinicians recorded the amount of time spent using iPrevent.

All patients were asked to complete the same pre- and post-iPrevent assessments as in Stage 1. Clinicians completed the SUS and Acceptability questionnaires 2 weeks after recruitment of 3 patients (or 3 months after familiarization, if full patient recruitment did not occur).

Statistical Analyses

All statistical analyses were performed in R 3.2.3 (R Core Team, 2015). The planned sample size of 20 clinicians and 60 patients was based on pragmatic estimates of the numbers it was considered possible to recruit over the available time period. The purpose of the study was to assess the acceptability and usability of iPrevent for clinicians and patients and not to test hypotheses. Therefore, descriptive statistics were used to summarize the data (mean, median, and range for continuous variables and counts and percentages for categorical variables). Patient and clinician data were analyzed separately. A pairwise t test was used to assess whether the STAI score changed from pre- to post-iPrevent assessment.


We recruited 20 clinicians and 43 patients (10 for Stage 1 and 33 for Stage 2). Clinicians only recruited 33 of the planned 60 patients (planned 3 per clinician). BS (n=6) recruited 16 of a planned 18 patients, GC (n=6) recruited 14 of a planned 18 patients (1 GC moved overseas during the study and was, thus, unable to recruit her 3 planned patients), and PCP (n=8) recruited only 3 of a planned 28 patients.

Participant Characteristics

Patient characteristics are shown in Table 1. Median age was 38 years (range 21-56 years), 74% (31/42) had a university education, and 51% (22/43 were at moderate risk for BC. Clinician characteristics are shown in Table 2. Their median age was 47 years (range 28-66 years); of all clinicians, 40% (8/20) were PCP, 30% (6/20) were BS, and all but 15% (3/20) were females. The majority used computers often and rated themselves as having good computer skills.

iPrevent Data Entry and Consultation Times

Patients took a median of 15 (range 5-60) minutes to enter their risk factor data. The median time taken for clinician consultations in which iPrevent data were discussed was 20 (range 5-45) minutes.

System Usability Scale

SUS responses are summarized in Figure 1. Data were missing for 6 patients and 1 clinician who did not return the questionnaire. Overall, 76% (28/37) patients and 68% (13/19) clinicians rated iPrevent usability as above average (SUS score >68).

Table 1. Patient characteristics (N=43).
Age in yearsa, median (range)38 (21-56)
Highest level of education, n (%)a

Secondary school6 (14)

Vocational training5 (12)

University31 (74)
Use of computers at work or elsewhere, n (%)a

Often33 (79)

Sometimes8 (19)

Rarely1 (2)
Computer skills (self-reported), n (%)a

Expert9 (21)

Good29 (69)

Poor4 (10)
Sources of information on breast cancer (BC) risk in the past, n (%)a

Health professional30 (71)

Family and friends19 (45)

Internet10 (24)

Health information booklets6 (14)

Support organizations4 (10)
BC risk category estimated by iPrevent, n (%)

Average14 (33)

Moderate22 (51)

High7 (16)
Do you feel like you know what your own risk of breast cancer is?” n (%)b

Don’t know my risk12 (29)

I think I know my risk25 (61)

Confident I know my risk4 (10)

aData missing for1 patient.

bData missing for 2 patients.

Table 2. Clinician characteristics (N=20).
Age in years, median (range)47 (28-66)
Gender, n (%)

Female17 (85)

Male3 (15)
Year of graduation, n (%)

1970-19791 (5)

1980-19897 (35)

1990-19994 (20)

2000-20097 (35)

2010-20151 (5)
Specialty, n (%)

Breast surgeon6 (30)

Genetic counselor3 (15)

Geneticist2 (10)

Primary care physician8 (40)

Medical oncologist1 (5)
Number of years working in specialty, median (range)16 (1–34)
How many patients per year would you discuss breast cancer risk with? Median (range)138 (4–960)
Use of computers at work or elsewhere, n (%)

Often20 (100)

Sometimes0 (0)

Rarely0 (0)
Computer skills (self-reported), n (%)

Expert0 (0)

Good18 (90)

Poor2 (10)
Figure 1. iPrevent System Usability Scale scores for clinicians and patients.
View this figure
Table 3. iPrevent acceptability among clinicians and patients.
Acceptability assessmentClinician, n (%)Patient, n (%)
The amount of information provided isa,b

Too much0 (0)1 (3)

A little too much2 (11)3 (8)

About right17 (89)33 (89)
The length of the tool isa,b

Much too long4 (21)0 (0)

A little too long6 (32)10 (27)

About right9 (47)27 (73)
Clarity of informationa,b

Very clear8 (42)7 (19)

Mostly clear5 (26)16 (43)

About right5 (26)13 (35)

Not clear1 (5)1 (3)
Regarding cancer prevention, how balanced did the information seemb,c

Biased toward prevention3 (17)9 (24)

Completely balanced14 (78)26 (70)

Biased against prevention1 (6)2 (5)
Any of the information new to youa,b

All0 (0)1 (3)

Most0 (0)11 (30)

Some12 (63)22 (59)

None7 (37)3 (8)
How helpful with regard to making a decision about BC risk managementa,b

Very helpful9 (47)19 (51)

Somewhat helpful7 (37)13 (35)

A little helpful3 (16)5 (14)
Recommend this tool to othersa,b

Definitely12 (63)18 (49)

Probably6 (32)18 (49)

Probably not1 (5)1 (3)
How simple to navigate through the toola,b

Very easy7 (37)21 (57)

Somewhat easy11 (58)15 (41)

Not easy1 (5)1 (3)
Easy to reada,b

Very easy8 (42)22 (59)

Somewhat easy11 (58)15 (41)

an=19 clinicians because of missing data for 1 clinician.

bn=37 patients because of missing data for 6 patients.

cn=18 clinicians because of missing data for 2 clinicians.

iPrevent Acceptability Questionnaire

Table 3 shows that iPrevent was generally acceptable to study participants. Of all, 89% (17/19) clinicians and 89% (33/37) patients reported that the amount of information provided by iPrevent was “about right.” Furthermore, 53% (10/19) clinicians and 27% (10/37) patients reported that iPrevent was too long. Only 1 patient and 1 clinician reported that the information was not clear and that they would “probably not” recommend iPrevent to others.

Exploratory Endpoints

Breast Cancer Risk Perception

Of patients who completed the relevant questions before iPrevent, 40% (14/35) correctly indicated their BC risk category, but 51% (18/35) overestimated and 9% (3/35) underestimated their BC risk category. Post-iPrevent, 86% (30/35) accurately estimated their risk category, although 11% (4/35) and 3% (1/35) continued to overestimate and underestimate their risk, respectively.

Breast Cancer Worry Scale

Pre-iPrevent, 26% (11/42) women reported worrying about BC “often” or “all the time,” while 19% (7/37) women reported this after iPrevent. Regarding the impact of BC worry on mood and daily activities, 69% (29/42) patients reported a low score (1-1.5 out of 4) pre-iPrevent. When this was compared before and after iPrevent, 25% (9/36) patients reported less impact, 47% (17/36) reported no change, and 28% (10/36) reported more impact.

Spielberger State-Trait Anxiety Inventory

The mean short form STAI score (maximum 24) pre-iPrevent was 11.3 (SD 3.8) with no significant change post-iPrevent (median increase of 1, 95% CI: 0.5-2; P=.14).

Breast Cancer Prevention Knowledge

Overall BC prevention knowledge improved for all risk groups. (Multimedia Appendices 1 and 2).

This pilot study of the iPrevent prototype has found good usability and acceptability without evidence of an adverse impact on anxiety or BC worry. The observation that the 8 PCP recruited only 3 patients between them in the required 3-month period suggests that implementation of iPrevent into primary care might be substantially more challenging than implementation into the specialist setting, where recruitment of patients was much higher. Another interpretation is that the study requirements (eg, obtaining written informed consent) were onerous, especially for PCP in busy practices, and thus, the low recruitment by PCP in this study might not reflect the uptake of iPrevent in routine practice. However, as earlier focus groups had highlighted that PCP generally do not see BC risk assessment and management as being in their domain, iPrevent might be able to contribute to overcoming provider unfamiliarity and lack of confidence for this group of clinicians [3].

The prototype was considered too long by a majority of clinicians and some patients, indicating another potential barrier to implementation. Patients took a median of 15 minutes and up to 60 minutes to enter their risk factor data, and the subsequent median time taken for the clinician consultation using the iPrevent output was 20 minutes. To address this issue, we have now incorporated changes to streamline the data entry for family history. This study also highlighted the need for patients to be able to enter their data into iPrevent at home prior to a consultation.

iPrevent may improve BC risk perception given an additional 46% (16/35) patients accurately estimated their BC risk category after using iPrevent. As a higher perceived risk of BC is associated with considering medical prevention and risk-reducing surgery among high-risk women [24-26], iPrevent could become a potential behavior-modifying tool. While this pilot study provides no information about the uptake of risk management strategies after using iPrevent, this issue will be an important endpoint for future larger studies. Other studies have found that women who have access to more thorough information from genetic counselors, combined with support to make decisions, have a higher uptake of risk reduction methods [27-29]; thus, we hypothesize that iPrevent might have a similar impact.

Use of iPrevent did not appear to increase patient worry or anxiety, consistent with the literature that has found that decreased anxiety and better psychological outcomes are associated with improved accuracy of perceived risk [24,30,31]. Use of iPrevent seemed to improve BC knowledge, a recognized critical first step in helping individuals understand screening options, weigh potential benefits and risks for risk-reducing measures, and make informed decisions [32-34]. In addition, 89% (33/37) patients indicated that some or most of the information contained in iPrevent was new to them (Table 3).

This pilot had several limitations. First, the sample was small and the study did not achieve its target patient recruitment. The majority of patients were young and highly educated, so the acceptability and usability of iPrevent might differ in the general community where computer literacy might be lower. Similarly, clinicians who chose to participate could have been more highly engaged with BC risk assessment and risk management than nonparticipant clinicians. Finally, only short-term outcomes were measured, and the impact on long-term satisfaction and uptake of BC risk-reducing measures could not be determined. As a result of this study, enhancements have been made to iPrevent with the aim of further increasing acceptability and usability.


We thank community representatives from the Breast Cancer Network Australia, Ms Debbie Sandler and Ms Leslie Gilham, for their advice on consumer issues during the development of iPrevent. We also thank Dr Alexis Butler (PCP liaison, Peter MacCallum Cancer Centre) for her assistance recruiting PCP and Professor Rod Jackson for his advice during the early phases of this project. International Breast Cancer Intervention Study computations for iPrevent are provided by the Risk Web Service developed jointly by the Hughes RiskApps Group at the Massachusetts General Hospital and the BayesMendel Lab at the Dana Farber Cancer Institute. Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm computations are provided by the Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK. This research was funded by the Australian National Health and Medical Research Council (NHMRC #1064244) and by Breast Cancer Trials Australia & New Zealand Discretionary Funding (formerly Australia and New Zealand Breast Cancer Trials Group). JLH is an NHMRC Senior Principal Research Fellow. KAP is an Australian National Breast Cancer Foundation Practitioner Fellow.

Conflicts of Interest

The International Breast Cancer Intervention Study model is offered for commercial use by Cancer Research UK, and JC receives a portion of the derived royalties. All other authors of this paper have declared no conflict of interest.

Multimedia Appendix 1

Breast Cancer Prevention Knowledge Questionnaire.

PDF File (Adobe PDF File), 40KB

Multimedia Appendix 2

Results of Pre- and Post-iPrevent® Knowledge Questionnaire.

PDF File (Adobe PDF File), 118KB

  1. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018 Sep 12 [FREE Full text] [CrossRef] [Medline]
  2. Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med 2015 Feb 26;372(9):793-795 [FREE Full text] [CrossRef] [Medline]
  3. Phillips K, Steel EJ, Collins I, Emery J, Pirotta M, Mann GB, et al. Transitioning to routine breast cancer risk assessment and management in primary care: what can we learn from cardiovascular disease? Aust J Prim Health 2016;22(3):255-261. [CrossRef] [Medline]
  4. Collins IM, Steel E, Mann GB, Emery JD, Bickerstaffe A, Trainer A, et al. Assessing and managing breast cancer risk: clinicians' current practice and future needs. Breast 2014 Oct;23(5):644-650. [CrossRef] [Medline]
  5. Amir E, Freedman OC, Seruga B, Evans DG. Assessing women at high risk of breast cancer: a review of risk assessment models. J Natl Cancer Inst 2010 May 19;102(10):680-691. [CrossRef] [Medline]
  6. Cintolo-Gonzalez JA, Braun D, Blackford AL, Mazzola E, Acar A, Plichta JK, et al. Breast cancer risk models: a comprehensive overview of existing models, validation, and clinical applications. Breast Cancer Res Treat 2017 Jul;164(2):263-284. [CrossRef] [Medline]
  7. Collins IM, Bickerstaffe A, Ranaweera T, Maddumarachchi S, Keogh L, Emery J, et al. iPrevent®: a tailored, web-based, decision support tool for breast cancer risk assessment and management. Breast Cancer Res Treat 2016 Feb;156(1):171-182 [FREE Full text] [CrossRef] [Medline]
  8. Tyrer J, Duffy SW, Cuzick J. A breast cancer prediction model incorporating familial and personal risk factors. Stat Med 2004 Apr 15;23(7):1111-1130. [CrossRef] [Medline]
  9. Lee AJ, Cunningham AP, Kuchenbaecker KB, Mavaddat N, Easton DF, Antoniou AC, Consortium of Investigators of Modifiers of BRCA1/2, Breast Cancer Association Consortium. BOADICEA breast cancer risk prediction model: updates to cancer incidences, tumour pathology and web interface. Br J Cancer 2014 Jan 21;110(2):535-545 [FREE Full text] [CrossRef] [Medline]
  10. Antoniou AC, Cunningham AP, Peto J, Evans DG, Lalloo F, Narod SA, et al. The BOADICEA model of genetic susceptibility to breast and ovarian cancers: updates and extensions. Br J Cancer 2008 Apr 22;98(8):1457-1466 [FREE Full text] [CrossRef] [Medline]
  11. Cancer Australia. Advice about familial aspects of breast cancer and epithelial ovarian cancer   URL: https:/​/canceraustralia.​​system/​tdf/​publications/​advice-about-familial-aspects-breast-cancer-and-epithelial-ovarian-cancer/​pdf/​2015_bog_familial_aspects_int.​pdf?file=1&type=node&id=2878 [accessed 2018-01-23] [WebCite Cache]
  12. Antill Y, Shanahan M, Phillips K. The integrated multidisciplinary clinic: A new model for the ongoing management of women at high genetic risk for breast and ovarian cancer. Cancer Forum 2005;29(2):A.
  13. Bunz U. The Computer-Email-Web (CEW) Fluency Scale-Development and Validation. International Journal of Human-Computer Interaction 2004 Dec;17(4):479-506. [CrossRef]
  14. Brooke J. SUS: A Retrospective. J Usability Stud 2013;8(2):29-40.
  15. Tullis T, Stetson J. Proceedings of UPA 2004 Conference. A comparison of questionnaires for assessing website usability   URL: [accessed 2018-01-24] [WebCite Cache]
  16. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: Adding an adjective rating scale. J Usability Stud 2009;4(3):114-123.
  17. Sauro J. A Practical Guide to the System Usability Scale. United States: CreateSpace Independent Publishing Platform; 2011.
  18. Smith SK, Trevena L, Barratt A, Dixon A, Nutbeam D, Simpson JM, et al. Development and preliminary evaluation of a bowel cancer screening decision aid for adults with lower literacy. Patient Educ Couns 2009 Jun;75(3):358-367. [CrossRef] [Medline]
  19. Meiser B, Butow PN, Barratt AL, Schnieden V, Gattas M, Kirk J, Psychological Impact Collaborative Group. Long-term outcomes of genetic counseling in women at increased risk of developing hereditary breast cancer. Patient Educ Couns 2001 Sep;44(3):215-225. [Medline]
  20. Lerman C, Trock B, Rimer BK, Jepson C, Brody D, Boyce A. Psychological side effects of breast cancer screening. Health Psychol 1991;10(4):259-267. [Medline]
  21. Marteau TM, Bekker H. The development of a six-item short-form of the state scale of the Spielberger State-Trait Anxiety Inventory (STAI). Br J Clin Psychol 1992 Sep;31 ( Pt 3):301-306. [Medline]
  22. Lerman C, Biesecker B, Benkendorf JL, Kerner J, Gomez-Caminero A, Hughes C, et al. Controlled trial of pretest education approaches to enhance informed decision-making for BRCA1 gene testing. J Natl Cancer Inst 1997 Jan 15;89(2):148-157. [Medline]
  23. Sivell S, Edwards A, Manstead AS, Reed MW, Caldon L, Collins K, BresDex Group. Increasing readiness to decide and strengthening behavioral intentions: evaluating the impact of a web-based patient decision aid for breast cancer treatment options (BresDex: Patient Educ Couns 2012 Aug;88(2):209-217. [CrossRef] [Medline]
  24. Meiser B, Butow P, Barratt A, Suthers G, Smith M, Colley A, et al. Attitudes to genetic testing for breast cancer susceptibility in women at increased risk developing hereditary breast cancer. J Med Genet 2000 Jun;37(6):472-476 [FREE Full text] [Medline]
  25. Hartmann LC, Lindor NM. The Role of Risk-Reducing Surgery in Hereditary Breast and Ovarian Cancer. N Engl J Med 2016 Feb 04;374(5):454-468. [CrossRef] [Medline]
  26. Haroun I, Graham T, Poll A, Sun P, Hill K, Weitzner E, et al. Reasons for risk-reducing mastectomy versus MRI-screening in a cohort of women at high hereditary risk of breast cancer. Breast 2011 Jun;20(3):254-258. [CrossRef] [Medline]
  27. Connors LM, Voian N, Shi Y, Lally RM, Edge S. Decision making after BRCA genetic testing. Down the road of transition. Clin J Oncol Nurs 2014 Jun;18(3):E58-E63. [CrossRef] [Medline]
  28. Bouchard L, Blancquaert I, Eisinger F, Foulkes WD, Evans G, Sobol H, et al. Prevention and genetic testing for breast cancer: variations in medical decisions. Soc Sci Med 2004 Mar;58(6):1085-1096. [Medline]
  29. Pal T, Lee J, Besharat A, Thompson Z, Monteiro AN, Phelan C, et al. Modes of delivery of genetic testing services and the uptake of cancer risk management strategies in BRCA1 and BRCA2 carriers. Clin Genet 2014 Jan;85(1):49-53 [FREE Full text] [CrossRef] [Medline]
  30. Watson M, Lloyd S, Davidson J, Meyer L, Eeles R, Ebbs S, et al. The impact of genetic counselling on risk perception and mental health in women with a family history of breast cancer. Br J Cancer 1999 Feb;79(5-6):868-874 [FREE Full text] [CrossRef] [Medline]
  31. Epstein SA, Lin TH, Audrain J, Stefanek M, Rimer B, Lerman C. Excessive breast self-examination among first-degree relatives of newly diagnosed breast cancer patients. High-Risk Breast Cancer Consortium. Psychosomatics 1997;38(3):253-261. [Medline]
  32. Braithwaite D, Sutton S, Mackay J, Stein J, Emery J. Development of a risk assessment tool for women with a family history of breast cancer. Cancer Detect Prev 2005;29(5):433-439. [CrossRef] [Medline]
  33. Cohn WF, Jones SM, Miesfeldt S. “Are you at risk for hereditary breast cancer?”: development of a personal risk assessment tool for hereditary breast and ovarian cancer. J Genet Couns 2008 Feb;17(1):64-78. [CrossRef] [Medline]
  34. Emery J. The GRAIDS Trial: the development and evaluation of computer decision support for cancer genetic risk assessment in primary care. Ann Hum Biol 2005;32(2):218-227. [CrossRef] [Medline]

BC: breast cancer
BS: breast surgeon(s)
GC: genetics clinician(s)
PCP: primary care physician(s)
PMCC: Peter MacCallum Cancer Centre
STAI: State-Trait Anxiety Inventory
SUS: System Usability Score

Edited by G Eysenbach; submitted 25.01.18; peer-reviewed by L Fallowfield, J Vilaplana; comments to author 07.08.18; revised version received 18.09.18; accepted 25.09.18; published 07.11.18


©Louisa L Lo, Ian M Collins, Mathias Bressel, Phyllis Butow, Jon Emery, Louise Keogh, Prue Weideman, Emma Steel, John L Hopper, Alison H Trainer, Gregory B Mann, Adrian Bickerstaffe, Antonis C Antoniou, Jack Cuzick, Kelly-Anne Phillips. Originally published in JMIR Formative Research (, 07.11.2018.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.