This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2353) Depression and Mood Disorders; Suicide Prevention (1341) Information Quality in Digital Media (334) Information Seeking, Information Needs (458)

Published on in Vol 6, No 9 (2022): September

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/36177, first published .
Quality of Late-Life Depression Information on the Internet: Website Evaluation Study

Quality of Late-Life Depression Information on the Internet: Website Evaluation Study

Quality of Late-Life Depression Information on the Internet: Website Evaluation Study

Authors of this article:

Teaghan A M Pryor1 Author Orcid Image ;   Kristin A Reynolds1, 2 Author Orcid Image ;   Paige L Kirby1 Author Orcid Image ;   Matthew T Bernstein1 Author Orcid Image
Article Authors Cited by (1) Tweetations (3) Metrics

    Original Paper

    1Department of Psychology, University of Manitoba, Winnipeg, MB, Canada

    2Department of Psychiatry, University of Manitoba, Winnipeg, MB, Canada

    *all authors contributed equally

    Corresponding Author:

    Kristin A Reynolds, BA, MA, PhD

    Department of Psychology

    University of Manitoba

    190 Dysart Rd

    Winnipeg, MB, R3T 2N2

    Canada

    Phone: 1 2044749528

    Email: kristin.reynolds@umanitoba.ca


    Background: The internet can increase the accessibility of mental health information and improve the mental health literacy of older adults. The quality of mental health information on the internet can be inaccurate or biased, leading to misinformation.

    Objective: This study aims to evaluate the quality, usability, and readability of websites providing information concerning depression in later life.

    Methods: Websites were identified through a Google search and evaluated by assessing quality (DISCERN), usability (Patient Education Materials Assessment Tool), and readability (Simple Measure of Gobbledygook).

    Results: The overall quality of late-life depression websites (N=19) was adequate, and the usability and readability were poor. No significant relationship was found between the quality and readability of the websites.

    Conclusions: The websites can be improved by enhancing information quality, usability, and readability related to late-life depression. The use of high-quality websites may improve mental health literacy and shared treatment decision-making for older adults.

    JMIR Form Res 2022;6(9):e36177

    doi:10.2196/36177

    Keywords

    late-life; depression; older adults; internet; websites; information quality; usability; readability 


    Background

    Late-life depression (LLD) can occur in adults aged ≥65 years, either for the first time or as a recurrent episode [1]. Major depressive disorder (MDD) is characterized by low mood or loss of interest in daily activities, changes in weight or appetite, trouble sleeping or sleeping too much, lack of energy, feelings of worthlessness or guilt, psychomotor agitation or retardation, difficulty focusing or making decisions, and thoughts of death or suicide. At least five of these symptoms must occur for most of the day, nearly every day, or for a period of at least 2 weeks [2].

    Approximately 2%-6% of adults aged ≥55 years have received a diagnosis of MDD or experienced a major depressive episode within the past year [3-5]. The prevalence of MDD in older adults may be higher when including individuals experiencing subclinical depression [6,7]. Furthermore, the severity of depressive symptoms among older adults is particularly concerning, given that adults aged ≥70 years, and older men, in particular, have the highest rates of completed suicide worldwide [8].

    It is important to note that older adults experiencing LLD can differ in symptomology compared with younger adults, such as presenting with fewer affective symptoms (eg, tearfulness), increased complaints of somatic symptoms, cognitive changes, and loss of interest [9]. Older adults are also more likely to experience comorbid health conditions and neurological disorders, which further affects the identification of LLD and influences the need for specialized treatment approaches [1,9].

    When older adults experience a mental health problem, such as depression, they are faced with a lack of information regarding the symptoms and how to manage them with effective treatment options [10,11]. This gap in knowledge leads to lower levels of mental health literacy (knowledge about recognition, prevention, and management), which can complicate or delay the mental health treatment–seeking process [12,13]. Despite such barriers, most older adults have positive feelings about seeking help for mental health problems and a desire for increased information and participation in the treatment decision-making process [14,15].

    Shared decision-making is a process that occurs between a patient and a health care provider, where diverse treatment options are shared by both parties to foster the agreement and implementation of a preferred treatment option [16]. Engaging in this process is beneficial for individuals with mental health problems, increasing their satisfaction and involvement in treatment decisions [17,18].

    The internet can be a valuable tool for meeting the information needs of older adults. By presenting a wide variety of treatment options, web-based information can facilitate engagement in the shared treatment decision-making process [19]. As noted previously, web-based depression information should reflect the differences in symptom presentation and treatment options that are relevant to the unique needs of the older adult population [1].

    Most internet users (58%-78%) use the internet to search for health information [20-25], and it is increasingly being used to access mental health information [26-29]. Older adults have high rates of internet use, with 73% of older adults using the internet [30,31] and 40% using the internet to access health information [32]. Little is known about specific internet use by older adults for mental health information, although a study found that 11% of older adults used it for finding information on mental health problems and expressed interest in using the internet as a tool for managing their mental health [33]. A more recent study found that 67% of adults with bipolar disorder aged ≥60 years who used the internet used it to access information about their disorder [34]. Despite this, some older adults feel as though they lack the knowledge and confidence to use the internet as a source of information [35].

    Therefore, caregivers of older adults are often involved in seeking information for their care recipient and play an invaluable role in facilitating the treatment-seeking and shared decision-making process [36,37]. Part of the information-seeking process involves using the internet to access essential treatment information. Recent research has shown that a high percentage of caregivers (67%-71%) use the internet to access health information on behalf of the individual they support [38,39], particularly for older adults [40].

    Despite the benefits of internet use for health information queries, including anonymity and accessibility, there are some disadvantages [41]. One of the drawbacks is the uncertainty of the quality of the information provided on the internet [31,42]. It may be difficult for individuals to determine whether the information presented is unbiased, accurate, and evidence based [43-46]. Inaccurate health information retrieved from the internet, which patients incorporate into their clinical requests, has been demonstrated to harm the physician-patient relationship and have detrimental effects on their health outcomes [47]. Therefore, evaluating the quality of the information provided on the web is essential in preventing the spread of misinformation and facilitating increased knowledge of balanced treatment options not only for older adults themselves but also for those who seek out information on their behalf.

    As a result, studies evaluating website quality are increasing [48] as internet use for mental health information becomes more prevalent. Furthermore, studies have raised concerns regarding the approach that researchers have taken to evaluate websites and the lack of consistency across studies. An important question stemming from this growing research base is what constitutes a high-quality website. Common criteria used in the literature to identify high-quality websites are based on the principles of quality, usability, and readability [48-50]. Website quality involves the extent to which a website provides clear, unbiased, evidence-based information regarding mental health diagnoses and treatment options [51,52]. Usability characteristics refer to the ease of use, navigation, and aesthetics of websites [53], and readability is the ease of reading written text [54-56], both of which contribute to the overall quality of web-based information. Therefore, high-quality websites should be relatively straightforward to navigate and comprehend.

    Website evaluations have been completed on a variety of health topics, including depression in the adult population [57-61]. A systematic review of studies evaluating the quality of mental health websites found that 23 of the 31 studies had poor quality overall [48]. A small number of studies have been conducted assessing the quality of depression information provided by websites. Generally, studies have shown that the quality of website information is poor [57-59]. However, one study examining the overall quality of websites for depression in adults aged 18 to 64 years found adequate quality, with most websites scoring higher than the mean score on a measure of content quality [61]. The discrepancy in findings could be because of variability in the website evaluation methodology used across studies. Nevertheless, this range of quality (poor to adequate) may not be helpful to consumers.

    Objectives

    Upon extensive review of the website evaluation literature, to the best of our knowledge, no study has examined the quality of websites specific to depression that appears or worsens in later life (LLD). Given the unique impact of aging on depression presentation, the involvement of caregivers in the treatment-seeking processes, and the influence of web-based information in the shared decision-making process, we deemed it essential to evaluate the quality of websites providing information on LLD. Websites were evaluated according to (1) quality of information, as evaluated by DISCERN [62], a standardized measure of website quality; (2) usability, as determined by the Patient Education Materials Assessment Tool (PEMAT) [63]; and (3) readability of information, as evaluated by the Simple Measure of Gobbledygook (SMOG) [64]. A secondary objective of this study was to determine whether website quality was related to usability and readability.

    On the basis of the existing mental health website evaluation literature, we hypothesized that (1) website quality would be adequate to poor according to DISCERN evaluations, (2) usability would be adequate to poor, and (3) the reading level would be higher than the recommended levels for health information (grades 7-8). Furthermore, we hypothesized that website quality, usability, and readability would be related. Specifically, the quality of websites would be positively associated with usability scores and negatively associated with reading levels.


    Website Selection

    Google Canada search engine was used to identify websites as it is the most widely used search engine worldwide [65] and has been identified as the starting point for most internet users seeking health information [30,42,66]. The search was completed on one of the computers in the research laboratory by the primary investigator (TAMP), where website cookies and search history were cleared before searching to prevent influencing the search results. The search terms “older” AND “depression” were used to complete the initial search to target websites presenting distinct information on LLD in this age group (presentation and treatment options).

    Websites within the first 3 pages of the search were evaluated if they did not meet exclusion criteria, as it has been found that most search engine users do not go past the first 3 pages of the search [67]. Websites were excluded from the evaluation if they were advertising or selling products; presented information from books or articles, as the purpose of this study is to evaluate websites; contained minimal information (<500 words) that was not substantial enough to evaluate; provided information that was not focused on LLD; and were not written in English, as this is the researchers’ language of origin. The website selection method and exclusion criteria were in line with prior depression website evaluation research [57-60].

    Procedure

    Measures of Website Quality

    The DISCERN instrument is a standardized measure comprising 16 items assessing the quality of written health information [62] and has been used in a variety of health website evaluation studies [68-71]. The reliability of DISCERN has been psychometrically evaluated in previous research [72-74] and is able to differentiate between low- and high-quality information [74]. DISCERN comprises 3 main sections focusing on how reliable the publication is, the quality of information for treatment options, and the overall quality of the publication [62]. Each item is rated on a 5-point scale from no to yes indicating the extent to which the information fulfills the criteria: 1 (criterion was not fulfilled at all), 2 to 4 (criterion was fulfilled to some extent), and 5 (criterion was completely fulfilled).

    Usability

    The PEMAT was used to assess usability. The PEMAT is a multi-item standardized tool used to assess the understandability (ability to understand materials) and actionability (ability to encourage consumers to take action on information presented) of materials educating patients on a variety of health topics [63]. The PEMAT has been used in numerous recent studies evaluating printed and web-based health-related materials [75,76]. The tool has been evaluated and found to have good internal consistency, reliability, and construct validity [77,78]. The tool is divided into 2 domains—understandability and actionability—with specific topic areas under each domain. The number of items used varies depending on the type of material used, either printed or audiovisual. For this study, PEMAT-printed was used, which comprises 34 items (17 understandability and 7 actionability items), as most of the website information can be printed from each website. Information was rated according to each item and scored either 0=disagree or 1=agree, with some items having the option of not applicable=NA. Separate usability and actionability scores were calculated by summing the total number of points given (excluding not applicable items) and dividing by the total number of possible points. This number was multiplied by 100 to obtain a percentage score to determine what percentage of the material is understandable or actionable.

    Readability

    A readability score was calculated for each website using the SMOG. The SMOG assesses the number of words with ≥3 syllables in 10 consecutive sentences sampled from the beginning, middle, and end of the text [79]. Although the Flesh-Kinkaid reading formula has been most frequently used and cited in assessing the readability of health information, the SMOG formula is recommended as a more appropriate formula to assess health information because of its consistency and ease of use when calculating reading level [79].

    Analyses

    The analysis component comprised (1) descriptive, (2) correlational, and (3) inferential statistics. All statistics were computed by the first author (TAMP) using SPSS (version 21.0; IBM Corp) for Windows. Descriptive analyses produced a mean score for each website, as well as a mean score and 95% CI for each DISCERN item. In addition, 2 percentages were computed for the domains of understandability and actionability according to the PEMAT. A 2-tailed Pearson correlation was calculated to determine the relationship between website quality (as determined by DISCERN) and website usability scores (as determined by the PEMAT) and between website quality and website reading level independently. An intraclass correlation coefficient was computed between the first half of the selected websites to determine the level of agreement between the primary (TAMP) and secondary (PLK) raters for DISCERN and the PEMAT, similar to previous research [60,80].

    Ethical Considerations

    This research did not involve human participants and involved the examination of mental health information in the public domain. Therefore, it was determined that no ethics approval was required to carry out this study.


    Website Characteristics

    Evaluated websites (N=19) are described in Table 1. This sample is consistent with previous research [57-60]. Most of the websites were from the United States; however, websites from Australia (Beyond Blue), Canada (HealthLinkBC), and Great Britain (Royal College of Psychiatry) also emerged in the initial search. The evaluated websites were hosted by hospitals (eg, Johns Hopkins Medicine), nonprofit organizations (Age UK and HelpGuide), and government organizations (National Institute of Mental Health). Most of the websites did not have a formal operationalized definition of LLD and used a variety of terms when defining LLD (eg, elderly or geriatric depression, depression and older adults or in older people, and aging and depression). Websites identified the ages of 60 to ≥65 years as a target age group within the late-life period.

    Table 1. Late-life depression website characteristics.
    Website (country)Search engine orderOverall qualitya,bUsabilityc,dReadabilitye



    		Understandability score (%)Actionability score (%)
    Age UK (United Kingdom)242.969.2609.5
    American Psychological Association (United States)122.353.84011.1
    Beyond Blue (Australia)44.153.86012.5
    Black Dog Institute (Australia)162.576.92012.4
    Canadian Coalition for Senior’s Mental Health (Canada)283.084.6606.9
    Centers for Disease Control and Prevention (United States)102.346.24010.7
    Health in Aging (United States)174.053.86011.5
    Healthline (United States)113.676.96010.8
    HealthLinkBC (Canada)303.976.9807.4
    HelpGuide (United States)13.471.4609.6
    Johns Hopkins Medicine (United States)132.669.24010.9
    MedlinePlus (United States)92.366.7408.8
    Mental Health America (United States)53.461.58011.2
    National Institute of Mental Health (United States)33.975.04010.2
    National Institute on Aging (United States)23.353.86010.6
    Royal College of Psychiatrists (United Kingdom)84.869.2507.1
    Substance Abuse and Mental Health Services (United States)293.957.12012.4
    WebMD (United States)63.646.2409.8
    World Health Organization251.530.8013.5

    aOverall quality was measured by DISCERN.

    bThe mean DISCERN score is a 1 to 5 rating averaged across 16 items.

    cUsability was measured by the Patient Education Materials Assessment Tool.

    dTo calculate the scores, items that are agreed upon are summed and divided by the total possible points then multiplied by 100 to get a percentage.

    eReadability was measured by the Simple Measure of Gobbledygook.

    Website Quality

    A mean score was provided for each website to better understand the quality of the information presented. Website quality varied greatly, with scores ranging from 1.5 (low quality) to 4.8 (high quality) out of a total score of 5 (Table 1). Websites that scored highly on DISCERN included the Royal College of Psychiatrists (4.8), Beyond Blue (4.1), and Health in Aging (4.0).

    An average score across the websites was also computed for each DISCERN item to better understand the criteria that were well addressed and criteria that needed improvement (Table 2). Most of the websites addressed the DISCERN items moderately well, with average DISCERN scores ranging from 2.4 to 4.0 (SD 1.0 to 1.8). The mean score of item 16, which served as an overall rating of the websites, was 3.2. This indicates that websites with information about LLD were of adequate quality. According to scores on specific DISCERN criteria, websites clearly showed that there were multiple treatment options available (average score of 4.0). Information presented by websites was relevant to the older adult population (average score of 3.7). Websites also encouraged shared decision-making with health care providers or family members (average score of 3.7). Websites lacked information on the risks of treatment options (average score of 2.4), describing how each treatment worked (average score of 2.5), and the benefits of each treatment (average score of 2.6). Most websites did not provide the sources used to create the publication (average score of 2.6). An intraclass correlation was computed to determine the reliability of the raters on the DISCERN measure, which determined that there was an excellent level of agreement (r9=0.90; P<.001).

    Table 2. Mean scores of DISCERN items across all websitesa.
    Item numberDISCERN itemScore, mean (SD; 95% CI)
    1Are the aims clear?3.4 (1.1; 2.9-3.9)
    2Does it achieve its aims?3.6 (1.0; 3.2-4.1)
    3Is it relevant?3.7 (1.2; 3.2-4.2)
    4Is it clear what sources of information were used to compile the publication (other than the author or producer)?2.6 (1.6; 1.9-3.3)
    5Is it clear when the information used or reported in the publication was produced?3.3 (1.3; 2.7-3.9)
    6Is it balanced and unbiased?3.6 (1.0; 3.1-4.0)
    7Does it provide details of additional sources of support and information?3.2 (1.4; 2.6-3.9)
    8Does it refer to areas of uncertainty?3.1 (1.3; 2.5-3.7)
    9Does it describe how each treatment works?2.5 (1.2; 2.0-3.1)
    10Does it describe the benefits of each treatment?2.6 (1.2; 2.1-3.2)
    11Does it describe the risks of each treatment?2.4 (1.3; 1.8-3.0)
    12Does it describe what would happen if no treatment was used?2.8 (1.8; 2.0-3.6)
    13Does it describe how the treatment choices affect the overall quality of life?3.4 (1.0; 3.0-3.9)
    14Is it clear that there may be more than one possible treatment choice?4 (1.3; 3.4-4.6)
    15Does it provide support for shared decision-making?3.7 (1.2; 3.2-4.3)
    16Based on the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices.3.2 (1.2; 2.6-3.7)

    aEach DISCERN item is rated on a 5-point scale with the anchors 1=did not meet criteria and 5=did meet criteria.

    Usability

    The understandability scores of the PEMAT (Table 1) varied, ranging from 30.8% to 84.6% (mean 62.8%). Only 32% (6/19) of websites met the 70% criteria, indicating that the website was understandable [77]. With regard to specific understandability criteria, most websites used an active voice and used a variety of visual cues to draw attention to important points of the websites. Websites lacked summaries of information and introduced complex medical terms within the text without definition. In examining the actionability section of the PEMAT, websites presented a range of scores from 0% to 80% (mean 47.9%), with only 10% (2/19) of the 19 websites meeting the minimum 70% threshold for websites to be deemed actionable. Upon further examination, most websites identified at least one action that individuals could take (eg, talking to their physician). Despite this, most websites did not provide any visual aids encouraging individuals to take action, lacked tools to aid individuals in taking action (eg, treatment planning sheet), and did not break down suggested actions into explicit steps.

    Two separate correlations were computed between the mean DISCERN scores and the usability and actionability percentages of the PEMAT to determine the relationship between website quality and usability. The correlation between DISCERN scores and the understandability scores of the PEMAT was not significant (r17=−0.30; P=.21). By contrast, the correlation between the DISCERN scores and the actionability scores of the PEMAT was found to be significant (r17=0.49; P=.04). An intraclass correlation was also computed to determine the interrater reliability, which established that there was an excellent level of agreement for the understandability section (r9=0.90; P<.001).

    Readability

    The readability of the websites was calculated using the SMOG readability formula, which produced a grade level score. The reading levels of the websites ranged from 6.9 to 13.5, with an average grade level of 10.4 across all websites (Table 1). Only 16% (3/19) of the websites met the National Institute of Health’s recommended grade level (grade 7-8). A correlational analysis was conducted to determine whether website readability was related to website quality, as measured by DISCERN. This analysis was nonsignificant (r17=−0.31; P=.20).

    Website Dimension Comparison

    Table 3 provides a simplified dimension description (good, adequate, or poor) for each website based on the evaluation measure scores, defined differently for each dimension: quality where good≥4, adequate=3 to 4, and poor≤3 (mean 1-5 rating scale); usability where good≥80, adequate=70 to 80, and poor≤70 (percentage understandable or actionable); readability, where good≤10, adequate=10 to 12, and poor≥12 (grade levels; Table 3). The rationale behind the cutoffs for these quality dimensions was based on how difficult it was for the websites to attain the recommended levels for each measure. For website quality, Good was used to describe websites that received a rating of >4 on the DISCERN measure as most websites were not able to achieve this. Similarly, with readability, most websites were not able to meet the recommended reading level (grade 7-8); thus, they were rated Good if they achieved a reading level of <10.

    Table 3. Website dimension comparisona.
    Website (country)Search engine orderOverall qualitybUsabilitycReadabilityd



    		Understandability scoreActionability score
    HelpGuide (United States)1AdequateAdequatePoorGood
    National Institute on Aging (United States)2AdequatePoorPoorAdequate
    National Institute of Mental Health (United States)3AdequateAdequatePoorAdequate
    Beyond Blue (Australia)4GoodPoorPoorPoor
    Mental Health America (United States)5AdequatePoorGoodAdequate
    WebMD (United States)6AdequatePoorPoorGood
    Royal College of Psychiatrists (United Kingdom)8GoodPoorPoorGood
    MedlinePlus (United States)9PoorPoorPoorGood
    Centers for Disease Control and Prevention (United States)10PoorPoorPoorAdequate
    Healthline (United States)11AdequateAdequatePoorAdequate
    American Psychological Association (United States)12PoorPoorPoorAdequate
    Johns Hopkins Medicine (United States)13PoorPoorPoorAdequate
    Black Dog Institute (Australia)16PoorAdequatePoorPoor
    Health in Aging (United States)17GoodPoorPoorAdequate
    Age UK (United Kingdom)24PoorPoorPoorGood
    World Health Organization25PoorPoorPoorPoor
    Canadian Coalition for Senior’s Mental Health (Canada)28AdequateGoodPoorGood
    Substance Abuse and Mental Health Services (United States)29AdequatePoorPoorPoor
    HealthLinkBC (Canada)30AdequateAdequateGoodGood

    aEach website was rated on each dimension as good, adequate, or poor, defined differently for each dimension.

    bOverall quality was measured by the DISCERN, where good ≥4, adequate=3 to 4, and poor ≤3 (mean 1-5 rating scale).

    cUsability was measured by the Patient Education Materials Assessment Tool, where good ≥80, adequate=70 to 80, and poor ≤70 (percentage understandable or actionable).

    dReadability was measured by the Simple Measure of Gobbledygook, where good≥10, adequate=10 to 12, and poor ≤12 (grade levels).


    Principal Findings and Comparison With Prior Work

    The purpose of this study was to evaluate the overall quality of LLD information provided by websites, as evaluated by standardized measures of website quality (DISCERN), usability (PEMAT), and readability (SMOG). A secondary purpose of this study was to examine the relationships among website quality, usability, and readability. Website quality ranged from low to high when examining the DISCERN mean scores. Furthermore, when looking at the average of the overall DISCERN rating (item 16), websites were of moderate quality (3.2/5), which indicates that LLD website quality is poor to adequate, providing support for the hypothesis that website quality would be poor to adequate. This finding is consistent with previous website evaluation studies [48,57-59,61].

    There were several high-quality websites, with the Royal College of Psychiatrists being identified as the website presenting the highest quality information according to DISCERN (4.8). This website also obtained a recommended reading level (grade 7.1), although usability (69.2%) and actionability (50%) were poor. The Royal College of Psychiatrists clearly identified the aims of the website and the sources used to create the information according to DISCERN. The same cannot be said for general trends across websites as the mean DISCERN score indicating identification of sources (item 4; “Is it clear what sources of information were used to compile the publication?”) was one of the lowest scores. Sources or references used to create web-based information should be clearly identified as consumers perceive them as positive content indicators, highlighting the trustworthiness and transparency of web-based health information [81].

    The Royal College of Psychiatrists provided relevant information on comorbid problems that occur late in life and how they might interact with depression, such as physical symptoms, long-term health problems, cognitive issues, and loneliness. This trend was also seen across other evaluated websites more generally, as they provided content that was highly relevant to depression in older adults (eg, sections addressing dementia, vascular depression, and insomnia), which was supported by one of the highest mean DISCERN scores evaluating relevance (item 3; “Is it relevant?”). Providing age-relevant depression information is particularly important as it can aid in increased knowledge about symptom recognition and differentiation of presenting problems (depression vs dementia), which has been shown to lead to more timely access to mental health services [82,83]. This finding is especially relevant for older adults in light of research demonstrating their lower levels of mental health literacy [84,85].

    Furthermore, the Royal College of Psychiatrists provided a range of unbiased treatment options, such as talk, medication, and complementary and brain stimulation treatments. This trend was also observed across all websites according to the high mean DISCERN score, indicating that the content was unbiased (item 6; “Is it balanced and unbiased?”). The Royal College of Psychiatrists included specific information to encourage the involvement of caregivers, specifically how to identify depression, encourage help seeking, and improve communication between caregivers and recipients. More broadly, information to encourage shared decision-making (eg, sections about talking with physicians or how to help someone with depression) had one of the highest scores across websites according to the mean DISCERN score evaluating support for shared decision-making (item 15; “Does it provide support for shared decision-making?”).

    This finding is important to note, as prior research supports older adults’ preference for involvement in decisions related to their health and treatment options [86], and caregivers have been identified as an important part of this process [36]. The inclusion of this information on websites could serve to encourage and improve the shared decision-making processes among older adults, caregivers, and health care providers [87]. As shown previously, websites that provide evidence-based information, diverse treatment options, and content that supports shared decision-making constitute a high-quality website [36,37,52].

    Websites generally failed to provide high-quality information about how different treatments work and the risks and benefits associated with those treatment options. A key premise of the shared decision-making process is to provide patients with different treatment options and allow them to weigh the risks and benefits associated with those treatments [16]. Incomplete treatment information may bias the shared decision-making process and limit individuals’ ability to fully weigh their treatment options [88]. Treatment information that clearly defines how treatments work and the associated risks and benefits serve to enhance mental health literacy, consumer empowerment, and shared decision-making [88-90]. Furthermore, when consumers have timely access to balanced, evidence-based information, they can make treatment decisions that align with their preferences [91].

    Most websites did not meet the minimum recommended levels of understandability and actionability. Websites with higher usability used formatting that facilitated better understanding, such as bolded main headings and subheadings, large fonts or the option to increase the font, information in short paragraphs or bullet points, boxes to highlight important information, and presenting most information on a single page. Despite providing formatting that promoted website usability, some websites had definitions of medical terms as hyperlinks, forcing individuals to navigate away from the original page the search brought them to. It was also difficult to locate treatment sections on certain websites (eg, Beyond Blue and Mental Health America), as they were not clearly marked or listed under the adult depression sections. These findings are an especially meaningful aspect of website quality for older adult populations as they may have different needs when using websites, such as the need for increased font size, darker letters with lighter backgrounds, short sections of information, and limited navigational steps when searching [92].

    Websites identified actions that individuals could perform to engage in their depression treatment (eg, behavioral strategies); however, most did not go on to further explain the steps and how to complete the recommended actions, limiting consumers’ ability to take the next steps in the help-seeking process. Although the understandability section of the PEMAT was not significantly correlated with DISCERN, the actionability section was significantly positively correlated with DISCERN. Specifically, as the quality of information increased, the actionability of the provided information increased. This finding suggests that higher-quality websites included information that encourages individuals to take action regarding their treatment, ideally resulting in prompt access to mental health services.

    Most websites did not meet the recommended reading level (grade 7-8), and therefore, the readability of the websites was poor overall, providing support for the hypothesis that the reading level will be higher than the recommended levels for health information (grades 7-8). This finding is in line with other web-based mental health information evaluation studies that have also found low readability levels among websites [36,48,50,54,60,93]. Website quality was not found to be associated with website readability. A possible explanation for this is that the websites used more complex medical terms to provide higher-quality information but at the cost of an increased reading level. This relationship has been observed in a recent study evaluating the reliability, readability, and quality of hip impingement information on the internet [94]. Furthermore, it is of interest that most of the websites failed to define complex medical terms. It is important to use simple and clear written content and provide definitions for more complex terms to make the content accessible to individuals at all reading levels.

    Limitations

    Although this study addresses significant gaps in website evaluation research, there are a number of limitations to note. First, search terms may not be representative of all web-based search strategies used to access information on LLD. Second, the websites included in this study were found by a routine Google search and do not represent the entirety of the websites that could be available to provide information on LLD. Third, raters’ pre-existing knowledge about the quality of particular organizations’ websites may have also biased raters’ perceptions of the quality and usability of the measures used in this study.

    Practice Implications and Recommendations

    This study identifies high-quality websites and provides valuable insights into which websites older adults and their caregivers should access to receive high-quality information about treatment options. It also highlights websites providers can recommend to their clients as a resource, such as the Royal College of Psychiatrists, Beyond Blue, and Health in Aging.

    Consumers of LLD information should seek web-based resources that discuss the impact of comorbid health problems and associated treatment options specific to this population. Consumers are also encouraged to engage with websites that provide clear evidence-based sources, as the identified sources can be an indicator of transparency and trustworthiness. Older adults and caregivers should seek websites that include content encouraging engagement in shared decision-making, such as how to discuss treatment options with their physician or involve a caregiver in their treatment.

    Finally, these findings provide guidance for organizations and website developers to consider when designing a website for older adults. Developers should consider using bolded headings, larger fonts, bullet points, short paragraphs, and text boxes to increase understandability and comprehension of the content. Moreover, developers should consider making navigation between adult depression pages and pages specific to older adults more cohesive, as well as sections of importance (eg, treatment sections) easy to find. Regarding content, developers should further break down actions into small measurable steps encouraging consumers to engage in behavioral changes or treatment seeking and provide tools to aid individuals in taking action (eg, treatment planning sheet). They should use simple and understandable language and provide in-text definitions or glossaries for more complex terms, subsequently increasing information accessibility to individuals with diverse educational backgrounds.

    Future Research

    As more people use the internet to access information about mental health problems, it is imperative to understand the quality of websites on the internet to understand whether older adults, caregivers, and health care providers are accessing easy-to-use, accurate, and comprehensive resources. Future research should examine older adults’ search strategies and, more specifically, whether older adults identify themselves as “older” when searching for mental health information on the web. Regardless of how older adults identify themselves, websites should better structure how their information is categorized on their websites to ensure that older adults are accessing the information relevant to them.

    It will also be important to look more in depth at the usability characteristics of websites that the PEMAT did not address. A measure such as the Visual Aesthetics of Websites Inventory, which examines more specific aesthetics of websites such as simplicity, diversity, colorfulness, and craftsmanship, would be useful in further evaluating usability [95]. Future researchers should fully evaluate the aesthetics of websites as it has implications on individuals’ first impressions and whether they will revisit the site or recommend it to others [96].

    Conclusions

    This is the first study to examine the quality of LLD information on the internet, and it addresses a gap in the literature by highlighting the quality of several LLD websites accessible on the internet. This study took a multifaceted approach to measuring website quality by using multiple measures to better understand different aspects that contribute to the overall quality of websites. The quality of LLD websites varied, ranging from low to high quality. Overall, the quality of the websites was adequate, and the usability and reading levels of the websites were poor. Websites provided information about particular problems that may affect depression in later life but lacked key information on how treatments work and the risks and benefits associated with treatments. Treatment sections were difficult to navigate or were found under adult depression sections. The ability of the websites to encourage understanding and action in individuals was also poor, and the information presented was higher than the recommended reading level. Websites were strong at providing multiple treatment options relevant to older adults and encouraging shared decision-making. They provided visual cues and formatting, which facilitated better understanding (eg, use of bolded headings, short paragraphs, or bullet points), and some websites were able to attain the recommended reading level. Website developers should consider increasing the quality, usability, and readability to produce high-quality information for older adults. High-quality websites may increase the mental health literacy of older adults and caregivers and improve the shared decision-making process. Health care providers should be aware of high-quality websites and should incorporate the use of high-quality websites into the shared decision-making process. They should direct older adults and caregivers to the high-quality websites identified in this study and use them as a decision-making tool by directing them to sections presenting different treatment options to further discuss in their ongoing care.

    Acknowledgments

    The authors would like to acknowledge funding from the University of Manitoba Centre on Aging (KAR, principal investigator) and Research Manitoba (KAR, principal investigator) for the completion and publication of this study.

    Authors' Contributions

    TAMP completed data collection, analyses, and writing. KAR participated in conceptualization, methodology, investigation, resources, supervision, and writing and editing of the manuscript. PLK participated in data collection and analyses. MTB participated in the conceptualization, methodology, supervision, and editing of the manuscript. All authors reviewed and approved the final manuscript.

    Conflicts of Interest

    None declared.

    1. Alexopoulos GS. Depression in the elderly. Lancet 2005;365(9475):1961-1970. [CrossRef] [Medline]
    2. American Psychiatric Association. Diagnostic And Statistical Manual Of Mental Disorders 5th edition. Virginia, United States: American Psychiatric Association; 2013.
    3. Kessler RC, Petukhova M, Sampson NA, Zaslavsky AM, Wittchen H. Twelve-month and lifetime prevalence and lifetime morbid risk of anxiety and mood disorders in the United States. Int J Methods Psychiatr Res 2012 Sep;21(3):169-184 [FREE Full text] [CrossRef] [Medline]
    4. Pearson C, Janz T, Ali J. Mental and substance use disorders in Canada. Statistics Canada.   URL: http://www.statcan.gc.ca/pub/82-624-x/2013001/article/11855-eng.htm#n9 [accessed 2022-09-02]
    5. Reynolds K, Pietrzak RH, El-Gabalawy R, Mackenzie CS, Sareen J. Prevalence of psychiatric disorders in U.S. older adults: findings from a nationally representative survey. World Psychiatry 2015 Mar;14(1):74-81 [FREE Full text] [CrossRef] [Medline]
    6. Gum AM, King-Kallimanis B, Kohn R. Prevalence of mood, anxiety, and substance-abuse disorders for older Americans in the national comorbidity survey-replication. Am J Geriatr Psychiatry 2009 Oct;17(9):769-781. [CrossRef] [Medline]
    7. Laborde-Lahoz P, El-Gabalawy R, Kinley J, Kirwin PD, Sareen J, Pietrzak RH. Subsyndromal depression among older adults in the USA: prevalence, comorbidity, and risk for new-onset psychiatric disorders in late life. Int J Geriatr Psychiatry 2015 Jul;30(7):677-685. [CrossRef] [Medline]
    8. Preventing Suicide: A Global Imperative. Geneva: World Health Organization; 2014.
    9. Fiske A, Wetherell JL, Gatz M. Depression in older adults. Annu Rev Clin Psychol 2009;5:363-389 [FREE Full text] [CrossRef] [Medline]
    10. Farrer L, Leach L, Griffiths KM, Christensen H, Jorm AF. Age differences in mental health literacy. BMC Public Health 2008 May 20;8:125. [CrossRef] [Medline]
    11. Wetherell JL, Petkus AJ, McChesney K, Stein MB, Judd PH, Rockwell E, et al. Older adults are less accurate than younger adults at identifying symptoms of anxiety and depression. J Nerv Ment Dis 2009 Aug;197(8):623-626 [FREE Full text] [CrossRef] [Medline]
    12. Jorm AF, Korten AE, Jacomb PA, Christensen H, Rodgers B, Pollitt P. "Mental health literacy": a survey of the public's ability to recognise mental disorders and their beliefs about the effectiveness of treatment. Med J Aust 1997 Mar 17;166(4):182-186. [CrossRef] [Medline]
    13. Kim YS, Lee HY, Lee MH, Simms T, Park BH. Mental health literacy in Korean older adults: a cross-sectional survey. J Psychiatr Ment Health Nurs 2017 Oct;24(7):523-533. [CrossRef] [Medline]
    14. Mackenzie CS, Scott T, Mather A, Sareen J. Older adults' help-seeking attitudes and treatment beliefs concerning mental health problems. Am J Geriatr Psychiatry 2008 Dec;16(12):1010-1019 [FREE Full text] [CrossRef] [Medline]
    15. Xie B, Wang M, Feldman R, Zhou L. Exploring older and younger adults' preferences for health information and participation in decision making using the Health Information Wants Questionnaire (HIWQ). Health Expect 2014 Dec;17(6):795-808 [FREE Full text] [CrossRef] [Medline]
    16. Charles C, Gafni A, Whelan T. Shared decision-making in the medical encounter: what does it mean? (or it takes at least two to tango). Soc Sci Med 1997 Mar;44(5):681-692. [CrossRef] [Medline]
    17. Fisher LJ, Goldney RD. Differences in community mental health literacy in older and younger Australians. Int J Geriatr Psychiatry 2003 Jan;18(1):33-40. [CrossRef] [Medline]
    18. Loh A, Simon D, Wills CE, Kriston L, Niebling W, Härter M. The effects of a shared decision-making intervention in primary care of depression: a cluster-randomized controlled trial. Patient Educ Couns 2007 Aug;67(3):324-332. [CrossRef] [Medline]
    19. Siegel CA. Embracing the internet for progress in shared decision-making. Inflamm Bowel Dis 2007 Dec;13(12):1579-1580. [CrossRef] [Medline]
    20. Bach RL, Wenz A. Studying health-related internet and mobile device use using web logs and smartphone records. PLoS One 2020;15(6):e0234663 [FREE Full text] [CrossRef] [Medline]
    21. Beck F, Richard J, Nguyen-Thanh V, Montagni I, Parizot I, Renahy E. Use of the internet as a health information resource among French young adults: results from a nationally representative survey. J Med Internet Res 2014 May 13;16(5):e128 [FREE Full text] [CrossRef] [Medline]
    22. Ducrot P, Montagni I, Nguyen Thanh V, Serry A, Richard J. Evolution of online health-related information seeking in France from 2010 to 2017: results from nationally representative surveys. J Med Internet Res 2021 Apr 14;23(4):e18799 [FREE Full text] [CrossRef] [Medline]
    23. Shahab L, Brown J, Gardner B, Smith SG. Seeking health information and support online: does it differ as a function of engagement in risky health behaviors? Evidence from the health information national trends survey. J Med Internet Res 2014 Dec 06;16(11):e253 [FREE Full text] [CrossRef] [Medline]
    24. Smail-Crevier R, Powers G, Noel C, Wang J. Health-related internet usage and design feature preference for e-mental health programs among men and women. J Med Internet Res 2019 Mar 18;21(3):e11224 [FREE Full text] [CrossRef] [Medline]
    25. Getting a second opinion: health information and the internet. Statistics Canada. 2015.   URL: https://www150.statcan.gc.ca/n1/pub/82-003-x/2008001/article/10515/5002590-eng.htm [accessed 2022-09-02]
    26. Brunault P, Bray A, Rerolle C, Cognet S, Gaillard P, El-Hage W. [Differences in access to Internet and Internet-based information seeking according to the type of psychiatric disorder]. Rev Epidemiol Sante Publique 2017 May;65(2):125-136. [CrossRef] [Medline]
    27. Kalckreuth S, Trefflich F, Rummel-Kluge C. Mental health related Internet use among psychiatric patients: a cross-sectional analysis. BMC Psychiatry 2014 Dec 24;14:368 [FREE Full text] [CrossRef] [Medline]
    28. Scott J, Hockey S, Ospina-Pinillos L, Doraiswamy PM, Alvarez-Jimenez M, Hickie I. Research to Clinical Practice-Youth seeking mental health information online and its impact on the first steps in the patient journey. Acta Psychiatr Scand 2022 Mar;145(3):301-314. [CrossRef] [Medline]
    29. Villani M, Kovess-Masfety V. How do people experiencing schizophrenia spectrum disorders or other psychotic disorders use the internet to get information on their mental health? Literature review and recommendations. JMIR Ment Health 2017 Jan 03;4(1):e1 [FREE Full text] [CrossRef] [Medline]
    30. Nikoloudakis IA, Vandelanotte C, Rebar AL, Schoeppe S, Alley S, Duncan MJ, et al. Examining the correlates of online health information-seeking behavior among men compared with women. Am J Mens Health 2018 Sep;12(5):1358-1367 [FREE Full text] [CrossRef] [Medline]
    31. Internet/broadband fact sheet. Pew Research Center. 2021 Apr 7.   URL: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/ [accessed 2022-08-31]
    32. Yoon H, Jang Y, Vaughan PW, Garcia M. Older adults' internet use for health information: digital divide by race/ethnicity and socioeconomic status. J Appl Gerontol 2020 Jan;39(1):105-110. [CrossRef] [Medline]
    33. Crabb RM, Rafie S, Weingardt KR. Health-related internet use in older primary care patients. Gerontology 2012;58(2):164-170. [CrossRef] [Medline]
    34. Bauer R, Glenn T, Strejilevich S, Conell J, Alda M, Ardau R, Miranda-Scippa, et al. Internet use by older adults with bipolar disorder: international survey results. Int J Bipolar Disord 2018 Sep 04;6(1):20 [FREE Full text] [CrossRef] [Medline]
    35. Moult A, Burroughs H, Kingstone T, Chew-Graham CA. How older adults self-manage distress - does the internet have a role? A qualitative study. BMC Fam Pract 2018 Nov 29;19(1):185 [FREE Full text] [CrossRef] [Medline]
    36. Elliott J, McNeil H, Ashbourne J, Huson K, Boscart V, Stolee P. Engaging older adults in health care decision-making: a realist synthesis. Patient 2016 Oct;9(5):383-393 [FREE Full text] [CrossRef] [Medline]
    37. Funk LM, Dansereau L, Novek S. Carers as system navigators: exploring sources, processes and outcomes of structural burden. Gerontologist 2019 May 17;59(3):426-435. [CrossRef] [Medline]
    38. Bangerter LR, Griffin J, Harden K, Rutten LJ. Health information-seeking behaviors of family caregivers: analysis of the health information national trends survey. JMIR Aging 2019 Jan 14;2(1):e11237 [FREE Full text] [CrossRef] [Medline]
    39. Fox L, Brenner J. Family caregivers online. Pew Research Center. 2012 Jul 12.   URL: https://www.pewresearch.org/internet/2012/07/12/family-caregivers-online/ [accessed 2022-04-10]
    40. Shaffer KM, Chow PI, Cohn WF, Ingersoll KS, Ritterband LM. Informal caregivers' use of internet-based health resources: an analysis of the health information national trends survey. JMIR Aging 2018 Dec 18;1(2):e11051 [FREE Full text] [CrossRef] [Medline]
    41. Cline RJ, Haynes KM. Consumer health information seeking on the internet: the state of the art. Health Educ Res 2001 Dec;16(6):671-692. [CrossRef] [Medline]
    42. Khazaal Y, Chatton A, Cochand S, Hoch A, Khankarli MB, Khan R, et al. Internet use by patients with psychiatric disorders in search for general and medical informations. Psychiatr Q 2008 Dec;79(4):301-309. [CrossRef] [Medline]
    43. Ek S, Eriksson-Backa K, Niemelä R. Use of and trust in health information on the internet: a nationwide eight-year follow-up survey. Inform Health Soc Care 2013 Sep;38(3):236-245. [CrossRef] [Medline]
    44. Fiksdal AS, Kumbamu A, Jadhav AS, Cocos C, Nelsen LA, Pathak J, et al. Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives. J Med Internet Res 2014 Oct 07;16(10):e224 [FREE Full text] [CrossRef] [Medline]
    45. Mitzner TL, Boron JB, Fausset CB, Adams AE, Charness N, Czaja SJ, et al. Older adults talk technology: technology usage and attitudes. Comput Human Behav 2010 Nov 01;26(6):1710-1721 [FREE Full text] [CrossRef] [Medline]
    46. Perrin A, Atske S. About three-in-ten U.S. adults say they are ‘almost constantly’ online. Pew Research Center. 2021 Mar 26.   URL: https:/​/www.​pewresearch.org/​fact-tank/​2021/​03/​26/​about-three-in-ten-u-s-adults-say-they-are-almost-constantly-online/​ [accessed 2022-09-02]
    47. Murray E, Lo B, Pollack L, Donelan K, Catania J, Lee K, et al. The impact of health information on the internet on health care and the physician-patient relationship: national U.S. survey among 1.050 U.S. physicians. J Med Internet Res 2003;5(3):e17 [FREE Full text] [CrossRef] [Medline]
    48. Reavley NJ, Jorm AF. The quality of mental disorder information websites: a review. Patient Educ Couns 2011 Nov;85(2):e16-e25. [CrossRef] [Medline]
    49. Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [CrossRef] [Medline]
    50. Kim H, Xie B. Health literacy in the eHealth era: a systematic review of the literature. Patient Educ Couns 2017 Jun;100(6):1073-1082. [CrossRef] [Medline]
    51. Fu K, Wong PW, Yip PS. What do internet users seek to know about depression from web searches? A descriptive study of 21 million web queries. J Clin Psychiatry 2010 Sep;71(9):1246-1247. [CrossRef] [Medline]
    52. Reichow B, Halpern JI, Steinhoff TB, Letsinger N, Naples A, Volkmar FR. Characteristics and quality of autism websites. J Autism Dev Disord 2012 Jun;42(6):1263-1274. [CrossRef] [Medline]
    53. Tsakonas G, Papatheodorou C. Analysing and evaluating usefulness and usability in electronic information services. J Inform Sci 2006 Jun 09;32(5):400-419. [CrossRef]
    54. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285(20):2612-2621 [FREE Full text] [CrossRef] [Medline]
    55. Coquard O, Fernandez S, Zullino D, Khazaal Y. A follow-up study on the quality of alcohol dependence-related information on the web. Subst Abuse Treat Prev Policy 2011 Jun 10;6:13 [FREE Full text] [CrossRef] [Medline]
    56. Reavley NJ, Mackinnon AJ, Morgan AJ, Alvarez-Jimenez M, Hetrick SE, Killackey E, et al. Quality of information sources about mental disorders: a comparison of Wikipedia with centrally controlled web and printed sources. Psychol Med 2012 Aug;42(8):1753-1762. [CrossRef] [Medline]
    57. Ferreira-Lay P, Miller S. The quality of internet information on depression for lay people. Psychiatr Bull 2018 Jan 02;32(5):170-173. [CrossRef]
    58. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000 Dec 16;321(7275):1511-1515 [FREE Full text] [CrossRef] [Medline]
    59. Lissman TL, Boehnlein JK. A critical review of internet information about depression. Psychiatr Serv 2001 Aug;52(8):1046-1050. [CrossRef] [Medline]
    60. Walsh K, Pryor TA, Reynolds KA, Walker JR, Mobilizing Minds Research Group. Searching for answers: how well do depression websites answer the public's questions about treatment choices? Patient Educ Couns 2019 Jan;102(1):99-105. [CrossRef] [Medline]
    61. Zermatten A, Khazaal Y, Coquard O, Chatton A, Bondolfi G. Quality of web-based information on depression. Depress Anxiety 2010 Sep;27(9):852-858. [CrossRef] [Medline]
    62. Charnock D. The DISCERN Handbook Quality Criteria for Consumer Health Information on Treatment Choices. Abingdon, Oxon: Radcliffe Medical Press; 1998.
    63. Shoemaker SJ, Wolf MS, Brach C. The patient education materials assessment tool (PEMAT) and user’s guide. Agency for Healthcare Research and Quality. 2020.   URL: https://www.ahrq.gov/professionals/prevention-chronic-care/improve/self-mgmt/pemat/index.html [accessed 2022-09-02]
    64. McLaughlin G. SMOG grading: a new readability formula. J Read 1969;12(8):639-646.
    65. Ratcliff C. What are the top 10 most popular search engines? Search Engine Watch. 2016 Aug 8.   URL: https://searchenginewatch.com/2016/08/08/what-are-the-top-10-most-popular-search-engines/ [accessed 2022-09-02]
    66. Sbaffi L, Zhao C. Modeling the online health information seeking process: information channel selection among university students. J Assoc Inform Sci Technol 2019 Apr 13;71(2):196-207 [FREE Full text] [CrossRef]
    67. iProspect blended search results study. iProspect. 2008 Apr.   URL: https://tinyurl.com/4m9rb58b [accessed 2022-08-31]
    68. Barnes C, Harvey R, Wilde A, Hadzi-Pavlovic D, Wilhelm K, Mitchell PB. Review of the quality of information on bipolar disorder on the internet. Aust N Z J Psychiatry 2009 Jan 01;43(10):934-945. [CrossRef]
    69. Batchelor JM, Ohya Y. Use of the DISCERN instrument by patients and health professionals to assess information resources on treatments for asthma and atopic dermatitis. Allergol Int 2009 Mar;58(1):141-145 [FREE Full text] [CrossRef] [Medline]
    70. Cerminara C, Santarone ME, Casarelli L, Curatolo P, El Malhany N. Use of the DISCERN tool for evaluating web searches in childhood epilepsy. Epilepsy Behav 2014 Dec;41:119-121. [CrossRef] [Medline]
    71. Grant N, Rodger S, Hoffmann T. Evaluation of autism-related health information on the web. J Appl Res Intellect Disabil 2015 Jul;28(4):276-282. [CrossRef] [Medline]
    72. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Mar;53(2):105-111 [FREE Full text] [CrossRef] [Medline]
    73. McCool ME, Wahl J, Schlecht I, Apfelbacher C. Evaluating written patient information for eczema in German: comparing the reliability of two instruments, DISCERN and EQIP. PLoS One 2015;10(10):e0139895 [FREE Full text] [CrossRef] [Medline]
    74. Rees CE, Ford JE, Sheard CE. Evaluating the reliability of DISCERN: a tool for assessing the quality of written patient information on treatment choices. Patient Educ Couns 2002 Jul;47(3):273-275. [CrossRef] [Medline]
    75. Balakrishnan V, Chandy Z, Verma SP. Are online Zenker's diverticulum materials readable and understandable? Otolaryngol Head Neck Surg 2016 Nov;155(5):758-763. [CrossRef] [Medline]
    76. Morony S, McCaffery KJ, Kirkendall S, Jansen J, Webster AC. Health literacy demand of printed lifestyle patient information materials aimed at people with chronic kidney disease: are materials easy to understand and act on and do they use meaningful visual aids? J Health Commun 2017 Feb 25;22(2):163-170. [CrossRef] [Medline]
    77. Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns 2014 Sep;96(3):395-403 [FREE Full text] [CrossRef] [Medline]
    78. Vishnevetsky J, Walters CB, Tan KS. Interrater reliability of the patient education materials assessment tool (PEMAT). Patient Educ Couns 2018 Mar;101(3):490-496 [FREE Full text] [CrossRef] [Medline]
    79. Wang L, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm 2013;9(5):503-516. [CrossRef] [Medline]
    80. Reynolds KA, Walker JR, Walsh K, Mobilizing Minds Research Group. How well do websites concerning children's anxiety answer parents' questions about treatment choices? Clin Child Psychol Psychiatry 2015 Oct;20(4):555-569. [CrossRef] [Medline]
    81. Sun Y, Zhang Y, Gwizdka J, Trace CB. Consumer evaluation of the quality of online health information: systematic literature review of relevant criteria and indicators. J Med Internet Res 2019 May 02;21(5):e12522 [FREE Full text] [CrossRef] [Medline]
    82. Brijnath B, Protheroe J, Mahtani KR, Antoniades J. Do web-based mental health literacy interventions improve the mental health literacy of adult consumers? Results from a systematic review. J Med Internet Res 2016 Jun 20;18(6):e165 [FREE Full text] [CrossRef] [Medline]
    83. Kim YS, Rhee TG, Lee HY, Park BH, Sharratt ML. Mental health literacy as a mediator in use of mental health services among older Korean adults. Int Psychogeriatr 2016 Oct 20;29(2):269-279. [CrossRef]
    84. Cronin M, O'Hanlon S, O'Connor M. Readability level of patient information leaflets for older people. Ir J Med Sci 2011 Mar;180(1):139-142. [CrossRef] [Medline]
    85. Nutbeam D, Wise M. Australia: planning for better health. Promotion Educ 2016 Aug 24;1(4):19-24. [CrossRef]
    86. O'Neal EL, Adams JR, McHugo GJ, Van Citters AD, Drake RE, Bartels SJ. Preferences of older and younger adults with serious mental illness for involvement in decision-making in medical and psychiatric settings. Am J Geriatr Psychiatry 2008 Oct;16(10):826-833. [CrossRef] [Medline]
    87. Luo A, Qin L, Yuan Y, Yang Z, Liu F, Huang P, et al. The effect of online health information seeking on physician-patient relationships: systematic review. J Med Internet Res 2022 Feb 10;24(2):e23354 [FREE Full text] [CrossRef] [Medline]
    88. Thapa DK, Visentin DC, Kornhaber R, West S, Cleary M. The influence of online health information on health decisions: a systematic review. Patient Educ Couns 2021 Apr;104(4):770-784. [CrossRef] [Medline]
    89. Alden DL. Decision aid influences on factors associated with patient empowerment prior to cancer treatment decision making. Med Decis Making 2014 Oct;34(7):884-898. [CrossRef] [Medline]
    90. Náfrádi L, Nakamoto K, Csabai M, Papp-Zipernovszky O, Schulz PJ. An empirical test of the Health Empowerment Model: does patient empowerment moderate the effect of health literacy on health status? Patient Educ Couns 2018 Mar;101(3):511-517. [CrossRef] [Medline]
    91. Krist AH, Tong ST, Aycock RA, Longo DR. Engaging patients in decision-making and behavior change to promote prevention. Stud Health Technol Inform 2017;240:284-302 [FREE Full text] [Medline]
    92. Nahm E, Preece J, Resnick B, Mills ME. Usability of health Web sites for older adults: a preliminary study. Comput Inform Nurs 2004;22(6):326-34; quiz 335. [CrossRef] [Medline]
    93. Grohol JM, Slimowicz J, Granda R. The quality of mental health information commonly searched for on the internet. Cyberpsychol Behav Soc Netw 2014 Apr;17(4):216-221. [CrossRef] [Medline]
    94. Küçükdurmaz F, Gomez MM, Secrist E, Parvizi J. Reliability, readability and quality of online information about femoracetabular impingement. Arch Bone Jt Surg 2015 Jul;3(3):163-168 [FREE Full text] [Medline]
    95. Moshagen M, Thielsch MT. Facets of visual aesthetics. Int J Human Comput Stud 2010 Oct;68(10):689-709. [CrossRef]
    96. Thielsch MT, Blotenberg I, Jaron R. User evaluation of websites: from first impression to recommendation. Interact Comput 2013 Jun 04;26(1):89-102. [CrossRef]

    ‎
    LLD: late-life depression
    MDD: major depressive disorder
    PEMAT: Patient Education Materials Assessment Tool
    SMOG: Simple Measure of Gobbledygook

    Edited by A Mavragani; submitted 06.01.22; peer-reviewed by A Tuck, S Sutherland; comments to author 15.02.22; revised version received 12.04.22; accepted 04.05.22; published 12.09.22

    Copyright

    ©Teaghan A M Pryor, Kristin A Reynolds, Paige L Kirby, Matthew T Bernstein. Originally published in JMIR Formative Research (https://formative.jmir.org), 12.09.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.