Published on in Vol 5, No 11 (2021): November

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/29146, first published .
Googling for Suicide–Content and Quality Analysis of Suicide-Related Websites: Thematic Analysis

Googling for Suicide–Content and Quality Analysis of Suicide-Related Websites: Thematic Analysis

Googling for Suicide–Content and Quality Analysis of Suicide-Related Websites: Thematic Analysis

Original Paper

1Department of Psychiatry, Xiamen Xianyue Hospital, Xiamen, China

2Department of Educational Science, University of Genoa, Genoa, Italy

3Department of Neuroscience, Rehabilitation, Ophthalmology, Genetics and Maternal/Child Sciences, University of Genoa, Genoa, Italy

4Department of Psychiatry, Istituto di ricovero e cura a carattere scientifico Ospedale Policlinico San Martino, Genoa, Italy

5College of Medicine and Medical Sciences, Arabian Gulf University, Manama, Bahrain

6Ministry of Health, Manama, Bahrain

7Higher Institute of Sport and Physical Education of Sfax, University of Sfax, Sfax, Tunisia

8Higher Institute of Sport and Physical Education of Kef, University of Jendouba, Jendouba, Tunisia

9Laboratory for Industrial and Applied Mathematics, Department of Mathematics and Statistics, York University, Toronto, ON, Canada

Corresponding Author:

Nicola Luigi Bragazzi, MPH, MD, PhD

Laboratory for Industrial and Applied Mathematics

Department of Mathematics and Statistics

York University

4700 Keele St

Toronto, ON, M3J 1P3

Canada

Phone: 1 416 736 2100

Email: bragazzi@yorku.ca


Background: Suicide represents a public health concern, imposing a dramatic burden. Prosuicide websites are “virtual pathways” facilitating a rise in suicidal behaviors, especially among socially isolated, susceptible individuals.

Objective: The aim of this study is to characterize suicide-related webpages in the Italian language.

Methods: The first 5 most commonly used search engines in Italy (ie, Bing, Virgilio, Yahoo, Google, and Libero) were mined using the term “suicidio” (Italian for suicide). For each search, the first 100 webpages were considered. Websites resulting from each search were collected and duplicates deleted so that unique webpages could be analyzed and rated with the HONcode instrument

Results: A total of 65 webpages were included: 12.5% (8/64) were antisuicide and 6.3% (4/64) explicitly prosuicide. The majority of the included websites had a mixed or neutral attitude toward suicide (52/64, 81.2%) and had informative content and purpose (39/64, 60.9%). Most webpages targeted adolescents as an age group (38/64, 59.4%), contained a reference to other psychiatric disorders or comorbidities (42/64, 65.6%), included medical/professional supervision or guidance (45/64, 70.3%), lacked figures or pictures related to suicide (41/64, 64.1%), and did not contain any access restraint (62/64, 96.9%). The major shortcoming to this study is the small sample size of webpages analyzed and the search limited to the keyword “suicide.”

Conclusions: Specialized mental health professionals should try to improve their presence online by providing high-quality material.

JMIR Form Res 2021;5(11):e29146

doi:10.2196/29146

Keywords



Suicide represents a public health concern, imposing a dramatic burden both in epidemiological and societal terms [1]. According to the latest available 2021 report of the World Health Organization (WHO) in 2019, Italy had a crude rate of 6.7 (95% CI 6.0-8.5) suicides per 100,000 people, with a male:female ratio of 2.9 [2].

Within the current eHealth era, in which the new information and communication technologies are being increasingly exploited to obtain health-related information [3], the internet can be characterized as having a Yin-Yang, paradoxical nature [4]. On the one hand, there exists a statistically significant correlation between a pathological use of the web and suicidality (either in terms of suicidal ideation or nonsuicidal self-injury). Prosuicide websites and online suicide pacts are “virtual pathways” that facilitate the emergence of suicidal behaviors, especially among socially isolated and susceptible individuals. On the other hand, the internet can serve as an effective tool for suicide monitoring and tracking [3,5] and can play a major role in suicide prevention [6,7], particularly for those vulnerable individuals who would be otherwise difficult to reach with conventional approaches [8,9].

Understanding the determinants of suicide-related web searches is, therefore, of crucial importance [10]. According to Wong and collaborators [11], the categories of the most visited suicide-related webpages can be classified into entertainment (30.13%), scientific information (18.31%), and community resources (14.53%). Among the 1314 accessed webpages in their study, prosuicide websites represented only a small fraction (0.15%). The most used search terms were “committing suicide with a gas oven,” “hairless goat,” “pictures of murder by strangulation,” and “photo of a severe burn.”

Suicide-related digital behavior has been assessed in different countries and different languages [12-15]. However, to the best of our knowledge, there is a dearth of information concerning suicide-related webpages in the Italian language. As such, the aim of this study is to address this gap in knowledge.


Suicide-Related Website Selection and Inclusion

The first 5 most commonly used search engines in Italy (namely, Bing, Virgilio, Yahoo, Google, and Libero) were mined with the search term “suicidio” (Italian for suicide). For each search, the first 100 webpages were considered. Websites resulting from each search were collected and duplicates deleted so that webpages could be analyzed and rated by 2 authors independently (MO and DP).

Quality assessment of the included webpages was performed with the HONcode instrument, which comprises the following sections: authoritative (if the qualifications of the authors are mentioned), complementarity (if the site aims at supporting but not replacing or substituting the physician-patient relationship), privacy (if visitors' data are protected and respected), attribution (if sources of published information in the medical and health-related pages are explicitly mentioned and referred to), justifiability (if claims related to potential benefits or performance are evidence based), transparency (if the presentation of the online material is accessible, with accurate email contacts), financial disclosure (if eventual funding sources are identified and mentioned), and advertising policy (if advertising and editorial materials are separate and identifiable) [16].

The agreement between the 2 authors was assessed with the Cohen κ statistic.

Furthermore, the 2 authors filled in an ad hoc predesigned Excel spreadsheet (Microsoft Corp) containing questions concerning the attitude of the webpage toward suicide (prosuicide, antisuicide, mixed or neutral), content and purpose of the webpage (supporting or emotional, informative, preventive), supervision or guidance (if the webpage was user-generated or handled by a professional figure, such as a psychiatrist or a psychologist), reference to age groups (adolescents or older people) targeted or referred to by the webpage (this simple dichotomic stratification was applied based on the availability of information provided and considering that the majority of internet users consist of adolescents), reference to other psychiatric disorders or comorbidities, the presence of figures or pictures related to suicide, and the presence of access restraint.

Moreover, for each included webpage, the number of comments’ likes, and shares, if any, was recorded.

Statistical Analysis

Before commencing any statistical handling and processing of the data, figures were visually inspected for potential outliers. The normality of data distribution was checked using the D’Agostino-Pearson omnibus test. Univariate analysis (chi-squared test and the chi-squared test for trend, Fisher’s exact test, t test, and analysis of variance or their nonparametric versions in case of violation of normal distribution) was performed for the variables under study. Multivariate linear regression analysis was performed to shed light on the determinants of the HONcode score.

All statistical analyses were performed with the commercial SPSS for Windows, version 24.0 (IBM Corp). Figures with P values less than .05 were considered statistically significant.


Sixty-four unique websites were retrieved and analyzed. Concerning the attitude toward suicide, 4 webpages were in favor of suicide (6.3%), whereas 8 sites were against suicide (12.5%). The remaining webpages (52/64, 81.2%) were neutral toward suicide. The overrepresentation of neutral pages in tone was statistically significant (χ22=30.55; contingency coefficient=0.44; P<.001; χ21 for trend=28.40; P<.001). In terms of content and purpose, 6 webpages were supporting or emotional (9.4%), 39 informative (60.9%), and 19 preventive (29.7%), with both emotional and preventive pages being underrepresented (χ22=13.83, contingency coefficient=0.31, P=.001; χ21 for trend=13.67, P<.001). Concerning the supervision or guidance of the websites, 45 (70.3% of the entire sample) were handled by a professional figure (a medical doctor or a psychologist, or an allied health professional). Consequently, the medical supervision or guidance of the webpages was predominant (P=.03). Thirty-eight webpages were targeted to or contained references to adolescents as an age group particularly vulnerable to suicide (59.4%; 26/64 websites referred to older people, 40.6%; P=.37). Forty-two (65.6%) webpages made references to other psychiatric disorders or comorbidities, such as depression (65.6%) or drug use (26.6%), while twenty-two (34.4%) websites did not. Webpages mentioning comorbidities were slightly overrepresented (P=.07). Twenty-three webpages (35.9%) versus forty-one (64.1%) contained figures or pictures related to suicide (not statistically significant, P=.15). Sixty-two webpages (96.9% of the sample) had no access restraint (P<.001).

The average HONcode score was 5.81 (SD 2.70, median 7), whereas the average overall number of likes, comments, and shares was 14,261.56 (SD 60,065.23, median 15.00). The correlation between HONcode and the overall number of likes, comments, and shares was 0.16 (95% CI –0.33 to 0.58; P=.53). Further details are reported in Table 1.

Webpages were generally compliant with the HONcode principles (range 68.8%-100.0%; authoritative principle: 44/64; principle of transparency: 64/64) except for the principle of justifiability (35/64, 54.7%, P=.72) and financial disclosure (40/64, 62.5%, P=.21), as shown in Table 2.

At the univariate analysis (Table 3), the HONcode score was higher in mixed or neutral webpages (P<.001), in websites with a preventive purpose (P=.02), with professional or medical supervision or guidance (P<.001), with references to other psychiatric disorders or comorbidities (P=.003), and without access restraint (P=.03). No statistically significant differences could be detected in terms of references to age groups (adolescents versus older people, P=0.81) or the presence of figures or pictures (P=.41).

Furthermore, webpages with medical supervision or guidance had more an informative and preventive content and purpose than did supporting or emotional ones (χ22=9.19, P=.01; χ21 for trend=3.19, P=.07). Those webpages with professional supervision or guidance also had a more positive or mixed/neutral attitude toward suicide, and this was statistically significant (P=.002). In contrast, no differences were found in terms of references to other psychiatric disorders or comorbidities, references to age groups, access restraint, and the presence of figures or pictures related to suicide.

In the multivariate linear regression analysis (Table 4), the predictors of the HONcode score were professional/medical or supervision/guidance of the webpages (regression coefficient=3.94; P<.001) and references to psychiatric disorders as comorbidities (regression coefficient=1.46; P=.003).

Meanwhile, no differences were found in terms of likes, comments, or shares either in the univariate or the multivariate analyses.

Table 1. Characteristics of the included websites (N=64).
ParameterValue
Attitude toward suicide, n (%)

Prosuicide4 (6.3)

Antisuicide8 (12.5)

Mixed/neutral52 (81.2)
Content and purpose, n (%)

Supporting/emotional6 (9.4)

Informative39 (60.9)

Preventive19 (29.7)
Supervision/guidance, n (%)

Religious figure4 (6.3)

Psychiatrist11 (17.2)

Other kinds of physicians3 (4.7)

Psychologist25 (39.1)

Pharmacist1 (1.6)

Other allied health professionals5 (7.8)

Lawyer1 (1.6)

User-generated webpage11 (17.2)

Volunteer organization/charity3 (4.7)
References to age groups, n (%)

Adolescents38 (59.4)

Older people26 (40.6)
Reference to other psychiatric disorders, n (%)

Depression42 (65.6)

Anxiety11 (17.2)

Panic disorder2 (3.1)

Bipolar syndrome9 (14.1)

Personality disorder8 (12.5)

Posttraumatic stress disorder3 (4.7)

Paranoia2 (3.1)

Schizophrenia12 (18.8)

Drug use17 (26.6)

Alcoholism6 (9.4)

Eating and weight disorders1 (1.6)

Sleep disorders1 (1.6)

Reference42 (65.6)

No reference22 (34.4)
Figures/pictures, n (%)

Yes23 (35.9)

No41 (64.1)
Access restraint, n (%)

Yes2 (3.1)

No62 (96.9)
HONcode score, mean (SD)5.81 (2.7)
Comments, n585
Shares, n98
Likes, n256,025
Table 2. Assessment of the suicide-related websites according to the HONcode principles (N=64).
HONcode principleYesNoP value
1. Authoritative44 (68.8)20 (31.2).047
2. Complementarily48 (75.0)16 (25.0).006
3. Privacy48 (75.0)16 (25.0).006
4. Attribution45 (70.3)19 (29.7).03
5. Justifiability35 (54.7)29 (45.3).72
6. Transparency64 (100.0)0 (0.0)<.001
7. Financial disclosure40 (62.5)24 (37.5).21
8. Advertising policy50 (78.1)14 (21.9).002
Table 3. Univariate analysis of the studied webpages according to the HONcode score.
ParameterHONcode score, mean (SD)P value
Attitude toward suicide<.001

Prosuicide0.75 (0.50)

Antisuicide3.50 (2.93)

Mixed/neutral6.55 (2.07)
Content and purpose.02

Supporting/emotional2.17 (2.93)

Informative6.08 (2.45)

Preventive6.42 (2.22)
Supervision/guidance<.001

User-generated webpage2.63 (2.27)

Professional figures7.16 (1.38)
References to age groups.81

Adolescents6.05 (2.36)

Older people5.46 (3.09)
Reference to other psychiatric disorders.003

No reference4.28 (3.16)

Reference6.41 (2.22)
Figures/pictures.41

Yes6.09 (2.83)

No5.66 (2.60)
Access restraint.03

Yes100 (0.00)

No5.97 (2.57)
Table 4. Multivariate linear regression analysis of the studied webpages according to the HONcode score.
Independent variablesValue

CoefficientSErpartialtP
Constant1.11N/AaN/A N/A N/A 
Content and purpose0.270.200.171.30.20
Reference to age groups0.180.440.060.42.78
Figures/pictures0.400.450.120.89.38
Supervision/guidance3.940.470.758.30<.001
Access restraint–1.991.42–0.19–1.40.17
References to other psychiatric disorders1.460.470.393.12.003
Attitude toward suicide0.520.570.120.91.37

aN/A: not applicable.


Main Findings and Comparison With the Literature

To the best of our knowledge, this is the first study to assess the quality and content of suicide-related webpages in the Italian language.

We found that the percentage of explicitly prosuicide websites was greater than that reported by Wong et al [11] but less than that reported by Chen et al (16.3%), who analyzed 375 linked webpages in Chinese [12]. According to the authors, 41.3% of the included webpages were antisuicide, and the majority of the prosuicide sites were user-generated (96.7%). Interestingly, searches using the search term “ways to kill yourself” (31.7%) and “painless suicide” (28.3%) generated much larger numbers of potentially harmful webpages than did the keyword “suicide” (4.3%).

A much larger portion of prosuicide websites (42%) was found by Sakarya et al [17], who analyzed 100 webpages. Thirteen percent of the studied sample was found to have content or material that may be considered protective against suicidal behavior, a result in line with our findings. However the authors found that “protective” websites did not have any kind of medical supervision or guidance from specialized mental health professionals. In contrast to this, we managed to find a statistically significant association between these 2 variables.

Til and Niederkrotenthaler [18] performed a cross-cultural study by comparing the suicide-related web searches in Austria and the USA. They found that in both countries, protective outweighed harmful website characteristics by approximately 2:1, a finding in line with our result. Authors were able to confirm the observation of Wong et al: websites retrieved with method-related search terms (eg, how to hang yourself) contained more harmful information (USA: P<.001; Austria: P<.05) and fewer protective characteristics (USA: P<.001; Austria: P<.001) compared to the term “suicide” [11]. Conversely, help-related search terms (eg, suicidal thoughts) yielded more webpages with positive and protective characteristics (USA: P=.07; Austria: P<.01). Interestingly, authors found that websites retrieved with the US search engines generally had more protective characteristics (P<.001) than did those retrieved with Austrian search engines. Moreover, resources with dangerous or harmful characteristics were better ranked than those with positive or protective characteristics (USA: P<.01; Austria: P<.05).

In another study, Thornton and colleagues [13] found that among Google searches retrieved related to suicide, a high portion of them were irrelevant webpages (n=136, approximately 26% of the entire sample). Of the 329 relevant websites, the majority were suicide preventive (about 68%); however, a considerable proportion of sites expressed mixed/ambiguous (22%) or neutral (8%) attitudes toward suicide, and 1% were explicitly prosuicide, figures which were slightly different from ours.

These inconsistent literature findings could be explained by the cultural differences across countries in which suicide-related web searches are performed, as shown by the research of Till and Niederkrotenthaler [15], as well as policies concerning the web. Overall, the combination of societal data and online behavior monitoring provide the best indication of risks [19].

The presence of antisuicide online material is of crucial importance for 2 major reasons: (1) according to a recent review of the literature [6], individuals at risk for suicide are probably more prone to look for suicide-related information online, and searching online information related to suicide is a proxy of suicidal ideation; and (2) according to a randomized controlled trial, education professional suicide prevention websites appear to increase and improve suicide prevention–related knowledge, especially among vulnerable and socially isolated individuals [18]. Therefore, exposure to high-quality curated websites may be associated with a reduction in suicidal thoughts and actions. Recent research has shown that the proactive suicide prevention online model is robust for identifying people that are at risk of suicide [20].

Strengths and Limitations of the Study

Despite some strengths, including being the first study to systematically assess the quality of suicide-related webpages in the Italian language, our investigation suffers from some limitations that should be properly acknowledged. The major shortcoming is the small sample size of webpages analyzed and the search limited to the keyword “suicide.” Future research should also explore other words indirectly related to suicide as well as other terms related to the ways suicide can be committed (ie, method-related search terms) and explore whether the semantic differential has an impact on suicide-related webpages in terms of content and quality. Moreover, caution should be used when generalizing the present findings to other countries, as our results could be affected by country-specific cultural issues and may not reflect the nature of other settings. Further, although we assessed the reference to other psychiatric disorders or comorbidities in the webpages included in our study, we were not able to establish a link between mental illness and suicide in terms of browsing websites. This warrants dedicated, ad hoc studies specifically exploring this topic. Another aspect that should be investigated concerns the size or burden of the psychological effect imposed by browsing suicide-related websites.

Conclusions

Sixty-four webpages were included in our analysis: 12.5% (8/64) were antisuicide and 6.3% (4/64) explicitly prosuicide. The majority of the included websites had a mixed or neutral attitude toward suicide (52/64, 81.2%) and had an informative content and purpose (39/64, 60.9%). Most webpages targeted adolescents as an age group (38/64, 59.4%), contained a reference to other psychiatric disorders or comorbidities (42/64, 65.6%), had medical or professional supervision or guidance (45/64, 70.3%), lacked figures or pictures related to suicide (41/64, 64.1%), and did not use any access restraint (62/64, 96.9%).

Specialized mental health professionals should take into account these findings, make an effort to improve their presence online, and provide high-quality material. However, given the aforementioned limitations, further research in the field is warranted.

Conflicts of Interest

None declared.

  1. Sinyor M, Tse R, Pirkis J. Global trends in suicide epidemiology. Curr Opin Psychiatry 2017 Jan;30(1):1-6. [CrossRef] [Medline]
  2. Suicide in the world. Global health estimates. World Health Organization. Geneva, Switzerland; 2019.   URL: https://apps.who.int/iris/bitstream/handle/10665/326948/WHO-MSD-MER-19.3-eng.pdf [accessed 2021-11-02]
  3. Bragazzi NL. A Google Trends-based approach for monitoring NSSI. Psychol Res Behav Manag 2013;7(6):1-8 [FREE Full text] [CrossRef] [Medline]
  4. Durkee T, Hadlaczky G, Westerlund M, Carli V. Internet pathways in suicidality: a review of the evidence. Int J Environ Res Public Health 2011 Oct;8(10):3938-3952 [FREE Full text] [CrossRef] [Medline]
  5. Solano P, Ustulin M, Pizzorno E, Vichi M, Pompili M, Serafini G, et al. A Google-based approach for monitoring suicide risk. Psychiatry Res 2016 Dec 30;246:581-586. [CrossRef] [Medline]
  6. Mok K, Jorm AF, Pirkis J. Suicide-related Internet use: A review. Aust N Z J Psychiatry 2015 Aug;49(8):697-705. [CrossRef] [Medline]
  7. Padmanathan P, Biddle L, Carroll R, Derges J, Potokar J, Gunnell D. Suicide and self-harm related internet use. Crisis 2018 Nov;39(6):469-478 [FREE Full text] [CrossRef] [Medline]
  8. Jacob N, Scourfield J, Evans R. Suicide prevention via the Internet: a descriptive review. Crisis 2014 Jan 01;35(4):261-267. [CrossRef] [Medline]
  9. Christensen H, Batterham PJ, O'Dea B. E-health interventions for suicide prevention. Int J Environ Res Public Health 2014 Aug;11(8):8193-8212 [FREE Full text] [CrossRef] [Medline]
  10. Marchant A, Hawton K, Stewart A, Montgomery P, Singaravelu V, Lloyd K, et al. A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS ONE 2017 Aug 16;12(8):e0181722. [CrossRef]
  11. Wong PW, Fu K, Yau RS, Ma HH, Law Y, Chang S, et al. Accessing suicide-related information on the internet: a retrospective observational study of search behavior. J Med Internet Res 2013 Jan 11;15(1):e3 [FREE Full text] [CrossRef] [Medline]
  12. Chen Y, Hung GC, Cheng Q, Tsai C, Wu KC. Searching for suicide-related information on Chinese websites. Psychiatry Res 2017 Dec;258:506-510. [CrossRef] [Medline]
  13. Thornton L, Handley T, Kay-Lambkin F, Baker A. Is a person thinking about suicide likely to find help on the internet? An evaluation of Google search results. Suicide Life Threat Behav 2017 Feb;47(1):48-53. [CrossRef] [Medline]
  14. Singaravelu V, Stewart A, Adams J, Simkin S, Hawton K. Information-seeking on the internet. Crisis 2015;36(3):211-219. [CrossRef] [Medline]
  15. Till B, Niederkrotenthaler T. Surfing for suicide methods and help. J. Clin. Psychiatry 2014 Jul 08;75(08):886-892. [CrossRef]
  16. HONcode: Principles. HealthOnNet. 1995.   URL: https://www.healthonnet.org/HONcode/Conduct.html [accessed 2021-11-02]
  17. Sakarya D, Güneş C, Sakarya A. Googling suicide: evaluation of websites according to the content associated with suicide. Turk Psikiyatri Derg 2013;24(1):44-48 [FREE Full text] [Medline]
  18. Till B, Tran US, Voracek M, Niederkrotenthaler T. Beneficial and harmful effects of educative suicide prevention websites: randomised controlled trial exploring Papageno v. Werther effects. Br J Psychiatry 2017 Aug;211(2):109-115. [CrossRef] [Medline]
  19. Barros JM, Melia R, Francis K, Bogue J, O'Sullivan M, Young K, et al. The Validity of Google Trends Search Volumes for Behavioral Forecasting of National Suicide Rates in Ireland. Int J Environ Res Public Health 2019 Sep 02;16(17):3201 [FREE Full text] [CrossRef] [Medline]
  20. Liu X, Liu X, Sun J, Yu NX, Sun B, Li Q, et al. Proactive suicide prevention online (PSPO): machine identification and crisis management for Chinese social media users with suicidal thoughts and behaviors. J Med Internet Res 2019 May 08;21(5):e11705 [FREE Full text] [CrossRef] [Medline]


WHO: World Health Organization


Edited by G Eysenbach; submitted 28.03.21; peer-reviewed by M Aljassas, R Treskes; comments to author 03.05.21; revised version received 11.08.21; accepted 01.10.21; published 11.11.21

Copyright

©Wen Chen, Andrea Boggero, Giovanni Del Puente, Martina Olcese, Davide Prestia, Haitham Jahrami, Nasr Chalghaf, Noomen Guelmami, Fairouz Azaiez, Nicola Luigi Bragazzi. Originally published in JMIR Formative Research (https://formative.jmir.org), 11.11.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.