Estimation of Psychological Distress in Japanese Youth Through Narrative Writing: Text-Based Stylometric and Sentiment Analyses

Introduction

Background

Adolescents often display premonitory symptoms of mental illnesses arising from psychological distress (such as depression and anxiety-related disorders) [1-3]. Noticing the onset of such symptoms is important to facilitate effective treatment and diagnosis. Any delay between the onset of psychopathological symptoms to diagnosis and clinical treatment can result in a worsening of the condition, leading to complications not only for the individual’s mental health but also with respect to their physical health and social relationships [4]. In particular, youth tend to report more symptoms of internalizing mental illnesses that are related to psychological distress [5] and accompanying physical or vegetative changes (eg, weight and appetite changes) compared to adults. Consequently, as with any psychopathology, early access to medical and psychological assistance is important for the recovery process. Accordingly, the development of methods for the early detection of psychological distress in youth is of crucial importance.

Text-Based Screening

In recent years, the rapid development of natural language processing (NLP) technology has enabled screening for dementia or depression through examinations of language use in free-text tasks [6-8]. One proposal to increase the effectiveness of such examinations is to limit and control the topics of these free-text tasks. This would reduce the occurrence of situations where differences in language use are due to differences in writing tasks and topics, which may confound language use detection with psychopathology. For example, the written narrative in recollecting a painful experience may be very different from the narrative recounting an interesting episode heard from a friend. Regardless of the underlying mental health of both instances, it is highly likely that different language and terms would be used in the recounting of these memories.

In this study, we adopted NLP techniques for detecting psychological distress disorders in young Japanese writers, and then describe possible language indicators that may reflect these differences. Accordingly, we consider the characteristics of youth with mental illness, and propose that writing fictional, imaginative narratives may appropriately allow for the generation of content that facilitates the detection of psychological distress.

Psychopathology and Autobiographical Narratives

Prior research on narrative writing and psychopathology detection has focused on autobiographical narratives stemming from self-disclosure. For example, when a patient with depression (ie, major depressive disorder) recalls their own experience, they tend to rely on general or repeated memory (eg, I played a game with friends last week) rather than concrete memory (eg, I played volleyball with Sato last Thursday) or extended memory (eg, looking at this souvenir reminded me of my cruise trip last winter). This phenomenon is referred to as overgeneral autobiographical memory [9-11] and has been well-documented even in youth with depression [12].

However, detection of psychopathology in autobiographical self-disclosure–based narratives may not be as effective in Japan, given that the Japanese are generally less accustomed to self-disclosure [13,14]. Furthermore, self-disclosure on online platforms comes with risks such as cyberbullying and social isolation [15]. Thus, an alternative method of detection is needed. Toward this end, we propose the use of fictional, imaginative writing that does not require any form of self-disclosure. As a first step, such research would need to examine which part of the narrative affords detection of psychological distress in the writer in detail. We consider that story creation relies on one’s past experiences and memory, but in a transformed, unrecognizable manner, resembling a generalized experience that can be shared with others.

Imaginative Narratives as Writing Topics

We propose that psychological distress in the writer may be observable through their written narratives. In other words, topic questions should encourage respondents to engage in creative, imaginative writing. Imagination, or fantasy, relies on stories and retrievals of one’s previous experiences that are combined and reworked based on certain elements of these past experiences in a creative manner to generate new propositions and scenarios [16].

We consider that creativity in writing imaginative narratives is important for psychological distress detection. A meta-analysis on trait creativity and psychopathology showed that mild psychopathology is associated with increased creativity, to the extent that it does not impair day-to-day functioning [17]. Similarly, external ratings of creativity in visual art were found to be associated with dehydroepiandrosterone-sulfate (DHEA-S), an endocrine marker of depression. Specifically, lower levels of DHEA-S (indicating an increased risk of depression) were predictive of higher creativity ratings, and this relationship was significantly moderated by the individual’s emotional vulnerability to social rejection or acceptance [18].

Accordingly, individuals’ mental health states should be reflected in the creativity of their narrative content, and analysis of these narratives may function as effective early detectors of psychological distress. Furthermore, imaginative free-text writing tasks may be more suited for mental illness detection in children and adolescents. As creativity refers to the ability to imagine new and useful things or situations [19], it involves imagining something that does not presently exist in reality. Thus, we propose that by constraining free-text narrative tasks to imaginative writing, we can effectively detect adolescent writers with possible psychopathological tendencies through examining the creativity of the written narratives as quantified by the amount of variation in their vocabulary and language use.

Objectives

Based on the hypothesis outlined above, our research focuses on the detection of psychological distress in written imaginative narratives. To quantify these variations in vocabulary and language use, we applied NLP tools such as stylometric measures and lexicon-based sentiment analysis to analyze these texts. Although the act of analyzing stories itself is not a new concept, the use of NLP tools to quantify qualitative text in analyzing imaginative narratives remains a challenging approach. Nevertheless, we propose that written stories from youth at high and low risk of psychological distress can be differentiated through stylometric text analysis methods.

Methods

Participants

A total of 634 participants were initially recruited from Yahoo! Crowdsourcing, a Japanese online crowdsourcing platform. Participants were reimbursed 5 yen (~US $0.05) through Yahoo! Crowdsourcing. After excluding participants with meaningless words and reprints of copyrighted works in their responses, a total of 629 (267 males, 335 females, 27 no disclosed gender; mean age 40.9 years, SD 12.3) participants were identified. Note that age was approximated from the mean of the class (group) data, since we only collected information on the age group. Of these participants, we further narrowed our focus to responses from young participants. Following the World Health Organization definition of “youth” as individuals aged between 15 and 25 years, we examined a final 52 participants (14 males, 29 females, 9 did not disclose gender; mean age 19.8 years, SD 3.1). Although the main focus of the analysis was on youth, we also examined supplementary data on 577 adults (253 males, 306 females, 18 did not disclose gender; mean age 42.4 years, SD 12.3) to allow for the possibility of identifying youth-specific characteristics. During the first 2 weeks, we limited recruitment to users under the age of 25 years, but had low participation rates. Therefore, for the next 2 weeks, we recruited users without any age restriction and then selected only users of the target age group. As a result, we collected data from the 52 youth and 577 adults. The adult data were not intended to be part of our research, but showed sufficient potential to provide a developmental perspective, and therefore were included for use in the supplemental analysis.

Materials

Participants were first instructed to “write an introduction to a story, with at least 200 (Japanese) characters.” This was an open-ended response task, designed to let participants engage in narrative writing by creating the introduction to an imagined story. Subsequently, they were assessed for psychological distress using the Kessler Psychological Distress Scale (K-10), a brief 10-item test widely used in screening mental illness (eg, in the World Health Organization World Mental Health Survey [20]). The cutoff value for K-10 was set at 30, which is a reference value to indicate a respondent’s state of severe psychological distress. Participants with a K-10 score ≥30 were assigned to the higher distress group and the remaining participants were assigned to the lower distress group.

NLP Measures

We adopted stylometric measures for authorship detection that have been shown to have relationships with the attitudes and psychological tendencies of authors. We utilized 12 types of stylometrics, as listed in Table 1, based on Japanese text metrics organized in Asaishi [21]. Additionally, we performed sentiment analyses to examine the ratio of specific emotion terms (eg, happiness, surprise, anger; see Table 1) to the total number of terms in the text.

To reduce confounds related to story length, we limited the open-ended responses to include only the first 200 characters while preserving the sentence unit.

That is, we kept the maximum number of the introductory sentences if the last sentence (a) did not exceed the 200-character limit or (b) exceeded the limit by fewer characters than the characters from the end of its previous sentence. During this process, sentences were separated by punctuation marks, except for those inside parentheses.

Because some metrics require grammatical information, we applied shallow natural language parsing techniques. Tokenization and part-of-speech tagging were processed through the morphological analyzer MeCab [22].

Note that Japanese text is not tokenized by authors with spaces. CaboCha [23] was used for syntax parsing.

Results

The results of the comparison of language indicators between youth and adults are shown in Table 2. See Multimedia Appendix 1 for examples of narrative writing in youth and adults.

In contrast to our hypothesis, participants in the higher distress group did not show significant increases in word richness or diversity. Most of the stylometrics that examined variation in word use (eg, type token ratio) did not significantly differ between groups. In exploring additional differences in language content between the higher and lower distress groups for youth, significant differences were observed in emotion terms for happiness-related word ratios. This suggests that narratives written by participants in the higher distress group were more likely to use happiness-related phrases and words than those in the lower distress group. No other significant differences were observed.

Although outside the scope of this study, we also carried out a supplementary analysis of narratives written by adults using the same procedure. In the adult participants, there were significant differences between higher and lower distress groups in content words, sadness, and the mean number of words per sentence clause. The higher distress group used fewer content words, fewer emotion words about sadness, and fewer words per chunk than the lower distress group (Table 2).

Discussion

Narrative Language Features and Youth Mental Health

We found significant differences between the lower and higher distress groups, particularly with regard to the relative frequency of happiness-related terms. Happiness is commonly viewed as a positively valenced emotion across cultures [28]. At a glance, the increased usage of happiness-related words in the higher distress group suggests that youth with increased tendencies to psychological distress or illness may prefer writing more positive narratives. Considering that the concept of happiness is not typically associated with psychological distress, our result suggesting the stronger presence of happiness-related terms in the higher psychological distress group appears contradictory. Nevertheless, we posit two potential explanations for this result. First, these findings may reflect the possibility that youth with higher distress prefer happier stories. Alternatively, these findings may suggest that more frequent priming of happiness may induce psychological distress in youth. Although a causal relationship cannot be investigated from this single cross-sectional study, we speculate on several interpretations. Prior research on self-directed narratives has revealed an association with depression. For example, those who tended to imagine a positively inclined future for themselves had lower depression measures (eg, Center for Epidemiological Studies Depression Scale, Children’s Depression Inventory, Beck Depression Inventory) at the time of their imagination, but in subsequent follow-up, they were found to have higher tendencies for depression than the group that had anticipated a less positive future [29].

By disambiguating these two concepts, one possibility could be that participants did not need to consider their “self” (current circumstances) in creative writing, and were able to imagine freely without these constraints. Thus, the increased positive affect (happiness scores) in their writing may suggest that this was more idealized or desired. Such an interpretation would be consistent with past research linking an increased desire for happiness with depression [30].

However, we reiterate that these explanations are speculations based on the pattern of results obtained from our data. More research is needed to confirm the applicability of these explanations in creative writing for individuals with tendencies toward psychological distress.

Type Token Ratio and Word Abstraction

Our results showed no significant difference in vocabulary usage, measured in this study through indicators such as the type token ratio, nor in the abstraction of the words for the narrative passages. One explanation could be that the memories of one’s own experiences do not necessarily affect the generation of a story. Differentiation according to the level of word abstraction alone is insufficient, because narratives may be conceptualized separately from one’s own experiences, referencing the collective knowledge from cultural media and the experiences of others.

Pros and Cons of Crowdsourcing

Several previous studies have shown that crowdsourcing is an appropriate tool for recruiting research participants [31,32]. Nevertheless, we acknowledge some problems with crowdsourcing data collection. For example, participants may attempt the same questionnaire twice, and incorrect comprehension of the instructional text may be problematic due to participation from nonnative speakers. Another problem we had to face was the phenomenon of “satisficing,” in that participants may tend to conserve cognitive resources in survey research [33]. In crowdsourcing, people are often motivated to work on multiple tasks in as short a time as possible to increase their monetary reward because of low unit costs.

In this study, we took steps to reduce these problematic effects from crowdsourcing by using a service in which native speakers (in this case, Japanese speakers) form the vast majority of users. We also excluded duplicated IDs and data (narratives) that were directly lifted from copyrighted works or lists of meaningless words. Although we cannot definitively rule out any bias or duplicate participants in our research, precautions were taken to safeguard against these potential problems.

Limitations and Future Directions

We note several limitations of our study. First, we had limitations with regard to the age of our sample. The crowdsourcing platform had a minimum age of 15, meaning that adolescents under 15 years old were not included in this study. Second, we used the K-10 for screening purposes, which is a general questionnaire and lacks sensitivity to specific diagnoses. Thus, there is a possibility that different types of psychopathological tendencies may exhibit differentiated effects or underlying mechanisms. Future studies should consider the use of actual clinical diagnoses as the criteria for mental illness beyond crowdsourced convenience sampling.

Finally, the linguistic indicators used in this study are widely used for author estimation and are usually applied to large volumes of text. In this study, narrative sentences of about 200 characters each were used, which may be insufficient for detecting the presence of true effects. Consequently, we are unable to rule out the use of creativity measures in imaginative writing as a means for detecting psychological distress, as our lack of a significant finding in this aspect can also be explained by an insufficiency in the length of our narrative data.

Nevertheless, our study lays the groundwork for psychological distress surveillance programs in sensitive populations, especially in situations where recollection of personal, self-related narratives or self-disclosure may be problematic or risky. This may be more relevant in the Japanese context. Japanese undergraduate students were less likely to self-disclose experiences of bullying compared to a US sample, and this was fueled by a concern for disrupting social and relational harmony [34]. As such, surveillance methods that rely on fictional narrative content without reliance on self-disclosure may be more suitable in a collectivistic Japanese context. Our exploratory result identifies happiness and surprise-related content as potential indicators of psychological distress in the writer. Future research should confirm these findings on a larger scale, with a larger, preregistered study involving more diverse samples and cross-cultural comparisons. Narrative writing may even provide therapeutic benefits for individuals suffering from depression [35], and future studies can also quantify the effectiveness of such methods in randomized controlled trials.

Conclusions

Youth with tendencies toward mental illness were found to write more positive stories that contained more happiness-related terms. Although the mechanisms underlying these differences in frequency of happiness-related term usage are speculative at present, these results may potentially have more widespread implications on screening, particularly in cultures such as Japan that are not accustomed to self-disclosure. This is a preliminary finding and more confirmatory research is needed to establish the robustness of these results.