This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2226) Natural Language Processing (650) Violence including Domestic Violence/Abuse (107) Behavior Change (655) Innovations in Mental Health Systems (371) Mindfulness and Meditation (197) Posttraumatic Stress Disorder (PTSD) (107)

Published on in Vol 4, No 9 (2020): September

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/17424, first published .
Natural Language Processing Tools for Assessing Progress and Outcome of Two Veteran Populations: Cohort Study From a Novel Online Intervention for Posttraumatic Growth

Natural Language Processing Tools for Assessing Progress and Outcome of Two Veteran Populations: Cohort Study From a Novel Online Intervention for Posttraumatic Growth

Natural Language Processing Tools for Assessing Progress and Outcome of Two Veteran Populations: Cohort Study From a Novel Online Intervention for Posttraumatic Growth

Authors of this article:

Kim P Norman1 Author Orcid Image ;   Anita Govindjee2 Author Orcid Image ;   Seth R Norman3 Author Orcid Image ;   Michael Godoy1 Author Orcid Image ;   Kimberlie L Cerrone3 Author Orcid Image ;   Dustin W Kieschnick1 Author Orcid Image ;   William Kassler2 Author Orcid Image
Article Authors Cited by (10) Tweetations (4) Metrics

    Original Paper

    1Young Adult and Family Center, UCSF Weill Institute for Neurosciences, University of California, San Francisco, CA, United States

    2IBM Watson Health, IBM Corporation, Palo Alto, CA, United States

    3Tiatros, Inc, San Francisco, CA, United States

    *all authors contributed equally

    Corresponding Author:

    Kim P Norman, MD

    Young Adult and Family Center

    UCSF Weill Institute for Neurosciences

    University of California

    401 Parnassus Ave

    San Francisco, CA, 94143

    United States

    Phone: 1 415 476 7402

    Email: kim.norman@ucsf.edu


    Background: Over 100 million Americans lack affordable access to behavioral health care. Among these, military veterans are an especially vulnerable population. Military veterans require unique behavioral health services that can address military experiences and challenges transitioning to the civilian sector. Real-world programs to help veterans successfully transition to civilian life must build a sense of community, have the ability to scale, and be able to reach the many veterans who cannot or will not access care. Digitally based behavioral health initiatives have emerged within the past few years to improve this access to care. Our novel behavioral health intervention teaches mindfulness-based cognitive behavioral therapy and narrative therapy using peer support groups as guides, with human-facilitated asynchronous online discussions. Our study applies natural language processing (NLP) analytics to assess effectiveness of our online intervention in order to test whether NLP may provide insights and detect nuances of personal change and growth that are not currently captured by subjective symptom measures.

    Objective: This paper aims to study the value of NLP analytics in assessing progress and outcomes among combat veterans and military sexual assault survivors participating in novel online interventions for posttraumatic growth.

    Methods: IBM Watson and Linguistic Inquiry and Word Count tools were applied to the narrative writings of combat veterans and survivors of military sexual trauma who participated in novel online peer-supported group therapies for posttraumatic growth. Participants watched videos, practiced skills such as mindfulness meditation, told their stories through narrative writing, and participated in asynchronous, facilitated online discussions with peers. The writings, including online postings, by the 16 participants who completed the program were analyzed after completion of the program.

    Results: Our results suggest that NLP can provide valuable insights on shifts in personality traits, personal values, needs, and emotional tone in an evaluation of our novel online behavioral health interventions. Emotional tone analysis demonstrated significant decreases in fear and anxiety, sadness, and disgust, as well as increases in joy. Significant effects were found for personal values and needs, such as needing or desiring closeness and helping others, and for personality traits of openness, conscientiousness, extroversion, agreeableness, and neuroticism (ie, emotional range). Participants also demonstrated increases in authenticity and clout (confidence) of expression. NLP results were generally supported by qualitative observations and analysis, structured data, and course feedback.

    Conclusions: The aggregate of results in our study suggest that our behavioral health intervention was effective and that NLP can provide valuable insights on shifts in personality traits, personal values, and needs, as well as measure changes in emotional tone. NLP’s sensitivity to changes in emotional tone, values, and personality strengths suggests the efficacy of NLP as a leading indicator of treatment progress.

    JMIR Form Res 2020;4(9):e17424

    doi:10.2196/17424

    Keywords

    natural language analysis; emotional tone; personality; values; PTSD; military sexual trauma; online interventions; internet-based cognitive behavioral therapy; narrative therapy; mindfulness 


    The lifetime risk of acquiring a mental illness diagnosis is 50%, yet over 100 million Americans lack affordable access to effective behavioral health care [1]. The economic costs of untreated mental illness are estimated in the hundreds of billions of dollars per year [2]. American military veterans are an especially vulnerable population. Up to 18% of Operation Enduring Freedom/Operation Iraqi Freedom veterans have posttraumatic stress disorder, and 25% have depression [3,4]. Furthermore, up to 41% of women and 4% of men are subjected to military sexual trauma, including sexual violence, sexual coercion, and severe and persistent sexual harassment [5]. It is estimated that 1.5 million of the 5.5 million veterans seen in US Department of Veterans Affairs (VA) hospitals in 2016 had a mental health diagnosis, and lack of access to mental health care is believed to contribute significantly to the high rate of suicide among US veterans, which tragically remains at 20 lives lost per day [6]. It is believed that at least half of veterans who need mental health care do not get the care they need, and more than half of those who would benefit from mental health care are unaware they need it [7].

    With or without a mental health diagnosis, most returning veterans struggle with the transition to civilian life, suffering painful and sometimes debilitating symptoms even if they do not meet full criteria for a diagnosis [8]. One factor that exacerbates the difficulties of transition is the sense of profound loss created by being disconnected from the military community [9,10]. For example, veterans transition from almost never being alone to feeling all alone, leaving a life with a clearly established identity, culture, mission, purpose, and ever-present peer support to suddenly having to reinvent themselves. As a result, they tend to isolate [11] and consequently typically believe they are the only ones who feel this way. It can lead to loss of meaning and purpose, resulting from the stark contrast of going from high-intensity deployment, where every decision matters, to civilian life, where decisions seem much more trivial if they matter at all. The failure to address the special needs of veterans may explain why, sadly, real-world completion rates for in-person VA-approved therapies for posttraumatic stress disorder (PTSD) are 2% [12].

    Real-world programs to help veterans successfully transition to civilian life must reach the many veterans who cannot or will not access care. These programs must have the ability to scale, as there is a shortage of services available to veterans, and they must address the loneliness that veterans experience while helping them reestablish their sense of mission, purpose, and connectedness. Therefore, successful interventions must also build a sense of community.

    In an effort to expand access to care, digitally based behavioral health initiatives have emerged within the past few years. Most mental illnesses, including anxiety, depression, PTSD, and addiction, are effectively treated by cognitive behavioral therapy (CBT) [13]. The knowledge base of CBT best practices lends itself to scale. Internet delivery of CBT via therapist-mediated video chat and mixtures of online therapy sessions with email homework assignments are as effective as in-person therapies [14-18]. However, while improving reach, these approaches offer little or no scale [19]. Completely automated, open access, self-paced internet protocols offer scale, but have very low completion rates of around 1% [20,21]. Guided self-help, where clinicians offer support via brief phone calls or emails, have completion rates of 50% or higher and outcomes equal to in-person therapies [17]. Such interventions, however, rely heavily on the contacts between research associates and research participants, and the actual level of guidance provided is not usually discussed in the methodologies of these studies. Furthermore, few trauma-focused intervention protocols incorporate peer or social support, which has been identified as a protective factor against onset of PTSD and an integral part of recovery [22]. Additionally, an overlooked aspect of recovery from trauma is incorporation of military values. Personal accountability and group accountability are highly valued characteristics in military culture [23], yet rarely used in the treatment of veterans. Our novel behavioral health intervention teaches mindfulness-based CBT and narrative therapy using peer support groups as guides, with human-facilitated asynchronous online discussions. Our intervention attempts to address the aforementioned difficulties of online treatment—those of scale, lack of social support, and heavy reliance on clinical contact—while incorporating group accountability values that resonate with veterans.

    Historically, the standard of assessing efficacy of a therapeutic intervention has been through administration of subjective symptom measures at various time points (usually pretreatment and posttreatment) [24-26]. These measures generate quantitative scores that become structured data [27]. Our online intervention, by virtue of administering written narrative and CBT exercises, generated large amounts of unstructured qualitative data that, in the course of successive weekly sessions, was longitudinal. Unstructured, qualitative data, while not generally the standard of intervention evaluation, may contain insights and nuances of personal change that are not captured by subjective symptom measures. Furthermore, data collection for posttreatment evaluation tends to suffer lower completion rates; thus, ground truth information about intervention effectiveness can be lost. The technological advancement of natural language processing (NLP) provides opportunities to generate insights in narrative- and CBT-based interventions by examining participants’ own language. In fact, use of NLP is a practice in the artificial intelligence (AI) field for attempting to cultivate emotional intelligence in conversational robotics [28]. Validity studies have shown that NLP reliably predicts personality characteristics, personal needs, and values, and monitors emotional tone and style of communication [29,30]. NLP offers the promise of enhancing scale by giving participants immediate feedback while providing clinicians with real-time, actionable observations. At the same time, NLP supplies researchers the data needed to assess the impact of the behavioral health intervention.

    The purpose of our study was to evaluate the feasibility and utility of NLP in evaluating change in a small pilot study of 2 veteran populations who completed 2 novel online behavioral health interventions.


    Participants

    A total of 23 participants were recruited for 2 studies of veterans, the Next Mission (NM) program and Women Warriors (WW) program. A total of 13 participants enrolled in the 14-week NM program, and 10 enrolled in the 8-week WW program. Inclusion criteria for the study were: (1) must be a military veteran, (2) must be aged 18 or older, (3) must speak English, (4) must be able to access the internet regularly, and (5) for the WW program, must be a woman. In the NM program, 11 of the 13 (85%) participants were men, while the WW program consisted of all (10/10, 100%) female participants. In total, 16 participants completed the courses, 9 in the NM program and 7 in the WW program.

    Procedures

    Participants were recruited via Facebook and LinkedIn ads and could sign up on a mobile, laptop, or desktop device of their choice. Recruitment was conducted over several months until enough participants were able to form cohorts for each course. The recruitment messages attempted to account for potential stigma by focusing on building resiliency skills and promoting posttraumatic growth rather than treating mental illness. Participants were informed that they would be helping others in their group while also getting help for themselves.

    Participants were also incentivized by the opportunity to earn University of California college credits for their work, although only 3 of 16 (19%) participants took advantage. Participants understood that the program was fully compliant to the Health Insurance Portability and Accountability Act (HIPAA) and that they could choose to participate anonymously. However, they also had the option of revealing their identity to the group at any time they wished during the course. Both programs, which we also refer to as courses, were entitled “Stress, Resiliency and Post Traumatic Growth.” The constellation of the groups was generationally diverse and allowed older veterans to connect with younger veterans.

    In each program, participants watched videos teaching principles of CBT, narrative therapy, behavioral activation, and mindfulness meditation. They submitted written homework, including journal entries and thought and mood logs, and participated in asynchronous discussions, all within a HIPAA-secure environment. Approximately 90 minutes of online class time and one hour per week of homework was completed by each participant. The asynchronous discussions were facilitated and monitored by a licensed, doctoral-level therapist. Of note, facilitators for the WW group were female. The content for the programs was created in various commercially available proprietary applications by faculty members in the Department of Psychiatry of a US University Medical Center and then assembled into the NM and WW programs and delivered on a commercially available proprietary platform.

    Measures

    Linguistic Inquiry and Word Count

    Linguistic Inquiry and Word Count (LIWC) is software designed to analyze word use within written text. It calculates the percentage of usage for sets of words, arranging them in 80 linguistic categories and generating output statistics for each of the categories [31,32]. LIWC uses a proprietary set of algorithms to produce 4 summary variables from the data: analytical thinking, clout, authenticity, and emotional tone [33]. Analytical thinking measures the degree to which words are used that suggest logical and analytic thinking patterns. Clout measures the characteristic of speaking from the perspective of high expertise and confidence. Authenticity measures the characteristic of more honest, personal, and self-disclosing language. Emotional tone measures the characteristic of expressing more positive emotions in an upbeat style and expressing less anxiety, sadness and hostility. LIWC has been shown across numerous studies to have internal consistency and validity [34].

    IBM Watson Personality Insights

    IBM Watson is a computer system that uses artificial intelligence to interpret unstructured data within natural language. IBM Watson’s Personality Insights is programmed to analyze natural language input and provide outputs of personality characteristics based on 3 models [35]: (1) the Big Five personality traits (openness, conscientiousness, extroversion, agreeability, and neuroticism) [36], (2) personal needs, and (3) personal values. Personal needs describe 12 dimensions of personality aspects that are likely to resonate with the participant [37]: excitement, harmony, curiosity, ideal, closeness, self-expression, liberty, love, practicality, stability, challenge, and structure. Personal values [38] describe 5 types of values that are important to an individual and likely to influence decision-making behavior [39]: helping others, tradition, life pleasure, achievement, and openness to change. Outputs generated from Personality Insights are likelihood ratios between 0 and 1 and are data driven, meaning an output for a variable is only produced if detected. A variable with a score above 0.5 indicates a greater than average tendency for the characteristic to be true [40]. Table 1 lists the definitions of detected levels of needs and values.

    Table 1. Definitions of personal needs and values outputs.a
    CharacteristicDescription
    Needs

    		ExcitementEmphasizes importance of getting out and living life, oriented toward having fun

    		HarmonyAppreciation for other people, their viewpoints, or feelings

    		CuriositySeeking discovery and desire for personal growth

    		IdealWanting perfection and seeking sense of community

    		ClosenessValuing connectedness with others

    		Self-expressionEmphasizes the importance of expressing oneself and asserting individual identities

    		LibertyHave a desire for fashion and new things, as well as the need for escape and freedom

    		LoveValuing social contact, either one-to-one or one-to-many

    		PracticalityHaving a desire to accomplish things, a desire for skill and efficiency, including physical expression and experience

    		StabilityValuing sensibility, equivalence, and balance

    		ChallengeHaving desire to succeed and take on challenges

    		StructureExhibit a grounded trait and a desire to hold things together. They need things to be well organized and under control
    Values

    		Helping othersShowing concern for the welfare and interests of others

    		TraditionEmphasizes self-restriction, order, and resistance to change

    		Life pleasureSeek pleasure and sensuous gratification for themselves

    		AchievementSeek personal success for themselves

    		ExcitementEmphasize independent action, thought, and feeling, as well as a readiness for new experiences

    aTable content was taken and aggregated from IBM Watson Personality Insights [29].

    IBM Watson Tone Analyzer

    IBM Watson Tone Analyzer is an artificial intelligence–enabled text analysis tool produced by IBM Watson that uses AI to infer emotional tone through written text. Tone Analyzer is based on psycholinguistics theory and examines how day-to-day word usage correlates to manifest emotions [30], based on the International Survey on Emotion Antecedents and Reactions [41] data set. Tone Analyzer has undergone studies for consistency and validity against human analysis of emotion and shown no statistically significant difference between human-labelled emotion of text and Tone Analyzer analysis of emotional tone [30]. Tone Analyzer–generated data outputs of the 5 basic emotions enabled us to evaluate the presence of 5 basic emotions: anger, disgust, fear, joy, and sadness [42].

    Qualitative Observations and Questionnaires

    Participants reported feedback from the course and filled out structured subjective symptom questionnaires. Structured data were not analyzed quantitatively in comparison with NLP due to variations of sample size and low completion rates. As such, quantitative changes in structured data were observed qualitatively. Facilitators also reported subjective qualitative observations of participant progress. Structured measures included the Positive States of Mind Scale (PSOM) [43], Posttraumatic Growth Inventory (PTGI) [44], PTSD checklist for the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (PCL-5) [45], Patient Health Questionnaires (PHQs) [46], Short Warwick-Edinburgh Mental Well-Being Scale (SWEMWS) [47], and the Brief Coping Orientation to Problems Experienced (COPE) [48].

    Statistical Analysis

    Participants’ writing samples and online posts were analyzed using the LIWC [33], IBM Watson’s Personality Insights [49], and Tone Analyzer [50]. Detection scores (NLP outputs) were generated as outputs, upon which we conducted a pretest-posttest analysis of these language attributes at the beginning and end of each course. Analysis was conducted in IBM SPSS Statistics (version 26). Initially, repeated-measures 1-tailed t tests were run and effect sizes were calculated using Cohen d. However, to reduce the risk of type I statistical error from multiple t tests, variables were grouped into 4 multivariate categories: (1) LIWC variables, (2) personality trait variables, (3) personality values and needs variables, and (4) emotional tone variables. Omnibus testing in the form of repeated-measures multivariate analysis of variance (rMANOVA) was conducted with Bonferroni correction [51]. Effect sizes were calculated as partial η2 [52] and Cohen d effect sizes were retained from t tests.


    Overview

    In total, 16 of the 23 (70%) participants completed the courses and 15 were able to have their unstructured data analyzed. Results of all rMANOVA analyses on NLP outputs are summarized in Table 2.

    Table 2. Results of natural language analysis for personality traits, values, emotional tone, and Linguistic Inquiry and Word Count for treatment groups of Next Mission combat veterans and Women Warriors military sexual assault survivors.a
    CharacteristicNMb combat veterans (n=9)WWc military sexual assault survivors (n=6)
     F testd (df)Cohen dP valueEffect size (partial η2)F testd (df)Cohen dP valueEffect size (partial η2)
    Personality traits








    		Openness6.372 (1,8)1.34.040.4439.740 (1,5)1.47.030.661

    		Agreeableness10.305 (1,8)1.77.010.77712.211 (1,5)1.68.020.709

    		Extroversion12.446 (1,8)1.52.0010.6097.105 (1,5)–1.58.0450.587

    		Conscientiousness27.923 (1,8)–2.83.0010.5631.929 (1,5)–0.53.220.278

    		Emotional range15.473 (1,8)–1.74.0040.6591.1014 (1,5)0.58.360.169
    Values and needse        

    		Curiosity3.227 (1,8)0.84.110.2875.971 (1,5)1.44.060.544

    		Harmony7.787 (1,8)1.42.020.493N/AfN/AN/AN/A

    		Structure1.865 (1,8)–0.44.210.1892.784 (1,5)0.87.160.358

    		Closeness17.672 (1,8)2.23.0030.68810.006 (1,5)1.23.030.667

    		Stability6.518 (1,8)1.49.030.449N/AN/AN/AN/A

    		Helping others21.715 (1,8)2.06.0020.7313.077 (1,5)0.35.140.381

    		Excitement23.635 (1,8)2.34.0010.7470.391 (1,5)0.15.560.072

    		Life pleasure18.926 (1,8)1.92.0020.7038.511 (1,5)1.30.030.630

    		Tradition9.005 (1,8)–1.54.020.5301.238 (1,5)0.12.320.198

    		Achievement9.414 (1,8)1.01.020.5417.294 (1,5)1.28.040.593

    		LoveN/AN/AN/AN/A13.474 (1,5)1.58.010.729

    		IdealN/AN/AN/AN/A3.173 (1,5)1.03.140.388
    Emotional tone        

    		Sadness10.852 (1,8)–1.01.010.5761.164 (1,6)–0.20.330.283

    		Disgust7.660 (1,8)–1.25.020.4891.413 (1,5)–0.74.290.220

    		Joy11.017 (1,8)1.81.010.5791.977 (1,5)0.84.220.283

    		Fear0.114 (1,8)–0.17.740.0144.365 (1,5)–0.80.090.466

    		Anger0.542 (1,8)–0.32.480.0630.518 (1,5)–0.40.500.094
    LIWCg analysis        

    		Analytical thinking0.229 (1,8)0.20.650.0280.094 (1,5)–0.12.770.018

    		Authenticity7.326 (1,8)0.92.030.4785.457 (1,5)0.37.070.522

    		Clout8.651 (1,8)0.90.020.5207.920 (1,5)1.47.040.613

    		Emotional tone6.016 (1,8)0.92.040.4294.597 (1,5)1.17.090.479

    aN=15.

    bNM: Next Mission.

    cWW: Women Warriors.

    dF values reported are from repeated-measures multivariate analysis of variance (rMANOVA), with effect sizes calculated using both Cohen d from repeated-measures t tests and partial η2 from rMANOVA tests. Significance calculated from rMANOVA test with Bonferroni correction.

    eFor each program group, 10 identified values were outputted by IBM Watson Personality Insights based on highest density, thus differ slightly between programs.

    fN/A: not applicable.

    gLIWC: Linguistic Inquiry and Word Count.

    NM Program

    A total of 9 of the 13 (70%) Next Mission participants completed the course and were able to have their unstructured data analyzed. Results of analysis on NLP data were grouped into 4 outcome variable groups: (1) personality traits, (2) personal values and needs, (3) emotional tone, and (4) LIWC. Results from rMANOVA were reported as univariate analyses. Cohen d effect sizes from initial t tests were retained and reported.

    Analysis of Personality Traits

    Participants showed significant increases in openness (F1,8=6.37; d=1.34; P=.03; partial η2=0.44), agreeableness (F1,8=10.3; d=1.77; P=.01; partial η2=0.78), and extroversion (F1,8=12.46; d=1.52; P=.02; partial η2=0.609), while conscientiousness (F1,8=27.92; d=–2.83; P=.001; partial η2=0.563) and emotional range (F1,8=15.47; d=–1.74; P=.01; partial η2=0.659) significantly decreased.

    Analysis of Personal Values and Needs

    Participants showed significant increases in the values and needs of helping others (F1,8=21.72; d=1.98; P=.002; partial η2=0.73), closeness (F1,8=17.67; d=2.23; P=.003; partial η2=0.69), excitement (F1,8=23.63, d=2.34; P=.001; partial η2=0.75), life pleasure (F1,8=18.93; d=1.92; P=.01; partial η2=0.70), harmony (F1,8=7.79; d=1.42; P=.01; partial η2=0.493), achievement (F1,8=9.41; d=1.01; P=.05; partial η2=0.54), and stability (F1,8=6.52; d=1.98; P=.02; partial η2=0.45). Furthermore, participants showed a significant decrease in valuing tradition (F1,8=9.01; d=–1.53; P=.03; partial η2=0.53).

    Curiosity (F1,8=3.23; d=0.84; P=.24; partial η2=0.28) increased but was not statistically significant. Likewise, structure (F1,8=1.87; d=–0.44; P=.10; partial η2=0.19) decreased but was not statistically significant.

    Analysis of Emotional Tone

    Participants showed significant decreases in sadness (F1,8=10.85; d=–1.01; P=.01; partial η2=0.58) and disgust (F1,8=7.66; d=–1.25; P=.02; partial η2=0.49), while joy significantly increased (F1,8=11.02; d=1.81; P=.01; partial η2=0.58).

    Anger (F1,8=0.54, d=–0.32; P=.24; partial η2=0.06) and fear (F1,8=0.11; d =–0.17; P=.37; partial η2=0.01) decreased but were not statistically significant.

    LIWC Analysis

    The application of LIWC tools showed significant improvement in authenticity, clout, and emotional tone. Authenticity (F1,8=7.33; d=0.92; P=.01; partial η2=0.48) improved by 49%. Clout (F1,8=8.65; d=0.90; P=.01; partial η2=0.52) increased by 80%. Emotional tone (F1,8=6.02; d=0.92; P=.02; partial η2=0.43) improved by 51%. However, analytical thinking did not show significant change (F1,8=0.23; d=0.20; P=.65; partial η2=0.03).

    Qualitative Observations

    Observations were derived from structured data completed by 3 of the participants (results found in Table 2). The PTGI showed a general increase in the domains of relating to others, new possibilities, personal strength, and appreciation for life. The PSOM indicated no overall changes but did show an increase from all 3 participants in responsible caretaking. Course feedback included comments such as:

    The site made me feel calm and cared for. It made me feel like people were looking out for me and that I had a network of resources and people if I needed.
    I got my laughter back.

    WW Military Sexual Assault Survivors

    In total, 7 of the 10 (70%) Women Warriors participants completed the course. Of these, 6 WW participants produced sufficient narrative to have their unstructured data analyzed. Results of analysis of unstructured data were grouped into 4 outcome variable groups: (1) personality traits, (2) personal values and needs, (3) emotional tone, and (4) LIWC. Results from rMANOVA were reported as univariate analyses. Cohen d effect sizes from initial t tests were retained and reported.

    Analysis of Personality Traits

    Participants showed significant increases in personality traits of openness (F1,5=9.74; d=1.47; P=.01; partial η2=0.66) and agreeableness (F1,5=12.21; d=1.68; P=.01; partial η2=0.71), while showing significant decreases in extraversion (F1,5=7.11; d=–1.58; P=.04; partial η2=0.59).

    Conscientiousness (F1,5=1.93; d=–0.53; P=.22; partial η2=0.28) decreased and emotional range (F1,5=1.10; d=0.58; P=.36; partial η2=0.17) increased but these were not statistically significant.

    Analysis of Personal Values and Needs

    Participants showed significant increases in the personal values and needs of closeness (F1,5=10.01; d=1.23; P=.03; partial η2=0.67), love (F1,5=13.47; d=1.58; P=.01; partial η2=0.73), life pleasure (F1,5=8.51; d=1.30; P=.02; partial η2=0.63), curiosity (F1,5=5.97; d=1.44; P=.03; partial η2=0.54), and achievement (F1,5=13.47; d=1.28; P=.02; partial η2=0.73).

    The values and needs domains of helping others (F1,5=3.10; d=0.35; P=.07; partial η2=0.38), stimulation (F1,5=0.39; d=0.15; P=.28; partial η2=0.07), tradition (F1,5=1.24; d=0.12; P=.16; partial η2=0.20), structure (F1,5=2.78; d=0.87; P=.08; partial η2=0.36), and idealism (F1,5=3.17; d=1.03; P=.07; partial η2=0.39) generally increased but were not statistically significant.

    Analysis of Emotional Tone

    None of the differences of emotional tone were statistically significant. Fear (F1,5=4.37; d=–0.8; P=.09; partial η2=0.47), disgust (F1,5=1.41; d=–0.74; P=.14; partial η2=0.22), and sadness (F1,6=1.16; d=–0.2; P=.16; partial η2=0.28) generally decreased, while joy (F1,5=1.98; d=0.84; P=.11; partial η2=0.28) generally increased.

    LIWC Analysis

    The application of LIWC tools showed significant improvement in clout (F1,5=7.92; d=1.47; P=.02; partial η2=0.61), which increased by 56%. Although not significant, emotional tone (F1,5=4.60; d=1.17; P=.08; partial η2=0.48) improved by 24%, and authenticity (F1,5=5.46; d=0.37; P=.07; partial η2=0.52) improved by 9.3%. Analytical thinking did not show significant change (F1,5=0.09; d=–0.12; P=.77; partial η2=0.02).

    Qualitative Observations

    Qualitative observations of structured data (results found in Table 3) were generally consistent with anecdotal reports from participants and facilitators. There were 7 participants who completed structured measures. The PHQ-15 showed general decreases in somatization levels, from moderately severe to minimal. The Brief COPE showed a general increase in acceptance, decreased denial, increases in getting support and asking advice, and increased feelings of hope (ie, not giving up). Of the 7 participants, 5 showed less anxiety (generalized anxiety disorder 7-item checklist) and 4 showed improvement in depression (PHQ-9). Results of the PCL-5 did not show overall improvement, but 2 of the 7 participants showed substantially reduced symptoms scores, while 3 others recorded no change. PTGI did not show overall improvement, but 4 participants showed a general improvement in relating to others and participation in life. Observations of the SWEMWS were mixed, as 2 participants improved and 5 remained the same. Course feedback included comments such as:

    I believe [course facilitator] when you say WE WILL get to the place of balance and peace. I returned home and I feel better than I have in a long time. I still have challenges and I know it's a day to day journey, but I ACTUALLY feel stronger…I'm honestly still in shock. I didn't think recovery would ever be a word used to describe me, but now I'm believing it.
    The good thing that came from this week's assignment was I found that I found support that I didn't realize that I had from sources I never would've thought.
    [The course platform] made me feel calm and cared for. It made me feel like people were looking out for me and that I had a network of resources and people if I needed.
    Table 3. Averages and change scores for treatment groups of Next Mission combat veterans and Women Warriors military sexual assault survivors.
    AssessmentNMa combat veterans (n=3)WWb military sexual assault survivors (n=7)
     Pretest, meanPosttest, meanDifference scorec, meanPretest, meanPosttest, meanDifference scorec, mean
    Posttraumatic Growth Inventory (PTGI)47.358.711.466.660.6–6.0

    		PTGI-I: relating to others10.717.06.321.018.6–2.4

    		PTGI-II: new possibilities13.014.71.717.01.7–15.3

    		PTGI-III: personal strength11.712.71.014.610.6–4.0

    		PTGI-IV: spiritual change3.03.70.72.62.70.1

    		PTGI-V: appreciation of life9.010.71.78.68.70.1
    SWEMWSdN/AeN/AN/A21.122.21.1
    Brief COPEfN/AN/AN/A36.330.3–6.0

    		Active copingN/AN/AN/A3.12.6–0.5

    		Positive reframingN/AN/AN/A3.32.9–0.4

    		PlanN/AN/AN/A3.62–1.6

    		Emotional supportN/AN/AN/A2.630.4

    		Self-distractionN/AN/AN/A3.32.7–0.6

    		VentN/AN/AN/A2.92.1–0.8

    		Behavioral disengagementN/AN/AN/A1.90.7–1.2

    		AcceptanceN/AN/AN/A1.93.41.5

    		HumorN/AN/AN/A2.91.9–1.0

    		ReligionN/AN/AN/A3.12.3–0.8

    		Instrumental supportN/AN/AN/A2.33.61.3

    		DenialN/AN/AN/A0.90.4–0.5

    		Substance useN/AN/AN/A1.10.9–0.2

    		Self-blameN/AN/AN/A2.01.8–0.2
    PCL-5gN/AN/AN/A57.660.93.3
    PHQh-15N/AN/AN/A13.610.9–2.7
    PHQ-9N/AN/AN/A13.410.7–2.7
    GAD-7iN/AN/AN/A11.48.3–3.1
    PSOMSj9.09.00.0N/AN/AN/A

    		Focused attention1.71.70.0N/AN/AN/A

    		Productivity2.01.3–0.7N/AN/AN/A

    		Responsible caretaking1.02.01.0N/AN/AN/A

    		Restful repose0.71.30.6N/AN/AN/A

    		Sensuous pleasure2.01.0–1.0N/AN/AN/A

    		Sharing1.71.70.0N/AN/AN/A

    aNM: Next Mission.

    bWW: Women Warriors.

    cDifferences in scores were used for qualitative observation. Differences in scores were not calculated for significance due to variation in completion and sample size.

    dSWEMWS: Short Warwick-Edinburgh Mental Well-Being Scale.

    eN/A: not applicable.

    fCOPE: Coping Orientation to Problems Experienced.

    gPCL-5: PTSD checklist for the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition.

    hPHQ: Patient Health Questionnaire.

    iGAD-7: generalized anxiety disorder 7-item checklist.

    jPSOMS: Positive States of Mind Scale (n=10).


    Principal Findings

    The purpose of our study was to evaluate the feasibility and utility of NLP in evaluating change in a small pilot study of 2 veteran populations who completed 2 novel online behavioral health interventions. Our aggregate results suggest that NLP can provide valuable insights on shifts in personality traits, personal values, and needs, as well as measure changes in emotional tone, in an evaluation of our novel online behavioral health interventions. The process of our participant recruitment suggests additional support for our findings. Recruitment was conducted over several months until critical mass was obtained for a cohort, with participants waiting variable amounts of time until start of the courses. Participants, regardless of waiting time, began the courses with similar symptom profiles. However, although our effect sizes are large, our small sample size must be considered in the interpretation of our results. Additionally, although it is reasonable to conclude that the large effect sizes seen are the result of our intervention, a limitation of our study was that we were unable to account for potential confounds by comparison with a control group. Thus, we cannot say with absolute certainty whether the effects seen were due to our intervention or additional factors. Future studies should seek to replicate the effects found in our pilot study with a control or comparison group, accounting for potential confounds.

    In general, results of NLP mirrored qualitative reports and feedback from participants and facilitators on the efficacy of these interventions. Furthermore, our study suggests that NLP may detect participant change with a greater sensitivity than that of subjective symptom measures. We note that checklists do not tell stories—only stories do. A common clinical observation is the phenomenon in treatment of depression or anxiety wherein an individual’s loved ones or a clinician will often notice change in the individual before the individual become self-aware of the change [53,54]. In a similar manner, natural language analysis may provide an indication that change is occurring through the person’s own written text before there is meta-recognition of change and before such change is detectable using subjective symptom measures.

    Findings such as decreased fear, sadness, disgust, and emotional range (neuroticism) with increased joy are consistent with posttraumatic growth and specifically tend to be associated with resilience. A common phenomenon in combat veterans is survivor’s guilt, more conventionally termed moral injury [55]. The nature of combat is such that it lends itself to situations that are morally ambiguous or unclear, potentially culminating in significant moral conflict after the fact. The constructs of moral injury and PTSD overlap, although they have characteristics that differentiate them [56]. A key component of moral injury is the presence of guilt or shame as a result of a real or perceived transgression [55]. From an evolutionary psychology perspective on emotions, guilt and shame are considered derivatives of the emotion of disgust [57]. Furthermore, there is evidence that elevated levels of disgust are associated with increases in intrusive thoughts of trauma [58] and with posttraumatic stress symptoms in general [59]. There is further evidence to suggest that inclusion in a supportive community of individuals with shared experiences is known to reduce feelings of guilt and shame while increasing feelings of acceptance and belonging [60]. Hence, the results of our study may suggest that reductions in the emotion of disgust could correlate with reductions in guilt and shame and could subsequently correspond to symptom change in PTSD. Although there is indication of this effect from our findings, further exploration and replication is required.

    A noteworthy finding of our study is revealed by the consistent increases in closeness, love, and life pleasure in WW military sexual assault survivors. PTSD research indicates that these domains are severely impacted by trauma in general and sexual assault in particular [61]. In fact, one of the main treatment targets for cognitive processing therapy of PTSD is reduced ability to trust [62], which is associated with negatively impacting closeness and intimacy. In sexual assault survivors, intimacy and closeness can become significantly more difficult in the wake of the trauma, especially when combined with other sequelae of PTSD symptoms [63,64]. The findings of our study suggest that this intervention is helpful in fostering an enhanced ability to become close and intimate, and indeed to rely on others for social support. A potential explanation is that the medium in which survivors of trauma interact, asynchronously through the web, provides a sense of psychological safety that enables taking healthy risks to interact with others. In other words, the interaction can feel safe because it removes potential external and environmental triggers, which can be physiologically and cognitively destabilizing in the moment. Additionally, knowing that social support is a significant protective factor for PTSD [22], the participants in our study could find additional psychological safety in knowing that the other members of the group had been through a similar experience. In fact, it is already established that support groups can be efficacious [65], but in populations with PTSD, avoidance can be a significant obstacle to treatment [66]. It suggests that a slow gradual approach to social interaction through a technological medium can produce significant changes in the ability to experience closeness and intimacy, as well as in needing and valuing others. Finally, we believe there is therapeutic value inherent in the task of writing about one’s adverse experiences [67].

    There is tentative evidence to suggest that the personality trait of conscientiousness is related to hypervigilance in decision making [68]. Thus, our findings may suggest that hypervigilance, a cardinal symptom of PTSD, may reduce in tandem with reduced conscientiousness. It is quite possible that on subjective symptom measures, individuals may not realize reductions in hypervigilance due to their identification as someone who is “always on guard,” a concept that has overlap and is consistent with military cultural values. However, additional research is needed to establish a clear association between the constructs of conscientiousness and hypervigilance.

    A possible explanation for increased need for stability (eg, a consumer who consistently likes the same choice in a product, not a variety of choices) is that as veterans connect and create community with each other, there may be a reflection back to shared military values. Stability and regimen are a hallmark of military culture [23]. In fact, one participant reported via course feedback that “the good thing that came from this week's assignment was I found that I found support that I didn't realize that I had from sources I never would've thought.”

    Further studies are needed and are underway to determine whether a higher rate of completion of subjective symptom measures will correspond to changes found with NLP. These studies are also needed to determine whether NLP changes will accurately predict future changes in subjective symptom measures.

    We recognize that with the attributes being measured by these tools, aggregating data is not always the best way to assess whether the impact of an intervention on an individual is positive, negative, or neutral. For example, authenticity and confidence in expression are generally understood in terms of “the more the better,” whereas for attributes like conscientiousness, the desired outcome would be more for someone who is irresponsible and less for someone who is overly obsessive or rigid. Future research should focus on evaluating these online behavioral health interventions with larger samples across different populations while also measuring effectiveness in comparing natural language analytics to conventional evaluation methods.

    A limitation of our study is that the generalizability is impacted by small sample sizes and the lack of a control group. Thus, we are unable to confirm whether results are due to the online intervention or another unidentified factor or set of factors. Furthermore, despite a growing number of validation studies, NLP is not widely accepted or implemented as a reliable indicator of therapeutic change. Thus, further validation studies that establish convergent and discriminant validity with therapeutic outcomes are warranted. An additional limitation of our study is that we were unable to statistically compare outputs from NLP to validated structured measures. The extant validation literature suggests that attempts at finding convergence of NLP with self-report measures of personality have produced mixed results. This could be due to the fact that NLP tends to measure both latent and explicit emotional tone and personality, whereas self-report measures are solely reliant on the perception of the individual. As a result, both methods have accrued criticism for being affected by bias [69]. However, the two methods may not measure the same construct, but different constructs from differing perspectives. Additionally, although there is initial evidence that NLP outputs can be predictors of consumer behavior [70], there is not yet certainty that those variables are predictors of therapeutic or behavioral change or that they impact psychosocial outcomes. On the other hand, symptom reduction measures have a better representation and acceptance as predictors of therapeutic and behavioral change within the extant literature. Thus, future studies should attempt to replicate our findings using NLP with larger sample sizes of veterans and construct a study design in such a way that NLP can be evaluated as a predictor of psychosocial outcomes.

    Conclusions and Implications

    Our interventions were impactful on attributes detected in writing, and the results of NLP provided tentative yet potentially valuable and provocative insights. By quantifying and aggregating these attributes, we have gained insights about which areas of emotional functioning are responsive to our intervention. By looking at individual analyses, we can readily see how each participant is progressing, for example by noting reduced emotional expression in someone who was emotionally dysregulated and increased emotional expression in another participant who was initially emotionally constricted. The use of natural language analytics tools opens up a completely new area of scientific inquiry. We are getting closer to entirely replicating what happens in in-person psychotherapy. We can now provide the benefit of both symptom checklists and patient narratives to expert clinicians who know how to interpret the data for clinical decision making and to researchers who can determine the impact and value of any given behavioral health intervention. We believe that using AI powered by natural language analytics will enable the creation of effective therapy bots that will assist facilitators and sustain participant engagement, as this intervention is scaled to make it accessible to everyone, anytime, anywhere. We also believe that using NLP applied to behavioral health interventions and other clinical situations creates an entirely new field of medical informatics.

    Acknowledgments

    The NM and WW programs were approved by the institutional review board of the University of California, San Francisco. The NM program was funded by Bristol-Myers Squibb Foundation Inc, a grant and gifts from the McCormick and the McKesson foundations, and individual donors. The WW program was funded by gifts by the Gallo Foundation and Veterans on Wall Street.

    Authors' Contributions

    All authors reviewed and edited the manuscript and approved of the final draft.

    Conflicts of Interest

    KN has a household interest in Tiatros Inc. KN is an advisor to Tiatros compensated by stock options. DWK is a project-based contractor with Tiatros Inc. WK is an employee of IBM. AG is an employee of IBM.

    1. National Projections of Supply and Demand for Selected Behavioral Health Practitioners: 2013-2025.: US Department of Health and Human Resources   URL: https:/​/bhw.​hrsa.gov/​sites/​default/​files/​bhw/​health-workforce-analysis/​research/​projections/​behavioral-health2013-2025.​pdf [accessed 2020-08-19]
    2. Rosch P. The Quandary of Job Stress Compensation. Workplace Psychology.   URL: https:/​/workplacepsychology.​files.wordpress.com/​2016/​07/​the-quandary-of-job-stress-compensation_rosch.​pdf [accessed 2020-01-10]
    3. Reisman M. PTSD Treatment for Veterans: What's Working, What's New, and What's Next. Pharm Ther 2016 Oct;41(10):623-634 [FREE Full text] [Medline]
    4. Posttraumatic Stress Disorder (PTSD).: US Department of Veterans Affairs   URL: https://www.research.va.gov/topics/ptsd.cfm [accessed 2020-01-10]
    5. Barth SK, Kimerling RE, Pavao J, McCutcheon SJ, Batten SV, Dursa E, et al. Military Sexual Trauma Among Recent Veterans: Correlates of Sexual Assault and Sexual Harassment. Am J Prev Med 2016 Jan;50(1):77-86. [CrossRef] [Medline]
    6. Hester RD. Lack of access to mental health services contributing to the high suicide rates among veterans. Int J Ment Health Syst 2017;11:47 [FREE Full text] [CrossRef] [Medline]
    7. National Academies of Science, Engineering, and Medicine. Evaluation of the Department of Veterans Affairs Mental Health Services. Washington, DC: The National Academies Press; 2018.
    8. Zogas A. US Military Veterans' Difficult Transitions Back to Civilian Life and the VA's Response. Brown University Watson Institute for International and Public Affairs. 2017.   URL: https:/​/watson.​brown.edu/​costsofwar/​papers/​2017/​us-military-veterans-difficult-transitions-back-civilian-life-and-va-s-response [accessed 2020-01-10]
    9. Derefinko KJ, Hallsell TA, Isaacs MB, Colvin LW, Salgado Garcia FI, Bursac Z. Perceived Needs of Veterans Transitioning from the Military to Civilian Life. J Behav Health Serv Res 2019 Jul;46(3):384-398. [CrossRef] [Medline]
    10. Morin R. The Difficult Transition from Military to Civilian Life. Pew Research Center Social & Demographic Trends. 2011.   URL: https://www.pewsocialtrends.org/2011/12/08/the-difficult-transition-from-military-to-civilian-life/ [accessed 2019-12-23]
    11. Solomon Z, Bensimon M, Greene T, Horesh D, Ein-Dor T. Loneliness Trajectories: The Role of Posttraumatic Symptoms and Social Support. Journal of Loss and Trauma 2014 Aug;20(1):1-21. [CrossRef]
    12. Najavits L. The problem of dropout from "gold standard" PTSD therapies. F1000Prime Rep 2015;7:43 [FREE Full text] [CrossRef] [Medline]
    13. Thase ME, Kingdon D, Turkington D. The promise of cognitive behavior therapy for treatment of severe mental disorders: a review of recent developments. World Psychiatry 2014 Oct;13(3):244-250 [FREE Full text] [CrossRef] [Medline]
    14. Azimi M, Moradi A, Hasani J. Effectiveness of cognitive behavioral therapy for insomnia (traditional and Internet-based) on everyday memory of people with insomnia and comorbid depression. Adv Cogn Sci 2019;20(4):20-34 [FREE Full text]
    15. Kuester A, Niemeyer H, Knaevelsrud C. Internet-based interventions for posttraumatic stress: A meta-analysis of randomized controlled trials. Clin Psychol Rev 2016 Feb;43:1-16. [CrossRef] [Medline]
    16. Spek V, Cuijpers P, Nyklícek I, Smits N, Riper H, Keyzer J, et al. One-year follow-up results of a randomized controlled clinical trial on internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years. Psychol Med 2008 May;38(5):635-639. [CrossRef] [Medline]
    17. Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007 Mar;37(3):319-328. [CrossRef] [Medline]
    18. Simon N, McGillivray L, Roberts NP, Barawi K, Lewis CE, Bisson JI. Acceptability of internet-based cognitive behavioural therapy (i-CBT) for post-traumatic stress disorder (PTSD): a systematic review. Eur J Psychotraumatol 2019;10(1):1646092 [FREE Full text] [CrossRef] [Medline]
    19. Van't Hof E, Cuijpers P, Stein DJ. Self-help and Internet-guided interventions in depression and anxiety disorders: a systematic review of meta-analyses. CNS Spectr 2009 Feb;14(2 Suppl 3):34-40. [CrossRef] [Medline]
    20. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res 2005 Mar 26;7(1):e7 [FREE Full text] [CrossRef] [Medline]
    21. Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res 2004 Dec 22;6(4):e46 [FREE Full text] [CrossRef] [Medline]
    22. Ozer EJ, Best SR, Lipsey TL, Weiss DS. Predictors of posttraumatic stress disorder and symptoms in adults: A meta-analysis. Psychological Trauma: Theory, Research, Practice, and Policy 2008 Aug;S(1):3-36. [CrossRef]
    23. Understanding Military Culture.: VA Community Provider Toolkit   URL: https://www.mentalhealth.va.gov/communityproviders/military_culture.asp [accessed 2020-03-19]
    24. Strategies for Measuring the Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation.: Assistant Secretary for Planning and Evaluation, US Department of Health and Human Services; 2015.   URL: https:/​/aspe.​hhs.gov/​report/​strategies-measuring-quality-psychotherapy-white-paper-inform-measure-development-and-implementation [accessed 2020-01-15]
    25. Hogue A, Dauber S, Henderson CE, Liddle HA. Reliability of therapist self-report on treatment targets and focus in family-based intervention. Adm Policy Ment Health 2014 Sep;41(5):697-705 [FREE Full text] [CrossRef] [Medline]
    26. Howie L, Hirsch B, Locklear T, Abernethy AP. Assessing the value of patient-generated data to comparative effectiveness research. Health Aff (Millwood) 2014 Jul;33(7):1220-1228. [CrossRef] [Medline]
    27. Borsboom D, Molenaar D. Psychometrics. In: Wright JD, editor. International Encyclopedia of the Social & Behavioral Sciences. Amsterdam, Netherlands: Elsevier; 2015:418-422.
    28. Bertl M. Towards emotional intelligent robots: How AI enables to understand human emotions. Research Gate. 2018.   URL: https:/​/www.​researchgate.net/​publication/​329400436_Towards_emotional_intelligent_robots_-_How_AI_enables_to_understand_human_emotions [accessed 2020-01-15] [WebCite Cache]
    29. The science behind the service Internet. IBM Watson Personality Insights.   URL: https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-science#science [accessed 2020-01-15] [WebCite Cache]
    30. The science behind the service. IBM Watson Tone Analyzer.   URL: https:/​/cloud.​ibm.com/​docs/​services/​tone-analyzer?topic=tone-analyzer-ssbts#the-science-behind-the-service [accessed 2020-01-15] [WebCite Cache]
    31. Robinson RL, Navea R, Ickes W. Predicting Final Course Performance From Students’ Written Self-Introductions. Journal of Language and Social Psychology 2013 Feb 13;32(4):469-479. [CrossRef]
    32. Tausczik YR, Pennebaker JW. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology 2009 Dec 08;29(1):24-54. [CrossRef]
    33. LIWC - Linguistic Inquiry and Word Count.   URL: http://liwc.wpengine.com/ [accessed 2020-01-15] [WebCite Cache]
    34. Pennebaker J, Boyd R, Jordan K, Blackburn K. The Development and Psychometric Properties of LIWC2015. 2015 Sep 15.   URL: https://repositories.lib.utexas.edu/handle/2152/31333 [accessed 2020-01-10] [WebCite Cache]
    35. IBM Watson Personality Insights.   URL: https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-models [accessed 2020-01-15]
    36. Norman WT. Toward an adequate taxonomy of personality attributes: replicated factors structure in peer nomination personality ratings. J Abnorm Soc Psychol 1963 Jun;66(6):574-583. [CrossRef] [Medline]
    37. IBM Watson Personality Insights - Needs.   URL: https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-needs [accessed 2020-01-15]
    38. Schwartz S. Basic Human Values: Theory, Measurement, and Applications. Revue Française de Sociologie 2006;47(4):929-968 [FREE Full text]
    39. IBM Watson Personality Insights - Values.   URL: https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-values [accessed 2020-01-15]
    40. IBM Watson - Interpreting Numerical Results.   URL: https://cloud.ibm.com/docs/services/personality-insights?topic=personality-insights-numeric [accessed 2020-01-15]
    41. Razek MA, Frasson C. Text-Based Intelligent Learning Emotion System. JILSA 2017;09(01):17-20. [CrossRef]
    42. Ekman P, Matsumoto D. Reading faces: The universality of emotional expression. In: Gernsbacher MA, Per RW, Hough LM, Pomerantz JR, editors. Psychology and the real world: Essays illustrating fundamental contributions to society. New York, NY: Worth Publishers; 2011:140-146.
    43. Horowitz M, Adler N, Kegeles S. A scale for measuring the occurrence of positive states of mind: a preliminary report. Psychosom Med 1988;50(5):477-483. [CrossRef] [Medline]
    44. Tedeschi RG, Calhoun LG. The posttraumatic growth inventory: Measuring the positive legacy of trauma. J Traum Stress 1996;9(3):455-471. [CrossRef]
    45. Blevins CA, Weathers FW, Davis MT, Witte TK, Domino JL. The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): Development and Initial Psychometric Evaluation. J Trauma Stress 2015 Dec;28(6):489-498. [CrossRef] [Medline]
    46. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA 1999 Nov 10;282(18):1737-1744. [CrossRef] [Medline]
    47. Ng Fat L, Scholes S, Boniface S, Mindell J, Stewart-Brown S. Evaluating and establishing national norms for mental wellbeing using the short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS): findings from the Health Survey for England. Qual Life Res 2017 May;26(5):1129-1144 [FREE Full text] [CrossRef] [Medline]
    48. Carver CS. You want to measure coping but your protocol's too long: consider the brief COPE. Int J Behav Med 1997;4(1):92-100. [CrossRef] [Medline]
    49. IBM Watson Personality Insights. 2016.   URL: https://www.ibm.com/watson/services/personality-insights/ [accessed 2020-01-14]
    50. IBM Watson Tone Analyzer. 2016.   URL: https://www.ibm.com/watson/services/tone-analyzer/ [accessed 2020-01-15]
    51. Armstrong RA. When to use the Bonferroni correction. Ophthalmic Physiol Opt 2014 Sep;34(5):502-508. [CrossRef] [Medline]
    52. Richardson J. Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review 2011 Jan;6(2):135-147 [FREE Full text] [CrossRef]
    53. Johnson C, Gunn J, Kokanovic R. Depression recovery from the primary care patient's perspective: 'hear it in my voice and see it in my eyes'. Ment Health Fam Med 2009 Mar;6(1):49-55 [FREE Full text] [Medline]
    54. Further Observations Regarding Major Depression Treatment and Special Issues – Suicide.   URL: https://www.mentalhelp.net/depression/suicide/ [accessed 2020-03-19]
    55. Frankfurt S, Frazier P. A Review of Research on Moral Injury in Combat Veterans. Military Psychology 2017 Dec 13;28(5):318-330. [CrossRef]
    56. Litz BT, Stein N, Delaney E, Lebowitz L, Nash WP, Silva C, et al. Moral injury and moral repair in war veterans: a preliminary model and intervention strategy. Clin Psychol Rev 2009 Dec;29(8):695-706. [CrossRef] [Medline]
    57. Giner-Sorolla R, Espinosa P. Social cuing of guilt by anger and of shame by disgust. Psychol Sci 2011 Jan;22(1):49-53. [CrossRef] [Medline]
    58. Bomyea J, Amir N. Disgust Propensity as a Predictor of Intrusive Cognitions Following a Distressing Film. Cognit Ther Res 2012 Jun;36(3):190-198 [FREE Full text] [CrossRef] [Medline]
    59. Badour CL, Feldner MT. The Role of Disgust in Posttraumatic Stress. Journal of Experimental Psychopathology 2018 Jul 11;9(3):pr.032813. [CrossRef]
    60. Andreas D, Ja DY, Wilson S. Peers Reach Out Supporting Peers to Embrace Recovery (PROSPER): A Center for Substance Abuse Treatment Recovery Community Services Program. Alcoholism Treatment Quarterly 2010 Jul 08;28(3):326-338. [CrossRef]
    61. Martinson AA, Sigmon ST, Craner J, Rothstein E, McGillicuddy M. Processing of intimacy-related stimuli in survivors of sexual trauma: the role of PTSD. J Interpers Violence 2013 Jun;28(9):1886-1908. [CrossRef] [Medline]
    62. Resick P, Monson C, Chard K. Cognitive processing therapy for PTSD: a comprehensive manual. New York, NY: Guilford Press; 2016.
    63. Connop V, Petrak J. The impact of sexual assault on heterosexual couples. Sexual and Relationship Therapy 2007 Jan 24;19(1):29-38 [FREE Full text] [CrossRef]
    64. Yehuda R, Lehrner A, Rosenbaum TY. PTSD and Sexual Dysfunction in Men and Women. J Sex Med 2015 May;12(5):1107-1119. [CrossRef] [Medline]
    65. Worrall H, Schweizer R, Marks E, Yuan L, Lloyd C, Ramjan R. The effectiveness of support groups: a literature review. Mental Health Social Inclusion 2018 Apr 09;22(2):85-93. [CrossRef]
    66. Kazlauskas E. Challenges for providing health care in traumatized populations: barriers for PTSD treatments and the need for new developments. Glob Health Action 2017;10(1):1322399 [FREE Full text] [CrossRef] [Medline]
    67. Pennebaker JW. Writing About Emotional Experiences as a Therapeutic Process. Psychol Sci 2016 May 06;8(3):162-166. [CrossRef]
    68. Heidari M, Arani M. Relationship Between Five Personality Factors with Decision Making Styles of Coaches. Sport Sci 2017 Feb 28;10(1):70-76 [FREE Full text]
    69. Stone AA, Turkkan JS, Bachrach CA, Jobe JB, Kurtzman HS, Kain VS, editors. The science of self-report: Implications for research and practice. Mahwah, NJ: Lawrence Erlbaum Associates Publishers; 2000.
    70. Liu Z, Wang Y, Mahmud J, Akkiraju R, Schoudt J, Xu A, et al. To Buy or Not to Buy? Understanding the Role of Personality Traits in Predicting Consumer Behaviors. In: Spiro E, Ahn YY, editors. Social Informatics. New York, NY: Springer International Publishing; 2016:337-346.

    ‎
    AI: artificial intelligence
    CBT: cognitive behavioral therapy
    COPE: Coping Orientation to Problems Experienced
    HIPAA: Health Insurance Portability and Accountability Act
    LIWC: Linguistic Inquiry and Word Count
    NLP: natural language processing
    NM: Next Mission
    PCL-5: PTSD checklist for the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition
    PHQ: Patient Health Questionnaire
    PSOM: Positive States of Mind Scale
    PTGI: Posttraumatic Growth Inventory
    PTSD: posttraumatic stress disorder
    rMANOVA: repeated-measures multivariate analysis of variance
    SWEMWS: Short Warwick-Edinburgh Mental Well-Being Scale
    VA: US Department of Veterans Affairs
    WW: Women Warriors

    Edited by G Eysenbach; submitted 11.12.19; peer-reviewed by S McRoy, Z Ma, K Schultebraucks; comments to author 06.01.20; revised version received 19.03.20; accepted 04.08.20; published 23.09.20

    Copyright

    ©Kim P Norman, Anita Govindjee, Seth R Norman, Michael Godoy, Kimberlie L Cerrone, Dustin W Kieschnick, William Kassler. Originally published in JMIR Formative Research (http://formative.jmir.org), 23.09.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.