A total of 23 participants were recruited for 2 studies of veterans, the Next Mission (NM) program and Women Warriors (WW) program. A total of 13 participants enrolled in the 14-week NM program, and 10 enrolled in the 8-week WW program. Inclusion criteria for the study were: (1) must be a military veteran, (2) must be aged 18 or older, (3) must speak English, (4) must be able to access the internet regularly, and (5) for the WW program, must be a woman. In the NM program, 11 of the 13 (85%) participants were men, while the WW program consisted of all (10/10, 100%) female participants. In total, 16 participants completed the courses, 9 in the NM program and 7 in the WW program.

Participants were recruited via Facebook and LinkedIn ads and could sign up on a mobile, laptop, or desktop device of their choice. Recruitment was conducted over several months until enough participants were able to form cohorts for each course. The recruitment messages attempted to account for potential stigma by focusing on building resiliency skills and promoting posttraumatic growth rather than treating mental illness. Participants were informed that they would be helping others in their group while also getting help for themselves.

Participants were also incentivized by the opportunity to earn University of California college credits for their work, although only 3 of 16 (19%) participants took advantage. Participants understood that the program was fully compliant to the Health Insurance Portability and Accountability Act (HIPAA) and that they could choose to participate anonymously. However, they also had the option of revealing their identity to the group at any time they wished during the course. Both programs, which we also refer to as courses, were entitled “Stress, Resiliency and Post Traumatic Growth.” The constellation of the groups was generationally diverse and allowed older veterans to connect with younger veterans.

In each program, participants watched videos teaching principles of CBT, narrative therapy, behavioral activation, and mindfulness meditation. They submitted written homework, including journal entries and thought and mood logs, and participated in asynchronous discussions, all within a HIPAA-secure environment. Approximately 90 minutes of online class time and one hour per week of homework was completed by each participant. The asynchronous discussions were facilitated and monitored by a licensed, doctoral-level therapist. Of note, facilitators for the WW group were female. The content for the programs was created in various commercially available proprietary applications by faculty members in the Department of Psychiatry of a US University Medical Center and then assembled into the NM and WW programs and delivered on a commercially available proprietary platform.

Participants’ writing samples and online posts were analyzed using the LIWC [33], IBM Watson’s Personality Insights [49], and Tone Analyzer [50]. Detection scores (NLP outputs) were generated as outputs, upon which we conducted a pretest-posttest analysis of these language attributes at the beginning and end of each course. Analysis was conducted in IBM SPSS Statistics (version 26). Initially, repeated-measures 1-tailed t tests were run and effect sizes were calculated using Cohen d. However, to reduce the risk of type I statistical error from multiple t tests, variables were grouped into 4 multivariate categories: (1) LIWC variables, (2) personality trait variables, (3) personality values and needs variables, and (4) emotional tone variables. Omnibus testing in the form of repeated-measures multivariate analysis of variance (rMANOVA) was conducted with Bonferroni correction [51]. Effect sizes were calculated as partial η² [52] and Cohen d effect sizes were retained from t tests.

In total, 16 of the 23 (70%) participants completed the courses and 15 were able to have their unstructured data analyzed. Results of all rMANOVA analyses on NLP outputs are summarized in Table 2.

A total of 9 of the 13 (70%) Next Mission participants completed the course and were able to have their unstructured data analyzed. Results of analysis on NLP data were grouped into 4 outcome variable groups: (1) personality traits, (2) personal values and needs, (3) emotional tone, and (4) LIWC. Results from rMANOVA were reported as univariate analyses. Cohen d effect sizes from initial t tests were retained and reported.

In total, 7 of the 10 (70%) Women Warriors participants completed the course. Of these, 6 WW participants produced sufficient narrative to have their unstructured data analyzed. Results of analysis of unstructured data were grouped into 4 outcome variable groups: (1) personality traits, (2) personal values and needs, (3) emotional tone, and (4) LIWC. Results from rMANOVA were reported as univariate analyses. Cohen d effect sizes from initial t tests were retained and reported.

The purpose of our study was to evaluate the feasibility and utility of NLP in evaluating change in a small pilot study of 2 veteran populations who completed 2 novel online behavioral health interventions. Our aggregate results suggest that NLP can provide valuable insights on shifts in personality traits, personal values, and needs, as well as measure changes in emotional tone, in an evaluation of our novel online behavioral health interventions. The process of our participant recruitment suggests additional support for our findings. Recruitment was conducted over several months until critical mass was obtained for a cohort, with participants waiting variable amounts of time until start of the courses. Participants, regardless of waiting time, began the courses with similar symptom profiles. However, although our effect sizes are large, our small sample size must be considered in the interpretation of our results. Additionally, although it is reasonable to conclude that the large effect sizes seen are the result of our intervention, a limitation of our study was that we were unable to account for potential confounds by comparison with a control group. Thus, we cannot say with absolute certainty whether the effects seen were due to our intervention or additional factors. Future studies should seek to replicate the effects found in our pilot study with a control or comparison group, accounting for potential confounds.

In general, results of NLP mirrored qualitative reports and feedback from participants and facilitators on the efficacy of these interventions. Furthermore, our study suggests that NLP may detect participant change with a greater sensitivity than that of subjective symptom measures. We note that checklists do not tell stories—only stories do. A common clinical observation is the phenomenon in treatment of depression or anxiety wherein an individual’s loved ones or a clinician will often notice change in the individual before the individual become self-aware of the change [53,54]. In a similar manner, natural language analysis may provide an indication that change is occurring through the person’s own written text before there is meta-recognition of change and before such change is detectable using subjective symptom measures.

Findings such as decreased fear, sadness, disgust, and emotional range (neuroticism) with increased joy are consistent with posttraumatic growth and specifically tend to be associated with resilience. A common phenomenon in combat veterans is survivor’s guilt, more conventionally termed moral injury [55]. The nature of combat is such that it lends itself to situations that are morally ambiguous or unclear, potentially culminating in significant moral conflict after the fact. The constructs of moral injury and PTSD overlap, although they have characteristics that differentiate them [56]. A key component of moral injury is the presence of guilt or shame as a result of a real or perceived transgression [55]. From an evolutionary psychology perspective on emotions, guilt and shame are considered derivatives of the emotion of disgust [57]. Furthermore, there is evidence that elevated levels of disgust are associated with increases in intrusive thoughts of trauma [58] and with posttraumatic stress symptoms in general [59]. There is further evidence to suggest that inclusion in a supportive community of individuals with shared experiences is known to reduce feelings of guilt and shame while increasing feelings of acceptance and belonging [60]. Hence, the results of our study may suggest that reductions in the emotion of disgust could correlate with reductions in guilt and shame and could subsequently correspond to symptom change in PTSD. Although there is indication of this effect from our findings, further exploration and replication is required.

A noteworthy finding of our study is revealed by the consistent increases in closeness, love, and life pleasure in WW military sexual assault survivors. PTSD research indicates that these domains are severely impacted by trauma in general and sexual assault in particular [61]. In fact, one of the main treatment targets for cognitive processing therapy of PTSD is reduced ability to trust [62], which is associated with negatively impacting closeness and intimacy. In sexual assault survivors, intimacy and closeness can become significantly more difficult in the wake of the trauma, especially when combined with other sequelae of PTSD symptoms [63,64]. The findings of our study suggest that this intervention is helpful in fostering an enhanced ability to become close and intimate, and indeed to rely on others for social support. A potential explanation is that the medium in which survivors of trauma interact, asynchronously through the web, provides a sense of psychological safety that enables taking healthy risks to interact with others. In other words, the interaction can feel safe because it removes potential external and environmental triggers, which can be physiologically and cognitively destabilizing in the moment. Additionally, knowing that social support is a significant protective factor for PTSD [22], the participants in our study could find additional psychological safety in knowing that the other members of the group had been through a similar experience. In fact, it is already established that support groups can be efficacious [65], but in populations with PTSD, avoidance can be a significant obstacle to treatment [66]. It suggests that a slow gradual approach to social interaction through a technological medium can produce significant changes in the ability to experience closeness and intimacy, as well as in needing and valuing others. Finally, we believe there is therapeutic value inherent in the task of writing about one’s adverse experiences [67].

There is tentative evidence to suggest that the personality trait of conscientiousness is related to hypervigilance in decision making [68]. Thus, our findings may suggest that hypervigilance, a cardinal symptom of PTSD, may reduce in tandem with reduced conscientiousness. It is quite possible that on subjective symptom measures, individuals may not realize reductions in hypervigilance due to their identification as someone who is “always on guard,” a concept that has overlap and is consistent with military cultural values. However, additional research is needed to establish a clear association between the constructs of conscientiousness and hypervigilance.

A possible explanation for increased need for stability (eg, a consumer who consistently likes the same choice in a product, not a variety of choices) is that as veterans connect and create community with each other, there may be a reflection back to shared military values. Stability and regimen are a hallmark of military culture [23]. In fact, one participant reported via course feedback that “the good thing that came from this week's assignment was I found that I found support that I didn't realize that I had from sources I never would've thought.”

Further studies are needed and are underway to determine whether a higher rate of completion of subjective symptom measures will correspond to changes found with NLP. These studies are also needed to determine whether NLP changes will accurately predict future changes in subjective symptom measures.

We recognize that with the attributes being measured by these tools, aggregating data is not always the best way to assess whether the impact of an intervention on an individual is positive, negative, or neutral. For example, authenticity and confidence in expression are generally understood in terms of “the more the better,” whereas for attributes like conscientiousness, the desired outcome would be more for someone who is irresponsible and less for someone who is overly obsessive or rigid. Future research should focus on evaluating these online behavioral health interventions with larger samples across different populations while also measuring effectiveness in comparing natural language analytics to conventional evaluation methods.

A limitation of our study is that the generalizability is impacted by small sample sizes and the lack of a control group. Thus, we are unable to confirm whether results are due to the online intervention or another unidentified factor or set of factors. Furthermore, despite a growing number of validation studies, NLP is not widely accepted or implemented as a reliable indicator of therapeutic change. Thus, further validation studies that establish convergent and discriminant validity with therapeutic outcomes are warranted. An additional limitation of our study is that we were unable to statistically compare outputs from NLP to validated structured measures. The extant validation literature suggests that attempts at finding convergence of NLP with self-report measures of personality have produced mixed results. This could be due to the fact that NLP tends to measure both latent and explicit emotional tone and personality, whereas self-report measures are solely reliant on the perception of the individual. As a result, both methods have accrued criticism for being affected by bias [69]. However, the two methods may not measure the same construct, but different constructs from differing perspectives. Additionally, although there is initial evidence that NLP outputs can be predictors of consumer behavior [70], there is not yet certainty that those variables are predictors of therapeutic or behavioral change or that they impact psychosocial outcomes. On the other hand, symptom reduction measures have a better representation and acceptance as predictors of therapeutic and behavioral change within the extant literature. Thus, future studies should attempt to replicate our findings using NLP with larger sample sizes of veterans and construct a study design in such a way that NLP can be evaluated as a predictor of psychosocial outcomes.

Our interventions were impactful on attributes detected in writing, and the results of NLP provided tentative yet potentially valuable and provocative insights. By quantifying and aggregating these attributes, we have gained insights about which areas of emotional functioning are responsive to our intervention. By looking at individual analyses, we can readily see how each participant is progressing, for example by noting reduced emotional expression in someone who was emotionally dysregulated and increased emotional expression in another participant who was initially emotionally constricted. The use of natural language analytics tools opens up a completely new area of scientific inquiry. We are getting closer to entirely replicating what happens in in-person psychotherapy. We can now provide the benefit of both symptom checklists and patient narratives to expert clinicians who know how to interpret the data for clinical decision making and to researchers who can determine the impact and value of any given behavioral health intervention. We believe that using AI powered by natural language analytics will enable the creation of effective therapy bots that will assist facilitators and sustain participant engagement, as this intervention is scaled to make it accessible to everyone, anytime, anywhere. We also believe that using NLP applied to behavioral health interventions and other clinical situations creates an entirely new field of medical informatics.