This study used the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD), a protected electronic medical record system. These data are not publicly available due to patient privacy and ethical restrictions, as stated in the Data Availability Statement. The NTUH-iMD includes electronic medical records such as admission and discharge notes as well as ICD (International Classification of Diseases) diagnostic codes. This study focused on 4836 discharge summaries from 3087 patients admitted to the psychiatric unit with a principal psychiatric diagnosis (ICD-9-CM [International Classification of Diseases, Ninth Revision, Clinical Modification]: 290‐319 or ICD-10-CM [International Classification of Diseases, Tenth Revision, Clinical Modification]: F00-F99) [10]. Discharge summaries in NTUH-iMD are predominantly written in English, consistent with routine clinical documentation practices at NTUH. Although some records contain limited Chinese expressions, transliterated terms, or code-mixed content, the corpus is best characterized as predominantly English clinical text with occasional multilingual elements rather than Chinese-language records. All personal identifying information, including names, addresses, and birthdays, were deidentified, and only sections related to the history of the present illness were included.

Due to the time-consuming nature of manual annotation, 500 discharge notes were selected. As previously described [10], the dataset included 189 (37.8%) notes on schizophrenia, 109 (21.8%) on bipolar disorder, and 117 (23.4%) on unipolar depressive disorder. Four (0.8%) patients had both schizophrenia and unipolar depressive disorder, and 8 (1.6%) had both schizophrenia and bipolar disorder. Additionally, 77 notes covered other diagnoses—dementia, substance use disorder, and anxiety disorders—that were not classified under the main 3 categories [10].

While basic personal information had already been deidentified, we applied an additional deidentification tool [11,12] to replace sensitive details, such as residence, school, workplace, and other elements, that could potentially disclose aspects of the patient’s private life. This tool uses advanced NLP to automatically detect and replace sensitive information [11]. Although effective, its random replacements can affect data completeness and readability. Therefore, after applying the tool, manual review and modification were conducted to ensure both deidentification and the integrity of the data for subsequent analysis.

This study was approved by the Institutional Review Board of National Taiwan University Hospital (NTUH-201610072RINA). The requirement for informed consent was waived because the study used deidentified retrospective electronic medical records. All data were processed and analyzed in accordance with institutional privacy and data protection regulations.

At the sentence level, key clinical course features were identified, including symptoms/episodes indicating disease onset, relapse, recurrence, remission/response, and hospitalization details. Symptoms/episodes encompassed the onset or worsening of psychiatric symptoms and dangerous behaviors such as suicide or violence. Hospitalization included both inpatient treatment and rehabilitation programs. Time information was extracted in formats such as year-month-date, age, vague terms (eg, “childhood” or “this year”), and duration formats (eg, “from A to B,” “between A and B,” or “persisted for A”). These definitions are outlined in Table 1.

At the chart level, we aimed to extract 4 key pieces of information:

Some charts did not provide precise episode counts or hospitalization details (eg, “The patient had several episodes in the past 10 years”). These were excluded from the corresponding analysis.

All medical chart annotations were conducted by a board-certified clinical psychiatrist with 20 years of experience (CSW) and a researcher with NLP expertise (CHS). Annotators were instructed to label each sentence based on content, identifying the development or exacerbation of psychiatric symptoms, behaviors related to suicide or violence as “symptom/episode,” treatment response or remission as “response/remission,” admissions to acute psychiatric or rehabilitation wards as “hospitalization,” or to mark the sentence as containing none of the above events. Timing information was also recorded. For example, in the sentence “He had insomnia and poor appetite for one month,” symptoms were annotated along with their duration. Additionally, chart data—including the first onset time, episode count, number of psychiatric hospitalizations, and the date of the most recent hospitalization—were annotated. To assess annotation reliability, a psychiatrist and an NLP researcher independently annotated all training documents. Interrater agreement was evaluated using Cohen κ before consensus adjudication. Disagreements were subsequently resolved through discussion to generate the final reference standard used for model development and evaluation.

Overall interrater agreement was substantial (Cohen κ=0.78). Interrater agreement varied across annotation categories (Table S1 in Multimedia Appendix 1). Agreement was highest for explicit temporal expressions, including calendar dates (κ=0.90), age-based expressions (κ=0.85), and hospitalization events (κ=0.87). In contrast, vague temporal expressions showed substantially lower agreement (κ=0.47), reflecting the inherent ambiguity of phrases such as “during childhood,” “recent years,” or “a long time ago.”

Due to the sensitive nature of psychiatric notes and privacy concerns, we opted for locally deployable models instead of cloud-based LLMs. We used the following models: LLaMA (Llama-3.1-8B-Instruct) [13], MentaLLaMA [14], OpenBioLLM [15], and Mistral (Ministral-8B-Instruct-2410) [16]. We chose these models because they are popular and can be fine-tuned on consumer-level hardware. In addition to the LLMs, we included Bio_ClinicalBERT as a conventional transformer-based baseline for sentence-level information extraction. Bio_ClinicalBERT is a clinical-domain BERT model widely used for clinical named entity recognition tasks [17]. Because Bio_ClinicalBERT is a sequence-labeling model rather than a generative model, it was evaluated only for sentence-level extraction tasks and was not included in chart-level prediction analyses.

Our approach employs a 2-stage extraction method to systematically derive clinical information from psychiatric medical records as illustrated in Figure 1. Stage 1—sentence-level extraction: For each sentence in a discharge note, the sentence-level extractor produces: (1) an event label (“symptom/episode,” “hospitalization,” “remission/response,” or “none”) and (2) associated temporal information (if present), in plain-text format. Stage 2—chart-level aggregation and prediction: all sentence-level predictions are assembled into a structured list maintaining original document order, with the format: [SENTENCE]: <text >| [EVENT]: <label >| [TIME]: <temporal info or “none”>. This structured list is provided as input to the Stage 2 chart—level extractor, which synthesizes this information to produce the 4 clinical course features (onset time, episode count, hospitalization count, and last hospitalization time).

For sentence-level extraction, we develop a specialized system prompt that clearly communicates the task objective: accurately classifying medical and temporal event types from sentences in clinical notes, as shown in Figure 2. The design of this prompt focuses on guiding the model to reliably identify and categorize relevant medical events, including episodes, symptoms, hospitalizations, and temporal indicators. By emphasizing context and temporal cues, the prompt facilitates precise extraction, ensuring consistency across varying clinical scenarios.

In the second stage, the chart-level extractor synthesizes aggregated sentence-level predictions to derive detailed psychiatric summaries from patient records. The chart-level prompt was crafted to systematically guide the extraction of key clinical features, including the first occurrence of symptoms, the number of episodes, hospitalization count, and the timing of the most recent hospitalization. This design aims to standardize the summarization process, clarify the criteria for data extraction, and minimize ambiguity, thereby enabling robust, consistent results suitable for clinical analysis. The detailed system prompt is shown in Figure 3.

Both extractors use low-rank adaptation (LoRA) [18] for fine-tuning. Annotated data were explicitly structured for fine-tuning, converting labeled annotations into corresponding prompt-response pairs aligned with each designed prompt. This structured data approach facilitates supervised learning and improves model performance.

The fine-tuning configurations were optimized to ensure training efficiency and accuracy. The batch size was set to 1, with gradient accumulation across 16 steps. We set the learning rate to 0.0001 and trained the model for 5 epochs, using a warm-up ratio of 0.1. The maximum sequence length was set to 8192 tokens. For LoRA, we set the rank to 8, the α to 16, and applied a dropout rate of 0.05. The LoRA hyperparameters were selected via a grid search over rank ∈ {4, 8, 16}, alpha ∈ {8, 16, 32}, and dropout ∈ {0.01, 0.05, 0.1}, with validation performance averaged across the first 3 folds.

All experiments were conducted on a single NVIDIA A6000 48 GB GPU. Fine-tuning each model for 1 fold required approximately 2 to 3 hours (sentence-level extractor: ~1.5 h; chart-level extractor: ~1 h). Inference (prediction) took approximately 20 minutes per fold per model. Models can run on consumer hardware with ≥24 GB of VRAM using 4-bit quantization, though fine-tuning would require proportionally more time.

Sentence-level extraction performance was evaluated separately for clinical events and temporal information. Precision, recall, and F₁-score were calculated for each annotation category. Performance was estimated using 10-fold cross-validation, and the mean performance across folds was reported. Clinical event extraction was evaluated for symptom/episode, hospitalization, and remission/response events. Temporal extraction was evaluated for calendar dates, age expressions, relative time expressions, duration, persistence, and vague temporal expressions.

Chart-level performance was evaluated for four clinical course features: first-episode onset time, number of episodes, number of hospitalizations, and most recent hospitalization. For temporal outcomes (first-episode onset time and most recent hospitalization), performance was assessed using exact-match accuracy and macroaveraged F₁-score. Predicted temporal expressions were normalized prior to evaluation, and partial matches (eg, “2010” vs “2010‐03”) were considered incorrect. For count-based outcomes (number of episodes and number of hospitalizations), both exact-match accuracy and mean absolute error (MAE) were reported. Lower MAE values indicate better performance.

To assess whether performance differences between methods were statistically meaningful, pairwise comparisons were conducted using bootstrap resampling. For each comparison, 1000 bootstrap samples were generated by resampling discharge summaries with replacement from the evaluation set, and 2-sided bootstrap P values were calculated from the resulting distribution of performance differences. Comparisons focused on the proposed 2-stage framework vs the direct and Joint approaches.

In an analysis of 500 discharge notes, a total of 12,947 sentences were extracted, with sentence counts ranging from 4 to 112 per note, averaging 25.9 (SD 17.3) sentences. Of these, 7177 sentences contained clinical events, and 4842 included time-related information.

Fine-tuning substantially improved sentence-level extraction performance across all evaluated LLMs (Table S2 in Multimedia Appendix 1). While performance gains varied across models, LoRA fine-tuning consistently increased F₁-scores for symptom/episode, hospitalization, and remission/response extraction. The largest improvements were observed in OpenBioLLM and MentaLLaMA, which demonstrated limited zero-shot performance but achieved competitive results after fine-tuning. These findings underscore the importance of task-specific adaptation when applying LLMs to psychiatric clinical narratives.

Tables 2 and 3 present the sentence-level extraction performance of all models after fine-tuning. Mistral achieved the highest F₁-scores across most clinical event categories, including symptom/episode (0.925), hospitalization (0.944), and remission/response (0.867). It also demonstrated the strongest performance for most temporal information categories, including age (0.983), relative time expressions (0.953), duration (0.976), time references (0.991), and vague temporal expressions (0.888). Bio_ClinicalBERT, a conventional transformer-based sequence-labeling model, achieved competitive performance for several extraction tasks but was generally outperformed by the best-performing fine-tuned LLMs. Nevertheless, LLaMA and OpenBioLLM showed performance comparable to Mistral for duration and time references. Persistent and vague temporal expressions remained the most challenging categories across all models, although Mistral achieved the highest performance in both tasks.

Table 4 summarizes chart-level performance in predicting key clinical course features, whereas Table 5 presents bootstrap-based pairwise comparisons between the proposed 2-stage framework and alternative approaches. Across all models, the 2-stage framework achieved the highest performance for the first-episode onset prediction. Mistral_Ours achieved the best onset accuracy (0.772), followed by LLaMA_Ours (0.744), OpenBioLLM_Ours (0.738), and MentaLLaMA_Ours (0.734). Detailed precision and recall results for the proposed 2-stage framework (ours) are provided in Table S3 in Multimedia Appendix 1.

For all 4 models, the 2-stage framework significantly outperformed both the direct and joint approaches for onset prediction (all bootstrap P≤.002). For episode count prediction, Mistral_Ours (accuracy=0.624) and LLaMA_Ours (accuracy=0.618) achieved comparable performance, with no statistically significant difference between them (bootstrap P=.85). For hospitalization count prediction, improvements over the direct approach were limited, with no significant differences observed for either LLaMA (P=.56) or Mistral (P=.99). However, the 2-stage framework consistently outperformed the joint approach for hospitalization count prediction across all evaluated models. For the most recent hospitalization, the 2-stage framework significantly outperformed the direct approach for all models (all bootstrap P<.05), although differences between the 2-stage and joint approaches were generally smaller.

As demonstrated in this study, modern LLMs provide a significant step forward in improving the efficiency and accuracy of the analysis of the course of psychiatric disorder. Our results highlight the potential of LLMs to extract critical clinical information from electronic health records, significantly reducing the manual burden on clinicians while enhancing the quality of the extracted data.

A notable finding of this study was the consistent advantage of the proposed 2-stage framework over both direct and joint extraction approaches. One possible explanation is that psychiatric discharge summaries often contain multiple temporally distributed events, diagnoses, remissions, and hospitalizations within a single document. Direct prediction requires the model to simultaneously identify relevant events, resolve temporal information, and infer chart-level clinical course features from the entire note, creating a highly complex reasoning task. Although the joint approach explicitly incorporates sentence-level extraction, it still requires event extraction and chart-level prediction to be performed within a single prompt-response process. In contrast, the proposed framework decomposes the task into 2 sequential steps. By first converting free-text narratives into structured event-time representations and then performing chart-level reasoning on the structured outputs, the model can focus on a simpler objective at each stage. This decomposition likely reduces cognitive load, improves interpretability, and facilitates the aggregation of longitudinal clinical information, particularly for first-episode onset prediction, where the largest performance gains were observed. The greatest improvements were observed for first-episode onset prediction, suggesting that separating temporal information extraction from chart-level reasoning is particularly beneficial for reconstructing longitudinal disease trajectories.

However, oracle analyses suggested that chart-level performance was not determined solely by sentence-level extraction accuracy. Although replacing model-generated sentence-level outputs with gold-standard annotations improved performance for some models and tasks, the improvement was not consistent. This suggests that many chart-level errors arose from the second-stage aggregation and reasoning process, such as selecting the correct onset time, distinguishing episodes from hospitalizations, and integrating multiple temporally distributed events across a discharge summary. In addition, the chart-level extractor was developed using model-generated sentence-level outputs, and gold-standard annotations may differ in format or information density from these predicted inputs. These findings suggest that future improvements should target both sentence-level extraction and chart-level clinical reasoning.

Our study did not use cloud-based APIs, such as OpenAI’s GPT, due to the highly sensitive nature of psychiatric notes, in which data privacy is a critical concern. As a result, we were unable to compare the performance of OpenAI’s GPT with other LLMs. Instead, we selected LLaMA and Mistral for this study, as they can be deployed locally and fine-tuned on private data, mitigating privacy risks. While LLaMA and similar models offer greater control and customization, they often require fine-tuning or retrieval augmentation to achieve performance comparable to larger proprietary models such as GPT-4 [19]. To address this, researchers have developed domain-specific LLMs for mental health, such as MentaLlama, which integrate psychiatric knowledge through fine-tuning [14]. With proper adaptation, open models can significantly reduce the performance gap.

To further investigate the impact of domain-specific open models, we evaluated the performance of MentaLlama [14] and OpenBioLLM [15]. However, the results did not show a clear improvement over LLaMA 3.1. A possible explanation is that MentaLlama is based on LLaMA 2, while OpenBioLLM is built on LLaMA 3; their effectiveness may, therefore, be limited by the capabilities of their respective base models.

Recent studies highlight the potential of Mistral-7B in clinical text extraction. A fine-tuned version of the model achieved 91% accuracy in identifying key clinical history details, closely matching GPT-4’s 92%, with no statistically significant difference [20]. Despite being an order of magnitude smaller than GPT-4, Mistral-7B, optimized with prompt engineering and in-context learning, reached near-human agreement (κ≈0.75) with radiologists to extract structured history elements.

These findings suggest that while GPT-4 excels in complex reasoning tasks, smaller open models such as LLaMA and Mistral can be competitive when fine-tuned for domain-specific applications. The choice of model depends on balancing performance, resource availability, and data security. In contrast, LLaMA and Mistral provide greater control through local deployment and fine-tuning, albeit with some trade-offs in raw performance.

Error analysis revealed several recurring sources of chart-level extraction errors. The most common error category was ambiguous temporal information (55/179, 30.7%), followed by missing information in the source notes (27/183, 14.8%), episode-hospitalization conflation (19/166, 11.4%), and confusion arising from multiple diagnoses with distinct onset times (11/108, 10.2%). Errors related to multiple clinical events described within a single sentence accounted for an additional 8.0% (9/112).

Missing information in the source notes often limited the accuracy of extraction, regardless of the model’s capabilities. For example, in the note “Since then, the patient has been repeatedly admitted to psychiatric services due to unstable mood and suicide attempts,” the number of hospitalizations was not specified, preventing accurate extraction. Ambiguous temporal information was another major source of error. In one case, a patient was evaluated in the emergency department and subsequently admitted, but no explicit hospitalization date was documented. The model occasionally inferred the outpatient visit date as the admission date, despite the absence of a clearly stated admission date.

The complexity of psychiatric narratives also contributed to extraction errors. When multiple diagnoses with distinct onset times were described in the same note, the model sometimes failed to correctly associate symptom onset with the appropriate diagnosis. For example, in a note describing autism diagnosed at the age of 4 years and mood symptoms beginning at the age of 15 years, the model occasionally conflated the onset times of the 2 conditions. Similarly, when multiple clinical events were described within a single sentence, the model sometimes captured only one event while overlooking others. For instance, in the sentence “Psychotic symptoms improved in March 2010 but worsened in May 2010,” the model often identified the symptom exacerbation but failed to recognize the preceding improvement.

Finally, episode-hospitalization conflation represented a distinct error pattern. In notes describing multiple psychiatric episodes alongside several hospital admissions, the model occasionally interpreted each episode as a separate hospitalization, leading to the overestimation of hospitalization counts. Overall, these findings suggest that many chart-level errors arise from incomplete documentation and the inherent complexity of psychiatric narratives rather than simple failures of event recognition.

Despite the promising results, several limitations were identified in this study. First, the dataset used was limited to discharge notes from a single hospital, which may not fully represent the diversity of psychiatric documentation across different clinical settings. Further research is needed to validate the model’s performance on larger and more diverse datasets, including outpatient and emergency room records, to ensure broader applicability. Second, the dataset is best characterized as primarily English clinical text with occasional code-mixed elements rather than fully Chinese-language records. Because the evaluated models were primarily pretrained on English corpora and the study used predominantly English discharge summaries, the generalizability of our findings to predominantly Chinese-language psychiatric records or other multilingual clinical settings remains uncertain. Future studies should evaluate the proposed framework on datasets with varying language compositions and across different health care settings. Third, the study does not differentiate between onset times for multiple diagnoses when patients have comorbid psychiatric conditions. This omission can lead to inaccuracies in real-world applications. Fourth, we identified only 4 items related to the clinical course. Expanding the scope of extracted features to include additional relevant clinical indicators—such as treatment responses, adverse effects for each regimen, drug compliance assessments, and identification of precipitating factors for relapse and recurrence—would provide a more comprehensive view of the patient’s psychiatric trajectory. Finally, several risks associated with deploying LLMs in clinical settings warrant explicit consideration. LLMs can generate plausible but incorrect outputs, and the extracted information should therefore be treated as decision-support rather than authoritative clinical data. We emphasize that the present system is a research prototype and must undergo prospective clinical validation before any real-world deployment. All predictions should be reviewed by qualified clinicians before informing treatment decisions.

This study demonstrates the potential of LLMs to automate the extraction of key clinical features from psychiatric discharge summaries. The model effectively captures structured data such as hospitalization events and onset dates, though further refinement is needed to handle vague temporal information and complex clinical nuances. Despite these challenges, LLMs show promise in enhancing psychiatric care by equipping clinicians with efficient, accurate tools for managing psychiatric disorders. As LLM technology advances, its integration into routine practice may improve the quality of care and patient outcomes.