A pseudonymized extract of clinical notes from the Japan Medical Data Survey (JAMDAS; M3 Inc), an outpatient health care database of nationwide primary care clinics curated by M3 Inc [17], was used for analysis. The 5 outpatient clinics were a convenience sample of facilities participating in the database; they were not randomly drawn from JAMDAS, and patients within these clinics were randomly sampled with no filtering by disease name or prescription. The dataset therefore includes patients with a variety of medical conditions. It comprised 2020 clinical notes from 1036 patients aged 0 to 84 years who visited these clinics for fever or suspected infectious disease between October 2020 and September 2023. The clinics served distinct populations (median patient age ranging from approximately 1 year at a pediatric-predominant clinic to approximately 37 years at an adult clinic; Table S6 in Multimedia Appendix 1).

After removing missing values and duplicate records, 3 physicians within the research team performed annotations on the following items: body temperature, fatigue, muscle and joint pain, headache and dizziness, respiratory symptoms, visible changes in the throat, gastrointestinal symptoms, sensory abnormalities, other symptoms, vaccination information, and onset date. Detailed annotation items, criteria, and extraction formats are described in Table 1. The raw onset description, parsed onset date, body temperature, and description of vaccination history were annotated as text data types, while all other items were classified as binary variables.

To assess interrater reliability, a pilot set of 216 notes was independently annotated by all 3 physicians; the remaining notes were annotated by a single physician. In the pilot set, the mean observed agreement for binary symptom flags was 93.9% (unanimous, 3/3, 91.1%), with a pooled Fleiss κ of 0.31; the low κ reflects the very low prevalence of most symptoms (the κ paradox) rather than poor agreement. For the fever flag, the physicians’ operational definitions initially diverged when no temperature had been measured (κ≈0.02); this was resolved by consensus into a single harmonized definition that was applied to the finalized reference standard. For free-text and numeric items, exact-match agreement among all 3 physicians was 71.8% for body temperature and 99.5% for vaccination descriptions (per-item Fleiss κ and observed agreement are given in Table S4 in Multimedia Appendix 1). Disagreements were resolved by majority vote (≥2 of 3 physicians). Of the 151 evaluation notes, 45 (30%) were triple annotated and 106 (70%) were single annotated; a sensitivity analysis showed comparable extraction performance between the 2 subsets (Table S8 in Multimedia Appendix 1).

Owing to the cost of manual annotation, only a portion of the 2020 candidate notes could be annotated: 802 (40%) notes (526 patients) were annotated, and 773 (96%) of these, after removing notes with missing or duplicated diagnostic free text, formed the analysis set. The remaining unannotated candidate notes were therefore excluded because of labeling resource constraints and not because the annotators failed to reach consensus.

The dataset was randomly divided into 622 (80%) training cases and 151 (20%) evaluation cases, ensuring no patient overlap between the two sets. We developed and compared the following approaches: a rule-based algorithm crafted by physicians; few-shot learning (FSL) using commercial and open-source LLMs; and fine-tuning of open-source LLMs. The rule-based algorithm was developed by physicians who carefully reviewed the training dataset and translated patterns for each extraction item into regular expressions.

After several preliminary performance experiments, this study selected Claude 3.5 Sonnet (version 20241022; Anthropic PBC) [18] as the representative commercial LLM and the Gemma 2 family (27B and 2B models; Alphabet LLC) [19] as the representative open-source LLMs. We tested several prompt patterns and settled on one that outputs results in a JSON format, where each extraction item serves as a key and the corresponding extracted value serves as its value (eg, {“fever”: 1, “temperature”: 38.3, “headache”: 0}). The prompt used is shown in Textbox 1.

Within each of 9 clinical categories, per-note item observations were pooled (microaveraged) to compute F₁-score; the overall value is the unweighted macroaverage across categories. Each cell shows the F₁-score point estimate with its 95% CI from a patient-level cluster bootstrap (2000 replicates). Categories with no positive cases were undefined and excluded from the macroaverage.

FSL is a technique in which a model is given a small number of example inputs and outputs to learn from before performing a task [20]. This allows the model to better understand the task and generate more accurate results without updating its parameters. We initially planned to compare zero-shot inference with FSL inference. However, we found that zero-shot outputs often did not adhere strictly to the JSON format, making reliable evaluation difficult. Therefore, we used the FSL results as the baseline for LLM performance. We randomly sampled 18 examples, ensuring that each extraction item had at least 1 positive instance included.

To further improve performance, we applied supervised fine-tuning to the open-source LLMs. The training dataset was split into 500 (80%) samples for training and 122 (20%) samples for validation, with the validation set used to monitor overfitting during training. We performed fine-tuning using the instruction-tuned Google Gemma 2 models (gemma-2-2b-it and gemma-2-27b-it). Full fine-tuning was conducted on the 2B model, while, owing to hardware limitations, we applied quantized low-rank adaptation (QLoRA) for fine-tuning the 27B model. Low-rank adaptation (LoRA) is a technique that reduces the number of trainable parameters by injecting low-rank matrices into a pretrained model’s weights, enabling efficient fine-tuning without updating the entire model [21]. QLoRA combines LoRA with quantization, compressing model weights to reduce memory use while maintaining accuracy, allowing large models to be fine-tuned on limited hardware resources [22]. The specific training parameters are detailed in Table S1 in Multimedia Appendix 1. Fine-tuning was performed using the transformers [23] and trl [24] libraries on a single RTX 6000 Ada GPU (NVIDIA Corp).

We grouped the extraction items into 9 clinical categories and computed performance within each category. Each extracted item in a note was treated as 1 binary observation; for items extracted as text or numeric values (eg, body temperature, onset date, and vaccination description), an exact-value match was required. Within each category, per-note item observations were pooled (microaveraging) to compute the F₁-score, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV); the overall value for each metric was the unweighted macroaverage across the 9 categories, which served as the primary performance metric. Categories with no positive cases in the evaluation set were undefined and were excluded from the average. Metrics were thus computed at the level of item presence within a note, pooled within a category and then macroaveraged across categories—not aggregated at the patient or note level. Moreover, 95% CIs were obtained by patient-level cluster bootstrap resampling with 2000 replicates; and paired between-model differences were assessed with the same patient-level bootstrap. Per-category sensitivity, specificity, PPV, and NPV are provided in Table S3 in Multimedia Appendix 1.

This study was approved by the Nihonbashi-Sakura Clinic Ethics Committee. The committee did not issue an approval or case number for this study, so none is reported. Because the analysis used pseudonymized, retrospective data recorded during routine clinical care, individual informed consent was not obtained; instead, in accordance with the Japanese Ethical Guidelines for Medical and Biological Research Involving Human Subjects, information about the study was disclosed to patients and an opt-out opportunity to decline participation was provided. All records were pseudonymized before they were made available to the research team, and the data were accessed and handled solely in accordance with the restricted-use data agreement with M3 Inc that governs the JAMDAS database. The study was conducted in accordance with the Declaration of Helsinki.

Of the 2020 candidate notes (1036 patients), 802 (526 patients) were annotated; the remainder were left unannotated owing to labeling resource constraints. After removing notes with missing or duplicated diagnostic free text, 773 notes from 526 patients formed the analysis set. Compared with the annotated notes, the unannotated notes came from patients who were significantly younger (median age 3, IQR 2-17 vs 15, IQR 7-20 years; P<.001) and were drawn disproportionately from particular clinics (P<.001); sex did not differ (P=.44; Table S5 in Multimedia Appendix 1). The data flow and an overview of the experiments are shown in Figure 1. Background information and the distribution of extraction items for the training and evaluation datasets are presented in Table 1.

The diagram shows the flow from 2020 candidate notes (1036 patients) through annotation (802 notes; 526 patients) to the analysis set (773 notes; 526 patients), and the random, patient-disjoint split into a training set (622 notes; 500 training and 122 validation) and a held-out evaluation set (151 notes) used to compare the rule-based, few-shot, and fine-tuned approaches.

Interrater reliability was assessed on a 216-note pilot set independently annotated by all 3 physicians; the remaining notes, including 106 (70%) of the 151 evaluation notes, were single annotated.

As shown in Table S2 in Multimedia Appendix 1, although there were some systematic differences in clinical note templates across clinics and physician preferences within clinics, the notes were generally written in the SOAP (subjective, objective, assessment, and plan) format, showing overall similarity. The performance evaluation results for each model are presented in Table 2. Some extraction items exhibited relatively low variability in their descriptions, indicating that their descriptions were similar across many physicians; consequently, the rule-based algorithm achieved high performance on these items. For items with high variability in descriptions, such as vaccination information and onset dates, the rule-based algorithm showed low accuracy; however, FSL with LLMs achieved high accuracy. In particular, the commercial model Anthropic Claude 3.5 Sonnet achieved a macroaverage F₁-score of 0.875 (95% CI 0.800-0.913), with a sensitivity of 0.929.

In the fine-tuning experiments, both the Google Gemma 2 2B and 27B models showed substantial performance improvements. The 2B model achieved a macroaveraged F₁-score of 0.822 (95% CI 0.752-0.874), while the 27B model fine-tuned with QLoRA reached the highest point estimate (0.906, 95% CI 0.833-0.945; specificity of 0.969 and PPV of 0.906). The fine-tuned 27B model’s advantage over Claude 3.5 Sonnet was small, and its CI included zero (ΔF₁-score=0.030, 95% CI −0.035 to 0.140); therefore, the 2 models were not statistically comparable. The 27B model did, however, significantly outperform the smaller fine-tuned 2B model (ΔF₁-score=0.084, 95% CI 0.040 to 0.163). Per-category sensitivity, specificity, PPV, and NPV for all models are provided in Table S3 in Multimedia Appendix 1.

Because the cohort was dominated by COVID-19 (approximately 80% of notes mentioned COVID-19), a fine-grained diagnosis breakdown was not feasible; we therefore compared COVID-19 (n=120, 79%) and non–COVID-19 (n=31, 21%) notes in the test set. The best fine-tuned model was stable across strata (macroaveraged F₁-score 0.900 vs 0.893), whereas the rule-based algorithm (0.702 vs 0.605) and Claude 3.5 Sonnet (0.886 vs 0.735) showed lower performance on non–COVID-19 notes. Given the small non–COVID-19 subgroup, these results are exploratory (Table S7 in Multimedia Appendix 1).

A qualitative error analysis of misclassifications by the LLM revealed that many errors occurred when multiple extraction targets were listed consecutively, separated by commas or periods, followed by a collective negation (eg, “Diarrhea, nausea negative.” and “Abdominal pain and vomiting. Wheezing negative.”). These cases can be interpreted in multiple ways and are examples of situations in which even human experts would find it difficult to reach a consensus.

We demonstrated that LLMs can extract infectious disease–related information and vaccination data from free-text clinical notes written in Japanese in EHRs with high accuracy. For items with high descriptive variability, LLMs markedly outperformed rule-based regular expressions. The commercial model Claude 3.5 Sonnet (with FSL) reached a macroaveraged F₁-score of 0.875 (95% CI 0.800-0.913) without additional training, while fine-tuning the open-source Gemma 2 27B model with QLoRA on only 500 notes achieved a comparable point estimate (0.906, 95% CI 0.833-0.945); the difference between the 2 was small and not statistically distinguishable (ΔF₁-score=0.030, 95% CI −0.035 to 0.140). The fine-tuned 27B model did, however, significantly outperform the smaller fine-tuned 2B model (ΔF₁-score=0.084, 95% CI 0.040 to 0.163). Given the security risks of sending clinical text containing sensitive personal information to external commercial models, achieving comparable performance with a smaller open-source model demonstrates the feasibility of processing data securely within a closed environment, enhancing real-world applicability.

Our findings are consistent with prior reports that LLMs outperform traditional NLP approaches for structured extraction from EHR free text [8,9,25] and extend them to Japanese primary care narratives, which, to our knowledge, have not previously been targeted directly; prior Japanese clinical NLP has largely used BERT (Bidirectional Encoder Representations from Transformers)-based models or symptom taggers on other document types [26,27]. In contrast to BERT-based COVID-19 case detection in Dutch general practice [10] and rule or curation pipelines in large US EHR systems [11], we show that a compact, locally deployable, fine-tuned generative model can perform comparably to a commercial model, consistent with recent reports that fine-tuned open-source LLMs reach human-level extraction performance from few examples [28] and that local open-source deployment mitigates the privacy risks of sending notes to external services [29], while benchmark studies find that fine-tuning still tends to outperform zero or few-shot prompting for extraction tasks [30].

Several categories scored low under FSL (eg, other symptoms and vaccination information for Gemma 2 2B). This likely reflects (1) class imbalance—rare symptoms such as chills, dizziness, and rash had very few positive cases; (2) ambiguity in clinical expressions, including consecutively listed findings followed by a collective negation—a well-known challenge for negation or assertion detection in clinical text [31,32]; and (3) the limited capacity of small models without parameter updates. Fine-tuning substantially mitigated these issues. The low interrater agreement observed for some common items (eg, fever), which is consistent with prior reports that annotation of clinical narratives is intrinsically variable [33], further indicates that part of the residual error stems from intrinsic annotation ambiguity rather than model failure.

At this feasibility stage, the extraction is fully automated: the structured outputs are intended for subsequent review and aggregation by public health or clinical staff rather than for autonomous clinical decision-making, and no specialized user interaction with the model is required beyond the domain expertise needed to interpret aggregated surveillance signals.

This study has several limitations. First, the 5 clinics were a convenience sample, and the annotated (analyzed) set overrepresented some clinics while leaving others almost entirely unannotated (eg, 2 clinics contributed almost no analyzable notes); the algorithm’s accuracy may not generalize to clinics with different documentation styles. Second, because manual annotation was resource-limited, only approximately 40% of the candidate notes (802 of 2020) were annotated, and these differed systematically from the unannotated notes: annotated patients were older (median age 15 vs 3 years; P<.001) and came disproportionately from particular clinics (P<.001), although sex did not differ (P=.44). Because the choice of which notes to annotate was not random, this raises the possibility of selection bias. Third, the evaluation set was predominantly single annotated (106/151, 70% of notes), so interrater reliability was quantified on the 216-note pilot subset; a sensitivity analysis showed comparable extraction performance between triple- and single-annotated notes (Table S8 in Multimedia Appendix 1). Fourth, we could not establish an external validation set, so the results represent internal validation only. Finally, most patients had a single note, so the ability to capture temporal symptom changes is unverified. Critically, this study demonstrates extraction accuracy, not downstream surveillance utility: we did not evaluate whether the extracted signals improve outbreak detection, trend monitoring, or timeliness in real-world settings.

Future research should (1) validate the algorithm on external clinics not involved in training, (2) link extracted symptom and vaccination signals to observed infectious disease trends to assess surveillance timeliness and accuracy, and (3) extend annotation to longitudinal records.

By fine-tuning the open-source LLM Google Gemma 2, we developed and internally validated a model specialized in extracting and structuring infectious disease–related information from Japanese free-text clinical notes with accuracy comparable to that of a commercial model, while remaining deployable in a closed environment. Because the model can run locally without transmitting sensitive clinical text to external services, it offers a practical and privacy-preserving foundation for EHR-based digital surveillance in settings where data cannot leave the institution. Realizing that potential will require external validation and, above all, demonstrating that the extracted signals improve real-world outbreak detection and timeliness—the essential next step toward broader public health impact.