This was a retrospective feasibility study conducted at a single tertiary neurosurgical center (Department of Neurosurgery at the University Hospital of Freiburg). The inclusion criteria comprised consecutive patients with histologically confirmed vestibular schwannoma who underwent surgical treatment between August and December 2023 and had complete documentation (discharge, surgical, pathology, and follow-up reports). Patients who lacked any of these reports were excluded from the analysis. Figure 1 illustrates the data flow and extraction process.

The primary outcome of the study was the accuracy rate in extracting structured information from the provided unstructured data. The ground truth for this evaluation was established by a medical team, which manually reviewed and assessed each response provided by the AI model. This ensured that the AI’s performance was rigorously compared to expert human judgment.

All personal data of both patients and health care professionals that were visible in the original documents were anonymized before being processed by the AI model (Figures 2 and 3). Consequently, it was not possible to identify any individual patients from the data provided to the AI model for analysis. It is also important to note that the original language of all medical reports was German, and these unstructured data were presented to the AI model in its native form for analysis and extraction.

Prompts were designed iteratively, meaning they were developed, tested, and refined to improve clarity and extraction precision. The initial zero-shot prompt served as a baseline framework for structured data extraction. On the basis of qualitative assessment of model responses, targeted refinements were introduced to better define certain variables that were prone to misinterpretation.

For example, the variable “intraoperative complications” was explicitly redefined in the prompt as “intraoperative damage to major vascular or neural structures,” and “new postoperative deficits” were clarified as “deficits developed after surgery, excluding preexisting ones.” These refinements were introduced to standardize semantic interpretation and reduce ambiguity during data extraction.

Refinement rounds continued until no further gains in extraction accuracy were observed, while ensuring that definitions of other variables remained unaffected. All prompts were executed in GPT-4o.

This prompt shown in Textbox 1 was used to generate a Microsoft Excel table with structured data extracted from medical reports written in natural language.

This standardized prompt was used across all patient records to ensure consistency in data extraction, with the AI tasked with transforming unstructured medical data into a format suitable for subsequent analysis.

All original medical reports were manually anonymized by the research team before being entered into the GPT-4o interface. Identifiers such as names and addresses were removed. The anonymized reports were then processed through the official ChatGPT web platform for data extraction. No identifiable or sensitive information was transmitted, stored, or shared, ensuring compliance with General Data Protection Regulation (GDPR) and the standards of the local ethics committee.

All analyses were performed using GPT-4o accessed through the official ChatGPT interface. At the time of data collection, research API (application programming interface) access was not available. The full dataset of 10 anonymized surgical cases was uploaded as a single compressed file for structured data extraction. Generated tables were manually reviewed and archived to ensure methodological traceability and reproducibility within the proof-of-concept framework.

Two board-certified neurosurgeons independently performed manual data extraction from the anonymized medical reports before the GPT-4o analysis, ensuring blinding to the AI-generated results. After completion of both manual and AI-based extractions, the datasets were compared to establish the ground truth. In cases where the 2 reviewers disagreed, a third senior neurosurgeon adjudicated the final classification. Interrater agreement between the 2 primary reviewers was calculated using Cohen κ.

The sample size of 10 patients was chosen based on feasibility for this proof-of-concept design, as the objective was not to achieve statistical power but to test the model’s capacity for accurate structured data extraction in a controlled, pilot environment.

For each binary variable, model outputs were compared to the ground truth to derive the counts of true positives, false positives, false negatives, and true negatives. On the basis of these, accuracy, precision, recall (sensitivity), and F1-score were calculated primarily for binary variables, where such metrics are meaningful.

For conditional text and categorical or numerical variables, performance was mainly summarized using accuracy and corresponding 95% CIs computed via the Wilson method for binomial proportions. Additionally, Δ accuracy (postrefinement and prerefinement) was calculated to quantify improvement following prompt refinement.

To explore potential differences in accuracy across variable types—binary, conditional text, and categorical or numerical—a Kruskal-Wallis test was applied. Given the small sample size and exploratory nature of this proof-of-concept analysis, no post hoc pairwise testing or correction for multiple comparisons was performed.

All analyses were performed using SPSS Statistics (version 27.0; IBM Corp) and R software (version 4.3.2; R Foundation for Statistical Computing).

This research was conducted in compliance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving human subjects. The local ethics committee of the University of Freiburg (23‐1393-S1-retro; approval date: October 31, 2023) approved the waiver of informed consent due to the retrospective design, full anonymization of patient identifiers, and minimal-risk nature of the study, consistent with institutional and national regulations. As this was a retrospective analysis of fully anonymized clinical data, no participants were contacted and no compensation was provided.

Ten consecutive patients diagnosed with intracranial acoustic neuromas who underwent surgical treatment at the Department of Neurosurgery at the University Hospital of Freiburg between August and December 2023 were included in this study. All screened patients met the inclusion criteria and were analyzed. GPT-4o generated an Excel table with structured data based on the zero-shot prompt described in the Methods section (Figure 4).

Regarding the collection of structured data that does not require text interpretation, such as date of birth, patient ID, date of surgery, histopathological diagnosis, and World Health Organization grade, and considering the physicians’ opinion as the ground truth, the accuracy rate was 100% (Table 1).

In the zero-shot prompt, the collection and processing of unstructured data into structured data provided heterogeneous results. The following variables presented a 100% accuracy rate when compared to the physicians’ evaluation: symptoms at presentation (yes or no), which symptoms, symptoms onset to surgery (in months), permanent deficits (yes or no), extent of resection (total, subtotal, and biopsy), and new symptoms at 3-month follow-up (Table 2).

In contrast, variables such as “intraoperative complications (yes or no),” “if yes, which complications,” “new postoperative deficits,” and “if yes, which deficits” achieved an accuracy rate as low as 50% in the zero-shot prompt. However, when the model was refined through additional prompt adjustments, as described in the Methods section, the accuracy rate improved to 90% to 100%, as presented in Tables 3 and 4.

Model performance across interpretative binary variables is summarized in Table 5. Accuracy ranged from 50% to 100%, reflecting variable complexity in contextual interpretation. While fields such as was the deficit permanent? and new symptoms at 3-month follow-up achieved perfect accuracy, others—particularly intraoperative complication and new postoperative deficits—showed lower precision due to false-positive predictions.

When comparing mean accuracies across variable types using the Kruskal-Wallis test, no statistically significant difference was observed (H=2.79; P=.25). Nevertheless, a consistent trend toward higher accuracy for structured categorical and numerical variables (eg, patient ID, date of birth, World Health Organization grade, extent of resection, and Karnofsky scores; mean 97.5%, SD 4.6%) and lower accuracy for conditional text variables (eg, which symptom? which complication? and which deficit?; mean 66.7%, SD 28.9% supports the descriptive results. Binary variables (eg, symptoms at presentation, intraoperative complication, new postoperative deficits, was the deficit permanent, and new symptoms at 3-month follow-up) showed intermediate performance, with a mean accuracy of 80.0% (SD 27.4%).

The findings of this study underscore the potential of GPT-4o and similar LLMs to revolutionize health care data mining. By achieving high accuracy rates for the extraction of structured information from unstructured medical records, GPT-4o demonstrated its utility in reducing the time and effort required for data analysis. Parameters such as symptom onset, tumor grade, and extent of resection were extracted with impressive accuracy, aligning with the work of Lee et al [7], who highlighted the power of NLP tools in parsing clinical trial data efficiently. However, these findings should be interpreted within the limitations of a small, single-institution feasibility study. While they demonstrate the technical viability of GPT-4o for structured data extraction, generalization to other clinical domains requires larger, multicenter validation.

However, not all data categories achieved uniform success. Variables such as intraoperative complications and new postoperative deficits presented challenges in zero-shot scenarios, emphasizing the need for prompt refinement and iterative learning to improve model performance. Similar observations have been reported by Adamson et al [8], who noted that model tuning significantly enhances the precision of clinical data extraction from electronic health records.

Beyond these technical aspects, the integration of LLMs into clinical workflows presents transformative opportunities. For instance, these models can support operational planning by predicting the allocation of resources such as surgical supplies and human labor based on historical data trends. This capability, as highlighted by Obermeyer and Emanuel [9], could reduce inefficiencies and improve overall patient outcomes. Moreover, automated coding of diagnoses and procedures, as discussed by Dong et al [10], could streamline billing processes, thereby alleviating administrative burdens for health care institutions. It should be emphasized that this study did not assess workflow integration, time efficiency, usability, or real-time clinical applicability. Any references to potential operational or efficiency benefits are therefore speculative and reflect broader opportunities described in the literature rather than outcomes directly evaluated in this analysis.

Despite these promising applications, ethical and practical challenges must be addressed. Ensuring data privacy and security is paramount, especially when handling sensitive patient information. Adherence to privacy regulations such as the Health Insurance Portability and Accountability Act and the GDPR is critical [11,12]. Furthermore, the “black box” nature of LLMs raises concerns about their interpretability and trustworthiness in clinical settings. Transparent AI frameworks, as advocated by Arrieta et al [13], could mitigate these issues by making model predictions more explainable. In this study, model interpretability was ensured by manually auditing GPT-4o outputs and verifying their correspondence with the original clinical context. Regarding privacy, all medical reports were manually anonymized prior to being uploaded to the GPT-4o interface. The analyses were conducted using the official ChatGPT web platform, with no identifiable patient information included at any stage. No data were stored, transmitted, or processed through third-party servers outside the OpenAI environment, ensuring full compliance with institutional and GDPR standards.

Recent scholarship has underscored the ethical and operational risks of deploying LLMs in health care. Ong et al [14] highlighted how bias propagation, lack of transparency, and accountability gaps remain major barriers to safe clinical implementation. Similarly, Elbattah et al [15] emphasized the need for transparent auditing mechanisms and robust validation pipelines to mitigate privacy and safety risks in medical AI applications. In line with these perspectives, this study reinforces the necessity of rigorous data governance, responsible model deployment, and continuous ethical evaluation when integrating LLMs into clinical research workflows.

The role of NLP in augmenting documentation accuracy and quality has been demonstrated in neurosurgical contexts. For example, Sastry et al [16] revealed that NLP models improved comorbidity documentation in inpatient admissions, illustrating their potential to enhance clinical record keeping. Similarly, Biswas et al [17] explored the application of NLP in automating the detection of intraoperative elements during lumbar spine surgery, further emphasizing the model’s utility in surgical workflows. In addition, prior work has shown that LLMs can outperform manual abstraction in specific domains and significantly reduce time and labor requirements [18,19]. Compared with manual chart abstraction, which is labor intensive and time consuming, GPT-4o enabled near-instantaneous data extraction. Rule-based NLP systems, although established, require extensive customization and perform inconsistently across heterogeneous text. GPT-4o offers greater flexibility but introduces new challenges related to prompt sensitivity, hallucination risk, and limited explainability.

Another pressing concern is the potential for bias in AI models. As noted by Obermeyer et al [20], algorithms trained on biased datasets can perpetuate health care disparities, underscoring the importance of curating diverse and representative training data. Additionally, regulatory hurdles must be navigated carefully to ensure that these tools are deployed responsibly and ethically, as discussed by Char et al [21].

Looking ahead, the potential applications of LLMs in medicine are vast. Real-time decision support during surgeries, enabled by the synthesis of evidence from medical literature and intraoperative data, could significantly enhance surgical outcomes. Hashimoto et al [22] have already explored the promises and challenges of AI in surgical contexts, laying the groundwork for future innovations. Combining LLMs with complementary AI technologies, such as computer vision for imaging analysis, further expands their utility, as emphasized by Topol [23].

While GPT-4o and similar models represent a significant advancement in health care data mining, their practical implementation requires careful consideration of ethical, technical, and regulatory challenges. LLMs have the potential to not only transform data-driven research but also enhance the quality and efficiency of patient care.

This study highlights the transformative potential of LLMs such as GPT-4o in health care, particularly for mining and structuring complex medical data. The demonstrated accuracy in data extraction underscores their utility in reducing manual workloads and optimizing research and clinical processes. However, challenges such as data security, bias, and regulatory compliance remain significant hurdles. By navigating these obstacles thoughtfully, LLMs could redefine how health care systems manage and use data, ultimately improving patient outcomes and operational efficiencies. As these technologies continue to evolve, their integration must be guided by ethical frameworks and multidisciplinary collaboration to ensure their full potential is realized responsibly.

The small sample size represents a central limitation of this study and results in wide CIs around several accuracy estimates. Consequently, the findings should be viewed as preliminary and interpreted with caution. Although some variables reached perfect accuracy, these were predominantly straightforward structured fields requiring minimal interpretative processing. As such, these results reflect a narrow and highly controlled context and should not be extrapolated as evidence that GPT-4o performs reliably across more complex or heterogeneous clinical data extraction tasks.

Furthermore, the analysis was conducted at a single institution, and all medical reports originated from one neurosurgical service and were written in German. This linguistic and institutional homogeneity may have influenced model performance and limits the generalizability of the results to other clinical environments, documentation styles, and languages. Given the known variability in LLM performance across languages and reporting conventions, external validation is essential.

Future studies involving larger, multi-institutional, and multilingual datasets, as well as more diverse clinical variables, will be necessary to assess reproducibility and determine whether these findings extend to broader clinical or operational contexts.