We created an LLM framework to extract oncological biomarker features and identify TNBC subtype statuses from pathology reports, as depicted in Figure 1.

The dataset used in this analysis was obtained from The Cancer Genome Atlas/Genomic Data Commons (TCGA/GDC) data portal. The reports were downloaded as PDF files using the GDC API. Each downloaded pathology report was assigned the corresponding GDC file’s universally unique identifier (UUID) to ensure traceability throughout preprocessing, feature extraction, and evaluation. A total of 532 pathology reports, including both scanned and text-based documents, were included in this study. Feature selection was based on a curated Breast Cancer Feature Checklist consisting of 4 clinically relevant domains, as shown in Figure 2. First, core receptor status, such as ER, PR, HER2, and TNBC definition, was selected, which helped differentiate hormone receptor–positive, HER2-enriched, and triple-negative breast cancer. Receptor status helps in choosing systemic therapy. Second, tumor status, such as tumor size category, histologic grade, and histology type, was selected, as this reflects primary tumor burden and intrinsic aggressiveness. Third, proliferation and molecular markers, such as Ki-67 index, p53 status, and the basal markers CK5/6 and EGFR, were selected because they are associated with tumor growth rate. Finally, nodal status, such as lymph node stage and lymphovascular invasion, was selected because it reflects metastatic potential. However, since the pathology reports were not uniform, not all features were obtained from all reports.

Deidentified pathology reports of patients with breast cancer were sourced from The Cancer Genome Atlas (TCGA) via the Genomic Data Commons (GDC) Data Portal. Since the experiment only analyzed secondary, deidentified data, no need for ethics committee approval and informed consent was required due to the nature of the experiment itself. All data were accessed and analyzed according to the rules set forth by TCGA/GDC. In future deployment in clinics, proper patient data privacy should be observed when using commercial LLMs.

Although this study evaluated commercial large language models through API-based inference, only deidentified pathology reports from The Cancer Genome Atlas were processed. In real-world clinical deployment, identifiable pathology reports should undergo appropriate deidentification before external processing. Institutions should ensure compliance with applicable privacy regulations and evaluate the security implications of transmitting clinical data to third-party AI services.

A critical consideration in deploying LLMs for clinical applications is the risk of hallucination. Getting results with greater accuracy and minimizing hallucinated results may be the scope of future research. Hallucination is a common complication during feature extraction. While direct hallucination quantification was beyond the scope of this study, the observed precision-recall trade-offs provide indirect evidence of extraction reliability. Zero-shot prompting of models with lower precision, such as GPT-4o mini for HER2 extraction (precision, 56.40%‐59.57%), may be at elevated risk of false-positive extractions. Conversely, models with lower recall, such as GPT-5 nano with zero-shot prompting for ER extraction (recall, 62.20%), may be more likely to miss true values than to fabricate them. These performance characteristics should inform model selection based on the clinical priority of minimizing false positives versus false negatives. Due to various factors determining tumor size, as well as indirect references to immunohistochemistry, it is essential to extract contextually and factually accurate information [24,25]. Therefore, a thorough ground-truth analysis was performed manually with domain experts to check the precision of the model extraction. For instance, various sizes of tumor were present in most of the pathology reports, and thus, determining the largest circumference and most detrimental carcinoma was extremely crucial. Prompt generation must be specific and succinct. CoT prompting applies this technique of performing a deeper analysis of the meaning of the IHC indicators and extracting the relevant values. Tumor size had different formats throughout the pathology reports, and hence, extracting not only the largest diameter but also the complete dimensions became necessary. This had a better outcome in few-shot prompting through the use of examples given on how to extract the dimensions [26,27]. Thus, prompt generation requires being extremely specific and providing a clear context for medical terms as well as pairing prompts with proper examples to extract information in the desired manner. These strategies help in reducing hallucinations and can also be used for training future models.

In the current study, results show that prompt engineering is crucial in boosting the performance of clinical information extraction from heterogeneous pathology reports. This observation is similar to the conclusions drawn in recent studies, in which reasoning greatly boosts the performance of LLMs in clinical natural language processing due to its ability to facilitate structured reasoning of the narrative [28-31]. The better performance demonstrated by CoT prompting is due to the ability of CoT prompting to help the LLM break down complex extraction tasks into reasoning steps prior to responding. The pathology reports of breast cancer have free-text narratives, inconsistency in style of reporting, multiple lesions, and variations in IHC format. This makes the task more complex, as it needs more than matching with keywords. CoT prompting is helpful for accurately identifying clinical biomarkers without any ambiguity. The advantages of reasoning-based prompts have also been highlighted by several studies related to clinical summarization, information extraction, and question answering in medicine [12,32].

From the analyzed models, Gemini-2.5 Flash-Lite showed the best overall accuracy in extraction despite being one of the smallest models, implying improvement in the domain of instruction following and reasoning skills. Therefore, small, instruction-tuned models are more practical due to their efficiency and better computational capabilities [28,30]. Moreover, the comparison showed that the prompt design impacted the extraction performance of all models, showing that the optimization of prompts is just as crucial as the choice of the model in healthcare tasks.

In line with previous studies, extraction accuracy was higher for ER, PR, and HER2 status compared to tumor size [33]. However, hormone receptor status is usually expressed in a standardized language, while tumor size may involve various measurements, macro- and microdimensions, or even multiple lesions in one report. Therefore, contextual reasoning is needed to recognize the clinically relevant measurement of the primary tumor. This demonstrates that the performance of extraction depends not only on the model but also on the complexity of pathology reports. Reasoning-based models for LLMs are able to provide help to pathologists as they are capable of extracting clinically relevant markers from unstructured pathology reports through automation. However, expert review is necessary prior to clinical application.

Hallucination is an important drawback that hinders the clinical implementation of LLMs. Ambiguous language, multiple numerical measurements, and indirect indication of immunohistochemistry in the pathology reports may lead to clinically plausible but incorrect results. To prevent this problem, manual verification of each extracted variable was conducted in the current study. Prompt engineering cannot solve the issue completely, and human verification is still needed [34,35].

This research faces certain limitations. It used 532 pathology reports and considered only 4 important clinical parameters (ER, PR, HER2, and tumor size). Generalizability needs to be enhanced through validation on more comprehensive datasets at multiple sites and through consideration of other pathological parameters. In addition, using proprietary LLM models may impact reproducibility due to the constant development of these commercial models. Thus, future research should explore medical LLM models for specific domains, as well as RAG methods and clinical validation.

In summary, the results show that reasoning-driven prompt engineering improved the performance of LLMs in identifying clinically important biomarkers from unstructured breast cancer pathology reports. The consistently superior results obtained using the CoT prompting method in various models illustrate the potential of prompt engineering as a viable approach to automating the extraction of clinical data. Innovations in the fields of prompt engineering, prevention of hallucinations, and external validation on different clinical datasets, as well as the creation of medically validated LLMs, will pave the way for the incorporation of AI in digital pathology and precision oncology pipelines [28,29,35].

One limitation of the current study is the absence of baseline evaluation when compared with other approaches for information extraction, which include the rule-based or regular expression–based approaches. Since the aim of the current research was to conduct evaluation of lightweight LLMs with different prompt formats, the comparison with rule-based approaches was out of scope. Future research should conduct analyses of the computational efficiency, implementation ease, and performance of LLM approaches compared with traditional rule-based approaches.

The current study was conducted on 1 dataset only, comprising 532 TCGA breast cancer pathology reports. While the dataset allowed us to perform a comprehensive comparison among different prompting approaches, future research must be conducted on multi-institutional pathology reports written in different styles.

Another limitation is that the current study concentrated on finding clinically relevant biomarkers (ER, PR, HER2, and tumor size) and TNBC. Classification of other molecular subtypes, such as Luminal A, Luminal B, and HER2-enriched breast cancer, and TNM staging based on American Joint Committee on Cancer criteria were out of the scope of the study. The future work would include integration of more pathological factors in order to perform classification of other molecular subtypes and TNM staging automatically.

In the current study, the tested LLMs produced categorical predictions but not probability-based predictions; thus, receiver operating characteristic–area under the curve analysis could not be performed. In future work, confidence-based prompting techniques or probability-based LLM output would need to be considered to perform threshold-based performance analysis.

While GPT-5 nano demonstrated 83% sensitivity for TNBC detection, this false-negative rate is not adequate for autonomous clinical decision-making. Thus, the proposed framework should be considered a clinical decision support framework that requires review by a professional pathologist. Finally, reproducibility is bound to change with future model developments of commercials LLMs. Validation through domain-specific medical LLMs, RAG, and clinical validation will add more robustness to the proposed framework.

This study did not follow a formal AI-diagnostic reporting checklist such as TRIPOD-AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis Using Artificial Intelligence) or STARD-AI (Standards for Reporting of Diagnostic Accuracy Studies Involving Artificial Intelligence). While model settings and error-handling procedures were reported, dataset flow and prompt templates were described narratively rather than as a fully reproducible pipeline, and ground-truth annotation protocol details (number of annotators, interannotator agreement) along with external validation on independent, multi-institutional datasets were outside the scope of the original study design. Future work will adopt a formal AI-reporting checklist and incorporate multiannotator ground-truth curation with agreement statistics, along with prospective external validation, to strengthen the reproducibility and clinical generalizability of this framework.

This study highlights the use of LLMs with appropriate prompting techniques for oncology feature extraction from pathology reports. Gemini 2.5 Flash-lite, GPT-5 nano, and GPT-4o mini achieved the highest accuracy in biomarker status extraction. Among these, GPT-5 nano paired with CoT prompting gave the best results in the detection of tumor size. This finding shows that smaller and more cost-efficient nano models can give better results compared to larger models for information extraction in clinical documents when prompted properly. However, GPT-5 nano had considerably higher processing time for all reports. This could make it less suitable for large-scale deployment. It has immense potential in assisting pathologists and oncologists by automating the extraction of key pathological features from unstructured reports, thereby increasing efficiency and reducing the possibility of human error. This would minimize manual efforts and improving clinical workflow efficiency. Furthermore, when integrated with agentic model context protocol–based workflows [36,37], such systems could continuously validate extracted results, flag TNBC cases in real time, and trigger clinical actions, leading to an AI-driven precision oncology system. This could transform how high-risk breast cancer subtypes such as TNBC are identified and managed in routine clinical practice [38,39].