This study did not involve human participants, patient data, protected health information, biological specimens, or animal subjects. The study evaluated computational models using publicly available anesthesiology board-style educational materials. Therefore, institutional review board approval and informed consent were not required in accordance with institutional and applicable regulations. No personally identifiable information was collected, analyzed, or reported.

To identify optimal configurations and establish a comprehensive benchmark, we conducted comparative experiments between RAG pipeline configurations and their nonaugmented baseline counterparts. To ensure robust evaluation, the study implemented and tested a diverse range of retrieval techniques and current model advancements. Figure 1 summarizes the overall experimental framework, benchmark datasets, and the relationship between the 5 experiments.

To assess the external validity of the RAG configurations on a comprehensive dataset, we assembled a 350-item evaluation set from Anesthesiology Examination and Board Review (7th ed) [19]. (Table S2 in Multimedia Appendix 1) The question pool was primarily derived from summary chapters 10 and 20, selected for their breadth and topical heterogeneity, and supplemented with items from other chapters to yield a balanced distribution. (Table 1)

The prompt templates originally depicted in Figure 2 were adapted to support a uniform 5-option format (A–E), replacing the 3-option format (A–C) used in the preliminary 46-item question set. To maintain structural consistency across the dataset, questions originally containing only four options (A–D) were normalized by appending a null placeholder (“E. Not a valid option”). This ensured that the prompt structure remained invariant for all 350 items.

To facilitate direct comparison with Experiment 1, the core architecture was preserved, using the Meta-Llama-3-8B-Instruct generation model and the same retrieval corpus (Miller’s Anesthesia).

To assess the efficacy of adaptive retrieval strategies, we implemented a modified self-reflective RAG (self-RAG) architecture. This approach applies a single-generation model to answer questions and perform self-critique [27]. For the purpose of controlled comparison, we simplified the setup by removing web-search and postgeneration critique modules, focusing solely on internal knowledge retrieval.

The workflow, illustrated in Figure 4A, uses a single LLM (Meta-Llama-3-8B) to function as both the relevance evaluator and the final answer generator. Consistent with previous experiments, the stella-en_400M_v5 embedding model is used for vector retrieval.

Following the initial retrieval, a document grader module evaluates the semantic alignment between the user query and the retrieved document chunks. As detailed in the prompt template in Figure 4B, the grader is instructed to assign a binary relevance score (“Yes” or “No”) via a JSON output. This decision node dictates the generation strategy: if the retrieved content is deemed relevant (“Yes”), the system executes the “Generate with Context” protocol (standard RAG). Conversely, if the retrieved content is classified as irrelevant (“No”), the system triggers a fallback mechanism, executing “Generate without Context” (Baseline), thereby relying exclusively on the model’s parametric knowledge to prevent hallucination induced by irrelevant context.

This section details a granular analysis of the RAG pipeline’s retrieval components to evaluate their impact on system performance and enhance explainability. To facilitate a controlled assessment, a representative subset of 10 questions is selected from the evaluation question set. These 10 questions were selected from the 350-question set (Table 1), where these specific questions were answered incorrectly (WRONG) across all or most settings (eg, question 139 is answered incorrectly across all tested RAG configurations) for deep retrieval analysis. The detailed locations for each question are presented in Table 2 below.

Given the high complexity of the primary corpus (Miller’s Anesthesia), we established a controlled environment to isolate the generation model’s reasoning capabilities from retrieval noise. The corresponding answers and explanations from the Anesthesiology Examination and Board Review served as a new retrieval source, categorized into two experimental conditions:

Concurrently, we examined the impact of model scaling on performance. As illustrated in Figure 5, the experimental setup included large embedding models (scaling up to 1.5 billion parameters) and generation models (up to 72 billion parameters) to test the hypothesis that increased parameter count correlates with improved accuracy. For these experiments, generation hyperparameters were fixed at the optimal values identified from the result of experiment 1 (temperature=0.1; top-p=0.1).

Finally, we evaluated the influence of text segmentation on retrieval quality. Beyond varying the retrieval depth (top-k), we compared two distinct chunking paradigms:

Building upon the optimal configurations identified in Experiments 1‐4, this phase integrates the most effective RAG components to benchmark the next generation of reasoning-oriented LLMs. The pipeline uses the high-capacity embedding model, gte-Qwen2-1.5B-instruct, and uses semantic chunking (threshold set at the 95th percentile) to maximize context coherence.

As illustrated in Figure 6, the study evaluates two distinct model categories. Conventional instruction-tuned models include the Llama series (Llama-3 to 3.3) and Qwen2.5-72B. Reasoning-enhanced models include the Qwen3 series (8B and 32B), which incorporate advanced chain-of-thought capabilities.

To ensure statistical robustness, evaluation was conducted on three randomly sampled subsets of 100 questions each, drawn from the 350-item corpus. (Table S3 in Multimedia Appendix 1) Unique random seeds were applied to ensure reproducible, independently generated sampling trials; because each 100-question subset was sampled from the same 350-item corpus, some questions could appear in more than one subset, and results are reported as the mean accuracy across the three trials. The combined 300-question analysis was used to provide more stable model-level comparisons across related retrieval-complexity conditions rather than to imply a single homogeneous statistical population.

To rigorously assess model robustness against retrieval noise, we designed a 6-level complexity spectrum. The “Bare” condition represents answers solely based on intrinsic parametric knowledge, without external context. The “Direct” condition injects the specific ground-truth explanation directly into the prompt. The “Combined” (needle-in-a-haystack) condition embeds the ground truth within a consolidated block of relevant clinical explanations. Because multiple questions in the 350-item corpus share the same source explanation block (eg, multipart vignette items that reference a common explanatory passage), the consolidated source corpus contains 263 unique explanations rather than 350. To simulate varying degrees of retrieval difficulty, the combined source is fused with increasing amounts of irrelevant distractor content (from unrelated ASA questions). This creates three graded difficulty tiers: easy, medium, and hard. With the above base combined set contains 263 explanations; we then inject 50, 125, and 274 additional (irrelevant) explanations, yielding totals of 313, 388, and 537 explanations for easy, medium, and hard, respectively. This stratification enables a granular analysis of how reasoning models distinguish relevant signals from clinical noise compared to conventional architectures.

The primary outcome measure was the accuracy rate, defined as the percentage of correctly answered questions. Evaluation results are presented as the mean across the randomized test subsets. To analyze performance differences across multiple model configurations, the Cochran Q test was first used to detect any overall statistically significant differences among the classifiers on the same dataset. Subsequently, the McNemar test was used for post-hoc pairwise comparisons to identify specific performance improvements between model pairs. To account for multiple hypothesis testing, a Bonferroni correction was applied to the significance threshold. All analyses were performed using Python (scipy.stats library), and a 2-tailed P value of <.05 was considered statistically significant.

This statistical analysis was conducted on the results obtained from experiments 2 and 5. In experiment 2, we analyzed performance differences among RAG configurations that varied in embedding models, while holding the hyperparameters (top-k, top-p, and temperature) constant. For instance, Cochran Q and McNemar tests were applied to compare the answer patterns of 350 questions between the Baseline model and the RAG0 configuration, with the hyperparameters set to top-k=4, top-p=0.1, and temperature=0.1. In contrast, Experiment 5 involved three distinct 100-question test sets for each RAG configuration, each corresponding to a different LLM model. To enhance the statistical power, the answers from these three test sets were aggregated into a single array of 300 answers (wrong or correct), enabling a more robust statistical analysis. For each analysis (eg, comparing different embedding models under the same RAG hyperparameters), we report the post-hoc pairwise McNemar results, including each configuration’s win count, Bonferroni-corrected α, and the computed P value.

All retrieval and generation experiments were conducted in a Rocky Linux 8.4 environment using Python (version 3.6.8). The implementation of the RAG pipeline and model inference used a combination of the HuggingFace, Transformer, and LangChain libraries, along with the PyTorch framework. The workstation server was equipped with eight V100 GPUs (32 GB each) and 767 GB of RAM. Models with fewer than 32 billion parameters were executed on a single GPU, whereas larger models, such as the Qwen2.5-72B, required up to 4 V100 GPUs to run both the LLM and the embedding model concurrently.

To provide an overview of the hyperparameter optimization results, we first summarize the distribution of top-performing temperature and top-p configurations across all experimental settings (Figure 7). Overall, the RAG-empowered framework consistently achieved higher accuracy, as indicated by the brighter colors in Multimedia Appendix 2. Across the 15 evaluated configurations, 30 maximum-performance occurrences were identified due to ties in peak accuracy; among these, 9 were observed at the (temperature, top-p) pair (0.1, 0.1) in the NORMAL zone. This finding is supported by the frequency distribution of optimal hyperparameter configurations (temperature and top-p) across different embedding models and top-k values (Figure 7). In general, peak performance aligned with the most conservative pair within each region: (0.1, 0.1), (0.05, 0.01), and (0.005, 0.001) for the normal, fine, and very fine zones, respectively. The (0.1, 0.1) pair achieves the maximum accuracy 9 times, which is significantly higher than any other configuration. Additionally, this pair achieves the highest overall accuracy of 71.74%, using the MedEmbed embedding model with a top-k of 8. Based on these comprehensive results, the 3 temperature and top-p pairs, (0.1, 0.1), (0.05, 0.01), and (0.005, 0.001), were selected for subsequent experiments.

In contrast to the strong performance observed on the 46-question set, the larger, more topic-diverse set yields markedly different results (Figure 8). The baseline (no-RAG) model achieved 50.29%, representing a decrease of 6.21 percentage points compared to its performance on the 46-question set (Table S1 in Multimedia Appendix 1). Furthermore, the performance gains achieved via RAG were modest on this corpus: the best configuration (stella-en_400M_v5 with top-k=12 across all three pairs of temperature and top-p) yielded a maximum improvement of 6.28 percentage points over the baseline, while other settings delivered marginal improvements ranging from 2 to less than 5 percentage points.

Although Experiment 1 indicated that a medical-tuned embedding (eg, MedEmbed-large-v0.1, top-k=12) could outperform a general-text embedding, Stella_en_400M_v5, on a narrower set (71.7% vs 65.2% for top-p=0.1, temperature=0.1, and top-k=8; Multimedia Appendix 2), this advantage did not transfer to the larger 350-question benchmark. Embeddings trained on published biomedical corpora or clinical-note–style data may not be explicitly aligned with the specific nuances of anesthesiology question-answering, resulting in relative underperformance (Figure 8), which is consistent with a domain and topic mismatch.

Cochran Q and post-hoc McNemar test results for Experiment 2 under retrieval depths top-k=4, 8, and 12 are demonstrated in Table S4 in Multimedia Appendix 1. For the most conservative retrieval setting (top-k=4) combined with the empirically stable sampling regime (temperature=0.1, top-p=0.1), Cochran Q was not significant (P=.06). This indicates that varying the embedding model does not produce a statistically detectable change in the 350-question answer pattern. By contrast, all other Cochran Q tests reached statistical significance.

After Bonferroni correction (corrected α=.005), most pairwise McNemar comparisons between Baseline and individual RAG configurations were not significant. (Table S4 in Multimedia Appendix 1) This indicates that conventional RAG does not consistently induce a statistically reliable shift relative to baseline (ie, unaugmented) generation on this heterogeneous 350-item set. Nevertheless, among the tested embedders, Stella exhibits the most pronounced divergence from the baseline in several settings (demonstrating larger discordant “win” counts and comparatively smaller McNemar P values), which is coherent with its generally higher observed accuracy across configurations. Specifically, at a retrieval depth of top-k=12, the performance of the Stella embedding model diverged significantly from the Baseline under the two more deterministic sampling regimes (P value<.01 and P value<.01, for temp=0.05 with top-p=0.01 and temp=0.005 with top-p=0.001, respectively), surviving the Bonferroni correction.

The performance of the Self-RAG pipeline compared with the baseline model is illustrated in Figure 9. As shown in Figure 9, the self-RAG implementation yielded only marginal improvements ranging from 2.28 to 4.85 percentage points, relative to the best baseline accuracy of 50.29%. This indicates no substantial benefit over the conventional RAG system (Figure 2B).

Figure 11 reports the mean accuracy across three 100-item sets for 7 LLMs as retrieval complexity increases. As demonstrated in the figure, accuracy consistently declined as the retrieved evidence became longer and contextually denser, progressing from the direct to the combined (hard) conditions.

The reasoning-oriented Qwen3-32B model achieved the highest accuracy, exceeding the largest tested conventional model, Qwen2.5-72B-Instruct by 1‐6 percentage points across conditions (eg, increasing from 92% to 96% in the Direct, from 86% to 89% in the combined, and from 81% to 87% in the Combined Medium setting). Following Qwen2.5-72B-Instruct, the Qwen3-8B model also demonstrated strong performance, achieving 67% in the Bare, 90% in the direct, and 84% in the combined setting, higher than all models in the Llama family. Within the Llama family, while the Llama-3.3-70B model outperformed the smaller Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.2-3B-Instruct models across all conditions (eg 80% vs 70%, 76%, and 56% in the combined condition; 69% vs 63% and 67%, and 48% in the hard condition), the gains were diminished when the retrieval difficulty increased, especially against Llama-3.1-8B-Instruct variant.

Figure 12 shows the relationship between model scale, reasoning capability, and accuracy across all retrieval conditions. Across the evaluated settings, larger models generally achieved higher correct ratios than smaller models within the same model family (ie, Llama or Qwen). This performance gap by model size became more pronounced as retrieval complexity increased. In addition, reasoning-enabled models maintained accuracy levels comparable to those of larger conventional models and exhibited a more stable accuracy across different retrieval conditions.

Experiment 5’s Cochran Q and post hoc McNemar comparisons across the 7 LLM-based RAG systems at multiple retrieval-complexity tiers are reported in Table S6 in Multimedia Appendix 1. Cochran Q was highly significant across all tiers (P<.05). Pairwise McNemar results demonstrated that the Qwen family, especially the reasoning models (Qwen3-8B and Qwen3-32B), exhibited the most consistent and substantial divergence from the Llama series. Specifically, Qwen3-32B significantly outperformed 4 nonreasoning models (Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, and Llama-3.3-70B-Instruct) in bare, combined, and easy conditions, 5 models in direct, medium, and hard conditions (Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Llama-3.3-70B-Instruct, and Qwen3-8B), but nonsignificantly outperformed nonreasoning Qwen2.5-72B-Instruct model.

We compared a RAG pipeline with an unaugmented baseline LLM and evaluated variations in core components (generator scale, embedding model, chunking strategy, sampling hyperparameters, and reasoning capability). On our benchmarks, the RAG framework delivered higher and more consistent accuracy than baseline generation when sampling temperature and top-p were tuned for stability; among the tested low-stochasticity configurations, temperature=0.1 and top-p=0.1 frequently yielded comparatively stable performance across experiments, although the exact optimal configuration varied across retrieval settings and embedding models. (Multimedia Appendix 2 and Figure 7) These findings suggest that lower-stochasticity sampling settings may improve performance stability in structured medical question-answering benchmarks, as excessively high parameter values can introduce variability that compromises factual consistency, whereas overly restrictive settings may reduce necessary inferential flexibility.

Retrieval quality emerged as the primary determinant of overall performance. As retrieval depth increased, configuration-dependent differences in model reasoning were amplified, highlighting the delicate balance between information recall and context noise. Although medical-tuned embedders demonstrated strong performance on narrow topical subsets, their advantage did not transfer to the broader 350-item corpus, a finding that is consistent with a domain and task mismatch. In contrast, a high-capacity GTE model (gte-Qwen2-1.5B-instruct) achieved higher retrieval precision and improved end-to-end accuracy. Table S4 in Multimedia Appendix 1 suggests that model capacity, combined with broad pretraining, can outweigh narrow biomedical fine-tuning when the target data distribution shifts, providing richer, more discriminative vector representations for precise knowledge extraction.

The choice of chunking strategy also critically influenced context quality. Naïve recursive chunking (fixed windows) produced fragmented, out-of-context segments from the Miller’s Anesthesia textbook [26], injecting headers and incomplete sentences into the vector store. Semantic chunking reduced these fragmentary chunks and retrieved passages with greater semantic alignment to the gold-standard evidence in illustrative cases. (Figure 10 and Table S5 in Multimedia Appendix 1) Consequently, we adopted it as the default processing method in subsequent experiments. Furthermore, our evaluation of a simplified Self-RAG variant demonstrated that simply increasing the architectural complexity of the pipeline does not necessarily yield clinically meaningful improvements, as the underlying generation models often inherit fundamental limitations in deep medical knowledge representation. (Figure 9) Therefore, within the present implementation, optimization of retrieval quality and contextual refinement appeared to contribute more substantially to performance than the additional postretrieval grading mechanism.

Reasoning-oriented LLMs demonstrated robust competitive performance. Among the conventional (nonreasoning) models, Qwen2.5-72B-Instruct served as the strongest baseline, achieving higher scores than its Llama model family counterparts, which is consistent with prior anesthesia benchmarks [30]. On the studied datasets, the reasoning-centric Qwen3-32B achieved even higher accuracy than the conventional Qwen2.5-72B-Instruct model. (Figures 11 and 12) These findings suggest that reasoning-oriented training and related model characteristics may improve robustness under complex retrieval settings, as increases in raw parameter capacity do not guarantee proportional performance gains on highly specialized anesthesiology examination benchmarks. Across increasing levels of retrieval difficulty, reasoning-oriented models demonstrated greater robustness to noisier and denser contextual inputs. This points to a complementary architectural approach, where a reasoning model, combined with optimized retrieval, can improve performance in structured medical question-answering benchmarks and may inform future development of medically oriented retrieval systems. However, these differences should not be interpreted as arising solely from reasoning capability, as variations in architecture, pretraining data, instruction tuning, and posttraining optimization may also contribute.

To further characterize failure modes, we reviewed all eligible cases in which Qwen3-32B answered correctly under the Direct condition but incorrectly under the Combined condition in Experiment 5. Among the 24 reviewed cases, 19 were associated with retrieval mismatch, in which the retrieved passages did not contain the key evidence needed to support the correct answer. Four additional cases reflected insufficient or only partially relevant retrieval, while 1 case reflected failure to correctly use partially relevant retrieved information despite the presence of related contextual evidence. These findings suggest that most residual errors under retrieval-complexity settings were primarily associated with retrieval quality and distractor-heavy contextual inputs rather than solely reflecting lack of underlying question-answering capability. Detailed examples and retrieved contexts are provided in Table S7 (Multimedia Appendix 1).

Initial explorations of LLMs in anesthesiology primarily evaluated the intrinsic parametric knowledge of standalone models. For instance, Shay et al [31] assessed ChatGPT-3.5 on ABA–style practice questions, reporting an overall accuracy of 56.2%, falling short of standard passing thresholds. Similarly, Angel et al [12] demonstrated that while early models like Bard (46.7%) and GPT-3 (58.3%) failed to pass the ABA written examination, the advanced closed-source GPT-4 achieved 78.3%. These findings closely mirror our baseline results; our unaugmented, open-source Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.2-3B models achieved comparable 49%, 54%, and 42% accuracies, respectively. (Figure 8) In addition, our larger-scale Qwen2.5-72B-Instruct also achieved a comparable 72% accuracy. (Figure 8) This comparison confirms that the raw parametric knowledge of smaller models aligns with earlier foundation models and is insufficient for complex anesthesiology examinations.

More recently, the literature has demonstrated how RAG and reasoning can elevate model accuracy on medical licensing thresholds. Recent research by Elkin et al [32] on the USMLE revealed that while large Llama-3-70B models passed the exams natively and achieved up to 92% accuracy with RAG, integrating semantic clinical knowledge was necessary to help a smaller Llama-2-13B model cross the 60% passing threshold on Step 3 (scoring 60.2%). This scale-dependent benefit closely aligns with our findings: our unaugmented Llama-3.1-8B-Instruct baseline (54%) saw only modest gains with RAG (76%), whereas our larger conventional Qwen2.5-72B-Instruct model equipped with RAG achieved 72% and 86% accuracies without and with RAG, respectively (Figure 8). Concurrently, meta-analyses of global medical exams show that advanced reasoning models dominate medical examination benchmarks, with models like GPT-o1 and DeepSeek-R1 achieving overall accuracy rates of 95.4% and 92%, respectively [33]. This emphasizes the critical role of robust test-time reasoning in specialized medical contexts [30]. Similarly, our reasoning Qwen3-32B model integrated with an optimized RAG pipeline achieved up to 89% accuracy on complex, distractor-heavy retrieval tasks (Figure 11).

Although this work evaluates model accuracy on curated anesthesia question sets, it does not constitute clinical validation in real-world patient care scenarios. In terms of document complexity control, the Direct context setting estimates a theoretical performance ceiling under perfect retrieval and is not deployable in practical applications. Several findings (eg, table-level retrieval metrics or single-question chunking illustrations) are based on small diagnostic subsets; broader replication is needed to confirm generalizability. Moreover, this study did not exhaustively audit failure modes (eg, hallucination typology and reasoning chain validity) or conduct clinical expert adjudication of the generated rationales. Finally, evaluations of model safety, bias, and calibration were out of scope and require a significant dedicated study before any clinical deployment can be considered. In addition, the evaluated benchmark consisted of structured text-based anesthesiology board-style questions. Accordingly, the present findings are most directly applicable to controlled medical question-answering benchmarks and may not generalize to open-ended clinical dialogue, real-time guideline retrieval, or multimodal anesthesiology tasks involving images, physiologic waveforms, or tabular data.

In conclusion, our results demonstrate that a RAG-powered LLM pipeline can achieve greater than 80% accuracy on complex anesthesiology board-style examinations under controlled retrieval conditions (oracle or curated context in experiments 4 and 5), whereas under open RAG retrieval on the heterogeneous 350-question benchmark in experiment 2, gains over the unaugmented baseline were modest (50.29% to 56.57%). These findings suggest its potential utility in benchmark-oriented evaluation settings. The relatively compact size of the evaluated open-source models (eg, 8 billion to 72 billion parameters), compared with massive, proprietary foundation LLMs like GPT-4, may facilitate local deployment and real-time responses. This mitigates critical infrastructural issues such as network latency, server overload, and patient privacy concerns. Moreover, the RAG framework facilitates dynamic knowledge updates, allowing new medical guidelines to be integrated into the document corpus without requiring expensive, computationally intensive model retraining. By explicitly outputting both the generated reasoning steps and the retrieved context, the system provides partial transparency into the mechanisms driving its final answer. However, optimizing a RAG system requires significant methodological effort to tune hyperparameters and test retrieval strategies to ensure reliable, accurate results in structured benchmarking settings.