Korean Medical Consultation With Open-Weight Large Language Models: Pilot Comparative Evaluation of Retrieval-Augmented Generation With Metadata Filtering

Introduction

Background

With the rapid advancement of artificial intelligence (AI), medical chatbots are playing an increasingly pivotal role in providing personalized health care consultations and managing patient data [1,2]. However, despite the overall progress of large language models (LLMs), a performance gap between English and Korean persists, as technical breakthroughs in high-resource languages do not inherently transfer to midresource languages such as Korean [3,4]. These limitations act as structural constraints when directly applying general-purpose LLMs to the Korean health care environment, particularly in their failure to sufficiently reflect local medical systems, legal regulations, and ethical standards [5,6].

Concurrently, South Korea’s medical ethics, guidelines, and legal norms are closely integrated with highly standardized prescription and treatment protocols, which are intrinsically linked to the National Health Insurance reimbursement system, licensing requirements, and clinical guidelines from professional societies [7,8]. Consequently, standardized procedures and criteria often carry more weight than individual clinical discretion [9]. As a result, recommendations from general-purpose GPT-based models—primarily trained on United States or European guidelines—may conflict with Korean regulations, potentially suggesting interventions that are prohibited, not covered by insurance, or legally sensitive [10-12]. Such discrepancies pose significant risks, including information errors, misdiagnosis, and institutional misalignment, thereby compromising both patient safety and legal liability [13,14]. This misalignment represents a substantial barrier, revealing the structural limitations of applying general LLMs directly to the Korean health care context.

Furthermore, Korean medical inquiries are characterized by patients describing their symptoms, progress, and concerns in detailed narrative forms [15,16]. A primary challenge for medical chatbots is accurately extracting clinically essential information from such verbose text and matching it with the appropriate Korean medical guideline documents [17,18]. Specifically, as Korean is an agglutinative language with high morphological complexity—where medical vocabulary consists of a mixture of Hanja, loanwords, and native Korean terms—specialized text processing and retrieval technologies optimized for the Korean language are required [19,20].

Current LLM-based medical chatbots, including commercial models, primarily rely on pretrained internal knowledge. This often leads to “hallucinations”—generating factually incorrect information—and a failure to ensure reliability based on the latest domain-specific evidence [21,22]. Specifically, when consultation content pertains directly to a specific country’s insurance standards, local medical systems, or locally used products and ingredients, simple learning-based response mechanisms struggle to guarantee the precision and up-to-dateness required for medical safety [10]. To address these limitations, the ecosystem has recently shifted toward state-of-the-art open-source LLMs such as Llama 3 (Meta), Mistral (Mistral AI), and Gemma (Google LLC), which allow for higher levels of customization, including on-premises deployment, domain-specific fine-tuning, and enhanced data privacy [23]. Nevertheless, even high-performance open-source models require architectural reinforcement, such as systematic integration with external knowledge, to ensure accuracy and safety in the medical domain. In this context, retrieval-augmented generation (RAG) is emerging as a promising solution [24]. By using dense retrieval techniques and Korean-specific embedding models, RAG enables chatbots to access refined medical databases, allowing this study to focus on generating fact-based responses grounded in prevalidated medical literature, guidelines, and local regulations [25].

However, RAG alone faces limitations in sufficiently processing the unique characteristics of Korean medical inquiries. As these inquiries are often long and narrative, with the patient’s core intent frequently obscured by peripheral context (such as emotions or treatment experiences), simple vector similarity-based retrieval often fails to select the most appropriate evidence documents [26]. As a sophisticated enhancement to overcome this, metadata filtering has been proposed. This approach involves the LLM extracting structured metadata—such as clinical department, disease category, age group, and specific drugs or procedures—from the query and filtering the database accordingly before the retrieval stage, thereby narrowing the search space [27]. This ensures that the system performs retrieval and generation only on the most relevant subsets of documents, which significantly improves response accuracy and consistency, especially for complex, multistep medical questions [28]. Despite these developments, there is a lack of empirical research systematically comparing and evaluating the performance of various open-source LLM architectures that combine RAG with metadata filtering within the Korean medical context [29]. Furthermore, benchmarks and evaluation frameworks that reflect the unique characteristics of Korean medical inquiries are not yet sufficiently established [30]. We identified a research gap that necessitates an empirical validation of open-source LLM-based architectures integrating RAG and metadata filtering to design safe and reliable medical chatbots tailored to the Korean health care environment.

Objectives

Therefore, we aimed to develop and comparatively evaluate a Korean medical consultation chatbot architecture that integrates RAG with LLM-based metadata filtering to better align responses with Korean clinical information needs and locally grounded evidence. We constructed a Korean medical document corpus (10.4 GB; 487,277 segments) and implemented an RAG pipeline using a Korean-specialized embedding model with an Elasticsearch-based retriever, augmented by metadata filtering that extracts structured clinical cues (eg, disease-, symptom-, or drug-related signals) to narrow the retrieval space before generation. We then compared 5 open-weight LLM backbones (Qwen3:4B, Mistral:7B, Llama-3.1:8B, Gpt-Oss:20B, and Gemma3:27B) under 3 configurations—baseline (no retrieval), RAG-only, and RAG plus metadata filtering—to determine when retrieval and filtering improve Korean health consultation responses. Performance was validated on 226 Korean consultation questions spanning 7 common disease categories, with an emergency medicine specialist rating each response for accuracy, safety, and helpfulness using a predefined scoring protocol.

Methods

Data Collection

This study focused on collecting and preprocessing Korean medical consultation data. To ensure a comprehensive dataset in diverse medical fields, we incorporated a variety of medical fields and consultation scenarios. The main data sources included publicly available disease information from the official websites of major Korean hospitals (n=3537; eg, disease encyclopedias accessible via the public health information portal of Asan Medical Center), legally usable open-source textbooks (n=2336), public data from the Ministry of Food and Drug Safety (n=87,635), and health-related knowledge data from major Korean search engines (n=49,681). These datasets were rigorously curated and refined through the supervision of health care professionals among the coauthors.

The data preprocessing process involved four key steps: (1) cleaning the data by removing unnecessary characters, duplicate entries, and nonmedical content to ensure dataset integrity; (2) ensuring consistency by correcting spelling errors, managing synonyms, and standardizing medical terminology; (3) conducting a rigorous screening process to ensure the complete absence of any personally identifiable information (PII) through a dual-verification approach, combining automated rule-based detection with manual review by medical professionals; and (4) classifying data by disease categories, examination and procedure types, body regions, medical specialties, and consultation types to enhance the efficiency of subsequent analyses. As a result, we constructed a medical document corpus of 10.4 GB, segmented into 487,277 text units for efficient retrieval within the RAG system. An overview of the data collection and management web user interface is shown in Figure 1.

This high-quality Korean medical consultation dataset, constructed through this process, is crucial for developing the RAG-integrated small large language models. This curated dataset includes frequently asked medical questions from the general public and answers based on the clinical expertise of medical professionals. By using this data for performance evaluation, we aimed to more accurately assess our service’s accuracy, safety, and helpfulness in a real-world medical consultation environment within Korea.

‎

Implementation of Default LLM

Concerning the LLM backbone, we built the LLM using Ollama as the backbone. The models were loaded from the Ollama library, allowing for model selection—including Qwen-4B (Alibaba Group), Mistral-7B (Mistral AI), Llama-3.1-8B (Meta), Gpt-Oss-20B (Google LLC), and Gemma-3-27B (Google LLC)—depending on the test. The service was run in an NVIDIA A100 (NVIDIA Corp) with 40 GB of video random access memory and an Ubuntu 22.04 (Canonical Ltd) environment to ensure the models could operate at optimal performance. Each model is briefly summarized in Table 1.

‎

Concerning conversation flow design, in health counseling, responding accurately and appropriately to user inquiries is crucial. Therefore, we carefully structured the conversation flow. This includes defining a step-by-step dialogue structure that incorporates initial greetings, symptom inquiries, requests for additional information, and guidance to visit a medical institution in case of an emergency, enabling users to interact smoothly with the chatbot. The chatbot also includes a memory function to remember the context of previous conversations and provide context-aware responses.

Concerning information extraction, when a user uploads an image of a prescription or medical report, the chatbot analyzes the image to extract medication and disease codes and presents the relevant information in a tabular format. This feature allows users to easily check detailed information about their health status and prescribed medications.

Concerning RAG, to use the database constructed in the data collection phase, we built an RAG and application programming interface system, enabling the LLM to generate answers based on this data. This approach ensures that the chatbot provides reliable and evidence-based information, consequently delivering more accurate and trustworthy medical responses to the user. A more detailed explanation of RAG will be provided in the following section.

About RAG

Concerning Korean embedding, in this study, we adopted the upskyy/BGE-M3-Korean embedding model [31], which is specialized for the Korean language, to maximize the accuracy and efficiency of information retrieval. The key reason for selecting this model is that BGE-M3 possesses a “multigranularity” feature, making it specialized for Korean embeddings. This means it can process texts of various lengths—from short user queries to long, specialized medical documents—with consistently high performance. Furthermore, this model is fine-tuned on Korean data, enabling it to accurately capture complex Korean medical terminology and the subtle semantic nuances of user queries.

Concerning the Elasticsearch store configuration, for efficient information retrieval, we configured an Elasticsearch store, an open-source search engine designed to maximize search performance in large datasets using an inverted index structure [32]. In this study, it was integrated with a Korean medical database, allowing for fast responses to user queries. This setup enabled quick searches across large-scale medical data and maximized the performance of the RAG technique. Elasticsearch is particularly effective in text-based query-answering systems, offering fast and reliable search results.

Concerning retriever configuration, we constructed the retriever by integrating the aforementioned Korean embedding model with the Elasticsearch store. When a user inputs a query, the retriever uses the embedding model to generate a vector representation and searches the Elasticsearch store for the most similar vectors. These retrieved documents are provided as input to our backbone model, which then generates an optimal answer based on that data. Through this process, the system provides information based on reliable evidence rather than offering simple conversational answers. This ensures the chatbot can deliver accurate and trustworthy medical information to the user.

Concerning metadata filtering, we enhanced search efficiency using metadata filtering technology. Due to the nature of our model, which must use large volumes of health care data such as textbooks and accumulated big data responses, we determined that metadata filtering technology was essential for speed optimization. Metadata filtering first recognizes specific disease names or symptoms from the user’s query and uses relevant metadata to reconstruct the query more accurately. This improves search accuracy and helps users quickly access the information they want [33]. Table 2 shows examples of keywords that the metadata actually retrieves and the information brought in through them.

Figure 3 depicts the overall workflow, illustrating the conversion of user queries into vector representations via the Korean embedding model for semantic retrieval.

‎

Evaluation for AI Model Performance

To validate this chatbot system, we reconstructed 226 representative questions based on question and answer data provided by a medical consultation platform, focusing on 7 common diseases among Koreans: hypertension, acute bronchitis, diabetes, indigestion or gastritis, atopic dermatitis, allergic rhinitis, and reflux esophagitis. Table 3 summarizes the information about the question list.

The main question types by disease include lifestyle, medication, symptoms, and treatment. For scoring, an emergency medicine specialist provided the necessary answer guidelines for each question. We quantitatively assessed each model’s responses using 3 criteria: accuracy, safety, and helpfulness. We used the following prompt script to guide the assessment process (Textbox 1).

We established accuracy, safety, and helpfulness as the core criteria for evaluating AI responses. Accuracy comprehensively evaluates whether the AI’s answer is medically consistent with the model answer reviewed by a medical expert, and whether it clearly aligns with the user’s question intent. In parallel, safety assesses whether the answer contains dangerous information that could be harmful to the user, and whether it ensures the information’s reliability by including essential warning statements, such as “AI advice cannot replace professional medical consultation.” Helpfulness measures how easily the user can understand the accurate and safe information and receive practical help from it. In other words, an answer is considered good only when it goes beyond informational correctness and also possesses user-centered delivery and practicality.

Ethical Considerations

All data used in this study were sourced exclusively from publicly accessible online platforms, including the official websites of major Korean hospitals, open-source medical textbooks, the Ministry of Food and Drug Safety, and Korean health search engines. These sources provide general disease-level medical knowledge intended for public health education and do not contain any patient-level records, clinical notes, or PII. No internal, proprietary, or private electronic health records were accessed.

As detailed in the Methods section, all collected data underwent a stringent verification process involving both automated screening and manual review by medical professionals to guarantee the total absence of any unintended PII. Accordingly, this study qualifies as research using publicly available information with no collection or recording of personal identifiers, and is exempt from Institutional Review Board review pursuant to Article 13, Paragraph 3 of the Enforcement Rule of the Bioethics and Safety Act in the Republic of Korea [34].

Results

Overview

The proposed model scored highest across accuracy, safety, and helpfulness. It significantly outperformed other models in terms of accuracy, safety, and helpfulness. The following table and graph visually represent the quantitative evaluation results for each model. To compare the performance differences between configurations (baseline vs RAG vs RAG+metadata filtering), we performed a paired t test for each model’s scores across the 226 validation questions.

Table 4 shows the results of analyzing the impact of RAG and metadata filtering technologies on the medical consultation response performance of various language models. The general trend indicates that applying RAG technology alone results in minimal performance improvement or even a slight decline. In contrast, when RAG and metadata filtering are applied together, most models show a significant performance enhancement.

The minimal effect of applying RAG alone appears to stem from the difficulty of accurately finding relevant and useful information from vast external data sources. Indeed, the change in average scores for the Qwen3:4B (4.25, SD 0.04 → 4.28, SD 0.04, P=.40) and Mistral:7B (3.79, SD 0.08 → 3.80, SD 0.11, P=.73) models was not statistically significant. Furthermore, the Llama3.1:8B and Gemma3:27B models actually experienced a decline in scores.

In contrast, when metadata filtering was combined, a clear improvement was observed, particularly in the midsized parameter models. The average score for the Mistral:7B model increased significantly from 3.79, SD 0.08, to 4.10, SD 0.10, and the Qwen3:4B model’s score rose from 4.25, SD 0.04, to 4.37, SD 0.05; both increases were statistically significant (P<.05). Gpt-Oss:20B also demonstrated a significant (P<.05) performance enhancement from 4.43, SD 0.05 to 4.51, SD 0.04, notably achieving the highest score of 4.61, SD 0.03 on the safety metric. This indicates that the effectiveness of RAG is highly dependent on the filtering process that controls the quality of the retrieved information.

Figure 4 compares the effects of RAG and metadata filtering against the original model. Each bar shows the average difference in accuracy, safety, and helpfulness compared to the baseline when RAG and metadata filtering are applied, with positive values indicating performance improvement. Overall, when the metadata filtering technique was included, the model showed improved performance in accuracy and helpfulness, which proves that it provides useful answers suited to the local Korean environment.

Statistical analyses were performed to evaluate the significance of performance changes. All continuous variables (accuracy, safety, and helpfulness) are reported as mean and SE of the mean. Differences between the baseline models and their RAG-augmented variants were assessed using a 2-tailed paired t test, assuming the scores for the same 226 questions are dependent. Statistical significance was defined as P<.05.

‎

Web Service Implementation

For this study, our development environment used Ubuntu 22.04 long-term support and an NVIDIA A100 40 GB graphics processing unit. We used the graphics processing unit for the computational operations of the LLMs. We applied Streamlit (Streamlit Inc) and LangChain (LangChain Inc) as development frameworks. Specifically, we used Streamlit to implement the web interface and LangChain to integrate and manage the LLMs.

The service developed through this process is named “Geongangi.” We aim for this service to provide information on disease symptoms, prevention, and health management, as well as posttreatment care conversations in the future. The intended target users are the general public and medical professionals. It can provide medical information to the general public and be used by medical professionals for the purpose of collecting patient information. Figure 5 shows the “Geongangi” web service interface, illustrating the user-friendly design that supports both patient and health care provider interactions.

‎

Discussion

Principal Findings

This study aimed to determine under what conditions RAG and metadata filtering improve Korean health consultation responses across open-weight LLMs of varying scales. Our findings validate the hypothesis set in this study by revealing that retrieval augmentation yields meaningful gains only when retrieval quality is actively controlled through metadata filtering. Standalone RAG failed to produce statistically significant improvements in any model—and, in 2 cases (Llama3.1:8B and Gemma3:27B), resulted in score decreases—indicating that unfiltered retrieval over a large-scale Korean medical corpus introduces contextual noise that can degrade response quality. In contrast, RAG combined with metadata filtering produced statistically significant performance gains (P<.05) in 4 of the 5 models evaluated. Notably, the magnitude of improvement was inversely associated with model scale: smaller models such as Mistral:7B showed the greatest absolute gain (3.79, SD 0.08 → 4.10, SD 0.10), while the largest model, Gemma3:27B, showed no significant benefit (P=.14)—suggesting that beyond a certain knowledge threshold, external retrieval provides diminishing returns. Collectively, these results confirm that effective RAG deployment in specialized medical domains requires not merely an augmentation of information, but a deliberate mechanism for aligning retrieved content with the clinical structure of the target health care environment.

Interpretations and Implications

The failure of standalone RAG to improve performance can be attributed to the unique linguistic characteristics of Korean medical inquiries. As demonstrated in Table 2, patient queries are frequently long and narrative, embedding core clinical signals within surrounding emotional expressions, personal histories, and contextual background. This structure creates a semantic mismatch between the patient’s actual clinical intent and the documents selected through vector similarity search alone; for example, a query describing weight loss, low mood, and loss of motivation was retrieved against a document on desquamative interstitial pneumonia rather than depressive disorder when filtering was absent. This finding is consistent with prior work identifying the limitations of dense retrieval in long, multitopic queries, where peripheral content dilutes the relevance signal and degrades retrieval precision.

Metadata filtering addresses this limitation not merely as a computational optimization, but as a mechanism that structurally aligns retrieval with the Korean clinical decision-making framework. South Korea’s health care system is characterized by highly standardized diagnostic and treatment protocols governed by National Health Insurance reimbursement criteria and professional society guidelines, in which institutional standards often take precedence over individual clinical discretion. As Korean medical knowledge is thus organized around well-defined categories in accordance with international standard guidelines—clinical department, disease classification, and drug and procedure type—ie, metadata filtering, which extracts these same clinical cues from the query before retrieval, naturally mirrors this organizational structure. This alignment explains why filtered RAG produced not only higher accuracy scores but also a meaningfully higher safety score in Gpt-Oss:20B (4.61, SD 0.03), whose reinforcement learning from human feedback–based alignment appeared to synergize with retrieved documents containing explicit clinical warnings and contraindication information.

Model-specific performance patterns further reveal that the benefit of metadata filtering is modulated by a model’s preexisting parametric knowledge. Mistral:7B, which showed the largest absolute improvement (3.79, SD 0.08 → 4.10, SD 0.10), appears to be well-suited to instruction-following but lacks the domain-specific knowledge to generate accurate Korean medical responses independently; metadata-filtered retrieval effectively compensated for this gap, representing the highest return on retrieval investment relative to model size. In contrast, Llama3.1:8B showed a score decrease under RAG-only conditions, likely because its limited context usage capacity caused unfiltered, noisy documents to interfere with generation rather than support it. Gemma3:27B, which exhibited no significant improvement even under filtered RAG (P=.14), likely reflects a knowledge ceiling effect—its 27B-scale pretraining corpus appears to have already internalized information equivalent to or exceeding the clinical scope of this study’s document database, rendering external retrieval marginally useful under current conditions. This effect may not generalize to domains requiring highly local, time-sensitive, or rare clinical knowledge not well-represented in general pretraining data.

These findings carry direct implications for practical deployment decisions. For resource-constrained settings that require on-premises deployment, smaller models (≤8B parameters) should be considered viable only in combination with high-quality metadata filtering, as their baseline medical knowledge is insufficient for safe clinical use. For midrange models (≈20B), the combination of filtered RAG and strong safety alignment offers a practical trade-off between computational cost and clinical reliability. For large models (≥27B), the marginal benefit of RAG may be limited unless the target domain involves highly specialized, locally specific, or frequently updated knowledge that is unlikely to be covered in pretraining data.

Limitations

We identified several challenges and boundaries in this study that should be considered when interpreting the findings. First, the system continues to struggle with complex clinical inferences involving the simultaneous consideration of multiple medications or comorbid conditions, as the current metadata schema does not yet capture interaction-level clinical relationships. Second, the construction and maintenance of metadata tag sets requires substantial manual effort from domain experts, which may limit the system’s scalability when extending to rare or highly specialized medical subspecialties underrepresented in the current corpus. Third, real-world user query analysis revealed frequent inquiries regarding diet, exercise, and lifestyle management, indicating that future iterations must expand data coverage beyond structured clinical literature to include validated nutritional and behavioral health information. Finally, this study did not systematically analyze the downstream impact of metadata extraction errors—cases where the LLM incorrectly identified or omitted clinical cues from the query—on final response quality, and the error propagation characteristics of such failures remain an important area for future investigation.

Conclusions

This quantitative evaluation confirms that architectural reinforcement through metadata filtering is essential for the reliable application of AI in the Korean medical domain. By demonstrating a framework that bridges the gap between general-purpose LLMs and the specific requirements of the local health care system, this study provides a blueprint for safer medical consultation tools. As these systems evolve to integrate broader health-related data, they will play a vital role in enhancing clinical efficiency and providing patients with high-quality, verified medical information that adheres to regional standards.