Evaluating a Customized Version of ChatGPT for Systematic Review Data Extraction in Health Research: Development and Usability Study

Introduction

The application of artificial intelligence (AI) in health research has the potential to innovate and optimize various research tasks [1]. Systematic reviews are now increasingly conducted as a gold-standard method for evaluating and synthesizing the evidence base [2]. However, this research methodology often takes up significant time, and therefore can be costly, and is subject to human error [3-5]. As large language models (LLMs) and their capabilities rapidly improve, they are now being used to assist with systematic reviews, streamlining a significant portion of health research methodology.

There has been a dramatic rise in the production of systematic reviews, as demonstrated by a 20-fold increase over the past 20-years, equivalent to 80 publications per day [6,7]. Although they have become a valuable resource for informing policy and clinical practice [8], systematic reviews require significant labor and financial costs. A recent analysis concluded that the mean estimated time to complete and publish a systematic review was 67.3 weeks, with an average of five authors per published text [3]. In addition to software and dissemination fees, organizations can expect to pay around USD $140,000 per review [4]. There is further concern for the potential of human error during the screening and extractions stages [5].

In response to this significant burden, there have been strong calls for the use of AI in systematic reviews, making the process more efficient [9]. The oldest applications have focused on the automation of title and abstract screening. Machine learning classifiers, such as support vector machines or complement naive Bayes have been trained to replicate human inclusion/exclusion decisions based on label training data [10,11]. Active learning methods, such as those implemented in softwares such as Covidence [12] or ASReview [13], prioritize the most relevant citations for initial screening. However, these traditional methods still require human involvement to make final decisions on study inclusion, and they do not automate other stages of the review (eg, extraction). Rule-based and natural language processing (NLP)–driven named entity recognition approaches have been used for structured data extraction from full-text articles [14]. Recent developments in deep learning, particularly transformer-based models such as BERT and SciBERT, have improved the performance of such tasks by capturing more nuanced linguistic features [15-17]. The recent advancement in NLP have further contributed toward the development of much-needed extraction tools [18-20]. While these pilot programs demonstrated strong performance, they were largely designed for the extraction of clinical trial data, limiting their application across study designs. Recent focus has shifted toward the popular platform, OpenAI’s ChatGPT [21]. This accessible LLM software is proficient at understanding and processing human language with high speed and accuracy. As the tool can effectively interpret information from complex texts, it may have strong potential for application in the extraction of systematic reviews. In addition, OpenAI recently released the functionality to build customizable versions of ChatGPT. Users can create a Generative Pre-training Transformer (GPT) and tailor it for certain tasks by providing it instructions and assumed knowledge [22]. These applications can be saved and accessed by other users. With the recent GPT functionality, health researchers can now create and share tools specifically designed to carry out systematic review data extraction.

Despite recent advances in AI technology, only three studies to date have reported using LLMs for data extraction. One study briefly reported on the feasibility of ChatGPT for data extraction [23], while others evaluated the performance of ChatGPT-4 [24] and ChatGPT-3.5 [25] ; these studies found moderate performance for extracting complex information but high accuracy for simpler extraction fields [24,25]. While these studies are the first to evaluate the performance of ChatGPT as an extraction tool, further efforts are needed to improve the availability of these programs.

We present a custom GPT program tailored for systematic review extraction, made available as open access. In the current study, we aim to provide a pilot evaluation of the extraction tool, using data from two systematic reviews. Performance of the custom GPT will be compared to that of a human extractor, examining rates of agreement and time taken to extract data.

Methods

GPT Building

OpenAI’s GPT (version 3.5) Builder allows users to create custom versions of ChatGPT that perform specific tasks by combining instructions, knowledge, and capabilities. The GPT builder interface was used to develop a specialized GPT to extract information from academic manuscripts to assist with the extraction phase of two systematic reviews: a methodological review and a systematic review of interventions. The developed GPT is named Systematic Review Extractor Pro (Figure 1).

‎

Prompt Engineering

Detailed prompt engineering guidelines were followed under the Role, Instruction, Steps, End goal, and Narrowing (RISEN) framework [22] to manually configure the GPT’s base context (purpose) and stepped actions to improve its performance in completing the required tasks. Each term refers to specific prompt components that improve the output quality. Role provides an identity to the GPT dictating the manner in which it acts, that is,“As an expert in the conduct of systematic reviews you are to....” Instructions inform the GPT what task it is to perform, while Steps provide a hierarchy of instructions that the GPT is to follow to perform the desired task. End goal informs the GPT on the format of desired output and content. Finally, Narrowing provides constraints to the GPT including key requirements for performing the given task (eg, setting a word limit on the GPT’s response output).

Iterative Development of the Tool

The RISEN prompt used to instruct the GPT was iteratively fine-tuned to improve its performance by working through five iterations of the detailed instructions. Differences between iterations were generally focused on providing greater specificity to the prompt and detailed steps to carry out the extraction. Any vagueness in the Steps would lead to diverse response ranges for extraction variables, and therefore required the greatest number of changes and troubleshooting. Any changes made to the prompt were systematically tested on dummy health research manuscripts to optimize performance. Due to higher inaccuracy in early stages of extracting statistical results in the template, OpenAI’s user guidelines were followed, to iteratively improve the tool [22]. These guidelines suggest ‘splitting’ or ‘chunking’ complex tasks into smaller ones.

Reviews for Extraction

A random sample of 10 studies from each of the two systematic reviews (total of 20 studies) were selected to evaluate the performance of the GPT. Review 1 is a systematic review of ecological momentary assessment (EMA) research methodologies for measuring substance use and associated behaviors. Full details are provided in the review protocol which was preregistered on the International Prospective Register of Systematic Reviews (PROSPERO: CRD42023400418). Of specific importance to this review was the extraction of methodological characteristics relating to ecological momentary assessment techniques. Review 2 is a systematic review of the effectiveness of wellbeing interventions for helping professionals. For comprehensive details, refer to the review protocol preregistered on PROSPERO (CRD42023422224). As a review assessing intervention effectiveness, this study included the extraction of statistical results.

Agreement

Total proportion of agreement was calculated by summing the number of instances where the GPT and human reviewer agreed on extracted data, divided by the total number of applicable fields. Agreement was computed for each study according to subsection, which included study characteristics, participant characteristics, methodological characteristics (only relevant for review 1), and statistical results (only relevant for review 2). See Multimedia Appendices 1 and 2 for the full list of variables in each subsection. A brief qualitative synthesis of the nature of errors was also conducted.

Timing

The time taken to extract data for each study was recorded for both the GPT and human reviewer. To enhance practicality, we also recorded the time taken for a human to review the GPT output.

Ethical Considerations 

This study is a methodological paper which conducts secondary analysis on deidentified and aggregated data and as such, is exempt from requiring ethical approval as per the Australian National Statement on Ethical Conduct in Human Research 2023 [26].

Results

Time Taken

The mean extraction time for the human reviewers was 36 minutes. For GPT, the mean time for extraction was 26.6 seconds, and an additional 13 minutes for human review.

Agreement

For the completed extraction templates for the GPT and human reviewers, see Multimedia Appendix 1 (review 1) and Multimedia Appendix 2 (review 2). As seen in Table 1, overall agreement between the GPT and human reviewer across the 10 studies in review 1 was 91.45%; agreement was the highest for study (95.25%) and participant characteristics (95.03%) and the lowest for methodological characteristics (87.07%). For the 10 studies in review 2, overall agreement was 89.31%, similar to review 1. The highest agreement was observed in study (90.83%) and participant (90.00%) characteristics. Lower agreement was recorded in extraction fields relating to statistical results (77.50%).

Qualitative Summary of Errors

Across both Review 1 and 2, the most common error made by the GPT was reporting an extraction field as “not specified,” despite the relevant information being present in the study manuscript (see Multimedia Appendices 1 and 2). These types of errors accounted for a high proportion of the inaccuracies, particularly where only p-values were sometimes reported rather than the required measures of effect.

While there were instances of errors resembling hallucination (eg, describing a randomized controlled trial as a pre-post study), these were few in number. When the GPT erroneously inserted information, it typically mistakenly drew data from other parts of the paper, rather than fabricating the information entirely, For example, several errors occurred in sample size extraction, such as reporting the baseline sample as the follow-up sample.

Instances Where the GPT Was More Accurate That the Human Reviewer

In review 1, there were 14 instances where the GPT correctly extracted fields, and the human reviewer was incorrect (equating to 3.25% of the overall number of fields). In Review 2, the GPT was correct in four fields where the human reviewer was incorrect with a rate of 1.16%.

Discussion

Principal Findings

This study evaluated the performance of a custom-configured LLM, specifically a GPT, in automating the data extraction phase of two systematic reviews. This application of AI for extraction provides initial insights into addressing challenges in systematic review methodology, such as time consumption, labor intensity, and potential for human error.

Even when accounting for human review, the custom GPT took less than 14 minutes to complete extraction of each study, 22 minutes faster than human extraction. The 26.6 seconds it took for extraction alone was comparable to a study by Gue et al [25], where ChatGPT-3.5 took approximately 17 seconds per study compared to 77 minutes by a reviewer. As such, the findings of prior research alongside the current study emphasize the processing speed of AI and its potential to significantly reduce the time demands and resultant costs associated with systematic reviews. This efficiency could facilitate more timely summaries of evidence-based practices and support researchers in managing large volumes of literature.

The high overall agreement rates observed in this study between the GPT and human extraction for review 1 (91.45% overall) and 2 (89.31%), suggest that LLMs may have potential to support automation of the extraction of data from academic manuscripts. Consistent across review 1 and 2, the GPT was particularly effective at extracting study (95.25% and 90.83%) and participant characteristics (95.05% and 90%, respectively). This is likely due to the homogeneity in participant and study characteristics that are reported in the extracted manuscripts, reflecting reporting standards upheld by academic journals.

However, agreement rates were lower when extracting methodological characteristics (77.88%) and results (77.50%) sections. Early iterations were improved by ‘splitting’ the task by extracting results separately to the rest of the extraction template. Results are challenging for GPT to extract, as studies used varied methods of analysis and some statistical results were only reported in tables. As LLMs favor extracting data from text information, additional prompting was required to ensure that tables were thoroughly and consistently incorporated into the information extraction sequence.

In this study, a small percentage of instances where the GPT outperformed the human extractor were observed. This demonstrates the potential for AI to identify patterns and insights that may be overlooked in human extraction. Human extractors can be prone to fatigue which may have led to these small inconsistencies. The preliminary findings of this study suggest that LLMs may have utility in identifying errors made by human extractors.

Overall, in accordance with previous research, LLMs excel in quickly extracting simple data (eg, participant demographics [25]. The purpose-built GPT evaluated in the current study was less accurate while extracting more complex fields compared to simple fields (agreement ranged from 77.50% to 95.25%) but appears to be relatively accurate in comparison to extraction using the generic ChatGPT model where agreement rates were as low as 41.2% [25]. A study of 87 human reviewers found error rates of 28.3% to 31.2%, corresponding to agreement rates of 68.8% and 71.7% with a gold-standard reviewer [27]. These rates remained relatively consistent across reviewer experience level and, similar to the findings of the current study, were more accurate for the extraction of participants and study characteristics compared to statistical results. Overall, the GPT evaluated in the current study demonstrated comparable error rates to those reported by human reviewers.

In its current format, researchers may consider the use of this tool as a second extractor, when they lack funding, resourcing, or time for a team member to perform the task (Textbox 1). The overall agreement rate of the custom GPT was approximately 90%, and the GPT had utility in identifying some human errors. Caution should be exercised while using the tool to extract information in areas where it showed lower accuracy, in particular results and methodological characteristics.

Strengths and Limitations

Building on previous work exploring the capacity of ChatGPT 3, 3.5 and 4 to perform data extraction [23-25], this study tested the performance of a custom-built GPT across two distinct systematic reviews. To the best of our knowledge, this study highlights the first empirical investigation of a custom OpenAI GPT to perform data extraction in a systematic review.

Our study has several limitations. When determining agreement between extractors, there is always a degree of subjectivity between coding any qualitative responses, which can either overestimate or underestimate agreement rates. The current study observed agreement rates across 20 studies. Testing a larger sample size would improve confidence in the tool. In terms of limitations of the tool itself, the tool’s construction on the GPT Builder may pose replicability challenges of the output due to the constantly evolving nature of the LLM. The benefit of this aspect is that the tool will automatically update as ChatGPT releases improved versions of their model [29]. However, future research should evaluate the test-retest validity of this custom GPT.

Conclusions

This study serves as promising early evidence for the application of a custom-built GPT in conducting systematic review data extraction. The findings of this study are consistent with the growing sentiment suggesting AI can enhance the efficiency of systematic reviews and reduce cost and time while maintaining accuracy [4]. By encouraging open discourse regarding the role of AI in research, researchers can contribute to the development of robust, transparent, and reproducible scientific practice. This is a rapidly advancing field of technology and OpenAI frequently releases new versions of ChatGPT that show improved performance and efficiency [29]. It is likely that with further advancement of AI, LLMs may be relied on as a sole reviewer for data extraction.