Machine Learning Approach to Identifying Wrong-Site Surgeries Using Centers for Medicare and Medicaid Services Dataset: Development and Validation Study

Introduction

Wrong-site surgery (WSS) is recognized as the fifth most severe medical error in the United States [1]. A recent study revealed that one-third of inpatient WSS incidents resulted in high-severity injuries, with 7.4% of these cases leading to death [2]. This preventable mistake not only causes a big impact on patients but also imposes considerable financial burdens on the health care system [3]. According to a review in the United States National Practitioner Data Bank, a total of US $1.3 billion was recorded in payouts for surgical never events for 2 decades (1990-2010) [4]. Despite efforts aimed at reducing surgical errors through preprocedure verification, site marking, and a preoperative timeout, WSS events continue to occur. Specifically, the services most frequently responsible for those cases were orthopedic, including spine and intervertebral disc surgery, arthroscopy, and surgery on muscles or tendons [2,5].

The underreporting of WSS further complicates efforts to address this issue. In the United States, only 2% of sentinel events are reported to The Joint Commission, contributing to the persistence of WSS [1]. Moreover, the World Health Organization’s 2024 Global Patient Safety Report highlights that only 38% of countries have implemented systems to report preventable and catastrophic medical errors, leading to the widespread underestimation of their true frequency [6]. Among the factors to WSS, inconsistent documentation and inappropriate history and examination information contribute significantly to the occurrence of WSS [2,3]. For example, a patient may have a radiology report indicating a right renal mass from an outside hospital, while a referral document refers to a recent ultrasound report of a left renal mass, leading to a left nephrectomy [2]. Due to the inescapable nature of human error, the ability to automatically identify such discrepancies with the emerging technology could substantially reduce the incidence of WSS.

To address issues of inconsistent documentation and inappropriate history and examination information, a machine learning (ML) model that can automatically detect uncommon associations between diagnosis and surgical procedures may offer a viable solution for identifying and preventing WSS. Recent studies have shown promising results using ML algorithms to detect medication errors, using unsupervised association rule learning to identify unreasonable medication-disease combinations [7,8]. We hypothesized that the similar characteristics between procedure-diagnosis and medication-disease associations could be used to train a model capable of detecting surgical errors. Therefore, this study aims to assess the transferability of an adapted ML model and validate its performance in detecting inconsistencies in surgical procedure documentation.

Methods

Data Source

The Centers for Medicare and Medicaid Services Limited Data Set (CMS-LDS) files provide detailed information on health care services given to beneficiaries. This database includes claims data that record the medical services provided, along with related diagnosis and procedure codes [9]. This dataset is crucial for health care research and quality assessments, helping to measure health care outcomes in various settings [10-12]. We included claims with complete records of both medical and surgical procedures.

Model Construction

In this study, we constructed the adapted version of the Association Outlier Pattern (AOP) model using the ML algorithms, with methods described in a previous paper [13]. We used CMS-LDS data from 2017 to 2019 as a training set to develop the model. This AOP model is designed to automatically detect uncommon or rare associations between specific diseases and surgical procedures, with a particular focus on discrepancies between the diagnosed side (left or right) and the side operated on. The model determined a procedure to be substantiated if the index procedure could be explained by a relevant diagnosis. However, if there was a procedure that could not be explained by any of the diagnoses, then the procedure-diagnosis combination would be viewed as unsubstantiated. All data collection and analysis procedures in this study fully comply with the principles outlined in the “Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research” [14].

Test Set Development

The study used inpatient CMS-LDS data from 2020 as a test set. We limited our testing data to orthopedic procedures, identified by the Attending Physician Specialty Code for Orthopedic Surgery. The study population was selected only if the ICD-10-PCS (International Classification of Diseases, Tenth Revision, Procedure Coding System) and ICD-10-CM (International Classification of Diseases, Tenth Revision, Clinical Modification) codes included identifiable body parts and laterality information. The fourth character of the ICD-10-PCS code was used to differentiate body parts and laterality. Cases in which the only procedure performed was the insertion of an infusion device were excluded, as the rationale for the insertion site is typically clinician-determined and often lacks a documented record.

Procedures and diagnoses were categorized by laterality: right, left, or both. “Right-side procedures” indicated that all procedures performed on the patient were on the right side, and “both-side procedures” indicated that the patient received both right- and left-side procedures. The same definitions applied to the laterality of diagnoses. Patients with complete laterality information for both procedures and diagnoses were classified into 9 subgroups based on the combination of procedural and diagnostic laterality, such as right-right, right-left, right-both, left-right, left-left, left-both, both-right, both-left, and both-both. The analysis focused on 2 subgroups with significant discrepancies between procedure and diagnosis laterality: right-side surgery with left-side diagnosis (right-left) and left-side surgery with right-side diagnosis (left-right).

Test Set Evaluation

The accuracy of these 2 subgroups was evaluated through a rigorous review process that examined the alignment between documented procedures and diagnoses. Two authors (YHC and CHL) independently reviewed the consistency between procedures and diagnoses. In cases of disagreement, both authors conducted a thorough secondary review of all available medical records and reached a consensus through discussion. A confirmed error case was defined as a scenario in which no direct association existed between the surgery performed and any recorded diagnosis. Clinical criteria for defining an error case included: (1) all diagnoses did not sufficiently justify the need for the surgery; (2) the anatomical areas mentioned in the diagnoses and procedures were either not the same, not in proximity (eg, shoulder joint and humerus are in proximity), or not appropriately related (eg, upper leg relates to femur bone); and (3) the anatomical areas were similar, but the laterality was opposite. These criteria ensured that discrepancies in laterality between diagnoses and surgical procedures were systematically identified.

Confirmed error cases were further categorized based on whether the procedure and diagnosis involved the same or different body parts. Confirmed nonerror cases were classified into five categories based on potential rationale: (1) same body part between procedure and diagnosis, but with the unspecified side in the diagnosis; (2) same body part between procedure and diagnosis, but with no laterality specified; (3) diagnosis without a specific body part that can explain the index procedure; (4) anatomical areas in the diagnoses and procedures were in proximity; and (5) the index procedure could be explained by a combination of diagnoses and concurrent procedures.

Model Performance Metrics

To assess the generalization and extrapolation performance of the model, we applied the trained AOP model to identify potentially unusual combinations of procedures and diagnoses among the 2 subgroups with significant discrepancies (right-left and left-right) in the test set. We plotted the precision-recall curves to obtain the optimal probability threshold to maximize the predictive capability of the AOP model. Precision-recall curves have been considered an effective metric for accessing the model, especially the data is an unbalanced dataset [15,16]. The optimal probability threshold is where a point can achieve high precision score while only sacrificing minimal recall (Figure S1 in Multimedia Appendix 1). By calculating the accurate prediction rate of the AOP model and the rule-based method (right-left and left-right), we can compare the performance between these 2 methods.

Ethical Considerations

This study used the CMS-LDS files, which provide beneficiary-level information for research aimed at improving the US health care system. Access to the data was granted through the standard Centers for Medicare and Medicaid Services (CMS) approval process, including entering into a Data Use Agreement with CMS. The study was exempt from human subject research ethics review, as the data used were deidentified in accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule. Since the study involved secondary analyses of deidentified data, informed consent was not required, and no direct interaction with human participants occurred.

To ensure privacy and confidentiality, all analyses strictly adhered to CMS policies, including the cell suppression rule, which prohibits the publication of results with cell sizes of 10 or fewer (eg, patients or services). Furthermore, no attempts were made to reidentify individuals from the data, maintaining compliance with ethical and privacy standards.

Results

Baseline Study Population

Data from a total of 346,382 claims were included in the study. Among them, the data indicated that 1106 patients had a left-side procedure with right-side diagnoses, and 1064 patients had a right-side procedure with left-side diagnoses (Figure 1).

‎

The study used inpatient CMS-LDS data from 2020 as the test set. Testing data was restricted to orthopedic procedures, and identified using the Attending Physician Specialty Code for Orthopedic Surgery. The study population included only cases where the ICD-10 (International Statistical Classification of Diseases, Tenth Revision) procedure and diagnosis codes provided clear information about the body part and laterality.

Test Set Population Characteristics

Among the 1106 patients with left-side procedures with right-side diagnoses, 603/1106 (54.5%) cases were confirmed as errors, while 503/1106 (45.5%) were deemed reasonable combinations after applying the clinical review criteria. Similarly, among the 1064 patients with right-side procedures with left-side diagnoses, 541/1064 (50.8%) cases were confirmed as errors, while 523/1064 (49.2%) were deemed reasonable combinations (Table 1). Most confirmed errors involved the same body part in both procedure and diagnosis but with opposite laterality. Most nonerror cases were considered reasonable due to the presence of a relatively general diagnosis code with an unspecified body part, or a diagnosis with a specified body part but no laterality.

This table summarizes the performance of the rule-based method and the AOP machine learning model in identifying surgical procedure-diagnosis inconsistencies in the CMS-LDS from 2020. The study focused on patients undergoing orthopedic procedures with discrepancies in laterality between procedures and diagnoses. Results are presented for 2 subgroups: left-side procedures with right-side diagnoses (n=1106) and right-side procedures with left-side diagnoses (n=1064). Clinical review outcomes are categorized into confirmed errors and nonerrors, with further classification based on the nature of the diagnosis and its alignment with the index procedure.

We focused on 2 subgroups with significant discrepancies regarding laterality between procedure and diagnosis, where combinations could be reasonable or unreasonable. For instance, a patient who underwent right knee joint arthroplasty but was diagnosed only with left knee osteoarthritis was deemed to have an error, as no diagnoses could justify the need for right knee joint replacement. Conversely, a nonerror case might involve a patient who underwent right lower leg detachment surgery with a diagnosis of diabetes-related gangrene (Table 2). Although the diagnosis did not specify the body part or laterality, it could explain the reason for the surgery and did not conflict with other diagnoses.

This table provides representative examples of confirmed errors and nonerrors from the test set of orthopedic procedures in the CMS-LDS data from 2020. Confirmed errors include cases where procedural laterality (eg, left or right) did not align with the diagnosis, while nonerrors involve reasonable justifications for the procedure based on provided diagnoses. Each example lists the index procedures, associated diagnoses, and rationale for classification based on clinical review.

AOP Model Performance Evaluation

Among the 1106 patients with left-side procedures with right-side diagnoses, the AOP model identified 697 potentially unusual combinations. After applying the clinical review criteria, 576 out of 697 (82.6%) cases were confirmed as errors, while 121 out of 697 (17.4%) were deemed reasonable. Similarly, among the 1064 patients with right-side procedures with left-side diagnoses, the AOP model identified 655 potentially unusual combinations. Of these, 524 out of 655 (80%) cases were confirmed as errors, and 131 out of 655 (20%) were reasonable combinations (Table 1). Most confirmed errors involved the same body part in both procedure and diagnosis but with opposite laterality. Most nonerror cases were considered reasonable due to a general diagnosis with an unspecified body part or a diagnosis with a specified body part but no laterality.

An example of a confirmed error among AOP model-identified cases was a patient who received right shoulder joint arthroplasty, while the diagnoses only included a left humerus fracture. Since a left arm fracture could not justify right shoulder joint replacement, this case was deemed an error. Conversely, an example of a nonerror case was a patient who underwent left glenoid cavity replacement, with a diagnosis of unspecified side rotator cuff tear or rupture (Table 3). As an unspecified side shoulder injury could justify left shoulder surgery, this case was deemed a nonerror.

This table illustrates specific examples of confirmed errors and nonerrors identified by the AOP machine learning model in the CMS-LDS 2020 test set. Confirmed errors are characterized by laterality discrepancies between the procedure and diagnosis, while nonerrors reflect plausible procedure-diagnosis relationships despite general or unspecified diagnoses. Each case includes the index procedures, associated diagnoses, and the rationale for classification.

Discussion

Principal Findings

In this study, we evaluated the effectiveness of the AOP model in detecting whether surgical procedures are substantiated with diagnoses records using CMS-LDS data. We found the AOP model is more accurate in identifying inconsistencies in surgical procedure documentation when compared with the traditional rule-based method. Meanwhile, the adapted AOP model also showed the transferability from detecting the medication-disease association and expanding to verifying the procedure-diagnosis association.

Our study adds to the existing literature that ML has penetrated the medical field with great success. ML allows for more detail to be mined from the data, allowing for the development of better diagnostic and prognostic tools than traditional approaches [17,18]. Similarly, the AOP model can detect associations between specific diseases and surgical procedures by learning patterns from a large database, while the rule-based method depends on the laterality of procedures and diagnoses. For example, among the 2 subgroups with significant discrepancies regarding laterality between procedure and diagnosis (left-right and right-left), a rule-based method classified all 2170 cases as inappropriate combinations due to the conflict in laterality. In contrast, the AOP model classified only 1352 of these cases as inappropriate. In terms of identifying inappropriate procedure-diagnosis combinations, the AOP model identified most errors similarly to the rule-based method (1100 for AOP vs 1144 for rule-based). In addition, the AOP performed with much higher accuracy (1100/1352, 81.4%) compared with the rule-based method (1144/2170, 52.7%).

The AOP model outperformed the traditional rule-based model, demonstrating a lower rate of false signals. Comparing the confirmed nonerror percentage between the rule-based method and the AOP model, there was a significantly lower rate of confirmed nonerror cases identified by the AOP model. In these cases, most confirmed nonerrors identified by the rule-based method could be explained by other diagnoses without mentioning body part or laterality. For example, cancer metastasis to bone may explain the reason for pelvic surgery, even without a specific pelvic metastasis diagnosis. This is because some diagnoses may not provide detailed information about the patient’s condition. Similarly, a patient who underwent lower leg detachment surgery with a diagnosis of diabetes-related gangrene could have the surgery justified, even if the gangrene diagnosis did not specify the body part. The rule-based method focuses solely on the laterality of the diagnosis, whereas the AOP model uses ML methods to explain the association between procedures and diagnoses. Therefore, when the association is deemed normal, the AOP model does not label them as inappropriate.

This study also showed several noteworthy findings. First, cases with fewer diagnoses were more likely to be labeled as inappropriate procedure-diagnosis combinations. For example, a patient who underwent surgeries for the left upper arm and left shoulder had only 4 diagnoses, including a right elbow problem and other chronic diseases. This may indicate incomplete diagnosis coding or an error in the surgical site. Second, the AOP model effectively identified unusual procedure-diagnosis combinations within subgroups where traditional rule-based methods failed to do so. Specifically, procedures that aligned with their corresponding diagnoses were deemed appropriate by conventional methods, such as left-side procedures with left-side diagnoses (left-left) and right-side procedures with right-side diagnoses (right-right). However, the AOP model continued to uncover atypical combinations within these groups. For instance, the model identified a patient who underwent left hip arthroplasty despite having a diagnosis of left knee osteoarthritis only.

The strengths of our study include its being the first to explore whether a ML model could play a surveillance role in detecting wrong site surgery. The effectiveness and fast pace to review the inconsistencies of surgical documentation help increase reporting mistakes in a timely manner without putting too much workforce on this issue. The findings from our study raise important questions regarding whether these errors stem from documentation mistakes or represent actual incidents involving patients. Therefore, in addition to retrospective reviews, it is essential to leverage modern technology to facilitate real-time error prevention. The potential of our model could add value on the current technology to encompass proactive prediction and analysis of diagnoses and clinical evidence to inform accurate surgical decisions and improve documentation practices [19]. In addition, the potential adaptability of the AOP model extends beyond orthopedic surgery to other specialties where procedural and diagnostic codes require logical alignment. For example, in ophthalmic surgery, the model could identify mismatches such as a diagnosis of left-sided cataract paired with a procedure for right-eye cataract surgery. Similarly, otolaryngology and other surgical fields with specific laterality could benefit. However, certain specialties may pose unique challenges due to differences in coding systems or the absence of laterality indicators, requiring further validation and specialty-specific refinements.

The AOP model has limitations. There were still nonerror cases identified by the AOP model. For instance, a patient who underwent left glenoid cavity replacement surgery with a diagnosis of unspecified rotator cuff tear was marked as inappropriate by the AOP model. However, the diagnosis could explain the necessity of the surgery. The reason the AOP model labeled this as an inappropriate combination may be due to the low frequency of this procedure-diagnosis combination in the CMS dataset. If the training dataset has a limited number of surgeries or diagnoses, then the association may be viewed as low and therefore marked as an inappropriate combination.

Another limitation is that the AOP model failed to identify some error cases labeled by the rule-based method. For example, a patient received an internal fixation surgery of the right upper femur with diagnoses of secondary malignant neoplasm of bone and left femur osteonecrosis. While the diagnosis of secondary malignant neoplasm could justify the surgical necessity, the diagnosis of left femur osteonecrosis conflicted with the right femur surgery. In such cases, the AOP model uses the maximum association within all procedure-diagnosis combinations, meaning that if the association between the surgery and a diagnosis is high, the model will view the combination as appropriate, even if the other diagnosis is in conflict. Therefore, in cases where one diagnosis can justify surgery, but another is conflicting, the AOP model may not accurately identify the combination. However, this aspect of the AOP model presents an opportunity for optimization, and future adjustments incorporating more detailed information through advanced ML techniques could yield a more accurate model, especially in cases where diagnoses are vague, incomplete, or lack clear specifications of body parts or sides.

Building on these strengths and limitations, future steps for implementing the AOP model in electronic health record systems could proceed along 2 directions. First, the model could support retrospective reviews by efficiently identifying WSS events, reducing review times and workforce burden, and enabling more timely reporting. Second, it could facilitate real-time consistency checks when clinicians input procedure codes. For example, if a surgeon enters a right-sided knee arthroplasty for a patient diagnosed with left knee osteoarthritis, the AOP model could flag the discrepancy and issue an alert. Such real-time notifications may prevent WSS before it occurs. However, to achieve successful implementation, the model must be optimized to address the limitations, and its performance must be validated across diverse electronic health record systems and datasets across different facilities.

Conclusion

Overall, the AOP model enhances patient safety by systematically analyzing the cooccurrence frequency of diseases and surgical procedures on opposite sides, thereby identifying potentially unusual procedure-diagnosis combinations. Our analysis demonstrated a significant improvement in accuracy rates when the AOP model was used, compared with scenarios where it was not applied. By improving precision in identifying and validating cases with discrepancies between the diagnosed and operated sides, the AOP model has the potential to reduce the risk of errors in orthopedic treatment planning. This enhanced accuracy enables health care providers to make more informed decisions, reducing the risk of WSS and ultimately contributing to improved patient outcomes in orthopedic health care settings.