AI-Enhanced 3D Models in Global Virtual Reality Case Conferences for Surgical Care in a Low-Income Country: Exploratory Study

Introduction

Background

A modeling study has demonstrated that approximately 5 billion people worldwide lack adequate access to surgical care [1]. This issue particularly affects regions in the Global South, including Sub-Saharan Africa, South America, and Asia. To target this issue, “Global Surgery” has been initiated and prioritized in the past decade by the World Health Assembly as a concept to improve and equalize surgical care across international health systems [2]. The Lancet Commission of Global Surgery particularly emphasizes the role of innovation and mobile health to address the accessibility of surgical expertise [3]. Rural surgeons in low-income countries, who must provide broad surgical skills, play a critical role in advancing Global Surgery [4]. In crisis regions and war zones, remote medical assistance, such as telemedical consulting, can enhance health service delivery, as currently demonstrated in the Gaza Strip [5].

Limited funding for medical technology is a key burden holding back the evolution of treatment practice as well as medical education in low-income countries [6]. In high-income countries, an increasing variety of high-end technologies in health care systems is available, whereas limited resources in low-income countries could especially necessitate frugal designs [7]. Uganda is an East African country with a population of 45.5 million people [8]. The World Bank classifies Uganda as a low-income country per its income classification [9]. The country has a severe shortage of qualified surgeons, with fewer than one per 100,000 people, exacerbating challenges such as geographic isolation, low health literacy, limited financial resources, and insufficient health care infrastructure [10-13]. In many urgent cases, for example, in frequent road traffic accidents, acid attacks, or fire burns, patients cannot receive the necessary immediate treatment. Individuals surviving the initial trauma later often become affected by preventable contractures and immobility, eventually necessitating plastic and reconstructive surgery [14]. A recent review analyzes the Ugandan health care system and emphasizes the need for investments and enhancements in both new and existing infrastructure as well as in the further education of health care workers [15].

In reconstructive surgery, a 3D understanding of individual pathologies located in surface anatomy is pivotal. This applies to treatment approaches for both traumatically induced injuries, such as burns, accident- or war-related wounds, as well as benign space-occupying lesions or oncological entities.

Modern software for creating 3D models from 2D images, such as photogrammetry, has made significant strides in recent years. This technique analyzes multiple 2D images taken from various angles to generate accurate 3D models. AI-driven 3D reconstruction technologies further improve model creation, enhancing both speed and quality [16,17]. These technologies allow high-quality 3D model creation even from frugal sources like smartphone cameras.

The application of extended reality (XR), which includes virtual reality (VR), augmented reality (AR), and mixed reality (MR), is expanding rapidly in the medical field [18,19]. However, access to these technologies is not equally distributed from a global perspective. In surgery, the technology allows preoperative planning using patient-specific imaging in 3D environments and the training of surgical procedures through immersive simulations [20-22]. Where available, XR has a positive impact on surgical training [23-25]. Computed tomographic (CT) and magnetic resonance imaging (MRI) scans can be reconstructed into a 3D image and displayed with the use of head-mounted displays (HMDs), commonly known as VR headsets. Novel explorations even demonstrate intraoperative holographic overlays of 3D reconstructed images [26]. For the user, visualizing the patient data in a 3D stereoscopic view reduces the cognitive work needed to assess the 3D information out of 2D scan slices and similar imaging technologies [27-29]. The use of 3D-reconstructed medical imaging has been analyzed in several studies, often in the context of addressing challenges in surgical strategies for complex oncological scenarios [26,30-32].

Therefore, in lower-income regions without access to radiological imaging, photogrammetric modeling could be an additional tool for patient data acquisition. Besides novel possibilities of 3D image representation, the assembly of health care professionals in immersive virtual spaces from different locations opens new approaches in future medical conferencing [33].

The employment of XR and AI-supported smartphone apps facilitates surface visualization in a manner that is uncomplicated, expeditious, cost-effective, and radiation-free, thereby offering a distinct advantage over radiological imaging methods. Our study emphasizes frugal and robust technological solutions by using widely available, consumer-grade hardware that is well-suited to low-resource and rural settings, where access to surgical infrastructure and education remains limited. By validating the feasibility, we propose a potential pathway for the broader adoption of surgical training and assistance tools, ultimately contributing to more equitable access to surgical care worldwide.

No original empirical data or scientific work that shows the integration of XR technology with AI-enhanced image reconstruction for the purpose of global collaborative surgical planning in the Global South has been identified. Thus, an explorative study design was carried out to provide firsthand and original data [34]. To the best of our knowledge, we describe a pioneering explorative study that uses collaborative XR for case conferencing and assessment using AI-enhanced 3D surface models of patients from rural Uganda in reconstructive surgery while executed only on affordable consumer-based hardware platforms.

Objective

This study intends to determine the feasibility of using new AI-enhanced image modeling technology within an immersive spatial XR scenario to collaboratively and remotely assess reconstructive patient cases in the resource-limited country of Uganda.

Methods

Study Cohort

Surgeons participating in the surgical camp by INTERPLAST-Germany e.V. at Lamu Medical Centre for Reconstructive and Global Surgery (Lamu) in Jinja, Uganda (October 22 until November 4, 2023) were invited to be involved in this exploratory study. Lamu is a private medical facility that regularly hosts international surgical camps. During this period, patients with a variety of potential reconstructive surgical conditions were included based on the following criteria.

Inclusion criteria:

• Having a condition that requires evaluation for reconstructive surgery (eg, fire or acid burns, amputations, scars, or skin and soft tissue masses or tumors).
• Being over 18 years of age, legally competent, and having provided written informed consent.

Exclusion criteria:

• Having an underlying condition that does not allow for anesthesia or surgery.
• Testing positive for COVID-19 or exhibiting symptoms of a COVID-19 infection.

Patient eligibility was assessed through a comprehensive anamnesis and physical examination relevant to the presenting pathology. When available, medical records were also reviewed.

Workflow

After participants’ enrollment, data acquisition and processing were conducted. Newly created datasets were prepared on-site while VR case conferences were scheduled. Figure 1 illustrates the study’s workflow, which encompasses 2 phases: the 3D model acquisition and the virtual reality case conference. Initially, the images were collected and then processed into a 3D reconstructed model using an artificial intelligence-enhanced smartphone app. Following the successful setup, the virtual reality case conference was conducted, during which each patient’s individual pathology and surgical strategy, including the marking of potential incision lines and flap approaches, were discussed.

‎

Creation of 3D Model

The 3D models of the participants’ conditions were created using the AI-enhanced smartphone app MagiScan (App version 1.8), developed by MagiScan Inc. A Google Pixel 7a smartphone, Google Inc., equipped with the Android operating system (version 13), was used for scanning. Each patient underwent a photographic session. A total of 45 images were taken from various angles and orientations as the photographer executed a 360° circumferential movement around the patient. The patient was instructed to maintain a stable position, either standing or sitting, to ensure a comprehensive visualization of the pathology from all relevant angles. Inadequate lighting conditions are the most common cause of artifacts, such as inaccurate surface discontinuities. It is essential to avoid spotlights and instead use indirect lighting to ensure optimal accuracy. These photos were subsequently uploaded into the app and autonomously processed on a dedicated secure storage in a German-based data center (Frankfurt, Germany), and maintain guidelines of the EU General Data Protection Regulation (GDPR). The patient data was anonymized by assigning a numeric pseudonym.

Raw photo data were transformed into 3D models, the resulting digital representation of the patient was exported in Graphics Library Transmission Format (GLTF) to be further imported into the VR platform.

VR Hardware and Software

Two different types of VR headsets (HMDs) were used to mitigate the risk of insufficient functionality in one of the headsets and to ensure flexibility in setup by including both wired and wireless options. The Meta Quest 2 operated wirelessly as a standalone device, whereas the Oculus Rift S was wired to a workstation. Manual interaction was facilitated through VR controllers wirelessly connected to the HMDs. Both headsets enabled a functional setup. The headsets provide a similar technical performance and user experience.

The used software was XR Dissection Master (Version V0.14.3) by Medicalholodeck, installed as a standalone device.

Virtual Case Conference

The virtual case conference initiated with a visual and audio check-up. In each setting, a Ugandan surgeon from Lamu (n=2) and a remotely integrated surgeon affiliated with INTERPLAST-Germany e.V. (n=2) were equipped with a set of VR glasses and controllers, allowing them to participate in the shared digital environment. All sessions were accompanied by the research assistant and principal investigator, who ensured sufficient setup and data acquisition. The 3D datasets were uploaded onto the XR platform and automatically synchronized among participants. Case discussions commenced with an introduction of the participating surgeons. Thereafter, the patient’s medical history and ongoing diagnosis were presented by the on-site surgeon. Possible procedures and tailored surgical approaches for each case were discussed. Throughout the case discussion, the app allowed users to zoom in and out of the 3D model, as well as rotate and move it as required. A toolbar provided a pen with various colors and stroke widths for highlighting and annotating specific areas of the object. During the interaction, the surgeons were represented by individually colored avatars, including virtual hands. After the mutual exchange, an individual treatment plan was proposed, and a final team revision was conducted for each case.

The case conferences were recorded on a server based at the University Hospital Bonn to later assess the durations of conference time sequences. The surgeons’ feedback was collected through semistructured interviews, including Likert scale items on user experience and content quality.

Ethical Considerations

The study was conducted in accordance with the Declaration of Helsinki and received ethical approval from the institutional review board of the Mildmay Uganda Research Ethics Committee (approved September 20, 2023; Ref. MUREC-2023‐299) and the ethics committee of the University of Bonn (approved August 15, 2023; Ref. 235/23-EP). Administrative clearance was granted by the Ugandan Ministry of Health on October 11, 2023. The study was registered with the Ugandan Council for Science and Technology (UNCST), which issued a research permit on October 23, 2023 (Ref. HS3198ES). Written informed consent was obtained from all participating patients and surgeons after they received comprehensive written and verbal information about the study. The participants were assured their right to withdraw from this study at any time. To safeguard anonymity and prevent any possibility of direct identification, a numerical pseudonym was assigned. Data access was limited exclusively to authorized members of the research team. No compensation was given to the participants.

Results

A total of 15 patients from the surgical camp conducted by INTERPLAST-Germany e.V. were included in this study (n=15). Four categories of working diagnosis were established according to the patient’s surgical condition. Each patient’s medical history and surgical procedure were discussed in a total of 3 intercontinental virtual case conferences. The meeting within an immersive virtual room facilitated real-time communication and interactive engagement with the 3D models of the patients. A recording of the virtual case conference is available as Multimedia Appendix 1.

The cohort consisted of 5 patients who endured burn contractures resulting from fire, hot water, or acid. Seven patients exhibited a variety of surface tumors, including lipoma, ganglion, cysts, or unknown lesions of the soft tissue. Two patients presented with chronic wounds. One patient showed soft tissue damage following trauma. The entities appeared in different locations that have been arranged into 3 distinct categories: Limb (8 patients), Head and Neck (7 patients), Trunk (1 patient). The summarized data of the cohort are presented in Table 1.

The median time for virtual case conferences was 3 minutes and 1 second, with a range from 1 minute and 33 seconds to 8 minutes and 14 seconds. Case introduction and medical history took a median of 50 seconds, while planning the surgical procedure had a median time of 1 minute and 20 seconds. Discussion and team revision took a median of 44 seconds. All time metrics related to data acquisition and 3D model creation are summarized in Table 2. The time metrics for the virtual case conferences are presented in Table 3. In some instances, team revision was not conducted if no additional questions or ambiguity arose following the discussion of surgical approaches and interacting with the 3D-reconstructed model.

Two general surgeons from Uganda and 2 reconstructive surgeons from Germany, affiliated with Interplast, conducted the XR case evaluations. During the virtual intercontinental case conferences, the surgeons actively engaged the app’s tools to outline anatomical landmarks and potential incision lines. Possible flap mobilizations and the placement of skin grafts were collaboratively marked on the patient models within the immersive setting to explore potential surgical strategies. The final surgical techniques included a jumping man flap, a supraclavicular flap, and a latissimus dorsi flap. All approaches were outlined and discussed.

Figure 2 demonstrates a VR image of a patient with postburn contractures. Major anatomical landmarks are highlighted on the 3D model in Figure 2A, vascular structures are marked in Figure 2B, and key vessel axes along with their origins are shown in Figure 2C.

Figure 3A presents a VR image of another patient with postburn contractures. The "jumping man flap," proposed as a therapeutic option, is marked in Figure 3B, and further areas of tissue mobilization are highlighted in Figure 3C.

Figure 4A depicts a patient with a large lipoma, along with the proposed incision line for surgical treatment Figure 4B.

The objective of the surgeons’ evaluation was to assess their detailed perception of the 3D-reconstructed patient models and their integration into a VR-based case conference. Specifically, the evaluation focused on the quality and visibility of the 3D-reconstructed models, communication features, and overall feasibility. Surgeons’ feedback was collected through Likert-scale items. The frequency distribution of the response options was analyzed to compare the responses. Likert-scale results of the surgeon’s feedback demonstrated that all 4 participating surgeons (4/4, 100%) strongly agreed that the 3D reconstructed models were of high quality. Regarding the visualization of large entities, all surgeons (4/4, 100%) found it satisfactory, while 2 surgeons (2/4, 50%) expressed satisfaction with the visibility of smaller entities, such as ganglions and finger contractures. All surgeons (4/4, 100%) agreed that the audio and visual communication tools were sufficient. No surgeons (0/4, 0%) reported any significant motion sickness, with only 1 surgeon reporting to have experienced slight motion sickness. However, at no point did this lead to the termination of the VR session or the need for a pause. Furthermore, all surgeons (4/4, 100%) affirmed that the technology could enhance surgical care in low-resource settings and could be integrated into routine practice for reconstructive surgery case discussions. The detailed results are provided in Multimedia Appendix 2.

‎

‎

‎

Discussion

Principal Results

Our findings demonstrated a novel, intercontinental immersive surgical health care approach. We initiated and explored a solution for remote case conferencing and interactive assessments of surgical strategies on digital patient models, providing insights into the requirements and effectiveness of AI-enhanced modeling and XR setups.

The VR-based case assessment process was time efficient, as reflected in the median metrics, enabling multiple surgeons to conduct remote, simultaneous, and detailed evaluations without being physically colocated. This collaborative setup supports improved resource allocation. Various surgical entities and conditions were assessed, with surgeons outlining individualized strategies for each case. The dynamic structure of the case conferences led to variations in timing across different assessment phases.

The accuracy of 3D reconstruction and representation of smaller lesions, such as small lipomas and ganglions, was limited, whereas larger entities were depicted with great accuracy. The smartphone-based 3D reconstructions were acquired in a reasonable timeframe but occasionally showed minor artifacts. Compared with conventional photogrammetry, AI-enhanced 3D reconstruction methods require significantly less time and computational resources [35], making them particularly valuable for rapid and cost-efficient 3D modeling in clinical settings. Ongoing developments in consumer mobile devices, particularly integrated light detection and ranging (LIDAR) technology, show promise for further applications and exploration in surgical practice, offering highly accurate 3D scanning capabilities that are already being used in the medical field [36].

Immersive Communication and Surgical Collaboration

Intercontinental communication in the immersive environment was effective during the real-time conferences in both visual gestures and speech. After their initial involvement in virtual case conferences, all participating surgeons recognized the high potential of 3D representation in XR to improve surgical care in the Global South. This approach may even serve as a future standard of care in remote planning of reconstructive operations. This underlines the quality of the patient models as well as the effectiveness of the VR communication platform.

Although no directly comparable studies currently exist, related literature underscores the influence of XR in surgery and digital communication. Numerous studies have demonstrated the potential benefits of XR in surgical planning, navigation, intraoperative guidance, and training [19,37]. In addition, the impact of digital communication solutions on case preparation and collaboration further emphasizes the potential of these technologies in surgical practice [38].

Limitations

This study aims to gain initial insights into the value and usability of the technology by implementing it during a temporal surgical camp to establish a standardized protocol. Consequently, the sample selection and composition may limit the representativeness of the patient sample compared to broader populations in public hospitals.

In addition, the small study population of this exploratory proof of principle limits the ability to draw meaningful conclusions about the technology’s impact on specific surgical pathologies or subentities. Furthermore, due to this design, the number of participating surgeons was limited, leaving room for a potential influence of individual bias.

Opportunities of XR for Surgical Care in Low-Income Regions

This study confirms the feasibility of XR-based case assessments for reconstructive surgery in a low-income country using frugal, off-the-shelf hardware. Multidirectional, immersive, and interactive communication between rural surgeons, specialists, and the broader community in the field of reconstructive surgery can reduce barriers to effective case discussions, facilitating global knowledge exchange based on real patient data [39,40]. Our study has shown that these goals are achievable with frugal equipment and limited training, with hardware costs kept below 1000 US$, making the setup accessible and scalable [41]. However, consistent, low-latency internet connectivity is essential for effective real-time VR use. XR has already proven valuable in surgical education, planning, increased precision, reduced operative times, and optimized outcomes while minimizing resource demands [42]. XR presents a significant opportunity to further educate health care professionals in anatomy and surgical skills, especially in resource-limited settings where extensive training on cadavers or live models is often unavailable. Particularly, in plastic and reconstructive surgery, XR training solutions can play a valuable role [43].

Conclusion and Outlook

Despite constrained access to modern medical technologies, this study demonstrates, for the first time, the feasibility of AI-enhanced 3D object reconstruction in conjunction with XR for reconstructive surgical assessment in a low-income country. Implementing such technologies in resource-poor settings across the Global South could significantly improve global surgery and reduce barriers to access to surgical care. It is suggested that medical AI applications need to be implemented equitably and inclusively from a global perspective [44].

While the affordability of VR headsets and AI-enabled mobile scanning continues to improve, financial barriers and infrastructural limitations, particularly internet access and initial investment, remain critical challenges [7]. Nonetheless, facilitating access to these technologies for rural health care providers could significantly expand their impact and adoption.

To scale this approach, larger prospective studies are needed to evaluate its effects on patient outcomes, cost- and time-effectiveness, and treatment workflows. Priorities for future research include exploring the implications of immersive planning on surgical decision-making, patient access, and rural outreach as patients often must travel long distances. As emphasized by the Lancet Commission, international collaboration and global visibility are essential to advance this solution beyond its pilot stage and realize its full potential in global surgery.