With an estimated 604,000 new cases and 342,000 deaths worldwide in 2020, cervical cancer is the fourth most frequently diagnosed cancer and the fourth leading cause of death from cancer among women. More than 85% of these women with cervical cancer live in low- and middle-income countries (LMICs) [1,2].

Nearly all cervical cancers are caused by an infection with human papillomavirus (HPV), which can be significantly reduced by HPV vaccination (primary prevention). However, since less than 30% of LMICs have a national HPV vaccination program [3], secondary prevention (ie, screening and treatment of precancerous lesions) remains the most important instrument in reducing the incidence of cervical cancer. In LMICs, visual inspection with acetic acid (VIA) is often the preferred method of screening [4].

In 2021, the World Health Organization (WHO) updated its guidelines for cervical cancer prevention and recommended HPV testing as the primary screening test because of its objectivity, high sensitivity, cost-effectiveness, and the possibility for self-sampling. VIA is recommended as an option for triage following a positive high-risk HPV (hrHPV) test to identify women’s eligibility for treatment of these precancerous lesions [5]. However, performance and diagnostic accuracy of screening programs based solely on VIA vary widely and often depend on the experience of the providers, with high intraobserver and interobserver variation [6] and low sensitivity and specificity (ranging from 62.5% to 80% and from 80% to 98.8%, respectively) for the detection of cervical intraepithelial neoplasia grade II or more advanced cervical lesions against the reference standard of histology or colposcopy followed by biopsy [7-9]. VIA is therefore associated with the potential risk of overtreatment and sometimes misses high-risk cases for cervical cancer [10]. Additional factors besides the experience and skill level of the provider that may contribute to inconsistent results of VIA are the light source and the quality of the acetic acid solution. The WHO guidelines offer countries the choice to perform presumptive treatment of HPV-positive women with cervical ablation to overcome issues of missed diagnosis in VIA.

To overcome the limitations of intraobserver and interobserver variation in detecting precancerous lesions, AI has increasingly been applied in medical imaging, including cervical cancer screening [11]. Several studies in both high-income countries and LMICs have demonstrated that the use of AI in VIA testing, cytology, or colposcopy improved detection rates of precancerous lesions with good accuracy [12]. The use of AI therefore could support the decision-making of providers who are performing VIA, with the potential of reducing intraobserver and interobserver variability [13]. However, most studies were carried out with existing pictures or in academic hospitals. No study was implemented in remote rural areas.

In this article, we share our analysis of the introduction of a new AI decision support system (AI-DSS) device under field conditions in women undergoing cervical cancer screening in 2 LMICs with the use of VIA after being tested HPV positive, which we carried out on the basis of secondary data after completion of the project. The goal of this study was to identify relevant factors of the implementation process and to understand barriers and facilitators in using an AI-DSS device in low-resource settings.

The Consolidated Framework for Implementation Research (CFIR) provides a structured, comprehensive method for analyzing various enablers and challenges of implementing and scaling up health interventions [14]. Factors are evaluated across 5 standard domains in CFIR (innovation, process, outer setting, inner setting, and individuals), each with multiple constructs and topics (Figure 1, adapted from Damschroder et al [14] and Toolbox [15]).

In our analysis, the CFIR domain “innovation” refers to the AI-DSS device, and the constructs are evidence base, relative advantage, trialability, design, and costs. The CFIR domain “implementation process” describes the activities that have taken place at the various study sites during the PRESCRIP-TEC (Prevention and Screening Innovation Project Towards Elimination of Cervical Cancer) project, with the constructs of preparation, training, implementation process, and monitoring. The CFIR domain “outer setting” is defined as the health system and health facility in which the cervical cancer screening takes place. The constructs are regulations, funding, and local conditions. The CFIR domain “inner setting” is the cervical cancer screening activity within the facility, with the constructs structure, communication, and mission alignment. The CFIR domain “individuals” consists of persons involved in various positions related to the use of AI-DSS, such as recipients, implementers, supervisors, and the technical support team.

For this study, an AI-DSS for interpreting VIA results (called SAKHI Manipal), developed by Manipal Academy of Higher Education (MAHE) in India, was introduced in Uganda and Bangladesh as part of a European- and Indian-funded research project on cervical cancer screening, the PRESCRIP-TEC, implemented from 2021 onward [16]. In this project, academic institutions and nongovernmental organizations (NGOs) from Uganda and Bangladesh collaborated with European and Indian universities and NGOs.

In Bangladesh, the research was conducted by the research institute icddr,b and the local NGO Friendship. HPV self-collected tests were offered at home and analyzed in a Friendship laboratory. For HPV-positive women, VIA with possible treatment was provided in the floating hospitals of Friendship in Kurigram, Gaibandha, Bogura, and Sirajganj districts in the North and Sathkhira district in the South, which are remote riverine nomadic islands (chars) and coastal areas.

In Uganda, the research was implemented by the Uganda Cancer Institute and the local NGO Uganda Rural Development and Training in 1 rural district. HPV tests were collected at home and analyzed in a local laboratory. VIA triage of HPV-positive women was performed in governmental health centers. Study sites differed among each other in terms of hrHPV prevalence, risk factors for HPV infection, and organization of the cervical cancer screening program.

In each country, 8000 women were targeted for cervical cancer screening, but only hrHPV-positive women were invited for VIA triage. In total, 1183 hrHPV-positive women (136 women in Bangladesh and 1047 in Uganda) underwent VIA triage supported by AI-DSS.

Women eligible for cervical cancer screening participated in different surveys, as explained below. In total, 5 midwives in Bangladesh and 10 midwives in Uganda performed VIA triage in combination with AI-DSS. Before starting the work, the midwives were given a 3-day practical and theoretical training session on the study protocol, including a refresher on VIA, HPV testing, and how to use the AI-DSS. During supervision visits, further training was provided twice per year. The work was regularly reviewed with the midwives during the project.

In each country, a physician with expertise in VIA triage provided regular supervision and could be consulted remotely or directly on difficult cases, as deemed by the screening nurse, or when there was discordance between the nurse and the AI-DSS. The number of consultations was not recorded, as the physician was often present in the examination room. Four gynecologists from a Dutch NGO (Female Cancer Foundation) provided field supervision once per year.

For the assessment of the quality of images captured by midwives, 6 gynecologists from Bangladesh, Uganda, India, and the Netherlands participated in a reference panel.

The CFIR framework (Figure 1) lists the constructs in each domain. We extracted data for each domain of this CFIR analysis from studies in the PRESCRIP-TEC project in Bangladesh and Uganda. Multimedia Appendix 1 provides an overview of the sources of information per domain and construct. Relevant information was extracted from routine data of participating patients (clinical data and data generated by the AI-DSS). Additionally, data were recorded from 3 types of surveys, which provided information about perceptions of patients and AI-DSS users (the adapted African Women’s Awareness about Cancer survey [17], the Health Service Trust Framework survey [18], and the preintervention and postintervention surveys developed in the project). Questionnaires were pretested and validated, translated, and back-translated. The WHO standardized Service Availability and Readiness Assessment survey for cervical cancer services [19] provided information about the health facilities. Country-specific model-based cost-effectiveness and budget impact analyses were conducted for the 2 countries, comparing the PRESCRIP-TEC strategy with the existing screening strategy in each setting. Data from initial project implementation informed the relevant model parameters. Costs were expressed in national currency units of each country, as well as 2022 US dollars. The results from the cost-effectiveness analyses were used to estimate the 5-year budget impact for defined target populations in each country [20].

A semistructured questionnaire was developed to evaluate experiences of health providers who worked with the AI-DSS device. Multimedia Appendix 2 provides more details about the quantitative data collection methods and results.

Local enumerators conducted in-depth interviews with 10 health workers (5 in Uganda and 5 in Bangladesh).

A logbook of evaluation meetings conducted with country managers supervising the health providers using the AI-DSS was updated quarterly. From the meeting minutes, we extracted relevant information, especially concerning the process. The project coordinator, the MAHE design team, and country implementation teams performed 2 online evaluation meetings in the beginning and midway through the implementation process with users of the AI-DSS.

Before the start of AI-DSS implementation, the skills of the selected health workers who would perform VIA were assessed [21].

An analysis of the quality of images and the performance of health providers applying VIA was performed using a panel of 6 expert gynecologists who assessed 285 of a total of 1183 preimages and postimages collected during the VIA procedures and provided a consensus reference assessment of VIA-positive (abnormalities of the cervix) or VIA-negative (no lesions identified). In total, 38 pictures out of 285 were rejected because of quality issues. Every third picture taken per country was assessed until September 2023. Pictures taken after that time were not included. First, the panel members analyzed pictures individually, and when no majority opinion was found, a consensus meeting was conducted. The MAHE design team also assessed the pictures to evaluate whether they met the quality criteria as described in the user manual of the AI-DSS (visibility of cervix, lighting, and reflection of speculum).

The authors aggregated incoming data and information during the project. Research assistants transcribed or summarized the interviews, which were later coded by the authors according to constructs in the CFIR framework. The research team performed regular analyses of facilitators and barriers using a learning-by-doing approach during the implementation research and made recommendations for improvement of the process.

In a consortium meeting in November 2023, a qualitative analysis of facilitators and barriers was performed for the introduction of the AI-DSS in practice.

This study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki. Patients and individual health providers provided written or electronic consent for the use of the information they provided in the research. Participants did not receive any financial compensation for participation in the study.

There was a strict separation between the medical data and research data. Patient data collected were not accessible to researchers. Data managers in the health services removed all personal patient data. Researchers received anonymized data with a random number per patient, which consisted of preimages and postimages, a data file with a random number, date of picture, code for device, judgment by health care worker, and judgment by AI-DSS.

There was role-based access control for all AI-DSS and mobile health (mHealth) app devices and computers used for data management, with a dedicated IT officer overseeing this access control, opening accounts for users, etc. The AI-DSS, an Android phone, was modified to ensure no other apps could be installed besides the AI-DSS algorithm by disabling the SIM card, Wi-Fi access, and other connections. The devices were kept in the clinic at all times.

The mHealth app used and servers used TLS/SSL encryption when transmitting data.

Participants were assigned participant ID numbers, which were used in place of identifying personal information in the research data. These IDs were kept in a separate master log where linkage to identifying information (names, addresses, phone numbers, etc) could be obtained for follow-up and clinical management of the patients as needed. The research team did not have access to the master log, except CN and SN, who were also involved in the clinical care of the participants. Furthermore, the cervical images were assigned a random number generated by the AI-DSS.

All members of the study team underwent research ethics training on human subject protection and good clinical practice.

This study received ethical clearance from relevant institutions in the participating countries: Bangladesh, icddr,b approval 21029; Uganda, National Council for Science and Technology HS2222ES. MAHE received ethical clearance for the AI-DSS study from Kasturba Medical College IEC 924/2020.

This trial was registered under ClinicalTrials.gov Identifier NCT05234112.

The costs per device were around US $600, which were the costs of the adapted mobile phone purchased by the PRESCRIP-TEC project, not including the costs of developing and maintaining the algorithm software and UI. No additional costs for the devices were incurred during the project.

Table 1 summarizes the barriers and facilitators in the 5 CFIR domains identified in the study.

This study provides insight into the implementation process of an AI-DSS for cervical cancer screening using VIA in 2 countries. In total, the AI-DSS was used 1183 times, and 285 pictures were analyzed, of which 38 were rejected due to quality issues. The device was easily accepted by health care workers and patients. Using the CFIR framework, with its 5 domains, barriers and facilitators were identified that are relevant to a smooth large-scale introduction of the device in cervical cancer screening (Table 1).

Under field conditions, the sensitivity and specificity of health care workers’ observations were slightly lower than those under laboratory conditions (assessing high-quality pictures on a computer screen). Time pressure and work stress during VIA may have contributed to lower accuracy. The AI-DSS was calibrated to find anomalies for further examination, and therefore, the negative predictive value was high (91.3%) but the positive predictive value was low (19.7%).

The rejection rate of pictures taken in Uganda (36/206, 17.5%) was much higher than in Bangladesh (2/77, 2.6%). This can be attributed to the less optimal preparation of health care workers in Uganda, as well as inadequate supervision in Uganda. After additional training, the rejection rate was reduced.

In recent years, several studies have been published about AI-DSS algorithms and their performance, including systematic reviews [24-28]. Research into the contextual factors, such as hardware used, UI, and human resources applying AI-DSS, however, is rare.

The use of the CFIR framework in this study helped identify relevant constructs across different domains (Multimedia Appendix 1) and provided an overview of barriers and facilitators (Table 1). The CFIR framework has been used in a few other studies for the analysis of AI-DSS in cervical cancer screening. One study in South Africa used the CFIR framework to compare 2 devices in a preclinical study at a research site [29]. Another study in Peru used the CFIR framework to evaluate the introduction of a device for remote and asynchronous expert colposcopist feedback (not AI-DSS) [30]. A third study in Uganda applied the CFIR framework in qualitative research to examine perceptions of stakeholders regarding the use of telemedicine and AI-DSS in cervical cancer screening [31]. According to these publications, the CFIR framework served as a solid and valuable framework for evaluating implementation research into AI-DSS in cervical cancer screening. This study confirms that the CFIR is a valuable tool for drawing lessons learned.

This study is relevant because AI-DSS is likely to be used frequently in future health care [32], and it provides insights into the user experiences of the UI of medical devices [33].

The findings of this research provide important lessons for introducing AI-DSS in cervical cancer screening in LMICs. Although the hardware (modified mobile phone) was equipped with a good camera, a clear screen, and sufficient lighting necessary for an AI application [34], a smaller and lighter type of device may have been preferable, just as fixing the device on a holder attached to the speculum would be helpful to secure its position, distance, and location vis-à-vis the cervix. Alternatively, a device on a tripod could be considered, which can be positioned after the cervix is visualized. Integrating the device into an ablation device could be another option that would allow for operation by a single health care worker. The offline mode of the device proved very useful, as the evaluations in Uganda and Peru highlighted issues with connectivity that prevented access to the algorithm. Additionally, using the mobile phone’s battery for lighting and picture capturing proved very helpful in field conditions without regular electricity.

Elhaddad and Hamam [32] state that UI design must be user-centered, fit into the user’s workflow, and be trusted, addressing attitudinal barriers among health care workers. The design of the SAKHI Manipal UI was indeed user-centered, guiding the health care worker through the workflow of cervical cancer screening. In general, the device’s use was intuitive. However, the zoom function could be made easier to operate when one person takes the picture while performing VIA.

The capacities of health care workers are crucial in operating the AI-DSS. Even if the task-shifting mode of the AI-DSS is used, the health care worker must be capable of performing VIA, including coloring, cleaning the cervix, and positioning the speculum correctly before taking the picture. Clear visibility of the SCJ is a precondition for using the AI-DSS. For future use in field conditions, it would be beneficial if the AI-DSS provided a mode that could indicate the quality of the picture before assessment, for example, by producing an orange traffic light that forces the provider to retake the picture.

The device produced a folder with a random number of the patient and pictures for further evaluation and research, which was very helpful in the research project. Discrepancies in assessment by health care workers and by AI-DSS led to discussions during supervision visits. Continuously comparing the AI-DSS assessment with that of the health care workers builds capacity and trust in the device.

The pictures were also used for communication with a supervisor working remotely. In this case, the limitations of the device with disabled Wi-Fi and SIM were a disadvantage, leading to delayed communication. The future introduction of passwords or facial recognition could facilitate wireless communication between devices, such as a phone and another phone or tablet.

Reassuring the patient about the confidentiality of the pictures and showing the pictures could be part of the standard procedure when using cameras for AI-DSS.

Future integration into regular screening procedures appears to be feasible, as it aligns with existing practice and does not necessitate a separate intervention. However, trust in the AI-DSS remains crucial. Both health workers and patients must be confident that technology can provide a reliable diagnosis. The algorithm used in our project requires further development to increase the accuracy of the AI-DSS and differentiate between precancerous lesions and other anomalies. Governments must develop clear policies and regulations concerning the use of AI in diagnostics, including procedures for task shifting (initial assessment by AI-DSS) and addressing disagreements between AI and health care workers. The costs of the device (now US $600) can be reduced in the future as mass production starts. Another solution is to combine medical apps, such as those for obstetrics and ultrasound, on a single device. Given the privacy issues, it is not recommended to install the software on a regular phone used by health care workers.

Overall, the study employed a process of learning by doing and identified issues and shortcomings that were subsequently addressed. This is an essential attitude in implementation research, which focuses on improving practice [35]. Our monitoring process was therefore successful.

First, in evaluating the introduction of the AI-DSS, we employed various data collection methods, as described in the methodology section. Therefore, this study presents a secondary analysis of data collected for various purposes, including patient satisfaction, user satisfaction, and project monitoring. The CFIR constructs could have been formulated more clearly upfront.

Second, we did not interview all health care workers who used the device and therefore may have missed viewpoints. Minutes of quarterly evaluation meetings and reports of supervision visits were used for monitoring. No transcripts of meetings were made, and relevant points may therefore have been overlooked. Better reporting formats could have been applied.

Third, the introduction took place in a research environment, and task shifting was not applied because of ethical considerations. Research into the real-life use of the device could reveal how disagreements in assessment between a health care worker and AI-DSS can be resolved. We were unable to measure the long-term impact of the AI-DSS on the prevention of cervical cancer [35]. Only after registration of the device as a medical device can the impact of task shifting be measured.

Fourth, due to COVID-19–related restrictions, initial capacity building was conducted online, which may have impacted the quality of training for health care workers. Later, face-to-face training was applied.

Introduction of AI-DSS in cervical cancer screening in low-resource settings is part of the future, certainly for cervical cancer screening using VIA [36]. Further research is needed into the facilitators and barriers when the AI-DSS operates in a routine cervical cancer screening program. Further research into the potential for task shifting once the AI-DSS device is licensed as a medical device is also recommended.

This study demonstrates that the introduction of an AI-DSS necessitates a broad spectrum of interventions and preconditions. We identified a suitable hardware device and developed a user-centered, intuitive UI that follows the workflow of health care workers. We recognize that software adaptations are required to increase the accuracy of the algorithm.

We showed that capacity building is crucial for the successful implementation of AI-DSS. Health care workers must be capable of performing a high-quality VIA and must be able to take excellent pictures that can be fed to the AI-DSS algorithm. We showed that the workload will not increase when AI-DSS is applied routinely.

We conclude that an AI-DSS device has the potential to assist health workers in better decision-making but requires careful introduction. Further research under field conditions is necessary.