Implementation Status and Usability of Digital Health Interventions Among Health Care Workers and End Users at the Primary Health Care Level in Chandigarh, North India: Cross-Sectional Study

Introduction

Primary health care is essential health care made universally accessible to individuals and families in the community by means acceptable to them, through their full participation, and at a cost that the community and country can afford, as per the World Health Organization (WHO) [1]. One of the 4 principles of primary health care is the use of appropriate technology, defined by WHO as “methods, procedures, technologies, and equipment which are scientifically validated, adapted to local needs, acceptable to users as well as recipients, and maintainable with local resources” [1]. The term digital health was coined as a “broad comprehensive term including eHealth, that is, the application of information and communication technology in health and health-related field, mobile health (the use of mobile and wireless technologies for medical care), as well as emerging areas such as the application of advanced computing sciences in “big data and artificial intelligence” [2]. Digital health interventions (DHIs) are services delivered electronically through formal or informal care, such as electronic medical health records, health management information systems (HMISs), mobile apps, and telemedicine. [3] In low- and middle-income countries, including India, the imbalance between demand and supply of health care services can be addressed through DHIs, as they can act as catalysts to provide affordable, accessible, and equitable health care services. Recognizing this, the Government of India embarked on the journey of digital health, as evident in the National Health Policy (NHP), National eHealth Authority (NeHA), and the National Digital Health Blueprint (NDHB). A flagship program, the National Digital Health Mission (NDHM), was launched on August 15, 2020, with a vision of establishing a digital ecosystem to achieve universal health coverage [4,5]. Over the past decade, the Ministry of Health and Family Welfare (MoHFW) has taken various digital initiatives for health information management, tracking of beneficiaries, establishing online inventory, and supply chain management. These also include platforms or mobile apps made to seek patient feedback, improve health literacy, and enhance the use of health services [6,7]. Recently, due to COVID-19, there was a major boost in the adoption of telemedicine services such as eSanjeevani OPD [8]. The Aarogya Setu mobile app, launched during the pandemic, also achieved the remarkable milestone of reaching 50 million downloads in just 13 days, making it the fastest-growing app globally [9]. As reported in various studies, the use of DHIs in primary health care has shown promising results in areas such as mental health issues, tobacco cessation programs, maintaining the vaccine supply chain, and hypertension management [10-12]. However, the implementation status of these DHIs at the primary health care level remains uncertain. A systematic review of previous studies reported barriers such as infrastructure and technical limitations, psychological and personal factors, and concerns regarding increasing working hours, which challenge the successful adoption of DHIs [13,14]. DHIs, as a tool to achieve universal health coverage, can be leveraged only if there is greater demand and use. This depends on factors such as the level of awareness among the community [15], clients’ or health care providers’ interaction with DHIs, and their belief that using technology will positively affect health [16]. However, there is little evidence on end-user perception (eg, satisfaction and motivation) of DHIs at the primary health care level, or awareness among the population in Chandigarh. The majority of available studies have evaluated either an independent intervention for a single condition or were part of nongovernmental organization or isolated projects. There is a paucity of evidence in the existing literature regarding the status of government-launched DHIs’ implementation at the primary care level and the usability of these interventions among the community and health care workers (HCWs) in Chandigarh, India. Hence, this study was planned with the objectives to assess the implementation status of existing government-launched DHIs in public primary health centers and to ascertain the usability, in terms of user response, of DHIs by target end users (general population and clients) and HCWs while delivering health service in primary health centers in Chandigarh. The results of this study might guide program managers regarding the status and scope for improvement of the digitization of health interventions at the primary care level.

Methods

Operational Definitions

Operational definitions as per the WHO guidelines for monitoring and evaluation of DHIs are given below [16]:

1. Digital health: “The use of digital, mobile, and wireless technologies to support the achievement of health objectives. Digital health describes the general use of information and communication technologies for health and is inclusive of both mHealth and eHealth.”
2. Usability: “The degree to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” In this study, only part of usability was evaluated in terms of user response.
3. Users or target end users: “The individuals who directly employ the technology using their mobile phones or wireless devices either to deliver health services or to receive services.”

Study Design and Settings

A cross-sectional study was conducted in Chandigarh, a union territory with an estimated population of 1,364,000 (2024). Approximately 97% of the population resides in urban areas. Chandigarh has 2 tertiary health care centers, 1 district hospital, 1 subdivisional hospital, 2 urban community health centers, and 34 health and wellness centers (HWCs; including 5 Ayurveda, Yoga and Naturopathy, Unani, Siddha, and Homeopathy centers) [17]. To assess the implementation status of existing government-launched DHIs, public primary health centers were selected. To ascertain the usability of DHIs in terms of user response by target end users, face-to-face interviews were conducted, and questions were administered in person to members of the general population visiting the study HWCs, as well as to the HCWs delivering health services in primary health centers in Chandigarh.

Study Health Facilities

To assess the implementation of DHIs, study health facilities were HWCs. Under the Ayushman Bharat Scheme, existing civil dispensaries and subcenters were upgraded to HWCs (recently renamed as Ayushman Arogya Mandir) [18]. These were established to provide universally accessible, free-of-cost, comprehensive primary health care. The primary focus is on promoting well-being and offering an extended range of services within the community. The comprehensive range of services now includes care in pregnancy and childbirth, neonatal and infant health care, adolescent health care, family planning, reproductive health care, management of communicable and noncommunicable diseases through National Health Programs, general outpatient care for acute simple illnesses, basic oral health care, care for common ophthalmic and ENT problems, elderly and palliative health care, emergency medical services, and screening with basic management of mental health ailments [17,18].

Study DHIs

Through a preliminary exercise, we enlisted health apps or portals that were implemented at HCWs and used by HCWs. For the general population, we enlisted health apps and portals that were launched for them and were functional. DHIs (apps or portals) that were included in the study were (1) Comprehensive Primary Health Care–Noncommunicable Disease (CPHC-NCD) portal; (2) Reproductive, and Child Health (RCH) portal; (3) Auxiliary Nurse Midwife OnLine (ANMOL) app; (4) Electronic Vaccine Intelligence Network (eVIN); (5) Family Planning–Logistics Management Information System (FP-LMIS); (6) eSanjeevani OPD; (7) Integrated Disease Surveillance Program–Integrated Health Information Program (IDSP-IHIP) portal; (8) Ayushman Bharat–Health and Wellness portal (AB-HWC portal); (9) HMIS portal; (10) Ni-kshay portal or app, (11) COVID-19 Vaccine Intelligence Network (CoWIN); (12) Arogya Setu app; (13) National Health Portal; and (14) My Health Records. The list of DHIs to assess the implementation and HCWs’ and clients’ responses is given in Table S1 in Multimedia Appendix 1.

Study Population

To assess the response of target end users, clients aged 18 years or older, irrespective of gender, who visited the study HWCs either seeking care for illness or as attendants, were contacted. HCWs, including data entry operators (DEOs), medical officers, auxiliary nurse midwives (ANMs), multipurpose workers (MPWs), and pharmacists, who were directly involved in the use and maintenance of DHIs, were enrolled in the study.

Sample Size

The sample size for assessing usability among both target end users and HCWs was calculated using the formula: n=4pq/e², where p=prevalence of usability of DHIs, which was assumed to be 50% (since there was a paucity of sufficient evidence in the existing literature on the prevalence of usability given the heterogeneity of selected DHIs, it was assumed to be 50% to estimate the sample size), q=(1p), and e=margin of error, assumed to be 10%. Considering a nonresponse rate of 20%, the final sample size was calculated as 120 for each group (target-end users and HCWs).

Sampling Technique

The criteria for the selection of HWCs were the mean number of patients seen per day. A list of HWCs with a mean number of patients seen per day was made, and 2 HWCs with high, 1 with moderate, and 1 with low patient footfall were randomly selected (a total of 4 HWCs). For target end-user enrollment, the catchment area of 1 HWC with moderate footfall was selected. Beneficiaries or persons who were visiting the facility and who consented were interviewed in person consecutively until the sample size was achieved. Similarly, HCWs in HWCs who were available and consented were interviewed in person consecutively. For this, we visited randomly selected 11 HWCs of Chandigarh.

Data Collection Tools and Methods

The data collection tools included a pretested observation checklist and data extraction form (Tool 1) to assess the implementation status of the DHIs, and pretested interview schedules (paper-based) for target end users (Tool 2) and HCWs (Tool 3) to assess the usability of DHIs. Tool 1 dealt with information on the functionality of DHIs, including technical factors such as connectivity, power, maintenance, data entry frequency, use, and management of currently operational DHIs. Portal-specific real-time reporting status, number of stock-out events, and number of clients enrolled under DHIs in the health facilities from April to June 2022 were collected. Data quality was checked by comparing the information recorded in the register (hard copies) with the corresponding data entries on the portal (DHI). The interview schedule for clients (Tool 2) was pretested in a health center in Haryana. During pretesting, it was found that there was low awareness of DHIs in the population except for the Aarogya Setu app, so questions were modified to first assess the awareness regarding DHIs. Tool 2 dealt with collecting information on the sociodemographic status of clients, language literacy, source of seeking health information, awareness of DHIs (ever heard of or aware of the mentioned DHIs), and whether they have visited portals or downloaded any mobile app from the listed DHIs. Data were collected using paper-based forms and entered into Epicollect 5 [19]. Tool 3 was used to collect information about the background characteristics of HCWs, the digital ecosystem in the working area, language literacy, training on DHIs, and reported responses on the usability of DHI with options such as “easy to use,” “motivated/intending to use DHIs,” “satisfaction with DHIs,” and “several minutes/hours spent daily on the portal.” Responses from Tool 3 were entered in an internet-based format (Google Forms).

Data Analysis

Data were entered in Microsoft Excel, and analysis was done using SPSS (trial version 25.0, IBM Corp) [20]. Descriptive statistics such as numbers and frequencies were calculated for awareness, apps downloaded, language proficiency, and user response in terms of ease of use, user satisfaction, and motivation. The chi-square test was applied to assess the association between sociodemographic factors and awareness of DHIs among clients. A P<.05 was considered significant. A logical framework model having input, process, and output indicators was used to assess the implementation status of the DHIs. Care was taken while deciding on these indicators from the review of literature, respective DHI guidelines, and public health expert opinions. A score-based system was developed to categorize the implementation status of DHIs. The scoring criteria for each indicator using the logic model are given in Table S2 in Multimedia Appendix 1. After scoring each indicator, cumulative scores of DHIs were calculated in all 3 categories—that is, input, process, and output indicators—and measured in percentages. Equal weight was given to each category. The status of DHI implementation, as per percentage cumulative scores obtained (ie, cumulative scores obtained divided by maximum cumulative scores×100), was categorized as follows: “not implemented=0%‐24%,” “poorly implemented=25%‐50%,” “fairly implemented=51%‐70%,” “effectively implemented=71%‐90%,” and “very effectively implemented=100%‐91%.”

Ethical Considerations

For this observational cross-sectional study, ethical clearance was obtained from the Institute’s Ethics Committee (IEC), Postgraduate Institute of Medical Education and Research (PGIMER), on March 17, 2022 (reference no IEC-INT/2022/MPH-59). Permission was obtained from the Chandigarh administration for the participation of HCWs. All ethical criteria were fulfilled before, during, and after data collection. Informed consent was obtained from participants before the interviews using a written informed consent form. A participant information sheet was given to all the participants before participation in the study. Personal information of participants was kept confidential, and there was no breach of confidentiality. The study data were anonymous. As this was an observational study, no compensation was provided, as per IEC guidelines. No identification of individual participants or users in any images in the paper or Multimedia Appendix was possible.

Results

Implementation Status of DHIs at the Primary Care Level

Regarding the implementation status of DHIs, eVIN and RCH were found to be effectively implemented (70%‐90%), whereas ANMOL was not implemented. HMIS, HWC, CPHC-NCD app, and FP-LMIS were found to be poorly implemented (25%‐50%). eSanjeevani and IDSP-IHIP portal fall in the category of fairly implemented (51%‐70%). HWC and HMIS were not operational in 2 HWCs because DEOs were not positioned at these centers. FP-LMIS and CPHC-NCD were in the poorly implemented category, as they were not operational due to technical issues or lack of training. Although inputs were available for the ANMOL app, it was not operational in any HWC because of technical issues. Regarding maturity level, all DHIs were in the mature category except the FP-LMIS app, eSanjeevani, and CPHC-NCD app, which were at the scale-up level (Table 1).

Awareness and Users’ Response Among Target End Users

Out of 120 target end users, 33.3% (40/120) were in the age group of 18‐30 years and 30‐45 years, and 86.7% (104/120) of participants owned a smartphone. Around 90% (108/115) of those with access reported having an adequate internet connection. There was low awareness (ever heard of DHIs or knew something about the respective DHI) of DHIs (range: 1%-18.3%; 1/120 to 22/120), except for Aarogya Setu (94/120, 78.3%) and the CoWIN app (43/120, 35.8%) in the community. Out of 94 participants, 56 (46.7%) had downloaded the Aarogya Setu app (Table 2). Clients with low socioeconomic status and low education were significantly more likely to be unaware of DHIs (P=.02 and P=.04, respectively). There was no statistically significant association between age, sex, and awareness of DHIs (Table 3). Since the awareness of DHIs was very low, we limited our questions to awareness only, and user response was not explored.

Status of HCWs’ Usability

Nearly 49.2% (59/120) of HCWs were aged 31 to 40 years, 87.5% (105/120) were females, 66.6% (80/120) were working as ANMs and MPWs, 5.8% (7/120) as DEOs, and 10% (12/120) as medical officers (Table 4).

In terms of the digital ecosystem, 85% (102/120) of participants reported an adequate number of computers or tablets at the workplace (perceived adequacy), and 41%(49/120) reported having satisfactory internet strength. In total, 66% (80/120) of HCWs rated working on DHIs as “easy,” and 89.2% (107/120) agreed that dual data entry has increased their workload. English (115/120, 95%) was the most preferred language while working on DHIs (Table 5).

ANMs and MPWs reported disagreement in most of the parameters for the ANMOL portal (78/80, 97%), mainly due to technical glitches. It was reported that even after entering data on the app, the data were not synced with the RCH portal. Therefore, they had to maintain similar details of one beneficiary at three places, that is, the RCH register, the RCH portal, and the ANMOL app. Triple data entry was the reason for lack of satisfaction (1/80, 1.3%), low motivation (1/80, 1.3%), and low perceived usefulness (5/80, 6.3%) of the ANMOL portal among ANMs. In total, 46.3% (38/85) of HCWs reported spending >90 minutes per day on entering data on the RCH portal. eSanjeevani, currently in the early phase of implementation, was used by medical officers (n=15). In addition, 60% (9/15) of them rated it as “easy to use,” but 33.3% (5/15) reported having technical issues (slow internet speed) most of the time. Help desk or feedback options were rarely used by health care providers for any of the DHIs. The most often cited reason for limited use of the feedback or help desk option was delayed responses. It was informed that verbal communication with superiors was the most common way to raise queries or provide feedback on DHIs. Technical issues were reported with all of the DHIs except for eVIN (34/82, 70% never; 14/82, 30% rarely). HCWs clearly understood the language of DHIs except for HMIS (5/10, 50%). ANMs or MPWs reported the IDPS portal (S form) as “easy to use” (74/80, 92%), and 66% (53/80) were satisfied with the functioning of the portal, albeit only 27% (23/80) of ANMs were motivated to use it. The primary reason was the belief among ANMs or MPWs that their primary duty was to provide MCH services, and that activities related to disease surveillance should be delegated to other staff (Table 5).

Discussion

Principal Findings

The results of this study highlighted that the RCH portal and eVIN app were effectively implemented; the IDSP-IHIP portal and eSanjeevani were fairly implemented, the HMIS portal, HWC portal, CPHC-NCD portal or app, and FP-LMIS were weakly implemented; and the ANMOL app was not operational in the study HWCs. There was very low awareness among clients (general population and beneficiaries) regarding DHIs launched by the Government of India to improve health literacy and health-seeking behavior (except for Aarogya Setu and the CoWIN app). This low awareness was found to be significantly associated with low socioeconomic status and low education level. The majority of the HCWs agreed on having an adequate digital ecosystem (number of computers, tablets, and connectivity) and considered operating data online to be easy. However, they reported increased workload due to dual data entry. Despite having a similar ecosystem at health care facilities, health care providers’ responses to each DHI varied depending upon technical issues, the time consumed to enter data, and individual factors.

Effective implementation of eVIN at the primary health care level is similar to a recent study that reported improvement in vaccines cold chain management after implementation of eVIN [21]. This could be because of the better user interface of eVIN or the smaller number of data elements to be entered on the eVIN portal. The RCH portal, although effectively implemented, needed improvement in its output indicators. Although the RCH portal was supposed to be a cloud-based system, data interoperability was up to the state level only, while its usage was at the primary care level. A beneficiary in the second or third trimester migrating to a UT needed to reregister herself again on the portal. This was one of the reasons for the inability to achieve full registration within the first trimester. A study conducted in Uttar Pradesh and Rajasthan on the assessment of an earlier version of the RCH portal, that is, the Mother and Child Tracking System (MCTS) portal, reported poor data quality assessment (overall performance numbers were 34% for pregnant women and 33% for children in Rajasthan, while performance numbers were 18% for pregnant women and 25% for children in Uttar Pradesh) [22]. However, a study on the MCTS portal in the Ambala district of Haryana reported that HCWs perceived that MCTS has ensured better data quality [23]. Although HMIS has evolved into a robust system for health information management up to the primary level, due to a lack of dedicated DEOs, reports were sent in paper-based format to the district health administration at 2 HWCs. A study on the implementation status of HMIS in hospitals of central Ethiopia reported overall implementation as poor with a 56% overall percentage score [24]. It was even lower, that is, 42% in our study. The parameters used in the Ethiopia study (2021) were availability, compliance, and use of HMIS in tertiary care hospitals.

At the community level, this study reported low awareness of DHIs among clients despite having a smartphone and adequate internet connection, except for the Aarogya Setu app and CoWIN. This could be because of the enhanced awareness during the COVID-19 pandemic in the community due to widespread mass marketing of the Aarogya Setu app and regulations mandating its download while traveling during the pandemic [25]. Around 46.7% (56/120) of participants had downloaded the Aarogya Setu app in this study. Responses of users on usefulness (53%), easy to use (70%), and satisfaction with functionality (50%) were similar to a review study by Kodali et al [26], which reported higher satisfaction level (>50%), acceptance rate (80%), and perceived usefulness (72.8%) as important parameters among clients for DHI usability. A study conducted in Singapore on awareness of the COVID-19 pandemic and adoption of digital health services in urban Asian populations reported a high level of awareness of COVID-19 infection but low acceptance of digital health services. They also reported that older and lower-income participants were less likely to use digital health services [27]. This finding is similar to our study regarding lower awareness among people from low socioeconomic status; however, no significant association was noted with age. These results were also consistent with a systematic literature review by Reiners et al [28], which explored sociodemographic factors affecting the use of eHealth. It was reported that people with lower income, less education, and those living in rural areas were less likely to use eHealth. Tailored delivery of eHealth was recommended to improve usage [28]. Hence, to increase awareness and empower Indian citizens with DHIs, a push in terms of awareness camps and counseling at health facilities is needed.

Use of feedback or help desk option for the HCWs was limited despite facing several technical issues in running the portals, such as the ANMOL app and the RCH portal. This might be due to a defective or underperforming feedback resolution mechanism, leading to HCWs rarely using it in the future. Despite the majority of participants perceiving data entry into the digital system as easy, HCWs reported (89%, 107/120) an increase in workload due to dual data entry (both in manual and digital form) and considered working on DHIs to be extremely time-consuming. A study on the usage of a health information portal at the primary health care level in Haryana also reported an increased workload of health workers as a result of additional time spent on entering physical records. There were limited opportunities for using the data produced at the primary health care level in web-based platforms [29]. Another study conducted in Uttar Pradesh and Rajasthan on the assessment of the MCTS portal [22], and in Ambala district, Haryana [23], reported overburdened HCWs due to data documentation, which was identified as a barrier to implementation of MCTS.

Strengths

This is the first study to assess the implementation status of DHIs at the primary health care level using a logic framework approach and WHO guidelines on monitoring and evaluation of DHIs in Chandigarh, India. Using input, process, and output parameters helped in identifying implementation gaps at these levels and providing specific recommendations that will strengthen the implementation of the DHIs at the primary care level. This study has also looked into the awareness and usability of digital health initiatives from the clients’ perspective to improve health literacy, awareness, and health service usage. The sample size and sampling technique for assessing the usability of the HCWs are representative of Chandigarh city, and hence those results can be generalizable to the city.

Limitations

There might be limitations in deciding the input, process, and output indicators, which might influence the implementation status of the DHIs. However, care was taken while deciding on these indicators from the review of literature, respective DHIs’ guidelines, and public health experts’ opinions. Due to time constraints and feasibility issues, the implementation status of DHIs was assessed in 4 centers; hence, selection bias cannot be ruled out, which may limit the external validity. Each of these DHIs needs further exploration on implementation, barriers, and facilitators specific to a program by conducting a qualitative study and including a greater number of HWCs for the generalizability of results. The generalizability of the results pertaining to the clients’ usability and awareness regarding DHIs is limited to the study area only. Due to feasibility and time constraints, we aimed only for descriptive analysis. The usability of DHIs by health care providers and associated factors can be further assessed more objectively in future studies.

Conclusions

Effective implementation and use are important for the sustainability and positive impact of DHIs. It is important to assess and monitor the currently operational DHIs in terms of inputs, processes, and outputs to identify gaps in implementation and ways to improve them. Also, how routinely HCWs use DHIs and their response to use will eventually affect the output, outcome, and impact of DHIs. As the Government of India is moving forward to a digitized health care system, it is important to know the awareness of DHIs at the community level, as they are the end-target users and receivers of digital health services. This study assessed the perspective of target-end users of DHIs. Awareness, acceptance, and demand for DHIs among marginalized people are of utmost importance for reducing health inequality and improving their health status. That will eventually help to achieve the Sustainable Development Goals and universal health coverage. The results of this study might guide policy makers to identify gaps and improve the implementation of DHIs by conducting enhanced training and timely refresher training sessions on the use or incentivization of HCWs, developing targeted awareness campaigns, or technical support initiatives. Further, adopting technological innovations such as automating data entry or improving system interfaces, along with a robust feedback or issues-resolution system, can minimize technological barriers. We recommend that there is a need for further in-depth assessment of barriers to implementation, awareness, and adoption of poorly performing government-launched DHIs, as well as evaluating long-term outcomes such as sustained adoption rates and improvement in health care delivery. This study also suggests the need to develop behavior-change communication interventions to increase the demand and uptake of DHIs in the community.