Published on in Vol 6, No 12 (2022): December

Preprints (earlier versions) of this paper are available at, first published .
Ethical Issues in the Use of Smartphone Apps for HIV Prevention in Malaysia: Focus Group Study With Men Who Have Sex With Men

Ethical Issues in the Use of Smartphone Apps for HIV Prevention in Malaysia: Focus Group Study With Men Who Have Sex With Men

Ethical Issues in the Use of Smartphone Apps for HIV Prevention in Malaysia: Focus Group Study With Men Who Have Sex With Men

Original Paper

1Department of Allied Health Sciences, University of Connecticut, Storrs, CT, United States

2AIDS Program, Yale School of Medicine, New Haven, CT, United States

3Department of Physical Therapy, University of the Philippines, Manila, Philippines

4Centre of Excellence for Research in AIDS (CERiA), University of Malaya, Kuala Lumpur, Malaysia

Corresponding Author:

Roman Shrestha, MPH, PhD

Department of Allied Health Sciences

University of Connecticut

358 Mansfield Road Unit 1101

Storrs, CT, 06269

United States

Phone: 1 8604862834


Background: The use of smartphone apps can improve the HIV prevention cascade for key populations such as men who have sex with men (MSM). In Malaysia, where stigma and discrimination toward MSM are high, mobile health app-based strategies have the potential to open new frontiers for HIV prevention. However, little guidance is available to inform researchers about the ethical concerns that are unique to the development and implementation of app-based HIV prevention programs.

Objective: This study aimed to fill this gap by characterizing the attitudes and concerns of Malaysian MSM regarding HIV prevention mobile apps, particularly regarding the ethical aspects surrounding their use.

Methods: We conducted web-based focus group discussions with 23 MSM between August and September 2021. Using in-depth semistructured interviews, participants were asked about the risks and ethical issues they perceived to be associated with using mobile apps for HIV prevention. Each session was digitally recorded and transcribed. Transcripts were inductively coded using the Dedoose software (SocioCultural Research Consultants) and analyzed to identify and interpret emerging themes.

Results: Although participants were highly willing to use app-based strategies for HIV prevention, they raised several ethical concerns related to their use. Prominent concerns raised by participants included privacy and confidentiality concerns, including fear of third-party access to personal health information (eg, friends or family and government agencies), issues around personal health data storage and management, equity and equitable access, informed consent, and regulation.

Conclusions: The study’s findings highlight the role of ethical concerns related to the use of app-based HIV prevention programs. Given the ever-growing nature of such technological platforms that are intermixed with a complex ethical-legal landscape, mobile health platforms must be safe and secure to minimize unintended harm, safeguard user privacy and confidentiality, and obtain public trust and uptake.

JMIR Form Res 2022;6(12):e42939




Between 2020 and 2021, HIV incidence around the globe witnessed an annual decline of 3.6%, the smallest since 2016 [1]. Although recent HIV incidence and mortality trends have declined globally, some countries in Southeast Asia still bear a disproportionate HIV burden [2-4]. Among the 1.5 million new HIV cases reported globally in 2021, a total of 260,000 were from the Asia and Pacific region, where HIV infection rates are now rising, even in locations where they had been previously declining [1].

Malaysia has one of the highest HIV prevalence in the Asia-Pacific region and remains one of the only a few countries globally where HIV-related mortality has been steadily increasing [5]. In addition, it remains one of the HIV hot spots in that region, with alarming increases in HIV infections [1] and 5500 new cases in 2021 among adults aged ≥15 years [6]. Malaysia’s HIV epidemic is rapidly expanding, with recent evidence suggesting accelerated sexual transmission, especially in men who have sex with men (MSM) [7,8]. Several factors may potentiate HIV transmission among Malaysian MSM, including condomless sex, sexually transmitted infections, and comorbid psychiatric or substance use disorders [9-14].

Same-sex behavior is illegal in Malaysia under the Sharia law and section 377 of the Malaysian Penal Code. Under section 377, “carnal intercourse against the order of nature” is punishable with imprisonment that may last for 20 years and include whipping [15,16]. Similarly, the Islamic Sharia law criminalizes anal sex among Muslims and carries similar punishments with imprisonment, fines, and whipping. Instances of Malaysian authorities raiding venues regularly visited by MSM to enforce the law have also been reported, along with cases of physical and psychological abuse during incarceration among MSM and other men perceived to be MSM who were arrested under section 377 [15].

In this hostile environment that criminalizes and punishes same-sex sexual activity, high levels of stigma and discrimination against Malaysian MSM ensue, including in health care settings [17-19]. The resulting fear of legal repercussions related to the disclosure of same-sex behavior and judgment from health care providers [17,20-22] hinder access to and use of HIV prevention and treatment services, contesting the role of in-person venues for HIV prevention services, such as HIV testing and pre-exposure prophylaxis (PrEP).

The recent advances in wireless technology, along with the change in the traditional health care service delivery model, have led to the development of mobile health (mHealth), which offers an unparalleled opportunity to deliver internet-based health care services. These technologies have increasingly been used in health care management and delivery to address various health conditions and diseases, including diabetes and hypertension management, smoking cessation, weight loss, increased physical activity, and sexually transmitted infections [23-28]. Such technologies have many potential health care benefits, such as monitoring users’ health status remotely and continuously, improving access to health care services, lowering health care costs, increasing patients’ awareness of health status, and improving patient-provider communications.

mHealth interventions can particularly benefit HIV prevention efforts in Malaysia, where smartphone ownership among MSM is nearly universal. Existing data indicate that over 97% of MSM in Malaysia have access to a smartphone, making mHealth a highly feasible platform for this subpopulation [29]. Furthermore, findings from recent studies demonstrate a strong preference among MSM for smartphone apps over other modalities (eg, text, phone calls, and emails) to engage in mHealth HIV prevention tools, thus supporting the development and deployment of smartphone apps for HIV prevention [29-31]. mHealth could serve as an innovative platform to improve access to and use of HIV testing, linkage to PrEP and antiretroviral therapy, and other support services (eg, mental health and substance use).

Despite the numerous benefits of app-based platforms, several ethical challenges accompany their use and must be considered to safeguard user safety, privacy, and rights. These include the protection of privacy and confidentiality, informed consent (including transparency with users about potential risks), data management and careful communication of data, mHealth product regulation and evaluation before public dissemination and uptake, and equity and equitable access [32-34]. Researchers, clinicians, and app developers must be knowledgeable about the ethical issues surrounding the use of HIV prevention apps to improve the perceived usefulness, interpretability, navigability, feasibility, and acceptability of such platforms and tailor them to the specific needs of their target population.


Although ethical considerations for telehealth have been well addressed in the literature, particularly in high-income countries [35,36], patient feedback on ethical issues around using smartphone apps for HIV prevention is sparse, especially in low- and middle-income countries such as Malaysia [31]. Therefore, this study aims to fill this gap by characterizing the attitudes and concerns of Malaysian MSM regarding HIV prevention mobile apps, particularly regarding the ethical aspects surrounding their use.

Study Design

The reporting of this manuscript was guided by the Standards for Reporting Qualitative Research [37]. This study used a qualitative exploratory design using focus group discussions (FGDs) as an interview technique. The authors first developed an interview guide, which was consolidated according to the available literature on key ethical issues related to mHealth [32,33,38].

Existing Ethical Principles and Interview Guide Development

To generate a comprehensive list of ethical principles related to HIV prevention mHealth app use, we first identified the general ethical framework’s 8 principles for biomedical research [39]. These included collaborative partnership, social value, scientific validity, fair subject selection, favorable risk-benefit ratio, independent review, and informed consent. We then explored ethical issues in the context of mHealth research among people living with HIV [32], which further stressed the importance of associated physical, social, behavioral, and psychological risks and privacy and confidentiality risks. In addition, we identified 2 mHealth ethics frameworks [33,38] that elaborated on the accessibility of mHealth platforms, informed consent, and regulation of mHealth products.

To develop the interview guide, we modified the general ethical framework for biomedical research principles by adapting it to mHealth-related research. For example, the social value and scientific validity principles were combined [32]. The collaborative partnership and independent review ethical principles were not addressed in the mHealth-specific ethics frameworks and were removed from the guide. Furthermore, Carter et al [33] described issues with access to mHealth, such as socioeconomic status and physical or mental impairments. These were subsumed under fair subject selection. Carter et al [33] and Fisher et al [38] discussed privacy and consent in more detail, which helped to populate probing questions. Finally, Shrestha et al [33] discussed regulation, which was not addressed in other frameworks and was thus added to the topic list. Finally, feedback from authors was taken until a consensus was reached. This was followed by 4 iterative rounds of revisions, rearrangements, and merging of topics between coauthors that ultimately generated the final version of the guide used in all FGDs.

Study Setting and Recruitment

We recruited a convenience sample of 23 MSM between August and September 2021. The eligibility criteria were as follows: (1) a self-reported negative or unknown HIV status, (2) aged ≥18 years, (3) identifying as male, and (4) the ability to read and understand English or Bahasa Malaysia. Participants were recruited using advertisements on geosocial networking (GSN) apps for MSM (ie, Hornet) and Facebook (a popular social networking website for the general population). The GSN apps pushed the advertisement as a message to the chat inboxes of all users in Malaysia. Targeted banner advertisements were used on Facebook. These banners appeared either as a static advertisement on the right-hand pane of the website or an advertisement that resembled a standard post that users could encounter while scrolling through their feed; clicking on the advertisements directed interested persons to another page to provide their contact information. A research staff member then contacted eligible individuals, assessed their willingness to join the study, and shared more details with eligible and willing participants.

Study Procedures and Measures

Owing to movement restrictions related to the COVID-19 pandemic in Malaysia, participant recruitment and FGDs were conducted via the internet. FGDs were conducted using a videoconferencing platform (ie, WebEx), and each session lasted approximately 90 minutes. Three FGDs were conducted until theoretical saturation was reached [40]. The first 2 FGDs included 8 participants, whereas the last session included 7. A trained facilitator led the FGD session, whereas a cofacilitator took notes, recorded nonverbal cues, and collected chat entries.

Each session began with a brief introduction in which a description of the purpose of the FGD was communicated to participants; time was dedicated to answering participants’ questions and concerns, and participants were reminded that the session would be recorded. Participants were informed that they were not required to use their real names and that they could keep their cameras off during the session to ensure privacy. Participants were encouraged to share their feedback verbally or through the chat function, which was frequently reviewed to ensure that all participant responses were collected. Each session was audio recorded and transcribed. Two authors reviewed the transcripts for accuracy.

Following informed consent, participants completed a brief web-based survey on Qualtrics that included sociodemographic information (eg, age, ethnicity, and sexual orientation), access to communication devices (eg, mobile phones, tablets, and laptops), and awareness of and previous use of HIV prevention services (eg, HIV testing and PrEP).

Before the discussion, participants were briefly provided with information regarding the use of smartphone apps for HIV prevention, including the common features and functions that apps for HIV prevention comprise to support HIV prevention efforts and care. Some key app features included receiving reminders to take PrEP; scheduling appointments with health care providers; ordering PrEP and HIV self-testing kits; and reviewing test results, medication intake, and other health-related information through the app.

Participants were then asked to provide insights into perceived individual- and community-level ethical challenges and concerns about using an HIV prevention app with such features. The FGDs started with a general question to participants regarding their impression and thoughts on the use of a mobile app that contains the features and functions that were just presented (“What is your general impression regarding the use of such an app with these features embedded?”). A more specific question followed, pertaining to the risks of using such apps for HIV prevention (“What do you think are some of the risks of using such an app for HIV prevention efforts?”). Participants were then probed on each of the 5 ethical principles (previously described) that were the focus of the FGD sessions through multiple questions on each ethical aspect.

The first ethical concern was privacy, specifically regarding anonymity and data deidentification. The subconstructs included app visibility (eg, “how would you feel about the app being visible on your phone?”), photos or avatars (eg, “how do you feel about using photo avatars and aliases in setting up your profile in the app?”), and reminders and notifications (eg, “what concerns might you have regarding PrEP reminders?”). The second ethical principle was the third-party use of data. Therefore, questions regarding potential confidentiality breaches were asked (eg, “do you have any concerns about your data being accessible to others?”). Questions regarding the storage and transmission of data were then addressed, including questions about server locations and types (eg, “if the data were stored on a server in Malaysia or in a foreign country, how would you feel about your data security/privacy?”), governmental interception of stored data (eg, “are you concerned about the government accessing your personal information from the app?”), and sharing personal health-related information (eg, “how comfortable do you feel about sharing personal health information via the app?”). Finally, questions about access to mHealth technology (eg, “do you think there are smaller groups in your community who will have difficulty using the app or getting access to a smartphone?”) and regulation of mHealth products in Malaysia (eg, “how do you feel about using an app that does or does not meet certain minimum standards for security, privacy or quality that leading companies or industry groups set?”) were raised. In addition, while each ethical issue was being discussed, participants were encouraged to provide recommendations or suggestions to help mitigate or solve the concerns they brought forth. For example, during the discussion on third-party use of data, participants were asked the following question: “What steps could app developers take to address the security and confidentiality breach concerns that you have?”

Data Analysis

Descriptive statistics for variables collected via a brief Qualtrics survey were computed, including frequencies and percentages for categorical variables, using SAS (version 9.4; SAS Institute, Inc). The transcripts were analyzed using abductive thematic analysis to inductively identify and interpret the concepts and themes that emerged from the interview transcripts. This method involves multiple readings of transcripts and interview notes and analytic induction via open and axial data coding using Dedoose software (SocioCultural Research Consultants, Los Angeles, CA, US) to organize transcripts thematically. Transcripts were checked for any inconsistencies or mistakes before coding was initiated. A codebook was developed with mutually agreed-upon codes derived from the interview transcripts, and coding was completed independently by 2 researchers (including a senior coder). To ensure reliability, codes were constantly compared for agreement and discussed between the 2 coders, and the senior coder cross-checked all codes [41,42]. The Cohen κ coefficient for agreement was estimated to assess the interrater concordance. Open coding, which involved assigning conceptual codes to small sections of words, phrases, and sentences in transcripts, was followed by axial coding, whereby relationships among similar concepts and categories were identified and combined into themes.

Ethics Approval

Participants provided verbal consent before starting the FGDs and were informed that participation in the study was voluntary. Participants were compensated with RM 45 (approximately US $10) per person for their participation. The study protocol was approved by the Institutional Review Board at the University of Connecticut (L21-0007). All FGD transcripts were deidentified before the analysis, and the web-based survey data were anonymous.


Table 1 provides information on participant characteristics. The mean age of the participants was 33.4 (SD 12.0) years. Most (13/23, 57%) participants were Chinese, identified as being gay (21/23, 91%), had daily access to a smartphone with internet (22/23, 96%), and had daily access to the internet (23/23, 100%). In addition, the vast majority (22/23, 96%) had taken an HIV test at least once and had heard of PrEP previously (22/23, 96%). Finally, 61% (14/23) of participants had already taken PrEP in the past.

Table 1. Characteristics of participants (N=23).
Age (years), mean (SD)33.4 (12.0)
Ethnicitya, n (%)

Chinese13 (57)

Malaya8 (35)

Indian2 (9)
Sexual orientation, n (%)

Gay21 (91)

Bisexual2 (9)
Access to communication technologyb, n (%)

Landline1 (4)

Mobile phone with internet access (ie, smartphone)22 (96)

Mobile phone without internet access2 (9)

Tablet13 (57)

Laptop19 (83)

PC7 (30)
Had daily access to internet, n (%)23 (100)
Primary device to access the internet, n (%)

Smartphone17 (74)

Tablet1 (4)

Laptop3 (13)

PC2 (9)
Ever tested for HIV, n (%)

Yes22 (96)

No1 (4)
Ever heard of PrEPc, n (%)

Yes22 (96)

No1 (4)
Ever taken PrEP, n (%)

Yes14 (61)

No9 (39)

aPercentages may not add to 100% owing to rounding.

bPercentages may not add to 100% because answers are nonexclusive.

cPrEP: pre-exposure prophylaxis.

Ethical Issues and Concerns Around the Use of HIV Prevention Smartphone Apps


Several themes were identified during FGDs regarding ethical concerns around using mobile apps for HIV prevention. Participants were generally concerned about privacy and confidentiality issues, including uploading personal information to mobile apps and how apps would be visible to others on mobile home screens. Another emerging theme was storage and data ownership, whereby participants raised concerns about data management and storage, including server types and locations and governmental access to data. Moreover, concerns about informed consent, access to web-based communication technology (eg, smartphones), and regulation were raised. Participants noted several subpopulations in Malaysia that might have trouble using or accessing mobile apps and demonstrated preferences for endorsing agencies, quality control checks, and locations of app developers and their affiliations. The major themes (privacy and confidentiality, storage and data ownership, access to mHealth technology, informed consent, and regulation) are discussed below in order of overall importance. Cohen κ coefficient showed strong agreement between the coders (Cohen κ=0.851).

Privacy and Confidentiality

The participants highlighted concerns regarding the privacy and confidentiality of the data collected by the app (Table 2). Participants were concerned about third-party access to the data collected via the app (or knowledge of their HIV app use) through incidental discovery, for instance, by someone accessing the phone (eg, family or friends):

Some people have a keen eye, and they might read the app’s name, for example, and guess that, it is related to HIV. Some people would discriminate against others based on their medications, like antiretroviral medications.
Table 2. Ethical concerns brought up in focus group discussions regarding privacy and confidentiality (N=23).
Ethical category themes and subthemesMentions, n
Uploading personal information12

Information for identity verification2

Uploading HIV test result10

Weary of uploading positive results3

Not concerned with uploading information6
Setting up account3

Concerns over disclosing personal information3
Doctors upholding privacy2
App visibility14

App icon8

PrEPa stigma within MSMb community1

Visibility is not an issue5

Discreet or silenced notifications preferred1

Notification customization options4

Pop-up display or message preview3

Sounds or ringtone5
Photos and avatars5

Avatars are an excellent option for discretion2

Customization options are preferred3

Photos are not an issue for younger MSM2
Not worried about privacy5
Level of security5
Breach via theft or hacking4

Access via log-in, password, or OTPc2

Not a concern2

aPrEP: pre-exposure prophylaxis.

bMSM: men who have sex with men.

cOTP: one-time password.

Although this was a relatively uncommon concern, some participants highlighted the need to ensure that information regarding the identity of the app developers or owners (eg, academic institutions, health care organizations, nongovernmental organizations, pharmaceutical companies, and government agencies) is transparent. For example, one individual mentioned:

I want to know “who owns this company” or whatnot. Suppose I feel it is associated with an academic institution or at least with one established in the health industry. In that case, I think it’ll be safer to upload my personal info on it.

Possible solutions to address privacy and confidentiality issues included a log-in screen on opening the app and multifactor authentication or fingerprint or facial recognition methods:

I wonder, sending out like a multi-factor authentication is quite good.

Privacy concerns regarding app visibility and tailoring notifications and avatars were also discussed. For example, 1 participant pointed out that app visibility should be minimized, suggesting that this “would be great for people who are still not comfortable with their sexuality.” One recommendation to address this issue was to incorporate a discreet app icon to make the app inconspicuous when notifications appear, thereby reducing the risk of accidental disclosure to family members or friends. Another suggestion was to deliver encrypted messages to the app users:

Maybe you could customize [the notifications] so that the app sends a message that says, “It’s 8 am, time to brush your teeth,” and you know, that doesn’t really mean brush your teeth, it means “take your PrEP.”

Another opportunity for discretion within the app, as suggested by the participants, involved the customization of profile photos and avatars. A participant suggested:

I think having the option to have avatars and stuff like that will help them to have the confidence to use the app, so I think it’s a great idea to have a few avatars as an option. For me personally, I wouldn’t be uploading my picture on your app, but I don’t think it really matters to have a personal picture on it.

In general, participants seemed to value discretion within the app, leaving no opportunity for accidental disclosure incidents.

Storage and Data Ownership

Another prominent concern that the participants raised was storing and sharing data collected via the app (Table 3). Participants expressly referred to the management of collected data and the data storage methods, including the entities in charge of data storage and individuals and organizations with access to it:

The communication itself is not a problem, but the custodian of the data is. I believe the chat will be safe. I mean, it will be encrypted, right? But it’s just a matter of how the data is being kept.
Table 3. Ethical issues and concerns brought up in focus group discussions regarding data storage and ownership (N=23).
Ethical category themes and subthemesMentions, n
Data ownership3
Government access to stored data3
Data management concerns34
How will data be used1
Data storage14

Location of the server2

Reputable operator3

Who stores or manages data3

Location of the server is not a concern5
Data transmission risks4

Avoid public Wi-Fi1

Not a concern2

Many participants expressed concerns that their personal health-related information may become publicly available and accessible by government agencies or that their sexual orientation may be disclosed to others, including health care providers:

I’m concerned about the custodian of the data that’s being collected by this app. Because it is not just a normal app, you will also be collecting data regarding health.

Participants further pointed out the threat to privacy through third-party access of the app data through government interception or hacking of information sent over the internet. For many participants, the underlying concern in this topic involved the likelihood of Malaysian government officials obtaining access to app data.

Interestingly, most participants who raised concerns about data storage servers were not too concerned about the server’s location. The need for reputable server hosting companies was discussed in several instances. Moreover, data transmission risks were not a major concern. Avoiding public Wi-Fi (or other telecommunication networks) was infrequently mentioned, and no other specific transmission risks were raised.

Access to mHealth Technology

Participants stressed that there is an ethical imperative to not exclude any individuals at risk for HIV (eg, discreet or hidden MSM subgroups, those with low income, those who reside in rural areas, and those who cannot afford technology) from benefiting from this platform (Table 4):

But then you also need to understand that the living standard itself, you know, people who are from the lower class might not be able to afford all this.
Table 4. Ethical issues and concerns brought up in focus group discussions regarding mobile health (mHealth) access and regulation (N=23).
Ethical category themes and subthemesMentions, n
Access to mHealtha technologyN/Ab

MSMc subpopulations10

College students3

Discreet MSM3

Low-income MSM2

MSM in rural areas1

MSM who use drugs1

Young MSM3

Endorsement by ministry or NGOd3

App liability or quality control of medication1

Assurance of HIV test kit quality2
App developer8

Local versus overseas4

Private versus public2

University versus company4

amHealth: mobile health.

bN/A: not applicable.

cMSM: men who have sex with men.

dNGO: nongovernmental organization.

One participant shared the challenges of young individuals (eg, college students) accessing HIV prevention services and indicated the need to tailor the app-based interventions to their specific needs. One potential solution proposed to ensure equitable access in this subgroup was to target college-level LGBTQIA+ (lesbian, gay, bisexual, queer, intersex, asexual, and others) advocacy and support groups to help them access HIV prevention and other health care resources:

They [college students] are often, um, either still under their family’s medical insurance and therefore have difficulty accessing PrEP or have concerns about privacy. So certainly, I think, making the app available to everybody, especially the younger, more vulnerable groups, would be valuable.

As mentioned previously, one of the MSM subpopulations identified by participants as potentially having trouble accessing or using mobile apps for HIV prevention was discreet MSM or those who are not out to their respective communities or social circles. It was noted that providing discreet MSM access to such apps could empower them, link them to care, and encourage them to voice their concerns. In addition, 1 participant noted that the app constitutes a way for MSM who use drugs to be potentially linked to HIV care and addiction treatment programs:

But I really hope that those people who are discreet can find an access to start off, you know, like, open up and voice out what is your concern and what is the care you need. That’s why I really hope it [referring to equitable access to MSM subpopulations] can be done.
Informed Consent

Several participants indicated the importance of informed consent (mentioned 7 times in total), whereby privacy policies and terms of services must be in place to maximize transparency and clarity between the app developers or owners and the front-end app users (Table 4). More specifically, participants suggested that this feature should be added before users sign up or create an account on the app and include details about the type of data that will be collected and the risks and benefits accompanying the use of the app. Furthermore, participants noted that these documents should ideally cover data collection and storage, whereby users would be able to consent to store their data on the mHealth platform or not:

I think at the beginning of the registration on the app, you must put an option to consent for app use, then it shouldn’t be an issue anymore. So, consent should include gathering data and stuff like that.

Participants were generally concerned about the regulation and evaluation of the app-based platforms and the services offered via the platform (Table 4). Participants expressed that one of the ways they would use to ascertain the app’s efficacy was to look at the location (ie, overseas or local) of the developers or owners and the type of the institution (ie, private company or academic institution owned) that owned or endorsed the app, with a preference for overseas entities and educational institutions. In addition, participants wanted to ensure that they could trust app developers or the app owner to conduct general quality control and ensure that the services provided through the app were of high quality and from trustworthy sources:

I am more concerned about the medication supply that I receive [through the app]. I would expect that the meds that are being supplied through the apps are legit, through the right source and as well as being approved by the regulators of Malaysia. I expect that this app will review all the medication being supplied.

Principal Findings and Comparison With Previous Work

mHealth platforms are promising tools that can change the HIV prevention landscape [43-47]. In countries where a hostile sociopolitical environment for MSM prevails, the ensuing stigma and discrimination around homosexuality hinder health care access and have been associated with reduced HIV prevention service uptake among MSM [48,49]. mHealth platforms are well placed to fill this gap, as they provide a web-based platform to deliver HIV services, especially in countries such as Malaysia where MSM are well connected to the internet and use smartphones to seek sexual health information on the web [29]. The findings of this study are critical to assist the development of ethically sound app-based HIV prevention programs, which require the engagement of MSM early on [31].

Among our sample of MSM, concerns about privacy and confidentiality were substantial, bringing forth the issue of third-party data interception, which can occur by accidental disclosure or deliberately through governmental access and subpoena. This finding accentuates previously reported concerns among MSM subgroups regarding the unintentional disclosure of sensitive information to nearby users [31] and further highlights the need for app developers to incorporate app features that address the privacy and confidentiality concerns. Other MSM subgroups have also emphasized their fear of being inadvertently “outed” within their respective communities if they use such platforms in public [50]. Interestingly, a discrepancy between privacy concerns and privacy practices (eg, privacy settings on mobile devices and apps) has also been noted, bringing forth a form of a privacy paradox [51].

In our study, participants stressed on incorporating additional security features in the app (eg, multifactor authentication) and additional features to adjust app visibility, notifications, and avatars to make them more discreet. Similar app features and recommendations were brought up by MSM in previous studies, including clear instructions in the app that prompt users to enable phone privacy features on their mobile device and warnings to view app content in private [52]. More importantly, using encrypted messages, colloquial expressions referring to same-sex behavior, or more neutral content seems to be a recurring suggestion [52,53]. To safeguard user privacy and confidentiality, app developers must communicate with the users about instances in which deidentification of information is not possible and the risk of third-party access to or interception of the data collected via hacking, legal interception, accidental discovery, or telecommunication companies (eg, Google) [33].

Study participants also expressed their concern that access to this platform should be equitable among members of the MSM community, emphasizing the importance of ensuring that specific subgroups, such as hidden MSM and those with low socioeconomic status, have equal access to these services. Similar concerns have been noted in the literature among people living with HIV, whereby ethical considerations such as the accommodation of literacy, infrastructure, and access to technology have been delineated and are viewed as necessary to guarantee equitable access to mHealth services and care [54]. Many established strategies can be implemented to ensure app equity. One potential solution discussed includes the need to market the app to college LGBTQIA+ advocacy groups with greater reach and aptness to share such a platform with vulnerable populations, such as younger MSM, who would be more attentive to privacy risks and, therefore, less able to access such platforms. Adopting a human-centered design approach is also crucial, whereby app developers should increase the participation of participants from underprivileged backgrounds in developing mHealth products to gain insights into their preferences and priorities. In addition to promoting inclusivity by integrating population-specific design or app features, app developers can also promote digital literacy by teaching users how to use mHealth platforms, especially in low-resource or underprivileged subpopulations and settings [55]. Although mHealth technologies are rapidly evolving and despite the impact of sociodemographic inequities on equitable access, there is still a lack of evidence on the equity implications of HIV mHealth platforms. Nevertheless, it is an ethical imperative not to exclude subgroups that lack or have limited access to mHealth, including those who cannot afford to buy a smartphone or have access to the internet and those who might have an impairment that hinders mHealth platform use [33].

A common emerging theme in FGDs was the storage and management of collected data. Participants also referred to entities who will oversee data storage and management, expressing fears regarding data dissemination to third parties (eg, governmental agencies or the public) while maintaining a higher level of trust in general for entities outside of Malaysia as opposed to local entities. MSM’s preferences for eHealth interventions listed in the literature seem to align with these findings, as they expressed fear of disclosure of their HIV status when the intervention originated in organizations or places that they did not know about in the past. In addition, participants would tend to be skeptical about the information disclosed or dismiss messages stemming from such organizations because of the lack of trust [50]. In general, app developers are required to store and transmit the least amount of data possible, which allows the app’s purpose to be fulfilled and to be sensitive to the amount and ways the data are stored on the device itself or transmitted to the clinical team [33]. This ethical consideration has been considered especially sensitive as the data collected are usually detailed and originate from multiple sources that can generate identifiable information if data streams are combined [56]. The unauthorized use of such data can jeopardize user safety and privacy if accessed by third parties [33,56].

Participants emphasized the importance of transparency via privacy policies and terms of service that clearly and explicitly detail which personally identifiable information will be collected, why it is collected, with whom it may be shared, and how users can control their data. It is essential that the participants be given a choice to consent to specific types of data collection and sharing activities [51], which corroborates the concept of “voluntariness” in the context of informed consent, whereby participants should be given a chance to consent to specific data collection streams or app modules but not others [51,56]. However, as there are instances in which opting out of some modules interferes with overall health or risk assessments, the risks and benefits associated with opting in or out of certain app modalities should be thoroughly explained [56]. In addition, using multimodel content, such as audio, video, or infographics, and summarizing key points in laypersons’ terms is encouraged to increase the engagement and comprehension of the users [56-59]. As data protection laws related to mobile apps change over time, these policy documents must be updated, and the users must be informed of such updates accordingly [56,60].

Finally, app-based platforms, particularly when intended to function as aids or alternatives to traditional medical services, carry increased risks to the users. There is a global debate on regulating mHealth apps if they are classified as medical devices. For example, in many high-income countries (eg, the United States and Australia), there are regulatory guidelines that provide oversight to mHealth apps that fulfill the definition of a medical device to ensure safety and effectiveness. However, these do not impede app dissemination, as most mobile apps can still be downloaded by patients via app stores without regulatory filters [33]. Moreover, there are no regulations in many low- and middle-income countries, including Malaysia, that guide app developers on consent, privacy protocols, and ethical practices in delivering health services. Therefore, a dedicated government body must be in place to provide regulatory oversight or “policing” to ensure the safety and effectiveness of such platforms. In addition, mHealth app developers must maintain transparency regarding the scientific evidence (or lack of evidence) underpinning the app’s effectiveness.

Limitations and Strengths

Our study has some limitations. First, our sample of MSM excludes those who are not active on GSN apps, do not have access to the internet, and do not read or understand English or Bahasa Malaysia. Furthermore, although theoretical saturation was reached after the third session, it is likely that additional ethical perceptions that were not brought up during the FGDs exist, which could be identified, for example, by recruiting participants in ways that differ from ours (ie, GSN app advertisements and direct profile inquiries). Second, most participants in our sample were of Chinese ethnicity, had tested for HIV, and were aware of PrEP, which limits the generalizability of our findings to the broader MSM group in Malaysia. Enrolling predominantly Malay participants or MSM who were not linked to HIV prevention services could yield different perspectives on ethical issues. Nevertheless, this study is among the first to examine the ethical issues surrounding the use of smartphone apps for HIV prevention among MSM in Malaysia, which is the first step toward understanding and engaging this specific subgroup. In addition, the web-based modality of the FGDs constituted a safe, confidential, and easily accessible platform for participants, thereby creating a safe environment and fostering a higher level of truthfulness among participants.


In countries such as Malaysia, where homosexuality is illegal, stigma and discrimination around HIV make traditional venue-based HIV prevention services less suitable for vulnerable populations, such as MSM. The findings from this study indicate the potential implications of ethical concerns and associated risks related to using app-based HIV prevention programs in low- and middle-income countries such as Malaysia. Such platforms can be kept safe by integrating security features (eg, multifactor authentication and discreet features to adjust app or notification visibility); storing and transmitting the least amount of data possible; ensuring transparency via clear privacy policies and terms of service; giving participants a choice to consent to specific types of data being collected; and, in the case of medical devices, ensuring adequate regulation by dedicated government bodies. Our findings indicate the need for the systematic engagement of all relevant stakeholders (eg, MSM, community members, and health care providers) and the adoption of an ethical framework in the design and development phases to ensure that the platform is ethical, safe, secure, equitable, and sustainable.


The authors acknowledge the financial support in part by a career development award from the National Institute on Drug Abuse (grant K01 DA051346) and a research grant from the Fogarty International Center (grant R21 TW011665). Data collection for this study was supported by the Fordham HIV Prevention Research Ethics Training Institute through a training grant sponsored by the National Institute on Drug Abuse (grant R25-DA031608). The funders had no role in the study design, data collection, data analysis, decision to publish, or manuscript writing.

Data Availability

The data sets generated or analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

None declared.

  1. UNAIDS Global AIDS Update 2022. Joint United Nations Programme on HIV/AIDS (UNAIDS). 2022.   URL: [accessed 2022-11-16]
  2. Global AIDS Update 2016. Joint United Nations Programme on HIV/AIDS (UNAIDS). 2016.   URL: [accessed 2022-08-05]
  3. Prevention Gap Report. Joint United Nations Programme on HIV/AIDS (UNAIDS). 2016.   URL: [accessed 2022-08-05]
  4. UNAIDS Data 2017. Joint United Nations Programme on HIV/AIDS (UNAIDS). 2017.   URL: [accessed 2022-08-05]
  5. Country Progress Report on HIV/AIDS 2019. Ministry of Health Malaysia. 2019.   URL: [accessed 2022-08-05]
  6. Country Factsheets Malaysia 2021. Joint United Nations Programme on HIV/AIDS (UNAIDS). 2022.   URL: [accessed 2022-11-08]
  7. Global AIDS Monitoring 2020, Malaysia HIV/AIDS Progress Report 2020. Ministry of Health Malaysia. 2020.   URL: https:/​/www.​​moh/​resources/​Penerbitan/​Laporan/​Umum/​Laporan_Global_AIDS_Monitoring_2020_new.​pdf [accessed 2022-08-10]
  8. Global AIDS Monitoring, Country Progress Report, Malaysia. Ministry of Health Malaysia. 2021.   URL: [accessed 2022-08-10]
  9. Chooi WT, Mohd Zaharim N, Desrosiers A, Ahmad I, Yasin MA, Syed Jaapar SZ, et al. Early initiation of amphetamine-type stimulants (ATS) use associated with lowered cognitive performance among individuals with co-occurring opioid and ATS use disorders in Malaysia. J Psychoactive Drugs 2017;49(4):326-332 [FREE Full text] [CrossRef] [Medline]
  10. Desrosiers A, Chooi WT, Zaharim NM, Ahmad I, Mohd Yasin MA, Syed Jaapar SZ, et al. Emerging drug use trends in Kelantan, Malaysia. J Psychoactive Drugs 2016;48(3):218-226. [CrossRef] [Medline]
  11. Singh D, Chawarski MC, Schottenfeld R, Vicknasingam B. Substance abuse and the HIV situation in Malaysia. J Food Drug Anal 2013 Dec;21(4):S46-S51 [FREE Full text] [CrossRef] [Medline]
  12. World Drug Report: 2016. United Nations Office on Drugs and Crime. 2016.   URL: [accessed 2022-08-10]
  13. Kanter J, Koh C, Razali K, Tai R, Izenberg J, Rajan L, et al. Risk behaviour and HIV prevalence among men who have sex with men in a multiethnic society: a venue-based study in Kuala Lumpur, Malaysia. Int J STD AIDS 2011 Jan;22(1):30-37. [CrossRef] [Medline]
  14. Lim SH, Akbar M, Wickersham JA, Kamarulzaman A, Altice FL. The management of methamphetamine use in sexual settings among men who have sex with men in Malaysia. Int J Drug Policy 2018 May;55:256-262 [FREE Full text] [CrossRef] [Medline]
  15. Habib S. Islam and Homosexuality. 2 volumes. Santa Barbara, CA, USA: ABC-CLIO; 2009.
  16. Itaborahy LP, Zhu J. State-Sponsored Homophobia, A world survey of laws: Criminalisation, protection and recognition of same-sex love. International Lesbian, Gay, Bisexual, Trans and Intersex Association (ILGA). 2014.   URL: [accessed 2022-08-25]
  17. Earnshaw VA, Jin H, Wickersham J, Kamarulzaman A, John J, Altice FL. Exploring intentions to discriminate against patients living with HIV/AIDS among future healthcare providers in Malaysia. Trop Med Int Health 2014 Jun;19(6):672-679 [FREE Full text] [CrossRef] [Medline]
  18. Earnshaw VA, Jin H, Wickersham JA, Kamarulzaman A, John J, Lim SH, et al. Stigma toward men who have sex with men among future healthcare providers in Malaysia: would more interpersonal contact reduce prejudice? AIDS Behav 2016 Jan;20(1):98-106 [FREE Full text] [CrossRef] [Medline]
  19. Jin H, Earnshaw VA, Wickersham JA, Kamarulzaman A, Desai MM, John J, et al. An assessment of health-care students' attitudes toward patients with or at high risk for HIV: implications for education and cultural competency. AIDS Care 2014;26(10):1223-1228 [FREE Full text] [CrossRef] [Medline]
  20. Schwartz SR, Nowak RG, Orazulike I, Keshinro B, Ake J, Kennedy S, TRUST Study Group. The immediate effect of the Same-Sex Marriage Prohibition Act on stigma, discrimination, and engagement on HIV prevention and treatment services in men who have sex with men in Nigeria: analysis of prospective data from the TRUST cohort. Lancet HIV 2015 Jul;2(7):e299-e306 [FREE Full text] [CrossRef] [Medline]
  21. Vu L, Tun W, Sheehy M, Nel D. Levels and correlates of internalized homophobia among men who have sex with men in Pretoria, South Africa. AIDS Behav 2012 Apr;16(3):717-723. [CrossRef] [Medline]
  22. Wei C, Cheung DH, Yan H, Li J, Shi LE, Raymond HF. The impact of homophobia and HIV stigma on HIV testing uptake among Chinese men who have sex with men: a mediation analysis. J Acquir Immune Defic Syndr 2016 Jan 01;71(1):87-93 [FREE Full text] [CrossRef] [Medline]
  23. Li R, Liang N, Bu F, Hesketh T. The effectiveness of self-management of hypertension in adults using mobile health: systematic review and meta-analysis. JMIR Mhealth Uhealth 2020 Mar 27;8(3):e17776 [FREE Full text] [CrossRef] [Medline]
  24. MacKinnon GE, Brittain EL. Mobile health technologies in cardiopulmonary disease. Chest 2020 Mar;157(3):654-664 [FREE Full text] [CrossRef] [Medline]
  25. Marcolino MS, Oliveira JA, D'Agostino M, Ribeiro AL, Alkmim MB, Novillo-Ortiz D. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018 Jan 17;6(1):e23 [FREE Full text] [CrossRef] [Medline]
  26. Mönninghoff A, Kramer JN, Hess AJ, Ismailova K, Teepe GW, Tudor Car L, et al. Long-term effectiveness of mHealth physical activity interventions: systematic review and meta-analysis of randomized controlled trials. J Med Internet Res 2021 Apr 30;23(4):e26699 [FREE Full text] [CrossRef] [Medline]
  27. Sales RO, Silva RM. mHealth in the prevention of sexually transmitted infections STIs. Cien Saude Colet 2020 Nov;25(11):4315-4325 [FREE Full text] [CrossRef] [Medline]
  28. Wang Y, Min J, Khuri J, Xue H, Xie B, Kaminsky LA, et al. Effectiveness of mobile health interventions on diabetes and obesity treatment and management: systematic review of systematic reviews. JMIR Mhealth Uhealth 2020 Apr 28;8(4):e15400 [FREE Full text] [CrossRef] [Medline]
  29. Shrestha R, Lim SH, Altice FL, Copenhaver M, Wickersham JA, Saifi R, et al. Use of smartphone to seek sexual health information online among Malaysian men who have sex with men (MSM): implications for mHealth intervention to increase HIV testing and reduce HIV risks. J Community Health 2020 Feb;45(1):10-19 [FREE Full text] [CrossRef] [Medline]
  30. Shrestha R, Maviglia F, Altice FL, DiDomizio E, Khati A, Mistler C, et al. Mobile health technology use and the acceptability of an mHealth platform for HIV prevention among men who have sex with men in Malaysia: cross-sectional respondent-driven sampling survey. J Med Internet Res 2022 Jul 25;24(7):e36917 [FREE Full text] [CrossRef] [Medline]
  31. Shrestha R, Fisher C, Wickersham JA, Khati A, Kim R, Azwa I, et al. Privacy and confidentiality concerns related to the use of mHealth apps for HIV prevention efforts among Malaysian men who have sex with men: cross-sectional survey study. JMIR Form Res 2021 Dec 16;5(12):e28311 [FREE Full text] [CrossRef] [Medline]
  32. Labrique AB, Kirk GD, Westergaard RP, Merritt MW. Ethical issues in mHealth research involving persons living with HIV/AIDS and substance abuse. AIDS Res Treat 2013;2013:189645 [FREE Full text] [CrossRef] [Medline]
  33. Carter A, Liddle J, Hall W, Chenery H. Mobile phones in research and treatment: ethical guidelines and future directions. JMIR Mhealth Uhealth 2015 Oct 16;3(4):e95 [FREE Full text] [CrossRef] [Medline]
  34. Keenan AJ, Tsourtos G, Tieman J. The value of applying ethical principles in telehealth practices: systematic review. J Med Internet Res 2021 Mar 30;23(3):e25698 [FREE Full text] [CrossRef] [Medline]
  35. Fields BG. Regulatory, legal, and ethical considerations of telemedicine. Sleep Med Clin 2020 Sep;15(3):409-416 [FREE Full text] [CrossRef] [Medline]
  36. Kaplan B. Revisiting health information technology ethical, legal, and social issues and evaluation: telehealth/telemedicine and COVID-19. Int J Med Inform 2020 Nov;143:104239 [FREE Full text] [CrossRef] [Medline]
  37. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med 2014 Sep;89(9):1245-1251 [FREE Full text] [CrossRef] [Medline]
  38. Fisher CB, Bragard E, Bloom R. Ethical considerations in HIV eHealth intervention research: implications for informational risk in recruitment, data maintenance, and consent procedures. Curr HIV/AIDS Rep 2020 Jun;17(3):180-189 [FREE Full text] [CrossRef] [Medline]
  39. Emanuel EJ, Wendler D, Killen J, Grady C. What makes clinical research in developing countries ethical? The benchmarks of ethical research. J Infect Dis 2004 Mar 01;189(5):930-937. [CrossRef] [Medline]
  40. Guest G, Namey E, McKenna K. How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field Methods 2016 Jul 24;29(1):3-22 [FREE Full text] [CrossRef]
  41. Creswell JW, Creswell JD. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 5th edition. Thousand Oaks, CA, USA: Sage Publications; 2017.
  42. Horsburgh D. Evaluation of qualitative research. J Clin Nurs 2003 Mar;12(2):307-312. [CrossRef] [Medline]
  43. mHealth: new horizons for health through mobile technologies: second global survey on eHealth. WHO Global Observatory for eHealth. Geneva, Switzerland: World Health Organization; 2011.   URL: [accessed 2022-08-25]
  44. Free C, Phillips G, Watson L, Galli L, Felix L, Edwards P, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med 2013;10(1):e1001363 [FREE Full text] [CrossRef] [Medline]
  45. Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja S, Chung MH, et al. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet 2010 Nov 27;376(9755):1838-1845. [CrossRef] [Medline]
  46. Milward J, Lynskey M, Strang J. Solving the problem of non-attendance in substance abuse services. Drug Alcohol Rev 2014 Nov;33(6):625-636. [CrossRef] [Medline]
  47. Mbuagbaw L, Mursleen S, Lytvyn L, Smieja M, Dolovich L, Thabane L. Mobile phone text messaging interventions for HIV and other chronic diseases: an overview of systematic reviews and framework for evidence transfer. BMC Health Serv Res 2015 Jan 22;15:33 [FREE Full text] [CrossRef] [Medline]
  48. Ayala G, Makofane K, Santos G, Beck J, Do TD, Hebert P, et al. Access to basic HIV-related services and PrEP acceptability among men who have sex with men worldwide: barriers, facilitators, and implications for combination prevention. J Sex Transm Dis 2013;2013:953123 [FREE Full text] [CrossRef] [Medline]
  49. Chesney MA, Smith AW. Critical delays in HIV testing and care: the potential role of stigma. Am Behav Sci 1999 Apr 1;42(7):1162-1174 [FREE Full text] [CrossRef]
  50. Hubach RD, O'Neil A, Stowe M, Giano Z, Curtis B, Fisher CB. Perceived confidentiality risks of mobile technology-based ecologic momentary assessment to assess high-risk behaviors among rural men who have sex with men. Arch Sex Behav 2021 May;50(4):1641-1650 [FREE Full text] [CrossRef] [Medline]
  51. Rendina HJ, Mustanski B. Privacy, trust, and data sharing in web-based and mobile research: participant perspectives in a large nationwide sample of men who have sex with men in the united states. J Med Internet Res 2018 Jul 04;20(7):e233 [FREE Full text] [CrossRef] [Medline]
  52. Zhao Y, Zhu X, Pérez AE, Zhang W, Shi A, Zhang Z, et al. MHealth approach to promote Oral HIV self-testing among men who have sex with men in China: a qualitative description. BMC Public Health 2018 Sep 27;18(1):1146 [FREE Full text] [CrossRef] [Medline]
  53. Nguyen LH, Nguyen HL, Larsson M, Tran BX, L Stein M, Rocha LE, et al. An exploratory assessment of the preference for eHealth interventions to prevent HIV and sexually transmitted infections among men who have sex with men in Hanoi, Vietnam. BMC Public Health 2020 Sep 11;20(1):1387 [FREE Full text] [CrossRef] [Medline]
  54. Duthely LM, Sanchez-Covarrubias AP. Digitized HIV/AIDS treatment adherence interventions: a review of recent SMS/texting mobile health applications and implications for theory and practice. Front Commun (Lausanne) 2020 Nov;5:530164 [FREE Full text] [CrossRef] [Medline]
  55. Fortuna KL, Kadakia A, Cosco TD, Rotondi A, Nicholson J, Mois G, et al. Guidelines to establish an equitable mobile health ecosystem. Psychiatr Serv 2022 Nov 15:appips202200011. [CrossRef] [Medline]
  56. Jacobson NC, Bentley KH, Walton A, Wang SB, Fortgang RG, Millner AJ, et al. Ethical dilemmas posed by mobile health and machine learning in psychiatry research. Bull World Health Organ 2020 Apr 01;98(4):270-276 [FREE Full text] [CrossRef] [Medline]
  57. Opportunities and challenges of developing information technologies on behavioral and social science clinical research. National Institute of Mental Health. 2017.   URL: https:/​/www.​​about/​advisory-boards-and-groups/​namhc/​reports/​opportunities-and-challenges-of-developing-information-technologies-on-behavioral-and-social-science-clinical-research [accessed 2022-09-01]
  58. Martinez-Martin N, Insel TR, Dagum P, Greely HT, Cho MK. Data mining for health: staking out the ethical territory of digital phenotyping. NPJ Digit Med 2018;1:68 [FREE Full text] [CrossRef] [Medline]
  59. Martinez-Martin N, Kreitmair K. Ethical issues for direct-to-consumer digital psychotherapy apps: addressing accountability, data protection, and consent. JMIR Ment Health 2018 Apr 23;5(2):e32 [FREE Full text] [CrossRef] [Medline]
  60. O'Reilly M, Parker N, Hutchby I. Ongoing processes of managing consent: the empirical ethics of using video-recording in clinical practice and research. Clin Ethics 2011 Dec 05;6(4):179-185 [FREE Full text] [CrossRef]

FGD: focus group discussion
GSN: geosocial networking
LGBTQIA+: lesbian, gay, bisexual, queer, intersex, asexual, and others
mHealth: mobile health
MSM: men who have sex with men
PrEP: pre-exposure prophylaxis

Edited by A Mavragani; submitted 24.09.22; peer-reviewed by K Macapagal, V Watzlaf; comments to author 28.10.22; revised version received 19.11.22; accepted 28.11.22; published 23.12.22


©Antoine Khati, Jeffrey A Wickersham, Aviana O Rosen, Jeffrey Ralph B Luces, Nicholas Copenhaver, Alma Jeri-Wahrhaftig, Mohd Akbar Ab Halim, Iskandar Azwa, Kamal Gautam, Kai Hong Ooi, Roman Shrestha. Originally published in JMIR Formative Research (, 23.12.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.