The innovative tools to expand HIVST (I-TEST) project is a National Institutes of Health (NIH)–sponsored research collaboration between the Nigerian Institute of Medical Research (NIMR), Saint Louis University, New York University, and the University of North Carolina at Chapel Hill [18]. To meet the challenge of using mHealth to promote HIVST among young people in Nigeria who do not have access to smart mobile phones, we engaged a team of developers to develop a USSD platform with the aim of verifying HIVST uptake by guiding users (youths) through the proper use of the HIVST kit and linking them to care and anonymous counseling.

First, the participant enters a numeric code (eg, *347*4929#) on their mobile phone. The network provider then validates the code, obtained from Africa’s Talking Ltd, a Kenya-based communication company that organizes USSD code service for a fee. After validation of the code, the process is repeated in reverse after contact with the third-party web-based app, with the response going back to the USSD gateway, which displays the response’s content on the screen of the user’s mobile phone (Figure 1).

The USSD code requires the user to enter key information required for the research, like a unique ID, the current state of residence, the current local government area of residence, age, sexual orientation, gender, and finally, the number of lines that appear on the test kit after the HIVST process has been completed. The USSD platform also provided an opportunity for posttest counseling by collecting information on participants’ mobile numbers (Multimedia Appendix 1).

On completing the test, the participant is sent an SMS notifying him or her of the test result. The information collected from the participant is saved into a database. Also, an administrative portal at the backend helps manage all records of participants involved in the research (Multimedia Appendix 2). The records on this admin portal can be exported.

This study took place in Lagos State, Nigeria, which is a state located in the South-West geopolitical zone of Nigeria with an estimated 24.6 million inhabitants as of 2015. Lagos State is a major financial center in Nigeria. It has 20 local government areas and 37 local council development areas.

The participants’ demographic characteristics (user testing and pilot study) were evaluated and reported as descriptive statistics. The recorded interview with the participants was transcribed verbatim, and inductive thematic analysis was done by members of the research team with experience in qualitative research. The inductive thematic analysis process ensured that the thematic analysis was completely data driven. The process involved repeated reading of the transcripts to become familiar with the data, generating initial codes relevant to the research questions, organizing the codes generated, arranging the subthemes into overarching themes, and defining and naming themes. A descriptive analysis of the findings from the heuristic questionnaire and the pilot study was also done. Variables were reported as proportions with bivariate analysis of the association between some variables through SPSS (version 26; IBM Corp).

The institutional review board of the Nigerian Institute of Medical Research (IRB-18-028) and Saint Louis University (approval number 31457) gave ethical approval for the Innovative Tools to Expand HIVST study, and informed consent was obtained from each participant before the commencement of the usability beta testing study.

A total of 7 of Nielsen’s 10 usability heuristics yielded positive results. The visibility of the system status of the USSD platform showed that the user could tell the state of the platform and the alternatives for action (19/19), and the response times were appropriate to the user’s cognitive processing (18/19). The assessment of the match between the USSD platform and the real world showed that menu choices were ordered in the most logical way, given the user, the item names, and the task variables (18/19), and questions were stated in clear, simple language for the question-and-answer interfaces (19/19). Assessment of consistency and standards of the platform revealed that the field labels are consistent from 1 data entry screen to another (19/19), and commands are used the same way and mean the same thing in all parts of the system (16/19). Also, on whether the platform helps users to recognize, diagnose, and recover from errors, all the participants reported that messages place users in control of the system, and the prompts were brief and unambiguous. Similarly, the assessment of recognition and recall revealed that there was a good color and brightness contrast between image and background colors (19/19), and there was an obvious visual distinction made between the “choose one” menu and the “choose many” menus (18/19). On the aesthetic and minimalist design of the USSD platform, pop-up or pull-down menus within data entry fields do have many, but well-defined, entry options (19/19), and the field labels are brief, familiar, and descriptive (100%), while in the help and documentation assessment, all the users reported that instructions follow the sequence of user actions and the information on the platform is relevant (Table 1).

However, 3 of Nielsen's usability heuristics yielded negative results. These were user control and freedom, error prevention, and autonomy. The assessment of the user’s control and freedom revealed that the majority of the participants felt that users could not easily reverse their actions (14/19), and 7 out of 19 felt that if the platform uses a question-and-answer interface, users cannot go back to previous questions or skip forward to later questions, and 8/19 felt that users cannot change their earlier menu choice if they can go back to a previous menu. Concerning error prevention, 6 out of 19 of the participants believe that the platform does not prevent users from making errors whenever possible, and 7 out of 19 felt that the platform does warn users if they are about to make a potentially serious error. While the assessment of user autonomy showed that 9 out of 19 participants believed that the platform does not give the option to return to factory settings.

Review and recommendations from the heuristic evaluation: 1 of the limitations of a USSD platform is the difficulty or challenge of reversing actions after they have been completed or when an error occurs while using the platform without an opportunity to go back to the beginning. It was recommended that the question prompts on the platform be simplified to reduce this challenge. Also, to address the concern about users' autonomy, it was recommended that a unique ID be assigned to the user instead of a name. These changes were incorporated into the pilot study.

A qualitative assessment of the user experience was done through exit interviews. A total of 13 youths out of the 19 who participated in the usability testing agreed to participate in the exit interview.

Pilot field testing of the USSD platform was done among 164 young persons. Females were in the majority (101/164, 61.6%), and the mean age of the participants was 17.5 (SD 3.18) years. Most participants were out-of-school AYAs involved in various unskilled and semiskilled work, trading, and nonprofessional work in the communities. Evaluation of their sexual orientation revealed that 92.1% (151/164) of the participants reported that they were heterosexual, while 7.9% (13/164) of them reported that they were gay (Table 2). All the participants in the pilot study were able to conduct HIVST, interpret their results, and use the linkage to care feature of the USSD platform without any challenge. A total of 7.9% (13/164) of youths had reactive HIV results (reactive to the OraQuick kit), and 92.1% (151/164) of them were nonreactive. However, among the 13 persons who were reactive and came for a confirmatory test at NIMR, none were confirmed HIV positive.

There was no statistically significant association between the gender of the participants and the HIV-positive result done with the test kit (χ²₁=0.4; P=.76). Also, older participants had positive results using the test kit with a mean age of 19 (SD 3.2) years compared to those who were negative with a mean age of 17 (SD 3.2) years. This difference in age and the HIV-positive test result was not statistically significant (t₁=1.60; P=.11). However, a statistically significant association existed between participants’ sexual orientation and HIV-positive test results using the OraQuick test kits. A total of 38% of persons who identified as gay were reactive using the test kit when compared to 5.3% among those who were heterosexual (χ²₁=18; P<0.001; Table 3).

This study reported on the development, usability, and field pilot of a USSD system to verify HIV self-testing results among AYAs in Nigeria. The study demonstrated the benefits of involving AYAs in the development and usability testing of an intervention aimed at addressing a health challenge peculiar to them. It extends the existing literature [2,14,24] on the feasibility of the use of mHealth among AYAs in Africa. From the study, we were able to demonstrate that the usability of the USSD platform as an alternative platform to the use of mobile apps in the verification of HIV self-testing among young people is good and well accepted.

The young persons who participated in the exit interviews reported that the platform was easy to use and had no significant challenges except for the occasional inability to access the USSD service on their phones, mainly due to mobile network service interruptions. This was said to occur when the user does not have call credit on their mobile phone. This is similar to a challenge with mobile network operators reported by a South African study that submitted that the USSD is prone to network and user timeouts [25].

One of the important features of the USSD platform in interpreting the result of HIVST and promoting linkage to care is that users of the platform could indicate how they would prefer to be contacted after performing HIVST. This could be either to understand the test result or to learn more about HIV prevention, in which case they could either call a mobile phone number or request a call from a trained AYA counselor.

The participants in the heuristic evaluation of the USSD platform reported 3 issues regarding the ease of reversing an action or making corrections if an error was committed when using the platform and the perceived challenge of the user’s autonomy. Though modifications were made to reduce these challenges in the final product, it is noteworthy that these were some of the challenges associated with using the USSD platform for health interventions, as reported by previous studies [25]. However, the benefits of improved health and access provided by the USSD platform were thought to outweigh these challenges [26].

We believe usability assessment using heuristic evaluations among potential end users is unique to this study because the traditional approach for heuristic evaluation uses few content experts, with its attendant shortcomings. However, as previously documented by Silverbratt et al [27], heuristic evaluations involving end users have a higher frequency and relevance value, and the potential of real-life assessment of a new app toward improved acceptability of the same [27].

The proportion of participants in the pilot study who tested positive after HIVST was about 7.9% (13/164), and none were confirmed to be living with HIV infection. This finding was not surprising because a previous study documented the sensitivity of the Oraquick HIVST kit to be between 92% and 93%, even though its specificity is as high as 99.9% [28,29]. Expectedly, most of the participants in the pilot study requested a phone call from the study team through the USSD platform after their HIVST was completed. All participants with reactive HIV test results were contacted by the AYA counselor and encouraged to do confirmatory HIV testing.

Furthermore, given that all categories of young people could benefit from the use of the USSD platform for HIVST because it is cheaper, and does not require the use of internet data, and can be used on all mobile phones (whether they are smartphones or not), we chose the population of out-of-school young persons who were not likely to have a smartphone in the inner-city of an urban area of Lagos State. Successful piloting of the USSD platform among this population suggests that the USSD platform would be useful in promoting HIVST among many Nigerian AYAs. Also, the iterative design that used input from end users in the design increased the acceptability of the platform in promoting HIVST.

Even though the piloting was done in an urban population among out-of-school young persons, caution needs to be expressed in generalizing the findings to the population of AYAs in Nigeria. Also, though the exit interviews were conducted with measures to ensure privacy, removing the potential for social desirability bias is difficult because the young people might want to say what they consider appropriate about using a new mobile phone feature.

This study demonstrated the usability and feasibility of the USSD system as an alternative to mobile phone apps for the verification of HIVST results among the population of Nigerian AYAs without access to smartphones. Hence, the scale-up of access to creative and youth-friendly mHealth interventions is urgently needed, especially among the AYA population, who would benefit from the USSD for HIVST services. Additionally, further research using implementation research methods is necessary to further document this evidence-based intervention’s adoption, feasibility, and sustainability in resource-limited settings.