The goal of the user interface is to provide clinical trial principal investigators and research coordinators with immediate access to weekly and cumulative counts at each stage of the study and weekly and cumulative transition rates from stage to stage. The interface uses compelling visual cues to bring the user’s attention to potential problems in the study workflow. The first version of the user interface for RecruitGPS seen in Figure 1 was the version from the successful multisite study dashboard user interface [32] with the generic names step 1 through step 9 substituted for the names of the stages screened through 18-month assessment. The multisite study user interface was collaboratively designed by the stakeholders in the study and the dashboard developer.

Think-aloud usability testing sessions were combined with a user survey to assess the effectiveness of, efficiency of, and satisfaction with the RecruitGPS dashboard. Effectiveness is the extent to which the user can achieve a goal with accuracy and completeness. Efficiency is the effort and resources necessary to achieve a complete and accurate goal. Satisfaction is a positive user experience and absence of discontent during task performance. Once the 3 are fulfilled adequately, the product can be considered to have attained an acceptable level of usability [38].

Participants in the study were recruited to use the RecruitGPS user interface while being video recorded and to complete a survey about their experience. Most participants were affiliated with CHIIS or were clinical researchers primarily based in the Midwest region of the United States. Recruitment and usability testing were conducted in August 2024.

Participants were recruited using convenience and snowball sampling techniques, with the CHIIS team supporting the process through digital advertisements, emails, and word of mouth. To be eligible, participants needed experience with clinical research studies, fluency in English, and access to the internet and Zoom (Zoom Video Communications Inc.). Although the literature suggests that 5 to 8 participants are sufficient for think-aloud usability testing, 12 participants were recruited to ensure diverse and comprehensive insights [39,40].

A total of twelve 10-minute usability testing sessions of the RecruitGPS dashboard were conducted via Zoom. During each session, the facilitator shared their screen, displaying the dashboard, and participants were provided with remote control access to interact with the interface. Participants were instructed as follows: “consider[ing] a research project of your own, explore the various tabs of the dashboard, identify challenges, and note features that work well while thinking aloud.” To identify specific problems, the researchers recorded the 10-minute usability testing sessions for a qualitative analysis of the challenges users encountered and features that worked well.

Following each usability test, participants received a survey link to provide additional feedback. The survey gathered insights on users’ overall experience, including their System Usability Scale (SUS) (Textbox 1) [41] and Net Promoter Score (NPS) [42] ratings. The NPS is based on the question “On a scale of 0 to 10, how likely are you to recommend this dashboard to a colleague or friend?” It is reported as a number ranging from −100 to +100, where a higher score is desirable. Respondents are categorized as detractors (giving scores of 0‐6), passives (giving scores of 7‐8), and promoters (giving scores of 9‐10) The percentages of promoters and detractors are calculated and then the percentage of detractors is subtracted from the percentage of promoters to obtain the NPS [42]. The survey also included open-ended questions about the most and least useful features, as well as overall impressions of the user experience and dashboard interface. To assess ease of navigation in the survey, users were asked the questions in Table 1.

This study was approved by the Human Research Protection Program at Indiana University (23755) prior to participant recruitment. Informed consent was obtained from all participants. Throughout the study, strict confidentiality and privacy protocols were followed. All data collected were deidentified before analysis to maintain participant anonymity. To protect privacy, secure data storage and anonymization procedures were used. Furthermore, no identifiable images of participants were included in the data. Participation was voluntary and no financial compensation or incentives were provided.

Twelve medical professionals completed the usability testing and surveys. Their demographics are summarized in Table 2.

The 10 questions on the SUS together assess overall usability, computed on a scale from 0 to 100, as described in detail by Brooke [41]. Table 3 reports individual item scores and overall SUS scores for each participant. Higher ratings on odd-numbered questions and lower ratings on even-numbered questions indicate better usability.

The mean SUS total score for all participants was 61.46 (SD 23.80; median 66.25) points. According to Bangor et al [43], a score of 85 and above is excellent and a score of 70‐84 is good.

Feedback on ease of navigation showed a mixed picture. While 25% (n=3) of the 12 respondents gave a rating of 5% and 58% (n=7) gave a rating of 4, 17% (n=2) rated ease of navigation a 1.

The NPS [42] for RecruitGPS in this study was 8, with 33% (n=4) of respondents classified as promoters and 25% (n=3) as detractors. This indicates a slightly positive satisfaction.

A second question asked participants to compare RecruitGPS to other recruitment tools they had used on a scale of 1‐5, with 1 meaning RecruitGPS is much worse and 5 meaning Recruit GPS is much better. The results were that 8 of 12 participants thought RecruitGPS was better. The results are summarized in Table 4.

The framework developed by Clarke and Braun [44] was used for thematic analysis of the recorded 10-minute usability testing sessions to identify and analyze patterns within the data, including challenges users encountered and features that worked well.

We found that 8 of 12 (67%) participants considered RecruitGPS better than other recruitment tools they had used, but the usability (SUS), satisfaction (NPS), and efficiency scores all indicated a need for improvement. Feedback on ease of navigation showed significant efficiency challenges faced by a minority of users, suggesting that some users struggled with navigation, despite overall positive feedback.

The recommendations for improvement based on the qualitative analysis are follows:

Although the quantitative analysis highlighted areas for improvement, the testing was conducted without any prior training for participants. The developers of RecruitGPS had not anticipated users attempting to operate the system without access to documentation or instructional materials, such as training videos. Despite this, RecruitGPS performed well enough. Additionally, the qualitative analysis offered valuable insights for enhancing the documentation and provided clear guidance for the development of instructional videos.

The first limitation of this study was the relatively small sample size, consisting of only 12 participants in the usability testing sessions and survey responses. Conducting future studies with a larger and more diverse user group could yield additional insights and a broader understanding of user needs.

A related limitation was the participants’ affiliations, as most were connected to 1 of 2 medical schools or their associated health care systems. While some participants had experience at other research institutions, their familiarity with alternative recruitment tools may have been limited to those used within these 2 medical schools, potentially restricting the study’s generalizability.

Another limitation was that the study did not include testing of the existing documentation, and the instructional videos had not yet been developed. Future testing of these resources could provide valuable feedback to refine and improve their effectiveness.

RecruitGPS demonstrates promise and potential in enhancing recruitment and retention for clinical trials by providing real-time insights, supporting agile decision-making, and using scalable design with an accessible user interface. Removing the cumulative funnel and creating instructional videos and documentation focused on navigation challenges are likely to enhance its usability. With these enhancements, RecruitGPS could become a key tool in clinical trial management, supporting successful participant recruitment and better research outcomes.