This study used a mixed methods, task-based approach to evaluate the overall usability of IVR calls and a workbook. The materials evaluated are components of an intervention (ie, HFC) aiming to culturally adapt a technology-delivered, family-based childhood obesity prevention program for rural Hispanic/Latino families. The HFC focuses on parents as agents of change and includes 2 parent classes, followed by 10 IVR automated telephone support calls over 6 months (Table 1). Program materials are provided in a parent workbook with activities to be completed at home that align with IVR calls to promote increased physical activity and consumption of fruits and vegetables, in concert with decreased sugary-drink consumption and screen time [22]. The HFC is an adaptation of the Family Connections program, originally designed for English speakers [23].

All UTs were conducted using materials in Spanish, and we have translated some examples of the resources used exclusively to present in this article.

All intervention materials evaluated are available in Spanish as part of the cultural adaptation characteristic of this program. The Workbook is divided into 9 chapters, guiding parents through understanding the health impact of childhood obesity and providing education and activities to support behavioral and environmental changes. Examples of activities include learning how to read a nutrition fact label, planning physical activities as a family, role modeling positive behaviors, and reflecting on current shopping and eating habits (Figure 1). The Workbook also includes extra resources and supplemental materials, such as BMI tables and a guide on how to use the calls.

Our recruitment goal was 5 to 10 participants, as recommended for this type of study [24]. Eligibility criteria for participation in this research included the following: parents/guardians/caregivers of children aged 6 to 14 years who were of Hispanic/Latino origin and literate in Spanish. The HFC research team partnered with 24 rural counties in Nebraska [22] to test the intervention’s feasibility throughout the study’s duration, and for the usability testing portion of the study specifically described in this paper, we had community health workers (CHWs) from 3 areas in rural Nebraska leading the recruitment efforts. Based on their client list and the study’s eligibility criteria, the CHWs contacted potential participants by phone and explained the research objectives and informed consent to those who were interested in participating. We conducted the UTs in a public library and in a meeting room at the health department. The places were selected and reserved by our CHW partners based on our requirements for a quiet location where we could set up the recording materials in their communities. More detailed information about the HFC design and materials can be found in the protocol paper [22].

We conducted UTs with 5 participants. Data collection occurred in May 2022, involving participants from 2 health departments in rural Nebraska. We applied the think-aloud method as a guide for conducting the UT because it allows testing observers to understand and track participants’ thought processes while navigating 1 randomly selected intervention session. To start the UT (process described in Figure 2), the trained, bilingual facilitator explained the activity and asked for the participant’s narrative consent. After agreeing to participate, they completed a short questionnaire exploring demographics and health literacy screening information [25]. We explained the think-aloud procedure in their preferred language (English or Spanish) and asked the participants to narrate their actions while they were performing tasks and to share their perceptions about the materials [14]. The facilitator led each session using a script developed by the research team with the support of an expert in user experience, while another staff team member acted as a notetaker. In some cases, a CHW was present to assist with the session facilitation.

Each UT consisted of the following: we asked participants to perform 3 tasks, the first 2 related to the Workbook and the third involving an IVR call. Tasks included engaging with the IVR system, accessing the workbook, calculating BMI, and setting exercise goals for the family. The facilitator provided the workbook in Spanish and a smartphone (iOS operating system) for the participants to navigate and test the calls. Additionally, after each attempted task, we used the standardized posttask questionnaire called After-Scenario Questionnaire (ASQ) to measure user satisfaction [26,27]. At the end of the UT, participants answered the System Usability Scale (SUS) [27,28] to assess overall usability of the materials (Figure 2). Each session was video recorded on Zoom and transcribed using NVivo (Lumivero) software.

Composed of 3 items, the ASQ [26] was used after every task to evaluate participants’ satisfaction regarding the ease of completing the task, the amount of time it took, and the support information to complete it. It contains a Likert scale from 1 (strongly agree) to 7 (strongly disagree). In this way, a lower score indicated a higher level of satisfaction with usability [26]. By the end of the test, participants responded to the 10-item SUS [28], that evaluates usability considering aspects related to effectiveness, efficiency, and satisfaction. The Likert scale goes from 1 (strongly disagree) to 5 (strongly agree). The score for odd questions (Q1, Q3, Q5, Q7, and Q9) is the scale position minus 1. For even questions (Q2, Q4, Q6, Q8, and Q10), the score is 5 minus the scale point. The overall value of SUS was calculated by multiplying 2.5 by the sum of the scores [27,28], and the classification ranges were as follows: worst (0‐25), poor (25.01‐52.00), ok (52.01‐72.74), good (72.75‐85.57), and excellent (85.58 or more) [29,30]. We chose these 2 specific scales because they are objective, easy to understand, and reliable [27]. Both questionnaires are widely used by researchers when evaluating user experience [31].

We used the FRAME [21] as a base for the data integration in this mixed methods research. We compiled the findings emerged from the UT within the 8 aspects of FRAME: (1) when the modification occurs; (2) if the modifications were planned (proactive or reactive); (3) who participated in the decision to modify; (4) what is modified; (5) at what level of delivery the modification is made; (6) nature of the content modification; (7) fidelity-consistency of the modification; and (8) reasons for the modification [21]. The FRAME framework has its roots in implementation science, which is why we chose this tool to help translate the research findings into practice, tracking, and planning the necessary adaptations to the HFC materials evaluated.

To address the 2 objectives proposed in this mixed methods study, we organized the data in two categories for integration and analysis: (1) data from the standardized questionnaires (ASQ and SUS) and (2) data from participants’ comments (think-aloud procedure) and researchers’ notes (the facilitator and notetaker) during the UT. For the descriptive analysis, average and SD were used for numerical variables, and relative and absolute numbers for categorical variables were used for the demographic questionnaires and scores on the SUS and ASQ.

For more in-depth analysis of the think-aloud data, we transcribed the UT-recorded sessions using NVivo software and uploaded the researchers’ notes to build a robust dataset. The materials were coded based on the FRAME [21] framework, with the primary aim of tracking and planning the necessary adaptations to the IVR calls and workbook. The researchers created a Microsoft Excel sheet containing the main aspects of the framework, adding the time necessary to complete each task and the total time spent on the UT. Conducting the UT using a mixed methods approach allows for tracking and planning improvements based not only on the questionnaires’ results but also on analyzing participants’ body language, visible comfort or discomfort, and time spent on specific tasks.

This study was approved by the University of Nebraska Medical Center Institutional Review Board (#0745‐20-EP) and is available for consultation [32] (identifier: NCT04731506). Narrative informed consent was obtained for the study, and each individual had the right to refuse to participate, answer any questions, or stop the UT at any time. We offered the participants a US $25 gift card for time compensation. To maintain participants’ anonymity during translation, data analysis, and results description, we referred to them in our materials as “staff,” followed by a study identification number.

This UT mixed methods study’s primary objective was to evaluate the usability of IVR calls and a workbook of HFC, and to document and plan adaptations to these materials. The ASQ results revealed the ease of completing the task, satisfaction with the time required to complete the task, and satisfaction with the support information. The ASQ, a widely used instrument by researchers, has been published in studies to analyze the usability of applications [33-35] and a telerehabilitation system [36], a patient-centered website [37], and others. However, no studies have used ASQ to assess the usability of printed workbook materials and IVR focusing on overweight children by the time of this publication. The SUS showed good usability of the workbook and the IVR calls, primarily when it refers to participants’ ability to quickly learn how to use the system (Q7), respondents’ confidence in using the system (Q9), and whether the system is simple (Q2) and friendly (Q8). The SUS, like the ASQ, is widely used to evaluate the usability of a product. The SUS has been applied to analyze a social robot-based platform to prevent childhood obesity [38], a mobile app offering techno–health care services and assistance with adhering to a Mediterranean diet [39], and a risk estimation tool for obesity prevention [40]. However, the studies are not targeted at Hispanic/Latino children who live in rural areas of the United States.

Advancing to the FRAME coding results, it is clear what the nature and stage of the research are, considering that all adaptations are in the preimplementation phase, and the majority are related to content improvement—wording, figures, layout. When looking at the adaptation’s nature and goals, it is explicit that the challenge and importance of working with Hispanics/Latinos from different backgrounds are significant. Our participants included Latinos from 4 different countries living in rural Nebraska, and even among Spanish heritage, there are multiple cultural differences within communication and language preferences that impose a need for a more neutral accent and grammar in the intervention materials.

There are multiple dimensions to consider when working on cultural adaptation for Hispanics or Latinos, especially for public health interventions. Therefore, interventions that include cultural adaptation processes, such as the HFC, must go beyond translation. The materials need to incorporate cultural expressions, using language and images that recipients can relate to; they must be meaningful, consider the context and reach, and reflect the target population [41,42]. Therefore, cultural adaptation should be ongoing and regularly refined to address evolving needs and preferences within Hispanic/Latino communities [43].

Another interesting fact we noticed during data analysis and result description was the number of adaptations that emerged from participants (recipients) when compared to the CHWs and the researchers. Only 6.67% (2/30) of the suggestions were made by the participants, and they were all contextual, such as having in-person meetings every 2 months, thereby showing more concern about the intervention itself than about the material specifications. Based on this, we highlight the importance of choosing the think-aloud procedure [14] for this study, and having the UTs video recorded, and aggregating the researcher notes to NVivo as part of the dataset and coded based on FRAME [21].

Sometimes, in technological studies, the participants may not think something needs to be changed or improved because it is their first contact with that type of resource. However, for the research team, it is important to draw on all relevant background and experience with the topic and develop the skills to conduct an in-depth analysis of qualitative and quantitative outcomes. Also, it is important to consider that some populations, specifically minorities, may think their opinions are not important or they do not share them freely for other reasons and barriers imposed by different contexts [44-46]. Therefore, when working with studies such as cultural adaptations, inclusive language, or anything related to minorities and health disparities, it can be essential to conduct mixed methods approaches independently of the sample, since data integration can provide specific, in-depth, essential information for data analysis [47-49].

Of the 30 adaptations tracked by our team using FRAME, 7 were not addressed immediately in this phase of the study. One of them was not addressed because it was already part of the program, having the first meeting in person, and the others because they could affect fidelity, that is, consistency with the intervention’s core elements. It is important to note that these UTs are part of the HFC. Therefore, any future adaptations that involve changes in fidelity will be assessed after the analysis of the HFC feasibility study’s results and before planning the intervention scale-up for a broader audience.

The methodological innovation proposed by the research team in this study is noteworthy. In technology studies (especially usability and satisfaction assessments), it is new to aggregate standardized questionnaires (such as the ASQ and SUS) with frameworks from other disciplines, such as the FRAME framework that we chose for the adaptations report and coding. Our goal in the incorporation of FRAME into the UT analysis is to help facilitate translation into practice, since this framework’s roots are in the precepts of Implementation Science, which is essential when working with technological applications for public health interventions, thereby permitting us to have a broader vision of the process of adaptation and improving delivery and effectiveness.

This study assessed the usability of the workbook and IVR, but we did not test the effectiveness of the intervention materials at this point. The results of this study indicate that participants evaluated our research materials positively. Task-based tests were conducted in controlled research settings, a different context from using the materials in real life. This requires the research team to be open to ongoing feedback from participants.

Also, the participants are from 2 different counties in rural Nebraska; all are women, and only Spanish-literate individuals were eligible, conditions that do not represent the diversity of Hispanics/Latinos we intend to reach with the intervention in the future. Our exclusively female sample may also limit the generalizability of our findings. This may be attributed to women being more directly connected with our recruiters or having flexible schedules to participate in the research activities. Childhood obesity interventions such as these are family-based; therefore, understanding the roles and perspectives of fathers is crucial for long-term success.

By testing the usability of the workbook and IVR calls, this study addressed part of the gap related to the development of culturally relevant interventions for Hispanic/Latino families living in rural areas. Our results from the SUS and ASQ show that the usability of HFC materials, before adaptations, was rated mostly as good. For both resources evaluated (the workbook and IVR calls), participants demonstrated higher satisfaction with the level of ease of task completion. Of the 30 adaptations tracked using FRAME, we addressed 23, primarily related to content changes, such as wording, layout, and figure adjustments. By understanding the strengths and limitations of the intervention materials, we aimed to enhance their accessibility and cultural relevance, and possibly their future effectiveness when applied in a feasibility study. The UT results, combined with the systematic documentation using FRAME, allowed us to identify and prioritize the adaptations needed to improve the intervention materials. These refinements will prepare the HFC intervention for implementation in a subsequent feasibility study.