The study follows a participatory design approach that required 2 iterations of development followed by a scenario-based evaluation. This methodology allows the identification of bottlenecks, additional system failures, ease of learning the tool, and the comprehensibility of the language, among others [15]. In the evaluation methodology by scenarios, the main functionalities and tasks included in the mental health care model of the Department of Córdoba, Colombia, were included. The Department of Córdoba is located in the northwestern region of Colombia, covering an area of approximately 25,020 km² (9660 square miles) and home to around 2 million people as of 2025 [16]. This model includes a system designed to provide a service for the timely detection of mood disorders, contact with mental health care personnel, and support for referring cases to the network of health care providers. At the time of the evaluation, the system had been built through participatory design and had not been released to end users. The study had three phases: (1) test scenario design and creation phase, (2) test user recruitment phase, and (3) test implementation phase (Figure 1).

Functionalities and scenarios to be evaluated were selected based on the Knowledge Domain Ontology, following the GOMS (Goals, Objectives, Methods, Selection rules) methodology [11]. Two activities were carried out to construct the functional Domain Ontology (DO): (1) cocreation exercise with the project’s coresearchers (16 psychologists, biomedical engineers, and communicators), in which the contributions of each of them were compiled on a collaborative platform; and (2) visits to governmental actors to identify the specific needs of the territory. This information was synthesized in Microsoft Excel. As a result, 8 basic functionalities of the system, containing 60 specific tasks, were found: (1) Telecounseling: the possibility of interacting with expert mental health personnel for patients and caregivers. (2) Directory of institutions: identification of georeferenced institutions that are part of the program. (3) Hotline: access through multiple communication channels to emergency mental health care. (4) Forum: the possibility of exchanging experiences among users of the mental health program. (5) Questions: compilation and display of frequently asked questions for quick reference. (6) Self-reported questionnaires: the possibility of completing scales, surveys, and screening and diagnostic tools reported to expert mental health personnel. (7) Information capsules: educational content about mental health and its care. (8) Population dashboard: display of active patients in the mental health program.

DO allows for prioritization of functionalities to be implemented through the frequency analysis of the potential users’ participation in the tasks identified in the functional analysis described in the section “Scenario Preparation and Creation Phase.” The DO identified main user groups: (1) Patients: individuals interested in receiving mental health services. Some of them formally join the program, which requires full identification of the individuals. These users have control over access to their own information. (2) Caregivers: individuals interested in getting mental health services for their care recipients. They may or may not be identified, and some formally join the program, which requires full identification of the individuals. Access to patient information is limited. (3) Clinical staff: personnel trained at different levels of education to support the diagnosis, treatment, rehabilitation, and monitoring of individuals with mental health conditions. It ranges from community health workers to highly specialized personnel. (4) Administrative personnel: staff responsible for coordinating the service provision and performing administrative activities such as monitoring, billing, scheduling services, and managing demand, among others. (5) Departmental Health Secretariat: the governmental body responsible for directing, coordinating, and administering public health actions across the department, including the oversight and integration of a network of public and private health care service providers.

For the purposes of the study, the patient and caregiver groups were merged into a patient group since caregivers’ tasks constituted a subset of patient tasks. Although the Departmental Health Secretariat was initially identified as a key stakeholder, it was not possible to include their participation in the study, as no officials were assigned to collaborate with the research activities.

The sample selection followed a convenience sampling. Potential users were recruited in 2 phases (September and October 2023) by the teaching team of the Faculties of Psychology of Universidad Pontificia Bolivariana at their sites in Medellín, Montería, and Palmira. The 3 types of roles (patient, clinical staff, and administrative personnel) with the greatest participation in the use of the mental telehealth system were included.

The inclusion criteria for the different types of participants were as follows: (1) Patients and relatives of patients older than 18 years (legal age in Colombia). (2) Mental health care personnel who were psychology or social work students in semesters of clinical internship (ninth semester of the degree or greater) or professionals in psychology or social work. (3) Administrative or logistical personnel with responsibilities in managing the population health program.

Potential users who did not have basic computer skills or who required support from others were excluded. After the initial call by the teaching team, the research team contacted the test participants to explain the evaluation procedure, and the synchronous testing spaces were coordinated through the Microsoft Teams platform. Participation was voluntary with verbal consent. The only data collected from individuals were those associated with the features of the type of user (age, gender, level of education, and occupation), excluding data that would allow their identification. The sessions were recorded on the Teams platform in case additional analysis was needed, and access was restricted to the research group.

The scenario-based evaluation methodology was used, accompanied by cognitive walk-throughs observations, in which the user expressed his or her actions, thoughts, and interpretations of the tool aloud [17,18]. The usability tests were conducted individually, and the permissions and access to the respective screens and interfaces were verified at the beginning of each test. Moreover, a document with instructions and proposed tasks to be performed was shared with the users.

To understand different sources of tasks performance in the solution and provide detailed feedback on the need to improve the interface, we conducted task analysis, followed by usability analysis. After each scenario, each user filled out a Google Forms survey containing a questionary for task load measurement (the NASA [National Aeronautics and Space Administration] Task Load Index [NASA-TLX]) [19]. At the end of all the assigned scenarios, users filled out the Post-Study System Usability Questionnaire (PSSUQ) [20].

This study was reviewed and approved by the health research ethics committee of Universidad Pontificia Bolivariana, Medellín, Colombia (Minute 09 of 2022). All participants were informed of the objectives and procedures of the research prior to their involvement, and verbal informed consent was obtained before initiating the usability testing sessions. Participation was voluntary. To ensure privacy and confidentiality, no personally identifiable information was collected. Only general demographic characteristics (age, gender, level of education, and occupation) were recorded. Usability sessions were conducted via Microsoft Teams and recorded exclusively for research purposes, with access restricted to the research team. All data were anonymized prior to analysis. Participants did not receive any financial or in-kind compensation for their involvement. No identifiable images of participants are included in this manuscript or in supplementary materials.

The summary of the scenarios created is shown in Table 1. Patients, clinical staff, and system administrators were identified as potential users. As shown, patients are the ones with the most types of interactions anticipated in the system.

A total of 30 individual users were recruited (14 patients, 8 people with a mental health care profile, and 8 people with administrative or logistical activities in the mental health care process). Some of the individuals participated in both evaluation cycles (Table 2).

The evaluations were conducted via Microsoft Teams, where users shared their screen. The tests per user lasted approximately 45-90 minutes. Table 3 contains the consolidated number of participants by cycle and their respective roles in the evaluated system.

A total of 6 test groups were conducted (3 types of users in 2 cycles), including the 4 functionalities with the highest scores in the prioritization and impact process: login, providing mental health care and monitoring (telecounseling), assistance to clinical staff in mental health emergencies (hotline), and population management (population dashboard).

All events were reported by the researcher (EPM) conducting the test. During the evaluations, 23 critical findings were identified that informed system improvements—9 were system errors and 14 were design errors. System errors refer to malfunctions in the application’s functionality, whereas design errors were limited to issues within the user interface. Figure 2 provides an example of interface modifications based on these findings.

This study evaluated the usability of a clinical telehealth system for mood disorder care developed through participatory design. The findings show that the system achieved usability scores (PSSUQ 2.2 and 2.3) and improvements in workload between evaluation cycles, particularly for patient-centered tasks, meeting the study’s objective of assessing usability through a scenario-based methodology.

One of the challenges in systems evaluation is the granularity of evaluation at the task level and the order in which such evaluation is performed. The scenario-based evaluation approach allowed planning an evaluation strategy informed by an objective prioritization of functions and tasks. In addition, a separation of user groups was achieved, and common tasks that did not require additional users for validation were identified.

One of the challenges of system evaluation is the granularity of evaluation at the task level and the order in which such evaluation is performed. The scenario-based evaluation approach allowed planning an evaluation strategy informed by an objective prioritization of functions and tasks. In addition, a separation of user groups was achieved, and common tasks that did not require additional users for validation were identified.

The system can be considered to have good usability, given an overall PSSUQ score between 2.2 and 2.3 in cycles 1 and 2, considering that the lower the score, the better the performance. The reference value is 2.8 [20]. These findings suggest that the participatory design process helped anticipate user needs and improve the quality of the product before deployment However, it was also possible to identify elements that previous heuristic evaluations were unable to detect. This confirms how user participation in the design and early usability evaluation contribute to good system usability and help identify opportunities for system improvement before the system goes into production, optimizing the use of development resources. This is consistent with findings from previous studies showing that iterative and participatory usability evaluations with both clinicians and end users can significantly improve system navigation, content structure, and user satisfaction prior to system deployment [22]. This reinforces the importance of iterative usability evaluations early in the system development life cycle to ensure alignment with user needs and task requirements [23].

It is interesting to note how system usability scores were better for users with more functional and task representation. Patients rated the system higher than administrators. This can be explained by the prioritization in the development of interfaces informed by the DO, where development was carried out according to task prioritization. It should also be noted that the differences observed between groups can be attributed to the different types of tasks assigned to each group. Tasks prioritized for patients tended to be simpler and more user-centered, whereas tasks assigned to administrative and clinical users were more complex, involving greater cognitive demands. The differences in scores between cycles 1 and 2 cannot be established as statistically significant given the sample size. However, changes may be due to individual variability. A possible future evaluation method to address this could involve repeated measurements with the same users or increasing the number of evaluations. The number of evaluations may be limited in scenarios where the number of potential users is insufficient. Optional usability evaluations may be implemented in the future for all individuals registered in the mental health program.

When analyzing the sources of task workload, there is an improvement in the NASA-TLX workload index for the vast majority of tasks performed in the second version of the system, except for the forum modeling and population management tasks (associated with clinical users and system administrators). This can be explained by the higher priority given to system changes benefiting patient and family-type users and by the addition of interface elements that had not been subject to group discussion. The most significant source of workload is mental workload, as expected. Future research could focus on improving the information available on-screen to reduce cognitive demands. As anticipated, physical workload is minimal since the interaction does not require extensive clicking.

This study has some limitations. The convenience sample size restricts the generalizability of findings, and the number of evaluations was insufficient to establish statistically significant differences between user groups. In addition, the evaluation of the synchronous interactive telehealth component was not carried out, as this would require a different experimental design. These limitations are consistent with findings from systematic reviews, which report that usability evaluations in mobile mental health interventions often rely on small sample sizes and primarily summative assessment methods [24]. In the future, the mobile interface of the mental health system will be evaluated, as it is considered a highly useful means of accessing and using the mental health system for end users (patients and relatives).

This study shows that combining participatory design with scenario-based usability evaluation enables early identification of usability challenges and iterative refinement of digital health systems. Beyond the specific case of Córdoba, Colombia, these findings have broader implications: engaging end users early, prioritizing patient-facing functionalities, and conducting iterative usability assessments can enhance adoption, optimize resources, and support sustainable models of mental health care in underserved regions.