Pediatric to adult health care transitions are a health promotion priority for youth with special health care needs in the United States [1]. Youths with special health care needs have chronic physical, developmental, behavioral, or emotional conditions that require health care and other services beyond typical use [2]. These youths need a higher level of well-coordinated health care, often across multiple providers and specialty types, to realize optimal health and wellness [3,4]. Common conditions among youth with special health care needs include a range of diagnoses that necessitate an even broader range of types and intensities of health care transition support. Diabetes, sickle cell disease, autism, and spina bifida are examples of the diversity of diagnoses among youth with special health care needs. They may need differing care transition support for a variety of aspects of their health care, such as medication management, health status monitoring, physical therapies, and behavioral therapies.

An estimated 5 million youths with special health care needs in the United States are between 12 and 17 years old—the transition age, during which health care transition planning should begin [5]. Seamlessly moving from pediatric to adult health care is a critical aspect of transitioning from childhood to adulthood for youth with special health care needs, and structured pediatric to adult care transition programs are associated with positive outcomes [6-8]. Without appropriate health care transition services, youths with special health care needs are vulnerable to adverse outcomes, such as loss of continuity of care [9,10], preventable morbidity, emergency department visits, and hospital visits [11,12].

Pediatric to adult health care transitions should be characterized by having an organizational policy in place that supports care transitions, developing tracking and monitoring processes that follow youths with special health care needs as they transition, assessing youth transition readiness, and transferring care [13]. According to the care transitions consensus statement by the American Academy of Pediatrics, the American Academy of Family Physicians, and the American College of Physicians, providers who care for youth with special health care needs should “(1) understand the rationale for transition from child-oriented to adult-oriented care; (2) have the knowledge and skills to facilitate that process; and (3) know if, how, and when transfer of care is indicated” [14]. It is also vitally important for health care institutions to have organizational policies that support care transitions at the pediatric and adult care levels. The adult team is crucial in supporting care engagement post transfer of care, but much of the emphasis is placed on the pediatric team.

Most youths with special health care needs do not receive adequate care transition services. Across the United States, only 17% of youths with special health care needs received appropriate care transitioning support [15], and access to health services decreased as the youths aged into adulthood [16]. Lack of adequate care transition support may be exacerbated by barriers to care. Although many barriers to care transitions exist for youth with special health care needs, providers also experience barriers to providing care transition services. These barriers can include lack of communication and coordination between the pediatric and adult sides of care, the perceived difficulty of working with youth with special health care needs, and provider hesitation to be involved in the care transition process [17,18]. Provider hesitation may be because of discomfort in providing care transition services, perceived lack of knowledge, lack of training on providing care for youth with special health care needs, and lack of resources (time and low or no reimbursement) associated with providing care transition services [17,19-22].

To assist in addressing issues related to lack of information, communication, and limited resources surrounding care transitions for youth with special health care needs, we tested the functionality and usability of an internet communication technology developed to assist health care providers in transitioning youths from pediatric to adult care. Funded by the Texas Department of State Health Services under the auspices of the federal Maternal and Child Health Bureau, this particular internet communication technology, the Texas Transition Toolkit [23], is an eHealth initiative providing three main services: a one-stop shop to research literature on transition care, a catalog of relevant tools to assist providers in creating care transition plans, and steps for how to develop and communicate a transition program. This tool is a state-level initiative that complements the federal Got Transition program [24] and is in line with the Health Resources and Services Administration Maternal and Child Health Bureau Title V Services Block Grant Program’s transition performance measure, specifically advocating for increasing the number of youths receiving appropriate care transition services [25].

Internet communication technologies such as this tool have the potential to provide rapid access to health and medical knowledge. Under one of the many definitions of eHealth, these health care system–oriented applications can improve access to quality health care at lower costs. Through eHealth applications, health-related information and knowledge can reach rural, isolated, low-population, or impoverished communities with very little marginal cost to the community or health service provider [26]. As these benefits of eHealth internet communication technology are mitigated by the quality of the application, it is important to evaluate the functionality and usability of eHealth internet communication technology with end users. Functionality assesses whether the technology works the way it is intended to work and delivers the expected results, whereas usability assesses users’ reactions to and interactions with the technology (including perceived utility and satisfaction) [27].

This paper describes the formative evaluation of the functionality and usability of this eHealth internet communication tool using a mixed methods usability test. As pragmatic researchers, we determined our research method through our research question: do representative users find the tool functional and usable? [28]. The project was exempted from the institutional review board of the Texas A&M University.

The usability test used a triangulated mixed methods case study approach consisting of a concurrent think-aloud phase, a satisfaction survey, and a survey of problem relevance and task performance to operationalize and capture functionality and usability [29,30]. The triangulation mixed methods design uses quantitative and qualitative methods to study an issue [29]. Triangulation considers the qualitative and quantitative aspects of a single study separately during the discovery and analyses stages but integrates the findings of each paradigm to develop new knowledge [31]. This approach examines multiple cases, in this case, multiple representative users, to more fully understand the phenomenon of interest—the functionality and usability of the tool [32]. The evaluation of the functionality and usability of the tool involved assessing for two aspects of the experiences of representative users: a qualitative exploration of problems encountered and a quantitative assessment of satisfaction. To confirm the outcomes of the usability test and further measure the functionality of the tool, the research team explored the relevance of the detected problems and representative users’ task performance. If the number and type of tasks and associated quality checks successfully completed or failed aligns with the concurrent think-aloud and the satisfaction analyses, it supports both the robustness of the results and indicates functionality and usability issues for representative users. Similarly, the presence of high-relevance problems in the concurrent think-aloud and satisfaction analyses adds further support to the robustness of the results in detecting important problems and where the functionality and usability of the tool possibly fails. Relevance, in this context, is how connected the type of problem is to the functionality and usability of the tool. It consists of the likelihood of the problem occurring combined with the impact the problem has on the functionality and usability of the tool for representative users, should the problem occur.

Two groups of participants were recruited for the usability test. One group consisted of representative users—providers working with youths with special health care needs as they transitioned from pediatric to adult care. This group participated in the concurrent think-aloud phase of the usability test and completed the satisfaction survey, thus providing information on problems encountered when using the tool and end user satisfaction with the tool. The second group consisted of website design professionals with experience in designing informational websites for governmental organizations. This group provided information on the relevance of the problems encountered by the representative users.

On the basis of the criterion sampling, 10 representative users were recruited from a medical home in Texas [33]. As the think-aloud method provides a rich source of data, a small sample of subjects generally suffice to discover individual (rather than population) knowledge and experiences [34]. The medical home was an opportunity sample that met the usability trial criterion established a priori, that is, the medical home expressed an interest in a care transition program for youth with special health care needs, had the ability to implement a care transition program, provided care for at least 25 youths with special health care needs, and was able to suggest a champion of care transitions. Through this champion, we were able to recruit a full sample of representative users during a professional conference on care transitions held at the medical home. To be eligible to participate as representative users, the providers must have experience working with youth with special health care needs on the pediatric or adult side of care.

Five website design experts from a research organization affiliated with higher education and a robust history of government website design were recruited to provide expert assessment of the relevance of each problem encountered by representative users during the concurrent think-aloud phase of the usability test. The experts were recruited via snowball recruitment, starting with a government website design consultant familiar with internet communication technology. These experts included software application developers (n=2), web application developers (n=2), and a web and information designer (n=1) with experience in the field ranging from 11 to 17 years.

Once the usability test was completed, the research team transcribed the verbalizations, abstracted data from the surveys, and charted the participants’ navigation through the tool. The functionality and usability of the tool were understood through a thematic analysis of the concurrent think-aloud data using NVivo 12 (QSR International) [39], and descriptive analyses of the Website Evaluation Questionnaire, task performance, and relevance data using Microsoft Excel [40] and Stata 15 (StataCorp) [41].

Given the small sample size and potential for identification, demographic data of representative users were fuzzed. Approximately 90% of the representative users who participated in the usability test were White, non-Hispanic women. Every user worked with at least one youth with special health care needs, and approximately 90% had a professional caseload where more than half of the patients were youths with special health care needs. More than one-third of the users worked solely with youths with special health care needs. Almost all (approximately 80%) of the representative users transitioned youths from pediatric to adult care. Half of the users worked with youths with special health care needs on both ends of the continuum—pediatrics and adult care. Approximately one-fourth of the users were physicians and approximately another quarter were nurse practitioners. The remaining representative users were social workers (approximately one-fourth), nurses, and transition specialists (clinical team coordinators, youth service specialists, and transition coaches). Some users served multiple roles (eg, nurses and certified educators for a particular chronic condition).

Table 1 displays the number of problems detected, the different types of problems detected, and the frequency of each type of problem. A total of 31 problems were detected through the analysis of the concurrent think-aloud transcripts. These problems were aggregated into 10 categories or type of problem, most of which (29/31, 93%) focused on issues associated with finding information on health care transitions, such as the utility of search criteria, finding the search bar, finding disease-specific resources, and finding resources blocked by a firewire. Among these problems, approximately one-fourth (8/31, 26%) centered on the utility of search criteria, meaning representative users found it difficult to create search terms that would return resources on a particular topic in health care transitions:

In total, 19% (6/31) of the problems resulted from the difficulty in finding the search bar. The representative users found the placement of the search bar confusing to the point that some thought the search bar was missing:

Approximately 10% (3/31) of the problems were associated with users encountering difficulties finding disease-specific resources:

When attempting to access resources behind a firewall, users encountered a number of problems, including blocked windows, software application openings, and an inability to access the contact information of the website manager:

The overall satisfaction score for the tool was 81.92% (1065/1300), indicating high satisfaction based on the overall satisfaction cutoff score of 73.33% (953.4/1300). The tool scored highest in relevance (138/150, 92%), followed by comprehension and layout (both 131/150, 87.3%), and the lowest in search options (114/150, 76%) and ease of use (114/150, 76%; Table 2 [35]). Satisfaction was high for each dimension.

Table 3 shows task performance. This is the number of tasks and quality checks successfully completed. Overall, 89% (89/100) of all the tasks and 75% (60/80) of the quality checks were completed successfully. Participants were least likely to find a specific tool for early adolescence, with a 70% (7/10) completion and a 70% (7/10) quality check success rate.

Table 4 provides the likelihood, impact, and relevance scores of each problem type. Focusing on relevance, every problem type was found to be relevant based on a cutoff score of 11.7 for relevance, and 60% (6/10) of the problems were found to be highly relevant to the functioning of the tool based on a cutoff score of 18.4 for highly relevant. Problems that interrupted the ability of end users to locate information because of the layout or mechanical workings of the tool were rated as having the highest relevance to the functioning of the tool. These involved the tags programmed on search terms (trouble finding search criteria that result in what user wants: relevance=18.4), organization of information (hard to find articles by disease: relevance=21.5), and inability to access information (email of website contact opens email software: relevance=20.4; need clarity on who to contact for article behind a firewall: relevance=21; some windows blocked by a firewall or files will not open: relevance=22).

The functionality and usability testing of the tool with representative users highlighted the usefulness of this eHealth internet communication technology among providers working with youth with special health care needs. Representative users found the tool to be highly relevant, as demonstrated by the satisfaction score for the dimension relevance (138/150, 92.0%). According to the users, the tool provided comprehensive information related to health care transitions for youth with special health care needs, with a satisfaction score of 87.3% (131/150) on the comprehensive dimension. This is important, as it suggests that the tool may help bridge the gaps in training on health care transitions for youth with special health care needs in US medical education [44]. Gaps in education on health care transitions exist at all levels of medical education [44], and providers often express concerns about the lack of knowledge on transitioning care for youth with special health care needs [45,46]. Furthermore, youths transitioning from pediatric to adult care also want access to medical education on health care transitions [47], which can be provided through the tool. There is also a need to provide information on health care transitions among youth with special health care needs to adult providers, who may lack knowledge of the process [48]. An eHealth internet communication technology such as the tool, which includes evidence-based literature on health care transitions and templates for transition protocols, can be used to create a structured tool to help improve collaboration across pediatric and adult care providers and continuity of care [49]. The tool does not rely on the ability of a health care organization to integrate the tool into their system; it can be implemented within an organization or external to an organization. Although integration within the systems of care is ideal [50], in other studies of educational transition tools, health care providers expressed concern over the ability to integrate tools, given the characteristics and deficits of health care systems [51].

Although the tool was well received by representative users, the usability test results identified areas of concern regarding functionality and usability. Users reported the most difficulty in two areas of functionality and usability: search and navigation. This was reflected in both the concurrent think-aloud and satisfaction survey results and was supported by the task performance and relevance analyses. Representative users reported lower satisfaction with search and navigation dimensions, relatively high number of search- and navigation-related problems and low task completion for tasks involving finding tools that require searching and navigation. The problems identified around the search and navigation functionality were also found to be relevant by web design experts. Each of these areas of analyses triangulates on search and navigation issues, suggesting the robustness of the results and allowing researchers to fine-tune the tool to optimize performance.

Uncovering these inefficiencies and subsequently optimizing the tool improves its functionality and usability, which may drive the future use of the technology for its specified purpose [52]. Studies of commercial technologies show that end users are more likely to utilize a technology if it is easy to use [53-55]. This is especially important for eHealth internet communication technology, such as the tool, which is intended to improve health outcomes in the long term through the dissemination of information and best practices. Although the results from the usability test provide a clear road map for optimizing the tool, they also highlight the importance of evaluating eHealth and internet communication technology with representative users. Often, internet communication technology evaluations focus too intensely on the technical aspects of the technology rather than the needs and expectations of end users [56]. Conducting functionality and usability assessments before widespread implementation is essential for forecasting usage and ensuring that internet communication technology has the intended impact [57].

Although the concurrent think-aloud methodology and the satisfaction survey provided a robust way to ascertain the functionality and usability of this eHealth internet communication technology, there are certainly other acceptable methods. In fact, even with the growing interest in website functionality and usability and the growing use of internet communication technology for eHealth applications, there is no consensus on the definition of usability. There exists a multitude of possible dimensions and measurement techniques in the field of internet communication technology research. Choosing between these dimensions and techniques to evaluate the technology involves a certain amount of bias, which could result in the website performing better in the laboratory than in a real-life setting. However, this usability test followed quality measures for usability studies of eHealth applications [58]. We used a valid and reliable tool, chose our study design based on the objectives of the study, used across-method triangulation, and included both representative users and experts in assessing usability.

Few evaluated measures exist that can be used to understand the impact of eHealth technology on health outcomes [59], which has potentially contributed to the limited evaluation of eHealth [60]. This is a particularly important limitation when the website is eHealth internet communication technology and is expected to affect health outcomes. Thus, further research is needed on the effect of the tool on care transition outcomes, particularly in isolated or resource-poor communities. Evaluation research on holistic care transition tools and programs is limited, with the majority of outcomes-focused evaluations targeting narrow, disease-specific populations of youth with special health care needs [6]. Although disease-specific instruments fall under the triple aim domain of population health [61], considering that most youths with special health care needs experience multiple comorbid conditions across their lifetime, more empirical evidence is needed on tools dedicated to broader care transitions. This tool, with its broad focus, answers calls to assist care providers in transitioning youth with special health care needs using a biopsychosocial model [62].