This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2226) Human Factors and Usability Case Studies (748) User Needs and Competencies (274) Usability Evaluation Case Studies (182) Developmental Problems (39) eHealth Literacy / Digital Literacy (358) User-centred Design Case Studies (564)

Published on in Vol 5, No 5 (2021): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/22915, first published .
A Provider-Facing eHealth Tool for Transitioning Youth With Special Health Care Needs From Pediatric to Adult Care: Mixed Methods, User-Engaged Usability Study

A Provider-Facing eHealth Tool for Transitioning Youth With Special Health Care Needs From Pediatric to Adult Care: Mixed Methods, User-Engaged Usability Study

A Provider-Facing eHealth Tool for Transitioning Youth With Special Health Care Needs From Pediatric to Adult Care: Mixed Methods, User-Engaged Usability Study

Authors of this article:

Darcy Jones McMaughan1, 2 Author Orcid Image ;   Sherry Lin1 Author Orcid Image ;   Jennifer Ozmetin1 Author Orcid Image ;   Judith Gayle Beverly1 Author Orcid Image ;   Joshua Brog1 Author Orcid Image ;   Emily Naiser3 Author Orcid Image
Article Authors Cited by (2) Tweetations (3) Metrics

    Original Paper

    1Department of Health Policy and Managment, School of Public Health, Texas A&M University, College Station, TX, United States

    2Oklahoma State University, Stillwater, OK, United States

    3Public Policy Research Institute, Texas A&M University, College Station, TX, United States

    Corresponding Author:

    Darcy Jones McMaughan, MA, PhD

    Oklahoma State University

    423 Willard

    Stillwater, OK

    United States

    Phone: 1 979 204 3983

    Email: darcy.mcmaughan@okstate.edu


    Background: There is a need for medical education on health care transitions for youth with special health care needs. The Texas Transition Toolkit (the tool) supports providers through a one-stop shop for researching literature on care transitions, a catalog of care transition tools, and guides for developing care transition programs.

    Objective: This study aims to assess the functionality and usability of the tool with providers working with transition-aged children and youth with special health care needs (representative users).

    Methods: The tool was evaluated using a triangulated mixed methods case study approach consisting of a concurrent think-aloud phase, a satisfaction survey, and a survey of problem relevance and task performance to operationalize and capture functionality and usability. Our mixed methods deep dive into the functionality and usability of the tool focused on 10 representative users from one medical home in Texas and 5 website design experts.

    Results: Representative users found the tool to be highly relevant, as demonstrated by the satisfaction score for relevance (138/150, 92%). According to the users, the tool provided comprehensive information related to health care transitions for youth with special health care needs, with a satisfaction score of 87.3% (131/150) for comprehensive. Overall satisfaction with the tool was high at 81.92% (1065/1300) with a cutoff score of 73.33% (953.4/1300) indicating high satisfaction, but users reported relatively lower satisfaction with search (114/150, 76%) and navigation (ease of use: 114/150, 76%; hyperlinks: 163/200, 81.5%; structure: 159/200, 79.5%). They experienced search- and navigation-related problems (total problems detected: 21/31, 68%) and, based on quality checks, had a relatively low task completion rate for tasks involving finding information (60/80, 75%), which required searching and navigation. The problems identified around search and navigation functionality were relevant (relevance scores ranging from 14.5 to 22, with a cutoff score of 11.7 indicating relevance).

    Conclusions: The tool may help bridge the gaps in training on health care transitions for youth with special health care needs in US medical education. The tool can be used to create structured protocols to help improve provider knowledge, collaboration across pediatric and adult care providers, and the continuity of care as youth with special health care needs transition from pediatric to adult care. The results provided a road map for optimizing the tool and highlighted the importance of evaluating eHealth technologies with representative users.

    JMIR Form Res 2021;5(5):e22915

    doi:10.2196/22915

    Keywords

    youth with special health care needs; health care transitions; eHealth; usability; concurrent think aloud method 


    Background

    Pediatric to adult health care transitions are a health promotion priority for youth with special health care needs in the United States [1]. Youths with special health care needs have chronic physical, developmental, behavioral, or emotional conditions that require health care and other services beyond typical use [2]. These youths need a higher level of well-coordinated health care, often across multiple providers and specialty types, to realize optimal health and wellness [3,4]. Common conditions among youth with special health care needs include a range of diagnoses that necessitate an even broader range of types and intensities of health care transition support. Diabetes, sickle cell disease, autism, and spina bifida are examples of the diversity of diagnoses among youth with special health care needs. They may need differing care transition support for a variety of aspects of their health care, such as medication management, health status monitoring, physical therapies, and behavioral therapies.

    An estimated 5 million youths with special health care needs in the United States are between 12 and 17 years old—the transition age, during which health care transition planning should begin [5]. Seamlessly moving from pediatric to adult health care is a critical aspect of transitioning from childhood to adulthood for youth with special health care needs, and structured pediatric to adult care transition programs are associated with positive outcomes [6-8]. Without appropriate health care transition services, youths with special health care needs are vulnerable to adverse outcomes, such as loss of continuity of care [9,10], preventable morbidity, emergency department visits, and hospital visits [11,12].

    Pediatric to adult health care transitions should be characterized by having an organizational policy in place that supports care transitions, developing tracking and monitoring processes that follow youths with special health care needs as they transition, assessing youth transition readiness, and transferring care [13]. According to the care transitions consensus statement by the American Academy of Pediatrics, the American Academy of Family Physicians, and the American College of Physicians, providers who care for youth with special health care needs should “(1) understand the rationale for transition from child-oriented to adult-oriented care; (2) have the knowledge and skills to facilitate that process; and (3) know if, how, and when transfer of care is indicated” [14]. It is also vitally important for health care institutions to have organizational policies that support care transitions at the pediatric and adult care levels. The adult team is crucial in supporting care engagement post transfer of care, but much of the emphasis is placed on the pediatric team.

    Most youths with special health care needs do not receive adequate care transition services. Across the United States, only 17% of youths with special health care needs received appropriate care transitioning support [15], and access to health services decreased as the youths aged into adulthood [16]. Lack of adequate care transition support may be exacerbated by barriers to care. Although many barriers to care transitions exist for youth with special health care needs, providers also experience barriers to providing care transition services. These barriers can include lack of communication and coordination between the pediatric and adult sides of care, the perceived difficulty of working with youth with special health care needs, and provider hesitation to be involved in the care transition process [17,18]. Provider hesitation may be because of discomfort in providing care transition services, perceived lack of knowledge, lack of training on providing care for youth with special health care needs, and lack of resources (time and low or no reimbursement) associated with providing care transition services [17,19-22].

    To assist in addressing issues related to lack of information, communication, and limited resources surrounding care transitions for youth with special health care needs, we tested the functionality and usability of an internet communication technology developed to assist health care providers in transitioning youths from pediatric to adult care. Funded by the Texas Department of State Health Services under the auspices of the federal Maternal and Child Health Bureau, this particular internet communication technology, the Texas Transition Toolkit [23], is an eHealth initiative providing three main services: a one-stop shop to research literature on transition care, a catalog of relevant tools to assist providers in creating care transition plans, and steps for how to develop and communicate a transition program. This tool is a state-level initiative that complements the federal Got Transition program [24] and is in line with the Health Resources and Services Administration Maternal and Child Health Bureau Title V Services Block Grant Program’s transition performance measure, specifically advocating for increasing the number of youths receiving appropriate care transition services [25].

    Internet communication technologies such as this tool have the potential to provide rapid access to health and medical knowledge. Under one of the many definitions of eHealth, these health care system–oriented applications can improve access to quality health care at lower costs. Through eHealth applications, health-related information and knowledge can reach rural, isolated, low-population, or impoverished communities with very little marginal cost to the community or health service provider [26]. As these benefits of eHealth internet communication technology are mitigated by the quality of the application, it is important to evaluate the functionality and usability of eHealth internet communication technology with end users. Functionality assesses whether the technology works the way it is intended to work and delivers the expected results, whereas usability assesses users’ reactions to and interactions with the technology (including perceived utility and satisfaction) [27].

    Objective

    This paper describes the formative evaluation of the functionality and usability of this eHealth internet communication tool using a mixed methods usability test. As pragmatic researchers, we determined our research method through our research question: do representative users find the tool functional and usable? [28]. The project was exempted from the institutional review board of the Texas A&M University.


    Study Design

    The usability test used a triangulated mixed methods case study approach consisting of a concurrent think-aloud phase, a satisfaction survey, and a survey of problem relevance and task performance to operationalize and capture functionality and usability [29,30]. The triangulation mixed methods design uses quantitative and qualitative methods to study an issue [29]. Triangulation considers the qualitative and quantitative aspects of a single study separately during the discovery and analyses stages but integrates the findings of each paradigm to develop new knowledge [31]. This approach examines multiple cases, in this case, multiple representative users, to more fully understand the phenomenon of interest—the functionality and usability of the tool [32]. The evaluation of the functionality and usability of the tool involved assessing for two aspects of the experiences of representative users: a qualitative exploration of problems encountered and a quantitative assessment of satisfaction. To confirm the outcomes of the usability test and further measure the functionality of the tool, the research team explored the relevance of the detected problems and representative users’ task performance. If the number and type of tasks and associated quality checks successfully completed or failed aligns with the concurrent think-aloud and the satisfaction analyses, it supports both the robustness of the results and indicates functionality and usability issues for representative users. Similarly, the presence of high-relevance problems in the concurrent think-aloud and satisfaction analyses adds further support to the robustness of the results in detecting important problems and where the functionality and usability of the tool possibly fails. Relevance, in this context, is how connected the type of problem is to the functionality and usability of the tool. It consists of the likelihood of the problem occurring combined with the impact the problem has on the functionality and usability of the tool for representative users, should the problem occur.

    Participants

    Two groups of participants were recruited for the usability test. One group consisted of representative users—providers working with youths with special health care needs as they transitioned from pediatric to adult care. This group participated in the concurrent think-aloud phase of the usability test and completed the satisfaction survey, thus providing information on problems encountered when using the tool and end user satisfaction with the tool. The second group consisted of website design professionals with experience in designing informational websites for governmental organizations. This group provided information on the relevance of the problems encountered by the representative users.

    On the basis of the criterion sampling, 10 representative users were recruited from a medical home in Texas [33]. As the think-aloud method provides a rich source of data, a small sample of subjects generally suffice to discover individual (rather than population) knowledge and experiences [34]. The medical home was an opportunity sample that met the usability trial criterion established a priori, that is, the medical home expressed an interest in a care transition program for youth with special health care needs, had the ability to implement a care transition program, provided care for at least 25 youths with special health care needs, and was able to suggest a champion of care transitions. Through this champion, we were able to recruit a full sample of representative users during a professional conference on care transitions held at the medical home. To be eligible to participate as representative users, the providers must have experience working with youth with special health care needs on the pediatric or adult side of care.

    Five website design experts from a research organization affiliated with higher education and a robust history of government website design were recruited to provide expert assessment of the relevance of each problem encountered by representative users during the concurrent think-aloud phase of the usability test. The experts were recruited via snowball recruitment, starting with a government website design consultant familiar with internet communication technology. These experts included software application developers (n=2), web application developers (n=2), and a web and information designer (n=1) with experience in the field ranging from 11 to 17 years.

    Qualitative and Quantitative Assessments

    Qualitative: Exploring Problems Encountered Through the Concurrent Think-Aloud Method

    eHealth internet communication tools can be evaluated using a variety of techniques; most originate from the fields of human-computer interaction and media design, which can be broadly grouped into two modes: expert-focused methods and user-focused methods. One example of a user-focused method is the think-aloud usability testing method [35]. This approach, which originates in the evaluation of physical tasks and then used primarily for software evaluations, is also frequently used for evaluating websites [36,37]. A form of the think-aloud approach, the concurrent think-aloud method, is an evaluation method that involves representative users completing tasks using the tool and simultaneously verbalizing their thoughts. During the recorded sessions, representative users completed tasks according to a predetermined scenario (a vignette) while verbalizing their experiences. Analyses of these verbal reports provided detailed insights into the functionality and usability problems encountered by representative users.

    To evaluate the functionality and usability of the tool using the concurrent think-aloud method, the research team developed a set of tasks related to the intended purpose of the tool. For example, representative users were asked to find a peer-reviewed article focusing on care transitions for youth with type 1 diabetes and to denote the completion of the task (yes or no). All tasks could be carried out independently to minimize dependency bias and reduce the risk that users would stall after one or two tasks. When applicable, a task contained an associated quality check. The quality check served to determine if the user actually completed the task successfully, in addition to perceived successful completion. These quality checks consisted of asking representative users to write down and speak aloud a few words from the completed subtasks—for example, when asked to find a peer-reviewed article on care transitions for youth with type 1 diabetes, users were subsequently asked to write and speak aloud the first three words of the article title. The tasks were introduced by a vignette, which explained the context and provided the details necessary to perform the tasks. The vignette described a youth with type 1 diabetes poised to graduate high school, transition to attending university, and transition to adult health care without the support of family. Each task represented an action that representative users were likely to perform when using the tool to provide care transition assistance for the youth in the vignette: navigating to pages containing information on care transitions, finding and downloading articles from the evidence base, locating care transition tools, and gathering information on key aspects of developing a care transition program (Multimedia Appendix 1).

    During the conference intermission, representative users received concurrent think-aloud tasks and oral instructions on how to carry them out. These instructions, which the facilitator read out loud from a script for consistency, instructed each user to “think aloud while performing your tasks, and pretend that the facilitator is not there. Do not turn to them for assistance. If you fall silent for a while, the facilitator will remind you to keep talking aloud. Finally, remember that it is the T3 Website, and not you, that is being tested.” All interactions between the representative users and the tool were recorded.

    Quantitative: Assessing Satisfaction With the Website Evaluation Questionnaire

    To assess satisfaction with the tool, representative users completed the Website Evaluation Questionnaire upon finishing the concurrent think-aloud phase of the usability test. The Website Evaluation Questionnaire is a 26-item Likert-type scale (Multimedia Appendix 2). Items from the questionnaire load onto one of eight dimensions: ease of use, hyperlinks, structure, relevance, comprehension, completeness, layouts, and search options. For each item, participants circled the response that best characterized their attitude on the item statement: strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree. Each dimension also contained one reverse-coded item. Responses to these items thus represented a disagreement, meaning disagreement with a negative statement about the tool (in other words, satisfaction with the tool along that measure; see Elling et al [35] for the structure of the Website Evaluation Questionnaire).

    The Website Evaluation Questionnaire was specifically designed for the evaluation of government websites and is based on user attitudes toward three global factors: the interaction process, the outcomes of the interaction process, and the esthetics of the website [38]. Each of these global factors provided the conceptual foundation for the four main quality factors assessed by the Website Evaluation Questionnaire. The first factor is navigation, which assesses user satisfaction regarding the process of seeking information via the tool. This is particularly relevant to measure the functionality and usability of the tool, given that the purpose of the tool is to assist representative users in finding information on care transitions for youth with special health care needs. Navigation is assessed through the dimensions of ease of use, hyperlinks, and structure, each of which was measured by three (ease of use) or four (hyperlinks and structure) items. The items interrogate users on, for example, whether the website is user friendly (“I consider this website user friendly”), whether they could find the information they needed via the hyperlinks (“under the hyperlinks, I found the information I expected to find there”), and whether the structure of the website supports information-seeking (“the convenient set-up of the website helps me find the information I am looking for”). The second dimension, content, measured the outcomes of the process of seeking information and contained the construct relevance, completeness, and comprehension. Each of these constructs was measured by three items per construct that focused on the perceived utility of the website (ie, “this website offers information that I find useful”), how easy it is to understand the website (“I find the information in this website easy to understand”), and whether the website provides enough information (“this website provides me with sufficient information”). The third dimension was layout, which assesses the esthetics of the website through three items, which focused on the appeal and attractiveness of the website (ie, “I like the way this website looks”). The fourth and final dimension, search options, contained three items assessing the usefulness of information retrieved through the search process (ie, “the search option on this website gives me useful results”). The Website Evaluation Questionnaire was tested in controlled and real-life settings and found to be valid and reliable [35].

    Quantitative: Assessing the Relevance of Problems Encountered and Task Performance

    To assess the relevance of problems encountered with the tool, the web design experts participating in this aspect of the study were asked to evaluate the detected problems in terms of likelihood and impact. Our research team explained to the web design experts that we analyzed recordings of a usability test of the tool and identified verbal indicators of problems experienced by representative users, such as expressions of doubt, task difficulty, incomprehensibility, or annoyance related to the use of the tool. We further explained that to measure the relevance of the problems detected, we asked them, as experts in web design, to rate each problem on a scale of 1 to 5 using two Likert-type scales—one for likelihood and one for impact. Likelihood was defined for the experts as how likely a typical user would experience the problem detected. Impact was defined as how much impact the detected problem would have on the functionality and usability of the website, should that problem occur. We included quotes from the representative users as examples of the detected problems and provided the tool website address for reference (Multimedia Appendix 3).

    Analysis

    Once the usability test was completed, the research team transcribed the verbalizations, abstracted data from the surveys, and charted the participants’ navigation through the tool. The functionality and usability of the tool were understood through a thematic analysis of the concurrent think-aloud data using NVivo 12 (QSR International) [39], and descriptive analyses of the Website Evaluation Questionnaire, task performance, and relevance data using Microsoft Excel [40] and Stata 15 (StataCorp) [41].

    Qualitative: Concurrent Think-Aloud Method

    This analysis identified the number and types of problems detected through deep readings of recordings of the usability test sessions. To analyze the data obtained from the concurrent think-aloud aspect of the usability test, the researchers (DJM and JO) used a summative content analysis to identify and quantify the content of the data and understand the contextual use of words and phrases within the content [42]. These words and phrases were synthesized into salient categories (problem types). During this analysis, the researchers interpreted and assigned meanings to the synthesized categories [42]. Problems identified were confirmed through team discussions.

    Quantitative: Satisfaction With the Tool

    Data from the Website Evaluation Questionnaire were coded based on the range of numerical values of each Likert-type item (1-5). Each numerical value in the range weighted the responses to produce a weighted score with strongly disagree weighted with 1 and strongly agree weighted with 5 and with negative items reverse coded. The weighted score was used to generate a satisfaction ratio based on the sum of the 26 items through a ratio of the total score to the highest possible score, with the highest possible score representing unanimous strong agreement across all representative users. The ratio was then multiplied by 100 to generate the overall satisfaction score. A dimension satisfaction score was similarly generated for each of the eight dimensions.

    Cutoff points were determined by subtracting the lowest score on the Likert scale (260) from the highest score on the scale (1300) and dividing by the number of levels of satisfaction (3—for low, medium, and high satisfaction), thus creating an interval value of 346.6. This value was added to each score to create three categories of satisfaction: low (260-606.6), medium (606.7-953.3), and high (953.4-1300). By applying these cutoff scores to the ratio of the total score to the highest possible score, we determined that a satisfaction score of 73.33% (953.4/1300) is the cutoff for high satisfaction. The same process was repeated across dimensions for satisfaction cutoff scores of low (30-70), medium (71-111), and high (112-150) for three item dimensions and low (40-93.3), medium (93.4-146.7), and high (146.8-200) for four item dimensions. The resulting high satisfaction cutoff score was 74.6% (112/150) for the three item dimensions and 73.4% (146.8/200) for the four item dimensions. Sensitivity analyses were conducted using cutoff scores based on mean and mode with no changes to the results.

    Quantitative: Relevance of the Problems Encountered and Task Performance

    The relevance score for each problem was created through the square root of the multiplied likelihood and impact scores [43]. Likelihood and impact scores were created by weighting each numerical value in the range of the responses to produce a weighted score with unlikely or no impact weighted with 1 and highly likely or high impact weighted with 5. The weighted score was used to generate a likelihood or impact score for each problem through a ratio of the total score to the highest possible score, with the highest possible score being 25. Cutoff points were determined by creating a three-level (low, medium, and high) interval value. A score of 5 to 11.6 indicated low likelihood, low impact, or low relevance. A score of 11.7 to 18.3 was indicative of likely, impactful, and relevant. A score of 18.4 to 25 signified that the problem type was highly likely, had high impact, or was highly relevant. Task performance was understood through the completion rate for each task and the associated quality checks.


    Representative Users

    Given the small sample size and potential for identification, demographic data of representative users were fuzzed. Approximately 90% of the representative users who participated in the usability test were White, non-Hispanic women. Every user worked with at least one youth with special health care needs, and approximately 90% had a professional caseload where more than half of the patients were youths with special health care needs. More than one-third of the users worked solely with youths with special health care needs. Almost all (approximately 80%) of the representative users transitioned youths from pediatric to adult care. Half of the users worked with youths with special health care needs on both ends of the continuum—pediatrics and adult care. Approximately one-fourth of the users were physicians and approximately another quarter were nurse practitioners. The remaining representative users were social workers (approximately one-fourth), nurses, and transition specialists (clinical team coordinators, youth service specialists, and transition coaches). Some users served multiple roles (eg, nurses and certified educators for a particular chronic condition).

    Qualitative: Problems Encountered Through the Concurrent Think-Aloud Method

    Table 1 displays the number of problems detected, the different types of problems detected, and the frequency of each type of problem. A total of 31 problems were detected through the analysis of the concurrent think-aloud transcripts. These problems were aggregated into 10 categories or type of problem, most of which (29/31, 93%) focused on issues associated with finding information on health care transitions, such as the utility of search criteria, finding the search bar, finding disease-specific resources, and finding resources blocked by a firewire. Among these problems, approximately one-fourth (8/31, 26%) centered on the utility of search criteria, meaning representative users found it difficult to create search terms that would return resources on a particular topic in health care transitions:

    Still looking...self-care management, maybe I’m looking in the wrong place. Care transition, EPIC transition planning tool...uh I’m not finding the self- care. Okay, I’ll probably give up on that.
    Table 1. Frequency of detected T3 problems by type (N=31).
    Problem typeExamplesFrequency, n (%)
    Utility of search criteria“Okay so if I type in the wrong thing it makes it more difficult. Still looking...self-care management, maybe I’m looking in the wrong place. Care transition, EPIC transition planning tool...uh I’m not finding the self- care. Okay, I’ll probably give up on that.”8 (26)
    Finding the search bar is difficult“I’m looking for that, scrolling. Hmm, is there a search bar? That would have been easy, oh here we go, I found it.”4 (13)
    Finding disease-specific resources“So, I still think that it would be better to organize this page by general versus...and then also you could have some general articles and then you could have some disease-specific articles and the disease-specific articles could be in alphabetical order to make it easier to find because I kinda gave up on that one.”3 (10)
    Email of website contact opens email software or application“Oh okay, so you have to add an account, so you have to actually put your email in? Okay can I close that?”3 (10)
    Cannot locate the email of the website contact person“Copy any contact email address. Let’s see. Okay so I’m going to the wrong place and I’m going back to look. Okay, so it has a contact person but no email address.”3 (10)
    Need clarity on who to contact for an article behind a firewall“I can’t really find who to talk to about getting this article.”3 (10)
    Some windows blocked by a firewall or files won’t open“There were, I’m going to score it a four because there were some windows blocked by a firewall.”3 (10)
    Search bar is missing“Is there a search bar here? That would be helpful under the tools, search for articles and tools.”2 (6)
    Clicking on aspects of the webpage results in no action“I should not be doing this because when I click on it, it doesn’t work.”1 (3)
    Difficulty returning to a previous page (going back)“Do I hit back-arrow or close?”1 (3)

    In total, 19% (6/31) of the problems resulted from the difficulty in finding the search bar. The representative users found the placement of the search bar confusing to the point that some thought the search bar was missing:

    Is there a search bar here? That would be helpful under the tools, search for articles and tools.

    Approximately 10% (3/31) of the problems were associated with users encountering difficulties finding disease-specific resources:

    So, I still think that it would be better to organize this page by general versus...and then also you could have some general articles and then you could have some disease-specific articles and the disease- specific articles could be in alphabetical order to make it easier to find because I kinda gave up on that one.

    When attempting to access resources behind a firewall, users encountered a number of problems, including blocked windows, software application openings, and an inability to access the contact information of the website manager:

    I can’t really find who to talk to about getting this article.

    Quantitative: Satisfaction With the Tool

    The overall satisfaction score for the tool was 81.92% (1065/1300), indicating high satisfaction based on the overall satisfaction cutoff score of 73.33% (953.4/1300). The tool scored highest in relevance (138/150, 92%), followed by comprehension and layout (both 131/150, 87.3%), and the lowest in search options (114/150, 76%) and ease of use (114/150, 76%; Table 2 [35]). Satisfaction was high for each dimension.

    Table 2. Representative user satisfaction with the tool based on the Website Evaluation Questionnaire.
    Dimension and itemaSatisfaction scores

    		Dimension score, n (%)Item score, n (%)
    Relevance138 (92)

    		I find the information in website helpful.
    		46 (92)

    		Website offers information I find useful.
    		47 (94)

    		Information in this website is of little use to me.b
    		45 (90)
    Comprehension131 (87.3)

    		Language used in website is clear to me.
    		46 (92)

    		I find the information in website easy to understand.
    		42 (84)

    		I find many words in website difficult to understand.b
    		43 (86)
    Layout131 (87.3)

    		I like the way this website look.
    		42 (84)

    		I find the design of this website appealing.
    		43 (86)

    		I think this website looks unattractive.b
    		46 (92)
    Structure159 (79.5)

    		I know where to find information I need on this website.
    		38 (76)

    		I find the structure of this website clear.
    		42 (84)

    		I was constantly redirected on this website.b
    		39 (78)

    		The convenient set-up of the website helps me find the information I am looking for
    		40 (80)
    Hyperlinks163 (81.5)

    		Homepage clearly directs me towards information I need.
    		39 (78)

    		Homepage immediately points me to information I need.
    		40 (80)

    		Under hyperlinks, I found information I expected to find.
    		43 (86)

    		It is unclear which hyperlink leads to information I need.b
    		41 (82)
    Completeness115 (76.6)

    		This website provides me with sufficient information.
    		41 (82)

    		I find the information in this website precise.
    		37 (74)

    		I find the information in this website incomplete.b
    		37 (74)
    Search options114 (76)

    		Search option helps me find the right information quickly.
    		38 (76)

    		Search option gives me useful results.
    		40 (80)

    		Search option gives me too many irrelevant results.b
    		36 (72)
    Ease of use114 (76)

    		I find this website easy to use.
    		38 (76)

    		I consider this website user friendly.
    		38 (76)

    		I had difficulty using this website.b
    		38 (76)

    aItem wording truncated for parsimony. Please see the Website Evaluation Questionnaire for complete item wording.

    bReverse-coded items. The score represents a disagreement score, meaning disagreement with a negative statement about the tool (thus satisfaction with the tool along that measure).

    Quantitative: Relevance of the Problems Encountered and Task Performance

    Table 3 shows task performance. This is the number of tasks and quality checks successfully completed. Overall, 89% (89/100) of all the tasks and 75% (60/80) of the quality checks were completed successfully. Participants were least likely to find a specific tool for early adolescence, with a 70% (7/10) completion and a 70% (7/10) quality check success rate.

    Table 3. Participant (n=10) task performance and associated quality checks.a
    Task and checkbCompletion, n (%)
    Navigate to page contains a database of peer-reviewed articles.9 (90)

    		Write the first 3 words of the database page title.8 (80)
    Find peer-reviewed article on care transition for T1 diabetes.10 (100)

    		Write the first 3 words of the article title.8 (80)
    Download the article and view the abstract.10 (100)
    Navigate back to the homepage.9 (90)
    Find the contact information of someone from the T3.8 (80)

    		Copy the contact’s email here.5 (50)
    A tool by Parent to Parent provides a timeline and action items for parents of youth with special health care needs. Find and view this tool.9 (90)

    		Write the first 3 words of the tool title.7 (70)
    Find a tool to discuss level of distress around chronic disease self- management.9 (90)

    		Write the first 3 words of the tool title.9 (90)
    You are charged with improving care transition for youth with special health care needs for your institution. Find the page that would be the most useful.9 (90)

    		Write the page you found this on.8 (80)
    According to the T3, is a champion important?9 (90)

    		Write the page you found this on.8 (80)
    Find a tool specific for early adolescence.7 (70)

    		Write the first 3 words of the tool title.7 (70)

    aAggregating by tasks and checks, we found an 89% (89/100) and 75% (60/80) completion rate, respectively.

    bItem wording truncated for parsimony.

    Table 4 provides the likelihood, impact, and relevance scores of each problem type. Focusing on relevance, every problem type was found to be relevant based on a cutoff score of 11.7 for relevance, and 60% (6/10) of the problems were found to be highly relevant to the functioning of the tool based on a cutoff score of 18.4 for highly relevant. Problems that interrupted the ability of end users to locate information because of the layout or mechanical workings of the tool were rated as having the highest relevance to the functioning of the tool. These involved the tags programmed on search terms (trouble finding search criteria that result in what user wants: relevance=18.4), organization of information (hard to find articles by disease: relevance=21.5), and inability to access information (email of website contact opens email software: relevance=20.4; need clarity on who to contact for article behind a firewall: relevance=21; some windows blocked by a firewall or files will not open: relevance=22).

    Table 4. Relevance of each problem type (n=10).
    Problem typeRelevancea=, weighted scores

    		LikelihoodImpactRelevance
    Trouble finding search criteria result in what user wants201718.4
    Finding the search bar is difficult181717.5
    Hard to find articles by disease212221.5
    Email of website contact opens email software or application19

    		22

    		20.4

    Cannot locate the email of the website contact person171717
    Need clarity on who to contact for article behind a firewall20

    		2221

    Some windows blocked by a firewall or files won’t open222222
    Search bar is missing202421.9
    Clicking on aspects of the webpage results in no action141815.9
    Difficulty returning to a previous page (“going back”)151414.5

    aRelevance= or the square root of the likelihood score×the impact score (Van den Haak et al [43]). Cutoff scores were 5-11.66 for low relevance, 11.67-18.33 for relevant, and 18.34-25 for highly relevant.


    Principal Findings

    The functionality and usability testing of the tool with representative users highlighted the usefulness of this eHealth internet communication technology among providers working with youth with special health care needs. Representative users found the tool to be highly relevant, as demonstrated by the satisfaction score for the dimension relevance (138/150, 92.0%). According to the users, the tool provided comprehensive information related to health care transitions for youth with special health care needs, with a satisfaction score of 87.3% (131/150) on the comprehensive dimension. This is important, as it suggests that the tool may help bridge the gaps in training on health care transitions for youth with special health care needs in US medical education [44]. Gaps in education on health care transitions exist at all levels of medical education [44], and providers often express concerns about the lack of knowledge on transitioning care for youth with special health care needs [45,46]. Furthermore, youths transitioning from pediatric to adult care also want access to medical education on health care transitions [47], which can be provided through the tool. There is also a need to provide information on health care transitions among youth with special health care needs to adult providers, who may lack knowledge of the process [48]. An eHealth internet communication technology such as the tool, which includes evidence-based literature on health care transitions and templates for transition protocols, can be used to create a structured tool to help improve collaboration across pediatric and adult care providers and continuity of care [49]. The tool does not rely on the ability of a health care organization to integrate the tool into their system; it can be implemented within an organization or external to an organization. Although integration within the systems of care is ideal [50], in other studies of educational transition tools, health care providers expressed concern over the ability to integrate tools, given the characteristics and deficits of health care systems [51].

    Although the tool was well received by representative users, the usability test results identified areas of concern regarding functionality and usability. Users reported the most difficulty in two areas of functionality and usability: search and navigation. This was reflected in both the concurrent think-aloud and satisfaction survey results and was supported by the task performance and relevance analyses. Representative users reported lower satisfaction with search and navigation dimensions, relatively high number of search- and navigation-related problems and low task completion for tasks involving finding tools that require searching and navigation. The problems identified around the search and navigation functionality were also found to be relevant by web design experts. Each of these areas of analyses triangulates on search and navigation issues, suggesting the robustness of the results and allowing researchers to fine-tune the tool to optimize performance.

    Uncovering these inefficiencies and subsequently optimizing the tool improves its functionality and usability, which may drive the future use of the technology for its specified purpose [52]. Studies of commercial technologies show that end users are more likely to utilize a technology if it is easy to use [53-55]. This is especially important for eHealth internet communication technology, such as the tool, which is intended to improve health outcomes in the long term through the dissemination of information and best practices. Although the results from the usability test provide a clear road map for optimizing the tool, they also highlight the importance of evaluating eHealth and internet communication technology with representative users. Often, internet communication technology evaluations focus too intensely on the technical aspects of the technology rather than the needs and expectations of end users [56]. Conducting functionality and usability assessments before widespread implementation is essential for forecasting usage and ensuring that internet communication technology has the intended impact [57].

    Limitations and Future Research

    Although the concurrent think-aloud methodology and the satisfaction survey provided a robust way to ascertain the functionality and usability of this eHealth internet communication technology, there are certainly other acceptable methods. In fact, even with the growing interest in website functionality and usability and the growing use of internet communication technology for eHealth applications, there is no consensus on the definition of usability. There exists a multitude of possible dimensions and measurement techniques in the field of internet communication technology research. Choosing between these dimensions and techniques to evaluate the technology involves a certain amount of bias, which could result in the website performing better in the laboratory than in a real-life setting. However, this usability test followed quality measures for usability studies of eHealth applications [58]. We used a valid and reliable tool, chose our study design based on the objectives of the study, used across-method triangulation, and included both representative users and experts in assessing usability.

    Few evaluated measures exist that can be used to understand the impact of eHealth technology on health outcomes [59], which has potentially contributed to the limited evaluation of eHealth [60]. This is a particularly important limitation when the website is eHealth internet communication technology and is expected to affect health outcomes. Thus, further research is needed on the effect of the tool on care transition outcomes, particularly in isolated or resource-poor communities. Evaluation research on holistic care transition tools and programs is limited, with the majority of outcomes-focused evaluations targeting narrow, disease-specific populations of youth with special health care needs [6]. Although disease-specific instruments fall under the triple aim domain of population health [61], considering that most youths with special health care needs experience multiple comorbid conditions across their lifetime, more empirical evidence is needed on tools dedicated to broader care transitions. This tool, with its broad focus, answers calls to assist care providers in transitioning youth with special health care needs using a biopsychosocial model [62].

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Texas Transition Toolkit think-aloud usability testing: vignette, tasks, and quality checks.

    DOCX File , 26 KB
    ‎
    Multimedia Appendix 2

    The website evaluation questionnaire.

    DOCX File , 19 KB
    ‎
    Multimedia Appendix 3

    Texas Transition Toolkit usability trial: website problem relevance.

    DOCX File , 22 KB
    1. Coller RJ, Berry JG, Kuo DZ, Kuhlthau K, Chung PJ, Perrin JM, et al. Health system research priorities for children and youth with special health care needs. Pediatrics 2020 Mar;145(3):e20190673 [FREE Full text] [CrossRef] [Medline]
    2. McPherson M, Arango P, Fox H, Lauver C, McManus M, Newacheck PW, et al. A new definition of children with special health care needs. Pediatrics 1998 Jul;102(1 Pt 1):137-140. [CrossRef] [Medline]
    3. Council on Children with DisabilitiesMedical Home Implementation Project Advisory Committee. Patient- and family-centered care coordination: a framework for integrating care for children and youth across multiple systems. Pediatrics 2014 May;133(5):1451-1460. [CrossRef] [Medline]
    4. Boulet S, Boyle CA, Schieve LA. Health care use and health and functional impact of developmental disabilities among US children, 1997-2005. Arch Pediatr Adolesc Med 2009 Jan;163(1):19-26 [FREE Full text] [CrossRef] [Medline]
    5. National Survey of Children's Health (NSCH). United States Census Bureau.   URL: https://www.census.gov/ programs-surveys/nsch/data.html [accessed 2019-05-04]
    6. Schmidt A, Ilango SM, McManus MA, Rogers KK, White PH. Outcomes of pediatric to adult health care transition interventions: an updated systematic review. J Pediatr Nurs 2020 Mar;51:92-107 [FREE Full text] [CrossRef] [Medline]
    7. Gabriel P, McManus M, Rogers K, White P. Outcome evidence for structured pediatric to adult health care transition interventions: a systematic review. J Pediatr 2017 Sep;188:263-269 [FREE Full text] [CrossRef] [Medline]
    8. Crowley R, Wolfe I, Lock K, McKee M. Improving the transition between paediatric and adult healthcare: a systematic review. Arch Dis Child 2011 Jun;96(6):548-553 [FREE Full text] [CrossRef] [Medline]
    9. Refaeli T, Mangold K, Zeira A, Köngeter S. Continuity and discontinuity in the transition from care to adulthood. Br J Soc Work 2016 Mar 21:325-342 [FREE Full text] [CrossRef]
    10. Montano CB, Young J. Discontinuity in the transition from pediatric to adult health care for patients with attention-deficit/hyperactivity disorder. Postgrad Med 2012 Sep;124(5):23-32. [CrossRef] [Medline]
    11. Jordan L, Swerdlow P, Coates T. Systematic review of transition from adolescent to adult care in patients with sickle cell disease. J Pediatr Hematol Oncol 2013 Apr;35(3):165-169 [FREE Full text] [CrossRef] [Medline]
    12. Shepard C, Doerge E, Eickmeyer A, Kraft K, Wan J, Stoffel J. Ambulatory care use among patients with spina bifida: change in care from childhood to adulthood. J Urol 2018 Apr;199(4):1050-1055 [FREE Full text] [CrossRef] [Medline]
    13. Got Transition.   URL: https://gottransition.org [accessed 2020-06-28]
    14. American Academy of Pediatrics, American Academy of Family Physicians, American College of Physicians-American Society of Internal Medicine. A consensus statement on health care transitions for young adults with special health care needs. Pediatrics 2002 Dec;110(6 Pt 2):1304-1306. [Medline]
    15. Lebrun-Harris LA, McManus MA, Ilango SM, Cyr M, McLellan S, Mann MY, et al. Transition planning among US youth with and without special health care needs. Pediatrics 2018 Oct;142(4):e20180194 [FREE Full text] [CrossRef] [Medline]
    16. de Silva PS, Fishman LN. The transition of the gastrointestinal patient from pediatric to adult care. Pediatr Clin North Am 2017 Jun;64(3):707-720. [CrossRef] [Medline]
    17. Okumura M, Heisler M, Davis M, Cabana M, Demonner S, Kerr E. Comfort of general internists and general pediatricians in providing care for young adults with chronic illnesses of childhood. J Gen Intern Med 2008 Oct;23(10):1621-1627 [FREE Full text] [CrossRef] [Medline]
    18. White PH, Cooley WC, American Academy of Pediatrics, American Academy of Family Physicians. Supporting the health care transition from adolescence to adulthood in the medical home. Pediatrics 2018 Nov;142(5):e20182587 [FREE Full text] [CrossRef] [Medline]
    19. Hergenroeder A, Wiemann C, Cohen M. Current issues in transitioning from pediatric to adult-based care for youth with chronic health care needs. J Pediatr 2015 Dec;167(6):1196-1201 [FREE Full text] [CrossRef] [Medline]
    20. Oswald D, Gilles D, Cannady M, Wenzel D, Willis J, Bodurtha J. Youth with special health care needs: transition to adult health care services. Matern Child Health J 2013 Dec;17(10):1744-1752 [FREE Full text] [CrossRef] [Medline]
    21. Treadwell M, Telfair J, Gibson RW, Johnson S, Osunkwo I. Transition from pediatric to adult care in sickle cell disease: establishing evidence-based practice and directions for research. Am J Hematol 2011 Jan 08;86(1):116-120 [FREE Full text] [CrossRef] [Medline]
    22. Peter NG, Forke CM, Ginsburg KR, Schwarz DF. Transition from pediatric to adult care: internists' perspectives. Pediatrics 2009 Feb;123(2):417-423. [CrossRef] [Medline]
    23. Texas transitions toolkit. 2019.   URL: https://www.texastransitiontoolkit.org/ [accessed 2019-08-10]
    24. Got transition (2014-2020). The National Alliance to Advance Adolescent Health.   URL: https://gottransition.org [accessed 2019-08-31]
    25. Lu M, Lauver C, Dykton C, Kogan MD, Lawler MH, Raskin-Ramos L, et al. Transformation of the title V maternal and child health services block grant. Matern Child Health J 2015 May;19(5):927-931 [FREE Full text] [CrossRef] [Medline]
    26. Taylor L, Waller M, Portnoy JM. Telemedicine for allergy services to rural communities. J Allergy Clin Immunol Pract 2019;7(8):2554-2559 [FREE Full text] [CrossRef] [Medline]
    27. Bertot J, Jaeger PT. User-centered e-government: challenges and benefits for government web sites. Gov Inf Q 2006 Jan;23(2):163-168 [FREE Full text] [CrossRef]
    28. Onwuegbuzie A, Leech N. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative research methodologies. Int J Soc Res Methodol 2005 Dec;8(5):375-387 [FREE Full text] [CrossRef]
    29. Creswell JW, Clark VL, Gutmann ML, Hanson WE. An expanded typology for classifying mixed methods research into designs. In: Taskakkori AM, Teddlie CB, editors. Handbook of Mixed Methods in Social & Behavioral Research. Thousand Oaks, California, United States: Sage Publications Inc; 2002:209-240.
    30. Merriam S, Tisdell E. Qualitative Research: A Guide to Design and Implementation. Hoboken, NJ: John Wiley & Sons; 2015.
    31. Whitehead D, LoBiondo-Wood G, Haber J. Nursing and Midwifery Research: Methods and Appraisal for Evidence Based Practice. Amsterdam, NL: Elsevier; 2016:1-450.
    32. Stake RE. Multiple Case Study Analysis. New York City, NY: Guilford press; 2013:1-978.
    33. Gall MD, Gall JP, Borg WR. Educational Research: An Introduction, 8th Edition. New York City, NY: Pearson; 2007:1-712.
    34. Fonteyn M, Kuipers B, Grobe SJ. A description of think aloud method and protocol analysis. Qual Health Res 2016 Jul 01;3(4):430-441 [FREE Full text] [CrossRef]
    35. Elling S, Lentz L, de Jong M, van den Bergh H. Measuring the quality of governmental websites in a controlled versus an online setting with the ‘Website Evaluation Questionnaire’. Gov Inf Q 2012 Jul;29(3):383-393 [FREE Full text] [CrossRef]
    36. Hoc J, Leplat J. Evaluation of different modalities of verbalization in a sorting task. Int J Man Mach Stud 1983 Mar;18(3):283-306 [FREE Full text] [CrossRef]
    37. Sienot M. Pretesting web sites: a comparison between the plus-minus method and the think-aloud method for the World Wide Web. J Bus Tech Commun 2016 Jul 25;11(4):469-482 [FREE Full text] [CrossRef]
    38. Elling S, Lentz L, De Jong M. Website evaluation questionnaire: development of a research-based tool for evaluating informational websites. In: Electronic Government. Berlin, Heidelberg: Springer; 2007:293-304.
    39. NVivo Qualitative Data Analysis Software. QRS International.   URL: https://qsrinternational.com/nvivo/nvivo-products/ [accessed 2019-08-10]
    40. Microsoft Excel. Microsoft Corporation.   URL: https://office.microsoft.com/excel [accessed 2019-12-30]
    41. Stata Statistical Software: Release 15. StataCorp. 2017.   URL: www.stata.com [accessed 2018-05-16]
    42. Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005 Nov;15(9):1277-1288 [FREE Full text] [CrossRef] [Medline]
    43. van den Haak M, De Jong M, Schellens P. Evaluation of an informational web site: three variants of the think-aloud method compared. Tech Comm 2007 Feb 01;54(1):58-71.
    44. Sharma N, O'Hare K, Antonelli RC, Sawicki GS. Transition care: future directions in education, health policy, and outcomes research. Acad Pediatr 2014;14(2):120-127 [FREE Full text] [CrossRef] [Medline]
    45. Dressler P, Nguyen TK, Moody EJ, Friedman SL, Pickler L. Use of transition resources by primary care providers for youth with intellectual and developmental disabilities. Intellect Dev Disabil 2018 Feb;56(1):56-68 [FREE Full text] [CrossRef] [Medline]
    46. Kuhlthau K, Warfield ME, Hurson J, Delahaye J, Crossman MK. Pediatric provider's perspectives on the transition to adult health care for youth with autism spectrum disorder: current strategies and promising new directions. Autism 2015 Apr;19(3):262-271 [FREE Full text] [CrossRef] [Medline]
    47. Sezgin E, Weiler M, Weiler A, Lin S. Proposing an ecosystem of digital health solutions for teens with chronic conditions transitioning to self-management and independence: exploratory qualitative study. J Med Internet Res 2018 Sep 06;20(9):e10285 [FREE Full text] [CrossRef] [Medline]
    48. Michaud S, Dasgupta K, Bell L, Yale JF, Anjachak N, Wafa S, et al. Adult care providers' perspectives on the transition to adult care for emerging adults with Type 1 diabetes: a cross-sectional survey. Diabet Med 2018 Jul;35(7):846-854 [FREE Full text] [CrossRef] [Medline]
    49. Tanner A, Philbin MM, Ma A, Chambers BD, Nichols S, Lee S, Adolescent Trials Network for HIV/AIDS Interventions. Adolescent to adult HIV health care transition from the perspective of adult providers in the United States. J Adolesc Health 2017 Oct;61(4):434-439 [FREE Full text] [CrossRef] [Medline]
    50. Bond J, Shanske S, Hoffman R, Ross AM. Piloting a structured developmental tool to assess transition readiness for youth with special health-care needs: a mixed-methods exploration of health-care provider experiences. J Child Health Care 2020 Mar 17;24(1):92-105 [FREE Full text] [CrossRef] [Medline]
    51. Morsa M, Gagnayre R, Pomey M, Deccache C, Lombrail P. Developmentally appropriate patient education during transition: a study of healthcare providers’ and parents’ perspective. Health Educ J 2019 Nov 18;79(4):377-389 [FREE Full text] [CrossRef]
    52. Venkatesh V, Hoehle H, Aljafari R. A usability evaluation of the Obamacare website. Gov Inf Q 2014 Oct;31(4):669-680 [FREE Full text] [CrossRef]
    53. Monsuwé TP, Dellaert BG, de Ruyter K. What drives consumers to shop online? A literature review. Int J of Service Industry Mgmt 2004 Feb;15(1):102-121 [FREE Full text] [CrossRef]
    54. Rose GM, Straub DW. The effect of download time on consumer attitude toward the e-service retailer. e-Serv J 2001;1(1):55 [FREE Full text] [CrossRef]
    55. Huang Z. The impact of usability, functionality and sociability factors on user shopping behavior in social commerce design. In: HCI in Business, Government, and Organizations. Switzerland: Springer; 2018:303-312.
    56. Verdegem P, Verleye G. User-centered e-Government in practice: a comprehensive model for measuring user satisfaction. Gov Inf Q 2009 Jul;26(3):487-497 [FREE Full text] [CrossRef]
    57. Tomlin C. The top 7 healthcare.gov UX failures caused by poor forecasting of usage. Useful Usability. 2013.   URL: http://www. usefulusability. com/top-7-healthcare-gov-ux-failures [accessed 2018-08-31]
    58. Silva AG, Simões P, Santos R, Queirós A, Rocha NP, Rodrigues M. A scale to assess the methodological quality of studies assessing usability of electronic health products and services: Delphi study followed by validity and reliability testing. J Med Internet Res 2019 Nov 15;21(11):e14829 [FREE Full text] [CrossRef] [Medline]
    59. Wakefield BJ, Turvey CL, Nazi KM, Holman JE, Hogan TP, Shimada SL, et al. Psychometric properties of patient-facing ehealth evaluation measures: systematic review and analysis. J Med Internet Res 2017 Oct 11;19(10):e346 [FREE Full text] [CrossRef] [Medline]
    60. Enam A, Torres-Bonilla J, Eriksson H. Evidence-based evaluation of ehealth interventions: systematic literature review. J Med Internet Res 2018 Nov 23;20(11):e10971 [FREE Full text] [CrossRef] [Medline]
    61. Prior M, McManus M, White P, Davidson L. Measuring the "triple aim" in transition care: a systematic review. Pediatrics 2014 Dec;134(6):1648-1661. [CrossRef] [Medline]
    62. Kuo AA, Ciccarelli MR, Sharma N, Lotstein DS. A health care transition curriculum for primary care residents: identifying goals and objectives. Pediatrics 2018 Apr;141(Suppl 4):346-354 [FREE Full text] [CrossRef] [Medline]

    Edited by G Eysenbach; submitted 03.08.20; peer-reviewed by A Abubakar, A Tanner; comments to author 02.12.20; revised version received 27.03.21; accepted 13.04.21; published 25.05.21

    Copyright

    ©Darcy Jones McMaughan, Sherry Lin, Jennifer Ozmetin, Judith Gayle Beverly, Joshua Brog, Emily Naiser. Originally published in JMIR Formative Research (https://formative.jmir.org), 25.05.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.