This usability pilot study used a mixed methods pre-post convergent design to evaluate the usability and preliminary effectiveness of the PHS program among health care providers from diverse disciplines. In this design, quantitative (eg, questionnaire ratings and pre-post outcomes) and qualitative (eg, open-ended feedback) data were collected concurrently and integrated during interpretation to provide a more comprehensive understanding of the user experience.

The PHS program was developed by an interdisciplinary team of investigators, including PC (psychology), SW (medicine), and EK (nursing). The PHS professional development program includes 4 sessions, each approximately 30 minutes in duration, which HCPs can complete in approximately 2 hours. The program focuses on assessing pediatric insomnia in children aged 5 to 12 years, information about other pediatric sleep disorders, and how to evaluate and manage insomnia in children (Figure 2 provides a description of each session). To ensure ease of access, scalability, and cost-effectiveness, the PHS program was created as an e-learning program. E-learning programs are delivered online via the internet and have the potential to reduce access barriers. Multiple studies have examined the delivery of e-learning programs, and e-learning has been shown to be effective in delivering professional development programs to HCPs [25-28].

For this study, our goal was to recruit 20 HCPs practicing in Canada whose general practice includes working with school-aged children, some of whom have insomnia. Eligible participants had to (1) be fluent in English (as the program is currently available only in English), (2) be a regulated HCP (ie, part of a regulatory body), and (3) have worked over the past 2 years with a minimum of 5 children aged 5 to 12 years for whom the primary concern expressed by the parent was that their child had symptoms of insomnia. Participants were ineligible for the study if they were considered experts in pediatric sleep, worked in a clinical practice that was mainly focused on pediatric sleep disorders, or were sleep coaches/consultants but not regulated HCPs. Our goal was to recruit 5 participants in each of the 4 targeted HCP groups: physicians (eg, family physicians, pediatricians, and psychiatrists), psychologists, nurses, and allied health professionals (eg, social workers, occupational therapists, and recreational therapists). In previous research, it has been suggested that about 85% of unique feedback and problems that are inherent in usability studies can be detected with only 5 participants [29], so the sample size of 5 participants for each of the 4 HCP groups should detect problems that need to be addressed in the PHS program.

A multipronged recruitment strategy was used, and recruitment was conducted from June to November 2023. The investigators recruited directly through their professional and disciplinary networks for this study. Advertising to recruit participants was done through hospital and professional body newsletters throughout Canada and was also advertised on LinkedIn, Instagram, Twitter, and Facebook.

Potential participants were provided with a link to a brief electronic screening questionnaire administered on REDCap [30] to ensure that they met the criteria for the study and to confirm the professional body with which they were registered to verify that they were licensed HCPs. Once their status as a licensed HCP was confirmed by a researcher (AI), eligible participants were automatically directed to the electronic Information and Consent Form. Once consent had been provided, participants were directed to complete the demographic questionnaire, sleep attitudes and beliefs questionnaire, and sleep knowledge questionnaire. HCPs who completed these steps were given access to the e-learning program for 1 month.

Session content was delivered via the internet in a lecture-style video format (ie, a speaker presenting program content with slides) and was displayed sequentially (following mandatory completion of the previous session), with each video session lasting approximately 30 minutes (Figure 2 provides the PHS session content). This usability study consisted of asking HCPs to provide feedback on their perceptions of the content and delivery of the program and their personal likelihood of using these materials in their respective work settings to support their work in assessing and treating pediatric insomnia. Once all 4 sessions were completed and all study questionnaires were completed, participants received a certificate of completion that they were able to download for their professional development portfolio.

For quantitative data, SPSS (version 28; IBM) was used to conduct all analyses. Descriptive statistics (means and SDs for continuous variables; percentages for categorical variables) were calculated for the HCP demographic questionnaire, the S-AUQ and P-AUQ, and the SQ. Paired 2-tailed t tests were conducted for the SKQ and SABQ (both administered preintervention and postintervention to HCPs), and effect sizes for both questionnaires were calculated using Cohen d.

Qualitative data from the P-AUQ, S-AUQ, and SQ were managed using Microsoft Excel and analyzed using inductive content analysis [33]. Content analysis involves three stages: (1) preparing the data (eg, deciding what units of data to code), (2) organizing the data (eg, open coding and creating categories), and (3) reporting themes (eg, discussion of the analysis process and results). Qualitative data were inductively coded into themes by the primary author (AI) and were checked for rigor and trustworthiness by a senior member of the research team (PC). For the HCP qualitative data, a theme was required to be identified by 3 or more individual participants, as established a priori for threshold inclusion (3/22, 14%) [34,35]; an a priori theme threshold has been used in similar qualitative studies evaluating programs [34-36].

The study involved human participants, and it was reviewed and approved by the Izaak Walton Killam Hospital Ethics Review Board (#1027643). The participants provided written informed consent to participate in this study, and all study data were anonymized. Participants received a CAD $50 (CAD $1=US $0.76 as of June 20, 2023) gift card honorarium for completing the study. The study adhered to the Declaration of Helsinki. A clinical trial number was not applicable.

Out of the 28 HCPs who consented to participate, 22 HCPs completed the study, which was slightly above our target of 20 participants and represents a 79% retention rate. Six participants withdrew because they did not have the time commitment available to complete the study. All those who participated in the study (N=22; 100%) reviewed the entire (ie, all 4 sessions) PHS program, and there were no missing data. Table 1 provides the participant-reported demographic characteristics and demographic characteristics divided by HCP occupation group.

When participants were asked about their perceptions of their ability to evaluate and treat sleep problems in children before completing the program, 82% (18/22) of participants stated that they felt they were able to evaluate/treat sleep problems but not as well as they would like, and the remaining 18% (4/22) stated that they felt they were unable to evaluate/treat sleep problems in children. Of note, there was variability in responses about their ability to evaluate/treat pediatric insomnia across HCP groups, with 100% (6/6) of allied health professionals, 83% (5/6) of psychologists, 80% (4/5) of nurses, and 60% (3/5) of physicians indicating that they were able to do this, but not as well as they would like.

Overall, participants rated the PHS program highly in terms of satisfaction. The mean ratings for all 8 items on the SQ ranged from 3.18 to 3.45, with a response of 3 indicating “mostly satisfied” and a response of 4 indicating “very satisfied” (Table 2). The majority of participants (19/22, 86%) reported that the PHS program met their needs, they would recommend the program to colleagues, they were satisfied with the program, and they would use the program outside the study. Table 2 provides a breakdown by item.

Participants’ evaluations of the PHS program’s overall usability are summarized in Table 3. When asked to rate the overall program’s usability, there were generally high ratings across the 7 usability dimensions. Credibility received the highest rating, whereas findability received the lowest rating across the usability dimensions. Overall, the participants viewed the program as useful, usable, desirable, findable, accessible, valuable, and credible (ie, they gave ratings of 3 or higher on these items). When program usability ratings by type of HCP were reviewed, it was found that physicians had the highest program ratings across all 7 usability dimensions; in contrast, psychologists rated the program the lowest across all 7 usability dimensions, though they still rated the program as moderately usable.

Generally, ratings across sessions were similar in that all means fell within the “moderate” to “extremely” usable range (Table 4). When looking at the ratings for all 4 sessions, session 4, which covered a variety of evidence-based treatments for pediatric insomnia, attained the highest ratings across all usability dimensions compared with the other sessions, except for credibility, which was high across all 4 sessions.

Participants’ acceptability of overall program content (P-AUQ) and session-specific content (S-AUQ) is summarized in Table 5. The entire PHS program and all 4 sessions were rated highly in terms of being educational and organized, with mean ratings falling between “agree” (rating of 4) and “strongly agree” (rating of 5). Overall, the majority (12/22, >50%) of participants did not perceive the need for content to be added, deleted, or changed in the program or for any individual sessions. All participants viewed the program as educational and organized, with most participants (82%; 18/22) reporting that it was ready to be used by other HCPs. In terms of material being removed, changed, or added to the program, 70% (15/22) of participants disagreed with this.

When ratings by HCP group were evaluated, physicians and allied health professionals had the highest ratings for the program, and psychologists had the lowest ratings for the overall program content compared with the other 3 HCP groups (Table 5).

A paired 2-tailed t test was conducted to examine changes in sleep knowledge, as measured by the SKQ scores before and after program completion. When a Shapiro-Wilk test of normality was conducted, nonsignificant P values were found for both variables, indicating that the data were approximately normally distributed. Participants’ preprogram SKQ scores (mean 10.27, SD 2.19) significantly differed from their postprogram scores (mean 13.50, SD 2.52; t₂₁=–4.93; P<.001), as scores significantly increased postprogram completion. This represents a large effect size (d=–1.05, 95% CI –1.57 to –0.52).

A paired 2-tailed t test was conducted to examine changes in sleep attitudes and beliefs, as measured by SABQ scores before and after program completion. When a Shapiro-Wilk test of normality was conducted, nonsignificant P values were found for both variables, suggesting that the data were approximately normally distributed. Participants’ postprogram scores (mean 53.91, SD 3.88; t₂₁=–2.55) were significantly higher than their preprogram SABQ scores (mean 51.18, SD 4.38; P=.02). This represents a moderate effect size (d=–0.54, 95% CI –0.99 to –0.09).

The PHS program was developed to address a significant evidence-to-practice gap: children with insomnia/insomnia symptoms rarely receive evidence-based care because HCPs are not adequately trained to screen, assess, and treat these sleep problems. After completing the PHS program, participants reported high levels of satisfaction with both the program and its content. The PHS program overall and each of the 4 sessions were rated as highly usable and had high acceptability ratings from HCPs. Overall, the feedback was mostly positive, with only minor suggested changes to the program. Moreover, the PHS program demonstrated preliminary effectiveness, with improvements in HCPs’ sleep-related knowledge as well as attitudes and beliefs from before to after program completion.

When evaluating HCPs’ satisfaction with the PHS program (research question 1), over 90% of participants reported that they would recommend the program to colleagues, they would use the program outside the study, the program met their needs, and they rated the program highly in terms of quality, content, and applicability. This supports our expected results that >80% of participants would be satisfied with the PHS program. Interestingly, physicians and allied health professionals reported the highest ratings for program satisfaction overall, and psychologists reported the lowest ratings, although these were still generally positive.

Previous research suggests that parents of children with sleep difficulties often turn to primary care providers for support [35], and pediatricians frequently encounter sleep-related concerns in practice [18]. Despite this, they receive little to no comprehensive formal training in sleep assessment and treatment [37-41]. This may explain why physicians and allied HCPs in our study reported the highest satisfaction with the PHS program, given that the program provided information that was missing from their formal training and was directly relevant to their clinical work. In contrast, while many clinical psychologists receive limited training in sleep management during their education and professional development [36], they are highly trained in behavioral management, which informs the foundation of sleep intervention strategies. As such, they may not have found the program to contain novel clinical information [19]. Of note, 5 of the 6 psychologists in this study reported feeling capable of addressing pediatric sleep problems, though they still expressed a desire to improve their skills. In other health care disciplines, previous research shows that most HCPs recognize the importance of sleep health [42]; however, their awareness of its relationships to psychosocial factors remains limited [43]. These findings highlight the need for training in the screening, assessment, and treatment of pediatric sleep problems, particularly for HCPs who believe they require additional training in this area.

Participants gave high ratings across all 7 dimensions of Morville and Sullenger’s [24] UX Honeycomb framework for both the program overall (research question 2) and the individual sessions (research question 3). The mean usability rating across all 7 user dimensions was approximately 4 out of 5, with over three-quarters of the participants rating the PHS program as useful, usable, desirable, findable, accessible, valuable, and credible. Though all sessions received very similar usability ratings, session 4, which focuses on pediatric insomnia interventions (eg, behavioral strategies and medication options), received slightly higher usability ratings. This is likely due to its direct relevance to HCPs dealing with children facing sleep challenges, making this session particularly valuable, desirable, and useful. Frequency ratings from the usability and satisfaction questionnaires indicated that 2 participants were more critical in their assessment of the program: one from the psychologist group and another from the nurse group. Their responses to the questionnaires suggested that the PHS program did not align well with their content needs. To address this in future studies, it could be beneficial to administer a preprogram questionnaire, which would inquire about participants’ objectives for enrolling in the PHS program in order to assess whether the program’s content and structure are likely to meet the specific needs and expectations of each participant.

The acceptability findings (research question 4) indicated that the overall program and all 4 sessions were highly regarded for being educational and well organized, and 82% (18/22) of participants believed that the program is ready to be used by other HCPs. While the majority of participants (12/22, 55%) did not indicate the need to modify the program, some participants did provide some constructive feedback (research question 5). One key area of feedback was related to the usability of the program (eg, reducing technical difficulties and program platform issues). In e-learning programs, technical difficulties have been found to decrease motivation and negatively impact program attrition [44]. As found in previous research, usability studies identify these types of problems so that they can be addressed before larger-scale testing [24,45]. Participants also provided specific content suggestions, such as adding more information about NDDs and sleep in school-aged children. Previous research demonstrates that NDDs and insomnia in children are highly comorbid, with up to 90% of children with NDDs experiencing insomnia symptoms [37,46]. To address this feedback, we are planning to create an additional module to add to the PHS program focusing on NDDs and sleep in school-aged children. Lastly, participants suggested adding additional resources for parents and HCPs, removing some program content, and increasing program interactivity, which aligns with other research where HCPs have found programs more effective when given applicable information that they can apply directly in practice [38,39]. Incorporation of user-centered design principles through usability testing is known to enhance engagement, completion rates, and implementation in clinical practice, ultimately increasing program effectiveness [22].

Lastly, our findings revealed a significant improvement in HCPs’ scores on the SKQ and the SABQ after completing the program, indicating an increase in their sleep knowledge, attitudes, and beliefs (research question 6). This underscores that the PHS program has the potential to boost HCPs’ understanding of pediatric insomnia and how to screen, assess, and manage these sleep issues in children. The questionnaires also showed preliminary evidence that HCPs’ perspectives on sleep had changed after completing the program. This change suggests that the program also positively influenced HCPs’ perceptions of sleep modifiability, children’s responsiveness to treatment, the overall impact of sleep issues, and the nature of children’s sleep problems.

A key strength of this study is that a user-centered design was used to evaluate the program by end users (ie, HCPs). Usability studies have been found to identify problems before larger-scale testing and to ensure that programs are usable to the end user, both of which are critical to user engagement with the program [24,45]. Another key strength of the study is that we recruited different types of HCPs to complete and evaluate the PHS program (ie, physicians, psychologists, nurses, and allied HCPs, such as occupational therapists, social workers, and recreation therapists), given that pediatric insomnia care is part of the clinical practice of many different types of HCPs. These HCPs often receive minimal to no training in pediatric sleep during professional training programs (eg, regular curriculum programming) or in continuing education [40-42], and as such, this program can help bridge the evidence-to-practice gap across different HCP disciplines.

There are some limitations to acknowledge in this study. Our sample predominantly identified as women, which limits our ability to generalize these findings. With this being said, approximately 70% of HCPs in Canada identify as women, so this reflects the current health care field [43]. Furthermore, we chose an a priori threshold for qualitative data analysis, meaning that comments not meeting the threshold were excluded from the final results. As a result, some important feedback may have been missed. Due to the small sample size, outlier ratings had a greater impact on questionnaire means, and we were underpowered to test whether excluding them would meaningfully change the results. Additionally, the SKQ and SABQ were author-developed and not psychometrically validated, potentially limiting their reliability and generalizability. The SKQ was based on PHS program content, which may further restrict its applicability beyond this context. Although improvements seen in sleep knowledge and attitudes after completing the program are promising, these findings need to be replicated in a larger-scale effectiveness trial. Lastly, the inclusion of HCPs who were likely interested in the content (sleep in children) and those recruited through the professional contacts of the developers of the PHS program raises the possibility of bias in their feedback.

Future research should evaluate the long-term effectiveness of the PHS program by conducting assessments at 6 and 12 months after program completion to determine whether increases in sleep knowledge lead to sustained behavioral changes in clinical practice. Investigating program outcomes across various health care disciplines could determine whether effectiveness differs by professional background. Additionally, the cost-effectiveness of the PHS program could be examined in future research, as previous research on sleep education programs has found them to be economically valuable [47]. To better understand how changes in knowledge and attitudes translate into practice, incorporating interviews with HCPs to identify barriers and facilitators to integrating evidence-based sleep assessment and treatment would be valuable. Furthermore, expanding e-learning HCP training programs to address different age groups (ie, young children, youth, and adults) and specific populations (eg, NDDs) is currently in development by the research team to address these needs.

In summary, this study demonstrated high satisfaction, acceptability, and usability ratings for both the overall program and the individual sessions of the PHS e-learning program from HCPs. Additionally, the study suggests that HCPs’ sleep knowledge, attitudes, and beliefs can improve following program completion. Qualitative feedback identified minor needed modifications, which will inform ongoing refinements. A health care system equipped with updated knowledge and skills regarding pediatric sleep issues is essential for providing better care to children and families facing sleep challenges. The next step involves a larger-scale effectiveness study across Canada, aiming to enhance insomnia treatment for school-aged children by better preparing HCPs. Ultimately, this user-centered online training program seeks to improve health care delivery and promote the long-term health and well-being of children in Canada [3]. From a public health perspective, an e-learning professional development program geared toward the screening, assessment, and treatment of insomnia and sleep problems in children holds immense potential to close the existing evidence-to-practice gap [45].