Outreach dissemination strategies over the 2-year period encompassed channels through grassroots, community, and professional networks. These information channels comprised monthly AI/AN Community of Practice Network meetings highlighting the HNY Website and Toolkit for professionals who plan, manage, and facilitate healthy youth programs in Indian country, social media posts through existing networks in the AI/AN community devoted to health messages and event promotion, field studies with AI/AN practitioners and community members and related manuscript publication, presentations at professional conferences (eg, American Public Health Association, Department of Health and Human Services Office of Population Affairs Teen Pregnancy Prevention) and Tribal meetings (eg, The Tribal Public Health Conference) reporting formative, outcome, and process data from the HNY Toolbox, and presentations on varied health topics that promoted the HNY website as a resource. Rollouts were planned to ensure optimal breadth (geographically) and depth of exposure (including local Tribal organizations, opinion leaders, national Federal decision makers, and funders).

Data from the HNY website and the embedded HNY Implementation Toolbox were assessed using Google Analytics collected from January 2022 to January 2024. Google Analytics is a web analytics service that tracks and reports website traffic and user behavior [27,28]. This tool is used extensively by digital marketers and businesses to test and improve digital marketing performance, assess customer behavior, and identify strategies to optimize and improve traffic to a website by providing insight into user behavior.

Data were collected to assess reach, user demographics, and user engagement through a real-time, interactive dashboard accessible through a registered account on the Google Analytics website [29].

Reach is the extent to which an intervention or program is delivered to the priority population and was assessed as the number of users from each US state [29]. It is a necessary antecedent for community exposure to intervention content and any opportunity to establish effectiveness [29,30]. A user was defined as a visitor to the sites who had an assigned unique identifier with one or more associated sessions [29]. User demographics were assessed by geographic location, user gender and age, and the type of device used to access the sites. User engagement was assessed by the number of page views, number of pages viewed per session, number of “tools” downloaded, average duration of sessions, bounce rate, and exit rate. Page views were assessed as the number of separate views each page received. A session was defined as a user visit to, and interaction with, the webpage (eg, button clicks, downloads, and page navigation). Pages per session were assessed as the number of web pages of the site that the user viewed in a single session [29]. Duration of a session was assessed as the average length of time in minutes that users spent on a website/webpage during a session [29]. Bounce rate, or nonengagement rate, was assessed as the percentage of single-page sessions a specific webpage received. If a user loaded a page but did not interact a second time (by viewing another page) within 30 minutes, this was considered a bounce [29]. For example, a bounce rate of 50% means that half the users engaged with the website on the visited page only. Exit rates were assessed as the percentage of users that left the website from a webpage (ie, the webpage was the last page visited by them before they exited the website) [29]. For example, an exit rate of 20% means that 20% of users left the website from this page. Reporting bounce rates and exit rates assess if the website/tool met the expectations of a user. If these are lower than industry standards then the site is considered to be engaging [29]. Downloads were assessed by user selections of a download option for a given resource template.

This study was reviewed and approved by the Committee for the Protection of Human Subjects at the University of Texas Health Science Center at Houston (institutional review board number: HSC-SPH-18‐0958). The HNY website also outlines its privacy policy in the footer section of the website as a means of informed consent from users and describes how user data are collected and may be used for research purposes. All data used are deidentified and anonymous. This manuscript, including related images and supplementary material, does not allow the identification of individual participants or users. No compensation was offered to participants as this paper reports analytics of a web-based resource that is accessible free of cost.

From January 2022 to January 2024 the HNY website had 10,237 total users. Most users were located in Washington (n=1515), California (n=1290), and Oregon (n=1019) (Figure 3A). The majority of users identified as female (n=5217, 70.0%), were mainly 25‐34 years (n=1676, 23.0%) or 18‐24 (n=1559, 22.0%) years of age, and accessed the website primarily via desktop (80.0%) and mobile (19.0%) devices (Table 1). The most frequently accessed sections (in decreasing order) were Curricula (9902 views), Resources (6397 views), HNY Implementation Toolbox home (4364 views), Community (4054 views), and For Caring Adults (3766 views) (Table 2). Page duration ranged from 1.7 minutes (Implementation Toolbox) to 3.06 minutes (Curricula), bounce rate from 50.2% (Curricula) to 65.5% (Community), and exit rates from 15.9% (Curricula) to 60.0% (For Caring Adults). Access to the HNY website increased from 2479 page views in the first quarter of 2022 to a cumulative total of 33,004 page views by the fourth quarter of 2023 (Figure 4).

From January 2022 to January 2024 the Implementation Toolbox had a total of 2566 total users (Table 3) and 11,762 page views (Figure 4). Most users were located in California (n=1238), Washington (n=1142), and Oregon (n=986) (Figure 3B). The majority of users identified as female (n=472, 66%), were mainly 35‐44 years (n=444, 35%) or 25‐34 (n=319, 25%) years of age and accessed the Toolbox mainly via desktop (81%) and mobile (19%) devices (Table 1). Phase page views (in decreasing order) were Gather (3278 views), Choose (1275 views), Prepare (907 views), Implement (843), and Grow (618 views) (Table 3). Page duration ranged from 1.67 minutes (Grow) to 2.08 minutes (Choose), bounce rate from 41.8% (Gather) to 72.9% (Prepare), and exit rates from 22.1% (Prepare) to 34.5% (Grow). The most frequently downloaded tools within the Toolbox were from the Gather phase and included the “Community needs and resource assessment” factsheet (136 views), the “Identify youth advocates and community partners” factsheet (111 views), and a “Bingo data collection” activity guide (83 views) (Table 4). Access to the HNY Implementation Toolbox increased from 1380 page views in the first quarter of 2022 to a cumulative total of 11,762 page views by the fourth quarter of 2023 in conjunction with dissemination activities of scientific presentations, community of practice sessions, social media posts, and webinars (Figure 4).

The HNY website and embedded HNY Implementation Toolbox provide web-based resources to support the adoption, implementation, and maintenance of culturally relevant, age-appropriate evidence-based adolescent health promotion programs in AI/AN communities. The HNY website is a “one-stop-shop” for Tribal health advocates to access culturally relevant and age-appropriate health curricula, programs, and related resources for AI/AN youth [14]. The embedded HNY Implementation Toolbox provides guidance to enable them to successfully implement these curricula in their communities. Over the 2-year evaluation period since the Toolbox’s “go live” date, approximately 1 in 8 users of the HNY website visited the Toolbox. This, and expanded geographic reach, corroborate the previously reported utility of the Toolbox [17]. The differential traffic and reach between the HNY website and the Toolbox may reflect the comparative maturity of the HNY website compared with the relatively newer embedded Toolbox. The greater reach in the western states is expected because the HNY website was developed and is housed in the NPAIHB located in Washington State, and early promotion was among affiliated Tribes in Washington State, Oregon, and Idaho. Demographically, the Toolbox users were older, with 35% (n=444) being 35‐44 years of age. This is likely a reflection of the demographics of community health advocates who are evaluating community needs, facilitating decision-making on the curricula best addressing these needs, and actively engaging community support to adopt and implement these curricula in their communities. This, and whether users had attended Tribal health conferences or Community of Practice gatherings in which the Toolbox was promoted could not be verified from the Google data.

Over the evaluation period, page views of the HNY Website and the HNY Toolbox increased 10-fold and 9-fold, respectively. Page views to the Toolbox witnessed a steady increase from 2022 to 2023. The reason for this is uncertain but is possibly attributable to the Toolbox being showcased in monthly Community of Practice meetings that gather dozens of advocates for Tribal youth in Indian country. This suggests the potential for embedded practitioner networks as a particularly utilitarian strategy to disseminate youth programs [27,28]. Given that the HNY website and Implementation Toolbox coexist, the promotion of one likely promotes traffic to the other. This is the strength of a collaborative approach that provides multiple resources that address a particular theme, whether it be a priority population, setting, or health domain. The cumulative page views over time are also suggestive of the need to provide a persistent and longitudinal promotional strategy. We will continue to build awareness nationally via presentations, workshops, and round table discussions at Tribal, education, and public health conferences.

The Toolbox’s average bounce rate (57.7%) was lower than that of the HNY website (62.1%) and also the industry average (62.3%); the average number of Toolbox pages visited per session was almost 5 times than that of the industry average (11.9 vs 2 pages) [14,31,32]. Further, the duration of use on both sites (2.4 and 1.8 respectively) are close to averages on comparable websites (2.58) [33]. These findings suggest that the Toolbox is “sticky,” and that users are finding material commensurate with their needs. This is also supportive of previously reported positive usability ratings for the site from AI/AN community decision makers.

User preferences within the Toolbox were for the Gather Phase and tools related to this phase that enable needs assessment and identification of community advocates. This may be indicative of Tribal communities being in this early phase of adopting and implementing adolescent health programs, that these tools are particularly useful for the work of health educators, or simply that community members are likely to start at the first step in their review and exploration of a novel support tool like the Toolbox. Future studies can investigate if progression through stages occurs over time, the association of phase use with “in-situ” community implementation, as well as challenges to implementation that can be a function of differing state or local policies, regional internet accessibility, funding, community readiness to adopt programs, and logistics of obtaining school board approvals [16].

Study findings need to be interpreted in the context of several methodologic limitations and biases due to data collection. Google Analytics was originally intended for marketing and sales purposes and is limited largely to descriptive data [34,35]. Guidelines for the use and interpretation of Google Analytics do not exist. Discrepancies in the data are possible due to a new client ID being assigned every time a user deletes cookies or signs in from a different device or browser [36]. This may produce an inaccurate count of “new” users because different IDs could be allocated to the same user [34]. Additionally, deletion of cookies may degrade data captured via Google Analytics which relies on user cooperation for successful data collection [20]. Google Analytics assigns a value of 0 seconds if a user visits a webpage until they navigate to a second page, which can lead to inaccurate page duration assessments [34]. Google Analytics does not collect demographic and contextual data on ethnicity, Tribe, or the context of the application of the HNY website and Toolbox. The gender of users captured by Google Analytics may only reflect a subset of users that log into their Google accounts when visiting the page and it is difficult to determine how the demographics align with the target population [34]. Page views and resource downloads do not necessarily equate to the actual adoption and implementation of programs. There is a risk of selection bias wherein the analytics reported may only reflect a subset of users that log into their Google accounts The purpose of use, whether in the context of seeking guidance on program implementation, review and feedback for research, or simply to assuage curiosity, was not ascertained in this study and will need to be if the utility of the HNY Website and Toolbox is to be fully understood. To address the above biases and limitations, reporting using Google Analytics may be applied in combination with other field data collection (qualitative or quantitative) to triangulate data sources. This study reported on an array of promotional strategies to achieve increased dissemination. Future studies, perhaps using a mixed methods design, are recommended to investigate their comparative effectiveness.

Despite limitations, the use of Google Analytics to assess and gather feedback on web-based decision support and to track user behavior on multiple devices is well accepted [20]. Advantages are that it can be used for self-comparison over time and provide a comparison to industry standards. This can provide a positive feedback loop wherein higher website traffic builds positive normative perceptions that generate further user traffic through “word of mouth.” An assessment of reach represents an early phase in understanding program dissemination and implementation that provides the foundation for future hybrid effectiveness-implementation evaluation, including network analysis to assess the dynamics of community diffusion.

The HNY website and HNY Implementation Toolbox provide AI/AN educators and youth advocates with step-by-step guidance to support the adoption, implementation, and maintenance of culturally relevant, age-appropriate evidence-based adolescent health promotion programs in their communities. Google Analytics provides insight into the use and reach of the HNY website and Implementation Toolbox among AI/AN youth advocates, which will help inform ongoing dissemination efforts. Future studies are needed to assess the Toolbox’s actual impact on the uptake and implementation of adolescent health promotion EBPs in native communities.