Published on 05.11.19 in Vol 3, No 4 (2019): Oct-Dec
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/14906, first published Jun 01, 2019.
Formative Evaluation to Build an Online Parenting Skills and Youth Drug Prevention Program: Mixed Methods Study
Background: Family-based drug prevention programs that use group-based formats with trained facilitators, such as the Strengthening Families Program (SFP), are effective in preventing underage drinking and youth drug use. However, these programs are resource-intensive and have high costs and logistical demands. Tailoring them for Web-based delivery is more cost-effective and makes it easier to scale these programs for widespread dissemination. This requires the active involvement of all key stakeholders to determine content and delivery format.
Objective: The aim was to obtain consumer, agency stakeholder, and expert input into the design of a Web-based parenting skills training and youth drug prevention program.
Methods: We conducted 10 focus groups with 85 adults (range 4-10, average 8 per group), 20 stakeholder interviews with family services agency staff, and discussed critical design considerations with 10 prevention scientists and e-learning experts to determine the optimal program content and technology features for SFP Online. Focus group participants also answered survey questions on perceived barriers to use, desired navigational features, preferred course format, desired content, preferred reward structures, course length, interactive components, computer efficacy, and technology use. Descriptive statistics were used to examine consumer characteristics; linear regression was used to examine relations between SFP exposure and four continuous outcome measures, including desired program content, interactive technology, and concerns that may inhibit future use of SFP Online. Logistic regression was used as a binary measure of whether consumers desired fun games in the SFP Online program.
Results: Three broad thematic categories emerged from the qualitative interviews enumerating the importance of (1) lesson content, (2) logistics for program delivery, and (3) multimedia interactivity. Among the many significant relations, parents who viewed more SFP lessons reported more reasons to use an online program (beta=1.48, P=.03) and also wanted more interactivity (6 lessons: beta=3.72, P=.01; >6 lessons: beta=2.39, P=.01), parents with less interest in a mixed delivery format (class and online) reported fewer reasons to use the online program (beta=−3.93, P=.01), comfort using computers was negatively associated with concerns about the program (beta=−1.83, P=.01), having mobile phone access was related to fewer concerns about online programs (beta=−1.63, P=.02), willingness to view an online program using a mobile phone was positively associated with wanting more online components (beta=1.95, P=.02), and parents who wanted fun games wanted more interactivity (beta=2.28, P=.01).
Conclusions: Formative evaluation based on user-centered approaches can provide rich information that fuels development of an online program. The user-centered strategies in this study lay the foundation for improving SFP Online and provide a means to accommodate user interests and ensure the product serves as an effective prevention tool that is attractive to consumers, engaging, and can overcome some of the barriers to recruitment and retention that have previously affected program outcomes in family-based prevention.
JMIR Form Res 2019;3(4):e14906
Youth alcohol and drug use continue to be top public health priorities. This emphasis stems from recognition that experimental drug use that often coincides with adolescent rebelliousness can easily transform into addiction if left unabated. Numerous trials have now shown that family-based drug prevention programs are an effective first line of defense preventing underage drinking and youth drug use [- ]. The evidence base also reinforces the utility of combining parenting skills training with program content that emphasizes youth drug prevention [ ]. An important consideration in this approach is that family dynamics, and especially parent-child social interactions, play a crucial role in a child’s developmental outcome [ , - ]. Most—if not all—of these programs have their conceptual roots in ecological [ ], social interactional [ ], and transactional [ ] models of human development. All these approaches underscore the close interpersonal alliance formed between parent and child and its incipient role in developmental outcomes. Regardless of conceptual underpinnings, most family-based programs strongly reinforce that parents shape their child’s behavior from a very early age.
The Strengthening Families Program (SFP) is one example of a parenting skills training program that combines youth drug prevention in a group-based, facilitator-led format for children aged 3 to 17 years. The program was first tested with 14 sessions designed to improve parenting skills among drug-addicted adults in treatment settings [, ]. Since then, the program has been retooled for universal settings [ ], applied to young children [ ], delivered in rural settings [ ], tested in urban schools [ ], culturally adapted [ , ], and examined using rigorous randomized controlled trials in international settings [ , ]. There is now also evidence the program can promote parent-child reunification in child welfare settings [ ] and produce other societal and economic benefits [ ].
In keeping with well-established research traditions, the program’s core components draw from family systems theory , relationship enhancement approaches [ ], family-based therapeutic approaches [ ], behavioral parent training [ ], and social learning theory [ ]. Collectively, these theoretical linchpins support the goal of teaching parents how to bond with their child, use more effective ways to communicate with their child, set reasonable boundaries and limits (controls and restrictions), reward their child for good behavior, and monitor their child’s activities [ ]. The child component addresses social and personal skills that will help them refuse negative peer pressure to use alcohol and drugs and improve their personal self-management skills. Children also practice ways to increase social-emotional regulation and impulse control and acquire better problem-solving and effective communication skills.
Similar to many other family-based programs, parents and children receive separate instruction in the first hour and then join together to practice and receive feedback on newly learned skills during a second hour. This portion includes supportive role play and games that the parent and child play together, which encourages parents to implement nonjudgmental dialog and increase parent-child attachment. Behavioral reinforcement techniques, such as situational role play and behavioral rehearsal with positive feedback given to families by highly trained facilitators, are a hallmark feature of SFP. The program uses curriculum guides, homework, and workbooks that can be practiced in the home to reinforce lesson material. Providing family meals before each weekly session, offering childcare for young siblings, and assisting with transportation are implemented to address attendance barriers that potentially inhibit participation and retention over time.
Internet-Delivered Parenting Programs
Notwithstanding evidence of their overall effectiveness, family-based programs remain resource-intensive with high costs and logistical demands. Any attempt to scale them for widespread dissemination has to address these implementation demands. In recent years, internet-based behavioral interventions have been making tremendous strides as alternatives to programs that customarily use face-to-face delivery, presenting a promising avenue for delivery [, ]. Web-based programs ensure standardized delivery and implementation fidelity, are convenient for end users, and are self-paced, cost-efficient, and reduce social or personal demands. Some parenting skills programs have taken on this challenge, and a few internet-based programs have produced promising findings underscoring their efficacy [ - ]. Two programs that have successfully migrated to the Web include Familias Unidas [ , ] and Triple P Online [ , ]. Familias Unidas targets parents with goals of preventing substance use, sexual risk behaviors, and sexually transmitted infections in high-risk youth from immigrant Hispanic and Latino families; Triple P is a population-based intervention that targets families with young children with early-stage conduct disorder or behavioral or emotional problems. Both programs use formative evaluation strategies in the process of developing an eHealth prototype. These efforts included focus groups to identify the program features desired by consumers and discussed with families novel ways to structure program content when delivered on a Web platform.
Challenges With Web-Based Delivery
Repackaging group-based, facilitator-led programs for Web delivery faces several formidable challenges. To begin with, a program developer must consider how to preserve the active ingredients of a program as it is transitioned to a Web-based platform. This entails finding ways to transform behavioral practice and positive reinforcement techniques into digital content that preserves the integrity of the intervention strategies. Then developers must consider finding the proper dosage and session length, particularly because internet-based programs are traditionally shorter than in-person, group-based formats. Developers must also find ways to assess exposure, including engagement, and use this information to establish a metric for fidelity of implementation and adherence—two important components related to program efficacy [- ].
Engagement can be defined as the adoption of the program by consumers, and then continued use can be measured through session exposure. In both cases, recruitment to participate and retention throughout delivery in family-based programs have faced several hurdles [- ]. Indeed, reviews of recruitment and retention in family-based programs highlight the problem of attrition, with a host of reasons cited by participants for their noncompliance [ , ]. Reasons for failing to attend or complete the program include that parents may not find the program accessible or they perceive the intervention demands as burdensome (ie, lengthy sessions) or intimidating. In the traditional group setting, additional logistical barriers to participation include transportation to and from the facility, costs associated with childcare if not provided by the program, and time away from family responsibilities [ ]. Scheduling demands can also interfere with participation. In some cases, parents have expressed reluctance to attend because of stigma, loss of privacy, or the appropriateness of the intervention for their essential needs [ , ]. Regardless of their stated reasons, dropping out of the intervention lowers dosage and adversely affects program outcomes, reduces power, and limits external validity.
Involving Users in Program Design
Transparency in the process of meeting these challenges would provide a template to guide future research examining the design and implementation of Web-based parenting skills training and youth drug prevention programs . Participatory action research strategies in the form of consumer preference studies represent one way to address the concerns that arise from the deployment of Web-based programs [ , ]. Consumer preference studies are a mainstay of community-based research, as they seek greater input from consumers in all phases of a program from its early conceptual stages through development, implementation, and even dissemination. There is now growing evidence that consumer preferences can be quite fruitful when designing programs for youth, especially those focusing on mental health [ ]. The strategy of gleaning information from consumers was also the tactic used by the developers of Familias Unidas [ ] and Triple P [ - ] when they solicited consumer’s input before the developers designed a Web-based delivery system for their family programs. By allowing participants to contribute and provide input in the early program design stages as part of “collective making,” consumers feel valued as co-creators [ ]. The goal here is to have the target consumer population take a more vested interest in program development, feel empowered, and take ownership, leading the developer to construct a product that is more reflective of the target consumer’s preferences and needs [ ]. Motivation to attend sessions should be higher among consumers when they feel the session content and delivery format are well designed and congruous with their needs.
This study uses a mixed methods formative evaluation design to explore consumer preferences in the prototype build stage for SFP Online. In choosing to develop an online version, several pressing questions needed to be addressed, including whether consumers felt the group-based format could be transformed in a way to maintain the active ingredients with no loss of fidelity in a digital environment. We also wanted to know what levels of interactivity and what type of program content are needed to offset factors that contribute to noncompliance. We applied both formative and summative strategies to obtain this information before developing and testing a prototype. We also posed specific questions to consumers regarding the feasibility, perceived utility, technology acceptance, desired instructional format (ie, screen layout and navigational features), and other utilization factors that could influence engagement. This type of in-depth feedback is essential during the development process because it can be a foundation for the design of intervention modules and help structure their delivery. Although we solicited and received input from both parents and youth, we only focus on the adult consumer responses in this study.
We also extended prior work by including agency staff (stakeholders) and subject matter experts because they often have tangible insight that can shed light on and give shape to the product’s functionality. Agency staff who routinely work with families offering services may have their finger on the pulse of what families want and need and may also recognize some of the barriers to attendance that affect program outcomes. Experts can weigh in on how to maintain close alignment with the developmental and family-based theories that drive most parenting skills training interventions. As detailed subsequently, a second angle that e-learning experts can weigh in on includes integrating various instructional design principles into a serious educational game to increase the program’s overall attractiveness. In addition to the formative strategies, we used a consumer preference survey administered to focus group participants, which is explained in detail.
shows the study design and gives a brief description of each arm corresponding to the formative and summative evaluation research plan. This study was part of a Phase I NIH-funded Small Business Innovative Research grant (NIDA; R43DA046238-01) and the study was approved by the Heartland IRB located in Belleville, IL (HIRB Project #180213-187) prior to engaging in any scientific work.
|Study arm||Sample size||Focus||Comments|
|Focus groups||85 adults in 11 separate groups||Necessary steps for translation of group-based program to a Web platform||Questions probed active ingredients, interactivity, cellular phone and computer efficacy, proposed content, games, and ways to connect parent and youth lessons|
|Consumer preference survey||85||Barriers and facilitators, computer technology familiarity, desired content and materials, interactivity, and desire for social media connections||45 questions assessing past experience with the Strengthening Families Program assessing whether prior program exposure guides consumer preferences for an interactive, multimedia Web platform|
|Key stakeholder interviews||20 agency staff||Suitability of Web platform for clientele, assessing required changes in content and delivery methods, whether program would be engaging (interactivity) and fit the agency goals||Semistructured interview with questions provided in advance, then teleconference call to expand on answers|
|Expert interviews||10 (5 prevention scientists, 5 e-learning experts)||Two themes: prevention science to bolster core active ingredients of program and e-learning emphasis to include recent technology and its influence on learning and behavior change||Semistructured interview with questions provided in advance, and then phone or video interview used to expand on answers|
We recruited 85 adults from nine geographically dispersed agencies in the US covering the western portion (NM, CA, WA, UT, and NV), the middle portion (KS, IN, and OH), the eastern seaboard (NC and NY), and the south (TX). These agencies were representative of the target agencies where SFP Online could be distributed, and they were willing to participate in the formative evaluation. The agencies conducted the focus groups between May and early June of 2018. Each agency had participated in prior SFP implementation training using the traditional group format or had implemented the DVD curriculum used in conjunction with group classes. An email was sent to all the agencies outlining the scope of the project and the requirements for participation. After the initial contact, we sent a memorandum of agreement to each agency outlining the participant recruitment requirements (emails, phone calls, and posted flyers) and what family participation would entail, including a description of incentives that would be provided to both the individuals and the agency.
We gauged the number of groups (approximately 4 to 10 members in each group) based on recent studies that confirmed thematic sufficiency with nonprobability sampling strategies can be achieved with three to six focus groups  and also accounted for potential dropout after recruitment. The inclusion criteria stipulated that parents (including guardians and caregivers) and youth had attended an SFP class or watched the SFP DVD. This requirement was intended to solicit input from families who had sufficient knowledge of the programs’ core content and could comment on the suitability of an online version, make suggestions for future content, and discuss the utility and feasibility of a Web-based version.
Each focus group lasted between 60 and 90 minutes and involved a moderator; refreshments were provided at the midpoint. Agency staff assisted in obtaining written informed consent, conducting the groups, and collecting survey materials. Moderators were given a seven-page instructional guide outlining how to conduct the group, preserve confidentiality, and solicit balanced input from the group members. Moderator probes queried the parents’ computer technology and mobile phone usage, their video game experience, desired program content, instructional materials, navigational features, interactive components, and whether parents wanted asynchronous social media connections (ie, blogs, chat, discussion board). Members of the research team were present during one of the focus groups and listened to others via GoToMeeting Web conferencing.
Key Informant Interviews
We also conducted 20 key informant semistructured interviews by teleconference with agency staff primarily drawn from the same agencies that participated in the focus group. One exception was a new agency in MA that participated in the interviews. One agency staff from each site participated, except for NY, UT, and KS, where more than one staff member was interviewed. Agency staff had been previously trained to deliver SFP as part of activities distinct from this project; they also provide, as part of their regular duties, additional psychoeducational and support services to the families. Their input was intended to address whether they felt the intended Web-based program could retain the core instructional strategies, how to structure the instructional modalities so they would be engaging, and what technology design features would be appropriate for the target families. Before the phone call, each stakeholder was provided a 20-item questionnaire with specific probes intended to elucidate their involvement with SFP, their organizational role, the demographic profile of the agency clientele (racial composition, income, risk level), their perceived feasibility and utility of implementing SFP Online, and the essential program components and skills training they felt should be included.
We recruited 10 subject matter experts to weigh in on prototype development. Five were prevention scientists who we asked to address the procedures for translating core active ingredients into online programming and ways to ensure engagement. An additional five e-learning experts shared ideas on different ways to capitalize on current trends in multimedia interactive programming and whether this should influence the design of SFP Online. A brief survey was emailed to the experts before the teleconference interview, and their written responses helped guide the interview. The first part included 10 brief questions intended to elucidate their experience designing interactive programs. This was followed by 19 interview probes addressing the feasibility of SFP Online, implementation strategies, use of live family coaches, social media, interactive activities to promote family problem-solving skills, dashboard and navigation features that might be attractive, methods to stimulate engagement and increase session exposure, and perceived barriers or problems when implementing eHealth programs (eg, recruitment, retention, session length, interactivity, and structuring programmatic content).
Consumer Preference Survey
Parents in the focus groups also filled out a 45-item consumer preference survey. Following completion, the anonymous surveys were collected by agency staff; placed in sealed, preaddressed envelopes; and mailed to the research team.shows the survey items (predictors) that were used in the summative analyses to examine consumer preferences. Additional outcome measures are explained subsequently. Fixed-choice or dichotomous yes or no response formats were used for some questions; the balance used five-point Likert response formats.
|Survey question||Response formata|
|How many Strengthening Families Program (SFP) DVD lessons did you watch?||0, 1-3, 4-6, 7-9, 10-11|
|How comfortable are you using a computer?||Very, somewhat, not very|
|Would you prefer doing the SFP course online, or would you rather attend a class?||Online, attend class, undecided|
|Would you prefer to track your skills practice on your computer instead of a paper handout?||Yes, maybe, probably not, no|
|Are you interested in using a game-like online version of SFP?||Yes, maybe, not really, not interested at all|
|Would you practice the skills at home without a live family coach to remind you?||Yes, no, maybe, maybe with points|
|Do you have access to a computer that connects to the internet?||Yes/no|
|Would you use SFP Online even if you also took a class?||Yes, maybe, probably not, not|
|Do you have a mobile or smartphone where you could view SFP Online?||Yes/no|
|If you have a smartphone, would you view SFP Online on your phone?||Yes/no|
|If you could access SFP Online, how likely are you to record your home practice assignments?||Likely, somewhat likely, somewhat unlikely, unlikely|
|How many lessons should be included in SFP Online?||1-5, 6-10, 11-15|
|How long should each individual lesson be?||6-8, 9-12, 13-20, 21-30, >30 minutes|
|Would including fun games in SFP Online help your family learn new skills?||Yes, maybe, probably not, no|
|Would you like family members to earn reward points after completing SFP assignments?||Yes, maybe, probably not, no|
|Which of the following would you prefer in a family game?||Points to reward progress, experience points to move up a level in game, both|
|Would you like SFP Online games to include a family competition for points when you practice the skills?||Yes/no|
|Do you think you would practice the skills at home without a live family coach to remind you?||Yes, no, maybe, maybe, if we earned points for doing it|
aSome variables recoded to avoid sparse cells. Age, gender, and race/ethnicity also asked. Race/ethnicity categories included African American or black, Asian, American Indian, Pacific Islander or Native Hawaiian, Alaska Native, white, Hispanic or Latino, and more than one race.
Qualitative Coding Schemes
Digital audiotapes from the focus groups and interviews were professionally transcribed verbatim, and the content of this material was then examined using thematic analysis as a research tool. This approach breaks down the open-ended answers into smaller units so the researcher can actively identify consistent and meaningful patterns in the text . We used a data-driven inductive approach similar to grounded theory [ , ] to quantify themes based on explicit content analysis. For example, thematic content from the focus groups might reference the word “games” used in the context of programmatic features that will attract youth and contribute to engagement. Likewise, stating the need for interactive components modeled after a coach would be considered a theme (ie, “the program needs a coach”). Frequency of how often a word or phrase appeared in different contexts provided a quantified metric of thematic sufficiency. We repeated this coding procedure across all groups with two members of the investigative team descriptively tabulating key terms and phrases verbatim and then synthesizing across groups until thematic saturation was obtained. The coders read the transcripts and compiled their thematic results independently, and then convened for discussion of emerging themes. The results would then be summarized and tabulated into a product specification plan, a written document presented to the product design team before the alpha prototype build.
Consumer Preference Survey: Outcome Measures
The survey included additional items assessing issues that might arise in using SFP Online, which were modeled as outcomes. Seven items assessed technology barriers that might present concerns when using SFP Online, including data plans, visual content, screen size, and navigation, with responses coded on a five-point Likert scale ranging from not at all concerned to extremely concerned. An additional 20 items assessed what subject material parents felt was important to include in the online version (eg, brain development, mindfulness, communication skills, giving compliments, and monitoring) with responses coded on a five-point Likert scale ranging from very important to not at all important. An additional 16 items assessed the importance of different program components for learning parenting skills (eg, reporting progress, short videos, family goals, virtual coach, and tracking progress) with responses coded on a five-point Likert scale ranging from very important to not at all important. A 10-item scale assessed reasons for using SFP Online (eg, self-paced, review lesson materials, monitoring family progress, and tracking family performance) with responses coded on a five-point Likert scale ranging from very important to not at all important. A single item asked parents if they were interested in a game-like version of SFP, with responses recoded as yes or no.
Both factor analysis and estimates of internal consistency indicated low reliability and lack of conceptual purity in the outcome scales. This is likely because parents treated each subset of items as a checklist, and this prevented us from obtaining homogenous unidimensional scales. As a result, we created unit-weighted indexes to represent the different technological, logistic, programmatic, and motivational reasons for using SFP Online. We dichotomously recoded the Likert response scales to 1 or 0 by collapsing “not at all concerned” and “slightly concerned” to 1 and coded the remaining response categories to 0. Likewise, we collapsed “very important” and “important” to 1 and all other responses to 0. For all analyses, the significance level was set at P<.05 with two-sided tests.
Adult Focus Group Characteristics
Eighty-five parents attended the 10 focus groups (77% female, 65/85). Their age breakdown included 11 parents (13%) between 18 and 31 years, 43 parents (51%) between 32 and 45 years, 19 parents (22%) between 46 and 52 years, 10 parents (12%) between 53 and 66 years, and 2 grandparents (2%) 67 years and older. In all, 56 (66%) were white, 8 (9%) were Native American or American Indian, 5 (6%) were Hispanic or Latino, 4 (5%) were African American, 9 (9%) were mixed race, 2 (2%) were Asian, and 1 (1%) indicated “other.” The family service agencies served a fairly homogeneous clientele, who were relatively poor, socially marginalized, experiencing family distress, and characteristically low in education. Therefore, we did not ask about their socioeconomic status using income or education.
Of the 85 parents, 79 (93%) said they had computer access with internet connection, and 69 (82%) said they were comfortable using computers, 11 (13%) said they were somewhat comfortable, and 5 (6%) said they were not comfortable. As anticipated, all the parents had some exposure to the SFP traditional group-based classes. Twenty-nine (34%) took a class with their family. Twenty (24%) viewed the SFP DVD video clips in a class setting, 21 (25%) viewed the DVD at home, and 11 (13%) viewed the SFP DVD with a coach. Thirty-three (39%) parents attended between zero and three lessons, 11 (13%) attended between four and six lessons, and 40 (48%) attended more than six lessons. Twenty-two (26%) parents said they preferred an online class, 35 (41%) said they wanted to attend an in-person class, and the remaining 23 (27%) were undecided between the two. Seventy-four (87%) parents said they would view SFP Online if it was available on a smartphone. When asked about including games in the new online version, 4 parents (4%) said they were not at all interested in games, 10 (12%) said they were slightly interested, 41 (48%) said they might be interested, and 30 (35%) said they would be very interested.
A total of 241 keywords were culled from the thematic content analysis of the focus groups and interviews. Two coders conducted the content analysis and reduced the pool of open-ended answers to 86 distinct keywords. An example of a word that parents provided that was highly prevalent was “activities,” and the corresponding commentary included “parent-and-child activities to do together.” Another example was “challenges,” and the commentary included “ability to customize challenges” and “challenges that spark our interest.” We then tallied the keywords by frequency of their use with the most popular terms, which included “games” (n=55), “unique” (n=40), “fun” (n=15), “challenges” (n=14), “points” (n=11), and “rewards” (n=9). The remaining terms had frequencies in the single digits. Using an inductive procedure, we then summarized these terms into three major categories: program content, logistics, and interactivity/engagement. Program content encompassed teaching points and lesson plans (eg, discussing risk and protective factors for drug use, practicing family management skills, and teaching families how to create strong bonds), and referenced specific SFP core activities (eg, communication boulders, mindfulness, and automatic negative thoughts).
Logistics encompassed program delivery features, such as what will make SFP Online engaging (technology and navigation control) and encourage future continued use (eg, natural-sounding voices, customization and feedback, accessibility on multiple devices, animation, and realistic role play). Interactivity and engagement encompassed multimedia features of the program (eg, avatars, chat or threaded discussion boards, expert coaching, pop-up notifications, and weekly expert blogs).to provide several examples corresponding to each category created from the three different formative evaluation strategies.
Consumer Preference Survey: Gender, Race, and Age Differences
There were no gender differences in the categorical measures assessing parents’ SFP course exposure, technology readiness, or course content preferences. Younger parents were more likely to want to view SPF Online using their mobile phone (χ21=4.0, P=.04, φ=−0.22), were more likely to want to track their skills practice on a computer (χ22=8.4, P=.01, φ=0.31), and were more likely to want a coach (χ22=7.6, P=.02, φ=0.29). White parents (compared with all other race/ethnic groups) viewed more SFP lessons (χ22=7.1, P=.03, φ=0.29) and preferred doing the classes online (χ22=6.7, P=.03, φ=0.28). There were no significant race/ethnicity, age, or gender differences for desiring a gamified version of SFP Online. There were also no significant demographic differences in the mean scores for the four outcome measures (ie, concerns about online use, desired components of online program, level of interactivity, and reasons to use an online program).
shows the complete results of the regression models with the three blocks of predictors and four outcome measures. Three distinct blocks were used to model predictors of the four outcome measures. Each block was configured to include between four and five predictors to create balance. The first block contained the number of lessons the parent viewed using the DVD, comfort using computers, whether they preferred taking SFP online or attending a class, whether they wanted to track their skills practice on a computer or a handout, and whether they would practice the newly learned skills at home without a coach. Comfort using computers was significantly associated with the index summing concerns accessing SFP Online (F2=5.34, P=.006). Parents who reported being very comfortable using computers expressed fewer concerns (mean 1.85, SE 0.42) compared to parents reporting they were somewhat comfortable (mean 3.03, SE 0.64) or not very comfortable (mean 3.68, SE 0.90), albeit these differences were not significantly different from each other by the Tukey-Kramer post hoc multiple comparison test.
The second block included whether parents had access to a computer with internet service, whether they would use SFP Online even if they had a class, if they had a mobile or smartphone to view SFP Online, would they view SFP Online if they had a smartphone, and whether they would record their home practice assignments on a computer or smartphone. Parents with access to a mobile phone reported significantly fewer concerns (F1=10.94, P=.001; those owning a phone: mean 2.23, SE 0.63; those not owning one: mean 5.34, SE 1.07). Those wanting to view SFP on their mobile phone reported fewer concerns (F1=4.76, P=.03, mean 2.97, SE 0.72) for parents who could view SFP on a mobile phone compared with those who could not view SFP on a mobile phone (mean 4.59, SE 0.82). The third block included a measure asking parents how many lessons should be included in SFP Online, the proposed length of the individual lessons, whether fun games would help their family learn new skills, and whether family members should earn reward points when engaging with the online program. None of the individual predictors were significantly associated with technological concerns using SFP Online.
The next model examined the associations between the same three blocks and an index assessing important programmatic components to include in SFP Online. For the first block, ease of using computers was the only significant measure in the model (F2=4.07, P=.02). The post hoc multiple comparison showed that parents expressing some comfort using computers wanted significantly fewer lesson components (mean 16.67, SE 0.68) compared with parents feeling very comfortable (mean 18.53, SE 0.44). Parents wanting to view SFP Online using a mobile phone wanted more online components (F1=10.46, P=.002; mean 20.19, SE 0.75) compared with parents who did not want to access the online program using a mobile phone (mean 18.24, SE 0.86). No predictors in the third block were individually significant.
The third index captured desired interactive components to include in the new online program. In the first block, the number of SFP lessons completed on the DVD was significant (F2=5.87, P=.004). Parents who completed an intermediate number of lessons (4 to 6) wanted more interactive components (mean 12.31, SE 1.32) compared with parents who watched the most lessons (>6 lessons or more: mean 10.98, SE 0.98) and parents who watched the fewest lessons (0-3 lessons: mean 8.59, SE 1.04). The second block included one significant measure; whether parents desired to take a class even if they took SFP Online was significant (F3=4.58, P=.005). Parents desiring to use SFP Online even with a class wanted more interactive components (mean 12.07, SE 1.31) compared with parents stating “no” (mean 6.18, SE 2.74, P=.09), although these post hoc comparisons were not significant.
The final block included three significant predictors assessing the number of proposed lessons for the new online program (F2=4.14, P=.02), wanting SFP Online to include fun games (F2=5.01, P=.009), and wanting rewards built into the program (F3=2.82, P=.04). Parents viewing between 1 and 5 lessons wanted fewer interactive components (mean 10.18, SE 1.63) compared with parents viewing between 6 and 10 lessons (mean 12.95, SE 1.27). Parents stating “yes” to wanting fun games also reported wanting more interactivity (mean 12.82, SE 1.13) compared with those stating “maybe” (mean 10.53, SE 1.09).
The final model included the same three blocks individually predicting reasons (motivations) to use SFP Online. In the first block, only tracking SFP skills practice on the computer (versus handouts) was significant (F2=4.38, P=.02). Parents not wanting to track their skills practice reported significantly fewer reasons to use SFP Online (mean 6.48, SE 0.81) compared with those saying “yes” (mean 8.17, SE 0.53) or “maybe” (mean 8.30, SE 0.47). In the second block, only wanting to take SFP Online in conjunction with a class was a significant predictor (F3=5.11, P=.002). Parents stating they would probably not do the online class in addition to a group class reported significantly fewer reasons to use SFP Online (mean 4.56, SE 1.17) compared with those stating “maybe” (mean 8.49, SE 0.67) and “yes” (mean 8.77, SE 0.69). In the third block, wanting rewards while using the new online program was significant (F3=5.17, P=.003). Parents who said they probably did not want rewards through the online program reported significantly fewer reasons to use SFP Online (mean 5.05, SE 0.99) compared with parents who did want rewards (mean 8.05, SE 0.47).
We then repeated the block entry procedure using logistic regression with a binary measure asking parents whether they were interested in using a game-like version of SFP (response categories “maybe” and “yes, very interested” combined to 1; “not really” and “not interested at all” combined to 0 as the reference category). To avoid convergence problems (maximum likelihood estimates that were not trustworthy), we examined each predictor one by one and collapsed response categories for the measures, reducing the number of cells in the estimation process. We then combined all the significant predictor measures culled from the individual models into a single model, controlling for race and age. Based on the individual models, parents who preferred taking SFP online (attend class was designated the reference or comparison class) were almost nine times as likely to want a game-like experience than those who were undecided (likelihood ratio [LR] χ22=8.8, P=.01; OR 8.88, 95% CI 1.06-74.48) to want a game-like experience. Those parents who were undecided were five times as likely (OR 5.10, 95% CI 1.02-25.29) to want a game-like experience compared with the reference class. Unadjusted, the C-statistic (a variant of Somers’ D) was 0.719, and the rescaled R2 (coefficient of determination) accounted for 16.6% of the variance.
Parents reporting they were very comfortable using computers were eight times as likely to be interested in a game-like online version of SFP (LR χ22=8.8, P=.01; rescaled R2=16.6%, C-statistic=0.654; OR 7.90, 95% CI 1.18-52.97) compared with those reporting they were not at all comfortable using computers. Parents with mobile phones on which they could view SFP Online were 11 times as likely to want a game-like online version than parents lacking mobile phone access (LR χ21=3.96, P=.046; OR 11.66, 95% CI 0.98-138.92; rescaled R2=8%, C-statistic=0.564). Parents who had a smartphone were six times as likely to want fun games than parents who did not have smartphones (LR χ21=6.1, P=.01; OR 6.02, 95% CI 1.52-23.84; rescaled R2=12%, C-statistic=0.636). Parents who stated they would be likely or somewhat likely to record home practice assignments on their computer were 78% more likely to want a game-like version compared with parents who reported they would be somewhat unlikely and unlikely (LR χ21=6.3, P=.01; OR 0.22, 95% CI 0.07-0.72; rescaled R2=12%, C-statistic=0.673).
A model with all the individually significant predictors combined fit well (LR χ26=23.2, P<.001; rescaled R2=40%, C-statistic=0.846, Akaike information criterion=66.84, Hosmer-Lemeshow goodness-of-fit test, χ25=1.0, P=.96). When all six predictors were modeled at once, only the measure of preferring the online version to a class was significant (LR χ22=6.4, P=.04; OR 16.27, 95% CI 1.49-177.02); adjusting for race and age did not improve the model fit.
Thematic content analysis indicated three key areas of concern raised by all study participants when asked about designing an online version of the SFP: (1) desired content of the program, (2) logistics affecting program delivery, and (3) interactive features that can stimulate engagement. Interestingly, these themes reflected commonality in how consumers, agency stakeholders, and experts view the essential components of an online parenting skills training program that also involves youth drug prevention. Importantly, experts emphasized features consistent with cognitive behavioral principles, including skills practice, positive feedback, low-stakes failure exercises, and using reward structures (ie, proficiency “bars”) that help families move toward various programmatic benchmarks. They also outlined several implementation challenges, including program length and finding ways to use multimedia and interactive features to reinforce the focus and core active ingredients of the program.
Experts also felt the online program content should be realistic, simple, and use video examples to teach problem-focused coping skills, reflect the body of literature on risk and protective factors, and provide parents with ready solutions to their problems. The parents, in particular, stated that all this should be crafted using animation that can exemplify role-playing skills and reinforce lesson content. Experts also felt that SFP Online should stimulate “authentic” collaboration between parent and child and offer coaching, discussion boards, and virtual interactions—all with the goal of increasing engagement to maintain high exposure over time.
Agency staff agreed in principle with the experts and felt that the online version should maintain a skills-oriented approach and be configured around real family experiences. Given their familiarity delivering the group format of SFP, they emphasized preserving many of the current fun activities that work well in the group setting. From a logistical standpoint, they listed numerous features that would enhance SFP Online, including shorter sessions, graphics emphasized over text, dashboards for the entire family, text message reminders, home practice materials, printable worksheets, and a means for agency personnel to track family progress. They also mentioned that parents may still want to meet in person with a facilitator, even if they had online access, and that boosters might be essential to reinforce program content over time.
Parents felt very enthusiastic about helping to design the program content and delivery format for the online version. They wanted shorter sessions and state-of-the-art navigation features, with pop-up notifications, rewind capabilities, drop-down screens and scrollable information, graphic display with virtual animation, and realistic role plays. The parents who had the most prior exposure to SFP provided the most feedback on program components that they believed could work online. They felt the program should include bulletin boards, quizzes, pause-and-practice features, proficiency rewards, weekly blogs from experts, and virtual role plays with instruction. They also felt the online version should appeal to a culturally diverse population.
The consumer preference survey augmented the inductive procedures by adding more fine-grained information that helped to clarify what factors are associated with technological barriers, course content, interactivity, and motivations to use the online version. Interestingly, technology-savvy parents expressed fewer concerns about using an online program and showed a clear preference for interactivity and designing SFP Online like a serious educational game. The parents reluctant to track their performance online and those not wanting performance rewards were less inclined to want a Web-based platform to learn parenting skills. Mobile phone access was ubiquitous in this sample, underscoring the narrowing of the digital divide in terms of access to mobile technology and the potential for eventually creating a mobile app to complement SFP Online . In all models tested, there was no evidence that any differences in the outcome could be attributed to race, gender, or age.
Even though the qualitative portion of the study reinforced tremendous overlap between the different groups, there were also unique contributions expressed by each group. For instance, agency stakeholders desired to find ways of connecting parent and child in the Web platform as this could be a strength of SFP Online. In addition, agency stakeholders were less inclined to address animation as crucial to the program, and more inclined to address interactive components that reinforce behavior change (ie, downloads, FAQs, discussion boards) and the use of technology to improve compliance (eg, text reminders). In this case, the focus of stakeholders appeared vested more in creating teachable moments and less focused on using technology to increase consumer involvement.
Strengths and Limitations
Several limitations should be mentioned. The data are cross-sectional, and we do not know, for example, whether any of the computer and mobile phone use questions actually predict subsequent use of SFP Online. Moreover, the purposive sampling intentionally recruited parents with prior exposure to SFP, including having used the DVD or participating in the group format. This can introduce bias because many of the parents who attended classes but desired an online version wanted the option to attend a class based on their prior exposure. Having the opportunity to receive immediate in-person feedback on their performance may be an attraction that is hard to duplicate on the Web. In addition, the small sample size altered the model testing sequence to avoid overfitting models. As a result, we could not test incremental variance attributed to each block of predictors. Related to this, some of the statistical relations may not have achieved significance merely because the study was underpowered. This problem underscores the trade-off between obtaining qualitative data from a small but sufficient number of focus groups and wanting a large enough and appropriately powered sample to produce robust parameter estimates. Notwithstanding, there is a wealth of information gathered using a mixed methods approach, and we certainly benefited from employing these different strategies.
One strength of this study is that we solicited collaborative input from prospective consumers as part of an active partnership preceding the development of the online program. This made participants feel valued and that they could contribute to program development in a meaningful way.
Curtailing the rising tide of addiction in America requires taking evidence-based family prevention programs to scale, which can best be achieved by providing high-quality, low-cost, online delivery. Creating an effective, online family-based prevention tool that parents and youth will enjoy and use requires knowledge of what parents and youth want to learn, how they want to acquire this knowledge, and what factors will keep them engaged so they can develop the appropriate skills and complete the course.
The authors would like to thank all the family service agencies that participated in the focus groups and assisted with the consumer preference surveys. We also want to thank the families who participated in this project, making it a collective effort that will ensure the final product is both engaging and addresses their parenting and child's developmental needs. This paper is dedicated to the memory of Steven P Schinke, a giant both in physical presence and in the field of prevention science. He was an inspiration and a true gentleman.
Conflicts of Interest
JLB and KLK have a principal financial interest in SFP 7-17. They receive financial remuneration from training group leaders, sales of the curriculum and evaluation contracts. Neither party has a financial interest in the SFP DVD, which is wholly owned by the Strengthening Families Foundation, a 501(c)3 not-for-profit corporation. All profits from sales of the DVD belong to the Foundation and no proceeds from sales are distributed in the form of wages or salary compensation. Further, neither author receives royalties from the SFP DVD. LMS is employed by LARS Research Institute, a not-for-profit company that offers professional consulting services in program evaluation, program development, and behavioral science technology transfer. Q-QH is a doctoral candidate in the Division of Public Health, University of Utah, and has no financial interest in the SFP product line.
Results of consumer preference survey regression models.PDF File (Adobe PDF File), 258 KB
- Kumpfer K, Hansen W. Family-based prevention programs. In: Scheier L, Hansen W, editors. Parenting and Teen Drug Use: The Most Recent Findings from Research, Prevention, and Treatment. New York, NY: Oxford University Press; 2014:166-192.
- Lochman J, van den Steenhoven A. Family-based approaches to substance abuse prevention. J Prim Prev 2002;23(1):49-114. [CrossRef]
- Forgatch MS, Patterson GR, Gewirtz AH. Looking forward: the promise of widespread implementation of parent training programs. Perspect Psychol Sci 2013 Nov;8(6):682-694 [FREE Full text] [CrossRef] [Medline]
- Allen ML, Garcia-Huidobro D, Porta C, Curran D, Patel R, Miller J, et al. Effective parenting interventions to reduce youth substance use: a systematic review. Pediatrics 2016 Aug;138(2):195 [FREE Full text] [CrossRef] [Medline]
- Baumrind D. Rearing competent children. In: Damon W, editor. Child Development Today and Tomorrow. San Francisco, CA: Jossey-Bass; 1989:349-378.
- Maccoby EE, Martin JA. Socialization in the context of the family: parent-child interactions. In: Mussen PH, Hetherington EM, editors. Handbook of Child Psychology: Volume 4. Socialization, Personality, Social Development. 4th edition. New York: John Wiley and Sons; 1983:1-101.
- Bell RQ. Parent, child, and reciprocal influences. Am Psychol 1979;34(10):821-826 [FREE Full text] [CrossRef]
- Bronfenbrenner U. Ecology of the family as a context for human development: research perspectives. Dev Psychol 1986;22(6):723-742. [CrossRef]
- Patterson GR. Coercive Family Process. Eugene, OR: Castalia Pub Co; 1982.
- Sameroff AJ. The Transactional Model of Development: How Children and Contexts Shape Each Other. Washington, DC: American Psychological Association; 2009.
- DeMarsh JP, Kumpfer KL. Family-oriented interventions for the prevention of chemical dependency in children and adolescence. J Child Contemp Soc 1985;18(1-2):117-151 [FREE Full text] [CrossRef]
- Kumpfer K, Demarsh J. Family environmental and genetic influences on children's future chemical dependency. J Children in Contemp Soc 1986 Nov 21;18(1-2):49-91 [FREE Full text] [CrossRef]
- Kumpfer KL, Magalhães C, Whiteside H, Xie J. Strengthening families for middle/late childhood. In: Van Ryzin M, Kumpfer KL, Fosco G, Greenberg M, editors. Family-Based Prevention Programs for Children and Adolescents: Theory, Research, and Large-Scale Dissemination. New York, NY: Psychology Press; 2016:68-86.
- Kumpfer KL, Whiteside HO, Greene JA, Allen KC. Effectiveness outcomes of four age versions of the Strengthening Families Program in statewide field sites. Group Dynamics: Theory, Research, and Practice 2010;14(3):211-229. [CrossRef]
- Kumpfer KL, Alvarado R, Tait C, Turner C. Effectiveness of school-based family and children's skills training for substance abuse prevention among 6-8-year-old rural children. Psychol Addict Behav 2002 Dec;16(4, Suppl):S65-S71. [CrossRef]
- Gottfredson DC, Kumpfer KL, Polizzi-Fox D, Wilson D, Puryear V, Beatty P, et al. The Strengthening Washington DC Families project: a randomized effectiveness trial of family-based prevention. Prev Sci 2006 Mar;7(1):57-74. [CrossRef] [Medline]
- Kumpfer KL, Magalhães C, Xie J. Cultural adaptation and implementation of family evidence-based interventions with diverse populations. Prev Sci 2017 Aug;18(6):649-659. [CrossRef] [Medline]
- Kumpfer KL, Pinyuchon M, de Melo AT, Whiteside HO. Cultural adaptation process for international dissemination of the Strengthening Families Program. Eval Health Prof 2008 Jun;31(2):226-239. [CrossRef]
- Bröning S, Baldus C, Thomsen M, Sack P, Arnaud N, Thomasius R. Children with elevated psychosocial risk load benefit most from a family-based preventive intervention: exploratory differential analyses from the German. Prev Sci 2017 Nov;18(8):932-942. [CrossRef] [Medline]
- Skärstrand E, Sundell K, Andréasson S. Evaluation of a Swedish version of the Strengthening Families Programme. Eur J Public Health 2014 Aug;24(4):578-584. [CrossRef] [Medline]
- Brook J, McDonald TP, Yan Y. An analysis of the impact of the Strengthening Families Program on family reunification in child welfare. Child Youth Serv Rev 2012 Apr;34(4):691-695. [CrossRef]
- Johnson-Motoyama M, Brook J, Yan Y, McDonald TP. Cost analysis of the strengthening families program in reducing time to family reunification among substance-affected families. Child Youth Serv Rev 2013 Feb;35(2):244-252. [CrossRef] [Medline]
- Forehand RL, McMahon RJ. Helping The Noncompliant Child: A Clinician's Guide To Parent Training. New York, NY: The Guilford Press; 1981.
- Guerney BG, Coufal J, Vogelsong E. Relationship enhancement versus a traditional approach to therapeutic/preventative/enrichment parent-adolescent programs. J Consult Clin Psychol 1981 Dec;49(6):927-939. [CrossRef] [Medline]
- Bowen M. Alcoholism as viewed through family systems theory and family psychotherapy. Ann NY Acad Sci 1991 Apr;1:94-102. [CrossRef]
- Bandura A. Social Learning Theory. Englewood Cliffs, NJ: Prentice-Hall; 1977.
- Van Ryzin KL, Kumpfer KL, Fosco G, Greenberg M. Family-Based Prevention Programs for Children and Adolescents: Theory, Research, and Large-Scale Dissemination. New York, NY: Psychology Press; 2016.
- Bennett GG, Glasgow RE. The delivery of public health interventions via the internet: actualizing their potential. Annu Rev Public Health 2009 Apr;30(1):273-292. [CrossRef]
- Lustria ML, Cortese J, Noar SM, Glueckauf RL. Computer-tailored health interventions delivered over the web: review and analysis of key components. Patient Educ Couns 2009 Feb;74(2):156-173. [CrossRef]
- Estrada Y, Molleda L, Murray A, Drumhiller K, Tapia M, Sardinas K, et al. eHealth Familias Unidas: pilot study of an internet adaptation of an evidence-based family intervention to reduce drug use and sexual risk behaviors among Hispanic adolescents. Int J Environ Res Public Health 2017 Mar 04;14(3):264 [FREE Full text] [CrossRef] [Medline]
- Prado G, Estrada Y, Rojas LM, Bahamon M, Pantin H, Nagarsheth M, et al. Rationale and design for eHealth Familias Unidas Primary Care: a drug use, sexual risk behavior, and STI preventive intervention for hispanic youth in pediatric primary care clinics. Contemp Clin Trials 2019 Jan;76:64-71 [FREE Full text] [CrossRef] [Medline]
- Metzler CW, Sanders MR, Rusby JC, Crowley RN. Using consumer preference information to increase the reach and impact of media-based parenting interventions in a public health approach to parenting support. Behav Ther 2012 Jun;43(2):257-270 [FREE Full text] [CrossRef] [Medline]
- Sanders MR, Kirby JN. Consumer engagement and the development, evaluation, and dissemination of evidence-based parenting programs. Behav Ther 2012 Jun;43(2):236-250 [FREE Full text] [CrossRef] [Medline]
- Yap M, Jorm A, Bazley R, Kelly C, Ryan S, Lubman D. Web-based parenting program to prevent adolescent alcohol misuse: rationale and development. Australas Psychiatry 2011 Aug;19(4):339-344. [CrossRef] [Medline]
- Breitenstein SM, Gross D, Christophersen R. Digital delivery methods of parenting training interventions: a systematic review. Worldviews Evid Based Nurs 2014 Jun;11(3):168-176. [CrossRef] [Medline]
- Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res Nurs Health 2010;33(2):164-173. [CrossRef]
- Byrnes HF, Miller BA, Aalborg AE, Plasencia AV, Keagy CD. Implementation fidelity in adolescent family-based prevention programs: relationship to family engagement. Health Educ Res 2010 Aug;25(4):531-541 [FREE Full text] [CrossRef] [Medline]
- Giles S, Jackson-Newsom J, Pankratz MM, Hansen WB, Ringwalt CL, Dusenbury L. Measuring quality of delivery in a substance use prevention program. J Prim Prev 2008 Nov;29(6):489-501. [CrossRef] [Medline]
- Heinrichs N, Bertram H, Kuschel A, Hahlweg K. Parent recruitment and retention in a universal prevention program for child behavior and emotional problems: barriers to research and program participation. Prev Sci 2005 Dec;6(4):275-286. [CrossRef] [Medline]
- Smokowski P, Corona R, Bacallao M, Fortson BL, Marshall KJ, Yaros A. Addressing barriers to recruitment and retention in the implementation of parenting programs: lessons learned for effective program delivery in rural and urban areas. J Child Fam Stud 2018 Jun 16;27(9):2925-2942. [CrossRef]
- Prinz RJ, Smith EP, Dumas JE, Laughlin JE, White DW, Barrón R. Recruitment and retention of participants in prevention trials involving family-based interventions. Am J Prev Med 2001 Jan;20(1 Suppl):31-37. [Medline]
- Mendez JL, Carpenter JL, LaForett DR, Cohen JS. Parental engagement and barriers to participation in a community-based preventive intervention. Am J Community Psychol 2009 Sep;44(1-2):1-14. [CrossRef] [Medline]
- Axford N, Lehtonen M, Kaoukji D, Tobin K, Berry V. Engaging parents in parenting programs: lessons from research and practice. Child Youth Serv Rev 2012 Oct;34(10):2061-2071 [FREE Full text] [CrossRef]
- Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med 2009 Aug;38(1):28-39 [FREE Full text] [CrossRef] [Medline]
- Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy 2002 Aug;61(2):213-236. [CrossRef]
- Whyte WF, Greenwood DJ, Lazes P. Participatory action research. Am Behav Sci 2016 Jul 27;32(5):513-551 [FREE Full text] [CrossRef]
- Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Human Factors 2015 Jul 09;2(2):e12. [CrossRef]
- Sanders EB, Stappers PJ. Co-creation and the new landscapes of design. CoDesign 2008 Mar;4(1):5-18. [CrossRef]
- Sanders MR, Kirby JN. Surviving or thriving: quality assurance mechanisms to promote innovation in the development of evidence-based parenting interventions. Prev Sci 2014 Mar 8;16(3):421-431. [CrossRef]
- Guest G, Namey E, McKenna K. How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field Methods 2017;29:3-22 [FREE Full text] [CrossRef]
- Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval 2006 Jun 01;27(2):237-246. [CrossRef]
- Glaser B, Strauss AL. The Discovery Of Grounded Theory: Strategies For Qualitative Research. Chicago, IL: Aldine; 1967.
- Strauss J, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Newbury Park, CA: Sage; 1990.
- Jackson LA, Zhao Y, Kolenic A, Fitzgerald HE, Harold R, Von Eye A. Race, gender, and information technology use: the new digital divide. Cyberpsychol Behav 2008 Aug;11(4):437-442. [CrossRef]
|NIDA: National Institute of Drug Abuse|
|NIH: National Institutes of Health|
|SFP: Strengthening Families Program|
Edited by G Eysenbach; submitted 01.06.19; peer-reviewed by A Nguyen, K Mummery, J Brož; comments to author 16.07.19; revised version received 09.09.19; accepted 24.09.19; published 05.11.19
©Lawrence Matthew Scheier, Karol L Kumpfer, Jaynie Litster Brown, QingQing Hu. Originally published in JMIR Formative Research (http://formative.jmir.org), 05.11.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.