Phase 1 used an iterative process involving consultation and two implementations of the classroom sessions, with the second implementation serving as a pilot study. After each iteration, the classroom sessions were revised.

Three changes to program materials were made in preparation for phase 1. First, session slide sets were revised to ensure representation of a variety of cultural/ethnic groups. Second, content on narrative writing was added to the final “Wrap-up & Review” session. Third, ordering of the Avoid Skill and Refuse Skill sessions was reversed relative to their positioning in the original program. This last change was driven by insights from implementing the original program [10]. Both sessions were adapted from an efficacious, early-adolescent substance use prevention program, “keepin’ it REAL” [14], which stands for the relationally competent resistance communication skills Refuse, Explain, Avoid, and Leave. Only Refuse and Avoid were taught in Mighty Girls because formative work indicated that Explain could elicit more peer pressure by inviting peers to argue [21], and Leave could not be used in DRAMA-RAMA.

Consultants included a female middle school staff member and three adult women in their early 20s (two African American women and one biracial African American/Native American woman). The latter were recruited as research assistants through Indeed, a worldwide employment website for job listings. These young women all lived in the same low-income, multicultural, urban metropolitan area in which the study was being performed. We specifically empowered them by emphasizing during recruitment and task directions that they brought a unique and valuable expertise critical to the success of the research effort (this message was repeated throughout phase 1 and phase 2 for all tasks involving these women). They were tasked with reviewing the Mighty Girls classroom session program manual and slide sets for language, terminology, and image inclusivity (ie, not heterosexist or specific to a particular ethnic group) and making recommendations.

Participants in both implementations were girls enrolled in the 7th grade (aged 11-14 years) at two multiethnic middle schools in an urban metropolitan area within central Florida. No participant had difficulty speaking, reading, or writing in English, and on average used only English, or English more than other languages, for these activities (ie, high level of acculturation). Participants of implementation 1 were recruited at a low-income (67% reduced/free lunch, in accordance with criteria to be designated as a “Title I” school: 65% of students/families qualify directly for supplemental nutrition assistance program, homeless, migrant, Medicaid, or foster care as verified by the state) middle school with 80% minority enrollment. Two girls dropped out after session 4 due to repeated disruptive behavioral issues. Participants of implementation 2 were recruited at a second low-income (52% reduced/free lunch) school with 69% minority enrollment (top 5% of state schools for enrollment diversity). Participants at both schools were recruited with: (1) flyers posted at the school; (2) a message sent out via phone by the principal to parents of 7th grade girls; and (3) the three research assistants discussing the study and handing out study packets during 7th grade lunch (packets contained the principal’s letter of support, parental consent forms, and screening form assessing parents’ culture/ethnicity and child’s English language fluency).

Each implementation involved a two-person multicultural teaching team consisting of either the first or third author in the teaching role and one of the three research assistants in the teaching assistant role (ie, Hispanic or non-Hispanic White teacher paired with African American or multiethnic [African American/Native American] teaching assistant). This team also acted as participant observers who made postsession notes about the session they had taught, identifying what worked and did not work, and providing feedback on the cultural relevance of the wording and examples used to explain concepts.

Teaching assistant role training involved completing session activities in the role of a participant, and provided an opportunity to try out a training approach that combined two 2-hour face-to-face sessions with a “homework” assignment similar to what might be used for training when the program is taken to scale. All three research assistants spontaneously reported how much they liked their training, particularly the “homework” (using communication techniques with friends and family).

Both implementations were videotaped. The first implementation involved postsession focus groups, and the second was a one-group, pretest-posttest pilot study (not powered for significance).

Participants at the first school completed a brief demographic questionnaire immediately prior to the first session that assessed age; birth country; race and ethnicity; acculturation (use of first language relative to English for those who spoke more than one language) [22,23]; and whether lunch was free, reduced in price, or full price.

Each Mighty Teens classroom session was delivered and then followed by a short focus group led by the teaching team. Focus group questions were displayed on slides. Participants wrote responses on large pieces of paper posted on walls or on anonymous index cards, depending on the question. They then voted (ie, agreed/disagreed) on themes identified in comments written publicly and anonymously, and responded to probes for more details or new thoughts emerging in response to other teens’ comments.

In lieu of focus groups, participants at the second school completed a paper-and-pencil pretest immediately prior to the first session (also containing the same demographic items used in the first implementation) and completed a posttest immediately after the last session. All measures had been pretested or used with this age group previously [10,24].

The pretest and posttest contained a short self-efficacy scale, along with items assessing resistance and sexual intentions, and intention to postpone sexual intercourse. The 9-item self- efficacy scale comprised 5 items from the 12-item measure reported by DiIorio et al [25] (eg, say “no” to sex even if the other person says they will break up with you if you don’t have sex) and 4 items created by the first author to assess self-efficacy regarding specific program resistance behaviors [10]. These 4 items were: (1) stop someone who is pressuring you to do sexual things without making them angry, (2) leave a party or dance club where sexual things are going on without being made fun of, (3) avoid situations where you know other people will be making out, and (4) say no in a serious way and stick to it when someone you like is pressuring you to do sexual things. Response options range from 1 to 4 with labels for endpoints and midpoints (1=not at all sure; 3=moderately sure; 5=completely sure). Higher scores indicate greater resistance self-efficacy. Cronbach α was .84.

All intention items used the same response options: definitely not (1), probably not (2), probably yes (3), and definitely yes (4). Higher scores indicate greater intention. Psychometric analysis argued for treating intention types as measures of separate constructs, not part of a single multi-item intentions scale. Resistance intentions were assessed by a question related to a particular response strategy when asked “to do something you do not want to do.” Response strategies included: not resisting (go along with what they want me to do), resisting by using relationally competent communication strategies (4 items, including suggest doing something else; α=.62), or resisting by using a nonrelationally competent strategy (tell them I don’t want to because it’s stupid). Kirby et al’s [26] 3-item sexual intention measure was used to assess willingness to engage in sexual intercourse (eg, I would have sex now to keep someone I cared about romantically as a boyfriend/girlfriend); Cronbach α was .79. Intention to postpone sexual intercourse was assessed with a single item: I want to wait to have sex until I am older.

Demographic responses were summarized with descriptive statistics. A paired t test was used to assess pretest-posttest differences in self-efficacy, resistance and sexual intentions, and intention to postpone sexual intercourse (second implementation only).

The first and second authors reviewed classroom session videos (both implementations) for signs of engagement, restlessness, and confusion. Engagement was defined as girls raising their hands to participate, smiling, looking at the teaching team or slide presentation when not engaged in a task, working on tasks, or easily redirected from chatting with peer(s) back to the main discussion and remaining attentive. Restlessness was defined as fidgeting, repeatedly talking with peer(s) and not easily redirected back to the task, complaining about the activity, being bored, or not having fun. Confusion was defined as a facial expression in which the nose and forehead were scrunched up in a type of frown, or complaints about being confused or not understanding task directions.

The first author also reviewed and performed a descriptive content analysis [27] of the teaching team postsession notes (both implementations) and postsession focus group videos (implementation 1 only). Content (eg, sentence or section of notes; participant response or portion of response if expressing multiple ideas) was sorted into one of three categories: (1) liking or positive, (2) disliking or negative, and (3) confusing or not clear. Improvement suggestions noticed in this process were also marked. Next, information within categories and improvement suggestions were each reviewed for themes and multiple instances, summarized, and then discussed with the study team.

Phase 2 began with app development that culminated in usability testing. The app prototype was initially developed for Android phones given Pew Foundation reports [28] indicating greater Android use by low-income families living in the urban metropolitan area within central Florida in which this research was being performed. However, the next iteration of the app will be built for both Android and iPhone cell phones.

Two theories guided app development. First, the theory of fun [29] stresses the importance of engagement for app uptake, and defines engagement as the result of positive affect, fun, low predictability, novelty, and the right amount of challenge (not too little or too much) experienced while interacting with the app. Second, narrative engagement theory [13] stresses the centrality of engagement to behavior change. According to this theory, engagement mediates overall effectiveness of narrative creation, a phenomenon that builds self-efficacy through cognitive rehearsal [12] while simultaneously imbedding intervention messages (eg, concepts) in a narrative [13]. App use was planned for nonschool settings (eg, home, public library), necessitating a simple and intuitive interface. Additionally, the interface needed to guide users in creating their own “Mighty Teens” electronic story of resistance (similar to the events that occurred with DRAMA-RAMA in the Mighty Girls program). Stories were to be recorded, mirroring natural storytelling.

Both the app and a secure, password-protected dashboard (backend accessible to administration staff) were developed using a collaborative approach, involving the authors and Margaret Broucek, Principal of Tapp.technology who was a phase 2 technology partner (Textbox 2). Tapp.technology iteratively created screen designs in response to feedback from the authors and the three research assistants, arriving at the designs used to create the prototype for usability testing (Figure 2).

Phase 1 participants were invited to participate in usability testing as part of their study participation. However, some were unable to attend due to transportation issues or schedule conflicts (eg, needed for sport team practice), resulting in a sample of 25 participants. Almost all participants (92%, 23/25) created and recorded at least one story. Two participants interacted with the app but did not record (or deleted recordings of) their stories. One of these participants did not want anyone to listen to her story. It is unclear if the other had difficulty with the recording feature or felt similarly. Although 25 girls participated in app testing, only 24 completed written assessments of usability and engagement.

Usability testing was performed at each school involved in phase 1 with hot spots, approximately 1 month apart. Participants accessed the app using either their own Android cell phone or a “testing” cell phone provided by the research team.

When participants arrived at their respective school, they were split into two groups with each group assigned to a different classroom. Once in their classroom, participants were split into dyads and a research team member was assigned to observe each dyad. Those that finished early were free to record additional stories. Research team members took ethnographic field notes describing participants’ verbal and nonverbal behaviors during app use, but were unable to see the participants’ screens unless shown to them. Research team members debriefed immediately after testing. Notes from this meeting became part of the ethnographic field notes.

After using the app, the girls completed written assessments of usability and engagement. Following this, a brief focus group discussion was held with the first or second author in the role of facilitator and a member of the research team in the role of cofacilitator/notetaker.

Ethnographic field notes, focus group notes, and participant responses to the open-ended usability question were analyzed using descriptive content analysis [27], similar to phase 1. However, the first author sorted units into categories, and then identified and presented a summary of themes represented by these categories to the usability testing team, along with copies of the raw qualitative data. The summary was finalized in the resulting discussion.

Participant stories were coded for narrative structure and reference to Mighty Teens program skills using two dichotomous (0, 1) variables. Narrative structure was defined as three complete components (1). A reference to a Mighty Teens program skill (1) required the participant to describe a character correctly using (or teaching another character to use) at least one skill. It was not necessary to use the exact program name for the skill. However, not referencing a skill or naming one without any application were both coded as no reference (0). Interrater agreement was 100%.

Mothers and fathers of 7th grade girls involved in the second implementation were invited to participate in a focus group (a) by flyers included in the study packets given out to their daughters, and (b) during the parental consent process for their daughters’ participation. No fathers expressed interest in participating or were involved in the consenting process.

Scheduling of focus group participation coincided with the first national lockdown in response to the COVID-19 pandemic. Parents were distracted, under distress, with other activities competing for their attention, complicating consenting and scheduling (eg, parents who had previously expressed interest did not return phone calls). Scheduling and COVID-19 accommodations resulted in two focus groups involving 6 mothers as participants.

Focus groups were conducted using Zoom conference technology, which was challenging because the participants seemed unfamiliar with Zoom and video conferences in general. However, the facilitator (second author) was able to assist by phone, and familiarize participants with the process and importance of confidentiality and privacy once they opened Zoom.

Focus group interviews consisted of open-ended questions about (1) what the parents had heard about Mighty Teen classroom sessions or the app from their daughters, and (2) the parents’ feelings about and challenges related to their daughters’ use of apps and cell phones. Information about classroom session topics and activities, and about the app was provided once parents shared what they had heard. Feedback was then solicited, followed by specific probing related to their daughters’ use of the app, her doing so privately, and supervision preferences and concerns.

Descriptive statistics are used to summarize the mothers’ ethnicities. Focus group recordings were transcribed verbatim and analyzed using descriptive content analysis [27], similar to phases 1 and 2. However, the first and second authors independently sorted units into positive/negative categories and summarized themes within the categories. No differences in categorization or meaning of theme labels were noted.

The consultants recommended taking more time in session 1 to explain the program, and making two changes to increase program language inclusivity: (1) replace “girls” with “teens” (including in the program title); and (2) use nongender-specific names and pronouns to avoid gender identity and sexual orientation biases. These changes were recommended to make teens feel included regardless of gender identity or sexual orientation. The only exception was in the Media Influences session, where the focus on how girls and women are depicted in advertisements was viewed as valuable. The three young adult consultants also reported (without prompting) how much they liked the program and wished it had been offered to them when they were in middle school, lending support for its multicultural potential.

We could not detect any substantive differences between the results obtained for the two usability testing sessions. Hence, results presented herein are combined across sessions. The observed time to complete a task ranged from 6.30 to 13 minutes (median 6.67). The prototype did not track the time spent between opening the app and submitting a story, and teens were offered the opportunity to record their stories outside if they did not want to be overheard. Hence, the observed time to complete a task may be confounded with the time spent visiting with friends, a behavior observed at the same time the recording was presumed to be occurring, arguing for it being a “high-end” estimate of the actual time required.

Only 24% (6/25) of the participants created a story with a beginning, middle, and end, and less than half of the participants (44%, 11/25) created a story in which Mighty Teens skills were used. The stories reflected both same and opposite sex orientations, consistent with the tailoring expected in self-generated narratives.

The System Usability Scale score was 76 despite 55% (12/22) of the participants agreeing that it was hard to think of an idea to use for the Mighty Teens app. Content analysis of open-ended question responses and focus group notes indicated that about half of the participants wanted the app to have a tutorial, more directions, or prompts, whereas about half felt that it was “simple,” “easy,” or “very straightforward.”

Analysis of ethnographic field notes described some participants as frowning and stopping, as if concentrating, and others clearly stuck at the initial step of identifying a title until told they could use “My Story” as the title if they wanted to. Regardless, these participants quickly transitioned to a rush of tapping and swiping with brighter affect, and focus group notes indicated that all but one participant (disgruntled by a preexisting peer conflict) were pleased with their experience. Nevertheless, many did not like how their recorded voice sounded.

Slightly more than a third (9/24, 38%) of the participants stated in their written responses to the usability open-ended question that there was nothing needed to improve the app (eg, “no,” “nope,” “nothing”). Less than half (11/24, 46%) made one or more comments concerning actual or desired app features, and 17% (4/24) made no comments.

Descriptive content analysis of responses to the usability open-ended question ethnographic field notes or focus group notes identified ways to improve usability (Textbox 3). Triangulation of these data sources indicated that a majority of participants (80%, 20/25) were frustrated or disappointed that they could not type their story. In contrast, 20% (5/25) expressed joy and astonishment that they could record (they had expected text entry) and had quickly begun using the app with gusto. However, this latter group agreed in the focus groups that text entry would be useful if a teen was concerned about being overheard, or was mute or stuttered.

Four unsubmitted stories were found in the testing phones after the session supported participants’ suggestion to add confirmation of a successful story submission. Additionally, the focus group discussion helped clarify other usability improvements. For example, all participants wanted the character set to be more representative but some also felt that the numbers of icons in the current character set was overwhelming, arguing for customizing over expanding the existing set.

Similarly, the importance of adding an “other” category for character emotions was emphasized by comments indicating that participants considered qualities such as courageous, shy, and determined to be emotions. In contrast, comments regarding places suggested adding both icons (eg, corner store, fast food place, “jumping place” [a place for teen parties with trampolines]) and an “other” option to accommodate places that might be more regional or seasonal (eg, pool, beach).

Descriptive content analysis indicated smiling, intense focus, and constant interaction with the app for 23 out of 25 (92%) participants with these same participants asking if they could use the app to make multiple stories. At least 4 out of 23 (17%) participants who requested this option did so in the time available with one participant creating 4 stories.

The word “fun” was used multiple times in all focus groups to describe the app; 83% (20/24) of participants reported they would recommend the app to their friends. Mean Fun and Like scores were 3.0 (SD 0.75) and 3.2 (SD 0.60), respectively, indicating that the participants agreed that the was app fun and liked using it. Additionally, both ethnographic and focus group notes indicated that participants found selecting icons for characters, character emotions, and places very appealing. Selecting emotions was stated as a “favorite part.”

Analysis of open-ended question and focus group data identified changes that could increase engagement (Textbox 3). One such change (sharing and chatting about stories with other teens) appeared part of a larger theme of wanting the app to have a purpose; otherwise, “why should we use it?”