This section is organized in two parts: system design and architecture, followed by the formative evaluation study design and procedures.

This study’s protocol was reviewed and approved by the Institutional Review Board of North Dakota State University (Protocol #IRB0005368, approved December 11, 2024, expiration December 10, 2027; Exempt, Category 3—Benign Behavioral Interventions). A protocol amendment covering additions to a subsequent phase of the broader study was approved on August 12, 2025. The research is supported by the NSF (National Science Foundation) RII (research infrastructure improvement) Track 2 FEC (focus areas competitive) grant. Participation was entirely voluntary. All prospective participants reviewed a digital information sheet detailing study procedures, potential risks, and benefits, and provided digital informed consent via checkbox confirmation on a secure online platform before any data collection. All study data were stored on encrypted, password-protected institutional servers and analyzed using deidentified datasets in which unique participant IDs replaced all personal identifiers. Research records will be retained for three years following study completion in accordance with North Dakota State University Institutional Review Board requirements, after which personally identifiable information will be permanently deleted.

The construction of the peer-support knowledge base involved data collected from public Reddit communities. Data were collected exclusively via the official Reddit API using the PRAW (Python Reddit API Wrapper), in compliance with Reddit’s terms of service. We adhered to safe harbor ethical guidelines for internet research and treated all content as sensitive given the intimate health nature of the topics discussed. A strict deidentification pipeline stripped author usernames, user flair, and specific timestamps during data ingestion. The system synthesizes retrieved narratives as aggregate, anonymous community perspectives, preventing reidentification of individual posters in the chatbot’s output. No attempt was made to contact original posters or interact with the platform beyond read-only data access.

Given the risks inherent in AI-mediated health information, the system was engineered with layered safety guardrails. Architecturally, retrieval from the Mayo Clinic corpus ensured that every response was grounded in vetted clinical content. At the prompt level, the model was explicitly prohibited from presenting community advice as medical fact and was required to attribute all information to its source. As a mandatory fail-safe, every response concluded with a safety disclaimer explicitly stating that HerCare is not a doctor and that the information provided is for educational support only and is not a substitute for professional medical advice.

This section moves from user acceptance (CUQ, NPS, and PIQ) to corpus-level dialogue analyses, quantifying sentiment polarity and decomposing emotion profiles (VADER and NRC) by comparing user inputs with system replies. We conclude by synthesizing these layers to describe how a trust-calibrated, empathy-oriented design manifests in practice.

The multistage pipeline and architectural integration of peer and clinical knowledge sources evaluated during this field study are illustrated in Figure 1. Regarding participant funnel, the public, uncompensated study link was accessed by 415 visitors during the one-month window (December 18, 2024, to January 17, 2025). The integration of this evaluation phase within the overall HerCare design and development workflow is depicted in Figure 3. Figure 3 provides an overview of the five-phase design and development process for HerCare. Phases 2a and 2b were conducted in parallel, reflecting an asymmetric corpus construction strategy in which each knowledge source received a distinct processing pipeline and embedding model optimized for its data type. Not all were participants in the intended sense: 80 did not provide eligible, informed consent (including 4 who were ineligible by age), consistent with casual exploration of a publicly advertised chatbot. Of the remaining 335 eligible, consenting visitors, 243 (72.5%) completed the full protocol (chatbot interaction, PIQ, CUQ, and NPS) and constitute the analytic sample; 92 (27.5%) did not complete the survey battery.

Regarding participant cohort, analyses draw on 243 completers (from 335 eligible, consenting visitors) in a single-session, remote study (median duration ≈16.9, IQR 11.8-26.3 min).

The sample skewed younger, predominantly US-based and Christian, with smaller Muslim and other representation (Table 3). This context informs the interpretation of acceptance and satisfaction scores.

To check for selection bias from the open recruitment, we compared completers (n=243) with consented noncompleters (n=92) on all prediscontinuation characteristics (Table 3). The groups did not differ in age (P=.11), religion (P=.52), or country (P=.052), but completers had longer sessions (median 16.9, IQR 11.8-26.3 vs 6.1, IQR 3.6-9.5 min; P<.001) and noncompleters exited at a median 40% survey progress, almost all at the questionnaire stage rather than during the chat. No noncompleter produced a complete CUQ and only one provided a PIQ or NPS response, so no analyzable partial data could be included. The shorter sessions and uniform survey-stage exit suggest many noncompleters sampled the interface out of curiosity rather than engaging in good-faith help-seeking; comparable demographics bound, without excluding, self-selection (see Limitations section).

The agent’s CUQ median was 78.1 (IQR 65.2‐87.5; mean 75.67, SD 15.50), indicating that most participants experienced the system as good to excellent in usability. The distribution was left-skewed (Shapiro-Wilk P<.001), with ratings concentrated at the upper end of the scale.

The seven PIQ facets clarify why top-line scores are strong. Table 4 gives details of the scores of the PIQ. The highest ratings—helpfulness, ease of use, and clarity—indicate that participants could quickly orient, obtain understandable guidance, and feel the interaction was productive. Across all facets, the median response was at the top or near-top of the scale (Table 4); accuracy, trustworthiness, and empathy showed the most spread (median 4, IQR 4‐5), suggesting these relational and correctness dimensions are judged somewhat more stringently than ease and helpfulness.

A VADER analysis of 1191 query-response pairs shows a clear, topic-general polarity shift from user inputs to assistant replies, illustrated in Figures 4-6 below.

Overall, user turns skew neutral-to-negative, which is expected in a help-seeking setting. Across topics, the neutral component dominates (often >0.80). This neutral dominance is concentrated in the LDA-derived topics whose keyword distributions are informational and symptom-oriented (eg, menstrual cycle concerns, menstrual cramps and pain experiences, and the information-seeking facet of birth control options and decision-making), where users predominantly describe symptoms and request factual guidance; we therefore read the neutral component as fact-seeking and symptom description rather than affective flatness. Negative sentiment, by contrast, rises in the distress-laden topics. The lowest compound score appears for stress, anxiety, and coping strategies (compound −0.181), followed by menstrual cramps and pain experiences (compound −0.131), indicating language of concern, discomfort, and uncertainty. In contrast, more proactive themes register higher positivity; birth control options and decision-making show the most positive user compound (+0.154), consistent with forward-looking, choice-oriented discourse.

In marked contrast, assistant turns are consistently and strongly positive across every topic. Compound scores range from +0.555 to +0.826, with positive proportions markedly higher than in user inputs (Figure 5). Negative sentiment in replies is low across the board (0.018‐0.069), even when addressing sensitive topics such as menstrual pain or stress. This pattern indicates that responses systematically emphasize supportive, reassuring language while avoiding alarming or discouraging phrasing. Notably, topics that begin with the most negative user tone (eg, stress, anxiety, and coping strategies, and menstrual cramps and pain experiences) still elicit robustly positive assistant compounds (≥+0.554), confirming that the system maintained a positive, supportive stance even under the most affectively challenging conditions.

This systematic gap between user-query sentiment (concern) and agent-response sentiment (support) is evident topic by topic across Figures 4-6, which contrast the aggregate sentiment scores of user queries and assistant responses for each topic.

Taken together, the VADER results depict a stable user→agent positivity transition: users frequently open with neutral descriptions punctuated by negative affect in distress topics, and the agent replies with uniformly positive, supportive language across all themes. The contrast is visible topic-by-topic (Figures 4-6): the lowest user compounds remain negative while the corresponding assistant compounds fall well into the positive range, and proactive topics show positivity on both sides with a further uplift in replies. We refrain from causal claims, and we note that these analyses pool user and agent turns rather than tracking individual sessions over time; the results therefore describe an aggregate difference between user-query sentiment and agent-response sentiment, not a within-session trajectory.

Using the NRC Emotion Lexicon, we profiled eight emotions—anger, anticipation, disgust, fear, joy, sadness, surprise, trust—for each turn and summarized them by topic (Figures 7 and 8). This analysis moves beyond polarity to describe the affective texture of the dialogue, indicating what users bring into the conversation and how the agent responds.

Regarding user emotional landscape, overall, user language reflects three recurring configurations. First, a proactive profile appears in birth control options and decision-making, where positive emotions co-occur at elevated levels—trust 0.966, joy 0.940, and anticipation 0.906—alongside substantial fear 0.846, consistent with high-stakes, forward-looking decision-making. Second, a distress profile characterizes stress, anxiety, and coping strategies—fear 0.444, sadness 0.437, and anger 0.356—and pain-centered topics such as menstrual cramps and menstrual cycle concerns, where fear and sadness are high, and anger is comparatively lower, indicating discomfort and uncertainty rather than frustration. Third, an ambivalence profile emerges in sexual and intimate health in religious contexts, which combines elevated sadness 0.351 and disgust 0.203 with moderate trust 0.500, suggesting emotionally conflicted help-seeking. Across topics, these distributions provide a concise map of users’ affective needs: hopeful yet apprehensive in choice-making; distressed in pain and anxiety; and conflicted when intimate concerns intersect with values and norms.

Regarding assistant emotional profile, in contrast, the agent’s replies are dominated by trust across nearly all topics, with consistently high anticipation and joy—a positivity triad that recurs throughout the corpus (Figure 8). Trust often exceeds other emotions by a wide margin, peaking at 5.281 for stress, anxiety, and coping strategies and 4.902 for period-related mood swings and symptom management. Negative emotions are present but comparatively low; importantly, sadness appears in contexts where acknowledgment is appropriate. For instance, in menstrual cramps and pain experiences, the agent shows its highest sadness 1.402, yet, this acknowledgment co-occurs with higher trust 3.825 and anticipation 1.567. This pattern is consistent with a validate-then-redirect response: the reply briefly reflects user distress (validation) before shifting emphasis toward confidence and forward-looking guidance (redirection). We observe this configuration across topics, including those with the most distressed user profiles, indicating a stable, supportive affective stance in the replies.

Taken together, the NRC results provide a granular description of how the emotional composition of conversations differs between users and the agent: users present with blends of fear, sadness, and (in places) anger or ambivalence, while replies emphasize trust and anticipation, with targeted acknowledgment where distress is salient. Figures 7 and 8 above visualize these per-topic patterns; full numerical data are provided in Tables S3 and S4 in Multimedia Appendix 3.

This formative study developed and evaluated HerCare, a dual-source RAG CA for women’s health. Across standardized usability instruments, postinteraction ratings, and corpus-level computational linguistic analysis, participants who completed this study rated the system highly on usability, trustworthiness, helpfulness, and empathy, consistent with good acceptability in this completer sample. Computational analysis of 1191 conversational turns confirmed that the system’s linguistic behavior aligned with its design goals: user queries skewing neutral-to-negative in affect were consistently met with strongly positive, trust-dominant responses. The designed validate-then-redirect behavior was consistently confirmed across topics, with the agent briefly acknowledging user distress before pivoting toward constructive guidance—an effective strategy enacted consistently without explicit per-topic tuning.

The high self-reported empathy score of 4.17 of 5, together with corpus-level evidence of a systematic polarity gap and trust-dominant agent responses, is consistent with HerCare enacting its empathic design goals at the linguistic level. Our work contributes a methodological approach for looking beyond self-report scores to characterize how empathy is expressed. By triangulating user perceptions with a computational analysis of the dialogue itself, we obtain a more granular, though still descriptive, view of the system’s affective behavior.

The VADER analysis provided a clear, quantitative measure of the polarity gap between user queries and agent responses. That the agent replied to neutral or negative queries with strongly positive language confirms the system behaved as designed; the contribution here is methodological rather than an independent behavioral discovery, in that it provides a blueprint for how researchers can quantify an agent’s response sentiment relative to user input.

The NRC emotion analysis allowed for an even deeper insight, revealing the specific texture of the agent’s persona. We observed that the AI was not generically happy but appeared to act as a trust engine. Its emotional output was overwhelmingly dominated by the language of trust and anticipation, a strategic choice designed to instill confidence and hope. This observation, made possible only through computational analysis, suggests that the system’s empathetic goals were successfully translated into a measurable linguistic strategy. The most informative cross-method observation is the convergence between the agent’s elevated NRC trust and anticipation scores and participants’ independently reported empathy ratings: two methods drawing on different data sources—one a corpus-level lexical count of the agent’s output, the other a subjective postinteraction judgment by users—point in the same direction. We frame this carefully. The convergence does neither establish that the NRC profile caused the empathy ratings, nor that NRC trust is a validated proxy for perceived empathy; the two were not statistically correlated at the participant level, since NRC scores are corpus-aggregated rather than per-participant. What it does offer is triangulation: a self-report signal and a behavioral-linguistic signal that would not necessarily agree but nonetheless align, lending convergent (though not confirmatory) support to the claim that the agent’s designed affective strategy was perceptible to users. We regard establishing a per-participant link between specific emotion profiles and perceived empathy as a valuable direction for future, appropriately powered work. This multimethod quantitative approach—combining subjective user ratings with corpus-level linguistic analysis—offers a practical framework for future digital health research aiming to evaluate the affective qualities of empathetic systems.

A descriptive contribution of our work is the characterization of an empathy strategy that the agent was designed to enact. The question is not just that users found the agent empathetic, but how that empathy was expressed at the linguistic level. The NRC analysis makes this concrete, confirming a consistent pattern we term validate, then redirect.

This pattern is most visible in distressing topics. For instance, in menstrual-pain discussions, the agent’s responses registered a notable level of sadness, consistent with acknowledging the user’s negative experience, alongside a substantially larger measure of trust and anticipation that shifts the response toward a constructive, hopeful frame. Rather than simple emotional mirroring, this resembles a multistep supportive-communication strategy: it signals “I hear your pain” before orienting the user toward potential solutions.

We propose validate, then redirect as a concrete and transferable design pattern for supportive AI. That this strategy was consistently operationalized by our architecture—and that users responded positively—suggests that with the right architectural foundation and prompting strategy, LLM-based systems can move beyond generic positivity toward context-aware emotional support. The articulation and refinement of such reusable design patterns, rather than their discovery as novel behaviors, is what we offer as a contribution toward more human-centered health AI.

Beyond the evaluation findings, the development of HerCare surfaced several design challenges and considerations that are broadly transferable to future empathy-attuned health AI systems. The most fundamental challenge was reconciling two epistemically different knowledge sources, peer narratives and clinical guidance, within a single response without falsely synthesizing them into a unified authoritative voice. The solution was architectural transparency: rather than merging sources into a seamless reply, the system was engineered to surface provenance explicitly in every response, allowing users to evaluate community and clinical content on their own terms. A second challenge was engineering affective responsiveness without requiring labeled emotional training data or fine-tuning: in our implementation, the NRC-based empathy-mapping layer—lexical inference over a standardized emotion lexicon combined with prompt-level goal injection—was associated with measurable empathic linguistic behavior across the corpus, suggesting this lighter-weight approach is a feasible alternative to supervised emotional modeling. Third, content quality control for Reddit data required a shift from post hoc moderation to corpus-level design relying on community voting, longitudinal engagement filters, and thread chaining to surface high-signal, collectively validated narratives rather than applying blanket content rules. Together, these design choices suggest a replicable pattern for building empathy-oriented health AI: treat knowledge provenance as a trust mechanism, use inference-based affective adaptation rather than supervised learning, and leverage community curation as a quality proxy.

Early reviews of CAs in health care showed promise but highlighted limited evidence for behavior change, small or quasi-experimental evaluations, and scarce safety reporting [33-35]. More recent syntheses continue to stress gaps in safety, evaluation quality, and appropriateness for sensitive contexts such as reproductive and intimate health [36,37]. Parallel work in women’s digital health (femtech) documents generic guidance, gaps in menopause support, and serious post-Roe privacy risks in period or fertility apps [38-44]. Studies of menstrual health apps further critique narrow personas, heteronormative defaults, and insufficient clinical grounding, despite high uptake and perceived utility [45-49]. A recent decade review consolidates these concerns, documenting Western-centric defaults and persistent underserving of Global South users, migrants, and marginalized genders, and advances a Reproductive Well-Being for All framework as a corrective lens for future systems [50]. Together, this work motivates systems that deliver accurate guidance while addressing the emotional and contextual realities of women’s health. These review gaps suggest that how information is delivered—its empathy and cultural fit—may be as consequential as correctness, motivating work on relational, tailored interactions.

Our finding that empathy ratings (4.17/5) were slightly lower than functional ratings aligns with this literature’s observation that relational qualities are judged more stringently than task performance, reinforcing the importance of investing in empathic design beyond surface-level pleasantries. Relational and empathic interaction has long been associated with better engagement, alliance, satisfaction, and even clinical outcomes [10,51,52]. HCI work on relational agents demonstrated that explicitly designed empathic behaviors (eg, responsiveness to emotion and social dialogue) can build durable alliances in health contexts [51,52]. Meta-analytic and theoretical work in health communication shows that personalization/tailoring reliably improves outcomes relative to generic messaging—when tailored to user characteristics and needs [7,8,53,54]. Public-health scholarship further distinguishes surface vs deep cultural tailoring [7]: beyond language and imagery, effective interventions embed community values, norms, and lived realities [33,34]. Within women’s intimate health, HCI studies foreground religious and cultural values shaping help-seeking and design needs, including privacy/modesty considerations and taboo navigation [6,55,56]. As a concrete Global South system example, HCI researchers co-designed a culturally appropriate AI combining community participation, professional moderation, and language sensitivity to improve fit while surfacing governance trade-offs designers must anticipate [57]. These strands collectively argue that conversational health systems should adapt to a user’s emotional state, background, and cultural context—not only to what is asked but to how it is asked. Yet, even carefully tailored clinical guidance can miss lived, context-specific concerns; online peer communities surface this missing layer.

HerCare’s dual-source architecture operationalizes what the peer support literature has long documented: that situated community knowledge complements clinical sources in ways that increase perceived relevance, particularly around taboo or culturally sensitive concerns. A large body of HCI or computer-supported cooperative work literature shows that peer spaces (eg, Reddit, WhatsApp [Meta], and Facebook groups) provide social support, reciprocity, and stigma-safe disclosure for sensitive topics including sexual abuse, postpartum depression, and broader reproductive health [58-64]. These communities often supply practical, situated knowledge that complements clinical guidance and increases perceived relevance, particularly around taboo or culturally sensitive concerns [61,63]. The design of digital safe spaces—including moderation, anonymity, and culturally appropriate norms—has been shown to be critical for women and gender minorities navigating health taboos [59,61]. This motivates architectures that can responsibly retrieve peer-support narratives alongside vetted clinical sources, while maintaining transparency about provenance and scope.

Users reported high trust in the system (4.35/5, 93% rating it trustworthy), a system that used explicit source attribution throughout. We did not, however, isolate the effect of attribution itself: participants were not asked whether attribution influenced their trust, so we cannot attribute the high trust ratings to attribution specifically. This remains an open question, of particular interest given the mixed findings on AI labeling effects in this literature. LLMs can exhibit strong performance on medical benchmarks; yet, reliability varies by task and evaluation practice—and human evaluations in health care remain uneven in design rigor [36,37,65,66]. RAG improves factuality and auditability by grounding responses in citable retrieved sources; yet, AI labeling has nuanced effects—labels can dampen perceived empathy and accuracy in some settings and often have limited impact on persuasiveness [67-69]. At the same time, disclosure and labeling present nuanced effects: AI-generated responses can make people feel heard, but explicit AI labeling can dampen perceived empathy or quality in some settings [70]; other studies find labeling has minimal impact on persuasiveness depending on context [37,70-72]. Collectively, the literature suggests health chatbots should (1) ground responses in transparent, citable sources (clinical + community where appropriate), (2) communicate uncertainty, and (3) calibrate trust through honest signaling of capabilities or limitations.

While our study offers preliminary evidence for the feasibility of the HerCare architecture, we acknowledge several limitations inherent to its formative nature. Our goal in this work was to characterize a novel system and establish feasibility; the following points outline the necessary next steps to build upon this foundation.

Most importantly, all outcomes are based on completers, an inherent limitation of this uncompensated, single-session design: 92 of 335 (27.5%) consenting visitors did not finish the survey battery. Completers and noncompleters did not differ significantly in measured demographics, and noncompleters overwhelmingly exited at the survey stage rather than during the interaction, but self-selection toward more engaged or favorably disposed users cannot be excluded. Usability, trustworthiness, empathy, and NPS estimates should thus be read as completer-only and may be optimistic; the NPS of 60, in particular, could be lower across all visitors. No analyzable partial data were available for noncompleters. Future deployments should add lightweight interaction measures and modest incentives to reduce survey-stage attrition.

First, this formative study provides evidence of the system’s positive reception but cannot make causal claims about its effectiveness relative to other approaches. The single-condition field study design, while maximizing ecological validity, does not include a control group. Future work should conduct a randomized controlled trial comparing our dual-source architecture against several key baselines: a single-source system using only expert clinical data, a single-source system using only peer narratives, and a generic, nonretrieval-based LLM with a similar empathetic persona. This would allow us to isolate the impact of the dual-source RAG architecture on user trust and perceived empathy.

Second, the sample’s demographic composition of the predominantly young (18-37 years), US-based (203/243, 83.5%), and Christian (195/243, 80.2%) limits generalizability in a domain where culture and religion directly shape help-seeking behavior. Women’s health experiences vary substantially across religious and cultural contexts, particularly around contraception, reproductive choices, and sexual health. The extent to which HerCare’s empathic strategies translate to non-Western, non-Christian, or older populations remains an open empirical question that future work must address through purposive sampling of underrepresented groups.

Third, both VADER and the NRC Emotion Lexicon are lexicon-based tools with known constraints in contextual sensitivity, cultural variability, and complex negation handling. They cannot detect sarcasm, irony, or culturally situated emotional expression, and their lexicons reflect primarily Western, English-language cultural associations. Findings from these analyses should be interpreted as descriptive corpus-level patterns rather than clinically precise affective measurements.

Fourth, the PIQ was developed by the research team specifically for this study and has not been externally validated. While its facets converged with validated instruments (CUQ and NPS), the lack of external validation limits the strength of construct validity claims for the trust, empathy, and accuracy facets specifically.

Fifth, as detailed in the Methods section, hallucination mitigation relied on architectural grounding and prompt-level guardrails—an approach validated in the team’s prior work rather than formal factual consistency evaluation or expert clinical review. The safety profile of HerCare’s outputs specific to women’s health topics remains to be established through independent clinical review. Additionally, this study’s window overlapped with major Western holidays, which may have elevated stress levels and introduced temporal confounds for anxiety and mood-related topics, a contextual factor that cannot be fully disentangled from the results.

Sixth, the peer corpus carries inherent demographic biases: Reddit’s user base skews younger, English-speaking, and predominantly Western, and score-based ranking may suppress minority perspectives. The five subreddits also underrepresent menopause, older women’s fertility concerns, and non-Western postpartum experiences—limitations that future studies should address through broader, more diverse corpus construction. Additionally, while transparent source attribution is a central design principle of HerCare, this study did not directly measure whether participants noticed or acted upon attribution labels; whether attribution meaningfully influenced trust calibration remains an open question for future evaluation.

Seventh, formal retrieval quality evaluation—including relevance assessment, precision metrics, and ablation of retrieval parameters—was not conducted; such benchmarking is recommended for future technical validation studies.

This formative study suggests that the central tension in digital women’s health AI between clinical accuracy and emotional resonance may be addressable through architectural design. By structuring the knowledge base as two distinct, purpose-built corpora and making their provenance visible in every response, HerCare was built to let users access vetted clinical guidance and lived community experience together rather than choosing between them; establishing whether this improves outcomes relative to alternatives will require controlled comparison. The most transferable contribution is the “validate-then-redirect” pattern characterized through the corpus analysis: a candidate design template for engineering empathy in health AI that can be instantiated through prompt engineering and retrieval architecture without fine-tuning or labeled emotional training data, and that warrants validation in future comparative work. More broadly, this work suggests that health AI should be evaluated not only on accuracy but on affective alignment, whether the system’s linguistic behavior matches the emotional needs users bring to the interaction. The multimethod quantitative evaluation framework used here, combining standardized usability instruments with corpus-level computational linguistic analysis, offers a replicable methodology for assessing this alignment in future systems. Future work should pursue controlled comparative evaluation, expanded cultural and demographic diversity, formal clinical safety review, and longitudinal study of whether short-term trust and acceptance translate into sustained engagement and health behavior change.