Based on a previous systematic literature review performed by our team [14], the Wake Forest Trust in Physician Scale (WFTPS) was selected as the starting point for item generation, with permission granted by the original authors for modifications [26]. This 11-item scale assesses fidelity, competence, honesty, confidentiality, and global trust, and demonstrates strong psychometric properties (Cronbach α=.93). Its prior use across diverse populations supported adaptation to the pediatric emergency and surgical contexts, as well as to physician self-reported contexts [27,28].

The patient family instrument incorporated elements of the WFTPS alongside newly developed relevant elements; this subsection was named the Trust-Experience–Patient-Family (TRUST-E-F). Review panel meetings comprising a pediatric general surgeon, a general surgery resident, 2 pediatric emergency physicians, and a quantitative researcher organized items into the 4 trust dimensions.

The corresponding physician subsection (Trust-Perception–Physician [TRUST-P-MD]) mirrored the TRUST-E-F items but was reframed from the physician perspective. Equivalent review meetings classified items into the 4 trust dimensions.

The patient family (Trust-Building Blocks–Patient-Family [TRUST-BB-F]) and physician (Trust-Building Blocks–Physician [TRUST-BB-MD]) subsections were designed to capture factors potentially important to trust building, intended for concurrent completion alongside subsection 1. The same review panel approach was applied to align items with the 4 trust dimensions.

Both instruments underwent two rounds of content validation, assessing whether item content was relevant and appropriately framed, prior to psychometric testing: (1) dimensional content validation, to ensure item alignment with the correct trust dimension; and (2) contextual content validation, to ensure items were appropriate for pediatric brief-encounter settings.

Content validation required expert panels to rate item relevance on a 9-point Likert scale (1=“not at all relevant” to 9=“extremely relevant”) and to provide open feedback. A combined synthetic and human scale–content validity index (S-CVI) threshold of ≥0.80 was set for each instrument [29,30].

A simultaneous parallel evaluation design was performed, wherein human and synthetic panels completed assessments independently to ensure that human experts remained blind to synthetic outputs [31]. Both panels received identical instructions and contextual information. Human experts were recruited based on relevant professional or personal experience and completed a survey form to provide their ratings.

The SIV process was governed by three human-centric principles: (1) complementary integration (LLMs augment rather than replace traditional human expert validation); (2) collaborative framework (human consensus serves as the primary reference standard, with synthetic processes serving a complementary role); and (3) human oversight (all SIV-suggested refinements require human expert confirmation and approval) [31].

All selected LLMs were accessed and operated within Perplexity Pro (Perplexity AI) workspaces, each provided with identical context, including the study protocol, the relevant trust instrument, and a relevant scoping review [14].

Three LLMs were selected based on observed reasoning abilities (September 2025) and accessibility within Perplexity AI. Claude Sonnet 4 was selected for its recognized lead in agentic reasoning and reduced tendency to rely on shortcuts or reasoning loopholes compared to earlier models [32]. GPT-5 was chosen for its advanced logical coherence, minimized hallucination rates, and slower, more deliberate reasoning process [33]. Grok 4 was selected for its demonstrated capacity for abstract reasoning, making it particularly relevant for evaluating conceptual domains such as the experience of trust in clinical settings [34].

LLMs were prompted to embody several expert roles: pediatric emergentologist, pediatric general surgeon, qualitative researcher, emergency department (ED) patient family representative, and surgical outpatient patient family representative (5 for patient family instruments); and pediatric emergentologist, pediatric general surgeon, and quantitative researcher (3 for physician instruments). Prompt design followed best practices using a 4-part structure: role details, prompt, instructions, and parameters (Multimedia Appendix 1) [8].

Each synthetic expert involved a persona-model pairing where the pairing functioned as a single synthetic expert. Duplicate prompt runs were averaged as repeated measurements rather than as separate raters. A run was not included if it failed to provide a rating for every item. When a run failed, it was rerun to ensure 2 valid measurements per expert.

Content validation assessed the relevance of each instrument to measuring trust specifically within pediatric emergency and surgical care contexts. It was conducted in 2 stages. First, to evaluate the relevance of each item to its internal latent constructs, and, second, to assess the relevance of items to the external clinical context.

Dimensional content validation involved assessing each item against its assigned trust dimension (dependability, caring, communication, and competence). A total of 3 human experts evaluated both types of instruments; in parallel, Claude Sonnet 4, GPT-5, and Grok 4 were persona-prompted to simulate patient-family and physician expert perspectives (see Multimedia Appendix 1 for detailed model parameters and prompt). Each item was displayed alongside its designated dimension, and experts rated using the 9-point Likert scale.

Items were further evaluated for their relevance to the broader pediatric emergency and surgical care context. Each item was rated on a 9-point Likert scale. The human panel included 7 experts for the patient-family instruments and 5 experts for the physician instruments. Welch t tests (1-tailed) were used to compare Item–Content Validity Index (I-CVI) rankings between each synthetic and human expert panel, identifying the optimal model for this stage.

Item-level (I-CVI) and scale-level (S-CVI) content validity indices were calculated for both validation rounds. I-CVI was calculated by dividing the number of experts rating an item as relevant (7 out of 9 or higher) by all experts, calculated independently for the human and synthetic panels, and then combined. S-CVI was calculated as the average of I-CVIs (S-CVI/Ave) for each instrument, representing the proportion of content-valid items [4].

Interrater reliability was assessed individually within human-only and synthetic-only panels for each round using fixed-marginal Fleiss κ, via Justus Randolph’s Fleiss κ calculator with a target of κ≥0.80 [35]. Items were categorized as either category 1 (ratings 0‐6) or category 2 (ratings 7‐9). Values were interpreted as κ<0.40 (“poor”), 0.40‐0.59 (“fair”), 0.60‐0.79 (“substantial”), and 0.80‐1.00 (“almost perfect agreement”) [36].

Percent agreement quantified the proportion of items classified equivalently as relevant or not relevant by both panels, per instrument and validation round. An item was classified as relevant for a given panel if at least two-thirds (66.7%) of that panel’s raters scored it ≥7. Items were flagged as a match if both panels’ classifications agreed.

To further characterize differences in rating tendencies between panels, mean relevance ratings on the original 9-point scale before binary categorization were calculated per expert across all items within a subsection and compared between the human-only and synthetic-only panels using Welch’s t-test consistent with the approach described for model selection in the contextual round.

Final expert deliberation determined instrument modifications based on open-ended comments. Substantive synthetic panel comments were confirmed with the human panel or a survey methodologist before implementation.

To evaluate the role of LLMs in quantitative tasks, instrument accessibility and semantic overlap were assessed using 3 parallel methods. The reference (“algorithmic”) method used Flesch-Kincaid indices for readability and embedding models with cosine similarity for semantic overlap. These metrics were then compared with 2 LLM-based approaches using Claude Sonnet 4.5 [37]: in the “LLM-instructed” condition, the model was explicitly directed to replicate the algorithmic procedure (ie, told to compute Flesch-Kincaid scores and cosine similarity using the same formulas as the reference method); in the “LLM-derived” condition, the model was asked to assess readability and semantic overlap directly, without being told which method or metric to use, allowing it to determine its own approach. Each LLM condition was run in triplicate, and the mean values were used for analysis.

Flesch-Kincaid Reading Ease and Grade Level scores were calculated for the patient family instrument, including the demographic section (Multimedia Appendix 1) [38]. Readability was assessed using the 3 approaches described above, with the Readable software implementation of Flesch-Kincaid serving as the specified blueprint formula for use in the algorithmic method [39]. Claude Sonnet 4.5 was instructed to account for the regional context of Quebec, Canada. Targets were set to a Grade Level of ≤8 and a Reading Ease score of ≥60 [40]. All LLM threads were examined to identify semantic fragmentation, Python code generated, methods selected, and the decision rules guiding each computational pathway.

To assess semantic overlap, pairwise cosine similarity was computed between all item pairs within each of the 4 instrument subsections, with pairs at a threshold of ≥0.90. As described in the Phase 2: LLM-Assisted Instrument Metric Assessment section, 3 methods were used, with OpenAI’s text-embedding-3-small model via a Python API serving as the reference algorithmic method [41].

Mean absolute error (MAE) was calculated to quantify the degree of deviation between the LLM-instructed and LLM-derived cosine similarity scores (each averaged across triplicate runs) and the corresponding algorithmic reference score, per item pair.

Each output across both tasks was evaluated to verify accurate computational pathways and results, and to characterize the reasoning underlying each approach. For the patient family readability analysis, all LLM threads were examined to identify the Python code generated, methods selected, and decision rules that determined each computational pathway and reasoning steps.

For the semantic overlap assessment, the researcher engaged LLMs in iterative discussion to derive reasoning pathways and document deviations from the algorithmic benchmark method. This iterative process yielded a prompt library for each LLM technique, which was then systematically reviewed by an adjudicator LLM to select optimal prompts based on reasoning quality and task alignment. The finalized prompts were subsequently tested to assess intertechnique performance variability and establish comparability with the algorithmic method. LLM discussions for each instrument and method type were transcribed, and thematic analysis was performed to identify key themes in LLM capacity to complete deterministic tasks.

The WFTPS was adapted for pediatric emergency and surgical contexts, as outlined in Table 1 and further detailed in Multimedia Appendix 1, from which TRUST-E-F originated. The original 5-point Likert scale was modified to a 4-point scale (1=“No, not at all“ to 4=“Yes, definitely”) by removing the neutral midpoint. Verb tense was modified to emphasize acute rather than longitudinal care, “your doctor” was modified to “my child’s doctor” in all items, and selected items were reframed in first-person phrasing (“I felt”) to capture the patient family’s perspective; 8 of 10 original items best fitting the study context were selected, and 4 new items were added.

Wording was modified to align each item with a single dimension, per COSMIN (Consensus-Based Standards for the Selection of Health Measurement Instruments) guidelines [42]. The instrument was directly adapted to the physician perspective for TRUST-P-MD.

Review panel deliberations generated new items identified as essential building blocks of trust, with a 5-point Likert scale (1=“not at all important“ to 5=“extremely important”).

The distribution of items across trust dimensions for each instrument is shown in Table 2.

For the dimensional round, 3 female human experts (ED patient family representative, pediatric emergentologist, and quantitative researcher) completed the pilot assessment for both patient family and physician instruments. This round took 2 weeks to complete, while synthetic experts provided evaluations within 2 hours. The distribution of synthetic experts can be found in Table 3. Synthetic patient family representatives were excluded from the physician instrument validation as they lack specialized clinical expertise, a gap that cannot be addressed in the absence of true lived experiences.

For the contextual round, 7 human experts were selected based on clinical or experience relevance to the instruments (1 pediatric general surgeon [male], 1 pediatric emergentologist [female], 1 pediatric nurse [female], 1 pediatric occupational therapist [female], 1 ED patient-family representative [female], and 2 doctoral students [females]). This group completed the patient family instrument assessment; five of the same experts (excluding 1 doctoral student and the pediatric nurse) completed the physician instrument assessment. This content validation round took the same amount of time as the dimensional round.

Based on the Welch t test (1-tailed) on the I-CVI scores of the dimensional content validation, Claude Sonnet 4 demonstrated fewer statistically significant differences from human expert panels (3 of 4 instruments) compared with GPT-5 or Grok 4 (2 of 4 instruments each) and was, therefore, selected for the contextual validation round.

S-CVI/Ave scores for the human, synthetic, and combined human-synthetic expert panels are shown in Table 4 (I-CVI scores of the panels; Multimedia Appendix 1). Each individual panel achieved S-CVI≥0.80 for both rounds and all instruments.

Fixed-marginal κ values were assessed at the subsection level for human-only and synthetic-only for the dimensional and contextual validation round as shown in Table 5.

In the human-only panel subsections, TRUST-E-F and TRUST-BB-F did not meet the κ≥0.80 threshold in both rounds (TRUST-E-F: dimensional κ=0.56, 95% CI 0.18-0.93 and contextual κ=0.67, 95% CI 0.41-0.92; TRUST-BB-F: dimensional κ=0.20, 95% CI –0.23 to 0.63 and contextual κ=0.75, 95% CI 0.50-1.00). Despite this, S-CVI met the threshold for both subsections and rounds, and mean raw relevance ratings did not differ significantly between human and synthetic panels in the dimensional round for TRUST-E-F and TRUST-BB-F (P=.21 and P=.65, respectively).

Percent agreement between human and synthetic panel classifications was calculated at the item level. In the dimensional validation round, agreement was 90.9% for the patient family instrument and 88.9% for the physician instrument. In the contextual validation round, agreement was 90.9% for the patient family instrument and 100% for the physician instrument; the lowest agreement observed at the subsection level was 80% in subsections TRUST-BB-F and TRUST-BB-MD in the dimensional round (Multimedia Appendix 1).

Mean raw relevance ratings (1‐9 scale) were higher among synthetic-only panels compared to human-only panels in 6 of the 8 subsections, where the mean difference for TRUST-E-F in the contextual round was statistically significant (mean difference=0.58; P=.039); the remaining comparisons did not reach significance (Multimedia Appendix 1). A pattern was observed for both rounds in which synthetic experts tended to rate the items as more relevant (“synthetic optimism”), whereas human experts tended to rate them more critically (“human conservatism”).

Refinements included adjusting wording for better contextual fit, removing double-barreled items where a single item covered 2 or more concepts, and bolding key aspects of the item to draw respondents’ attention to the main points (Multimedia Appendix 1) [43,44]. One TRUST-E-F item (“I felt my child’s doctor was honest with me about my child’s condition”) was removed based on human expert majority consensus (4 of 7), despite favorable I-CVI scores (dimensional: 0.97, contextual: 0.94), because of redundancy with item 3 (“My child’s doctor seemed transparent when discussing my child’s condition, even when the news was challenging”), which received a perfect I-CVI in both rounds.

Flesch-Kincaid readability scores across the patient family instrument and its subsections are shown in Table 6. The algorithmic method meets the target threshold for Grade Level (6.9) but falls marginally below the target Reading Ease score (58.2).

Analysis of LLM-generated code confirmed appropriate computation of Flesch-Kincaid in all runs where follow-up queries to address discrepancies identified syllable, sentence, and word counting rules as sources of deviation; for instance, certain runs used manual counting while others used a library to determine syllable counts. Further analysis of the coding logic confirmed that no glitches or incorrect routes were taken (eg, incorrect or alternative formula usage or variable assignments). The average of the 3 methods for the overall instrument met target thresholds (Grade Level 8.1 SD 1.1; Reading Ease 60.1 SD 5.6).

Cosine similarity scores for the trust instrument subsections can be visualized in Figure 2. The highest semantic overlap based on the algorithmic method for each instrument is listed in Table 7. The highest cosine similarity scores for item pairs identified by the mean across the 3 approaches (algorithmic, LLM-derived, and LLM-instructed) were between TRUST-E-F items 5 and 7 (0.61, SD 0.41), TRUST-BB-F items 7 and 8 (0.69, SD 0.24), TRUST-P-MD items 1 and 2 (0.53, SD 0.28), and TRUST-BB-MD items 7 and 8 (0.63, SD 0.27). Item-to-item comparison values for all pairs are provided in Multimedia Appendix 1.

The MAE for the LLM-derived technique was TRUST-E-F of 0.12, TRUST-BB-F of 0.15, TRUST-P-MD of 0.17, and TRUST-BB-MD of 0.13. For the LLM-instructed technique, the MAE was TRUST-E-F of 0.46, TRUST-BB-F of 0.39, TRUST-P-MD of 0.42, and derived TRUST-BB-MD of 0.40.

The adjudicator selections for the LLM-derived and LLM-instructed methods are visualized in Multimedia Appendix 1. Based on iterative discussions with LLMs and their inability to execute appropriate methods with the selected prompts, the final cosine similarity scores relied solely on the algorithmic scores.

Following thematic analysis of the 8 LLM transcripts, 4 key themes were identified as factors contributing to inappropriate execution: tool constraints pushing incorrect approximations, LLMs being probabilistic rather than deterministic, helpfulness being prioritized over accuracy, and an inability to maintain consistency, as highlighted in Multimedia Appendix 1.