Clinical psychologist A, who is the sole author of a book on CF in family therapy [24], created 25 fictional parenting stress vignettes. Various episodes were extracted from parenting blogs on Ameba Blog, a popular blogging platform in Japan [25]. Each episode was summarized after removing personal information and commercial content (prior confirmation was obtained from the administrators of the Ameba Blog, indicating there were no issues with such usage). An example of a vignette is shown in Textbox 1. Two additional fictional parenting stress vignettes were created using the same method as that used for the preliminary datasets in order to develop the evaluation criteria.

Semantic Scholar [26] was used to search for research studies related to the following 10 phrases: “maternal anxiety meta-analysis,” “parental anxiety meta-analysis,” “maternal depression meta-analysis,” “parental depression meta-analysis,” “autism spectrum disorder parenting meta-analysis,” “attention deficit hyperactivity disorder parenting meta-analysis,” “developmental disorder parenting meta-analysis,” “intellectual disabilities parenting meta-analysis,” “parenting child with chronic illness meta-analysis,” and “parenting child with chronic medical problem meta-analysis.” Up to 500 studies were retrieved per phrase, resulting in a total of 4251 studies. Subsequently, duplicate studies based on the article IDs were removed. Additionally, studies whose titles did not include both “meta” and “analysis” were excluded, resulting in 2233 selected studies. Finally, studies without available abstracts were excluded, leaving 1544 studies. The text used in the titles and abstracts of these studies (410,654 words) served as the corpus for constructing the knowledge graph.

Next, each sentence within this corpus was translated into relational data for the knowledge graph using an LLM called Claude (Sonnet 3.5; “Claude-3‐5-sonnet-20240620”; Anthropic) [27], which has been frequently used in clinical contexts [28]. The relational data in a knowledge graph explicitly indicates the relationships between entities. For example, the sentence “Parent Distress was positively associated with their Child Distress” translates into relational data as “[Parent Distress] -[:ASSOCIATED_WITH]-> [Child Distress],” illustrating 2 entities and their connecting relationship. This resulted in the extraction of 7740 entities and 2348 relationships. The constructed knowledge graph is depicted in Figure 1, where individual nodes represent entities and the edges connecting these nodes represent relationships. Knowledge graphs have also been used in clinical contexts [19].

First, for the control group, an LLM named Claude (Sonnet 3.7; “Claude-3‐7-sonnet-20250219”) [27], which has also been used in clinical contexts [29], was used with a basic prompt. It read the 25 vignettes (Textbox 1) and generated 25 CFs (Figure 2). Next, the personalization group incorporated a personalization prompt into the basic prompt; otherwise, the procedure was identical to that of the control group (Figure 2). For the knowledge graph group, the Japanese text was translated into English via Claude (Sonnet 3.7), and the previously constructed knowledge graph was queried using this translated English text. Based on prior studies [30,31], the similarity threshold was set at 0.9. Clinical psychologist A also judged the literature citations retrieved using this threshold to be appropriate. Relationships with similarity scores exceeding 0.9 between the English sentences and the relational data were extracted, compiled, and summarized as prior study findings (Table S1 in Multimedia Appendix 1). These findings were used for the knowledge graph group. Additionally, the knowledge graph group used a knowledge graph prompt alongside the 25 vignettes to generate 25 CFs (Figure 2). In the knowledge graph and personalization groups, a personalization prompt was added to the knowledge graph group’s prompt. Otherwise, the procedures matched those of the knowledge graph group. Finally, clinical psychologist B, a human expert who has presented as a symposium speaker on CF in family therapy [32], read basic prompts, excuse prompts, and 25 vignettes, and then generated 25 CFs (Figure 2). Each of the 5 groups performed 25 CFs for the 25 vignettes, producing 125 CFs (5×25).

Clinical psychologist C, a Japanese expert who validated the assessment tools for stepfamilies in Japan [33], evaluated the previously created preliminary datasets (2 vignettes) across the 5 groups. These expert CFs were conducted only for the preliminary datasets by clinical psychologist A, rather than by clinical psychologist B, and were not used in the main evaluation of the 25 vignettes. The evaluation criteria consisted of 4 items assessing the general quality of CFs (on a 7-point scale) [1] and 18 items evaluating key components (presence or absence, coded as 0 or 1) [2]. Based on this evaluation, clinical psychologist A suggested adjustments to ensure broader variability in general quality scores and recommended broadly defining the presence of the 18 key components. Clinical psychologist C agreed with these recommendations. Using these criteria, clinical psychologist C evaluated the 125 CFs, which were presented randomly to ensure that clinical psychologist C remained unaware of the group assignments, thus ensuring a single-blind condition.

A one-factor ANOVA was performed to determine the effects of the 5 groups on the quality of the CF. The Tukey multiple comparison test was used for comparisons between groups.

This study did not involve experimental participants, nor did it include personal data; therefore, an ethical review of personal data was not required.

Before testing our hypotheses, we compared the length of the CF texts across the 5 groups (Table 1). As shown in Table 1, the number of characters in CFs produced by human experts was significantly lower than that of the 4 LLM conditions. Although the evaluator was blinded to the source of each CF, it is possible that the evaluator inferred human authorship based on text length. Therefore, the results should be interpreted with caution.

The general quality of CF was evaluated (Table 1). Regarding consistency, the expert CFs showed significantly higher scores than the machine-generated CFs (Figure 3A). Regarding correctness, both the knowledge graph group and the knowledge graph with personalization group demonstrated significantly higher scores than the control group (Figure 3D). Surprisingly, the correctness scores of the knowledge graph and knowledge graph with personalization groups were significantly higher than those of the expert group (Table 1). Similar results were confirmed for completeness, indicating that both the knowledge graph group and the knowledge graph with personalization group had significantly higher completeness scores than the control group (Figure 3C). The completeness scores of these 2 groups were also significantly higher than those of the expert group (Table 1). Regarding feasibility, the knowledge graph group and the knowledge graph with personalization group showed significantly higher scores than the control group (Figure 3B). However, there was no significant difference in feasibility between the 2 groups. These results suggest that the correctness and completeness of CF can be improved by incorporating a meta-analysis knowledge graph. Although the consistency of the machine-generated CF did not reach human expert levels, providing machines with knowledge graphs allowed the feasibility to reach levels comparable to those of experts.

The key components of CF were compared (Table 1). For the accessible language component, the expert group scored significantly higher than the control group and the knowledge graph with personalization group (Figure 4A). Regarding the mean-making component, the personalization group scored significantly higher than the expert group (Figure 4B). These findings suggest that human experts are more likely to use accessible language than machines and that personalization prompts might increase the mean-making components within the CF.

A qualitative analysis of the CFs from the 5 groups based on the vignette in Textbox 1 was conducted (Table 2). Concerning the use of prior study findings, citations were present in the knowledge graph and the knowledge graph with personalization groups, but absent in the control, personalization, and expert groups (Table 2). Considering that human experts typically require more time to conduct literature searches, leading to fewer citations during CF [14], it is possible that the control, personalization, and expert groups did not perform adequate literature searches during CF creation. In contrast, the knowledge graph and knowledge graph with personalization groups were provided with a list of relevant prior study findings in advance (Figure 2; Table S1 in Multimedia Appendix 1), facilitating easier use of prior study findings.

The CFs were also examined from a reframing perspective. Reframing involves therapists actively attempting to transform their clients’ thoughts, which is a common technique in family therapy [34]. The vignette in Textbox 1 depicts a mother expressing self-blame, making reframing particularly applicable [35]. All 5 groups applied reframing to this aspect (Table 2). While reframing recommends directly using client phrases [36], only the expert, personalization, and knowledge graph with personalization groups adhered to this recommendation (Table 2). The control group did not use client expressions during reframing and the knowledge graph group used only single words, suggesting that they did not fully adhere to this recommendation (Table 2). This analysis indicates that personalization prompts facilitate the extraction of client-specific expressions, enhancing mean-making within CF.

Table 2 also includes comments from the experts following their responses and evaluations. The responding expert (clinical psychologist B) indicated a deliberate practice of writing only content considered easily acceptable to the recipient rather than exhaustively documenting the entirety of the CF. This practice is thought to contribute to the higher consistency observed in human-generated CFs compared to those generated by machines. Furthermore, an evaluation expert (clinical psychologist C) pointed out the unnatural expressions present in machine-generated CFs when directly returned to clients. This observation aligns with the lower scores for accessible language found in machine-generated CFs compared to human-generated CFs.

This study demonstrates that the correctness and completeness of CF increase when using a knowledge graph based on meta-analyses. In addition, the use of a knowledge graph resulted in the feasibility of CF reaching a level comparable to that of human experts. These findings support the validity of the evidence-based CF model [12]. Incorporating the findings of meta-analytic studies into CF has 3 advantages. First, integrating academic findings enhances the accuracy of CF content [12]. Second, the inclusion of diverse academic findings increases CF completeness [15]. Third, because meta-analyses often include specific therapeutic methods, their incorporation into CF enhances feasibility [13]. This study provides concrete data to support the validity of the evidence-based CF model [12]. This study is valuable in that it enhances CF by incorporating a knowledge graph derived from meta-analytic findings into an LLM. Although previous studies have applied knowledge graphs to diagnostic assessment [30,31], research using knowledge graphs specifically for CF remains scarce. This study is novel in demonstrating that the use of a knowledge graph can improve the quality of CF. Furthermore, previous CF research has predominantly focused on cognitive-behavioral therapy in Western countries [2,23]; incorporating Asian family therapy has expanded the cultural diversity of CF research [1].

Regarding consistency, the influence of the knowledge graph was minimal, and human expert CF generally scored higher than machine-generated CF. Human experts summarized the content in paragraphs rather than bullet points, potentially contributing to their higher consistency scores. In addition, human experts included more elements of accessible language than machines. Although the machines were prompted to use simple language and adhere to these instructions, the human experts still exhibited more explicit emotional expressions. The experts effectively combined the expressions presented in the vignettes to convey nuanced meanings. Nuanced expression is a difficult area for LLMs. Although LLMs may have an advantage over the general population [37], human experts currently show an advantage, and a similar trend arises in a variety of specialties [38].

Personalization had a limited impact on the general quality or specific key components of CF, which is inconsistent with previous findings [4,21]. However, qualitative analysis indicated that personalization facilitates the extraction of client-specific expressions, promoting mean-making within CF. During vignette creation, writer-specific expressions were likely diminished because of the removal of proper nouns, potentially obscuring the personalization effect. Individual-specific expressions may also have been lost during the summarization process. One previous study has demonstrated that a 2-stage approach—first summarizing individual-specific characteristics and subsequently performing diagnostic reasoning—yields higher diagnostic accuracy in LLMs than direct diagnosis alone [28]. Accordingly, adopting a 2-stage approach in which individual-specific expressions are first extracted and then used to generate CFs may facilitate the development of CFs that better capture individual-specific characteristics. Future research should use longer vignettes with more writer-specific expressions and a 2-stage approach to better examine personalization effects [4].

This study has 2 practical implications. First, knowledge of graph-generated CF can be used to train novice therapists [11]. Novice therapists can compare their CFs with machine-generated CFs based on knowledge graphs, which helps them identify missing perspectives or components in their formulations [39]. Second, knowledge of graph-generated CFs can support expert therapists. Experts often lack time for literature searches, resulting in limited citations of previous research [14]. Machines can automatically extract meta-analytic literature data and present experts with current evidence, thereby improving CF quality through the timely integration of such evidence [15]. Since the knowledge extracted from meta-analyses may remain at a relatively superficial level, and even when the cited knowledge is accurate, its application in practice requires careful consideration to avoid placing undue burden on clients [30,31]. Therefore, when such knowledge is used in clinical settings, professional experience and ethical judgment are essential [28]. Indeed, the clinician involved in this study reported intentionally reducing the length of the CF to minimize the client’s burden. Because CF generation by machines is less time-consuming and more cost-effective [8], appropriately reviewing machine-generated CFs by human experts could contribute to improving the quality of CFs produced by humans.

This study has 4 limitations. First, there was a potential bias in the constructed data, as only 1 individual was involved in vignette creation, 1 in response, and 1 in evaluation. Because this study used only a single evaluator, interrater reliability could not be calculated. The scores used as the evaluation metric could not be examined in relation to other external measures, and thus the validity of the metric was not established. In addition, because test-retest assessments were not conducted for this measure, test-retest reliability could not be calculated. To ensure the reliability and validity of these measures, future research should incorporate test-retest evaluations and CF measures with established validity [8]. Securing multiple vignette creators, respondents, and evaluators can prevent such bias [2]. Second, human experts’ CFs differed significantly from machine-generated CFs in style and length, potentially enabling evaluators to infer human authorship despite the blinded condition. Evaluators may have identified and preferentially rated the machine responses negatively [40]. Because the CFs produced by human experts were shorter, the evaluator may have recognized them as being authored by human experts. This recognition may have contributed to the higher consistency ratings observed for the human expert condition. Future research should standardize the length and style of responses to ensure more robust blinding. Third, vignette-style presentation may not be the optimal way to present clinical cases. Presenting clinical cases in a vignette style reportedly resulted in lower diagnostic accuracy among human experts compared with presentations that list symptoms alone [41]. Similarly, when LLMs evaluate cases presented in a vignette style, they tend to generate an excessive number of diagnoses [42]. Therefore, future studies should examine the quality of CF using a format in which symptoms are presented solely as structured lists. Fourth, qualitative analysis in this study did not use thematic analysis [43]. In future research, more comprehensive qualitative analyses could be achieved by conducting thematic analysis and ensuring transparency in the coding process.

Despite these limitations, our study demonstrated the effectiveness of knowledge graphs in enhancing the general quality of CF, which is consistent with studies showing the usefulness of knowledge graphs in the clinical domain [9,10,16-19]. A comparison of human and machine-generated CFs also highlighted human strengths in terms of clarity and machine strengths in terms of completeness and correctness. Future studies should demonstrate improvements in the quality of therapist-generated CFs by presenting machine-generated CFs with meta-analysis knowledge graphs to novice and expert therapists [5]. Using machines as supplementary tools to enhance CF quality will enable therapists to rapidly produce high-quality CFs [3], ultimately improving the mental health services provided to many individuals [20].