The analysis of PSYHAMM data aims to evaluate the precision of a computerized system in identifying, among system-flagged cases, pathologies and associated off-label treatments. This evaluation is based on comparisons with manual checks performed by a psychiatrist (EC) using a structured evaluation grid based on 4 predefined binary criteria applied systematically to each record: (1) whether the diagnosis identified by PSYHAMM was documented in the medical record (excluding negated, hypothetical, or family history mentions); (2) whether the diagnosis was accurate at the precise or broad level (with a predefined tolerance for schizoaffective disorders being classified as either schizophrenia or bipolar disorder); (3) whether the identified treatment was current or had been prescribed during the relevant period; and (4) whether the drug-pathology pair constituted a confirmed off-label situation, with no other accepted indication present. Borderline or ambiguous cases, particularly those involving atypical diagnostic categorizations, were discussed with a senior psychiatrist (MOK) to establish consistent decision rules for the main ambiguities encountered. Because this reference standard relied on a single primary rater (EC) rather than independent, blinded double coding, no interrater agreement statistic could be computed.

A retrospective analysis of pathology-medication pairs was conducted to evaluate precision among system-flagged cases in a clinical context. The data were extracted from a detailed file containing information on diagnoses, administered treatments, and off-label situations. The main variables studied by the expert included the precise diagnosis, which is the exact diagnosis identified by PSYHAMM, and the broad diagnosis. Other variables included medication treatment, referring to the specific treatment prescribed, and off-label detection, which pertains to the identification of off-label prescriptions.

A sample of the database, which automatically detected off-label prescription situations, was extracted to identify unique pathology-medication pairs and calculate their frequency. Precision was measured for the 4 evaluation variables: precise diagnosis, broad diagnosis, medication treatment, and off-label detection. Precision was defined as the number of relevant documents retrieved divided by the total number of documents proposed for a given query, calculated as follows:

This study was conducted in accordance with the principles of the Declaration of Helsinki. It consisted of a retrospective, secondary analysis of routinely collected, deidentified clinical data extracted from the EDSaN CDW, without any intervention or direct contact with patients. The study was reviewed and approved by the ethics committee of GHU Paris Psychiatrie et Neurosciences (Sainte-Anne). Because the study was classified as research not involving human participants (recherche n’impliquant pas la personne humaine) and used exclusively preexisting deidentified records for secondary purposes, individual informed consent was not required. Patients managed at GHU Paris Psychiatrie et Neurosciences (Sainte-Anne) were informed, through institutional information notices, of the possible reuse of their data for research and of their right to object, in accordance with the nonopposition (opt-out) regime applicable to research not involving human participants. All data were deidentified prior to analysis by the automated extract-transform-load pipeline of the EDSaN warehouse, which removes direct identifiers and applies natural language processing to suppress identifying information in free-text documents. Data were stored and analyzed within a secured, access-controlled environment accessible only to authorized members of the research team, and no identifiable patient data are reported in this manuscript.

The 14 drug-pathology combinations identified among the sample of 197 records are presented in Table 1. Bipolar disorder treated with sodium valproate represented the majority of cases, with 108 (54.8%) occurrences, followed by schizophrenia treated with sodium valproate, with 37 (18.8%) occurrences. Other notable combinations include obsessive-compulsive disorder treated with venlafaxine, with 11 (5.6%) occurrences, and attention-deficit/hyperactivity disorder in adults treated with methylphenidate, with 9 (4.6%) occurrences. Each of the remaining combinations accounted for less than 4% of the cases (6/197 or fewer).

The precision of the diagnostic parameters among system-flagged cases (N=197) is presented in Table 2. The overall precision for confirmed off-label situations was 51.3% (n=101). Precision was 75.6% (n=149) for the precise diagnosis, 84.8% (n=167) for the broad diagnosis, and 61.4% (n=121) for the identified treatment.

Among the 197 drug-pathology pairs studied, 101 (51.3%) were evaluated by the expert as true positives, meaning they were indeed off-label prescriptions (Figure 2). Conversely, 96 (48.7%) pairs were incorrectly identified by the system as off-label (Figure 2). Among these 96 cases, in 76 (79.2%) cases, the patient was not exposed to the drug involved in the off-label prescription. The reasons for this nonexposure are as follows: in 50 (65.8%) cases, it was a prescription prior to the concerned stay, indicated as a history of exposure; in 23 (30.3%) cases, there was a treatment discontinuation during the stay; and in 3 (3.9%) cases, the treatment was mentioned in the report only as a therapeutic possibility (hypothetical; Figure 2). Finally, in 20 (20.8%) of the 96 cases, the diagnosis was incorrect, meaning another diagnosis should have been considered for the studied drug. The results are summarized in Figure 2.

The evaluation of diagnoses and treatments, detailing the precision of precise and broad diagnoses, identified treatments, and off-label situations for various pathology-medication pairs, is presented in Table 3. Bipolar disorder treated with sodium valproate showed the highest precision in both precise (90/108, 83.3%) and broad diagnoses (104/108, 96.3%). Schizophrenia treated with sodium valproate had a high treatment identification rate (25/37, 67.6%) but lower off-label situation precision (18/37, 48.6%). Overall, the precision varied substantially across different pathology-medication pairs, highlighting areas for improvement in precise diagnosis and off-label detection (Table 3).

Despite the promising results, there are several limitations to the current study that must be addressed in future research. One major limitation is the reliance on the accuracy of EHRs, which can vary significantly in terms of completeness and consistency. Incomplete or inaccurate records can lead to false positives or negatives in off-label detection.

Performance was characterized using precision (positive predictive value) alone. Because manual review was restricted to system-flagged candidates, the reported 51.3% precision—and the corresponding 48.7% false-positive rate—quantify only the proportion of flagged candidates that were genuine off-label situations, and not the system’s ability to retrieve all true off-label prescriptions. A complete performance characterization, including recall against an exhaustively annotated reference standard, remains necessary and is planned for future work.

Additionally, the manual verification process, while essential for ensuring precision, is time consuming and subject to human error. A key limitation of the present study is that the reference evaluation relied on a single primary rater (EC), with ambiguous or borderline cases adjudicated jointly with a senior psychiatrist (MOK) according to predefined decision rules. Although this approach provided expert-level judgment and ensured consistent handling of the main ambiguity encountered (the categorization of schizoaffective disorders), it does not constitute an independent, blinded double-coding procedure. Consequently, no formal interrater agreement statistic (eg, Cohen κ) could be computed, and the possibility of single-rater bias cannot be excluded. This is an important constraint on the robustness of the validation; independent double coding with measurement of interrater agreement is a priority for future work and a prerequisite before any deployment for clinical or regulatory use.

Another limitation is the focus on a single institution (Sainte-Anne Hospital), which may limit the generalizability of the findings. Expanding the study to include multiple institutions with diverse patient populations and health care practices could provide a more comprehensive understanding of off-label prescribing patterns and the generalizability of the PSYHAMM approach [32].

A further limitation concerns the size and composition of the evaluation sample. The analysis was based on 197 system-flagged drug-pathology pairs, of which more than half (54.8%) corresponded to a single pair (bipolar disorder treated with sodium valproate) and 73.6% to only 2 valproate-related pairs. This distribution reflects the real-world frequency of off-label prescribing at the study site, where valproate is used for several psychiatric indications, rather than a sampling artifact; nonetheless, it limits the precision and stability of per-pair estimates for less frequent combinations, so results for rare pairs should be regarded as preliminary. As a single-center, proof-of-concept evaluation, the study was designed to assess feasibility rather than to provide population-level prevalence estimates; larger, multicenter samples with a more balanced, prospectively defined sampling strategy will be required to confirm and generalize these findings.

Future research should also investigate the integration of additional data sources, such as pharmacy records, to provide a more complete view of medication use, and the use of longitudinal, temporally aware models better able to handle the timing of diagnosis and treatment.

The work presented here was developed solely at Sainte-Anne Hospital. However, EDSaN has been developed and is routinely used at RUH, which covers numerous medical specialties, suggesting that it could be widely adopted. Nevertheless, RUH does not have a psychiatric department, and this possibility will need to be evaluated.

The PSYHAMM project demonstrates a promising approach to the automated detection of off-label drug prescriptions in psychiatry. By leveraging advanced computational tools and integrating clinical oversight, the system achieves a significant level of precision in identifying off-label use, although challenges remain in addressing temporal issues and ensuring data completeness. The findings underscore the importance of continuous refinement and expansion of automated systems to enhance their reliability and effectiveness in clinical practice. Future research should focus on integrating additional data sources, expanding to multiple institutions, and incorporating advanced analytics to further improve the precision and utility of off-label detection systems.