This study was designed as a comparative methodological evaluation of two AI-assisted screening approaches within ASReview compared with the manual screening approach used in a previously published scoping review on how AI can support quality of life in people with dementia [26]. Reporting of the present study was guided by the PRISMA-trAIce (Preferred Reporting Items for Systematic Reviews and Meta-Analyses–Transparent Reporting of AI-Assisted Evidence Synthesis) guidelines for the transparent reporting of AI-assisted evidence synthesis, where applicable [27].

The unit of analysis was the individual title and abstract record. The data were derived from the title and abstract dataset of the original scoping review, including both the manual screening decisions and the final full-text inclusion decisions. Manual screening and inclusion decisions from the original review served as the reference standard against which the two ASReview approaches were evaluated. The study size corresponded to that of the full screening dataset from the original scoping review. After deduplication and preparation for screening, the dataset consisted of 4690 titles and abstracts, of which 283 were selected for full-text review and 30 were included in the final review.

The screening procedure and stopping rule were informed by selected elements of the SAFE procedure [6,24,30]. The SAFE procedure provides practical guidance for active learning–based screening, including the use of prior knowledge and stopping heuristics that combine a minimum screening proportion with a run of consecutive irrelevant titles and abstracts. Because the present study was a retrospective methodological evaluation rather than a live review selection process, the full SAFE procedure was not implemented. Instead, selected SAFE elements were used to define the prior-knowledge strategy and stopping rule for the ASReview screening approaches.

For the present study, 2 separate ASReview projects were created in the ASReview environment, 1 for approach A and 1 for approach B. For each project, the complete title and abstract dataset from the original scoping review were exported from Rayyan and uploaded to ASReview. Approach A and approach B were conducted as independent comparative screening projects. The screening decisions from approach A were not used as training data for approach B.

To ensure consistency in the application of the eligibility criteria, the same reviewer who conducted the majority of the title and abstract screening in the original scoping review (DS) also screened the titles and abstracts presented by ASReview for each approach. The same eligibility criteria as in the original scoping review were applied. Titles and abstracts were labeled as relevant or irrelevant independently for approach A and approach B.

Screening continued until the predefined stopping rule was met. This stopping rule required that 2 conditions were fulfilled: first, at least 10% of the complete dataset had to be screened; second, after this minimum threshold had been reached, screening continued until 50 consecutive titles and abstracts had been labeled as irrelevant by the reviewer. The threshold of 50 consecutive irrelevant titles and abstracts was selected as a pragmatic stopping criterion. The authors acknowledge that more conservative thresholds, such as 100 consecutive irrelevant titles and abstracts, may increase sensitivity by extending the screening process. The same stopping rule was applied to both ASReview approaches. Because each ASReview configuration generated a different ranking of titles and abstracts, the stopping rule was reached after a different number of screened titles and abstracts in each approach. After stopping, the titles and abstracts labeled as relevant by each ASReview approach were compared with the manual title and abstract screening decisions and final inclusion decisions from the original scoping review.

To assess performance, both ASReview approaches were compared with manual screening using sensitivity, specificity, precision, accuracy, screening time, and agreement. Sensitivity was used to assess the proportion of manually selected titles and abstracts that were also identified by each ASReview approach. Specificity was used to assess the proportion of manually excluded titles and abstracts that were also excluded by ASReview. Precision was used to assess the proportion of titles and abstracts labeled as relevant by each ASReview approach that were also relevant according to the manual approach. Accuracy was used to assess the overall proportion of titles and abstracts that were correctly classified by each ASReview approach, including both records considered relevant and records considered irrelevant according to the manual approach. Agreement between each ASReview approach and the manual approach was measured using the Cohen κ and observed agreement (P_o) [31], and the McNemar test was used to assess directional imbalance in discordant classifications [32]. Total screening time was estimated using the time-tracking information displayed within Rayyan for the manual approach and within ASReview for both AI-assisted approaches. Screening time did not include full-text screening or data extraction. False negatives were examined descriptively to explore potential screening-related selection bias by assessing whether missed titles and abstracts shared common characteristics, such as dementia care context, AI methods, or failure to report AI performance. The evaluation level at which each metric was calculated is described in the Data Analysis section.

Performance was assessed at 2 levels. At level 1, manual title and abstract screening decisions were used as the reference standard. Thus, titles and abstracts selected for full-text review during manual screening were considered relevant, whereas titles and abstracts excluded during manual title and abstract screening were considered irrelevant. At this level, sensitivity, Cohen κ, observed agreement, and the McNemar test were calculated.

At level 2, the final full-text inclusion decisions from the original scoping review were used as the reference standard: titles and abstracts included in the final review were considered relevant, whereas all other records were considered irrelevant. At this level, sensitivity, specificity, precision, accuracy, Cohen κ, observed agreement, and the McNemar test were calculated. This approach aligns with prior research, suggesting that although ASReview retrieves fewer abstracts, a higher proportion may be included in the final inclusion set [25,33]. For binary classification metrics, 95% CIs were calculated using Wilson score intervals [34]. At both levels, prior knowledge titles and abstracts were included in the statistical analysis of the performance metrics.

Screening time referred only to the title and abstract screening process and did not include full-text screening (during the manual approach). For all approaches, total screening time was recorded in minutes at the end of the screening process.

This study evaluated the performance of two AI-assisted screening approaches within ASReview compared with manual title and abstract screening in a previously completed dementia care scoping review. Both ASReview approaches substantially reduced screening time and the number of titles and abstracts that required screening, but neither approach retrieved all studies included in the final scoping review. Approach B (custom configuration), which used a semantically richer feature extraction method and a larger prior knowledge set, performed better than approach A (default configuration). However, approach B still missed several final included studies. These findings illustrate that ASReview can support workload reduction in the title and abstract screening process. However, the use of ASReview may be insufficient for reviews in which the aim is to identify and capture as much evidence as possible.

The finding that ASReview can reduce screening workload is in line with previous evaluations of active learning-based screening tools for title and abstract screening [13,14,17,22,35,36]. However, there is no solid overview of how well active learning-based screening tools perform in the title and abstract screening process [23], and previous evaluations show that performance varies across review topics, dataset size, model configurations, prior knowledge strategies, and stopping rules [7,19,37]. This means that ASReview performance cannot be inferred from previous evaluations alone but should be assessed within the specific review context in which it is used. For active learning-based screening workflows, this has implications: a fixed minimum screening proportion, such as 10% or 50% of the total dataset, cannot by itself ensure full identification of relevant titles and abstracts [19,38]. Sensitivity also depends on the model configuration and the composition of the prior knowledge set [19,36]. All these implications may be particularly important in review topics with heterogeneous terminology, where relevant titles and abstracts may be described indirectly or inconsistently [19].

The high specificity and accuracy observed for both ASReview approaches should be interpreted in light of the highly imbalanced dataset. In title and abstract screening, irrelevant titles and abstracts typically form the large majority of the dataset [39]. As a result, high specificity and accuracy can mainly reflect the correct exclusion of irrelevant titles and abstracts rather than the successful identification of relevant titles and abstracts [40]. Therefore, sensitivity, false negatives, and the number of missed final included studies are particularly important for assessing whether an AI-assisted screening workflow is suitable for evidence synthesis.

The 70% (21/30) sensitivity observed for the best-performing ASReview approach represents a critical limitation for using the evaluated workflow as a screening strategy. Approach B retrieved 21 of the 30 final included studies from the original review, meaning that 9 studies would have been missed if this approach had been used as the primary screening approach during the original review. This level of sensitivity would be insufficient for reviews in which near-complete capturing of evidence is expected [41]. By comparison, dual-reviewer screening of titles and abstracts misses around 2.5% of the relevant titles and abstracts [42]. However, dual-reviewer screening is not always feasible because it requires substantial resources [5,13,15]. This trade-off could be acceptable for rapid reviews, where reviewers are willing to give up some degree of certainty regarding the comprehensiveness of the included evidence [42]. However, it remains unclear what level of reduced sensitivity is acceptable when AI-assisted screening tools are used. Further research is therefore needed to determine what level of reduced sensitivity is acceptable when using AI-assisted screening tools in different review contexts.

Agreement analysis highlighted the effect of class imbalance in title and abstract screening. Observed agreement between ASReview and manual screening was high; this was largely driven by shared exclusions of titles and abstracts. In contrast, Cohen κ values ranged from 0.24 to 0.35, which indicate poor agreement beyond chance [43]. This suggests that, despite high observed agreement, ASReview showed limited agreement with manual screening on which titles and abstracts should be selected as relevant. This is expected in title and abstract screening, where relevant titles and abstracts are typically a small proportion of the dataset, but the low kappa values should not be interpreted as acceptable agreement [39,43,44]. The McNemar test provided complementary information by showing that the direction of discordance differed between evaluation levels [45]. At level 1, ASReview missed more titles and abstracts selected for full-text review by manual screening than it additionally labeled as relevant. At level 2, the direction was reversed: ASReview labeled more titles and abstracts that were not included in the final review as relevant than it missed among the final included titles and abstracts. Together, these findings show that high observed agreement could mask practically important disagreement. Missed final included titles and abstracts may reduce the completeness of the evidence base, whereas level 2 false-positive titles and abstracts indicate that workload reduction was accompanied by limited precision among ASReview-selected titles and abstracts.

Current study findings should be interpreted in the context of a broader movement toward responsible use of AI in evidence synthesis. The RAISE (Responsible Use of AI in Evidence Synthesis) recommendations, developed by Cochrane, among others, provide a framework for ensuring the responsible use of AI across all roles within the evidence synthesis process [46]. Similarly, PRISMA-trAIce frameworks provide a reporting framework to improve transparency when AI is used as a methodological tool in evidence synthesis [27]. These developments may support broader adoption of AI-assisted screening tools in the title and abstract screening process but also increase the need for transparent reporting and validation of how such tools are used in specific review contexts [37]. Retrospective validation studies may help identify context-specific best practices for using active learning during title and abstract screening [47]. However, a lack of dataset availability can hinder reproducibility [23,33]. The present study contributes to this evidence by evaluating ASReview’s performance in the context of a dementia care scoping review and by adopting an open science approach, making the full dataset and reviewer decisions publicly available so that the current study is reproducible.

To our knowledge, this is the first study to evaluate the performance of ASReview in the dementia care literature research. By making the full title and abstract dataset and all screening decisions available, this study also contributes to transparency, reproducibility, and future validation of AI-assisted screening in this review context. Another strength is that the same researcher conducted both the manual and AI-assisted screening, which helped ensure consistent adherence to the eligibility criteria across all screening approaches. However, the current results need to be viewed in light of some possible limitations. This study focused on a single scoping review, which limits generalizability to other review topics. Second, the manual screening and final inclusion decisions from the original scoping review were used as the reference standard for evaluating both ASReview approaches. Although the original review included an initial dual-screening validation procedure, the final screening decisions should not be interpreted as an error-free gold standard. Third, the ASReview screening approaches were conducted retrospectively by a reviewer who had been involved in the original scoping review title and abstract screening process. Therefore, the reviewer may have been more familiar with the review topic, eligibility criteria, and terminology during the ASReview screening. This familiarity may have influenced labeling decisions and may also have reduced the time needed to assess titles and abstracts compared with the original manual screening process, in which the reviewer still had to become familiar with the literature and screening criteria. Finally, this evaluation was conducted using ASReview version 1.3. The exact extension package name and version used to enable sBERT-based feature extraction in approach B were not retained. Because ASReview and its extensions are under active development, findings may differ with newer releases, plugin versions, alternative model configurations, or updated workflow options.

This study evaluated two AI-assisted screening approaches within ASReview against manual title and abstract screening in a completed dementia care scoping review. Both ASReview approaches reduced screening time and the number of titles and abstracts requiring manual screening, but neither approach retrieved all titles and abstracts included in the final review. These findings suggest that AI-assisted screening tools, in particular ASReview, can support workload reduction in title and abstract screening but that both evaluated ASReview approaches may be insufficient for reviews in which near-complete capturing of evidence is required. Further research is needed to determine how AI-assisted screening tools perform in review contexts and to determine what level of reduced sensitivity is acceptable when using AI-assisted screening tools in different review contexts.