Cognitive impairment encompasses both mild cognitive impairment (MCI) and dementia. MCI is considered to be an intermediate state between normal aging and dementia [1] and may initially not interfere with daily life but potentially progress to dementia and cause significant functional impairment [2]. Recent biomedical research continues to yield novel treatments, such as lecanemab (Eisai and Biogen) and Donanemab (Eli Lilly and Company) [3,4], which can effectively slow cognitive decline if applied at its early stages (MCI or mild dementia). This underscores the importance of detecting cognitive impairment early for the cognitive health of older adults.

Nevertheless, cognitive impairment underdiagnosis remains a concerning issue around the world. In the United States, cognitive impairment assessment is included in Medicare’s Annual Wellness Visit [5], but less than 20% of older adults use this preventive service [6,7]. In countries such as the United Kingdom, routine cognitive impairment assessment is not recommended for all older adults [8]. This underuse or lack of provision of preventive cognitive impairment assessment, combined with challenges faced by primary care providers in identifying early cognitive impairment on top of managing other health issues, often leads to cognitive impairment underdiagnosis [9,10].

Considering these challenges, health professionals working in community environments such as community health and social workers (CHSWs) have become increasingly important in cognitive health management for older adults. CHSWs primarily focus on promoting overall well-being within community settings, such as nonprofit organizations, government agencies, and community centers, and often work with specific populations such as older adults. The World Health Organization’s Global Action Plan has made dementia a priority for public health action, accelerating a paradigm shift in the prevention and care of cognitive impairment from clinical systems toward family and community-based services [9]. This shift emphasizes the essential roles of CHSWs in identifying and referring older adults with early cognitive impairment symptoms to clinical care providers for further assessment and treatment.

Although CHSWs have a growing role in cognitive health management, several challenges may hinder the implementation of timely cognitive impairment assessment in community environments. Administering a cognitive impairment assessment, even with a so-called “brief” tool such as the Telephone Interview for Cognitive Status (which consists of 11 items and takes about 10 minutes with a trained administrator), is easier said than done [11]. Given the broad range of responsibilities of CHSWs, integrating cognitive impairment assessments into their existing tasks could stretch their capacity and limit the time available for other essential services. Besides, CHSWs may encounter challenges related to a lack of specific training and expertise in cognitive health assessment, because their educational background often focuses on social work and public health. This knowledge gap could decrease their ability to accurately identify symptoms of cognitive impairment and confidently refer individuals for further clinical evaluation. Finally, stigma and cultural barriers surrounding cognitive impairment may also present challenges for CHSWs [12]. Older adults or their families might be hesitant to disclose cognitive impairment symptoms within community settings, leading to inaccurate results of cognitive impairment assessment [12].

In conducting surveys or questionnaires, researchers usually expect respondents to answer questions honestly and accurately. However, some respondents might lack the cognitive capability to think carefully and answer each question genuinely, leading to a lower quality of response [13-16]. The definition of low-quality response (LQR) in this paper is relatively neutral and is not motivated by intentional deception, but rather by invalid, unreliable, or erroneous responses [17,18]. Manifestations of the LQR are diverse [19-21], including but not limited to (1) skipping questions—respondents may selectively answer and bypass certain queries; (2) contradictory answers—for similar questions, respondents might provide conflicting responses; (3) oversimplified responses—for instance, always choosing “agree” or “do not know” for all questions; and (4) inaccurate or unreliable answers. This could be due to the respondents not fully understanding the question or lacking the time or motivation to ponder over it.

Completing a questionnaire is a task that requires multiple cognitive functions working in synergy [22], such as attention, working memory, and executive functions. Respondents with subtle cognitive deficits might resort to suboptimal answering strategies to cope with the psychological demands of responding to questions [17,23]. Some studies have found that older adults with cognitive deficits may exhibit stronger signals of LQR during questionnaires [19-21,24]. For instance, they might more frequently skip questions or opt for the “do not know” answer. Early cognitive decline might first manifest in complex tasks in the daily life of people, among which, completing a questionnaire is one.

Recent research on early identification of cognitive impairment has been using advanced computational statistics techniques such as machine learning to develop predictive models of cognitive impairment [25-28]. Nevertheless, challenges persist in effectively implementing such cognitive impairment predictive models within community environments. Existing studies on machine learning and cognitive impairment classification mostly used clinical biomarkers such as magnetic resonance imaging and cerebrospinal fluid analysis as predictors [29-31]. These approaches are not feasible in community settings due to their reliance on high-cost or invasive methods for obtaining predictor information. Furthermore, most current studies on machine learning and cognitive impairment classification have been based on small clinical samples (sample size typically ranged from 140-550), where resampling or case-controlling techniques were widely used to derive analytic samples with balanced cognitive impairment and noncognitive impairment proportions [32-34]. The proportion of cognitive impairment cases in such datasets can be significantly higher than the estimated prevalence (about 15%-20%) observed among older adults in communities [35]. While these resampling and case-controlling techniques may enhance the model’s sensitivity to the minority class (ie, the cognitive impairment) [36,37], existing research has shown that reliance on these techniques can lead to discrepancies in the distribution of data and introduce model bias [38,39].

It is worth noting that recent studies have started to explore the use of large community-based samples with questionnaire data to develop machine learning models for cognitive impairment classification [40-43]. However, there remain several important gaps in this genre of research. First, issues related to LQRs, such as nonresponse, extreme answers, and acquiescence, are ubiquitous in the questionnaire data of older adults [19,44-46]. Direct use of questionnaire response data without proper handling of the potential LQR issue may reduce the validity of the prediction models. Second, the number of predictors used in these studies was generally substantial (approximately 22-44), which can make rapid data collection and cognitive impairment assessment challenging. Lastly, widely used machine learning techniques such as regularization are not designed to produce unbiased predictions, because bias-variance trade-off is an essential component of these techniques [47]. Consequently, the predicted probability from these models does not necessarily reflect the true likelihood of cognitive impairment. This poses a challenge for end users, such as CHSWs, in selecting an appropriate threshold. Existing studies often fall short in providing guidance on determining this threshold.

This study aimed to develop a novel machine learning tool for the early identification of cognitive impairment in community settings using psychometric indices generated from subtle inconsistencies in questionnaire responses. A distinctive feature of this study is that rather than filtering out the original questionnaire responses due to potential LQR issues (eg, eliminating low-quality answers), we relied on LQR indices, derived from psychometric methods, as predictors in our machine learning models to classify cognitive impairment. In community environments, CHSWs routinely administer brief questionnaires to identify well-being issues [48], generating a rich amount of questionnaire response data despite that the contents of these questionnaires are usually not directly related to assessing cognitive impairment. This study will investigate whether the relationships between LQR and cognitive impairment could be exploited to develop a machine learning tool for early cognitive impairment identification.

An overview of the methodology adopted in this study is provided in Figure 1. We followed the guidelines outlined in the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) statement for reporting the development and validation of the multivariate predictive models (Multimedia Appendix 1) [49].

Development of the machine learning tool was based on individuals aged 50 years and older from the Health and Retirement Study (HRS), a community-based longitudinal aging study involving a representative sample of older adults in the United States [50]. As a part of this study, demographics including race and ethnicity are collected using surveys. In 2006 (wave 8) and 2008 (wave 9), the HRS administered the leave-behind survey, which explored various aspects of participants’ psychological well-being and life experiences [51,52]. A random half (n=8681, 51.13%) of participants completed the survey in 2006, and the remaining participants (n=8296, 48.87%) completed it in 2008. Our analysis included participants from the leave-behind survey, excluding those whose questionnaires were completed by proxy, those with missing responses to all scale items, or those lacking key follow-up records (such as survival status). This resulted in a total of 12,942 individuals included in our analysis. While our analysis examined all 21 available questionnaires in the HRS leave-behind survey (Multimedia Appendix 2) [51], 4 questionnaires, that is, the optimism, hopelessness, life purpose, and life satisfaction scales, were used as the primary data source to develop the machine learning tool. These questionnaires were brief and assessed aspects that are closely related to the common focus of CHSWs on overall well-being as a positive psychological attitude, satisfaction with life, clear life goals, and optimism significantly contribute to enhancing the quality of life, extending the anticipated lifespan, and reducing mortality risk in those aged 50 years or older [53-56].

To model subtle inconsistencies in questionnaire responses as predictors in our machine learning model, we created two types of LQR indices using psychometric methods. For each questionnaire, we first fitted a graded response model—a common psychometric model based on the item response theory—to the data to obtain an estimate of the respondents’ “latent trait” score [57]. We did this to remove information related to content (eg, people’s optimism levels) and to isolate statistically “misfitting” response patterns indicative of LQR regardless of question content. The first index, the squared residuals, was then derived by calculating the squared differences between the observed questionnaire item responses and the statistically expected responses given a respondent’s latent trait [58]. Larger differences between observed and expected responses indicate severe problems with LQR. The second index was defined as the probability of a respondent choosing the observed response given their latent trait [59]. Selecting statistically fewer probable responses can indicate problems with LQR. The LQR indices were generated using R (version 4.2.2; R Foundation), mirt (version 1.38.1; York University), and tidyverse (version 2.0.0; RStudio) [59,60].

Cognitive impairment status was ascertained using the validated criteria developed by Langa et al [61] for the HRS. In brief, a cognitive status score was calculated from cognitive tests of immediate and delayed word recall, an attention and working memory task (serial sevens), and counting backward from 20. The total score has a range from 0 to 27 with higher scores indicating better cognitive performance. Thresholds of dementia (0-6 points), cognitively impaired with no dementia (7-11 points), and cognitively normal (12-27 points) were applied [61]. The cognitively impaired with no dementia and dementia categories were then combined into a single cognitive impairment category to indicate whether the individual currently has cognitive impairment. To investigate the longitudinal prognostic utility of the LQR indices, we also created a secondary predicted outcome in the supplemental analysis, that is, dementia or mortality in the next 10 years, which was derived from the HRS follow-up records.

Using the LQR indices derived from each questionnaire, we trained and compared seven machine learning models for the binary classification task of forecasting cognitive impairment. Specifically, we used the multilayer perceptron (MLP) technique. MLP is a feed-forward neural network that is structured with input, hidden, and output layers [62]. Compared to conventional machine learning, it possesses robust adaptive learning capabilities, enabling the automatic extraction of abstract information from raw predictor variables without the need for labor-intensive manual transformations [62]. Moreover, the MLP is particularly adept at discerning nonlinear relationships between predictors and outcomes [63], facilitating the modeling of potential complex associations between the LQR indices and cognitive impairment. In contrast to certain advanced hybrid deep learning models, the MLP boasts a lighter architecture and parameter setup, which reduces computational burdens [64]. In addition to training the models using only LQR indices, another version of the models was developed using the LQR indices in combination with two easily obtained demographic variables, that is, age and gender, as predictors.

The dataset had 1.25% missing values (3898 of 310,608 data points), with 1499 of 12,942 (11.58%) participants having at least one missing item. These missing values were imputed using a regression-based iterative imputation method (details in Multimedia Appendix 3) [65]. The imputed data was then split into training (n=9059, 70%) and testing sets (n=3883, 30%) using stratified random sampling to ensure a consistent cognitive impairment prevalence (n=1641, 18.11% for the training set and n=704, 18.1% for the testing set) across both sets. We used batch normalization and class weighting strategies during the training process [66,67]. This allowed our model to support raw data input without scaling and resampling steps.

The MLP model comprised four hidden layers with a rectified linear unit activation function [68]. To reduce overfitting, we added dropout layers after each hidden layer [69]. The output layer used a sigmoid activation function with an initialization strategy to accelerate model convergence [68,70]. During the training phase, we used 10-fold cross-validation on the training sets [71]. Model weights and biases were adjusted based on the binary cross-entropy loss. The data were further split into 80% (n=7247) training subsets and 20% (n=1812) validation subsets in each fold. The area under the curve (AUC) was monitored on the validation subset to guide mechanisms such as early stopping [72]. We also used the Adam optimizer with a dynamic learning rate to optimize the training process, ensuring that only the best-performing model was retained for the final evaluation of the test dataset [66,73]. Hyperparameter selection was initially guided by KerasTuner (Google) [74]. Based on its recommended values, extensive iterative experiments were conducted to ascertain the selected configurations, as listed in Multimedia Appendix 4. The model training was carried out using Python (version 3.9.16; Python Software Foundation), TensorFlow (version 2.12.0; Google, Google Brain), and scikit-learn (version 1.2.1; French Institute for Research in Computer Science and Automation) [66,75,76]

We compared the performance of the MLP to 6 different predictive models. These included 2 classical machine learning models (logistic regression and decision trees) [77,78], 2 ensemble learning models (XGBoost and LightGBM) [79,80], and 2 hybrid deep learning models bidirectional-gated recurrent unit and convolutional neural network-long short-term memory [81-83]. We compared bidirectional-gated recurrent unit and convolutional neural network-long short-term memory models due to the potential dependencies among LQR indices. The LQR indicators are derived from multi-item questionnaires, where multiple items are designed to load onto a single latent factor. Consequently, there may be correlations between LQR indices within items from the same questionnaires. For the classical machine learning and ensemble learning models, we used grid search to determine the optimal hyperparameters. In the case of the hybrid deep learning models, we adopt the same training approach as the MLP and a similar architecture for the fully connected layers. All models were trained by 10-fold cross-validation and evaluated on the same unseen test dataset.

We determined the threshold for cognitive impairment risk scores predicted by the model based on the ratio of underdiagnosis cost to overdiagnosis cost, aiming to minimize the total cost of underdiagnosis and overdiagnosis [84]. Underdiagnosis refers to when the assessment tool identifies an individual with cognitive impairment as cognitively normal, while overdiagnosis refers to when the tool identifies a cognitively normal individual as having cognitive impairment. When applying the cognitive impairment assessment tool in community settings, individuals whose predicted risk scores are higher than the threshold would be referred to follow-up clinical cognitive impairment assessment. Thus, the cost of overdiagnosis would be equal to the cost of conducting one follow-up clinical cognitive impairment assessment. The cost of underdiagnosis would be more profound, including the cost of not receiving timely clinical assessment and treatment of cognitive impairment. Users of the cognitive impairment assessment tool may choose different cost ratios tailored to the needs of their practices. A larger cost ratio would be appropriate when the follow-up clinical assessment and treatment of cognitive impairment is readily available. With the emergence of effective medications such as lecanemab and donanemab [3,4], this cost ratio would more likely to become larger in the future.

Predictive accuracy of the assessment tool was evaluated using the AUC value. AUC values range from 0 to 1, with a higher score indicating better predictive performance [72]. Moreover, the efficiency of using the tool in identifying individuals at high risk of cognitive impairment was evaluated using an efficiency curve approach. The first step was to calculate the proportion of individuals referred to further clinical assessment for cognitive impairment based on predicted results from the tool, which served as a proxy measure of the resources required to conduct follow-up clinical assessment and treatment. Then, the proportion of individuals with cognitive impairment identified by the follow-up clinical assessments was calculated, which acted as a proxy measure for the desired outcome of the tool’s implementation and subsequent cognitive impairment assessment. These proportions were computed at different predicted risk score thresholds from the tool, and the paired proportion values were plotted on a curve. The tool’s efficiency was then compared with plausible rule-based screening strategies, which included conducting cognitive impairment assessments for all individuals aged 65 years and older, or those with cardiovascular diseases such as high blood pressure, heart diseases, possible stroke or transient ischemic attack, or diabetes that is one of major risk factors for cardiovascular disease. This approach of comparing screening efficiency based on proxy measures of needed resources and desired outcomes has been previously implemented by research to compare machine learning–based assessment against rule-based screening strategies [85].

This study is approved by the University of Southern California institutional review board (UP-22-00147) and the University of Surrey Research Integrity and Governance Office (FHMS 21-22 216 EGA).

In this study, we developed a novel machine learning tool for predicting cognitive impairment based on LQR indices derived from subtle inconsistencies in questionnaire responses. One unique advantage of the tool is its reliance solely on the brief questionnaire that does not directly assess cognitive impairment. Of all the questionnaires evaluated, the best performer was the optimism scale, which comprises just six questions. Crucially, the content of these questionnaires does not need to be directly related to cognitive impairment. This flexibility allows health professionals, such as CHSWs, to select questionnaires that resonate more with their practical emphasis, such as focusing on overall well-being aspects such as optimism, hopelessness, life purpose, and life satisfaction [53-56]. Powered by our machine learning models, these questionnaires now enable a “passive or backend” cognitive impairment assessment with acceptable accuracy (AUC around 0.7) for prescreening and screening purposes. This approach alleviates the need for health professionals such as CHSWs to undergo specialized cognitive impairment assessment training and addresses potential stigmatization concerns. Residents can engage without fearing labels such as “potentially cognitively impaired” or being seen as “likely to develop dementia” from receiving cognitive impairment evaluations [12]. In essence, our tool streamlines the cognitive impairment assessment process, making it more adaptable and less burdensome for both professionals and the community.

Compared with existing machine learning models for cognitive impairment prediction, our machine learning model is developed from a large epidemiological sample without using class-balancing techniques. This ensures that the proportion of cognitive impairment in both training and testing sets mirrors the actual prevalence of cognitive impairment in communities. As mentioned above, our model demands fewer efforts in gathering data on predictors compared to its counterparts. The tool also allows for flexible threshold selection based on the ratio of costs related to underdiagnosis to the costs related to overdiagnosis. Such flexibility facilitates end users to navigate the trade-off between underdiagnosis and overdiagnosis when using a risk assessment tool.

Compared to clinical studies using clinical biomarkers as predictors for dementia [26,29], our tool still has a gap in predictive performance. However, our goal does not intend to replace existing clinical cognitive impairment assessment tests or tools. Instead, we hope that the tool we have developed will lower the threshold for the usage of cognitive impairment risk assessment, thereby adapting it to the needs of community settings. The tool may establish an effective bridge between the community end and the clinical end, allowing the clinical end to align resources more productively, resulting in fewer unnecessary inputs and greater efficiency in the health care system. In addition, by comparing to recent studies that use similar HRS data to predict cognitive impairment [43,87,88], our tool demonstrated predictive performance close to those studies, even though other studies used clinical or cognitive impairment-related risk factors as additional predictors. For instance, a study achieved an AUC of 0.78 but used 13 additional cognitive impairment risk factors (eg, race, stroke history, and glycated hemoglobin) alongside questionnaire data [87].

Finally, our approach expands the boundary of current research by innovatively integrating knowledge and techniques from data sciences and psychometrics to enhance the cognitive health of older adults. Central to our methodology is the machine learning tool rooted in LQR indices that are designed to detect latent cognitive deficits through modelling subtle inconsistencies in questionnaire responses. Unlike clinical markers of cognitive impairment [29-31], these psychometric indices are not only cheaper but also simpler to obtain, making them ideal for community settings. More importantly, the psychometric method allows for the evaluation of cognitive impairment independently of the questionnaire’s content. Further, while conventional research methodologies typically handle low-quality data by either eliminating or statistically adjusting them by methods such as weighting [89-92], our approach presents a new perspective: leveraging, rather than dismissing, low-quality data. This strategy underscores the potential of our methodology to enhance data utility for future research.

In addition to the development and validation of this tool, we further explored the possibility of improving predictive performance to provide insights for future research. We experimented with two approaches: one was to incorporate education level as an additional predictor, and the other was to combine four selected scales. The results showed that both methods could enhance the prediction performance of current cognitive impairment using LQR as predictors (Multimedia Appendix 9). It is worth pointing out that the number of items in each questionnaire had no direct impact on the predictive effect (Multimedia Appendix 6), but combining different questionnaires did improve the discriminative ability of the model. The application of these two approaches may increase the burden of assessment as more predictors are added to the model. Furthermore, incorporating other indicators of LQR, such as prolonged response time, may increase the predictive accuracy [93].

This study is constrained by the questionnaires available in the HRS, potentially limiting its generalizability to other questionnaire response data. However, an ongoing comprehensive meta-analysis of longitudinal aging surveys from ten countries indicates that the relationships between LQR indices and cognitive functioning remain consistent across various questionnaires and nations [94]. Additionally, ascertainment of cognitive impairment in this study is based on empirically established thresholds rather than detailed clinical evaluations. Clinical diagnosis of cognitive impairment is intricate, often requiring a mix of cognitive assessments, health assessments, laboratory tests, and brain imaging. Given this complexity, large aging survey studies such as the HRS find it impractical to clinically evaluate every participant. Consequently, these studies typically perform clinical evaluations for a select group and then use empirically derived standards, such as the Langa et al [61], to determine cognitive impairment in more extensive samples. Lastly, this study is limited by its nature as a secondary data analysis. Future research should implement the tool in real-life settings and conduct external validation and impact assessment to ensure its effectiveness.

The machine learning tool developed in this study provides a novel yet practical solution for tackling the challenges of early identifying cognitive impairment in community environments. The approach adopted in this study innovatively integrates psychometric methods with data science techniques and large questionnaire response data, resulting in a risk assessment tool that can facilitate health professionals working in community environments to conduct “passive or backend” cognitive impairment assessment and therefore better collaborate with medical systems to promote early identification and treatment of mild cognitive impairment and dementia.