This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
With the aging of populations worldwide, early detection of cognitive impairments has become a research and clinical priority, particularly to enable preventive intervention for dementia. Automated analysis of the drawing process has been studied as a promising means for lightweight, self-administered cognitive assessment. However, this approach has not been sufficiently tested for its applicability across populations.
The aim of this study was to evaluate the applicability of automated analysis of the drawing process for estimating global cognition in community-dwelling older adults across populations in different nations.
We collected drawing data with a digital tablet, along with Montreal Cognitive Assessment (MoCA) scores for assessment of global cognition, from 92 community-dwelling older adults in the United States and Japan. We automatically extracted 6 drawing features that characterize the drawing process in terms of the drawing speed, pauses between drawings, pen pressure, and pen inclinations. We then investigated the association between the drawing features and MoCA scores through correlation and machine learning–based regression analyses.
We found that, with low MoCA scores, there tended to be higher variability in the drawing speed, a higher pause:drawing duration ratio, and lower variability in the pen’s horizontal inclination in both the US and Japan data sets. A machine learning model that used drawing features to estimate MoCA scores demonstrated its capability to generalize from the US dataset to the Japan dataset (
This study presents initial empirical evidence of the capability of automated analysis of the drawing process as an estimator of global cognition that is applicable across populations. Our results suggest that such automated analysis may enable the development of a practical tool for international use in self-administered, automated cognitive assessment.
With the aging of populations worldwide, early detection of cognitive impairments has become a research and clinical priority. In particular, early identification of prodromal dementia is essential for providing secondary prevention and disease-modifying treatments [
Drawing ability is a promising means for developing such an automated cognitive assessment tool. Drawing tests have been widely used for screening cognitive impairments and dementia (eg, trail making [
In this study, we evaluated the applicability of automated analysis of the drawing process for estimating global cognition in community-dwelling older adults across populations in different nations. Specifically, we collected drawing data with a digital tablet, along with MoCA scores for assessing global cognition, from community-dwelling older adults in the United States and Japan. We then investigated the associations between the MoCA scores and drawing features across the 2 data sets. Finally, we built a machine learning model that used the drawing features to estimate MoCA scores, and we evaluated the model’s generalizability from the US data set to the Japan data set.
The study was approved by the University of California San Diego Human Research Protections Program (HRPP; project number 170466) and the Ethics Committee of the University of Tsukuba Hospital (H29-065). All participants provided written consent to participate in the study after the procedures of the study had been fully explained.
The participants were community-dwelling older adults recruited in San Diego County, California and in Ibaraki prefecture, Japan. For the US data set, the participants were residents of the independent living sector of a continuing-care senior housing community and were recruited through short presentations using an HRPP-approved script and flyer. For the Japan data set, the participants were individuals recruited through local recruiting agencies or community advertisements in accordance with the approved protocol. Both data sets represented subsets of larger cohort studies [
Participants’ characteristics (n=92).
Characteristics | United States (n=55) | Japan (n=37) | |
Age (years), mean (SD) | 83.4 (6.9) | 73.3 (4.5) | <.001a |
Sex (female), n (%) | 39 (71) | 19 (51) | .06b |
Education (years), mean (SD) | 16.3 (2.3) | 13.8 (2.0) | <.001a |
Montreal Cognitive Assessmentc, mean (SD) | 24.4 (3.2) | 24.4 (2.6) | .98a |
Trail Making Test part B time (seconds), mean (SD) | 131.9 (65.1)d | 96.9 (50.1)d | .008a |
Trail Making Test part B errors, mean (SD) | 1.7 (2.5)d | 0.9 (1.5)d | .07a |
aCompared using 2-sided
bCompared using a chi square test.
cTotal possible score ranges from 0 to 30.
dData were missing for 1 participant because of incomplete trials.
All participants performed the Trail Making Test part B (TMT-B) [
The TMT-B was selected as a representative cognitive task that involves drawing motions and is commonly used in clinical practice for screening AD and MCI [
Next, we extracted drawing features from the drawing data and examined their associations with the MoCA scores. Specifically, we investigated the following 6 automatically extracted drawing features: the drawing speed and its variability, the pressure variability, the variabilities of the pen’s horizontal and vertical inclinations, and the pause:drawing duration ratio. These features were selected because they have been reported as significant indicators of changes in cognitive or motor functions [
To investigate the associations of each drawing feature with the MoCA scores, Pearson correlation coefficients were computed after controlling for the age, sex, and years of education for the entire data set and for the US and Japan data sets separately. The 3 sociodemographic variables were considered as covariates, because they have been suggested to affect performance on cognitive screening tests, including the MoCA [
We also developed a supervised machine learning model that used drawing features to estimate MoCA scores, and we then evaluated the model’s applicability across data sets. The analysis workflow is illustrated in
Study overview: (A) workflow of the automated analysis in which drawing data were collected with a digitizing tablet and pen, 6 drawing features were extracted from the drawing data, and a regression model for estimating Montreal Cognitive Assessment (MoCA) scores was trained on the US data set and tested on the Japan data set; (B) plot of the drawing speed variability with respect to the MoCA score for the US and Japan data sets, in which each point represents 1 participant and the solid line represents the regression line for the combined data set; (C) plot of the estimated and actual MoCA scores in the Japan data set, in which each point represents 1 participant and the solid line represents the regression line; (D) comparison of the features’ importance with standard deviations, as assessed via the mean absolute Shapley Additive Explanations (SHAP) values.
The mean MoCA score was 24.4 (SD 3.0; range for participants: 16-30; possible range: 0-30), and the scores did not differ statistically between the 2 data sets (
For the correlation analysis between the MoCA scores and each drawing feature in the entire data set, we found that 4 of the 6 features were significantly associated after controlling for age, sex, and years of education (absolute Pearson
The random forest model trained on the US data set could estimate MoCA scores from drawing features for the Japan data set with an
Partial correlations between drawing features and Montreal Cognitive Assessment (MoCA) scores after controlling for age, sex, and years of education.
Drawing features | All (n=92) | United States (n=55) | Japan (n=37) | |||
|
Pearson |
Pearson |
Pearson |
|||
Drawing speed | 0.08 (−0.14 to 0.28) | .48 | 0.09 (−0.19 to 0.35) | .53 | 0.14 (−0.21 to 0.45) | .44 |
Drawing speed variability | −0.42 (−0.58 to −0.23) | <.001 | −0.33 (−0.55 to −0.06) | .02 | −0.58 (−0.77 to −0.31) | <.001 |
Pause:drawing duration ratio | −0.49 (−0.63 to −0.31) | <.001 | −0.32 (−0.55 to −0.06) | .02 | −0.73 (−0.86 to −0.53) | <.001 |
Pressure variability | −0.34 (−0.51 to −0.14) | .001 | −0.26 (−0.49 to 0.02) | .07 | −0.49 (−0.71 to −0.18) | .003 |
Variability of pen's horizontal inclination | 0.33 (0.13 to 0.50) | .002 | 0.30 (0.03 to 0.53) | .03 | 0.38 (0.04 to 0.63) | .03 |
Variability of pen's vertical inclination | 0.17 (−0.04 to 0.37) | .11 | 0.26 (–0.01 to 0.50) | .06 | 0.16 (–0.19 to 0.47) | .37 |
We collected drawing data from 92 community-dwelling older adults in the United States and Japan, and we investigated the associations between features characterizing the drawing process and global cognition as assessed by MoCA. We obtained 2 main findings, as follows. First, we found drawing features that showed consistent trends with respect to the changes in MoCA scores across the US and Japan data sets. Specifically, with low MoCA scores, there tended to be higher variability in the drawing speed, a higher pause:drawing duration ratio, and lower variability in the pen’s horizontal inclination. Our second finding was that the automated machine learning model trained on the drawing data in the US data set could estimate the MoCA scores for the Japan data set with an
Regarding the correlations of drawing features with MoCA scores across the US and Japan data sets, the correlations persisted even after controlling for age, sex, and years of education. In post hoc power analysis, the power exceeded 0.90 with a significance level of .05 (2-sided). The trends were consistent with those observed in previous studies with individuals with impaired global cognition [
We have presented preliminary evidence suggesting that automated analysis of the drawing process for estimation of global cognition can be applied across populations. We trained the machine learning model on drawing data in the US data set, and we then evaluated its performance on unseen drawing data in the Japan data set. In this context, the model could estimate MoCA scores with an
With the aging of populations worldwide, there is a growing interest in using digital technology to assess cognitive function in nonclinical settings like the home for early detection of dementia [
Furthermore, the approach using behavioral data is expected to support future efforts toward the development of continuous, passive monitoring tools for early detection of dementia from data that can be collected in everyday life [
Regarding the device used for drawing data collection, previous studies have shown the usefulness of a range of devices, including a mobile tablet with a stylus [
This study had several limitations. First, it was limited in terms of the numbers of participants, drawing tasks, and data sets. Our findings were based on drawing data from a single task, and the applicability to other types of drawing data thus remains unexplored. In addition, the international applicability of our model was only evaluated between 2 data sets, and the details of how the model performance is influenced by cultural differences have not been thoroughly investigated. Together, our findings have yet to be confirmed with larger samples that provide cross-cultural insights. Second, we did not investigate the participants' sensory and physical functions (eg, eyesight, grip strength), even though those functions might affect drawing performance. Moreover, other residual confounders might exist. Third, the drawing data were collected in a laboratory setting with a tester; accordingly, a future study will need to establish the validity of fully self-administered tasks. Finally, further research will also be needed to obtain a mechanistic understanding of how drawing features relate to the neural changes underlying cognitive impairments.
In summary, we have presented empirical evidence of the capability of automated analysis of the drawing process as an estimator of global cognition that is applicable across populations. Although no causality could be inferred from our results with cross-sectional data, the results nevertheless suggest that automated analysis of the drawing process could be a practical tool for international use in automated cognitive assessment. Consequently, this approach may help lower the barrier to early detection of cognitive impairments in a variety of settings and populations.
Alzheimer disease
Human Research Protections Program
mild cognitive impairment
Mini-Mental State Examination
Montreal Cognitive Assessment
Shapley Additive Explanations
Trail Making Test part B
University of California San Diego
This work was supported by the National Institute of Mental Health T32 Geriatric Mental Health Program (grant MH019934), by the Sam and Rose Stein Institute for Research on Aging at the University of California San Diego (UCSD), by IBM Research AI through the AI Horizons Network IBM-UCSD AI for Healthy Living program, and by the Japan Society for the Promotion of Science, KAKENHI (grant 19H01084).
YY, KS, MK, and HCK are employees of IBM. The other authors report no conflict of interest regarding this study.