Study Design
The data for this study were obtained from the SMARTSENSE-D project [36]. Participants were recruited from primary and specialized youth mental health services in Melbourne, Victoria, Australia, over a 2-year period between 2020 and 2021. Recruitment took longer than anticipated due to the COVID-19 pandemic situation in Melbourne. In total, 41 young people (aged 16-25 years) with a diagnosis of MDD consented to participate in data collection for 8 weeks.
With consent from their clinicians, we contacted potential participants who met the diagnosis criteria for MDD based on results from the Mini-International Neuropsychiatric Interview [37] and Patient Health Questionnaire–9 [38]. We included participants with an MDD diagnosis based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition [39], who regularly used a smartphone, understood smartphone data tracking, and were willing to participate. We excluded those at high risk of suicide, actively manic or actively psychotic, not familiar enough with smartphones to complete daily EMA surveys, or with insufficient ability to communicate in English.
Each participant provided demographic information, psychometric assessments (before and after the 8-week period), continuous smartphone sensing data (Android phones only), and data from a wrist actigraphy device during the 8-week trial period. We included in our investigation score results from the following measures and questionnaires that were administered in the study: (1) a demographic questionnaire (baseline assessment only) that captured demographic information such as age, gender, educational level, and other relevant background information; (2) the Depression, Anxiety, and Stress Scale–21 (DASS-21) [40,41], a set of 3 self-reported scales designed to measure the emotional states of depression, anxiety, and stress; (3) the Quick Inventory of Depressive Symptomatology (QIDS) [42,43], a brief self-report tool that assesses the severity of depressive symptoms; (4) the University of California, Los Angeles (UCLA), Loneliness Scale [44,45], which is designed to measure subjective feelings of loneliness and social isolation; (5) the Penn State Worry Questionnaire (PSWQ) [46,47], a self-report measure that assesses the tendency to worry excessively; (6) the Rumination Response Scale (RRS) [48,49], which assesses the extent to which individuals focus on their depressive symptoms and the possible causes and consequences of these symptoms; (7) the NMP-Q [3,50], which measures the severity of nomophobia, which is the fear of being without a mobile phone; and (8) a debriefing questionnaire (follow-up assessment only), which used a Likert scale from 1 (strongly disagree) to 7 (strongly agree) to capture participant attitudes toward such smartphone sensing and their perspectives on its feasibility and acceptability and also captured participant attitudes regarding each of the sensors being activated on their phones using a Likert scale from 1 (extremely uncomfortable) to 7 (extremely comfortable).
The AWARE-Light smartphone sensing app [51] was installed on each participant’s Android phone and continuously collected geolocation, keyboard, communication (through calls and messages), screen status, app use, and touch operation data using sensors [52,53]. In addition, the AWARE-Light app administered EMA surveys twice daily at noon and 8 PM. The debriefing questionnaire items and EMA questions can be found in Multimedia Appendices 1 and 2, respectively.
Data Processing
Preprocessing
As participants installed the AWARE-Light sensing app at different times on their first study day, we discarded the data collected during the initial day of the 56-day data collection period. Therefore, the analysis was based on 55 full 24-hour days for each participant starting from the second day of installation.
We handled missing values in 2 steps. First, we only retained the sensor data for a participant if the data were recorded for at least 27 out of the 55 days; if a participant’s sensor captured data for ≤26 days, then the data were excluded. The phone call and SMS text message sensors were exempt from this requirement as they may not be used every day. As such, we had different numbers of participants with valid data for each sensor dataset. Given this filtering, we ended up focusing on the smartphone sensor data of participants whose location and screen sensors met the threshold, resulting in a total of 27 valid sets of participant data.
Second, we used imputation to handle missing values in active data, including EMA questions and psychological assessments. In sum, <2% of the psychometric data were missing and imputed, and approximately 30% of EMA data were missing values. The days with missing EMA answers were excluded from feature calculations, such as the average and SD.
To resolve the missing data in the QIDS, we left the missing field as null to represent the missing data. Because the questions in these scales vary significantly, we believe that participants’ attitude toward certain questions cannot be represented by the ratings for other questions in the same scale. Note that the total value of the QIDS remained unchanged after resolving the missing data.
To resolve the missing data in assessments other than the QIDS, we first calculated the average ratings of nonmissing questions for each participant and then used the average score to replace the missing data. In this way, we assumed that participants had no special preference toward the questions they skipped.
For psychometric assessments, missing data arose from participants selecting “Prefer not to say,” which we treated as missing data. This suggests that the missingness might be related to the participants’ unwillingness to disclose certain information, implying that the data were missing not at random. Given that <2% of the data were missing, we opted for a straightforward imputation method. While this approach does not explicitly account for the potential missing-not-at-random nature of the data, the small proportion of missing data minimized the impact on our overall analysis. Nevertheless, we acknowledge that more sophisticated methods such as pattern-mixture models or sensitivity analysis could be considered in future studies to address any potential biases.
We verified the correlation and root mean square error (RMSE) values for the psychometric data and found that the lowest correlation between the original and imputed datasets was 0.89, whereas the highest correlation corresponded to an RMSE of 0, indicating a near-perfect match. Specifically, the RMSE values for the baseline DASS-21 total, baseline UCLA Loneliness Scale total, baseline PSWQ total, baseline RRS total, and follow-up RRS total were 0.61, 0.94, 2.15, 0.78, and 2.56, respectively. This indicates that the overall patterns remained robust and intact after imputation.
Feature Generation
The smartphone sensor data collected were used to generate a variety of information features. Message, keyboard, and screen status features were generated using the RAPIDS feature generation tool (AWARE Framework) [54], and the library by Doryab et al [55] was used to generate geolocation features such as average speed, maximum length staying at clusters, and number of frequent places [56]. Specifically, the Hierarchical Density-Based Spatial Clustering of Applications With Noise (HDBSCAN) [57] algorithm was used for cluster generation, effectively handling noise and clusters of varying densities.
We first generated daily smartphone sensor data and then averaged the data across the study period. For instance, to calculate the “maximum time spent at significant locations” feature, we first determined the maximum length of time that each participant stayed in various clusters per day, and then we computed the average duration over the 56 days for each participant.
In addition, we created our own Python scripts to work with certain sensors or generate features that were not available using the aforementioned tools, including touch operations, screen status, app use, and daily EMA questions. The fundamental features are the counts and average values of the interaction with the phone, such as counts of clicks and scrolling from touch operations, average and SD of EMA values, and the number of screen unlocks during each hour of the day.
The features derived from passively collected smartphone sensor data were preselected based on their relevance and potential to provide meaningful insights into user behavior and nomophobia. Location-based features offer detailed insights into the daily movement and habits of participants, encompassing various aspects of mobility, such as variability, speed, distance, and location diversity. Message traces and keyboard use for information retrieval or exchange indicate the extent of users’ reliance on their smartphones. Screen use time and frequency, count of touch actions, and app use for particular app types not only reflect smartphone engagement but also serve as potential indicators of problematic smartphone use [27].
Regarding the psychometric measures investigated (DASS-21, QIDS, UCLA Loneliness Scale, PSWQ, RRS, and NMP-Q), apart from using the aggregate score of an assessment scale, in certain cases, we also investigated the subscores based on the dimensions or domains in each scale. For example, the four dimensions in the nomophobia scale [27] are (1) not being able to communicate (belonging and connectedness), (2) losing connectedness, (3) not being able to access information, and (4) giving up convenience. Similarly, several of the other psychometric assessments, such as the DASS-21, QIDS, and RRS (see the Models section), were also composed of different dimensions.
It is worth noting that we focused on analyzing the psychometric measures from the follow-up assessment for several reasons. First, the follow-up assessment results aligned better with the phone sensor data in terms of chronological order. The follow-up assessments were chronologically recorded after the passive sensing and could reveal more information about the entire study period. Second, the follow-up assessments from various questionnaires aligned better with each other as they were taken at the same time point. Third, the psychometric measures changed little throughout the 56-day study duration. The ratio of the absolute scale difference (calculated using the formula [after − before]/before) for the DASS-21, PSWQ, RRS, and QIDS was approximately 0.1, whereas the ratio for the UCLA Loneliness Scale was <0.05. This subtle difference between the baseline and follow-up absolute values indicates only minor variations across these scales, suggesting stability in the measured psychometric attributes over the study period.
Apart from the total scores for each psychometric measure, we explored the subfactors because they offered a more detailed and sensitive view of the data. While the total score provides a holistic perspective, the subfactors reveal detailed insights into different aspects of the psychometric constructs. These subfactors can highlight specific areas of change that may not be apparent in the overall scores, offering a more granular understanding. Therefore, we were interested in examining the correlation and predictive power of both the total scores and the subfactor scores of the psychometric measures.
Models
We used the Spearman rank correlation [58] to investigate the statistical associations between nomophobia scores and other variables of interest. The Spearman correlation was chosen due to the nonnormal distribution of ordinal Likert-type questionnaire measures and continuous smartphone-collected features. We adjusted all P values using the original Benjamini-Hochberg method [59] to control the false discovery rate at 20%. This adjustment allows up to 20% of the rejected null hypotheses to be false positives and was chosen to ensure a reasonable balance between sensitivity and specificity in our exploratory analysis.
To test the correlations between smartphone sensor features and nomophobia, we only included a participant’s dataset for a sensor if it had at least 27 out of the 55 days of availability (location and screen sensors were selected after this filtering process as they were the 2 sensors with sufficient data). Correlations between psychometric and debriefing questionnaire features and the NMP-Q were explored using valid data for 41 participants. Correlations between EMA features and nomophobia were explored using valid data for 85% (35/41) of the participants.
In total, we selected 48 correlation pairs to investigate, as shown in Textbox 1. The significant feature pairs are presented in the Results section.
We generated the smartphone sensor and psychometric features (Multimedia Appendix 3) shown in Textbox 2.
Selected pairs of features for correlation investigation.
A total of 9 location sensor features were each paired with the Nomophobia Questionnaire (NMP-Q) total score (Multimedia Appendix 3).
A total of 3 screen sensor features were each paired with the NMP-Q total score (Multimedia Appendix 3).
A total of 8 ecological momentary assessment (EMA) features were each paired with the NMP-Q total score. Each feature was obtained by averaging the morning and evening scores of an EMA affect question (Multimedia Appendix 3).
A total of 2 debriefing questionnaire features—one based on the question concerning overall comfort with using AWARE-Light (statement: “I felt comfortable using the app”) and the other based on the average of questions concerning comfort with the accelerometer, app, communication, location, light, keyboard, network, and screen sensors—were paired with the NMP-Q total score as well as the NMP-Q subdimension scores for a total combination of 10 pairs (Multimedia Appendix 3).
A total of 4 Depression, Anxiety, and Stress Scale–21 (DASS-21) features were each paired with the NMP-Q total score (the DASS-21 total score, as well as the scores for the 3 subdimensions of depression, stress, and anxiety). Each subdimension or factor was added to the exploration given that they each represented separate mental health conditions. Furthermore, this investigation had a particular focus on depression given the cohort of young adults with depression (Multimedia Appendix 3).
A total of 11 Quick Inventory of Depressive Symptomatology (QIDS) features were each paired with the NMP-Q total score (the QIDS total score, as well as the scores for the 10 subdimensions). As with the DASS-21, given the focus on depression, it was decided to investigate the subdimensions of this scale as well (Multimedia Appendix 3).
One Rumination Response Scale total score feature was paired with the NMP-Q total score (Multimedia Appendix 3).
One Penn State Worry Questionnaire total score feature was paired with the NMP-Q total score (Multimedia Appendix 3).
One University of California, Los Angeles, Loneliness Scale total score feature was paired with the NMP-Q total (Multimedia Appendix 3).
Smartphone sensor and psychometric features generated.
Location
The RAPIDS tool (AWARE Framework) used Density-Based Spatial Clustering of Applications With Noise algorithms (HDBSCAN) to calculate location points into clusters. On the basis of the time spent in each location cluster, 8 features derived from location clusters were explored in correlation with the Nomophobia Questionnaire (NMP-Q) scale. In total, 5 of these features describe significant places (the minimum, maximum, average, and SD of the time spent per day at significant locations), as well as the number of visited significant places per day. The other 3 features include the average speed (indicating how fast a person moves) per day, total traveled distance per day, and time spent at home per day. There may be some overlap of information if the home is considered the most significant location for the person. The maximum period of staying at significant locations refers to the longest duration that a user stays at any identified significant location over a given period, identifying key locations that are important to the user, such as their home, workplace, or frequently visited places. Extended stays at certain locations can provide insights into routines and lifestyle patterns. The SD of the time spent in a cluster reflects the degree of variability in the time participants spend at different location clusters, with a higher SD indicating greater variation.
Screen
A screen use episode is defined as the period from when the phone is unlocked until the screen is turned off. A total of 3 screen features were explored for correlation: the total duration of unlock episodes, considered as the time when smartphones are actively used; the longest duration of any unlock episode, representing the longest use episode per day; and the number of all unlock episodes, indicating how many times the person unlocked their smartphone in a day. For predictive models, 3 additional features were considered: the minimum, average, and SD of unlock episode duration in a day.
Messaging
In total, 3 pairs of messaging features were explored for predictive models, including the total number of messages received from or sent to the contact who sent messages most frequently per day, the total number of messages received and sent by the participant per day, and the number of different contacts from whom the participant received and sent messages per day.
App use
A total of 4 app use features were included in the candidate pool for the predictive model: the total number of times that social media apps were used by the participant per day, the total number of times that communication apps (eg, messaging and email) were used per day, the total number of times that entertainment apps (eg, games and videos) were used per day, and the total number of times that music or audio apps were used per day.
Touch actions
One feature derived from collected smartphone touch data was the average number of items scrolled. This metric is calculated by tracking the number of distinct scroll actions a user performs on their smartphone screen during a specified period.
Psychometric subfactors
Our study included both the total scores and subfactor scores of the following psychometrics: Depression, Anxiety, and Stress Scale–21 (DASS-21); Quick Inventory of Depressive Symptomatology (QIDS); University of California, Los Angeles (UCLA), Loneliness Scale; Penn State Worry Questionnaire (PSWQ); Rumination Response Scale (RRS); and NMP-Q. However, not all combinations across the subfactors were explored. We primarily focused on the total score on the NMP-Q for exploring correlations with smartphone sensor features and other psychometrics. In addition, for specific debriefing questions, we examined subfactors of participants’ NMP-Q scores to understand why they felt comfortable with sensitive data being collected via smartphones. The DASS-21 is designed to be interpreted using 3 subscales: depression, anxiety, and stress. The QIDS also includes subfactors even though some subfactors are represented by a single question. In contrast, the RRS, PSWQ, and UCLA Loneliness Scale do not have subfactor representations and, thus, were not explored in this context.
In general, we investigated the correlation between each of these features and the NMP-Q total score. The one exception to this was the debriefing questionnaire features, which we also tested against the NMP-Q subdimensions because of our preassumption that using AWARE-Light and sensor activation might naturally correlate with certain aspects of nomophobia, such as the need for communication or convenience.
Apart from the correlation analysis, we also experimented with constructing some supervised machine learning models with the follow-up nomophobia scores as the outcome to be predicted. In our supervised learning models, we only included data from participants whose location and screen sensor data met the threshold of at least 27 out of the 55 days of availability (which resulted in 27/41, 66% of the participants included in the analysis). If a participant met this requirement for both sensors, then data for all their sensors, their psychometric results, and their EMA data were retained for regression.
After data cleaning, linear regression was applied to 47 extracted features (Multimedia Appendix 4) from 66% (27/41) of the participants, with filtering based on the combination of location and screen sensors. We focused on exploring two general models: (1) those whose input was only passive smartphone sensor features and (2) those whose input combined passive and active data, including EMA, phone sensors, and psychometrics. We implemented both wrapper methods and filter methods to test various combinations of features using R2 to evaluate the models. Of the 47 preselected features, we included those that showed a potential improvement in model performance of >0.01 in terms of R2 results. To predict NMP-Q scores, we used leave-one-out cross-validation for model evaluation due to the small sample size. In addition to the R2 score, we used mean absolute error (MAE), mean absolute percentage error (MAPE), and RMSE to assess the regression errors.