Depression is a pervasive mental health disorder affecting over 300 million people worldwide and consistently ranks among the leading contributors to the global disease burden [1]. Beyond its psychological toll, depression adversely impacts behavior and physical health, increasing the risk of cardiovascular diseases and other chronic conditions [2]. Early detection and timely intervention are crucial to improving outcomes; however, many individuals avoid professional help due to stigma, shame, or misconceptions [3], leading to delayed treatment and a higher suicide risk [4].

Traditional diagnostic methods, such as structured clinical interviews and self-report questionnaires (eg, Patient Health Questionnaire-8, PHQ-8 [5]), are reliable but limited in scalability. They rely on disclosure, clinician availability, and structured settings, making them impractical for frequent or large-scale screening. These limitations have motivated research into artificial intelligence (AI)–based depression detection, which leverages multimodal behavioral cues—including linguistic, acoustic, and visual signals—to provide scalable and more objective assessments.

Although promising, current AI approaches face several challenges. First, most models are trained on structured clinical interviews and generalize poorly to spontaneous conversations, limiting adaptability. Second, high-quality annotated datasets remain scarce, restricting the ability to capture subtle linguistic and emotional cues. Third, emotional signals in natural settings are often weak or suppressed, reducing sensitivity without external stimulation. Finally, many state-of-the-art architectures are computationally heavy and unsuitable for real-time deployment on resource-constrained platforms such as socially assistive robots (SARs).

Existing research on depression detection primarily relies on 3 modalities: text, audio, and visual. Advances in machine learning and deep learning have improved predictive performance; however, challenges related to data scarcity, feature robustness, and model generalizability continue to limit clinical deployment. Although text remains the most informative modality, incorporating acoustic and visual cues provides complementary behavioral signals, which motivates multimodal approaches.

Textual analysis plays a central role in automated depression detection, as linguistic content directly reflects psychological states. Prior studies have leveraged word embeddings and semantic representations to capture depression-related language patterns [6], while affective and mental health lexicons have been introduced to enhance emotional interpretability [7]. Analyses of clinical interview datasets consistently show that text-based features outperform acoustic and visual modalities in predictive accuracy [8-10]. This advantage arises from both the respondent’s verbal expressions and the structured conversational context shaped by interviewer prompts, enabling fine-grained semantic and emotional analysis.

Speech-based approaches commonly extract low-level acoustic descriptors, such as Mel-frequency cepstral coefficient (MFCCs) and extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), to capture prosodic and voice-quality cues associated with depression [8,9]. Deep learning models, including convolutional neural networks, further learn discriminative speech representations [10]. However, acoustic features are sensitive to speaker variability, background noise, and contextual factors, which hinder generalization.

Visual cues, including facial expressions, gaze patterns, and action units (AUs), provide additional nonverbal indicators of depressive behavior. While these features capture subtle expression dynamics, their reliability is limited in practice, as individuals with depression often exhibit subdued or inconsistent facial activity influenced by personal traits and environmental conditions.

Multimodal fusion frameworks integrate textual, acoustic, and visual signals to enhance robustness and sensitivity. Empirical evidence from clinical interview studies indicates that although speech and visual cues contribute auxiliary information, textual features remain the dominant predictors [8-10]. Interview transcripts encode not only linguistic content but also interaction patterns and contextual flow, making them essential for depression assessment. Nevertheless, most existing multimodal systems rely on structured interview settings, limiting their adaptability to spontaneous, everyday conversations.

To address weak emotional expression in natural settings, recent work has emphasized emotional reactivity as a key factor in depression [11]. Emotion-induction techniques help elicit observable affective responses, enriching behavioral signals and improving model interpretability.

Depression involves an interaction between mood—a persistent affective state—and emotion, which refers to stimulus-driven responses. A meta-analysis of laboratory studies on major depressive disorder demonstrated attenuated emotional reactivity to both positive and negative stimuli, such as affective images and videos [12]. Emotion-induction methods range from film clips and images to autobiographical recall and immersive environments. Although effective in eliciting transient emotional responses, these approaches often lack ecological validity and are rarely integrated into conversational or cross-cultural AI systems.

Recent studies have explored multimodal and affective conversational systems. Large language model (LLM)–based models [13] excel at generating coherent and empathetic dialogue but are typically resource-intensive, lack multimodal perception, and do not incorporate explicit emotion-induction mechanisms. Multimodal fusion frameworks such as Zhang et al [14] focus on integrating multiple modalities but do not prioritize lightweight deployment or hierarchical fusion. Hsieh et al [15] emphasize efficiency but lack support for natural conversation, while Li et al [16] introduce compact multimodal deep fusion architectures without emotion induction or support for free-form dialogue beyond structured interviews.

In contrast, our framework integrates natural conversational interaction, multimodal perception, hierarchical deep fusion, lightweight deployment, and explicit emotion induction. Support for natural conversation enables the analysis of unscripted dialogue; multimodal fusion enhances robustness through complementary signals; lightweight design allows deployment on resource-constrained platforms; deep fusion facilitates cross-modal interaction at multiple levels; and emotion induction actively elicits affective responses, improving detection sensitivity. To position our framework within the existing literature, we compare representative systems across 5 key dimensions: natural conversational support, multimodal fusion, lightweight deployment, hierarchical deep fusion, and emotion induction. As prior work typically focuses on only 1 or 2 of these aspects, Table 1 provides a consolidated comparison to clarify the distinctive contributions of our approach.

To address these gaps, we propose Depression Social Assistant Robot (DEPRESAR)–Fusion, a lightweight multimodal depression detection framework designed for naturalistic interaction with SARs. The framework is built upon 3 primary objectives. First, it enables emotion-aware SAR interactions by supporting spontaneous, everyday conversations as the foundation for depression detection. Second, it incorporates emotion induction and adaptive engagement mechanisms by integrating evocative video stimuli with an LLM-based conversational response module, thereby enhancing emotional expressiveness and contextual sensitivity during dialogue. Third, it uses a lightweight multimodal deep fusion architecture that combines linguistic, acoustic, and visual features within a resource-efficient design suitable for real-time deployment on SAR platforms.

By bridging the gap between structured clinical assessments and natural conversations, DEPRESAR-Fusion advances scalable, nonintrusive, and emotionally aware AI-driven mental health support.

Our proposed system, DEPRESAR-Fusion, was designed for real-time depression detection through natural user interactions (Figure 1). It began with an emotion-induction video to encourage spontaneous emotional responses, followed by a casual conversation with an SAR. During the interaction, multimodal data—including speech, facial expressions, and text—were captured and processed in real time.

Key features were extracted by a data preprocessing module, which encoded audio (eg, MFCCs and eGeMAPS), visual (eg, gaze, head pose, and facial AUs), and transcribed text data. If the user explicitly ended the conversation, the system proceeded to depression prediction. Otherwise, an LLM-powered, emotion-aware social robot dynamically responded via the emotional support conversation (ESC) framework, enhanced with self-consistency prompting to ensure reliable, context-aware, and empathetic interactions. The extracted multimodal features were then passed to a dual-head depression detection module that jointly performed binary classification and PHQ-8 score regression. This architecture enabled simultaneous estimation of depressive presence and severity while maintaining an engaging and supportive user experience.

Our experiments were conducted on the Extended Distress Analysis Interview Corpus (E-DAIC), an enhanced version of the Distress Analysis Interview Corpus-Wizard-of-Oz (DAIC-WOZ) dataset, both designed for multimodal depression and anxiety detection from structured clinical interviews. DAIC-WOZ consisted of semistructured interviews conducted via a Wizard-of-Oz setup and included synchronized audio, video, and transcribed speech annotated with psychological assessment scores.

E-DAIC extended DAIC-WOZ with curated splits and standardized multimodal features for machine learning research. It contained 189 interview sessions from 100 participants, providing audio recordings (for acoustic features such as MFCC and eGeMAPS), video recordings (for facial expression and gaze analysis), time-aligned transcripts, and expert-rated PHQ-8 scores as ground truth labels.

Depression severity was measured using the PHQ-8 [5], a validated 8-item instrument ranging from 0 to 24, with scores ≥10 indicating clinically significant depressive symptoms (approximately 88% sensitivity and specificity).

In this study, E-DAIC served as the primary dataset for model training and evaluation, while the DAIC-WOZ benchmark subset was used to assess generalizability and facilitate comparison with prior work.

To enhance model generalization under limited data conditions, we implemented 2 complementary data augmentation strategies involving both text-only and multimodal synthetic data generation.

First, synthetic text-only interview samples were generated using publicly available social media depression datasets, including Reddit Self-reported Depression Diagnosis and Longitudinal Twitter Dataset of Depression in the COVID Era [17,18]. In this pipeline, user posts were segmented into response-style utterances using spaCy to simulate interview dialogue structure. Each segmented sample was temporally aligned to fixed 3-second intervals, while nontextual modalities were padded to maintain input format consistency. Because these datasets provided only binary depression labels, PHQ-8 regression scores were approximated using a thresholding strategy in which depressed samples were assigned a score of 10 and nondepressed samples a score of 0.

Second, synthetic multimodal samples were generated by paraphrasing and restructuring the original structured interview transcripts from E-DAIC [19]. Paraphrasing was performed using ChatGPT (GPT-4o) while preserving the original semantic meaning. Importantly, all paraphrasing was strictly applied only to the training split after dataset partitioning. Test set transcripts were never paraphrased or used in any form during synthetic data generation, ensuring that no semantic leakage occurs. Audio and visual features were resampled from corresponding original samples and temporally aligned with the modified transcripts to construct fully synthetic multimodal instances.

Together, these augmentation strategies improved robustness and enabled the model to better accommodate linguistic variability and multimodal diversity encountered in real-world conversational settings.

The system captured and processed 3 complementary modalities—audio, visual, and text—to extract depression-related features. For the audio modality, acoustic features, including MFCCs and eGeMAPS descriptors, were extracted using openSMILE [20], enabling the capture of prosodic patterns and voice quality characteristics associated with depressive states.

For the visual modality, nonverbal behavioral markers, such as gaze direction, head pose, and facial AUs, were obtained using OpenFace [21]. These features provided cues related to affective expression and social engagement.

For the textual modality, semantic representations were derived from speech transcriptions using pretrained MiniLM [22] and bidirectional encoder representations from transformers [23] embeddings. Both word-level and sentence-level embeddings were incorporated to capture contextual semantics and linguistic indicators of depression.

All extracted features were temporally aligned and encoded to facilitate downstream multimodal fusion and depression assessment.

To integrate heterogeneous data, the system first normalized and encoded each modality into a shared embedding space. Audio and visual features were transformed using a bag-of-audio-words or bag-of-visual-word framework [24,25], which mapped variable-length sequences into fixed-size distributional vectors via modality-specific codebooks. These normalized embeddings were then temporally aligned and passed to an attention-based fusion module. We used both cross-attention and self-attention mechanisms [26] to model intermodal and intramodal relationships: cross-attention aligned textual features with relevant nonverbal cues, while self-attention captured contextual dependencies within each modality. The fused multimodal representation was then processed by a dual-head architecture (Figure 2). The classification head output the probability of clinical depression, while the regression head predicted a continuous PHQ-8 score. This unified design supported both categorical and continuous interpretations of depression severity, and attention-based fusion enhanced robustness to noise and missing data, improving overall performance. To handle data scarcity in regression targets, we scaled the regression loss.

LregL_reg (around 15), higher than the classification loss L_cls (around 0.5) during training, ensured that the model gave adequate learning attention to the regression head. Training setup—we minimized a multitask objective combining binary cross-entropy (for classification) and mean squared error (for regression): L=α·L_cls+β·L_reg with α=1.0, β=1.0. We used the Adam optimizer (learning rate 10^–5, weight decay 10^–4, batch size 8, and dropout rate 0.1).

DEPRESAR-Fusion operated on a 2-tier architecture: an Android application running on the Kebbi robot by Nuwa Robotics and a remote Windows-based server. This structure balanced real-time interaction with computational efficiency.

We evaluated the proposed multimodal fusion model for depression detection on the E-DAIC dataset, supplemented with synthetic data derived from the Reddit Self-reported Depression Diagnosis and Longitudinal Twitter Dataset of Depression in the COVID Era datasets to enhance generalizability. Performance was assessed using both classification metrics (eg, F₁-score) and regression metrics (eg, root mean square error [RMSE], mean absolute error [MAE], and concordance correlation coefficient [CCC]) to evaluate binary depression detection and continuous PHQ-8 score prediction, respectively.

To benchmark our approach, we compared it against a diverse set of representative unimodal and multimodal baselines spanning different architectural paradigms. Text-based approaches included the graph convolutional network proposed by Burdisso et al [29], which classified depressive symptoms from interview transcripts, as well as our prior lightweight text-only model (Hsieh et al [15]), which leveraged linguistic signals for depression detection.

Several multimodal deep learning frameworks were also considered. Dham et al [30] integrated audio, visual, and textual features for depression severity estimation, while Li et al [16] proposed a Flexible Parallel Transformer that incorporated handcrafted audiovisual features for enhanced multimodal fusion. Yin et al [31] introduced a hierarchical recurrent neural network designed to capture temporal dependencies across audio, visual, and text modalities, and Zhang et al [14] combined speech and linguistic features for PHQ-8 regression. Additionally, Lau et al [32] developed a parameter-efficient bidirectional long short-term memory with attention, adapted via prefix-tuning for PHQ-8 prediction.

We further evaluated LLMs, including GPT-4o, o1, and o3-mini (OpenAI and Anthropic), under zero-shot settings for textual inference of depressive symptoms.

Together, these baselines cover a broad spectrum of architectures and modality configurations, enabling a rigorous and comprehensive comparison with our proposed lightweight multimodal framework.

This study protocol was approved by the institutional review board of the National Taiwan University Hospital (202105013RINB), and written informed consent was obtained from all participants prior to participation. To protect participant privacy and confidentiality, all collected data were deidentified and securely stored, and access to the data was restricted to authorized members of the research team only. Participants received NT $200 (US $6.29) as compensation for their participation.

To evaluate the ability of the models to estimate depression severity as a continuous score, we assessed regression performance using 3 standard metrics: RMSE, MAE, and CCC [33]. RMSE captured the average squared difference between predicted and actual PHQ-8 scores, placing greater emphasis on larger errors. MAE measured the average absolute deviation between predictions and ground truth, providing a more interpretable indication of overall prediction error. CCC evaluated both precision and accuracy by assessing the degree to which predicted values conformed to the ground truth in terms of scale and location, thereby offering a more comprehensive measure of agreement.

As shown in Table 2, on the E-DAIC dataset, our proposed model achieved the best RMSE and MAE scores, outperforming all other methods, including those using larger or specialized multimodal architectures. While OpenAI’s o3-mini attained the highest CCC, it did so at the cost of significantly more parameters and computational overhead. In contrast, our model offered competitive concordance with minimal complexity.

Table 3 further validates the strength of our method, which achieved state-of-the-art RMSE and MAE results on the DAIC-WOZ benchmark. Notably, our model also achieves the highest CCC (0.6603), suggesting that it better aligns with ground-truth depression scores in both magnitude and scale.

As shown in Table 4, we evaluated the binary classification performance of our proposed method and several baselines on the E-DAIC test split. The objective was to distinguish between depressed and nondepressed individuals based on the standard PHQ-8 cutoff threshold (≥10). Due to the limited public availability of directly comparable classification results on other datasets, we restricted our binary classification evaluation to E-DAIC, which was widely adopted in prior research.

To account for class imbalance, we reported 3 standard F₁-score–based metrics: depression F₁-score, control F₁-score, and macro F₁-score. Depression F₁-score treated the depression class as the positive class, while control F₁-score treated the nondepression (control) class as the positive class. Macro F₁-score was computed as the unweighted average of the depression F₁-score and control F₁-score, providing a balanced evaluation across both classes.

The proposed model achieved state-of-the-art performance in both regression and classification tasks while maintaining an exceptionally small footprint of 617K parameters. On the E-DAIC benchmark, our model attained an RMSE of 5.46 and an MAE of 4.35, outperforming prior multimodal approaches such as Zhang et al [14] (RMSE 6.11) and Yin et al [16] (RMSE 5.50). Similar trends were observed on the DAIC-WOZ dataset, where our method achieved an RMSE of 4.38, an MAE of 3.40, and the highest CCC of 0.66, indicating stronger agreement with ground-truth PHQ-8 scores compared to existing baselines.

In binary classification on the E-DAIC dataset, the proposed framework achieved a macro F₁-score of 0.8297, substantially exceeding both traditional multimodal models and zero-shot LLMs. Notably, the model demonstrated balanced detection capability, with a depression F₁-score of 0.7647 and a control F₁-score of 0.8947, which was critical in clinical screening scenarios to minimize both false negatives and false positives.

Despite its compact size, the proposed model consistently outperformed parameter-heavy LLM baselines such as GPT-4o and o3-mini, which required orders of magnitude more parameters. These quantitative results highlighted three key strengths of our approach: (1) state-of-the-art accuracy with minimal parameters, establishing a new lightweight benchmark for multimodal depression detection; (2) effective joint modeling of regression and classification through a simple multitask 2-head architecture; and (3) robust generalization across both DAIC and E-DAIC datasets, supporting deployment on resource-constrained platforms such as SARs.

To quantify the contribution of each modality in our multimodal framework, we performed an ablation study by selectively disabling feature groups during both training and testing. Each ablated modality was replaced with random noise, ensuring that the architecture and learning dynamics remained consistent across settings.

This study introduces DEPRESAR-Fusion, a lightweight multimodal depression detection framework designed for naturalistic interaction with SARs. Across both regression and classification tasks, the proposed model demonstrated strong predictive performance while maintaining an exceptionally compact architecture (617K parameters).

On PHQ-8 regression tasks, the model achieved the lowest RMSE and MAE on the E-DAIC dataset and state-of-the-art performance on the DAIC-WOZ benchmark, while maintaining competitive CCC values. These results indicate that the proposed dual-head architecture can effectively estimate both depressive severity and categorical status within a unified framework.

In binary classification, the model achieved a macro F₁-score of 0.8297 on E-DAIC, demonstrating balanced performance across depressed and nondepressed classes. Importantly, the system maintained stable performance during real-world deployment on a commercial SAR platform, suggesting practical feasibility beyond controlled benchmark settings.

The ablation study revealed that linguistic features—particularly word-level embeddings—contribute the most substantial predictive signal. Acoustic and visual modalities provided complementary improvements, enhancing robustness without dominating the predictive process. Furthermore, the pilot deployment indicated that emotion induction may increase expressive signal strength and improve downstream detection performance, suggesting the practical value of integrating affect elicitation into conversational AI systems.

Our findings are consistent with prior research demonstrating the dominant predictive role of textual features in depression detection. Studies based on clinical interview transcripts have shown that linguistic content often outperforms acoustic and visual modalities in predictive accuracy [8-10,34,35]. Our ablation results support this observation, as removing word-level embeddings resulted in the largest degradation in performance.

Compared with multimodal architectures, such as Yin et al [31], Zhang et al [14], and Li et al [16], our model achieves lower RMSE and MAE while maintaining a substantially smaller parameter footprint. While transformer-based or hierarchical recurrent frameworks emphasize deep cross-modal modeling, our attention-based fusion module demonstrates that efficient structured integration can achieve competitive or superior results with significantly reduced computational cost.

Relative to parameter-efficient text-only approaches such as Lau et al [32], our multimodal framework provides improved regression consistency, suggesting that incorporating behavioral cues beyond text enhances severity estimation. Additionally, compared with zero-shot LLMs (eg, GPT-4o, o3-mini), our model achieves stronger classification performance despite operating at a fraction of the parameter scale, highlighting the value of task-specific multimodal design over parameter scaling alone.

Finally, while psychological literature has established altered emotional reactivity in depression [11,12], few computational systems explicitly integrate emotion induction into multimodal detection pipelines. Our pilot findings suggest that embedding affect elicitation within conversational interaction may enhance behavioral signal salience in naturalistic settings.

This study has several limitations that warrant careful consideration. The real-world deployment involved a relatively small sample size (n=22), which constrains statistical power and limits the precision of effect size estimation. Although certain differences—particularly in the emotion-induction condition—reached statistical significance, the modest sample increases uncertainty regarding robustness and reproducibility. Larger-scale studies are necessary to confirm stability across broader populations.

Participants were primarily young adults affiliated with a university setting, resulting in a relatively homogeneous demographic profile. Such sampling may introduce selection bias and restrict external validity. Depression presentation varies across age groups, socioeconomic contexts, and cultural backgrounds. The generalizability of the current findings to older adults, clinical psychiatric populations, or cross-cultural environments, therefore, remains to be established.

PHQ-8 was used as the sole ground truth indicator of depressive symptom severity. While widely validated and frequently adopted in computational depression research, PHQ-8 is a self-report instrument rather than a clinician-administered diagnostic interview. Self-reported measures may be influenced by reporting bias, transient emotional states, or social desirability effects. Incorporating structured diagnostic interviews or multimethod assessment frameworks would strengthen clinical validity.

The emotion-induction protocol relied on a fixed set of emotionally evocative video stimuli. Emotional reactivity varies across individuals, and standardized stimuli may not elicit comparable affective responses in all participants. Although improved predictive performance was observed following induction, the study design does not fully disentangle whether the performance gain reflects increased emotional expressiveness, cognitive priming, or heightened engagement. More controlled experimental paradigms—including neutral baseline conditions or counterbalanced stimulus designs—would allow for clearer causal interpretation.

The data augmentation strategy also introduces considerations. Synthetic text samples were generated exclusively from training data to prevent information leakage; however, approximating PHQ-8 regression targets through threshold-based assignment reduces label granularity. This simplification may limit the precision of severity calibration. In addition, paraphrased transcripts, while semantically constrained, may differ distributionally from naturally occurring spontaneous conversations, potentially affecting generalization.

The evaluation framework focuses on cross-sectional assessment rather than longitudinal monitoring. Depression is dynamic and episodic, and a single interaction may not capture temporal symptom fluctuations. Long-term validation is needed to determine sensitivity to clinical change and suitability for continuous monitoring scenarios.

Finally, while attention mechanisms provide partial insight into modality weighting, the interpretability of specific multimodal feature contributions remains limited. Greater integration of explainability tools and clinician-in-the-loop evaluation would be necessary before large-scale clinical deployment.

Future research should explore cross-cultural and multilingual generalization to broaden applicability across diverse populations. The emotion-induction mechanism may be further optimized through adaptive or reinforcement-based strategies to enhance user engagement.

In addition, incorporating continual and personalized learning mechanisms could support longitudinal adaptation without full retraining. Future work should also integrate explainability tools and conduct systematic failure case analyses to improve robustness and clinical interpretability. Finally, more controlled experimental designs, such as neutral-video conditions, are needed to better isolate emotion-specific induction effects.

DEPRESAR-Fusion demonstrates that compact multimodal architectures can achieve state-of-the-art depression detection performance while remaining suitable for real-time SAR deployment. By integrating multimodal perception, lightweight deep fusion, and emotion-aware interaction mechanisms, this work advances practical, scalable, and field-validated approaches to AI-driven mental health support.