1. Introduction

The number of people suffering from depression is increasing worldwide, seriously affecting the health and well-being of individuals. According to the World Health Organization (WHO), 5 percent of adults globally suffer from depression [1]. However, due to limitations in medical resources, the diversity of symptoms, social stigma, and other factors, many patients with depression are not diagnosed at an early stage. Traditional approaches rely on clinical consultations or questionnaires, such as the Patient Health Questionnaire (PHQ-9) and the Montgomery-Åsberg Depression Rating Scale (MADRS), which are commonly used to assess depression [2]. In recent years, social media use has been on the rise, and the emotions expressed by individuals with depression on these platforms often appear more authentic than those captured by traditional methods, like questionnaires. Therefore, the abundance of real-time, diverse data available on social media offers new opportunities for predicting depression. By collecting data from a single modality on social media (e.g., text, images, audio, or video), previous researchers have analyzed these datasets to assess user characteristics and predict their likelihood of depression. For instance, M. Trotzek et al. employed machine learning models based on social media posts to address the problem of early depression detection [3]. However, single-modal methods, which rely on one type of data, often result in insufficient predictive accuracy and struggle to account for individual differences. To achieve more accurate and effective predictions of depression, researchers have begun exploring multimodal approaches to address these limitations. Multimodal models have indeed been shown to predict depression more effectively than single-modal methods. By effectively integrating data from multiple modes, such as text, images, and audio, researchers can extract more accurate features, leading to improved classification and analysis. As a result, the accuracy of depression predictions is significantly enhanced. However, multimodal approaches still face several challenges, including a lack of publicly available datasets, difficulty in selecting features from different data modalities, and issues with model overfitting. To help researchers better understand and refine multimodal methods, we analyze recent research on both single-modal and multimodal approaches for depression prediction and provide recommendations for future research.

2. Literature Review

2.1. Depression Detection Methods Using Single-Modal Social Media Data

Earlier studies aimed to predict depression using single-modal techniques. Researchers collected text, image, audio, or video information from social media users and analyzed their depression levels by extracting features from a single modality. Single-modal research primarily employed machine learning methods, which included techniques such as linear regression, decision trees, K-nearest neighbors, Naive Bayes, random forests, and support vector machines. Many studies focused on text analysis. For instance, R. Chiong et al. demonstrated the effectiveness of using machine learning methods in their research [4]. However, their approach relied on supervised machine learning classifiers, which require labeled datasets for training. This limitation could be addressed in the future by introducing unsupervised classifiers. Later, Y. Wu et al. introduced a novel approach by defining the basic components of personalized information within the domain of text-based depression prediction and proposing the Personalized Information Embedded (PIE) model [5]. This model emphasized the extraction of personalized information, thereby improving the accuracy of predictions.

As technology advanced, researchers began to apply deep learning methods. Compared to traditional machine learning, the primary advantage of deep learning is its ability to achieve higher accuracy when processing large-scale data. For example, A. Pérez et al. investigated the use of neural language models to detect depression by designing a classification framework [6]. Additionally, J. Singh et al. proposed an embedded long short-term memory (LSTM)-based scheme that utilized natural language processing technology [7]. This embedded neural network scheme effectively predicted users’ emotional states. Furthermore, V. Tejaswini et al. introduced a novel hybrid deep learning neural network model combining FastText, convolutional neural networks, and LSTM [8] [9]. This model integrated NLP with deep learning to represent both global and local features of depression effectively.

Although single-modal methods have produced promising results, they also have limitations, such as unstable classification performance and incomplete feature extraction, leading to inaccurate depression predictions. As a result, researchers have turned to multimodal approaches to improve the prediction of depression.

2.2. Depression Detection Methods Using Multimodal Social Media Data

The multimodal research method analyzes the characteristics of different modalities more comprehensively, resulting in more accurate predictions. Multimodal prediction primarily relies on multimodal fusion methods, which can be broadly categorized into three types: feature-level fusion (early fusion), decision-level fusion (late fusion), and model-level fusion. Most current research focuses on feature-level fusion.

The fusion of text and images has shown promising results. For example, T. Gui et al. proposed a new cooperative multi-agent model [10] that automatically selects relevant text and images from users’ historical posts. These selected posts effectively indicate a user’s depression, and the model demonstrated strong and stable performance in real-world scenarios. Additionally, C. Lin et al. designed a system called SenseMood [11], which used a deep visual-textual multimodal learning approach to reveal users’ psychological states on social networks by analyzing the pictures and text they posted. This system significantly improved depression detection performance on social networks. Some studies also incorporate features from text, video, and audio. For instance, M. Li et al. proposed a Multimodal Fusion Module (MFM) and introduced the IISFD method to enhance comparative learning between samples [12]. There are also other feature fusion methods, such as the one proposed by H. Fan et al., which integrates video, audio, and rPPG signals using a Transformer-based multimodal feature enhancement network (TMFE). This approach was the first to combine rPPG signals with video and audio modes for multimodal depression detection, proving helpful in detecting depression [13]. Z. Li et al. proposed a Multimodal Hierarchical Attention (MHA) model for depression detection on social media [14]. Using the depression detection dataset from Weibo, this model demonstrated good classification performance and accurately identified depressed users on social media.

Feature fusion has also been combined with deep learning techniques. A. Malhotra and R. Jindal proposed a real-time, deep learning-based system for sentiment analysis of users’ multimodal social media posts. The system extracts features from images, text, and videos on a user’s social media through multiple patterns to detect episodes of depression, suicidal ideation, or self-harming behavior. They used deep learning techniques to detect depression and suicidal tendencies in real-time by analyzing multimodal user-generated content [15]. Additionally, P. Mann and A. Paes, along with their colleagues, detected depression using multimodal social media data. They compared deep learning models to feature-engineered models derived from Instagram images and their captions. By creating three types of deep learning and feature-engineering models—text-only, image-only, and text+image fusion—they achieved the best prediction results using ELMo and ResNet-34 classifiers to combine features. This approach could assist in large-scale depression screening [16]. Furthermore, H. Zogan et al. proposed a new computational framework for automated depression detection. The framework automatically summarizes a user’s post history to select relevant user-generated content. This content is then processed through a novel deep learning framework, consisting of a CNN-GRU model, which outperformed existing robust baselines [17]. H. Zhang et al. introduced the neural network hybrid model MTDD to identify depressive tendencies [18]. Based on social platform data, this deep neural network hybrid model utilized multimodal features to learn the vector representation of depression-prone text, proving effective in detecting depressive tendencies.

Using feature fusion methods to construct multimodal sentiment dictionary models has also shown promising results. M. P. M. Tadlagi et al. proposed a multimodal depressive dictionary learning method to detect depressed users on Twitter [19]. This method was developed using benchmark depression and non-depression datasets, along with well-defined discriminative depression-oriented feature groups. It revealed potential differences in online behavior between depressed and non-depressed users on social media.

Recent studies have shown that hybrid fusion methods yield better results than simple early or late fusion approaches. For example, X. Zhang et al. proposed a novel trilateral bimodal encoding model (MEN), attentional decision fusion (ADF), and a feature extraction fusion strategy [20]. This method combined early intra-modal fusion with late inter-modal fusion, representing a hybrid fusion approach.

Although the accuracy of depression prediction has significantly improved with the use of multimodal methods, there are still some limitations: public datasets suitable for research remain insufficient, there is a risk of disclosing the privacy of depression patients, and the constructed models may suffer from overfitting. Therefore, the multimodal approach still has considerable room for improvement, offering opportunities for further exploration by researchers.

3. Discussion

3.1. General Process of Multimodal Depression Detection Methods

Based on recent literature, we summarize the general steps of multimodal prediction methods, as shown in Fig. 1:

(1) Multimodal data acquisition: Collecting text, image, video, audio, and other data from users on social media.

(2) Data cleaning and preprocessing: Cleaning the collected data to remove noise and outliers, and preprocessing it to facilitate subsequent analysis.

(3) Single-mode feature extraction: Using the appropriate techniques for each mode to extract relevant features.

(4) Multimodal feature fusion: Fusing the extracted features, which can involve simple feature fusion or deep learning-based methods.

(5) Model construction and training: Selecting an appropriate model and training it using the datasets.

(6) Model evaluation and optimization: Evaluating the model’s performance using suitable metrics and further optimizing it.

Figure 1

3.2. Comparative Study of Multimodal and Single-modal Methods

Table 1. Comparison of single-modal and multimodal datasets and techniques

Table 1 shows that datasets used in single-modal methods typically contain only one type of data, and the techniques applied are relatively simple. However, relying on a single source of information may not fully capture the patient’s true condition. In contrast, multimodal datasets integrate multiple types of data and employ more advanced techniques, which significantly enhances prediction accuracy.

3.3. Comparative Study of the Modes and Techniques Used in Multimodal Methods

Table 2. Comparison of Modes and Techniques Used in Multimodal Approaches

According to the comparison shown in Table 2, it is evident that multimodal feature fusion is the most commonly used method among multimodal approaches. The most frequently used modalities are text, images, and videos. Feature extraction techniques predominantly include the TF-IDF model, Word2Vec model, GloVe, BERT model, SVM, CNN, and others. For feature fusion, commonly used techniques are FC (Fully Connected layer), LSTM, and CNN. Multimodal feature fusion methods can range from simple feature fusion to combinations of machine learning and deep learning models.

4. Conclusion

By analyzing and comparing single-modal and multimodal depression detection methods on social media, it is apparent that while single-modal depression prediction has certain application value, it is limited by issues related to information comprehensiveness and prediction accuracy. Multimodal depression prediction, on the other hand, provides more comprehensive and rich information by combining multiple data types, thereby improving prediction accuracy. However, the multimodal approach still faces challenges, including insufficient publicly available datasets suitable for research, complexities in data feature extraction and fusion, and overfitting in constructed models. To enhance the predictive performance of multimodal methods, researchers should focus on developing efficient multimodal data acquisition techniques, optimizing the fusion methods for multimodal data, and addressing individual differences between samples. The application of multimodal prediction for detecting depression in patients on social media holds significant potential. Researchers should foster interdisciplinary collaboration among psychology, computer science, neuroscience, and other fields, and increase clinical testing to contribute to the early detection of depression.