1. Introduction

Advances in single-cell measurement technologies have enabled cell-resolution acquisition of multimodal biological data, including transcriptomes and chromatin accessibility. The rapid growth of these datasets has spurred the development of deep learning methods for multimodal biological data analysis, aiming to elucidate biological states and facilitate applications such as drug discovery [1]. Recent studies have identified fundamental tasks for multimodal cell data analysis [2]: ​​integration​​ and ​​imputation​​. Figure 1 illustrates definitions of multimodal integration and imputation.

​​Integration​​ involves bridging modality gaps and identifying biological clusters in a unified multimodal atlas to reveal cell identity and function. ​​Imputation​​ aims to recover missing signals caused by measurement noise. While intra-modal imputation restores zero or low-expression values in the original data, ​​cross-modal imputation​​ reconstructs data across modalities in the absence of joint measurements. Although methods exist for separate integration and imputation, no unified benchmark addresses both tasks. This study reviews existing methods and establishes a joint benchmark to promote multimodal biological data analysis.

For integration, multimodal datasets typically consist of unpaired cells with cross-modal feature mismatches. Matrices represent cell data, where rows are samples (cells) and columns are features (genes/proteins). For different modalities, feature spaces are independent, and inter-modality samples lack pairing information. Traditional methods project multimodal data into a joint feature space, but this can disrupt biological heterogeneity [3-5]. Recent approaches transform integration into end-to-end cell representation learning [6-8]. Techniques like Concerto (contrastive learning) [10], scBERT (Transformer-based gene representations) [11], and CLUE (cross-modal guided graphs) [12] demonstrate advancements in cell representation learning. Monae combines these perspectives while ensuring computational efficiency [13].

For imputation, dropout events in scRNA-seq data introduce high noise, complicating downstream analyses. Intra-modal imputation recovers true values from zero counts, while cross-modal imputation predicts missing modalities using complementary signals [14]. Most existing methods target RNA modality, divided into model-based [15-19] and deep learning-based [20-23] approaches. Recent methods support multimodal integration and cross-modal imputation [24]. Monae uniquely handles unpaired scenarios and jointly performs integration and imputation [13].

​In summary​​, current methods for multimodal biological data integration and imputation have established their own benchmarks independently, lacking unified datasets and standardized evaluation metrics. This fragmentation hinders direct comparisons and progress in the field. Therefore, this study aims to collect and systematically evaluate a broad range of existing methods for integration and imputation, establishing a reliable and comprehensive benchmark. This effort will provide a foundation for future research in multimodal biological data analysis, fostering the development of more robust and generalizable tools.

2. Methods

We summarize representative methods for integration and imputation tasks, selecting benchmarking candidates as shown in Table 1. Datasets used in benchmarks are detailed in Table 2.

2.1. Integration

Integration of multimodal cell data can be categorized into two primary types: ​​semi-supervised​​ and ​​unsupervised​​ methods. Semi-supervised methods rely on paired observations across modalities. For instance, ​​MultiVI​​ employs a series of variational autoencoders to learn from multimodal data and align joint embeddings [6]. ​​CLUE​​ introduces auxiliary tasks to facilitate alignment between heterogeneous feature spaces [12]. ​​scMoGNN​​ leverages graph neural networks (GNNs) to identify connections between different cell modalities and integrates them using established single-modal embeddings [25]. However, due to the high cost of obtaining paired cell data, unsupervised methods have gained popularity. These methods merge multimodal measurements without relying on paired data. Techniques like ​​Seurat​​ [3], ​​LIGER​​ [4], and ​​Harmony​​ [5] first perform functional transformations to resolve differences within shared spaces, but such transformations may compromise inherent biological heterogeneity. The matrix factorization method ​​iNMF​​ [26] circumvents the need for projections but lacks the ability to capture complex biological interactions due to its linearity. Other methods, such as ​​bindSC​​ [27], are limited by dependencies on specific modalities (e.g., RNA and ATAC) and scalability issues. ​​Pamona​​ [28] uses nonlinear manifold alignment to synchronize data across modalities but struggles with uniformly distributed data. ​​GLUE​​ [29] fuses multimodal information via guided graphs, though its integration outcomes remain suboptimal. ​​CoVEL​​ [9] enhances cell representation by fusing information from single-cell multimodal data, demonstrating improved integration performance through multi-view embedding learning. However, its complex architecture limits applicability across diverse datasets. ​​Monae​​ addresses these limitations by refining CoVEL into an efficient integration framework [13].

2.2. Imputation

Most imputation methods focus on the ​​RNA modality​​ and are classified into ​​model-based​​ and ​​deep learning-based​​ approaches. Model-based methods recover gene expression counts using similarity relationships (e.g., gene-gene or cell-cell correlations). ​​SAVER​​ [15] improves expression estimates by leveraging gene-cell associations, while ​​MAGIC​​ [16] shares information across similar cells via data diffusion to account for dropout events. Deep learning methods, such as ​​AutoImpute​​ [30], learn inherent data distributions through autoencoders. ​​DCA​​ [23] introduces a zero-inflated negative binomial model to capture nonlinear gene-gene correlations and sparse count distributions. However, these single-modal methods lack signals from complementary modalities. Only a few techniques support multimodal data. ​​UnitedNet​​ [14] performs cross-modal imputation via multitask learning but requires paired cell data. ​​scButterfly​​ [31], ​​MultiVI​​ [6], and ​​JAMIE​​ [32] enable multimodal integration and cross-modal imputation but depend on partial pairing. ​​Monae​​ uniquely supports unpaired scenarios and simultaneously handles intra-modal and cross-modal imputation [13].

3. Experiments

3.1. Experimental settings

We employed four publicly available multimodal datasets in our study. Dataset 1, ​​10xMultiome[33]​​, comprised 9,631 human peripheral blood mononuclear cells (PBMCs) profiled using 10x Genomics Multiome technology to jointly measure transcriptomic and chromatin accessibility features. Dataset 2, ​​Chen2019​​[34], included 9,190 mouse cortical cells generated via SNARE-seq, with cell pairing modified to simulate unpaired multimodal scenarios. Dataset 3, ​​Ma2020[35]​​, contained 32,231 mouse skin cells assayed by SHARE-seq across four batches per modality, enabling batch effect evaluation. Dataset 4, ​​Muto2021[36]​​, featured 44,190 human kidney cells (19,985 RNA and 24,205 ATAC profiles) generated through snRNA-seq and snATAC-seq, with five batches per modality to test scalability under complex unpaired conditions. For integration performance assessment, we computed a composite score integrating batch correction metrics (SAS, GC, batch.r) and biological fidelity metrics (MAP, cell type ASW, NC). Imputation efficacy was evaluated using AMI, ARI, NMI, and HOM metrics. All experiments adopted a standardized 10% random cell split for validation to ensure robustness.3.2.Multimodal Integration Benchmarks.

Results on four datasets are shown in Tables 3–6.

Based on the assessment results of four multimodal integration methods across four representative datasets, significant disparities in method performance were observed across key metrics. The detailed findings are summarized as follows:

3.2.1.Methods with outstanding overall performance​​

3.2.1.1. CoVEL​​

CoVEL demonstrated exceptional cross-modal prediction accuracy (MAP scores of 0.857 in Dataset 3 and 0.937 in Dataset 4), ranking among top-tier methods with consistent overall scores (0.679–0.686). Despite a relatively low Normalized Mutual Information (NC) score of 0.2 in Dataset 3, its robustness in cross-modal data fusion stood out, achieving high Shared Neighbors Consistency (SAS = 0.914).

3.2.1.2. Monae series​​

​​Monae-E​​ exhibited superior performance in ​​graph connectivity (GC)​​ and ​​biological consistency (bio.c)​​ metrics (e.g., GC=0.978, bio.c=0.645 in Dataset 4), underscoring its strength in integrating biological features.Monae​​ achieved a balanced overall score of ​​0.736​​ (Dataset 3) and ​​0.757​​ (Dataset 4), with a notable MAP value of ​​0.974​​. However, its NC score remained relatively low (0.153–0.193), indicating room for improvement in cross-modal cell type annotation consistency.

3.2.2. Methods with specific metric advantages​​

3.2.2.1. GLUE​​

Excelled in batch effect removal (batch.r) and shared neighborhood consistency (SAS) (e.g., batch.r=0.908 in Dataset 4, SAS=0.9 in Dataset 2). However, its NC (0.2 in Dataset 3) and MAP (0.829 in Dataset 3) scores were suboptimal, limiting its utility in cell annotation and cross-modal prediction tasks.

3.2.2.2. scButterfly​​

Achieved high MAP values (0.812 in Dataset 1, 0.875 in Dataset 2) but underperformed in NC (0.181 in Dataset 4) and overall score (0.687 in Dataset 4). It is best suited for scenarios prioritizing cross-modal mapping consistency over global integration.

3.2.2.3. JAMIE​​

Showcased strong performance in omics layer ASW (0.831 in Dataset 2) but lagged in NC (0.099 in Dataset 3) and overall score (0.443), reflecting a focus on local structure preservation rather than holistic integration.

3.2.3. Characteristics of other methods​​

3.2.3.1. MultiVI​​

Delivered moderate performance in biological consistency (GC and bio.c) (e.g., GC=0.872 in Dataset 4) but struggled with NC (0.167–0.495) and MAP (0.495 in Dataset 4). It is recommended for applications emphasizing biological fidelity over cross-modal alignment.

3.2.4. Summary and recommendations​​

3.2.4.1. Overall performance recommendations​​

Monae-E​​ is ideal for tasks requiring balanced biological interpretability and cross-modal prediction (e.g., GC, bio.c, MAP).CoVEL​​ excels in prediction accuracy and cross-modal consistency, making it suitable for cell state mapping-intensive scenarios.

3.2.4.2. Scenario-specific selections​​

GLUE​​ is preferred for batch effect mitigation.scButterfly​​ is advisable for optimizing cross-modal mapping consistency.

3.2.4.3. Limitations and future directions​​

Most methods struggle to balance NC (cross-modal annotation consistency) and batch.r (batch effect removal), highlighting the need for algorithms that synergize local precision with global stability. Future work should focus on unified frameworks addressing these trade-offs.

3.3. Intra-modal imputation benchmarks (Muto2021 dataset)

The intra-modal imputation benchmark focused on the Muto2021 dataset, where RNA and ATAC modalities are completely unpaired. This challenging setup evaluates algorithms’ ability to recover missing signals without direct cross-modal correspondence. Results are presented in Table 7.

The unpaired Muto2021 dataset posed challenges for imputation robustness. Monae-E ranked first in HOMe (0.918) and NMI (0.735), excelling in cross-modal distribution alignment. Monae closely followed (AMI=0.729, NMI=0.730). Traditional methods like MultiVI (HOMe=0.929) and MAGIC (HOMe=0.931) underperformed (NMI <0.70), while DCA achieved moderate NMI (0.591) but lagged in HOMe (0.907). These results validate deep learning’s superiority for unpaired intra-modal imputation, with Monae-E’s optimized joint learning strategy achieving superior cross-modal representation.

3.4. Cross-modal imputation benchmarks (Muto2021 dataset)

Cross-modal imputation experiments used the Muto2021 dataset, featuring completely unpaired RNA and ATAC modalities, to assess methods’ robustness in reconstructing missing modalities without direct pairwise correspondence. Results are shown in Tables 8–9 (RNA→ATAC and ATAC→RNA directions).

3.4.1. Overall performance comparison​​

3.4.1.1. ​​scButterfly excels in RNA→ATAC direction​​

NMI reached 0.645​​, and ​​HOM achieved 0.923​​, significantly outperforming other methods (e.g., MultiVI: NMI=0.646, HOM=0.872).AMI (0.703)​​ and ​​ARI (0.297)​​ also demonstrated strong capability in capturing cross-modal joint distributions.

3.4.1.2. ​MultiVI shows robustness in ATAC→RNA direction​​

NMI=0.637​​ and ​​HOM=0.883​​ highlighted its strength in preserving global structure.However, ​​ARI (0.211)​​ and ​​AMI (0.61)​​ were relatively low, indicating weaker fine-grained alignment.

3.4.1.3. ​Monae exhibits directional variability​​

RNA→ATAC​​: NMI=0.764, HOM=0.934 (Table 8).ATAC→RNA​​: NMI=0.66, HOM=0.895 (Table 9).Performance discrepancies likely reflect directional data distribution shifts, yet Monae still outperformed most traditional methods.

3.4.2. ​Methodological insights​​

3.4.2.1. ​Deep learning methods dominate​​

scButterfly​​ achieved near-perfect ​​HOM (0.923)​​ in RNA→ATAC, showcasing exceptional consistency in modeling cross-modal cell states.scButterfly​​ and ​​MultiVI​​ surpassed traditional methods (e.g., BABEL: NMI=0.235) by wide margins in HOM and NMI.

3.4.2.2. ​Limitations of traditional approaches​​

BABEL​​ underperformed across all metrics (RNA→ATAC: NMI=0.235, HOM=0.32; ATAC→RNA: NMI=0.123, HOM=0.147), underscoring challenges in aligning heterogeneous modalities.JAMIE​​’s HOM=0.413 in RNA→ATAC revealed weaknesses in local structure preservation.

3.4.3. ​Key conclusions​​

3.4.3.1. ​​Directional sensitivity of cross-modal mapping​​

Performance disparities between RNA→ATAC and ATAC→RNA tasks (e.g., scButterfly’s NMI drop from 0.645 to 0.683; Monae’s HOM decline from 0.934 to 0.895) emphasize the need for direction-specific optimizations.

3.4.3.2. ​Deep learning methods excel in complex scenarios​​

Generative models (e.g., scButterfly) and joint embedding frameworks (e.g., MultiVI) outperformed traditional methods in capturing ​​global dependencies (HOM)​​ and ​​distribution consistency (NMI)​​.

3.4.3.3. ​Practical trade-offs​​

For ​​high-precision cell state alignment (HOM > 0.9)​​, prioritize ​​scButterfly​​.For ​​balanced efficiency and robustness​​, ​​MultiVI​​ remains a viable alternative.

3.4.4. Summary​​

These experiments validate the effectiveness of deep learning in unpaired multimodal integration. ​​scButterfly​​ leads in RNA→ATAC imputation, while ​​MultiVI​​ excels in ATAC→RNA stability. Future work must resolve directional sensitivities to achieve generalized cross-modal solutions.

4. Conclusion

This study establishes a benchmark for multimodal biological data integration and imputation, revealing significant methodological disparities. CoVEL excels in cross-modal prediction accuracy (MAP) and shared neighborhood preservation (SAS), while Monae variants balance biological interpretability (graph connectivity, biological conservation) with integration efficacy. Deep learning methods (e.g., scButterfly, Monae-E) outperform traditional approaches in unpaired imputation tasks. Task-specific recommendations include GLUE for batch effect removal, scButterfly for RNA→ATAC mapping, and Monae-E for holistic integration. Key challenges remain in reconciling local precision with global stability and addressing directional biases. Future work should prioritize unified frameworks, scalable architectures, and direction-agnostic models to advance robust, generalizable solutions in multimodal biological analysis.