1. Introduction

Super-resolution (SR) seeks to generate high-resolution (HR) images from low-resolution (LR) observations[1]. A challenge of SR lies in its ill-posedness, meaning that a single LR image can correspond to more than one valid HR reconstructions. SR, extensively applied across various domains, finds significant use in fields like medical imaging [2] and remote sensing imaging [3].

Amid the swift advancements in deep learning, this technology has been extensively integrated into a wide range of artificial intelligence tasks, including image classification and object detection, leading to remarkable breakthroughs. Simultaneously, numerous deep learning approaches have been employed to address super-resolution challenges. Both early methodologies utilizing Convolutional Neural Networks (CNNs) and subsequent techniques grounded in Generative Adversarial Networks (GANs) have demonstrated exceptional performance. More recently, diffusion-based super-resolution has shown great potential in enhancing the resolution of LR images. Recently, the problem of artifacts in diffusion-based super-resolution was confronted by Zheng et al. [4] through the introduction of innovative techniques known as Reality-Guided Refinement (RGR) and Self-Adaptive Guidance (SAG). Wang et al. [5] introduced a one-step bi-directional distillation approach, which is specifically designed to elucidate the deterministic mapping between input noise and the resultant high-resolution image, along with the reverse process, by utilizing a teacher diffusion model in conjunction with their developed deterministic sampling method.

This article presents an extensive review of the latest advancements in deep learning-based super-resolution techniques. The SR methods based on deep learning are categorized according to the learning approaches and network types used. These categories include SR methods based on Convolutional Neural Networks (CNNs), Residual Networks (ResNet), Generative Adversarial Networks (GANs), and diffusion-based SR networks, each of which is introduced and discussed in detail.

2. Introduction to Deep Learning SR Method

2.1. CNN-based Super-Resolution Networks

2.1.1. Super-Resolution Convolutional Neural Network (SRCNN). The Super-Resolution Convolutional Neural Network (SRCNN) [6] is different from traditional interpolation and reconstruction algorithms. It employs a three-layer neural network to extract feature information from images, thereby enabling the learning of the mapping relationship between LR images and HR images in an end-to-end manner. Ultimately, the system, in a significant development, reconstructs the image while simultaneously simulating the traditional super-resolution process. As the pioneering deep learning-based image super-resolution model, SRCNN significantly enhanced the performance of image resolution reconstruction at the time, notwithstanding its relatively simple architecture.

2.1.2. Fast Super-Resolution Convolutional Neural Network (FSRCNN). Fast Super-Resolution Convolutional Neural Network (FSRCNN) [7] is an improved version of SRCNN, designed to enhance both the computational speed and performance of image super-resolution. Compared to SRCNN, FSRCNN incorporates deconvolution layers at the final part of the network, utilizes a deeper network architecture, and employs smaller convolutional kernels. This design enables the model to effectively acquire the mapping from the original LR images to HR images, significantly increasing the restoration speed while maintaining the quality of the recovery.

2.1.3. Efficient Sub-Pixel Convolutional Neural Network (ESPCN). Unlike SRCNN, Efficient Sub-Pixel Convolutional Neural Network (ESPCN) [8] does not compute directly at the HR level. At the heart of this network lies the incorporation of a sub-pixel convolutional layer structure, which is placed at the last layer of the network. This method regulates the quantity of feature maps through convolutional operations and integrates multiple feature maps by rearranging the pixels, achieving different upscaling effects, thereby improving both reconstruction efficiency and quality.

2.1.4. Residual Channel Attention Network (RCAN). The Residual Channel Attention Network (RCAN) [9] achieves significant advancements by introducing the Residual in Residual (RIR) structure, which constructs deep networks to enhance visual recognition performance. This approach allows the model to capture high-frequency information and combine it with low-frequency information to improve resolution. Additionally, RCAN also marks the inaugural introduction of the attention mechanism in LR image tasks, adjusting the allocation of resources to more information-rich areas through channel attention and improving the resolution improvement effect.

2.1.5. Holistic Attention Network (HAN). The Holistic Attention Network (HAN) enhances the generation of clearer and more detailed high-resolution images by introducing a holistic attention mechanism that captures both global and local information at multiple scales [10]. The Spatial Attention Mechanism focuses on the spatial relationships within the image, enhancing spatial features. The Channel Attention Mechanism, similar to RCAN, underscores the significance of features across various channels.

2.2. ResNet-based SR Networks

2.2.1. Very Deep Convolutional Networks for Super-Resolution (VDSR). Very Deep Convolutional Networks for Super-Resolution (VDSR) utilizes an exceptionally deep convolutional network [11]. The increase in network depth leads to a significant enhancement in image clarity. VDSR is characterized by the use of 20 weight layers, which effectively capitalize on contextual information across extensive image areas. However, in extremely deep networks, the speed of convergence emerges as a critical issue during the training process.

2.2.2. Enhanced Deep Residual Networks (EDSR). Unlike typical generative network components, Enhanced Deep Residual Networks (EDSR) removes unnecessary batch normalization operations [12]. This allows EDSR to stack more network layers without more resources. Consequently, each layer of the network can extract more features to enlarge the model size and improve reconstruction quality. Additionally, this method integrates residual scaling by incorporating a constant scaling layer following the convolutional layers in each residual block, which serves to stabilize the training of large models.

2.2.3. Multi-Scale Residual Network (MSRN). The Multi-Scale Residual Network (MSRN) advances the residual block by incorporating convolutional kernels of diverse scales. This enhancement facilitates the adaptive detection of image features across varying scales [13]. The Multi-Scale Residual Blocks employs multi-scale feature fusion to capture image feature information at various scales, thereby significantly reducing computational complexity. The Hierarchical Feature Fusion Structure filters and fuses all feature outputs from above blocks, minimizing unnecessary redundant information while adaptively highlighting useful information.

2.2.4. Deep Back-Projection Networks (DBPN). Deep Back-Projection Networks (DBPN) progressively enhances image resolution through a multi-level forward and backward projection mechanism, thereby better recovering image details [14]. The core of DBPN involves using forward projection to map low-resolution features into high-resolution space, followed by backward projection to remap them back into low-resolution space. This iterative process helps reduce error accumulation and improves reconstruction accuracy. The multiple bidirectional projection operations enable DBPN to excel in detail recovery and edge reconstruction, particularly demonstrating outstanding performance in high-magnification tasks.

2.2.5. Cascading Residual Network (CARN). Cascading Residual Network (CARN) applies a progressive learning approach to cascading residual networks [15]. In the training method for extreme super-resolution scenarios, the model initially generates relatively low-resolution outputs and then incrementally increases the output resolution by adding additional networks to the model. This progressive upsampling of images helps to reduce abrupt size changes in the model, thereby alleviating training instability.

2.3. GAN-based SR Networks

2.3.1. Super-Resolution Generative Adversarial Network (SRGAN). Super-Resolution Generative Adversarial Network (SRGAN) introduces GANs into the image SR field, proposing a network model for image SR reconstruction that utilizes generative adversarial networks [16]. Through adversarial training between the generator and discriminator, the model generates high-resolution images from low-resolution inputs. The generator aims to create visually realistic high-resolution images, while the discriminator evaluates the authenticity of these images.

2.3.2. Enhanced Super-Resolution Generative Adversarial Network (ESRGAN). Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) [17] builds upon the foundation of SRGAN with several improvements. The paper introduces a network unit called the Residual-in-Residual Dense Block (RRDB), which eliminates the Batch Normalization layers. Additionally, the authors enhance the discriminator's objective in the GAN framework and utilize an activation-based feature composition for the perceptual loss function. These modifications contribute significantly to improving the visual quality of the output images.

2.3.3. Blind Super-Resolution Generative Adversarial Network (BSRGAN). Blind Super-Resolution Generative Adversarial Network (BSRGAN) employs random permutations of various degradation factors, including blurring, downsampling, and noise [18]. This approach enables the trained BSRGAN to exhibit significant performance improvements when handling real-world degraded images.

2.4. Diffusion-based SR Networks

2.4.1. Super-Resolution Diffusion Probabilistic Model (SRDiff). While learning-based SISR methods significantly outperform traditional techniques, approaches focusing on PSNR, GAN-driven methods, and flow-based methods each face issues such as over-smoothing or excessive model size. To address these challenges, the Super-Resolution Diffusion Probabilistic Model (SRDiff) is proposed [19]. This model is the first single-image super-resolution model based on diffusion processes, gradually transforming Gaussian noise into super-resolution images conditioned on low-resolution inputs through a Markov chain, while incorporating residual prediction to accelerate convergence.

2.4.2. Frequency Domain-guided Multiscale Diffusion Model (FDDiff). The Frequency Domain-guided Multiscale Diffusion Model (FDDiff) [20] decomposes the high-frequency information complementary process into more granular steps. FDDiff directs the backward diffusion process to supplement the missing high-frequency details over time steps. Furthermore, a multiscale frequency refinement network is designed, which is capable of processing information across multiple scales, allowing for more precise identification and prediction of high-frequency details in signals or images.

2.4.3. Implicit Diffusion Models for Continuous Super-Resolution (IDM). Implicit Diffusion Models for Continuous Super-Resolution (IDM) [21] integrates implicit neural representations with denoising diffusion models into a unified end-to-end framework. During the decoding process, implicit neural representations are utilized to learn continuous resolution representations. A scalable adjustment mechanism is designed, which includes a LR adjustment network and a scaling factor.

3. Discussion and Analysis

3.1. Evaluation Metrics

Evaluation metrics typically employ established computational formulas to assess the errors between SR images and HR images, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) . As the ratio of the maximum power of a signal to the power of noise in the signal, PSNR measures the quality of the compressed reconstructed image of the input. SSIM is a method which can quantify the similarity between two images. Higher values of PSNR and SSIM indicate better quality of the reconstructed images, reflecting superior performance of the SR methods.

3.2. Datasets

The datasets used include Set5[22], Set14[23], B100[24], Urban100[25], and Manga109[26]. These datasets contain varying numbers of high-resolution images, ranging from 5 to 2,650, covering a wide range of content such as environments, flora and fauna, racing cars, natural landscapes, and people. They effectively and comprehensively test the performance of algorithms. Additionally, other related algorithms have been designed to improve the practical application value of SR in more realistic life scenarios by utilizing image datasets that better reflect real-world conditions for experimentation.

3.3. Analysis of Experimental Results

This study conducts a comparative analysis of the experimental results from traditional methods and deep learning approaches based on different network types, as reported in relevant literature. The public datasets used for the experiments include Set5[22], Set14[23], and BSD100[24]. The performance of several mainstream deep learning-based super-resolution methods is compared, including Bicubic interpolation [27], SRCNN[6], FSRCNN[7], VDSR[11], EDSR[12], RCAN[9], MSRN[13], HAN[10], and DBPN[14], with Bicubic serving as the benchmark method for comparison. Since the Set5, Set14, and BSD100 datasets only provide HR images, simulated LR images need to be generated from the high-resolution images beforehand. In this study, Bicubic interpolation is used to downsample each high-resolution image by factors of 2, 3, and 4, resulting in the corresponding low-resolution images. Subsequently, experiments on super-resolution using various deep learning-based models are conducted on the LR images for 2x, 3x, and 4x upscaling, with the results presented in Tables 1-3.

Table 1. Quantitative comparison with SR methods at ×2 scale

Table 2. Quantitative comparison with SR methods at ×3 scale

Table 3. Quantitative comparison with SR methods at ×4 scale

Deep learning-based methods consistently outperform the traditional Bicubic method across all metrics at ×2, ×3 and ×4 scales, and this advantage becomes more pronounced as the scaling factor increases. While SRCNN, as one of the earliest deep learning SR methods, demonstrated excellent performance at its inception, its efficacy has gradually been surpassed by newer methods in recent years.

Across all scales and datasets, the HAN method performs well in terms of PSNR, and in most cases, its SSIM values also rank among the top. This indicates that HAN possesses strong capabilities in image super-resolution tasks. Simpler models like SRCNN and FSRCNN, while advantageous in computational efficiency, perform worse than more complex deep learning models when handling high-resolution images and intricate scenes. At higher scaling factors, the MDSR method excels in SSIM, particularly on the Set5 dataset. The introduction of attention mechanisms in both the RCAN and HAN methods generally results in outstanding performance across all datasets and scaling factors, suggesting that attention mechanisms can effectively enhance model performance.

As the scale increases, the difficulty of image reconstruction rises. Taking a scaling factor of 4 as an example, the PSNR and SSIM values of nearly all algorithms show a significant decline. This indicates that the scaling factor has a substantial impact on the effectiveness of super-resolution reconstruction. The PSNR and SSIM values for SRCNN and FSRCNN decrease relatively gradually with increasing scaling factors, whereas more complex models like VDSR, EDSR, and RCAN are better at preserving image quality, particularly performing exceptionally well at factors of 3 and 4.

4. Conclusion

As a fundamental task in computer vision, image SR reconstruction plays a crucial role in practical applications such as criminal investigation, medical diagnosis, and smart cities. This paper reviewed recent deep learning-based image SR methods and compared the performance of mainstream approaches based on two metrics: PSNR and SSIM. Although SR technology has achieved substantial research progress, several challenges remain in the following areas:

(1) Model Lightweight Design. As the depth and complexity of models increase, so does their computational load. Designing lightweight network architectures that maintain high-quality high-resolution images while reducing computational demands is an important direction in current research.

(2) Generalization Ability. Most current SR models perform exceptionally well in specific scenarios but struggle to adapt to diverse and complex environments. Therefore, further research is needed on how to maintain strong generalization capabilities across different contexts.