Dual-Domain Cooperative Recovery of Atmospheric Turbulence Degradation Images

Abstract

Atmospheric turbulence is a key factor contributing to data distortion in mid-to-long-range target observation tasks. Neural networks have become a powerful tool for dealing with such problems due to their strong ability to fit nonlinearities in the spatial domain. However, the degradation in data is not confined solely to the spatial domain but is also present in the frequency domain. In recent years, the academic community has come to recognize the significance of frequency domain information within neural networks. There remains a gap in research on how to combine dual-domain information to reconstruct high-quality images in the field of blind turbulence image restoration. Drawing upon the close association between spatial and frequency domain degradation information, we introduce a novel neural network architecture, termed Dual-Domain Removal Turbulence Network (DDRTNet), designed to improve the quality of reconstructed images. DDRTNet incorporates multiscale spatial and frequency domain attention mechanisms, combined with a dual-domain collaborative learning strategy, effectively integrating global and local information to achieve efficient restoration of atmospheric turbulence-degraded images. Experimental findings demonstrate significant advantages in performance for DDRTNet compared to existing methods, validating its effectiveness in the task of blind turbulence image restoration.

1. Introduction

Imaging systems within the atmospheric medium are susceptible to geometric distortions and optical blurring, primarily due to atmospheric turbulence [1]. Further analysis indicates that the severity of turbulence disturbances correlates with the optical path integral, with turbulence-induced image degradation becoming more pronounced at greater imaging distances [2]. Specifically, atmospheric turbulence causes random fluctuations in the air refractive index, introducing stochastic disturbances in amplitude and phase during light propagation. This complexity and unpredictability in the imaging process pose a challenging inverse problem for computer vision and adaptive optics to mitigate atmospheric turbulence effects [3,4]. The significance of this issue lies in the fact that if degraded images are utilized for advanced visual tasks such as target detection, classification, remote sensing, surveillance, autonomous driving, and high-resolution Earth observation, the deterioration in image quality will directly compromise the performance and accuracy of these tasks, potentially leading to erroneous outcomes [5]. Such effects are intolerable in demanding application scenarios. Therefore, restoring turbulence-degraded images is a critically important low-level visual task to meet the quality requirements of advanced visual tasks.

In recent years, with the rapid advancement of deep learning technology, end-to-end deep learning-based image restoration techniques have increasingly become mainstream. Consequently, the application of deep learning methods to mitigate atmospheric turbulence effects in degraded images has garnered significant attention [6]. Unlike traditional adaptive optics methods, deep learning offers substantial potential in addressing nonlinear problems by leveraging data-driven learning of prior knowledge. Through extensive training on large datasets, deep neural networks can establish robust nonlinear mappings between inputs and outputs. With sufficient data and strong generalization capabilities, these networks can tackle challenges that traditional methods cannot, such as image denoising [7,8], deblurring [9,10], and super-resolution [11,12]. Unlike non-blind turbulence image restoration tasks, which often require cumbersome extraction of prior knowledge [13,14], neural networks can autonomously learn the necessary prior information through data-driven approaches. As a result, deep learning methods based on various neural network architectures have emerged as the leading approach for blind turbulence image restoration tasks in recent years. For instance, Kangfu Mei et al. [15] proposed LTT-GAN, based on generative adversarial networks, to mitigate turbulence interference in long-distance facial recognition. Zihao Cai et al. [16] addressed turbulence interference in solar telescopes by enhancing CycleGAN. Santiago Lopez-Tapia et al. [17] introduced variational deep networks for the first time to correct atmospheric turbulence interference. P. Hill et al. [18] introduced the concept of “accelerated Deep Image Prior” in the context of blind turbulence image restoration. Xingguang Zhang et al. [19] proposed the Turbulence Mitigation Transformer, while Xijun Wang et al. [20] designed the “Real-World Atmospheric Turbulence Mitigation Model” by combining supervised and unsupervised methods. Abu Bucker Siddik et al. [21] utilized a deep neural network architecture based on AlexNet to predict corrected Zernike coefficients. Xingguang Zhang et al. [22] introduced “DATUM”, which efficiently performs long-distance spatiotemporal aggregation using a recursive approach. Lizhen Duan et al. [23] tackled interference in the pre-denoising step of turbulence-degraded images using a novel deblurring kernel. Viktor Sineglazov et al. [24] developed a “fully convolutional encoder-decoder network” to address turbulence blur in drone-captured images. Yiming Guo et al. [25] designed “a novel blind restoration network” to improve the generalization capabilities of existing models. Huimin Ma et al. [26] employed “GoogLeNet” to develop a method for correcting turbulence aberrations between two image frames. Ripon Kumar Saha et al. [27] pioneered a segmented restoration channel technique to restore dynamic scenes in turbulent environments. Nantheera Anantrasirichai et al. [28] proposed a new framework that supports dynamic scene restoration by utilizing short time spans. Weiyun Jiang et al. [29] introduced a universal implicit neural representation (NeRT) for unsupervised turbulence mitigation. Shengqi Xu et al. [30] won the “CVPR 2023 UG²⁺ Track 2.2” by combining image matching and deblurring methods. Shuyuan Zhang et al. [31] proposed a dual-patch pixel (DPP) prior to effective blind deblurring of turbulence images. Inspired by the connection between turbulence phase diagram characteristics, Xiangxi Li et al. [32] introduced a novel deep neural network named DeturNet. Ajay Jaiswal et al. [33] integrated a physics-based simulator directly into the training process to help the network separate turbulence randomness from degraded and underlying images. Zhiyuan Mao et al. [34] proposed a transformer model inspired by physics to improve atmospheric turbulence imaging and introduced a unified approach for mitigating atmospheric turbulence in both static and dynamic sequences [35]. In summary, although existing neural network models have achieved impressive results in restoration tasks, many primarily rely on redundant spatial domain information for representation learning, often overlooking the potential value of frequency domain information in blind turbulence image restoration tasks.

In image restoration theory, high-frequency information in the frequency domain represents textures and details, while low-frequency information is primarily associated with smooth areas of an image [36]. According to signal processing theory, filtering different frequency domain components can more effectively restore degraded images. Additionally, advancements in fast Fourier transform (FFT) technology have significantly enhanced the speed of processing frequency domain signals. As a result, the role of frequency domain information in neural networks has garnered increasing attention. For example, to address spatial domain degradation in deep learning-based exposure correction algorithms, Jie Huang et al. [37] proposed the Fourier Transform-Based Deep Exposure Correction Network (FECNet). In end-to-end image-deblurring networks, Xintian Mao et al. [38] introduced the Residual Fast Fourier Transform-Convolution Block (Res FFT-Conv Block). Chongyi Li et al. [39] enhanced low-light images by correcting phase and amplitude through neural networks, while Shi Guo et al. [40] proposed the Spatial-Frequency Attention Network (SFANet) to improve convolutional networks’ ability to capture long-range dependencies. Xuanhua He et al. [41] achieved dual-domain complementary learning by using spatial information for local feature learning and frequency domain information for global feature learning. Lei Lu et al. [42] introduced the Denoising Frequency Attention Network (DFANet) based on frequency differences, and Kangzhen Yang et al. [43] separated and enhanced high-frequency and low-frequency information in image enhancement tasks. Xin Yuan et al. [44] proposed SFUNet, which employs frequency-aware convolution and attention modules to jointly model complementary information from both spatial and frequency domains in wavelet data. Tian Zhou et al. [45] demonstrated that applying the Fourier transform in transformers can better capture the global characteristics of time series, while Badri Narayana Patro et al. [46] combined frequency domain information with multi-head attention layers to create a more efficient transformer architecture. These studies collectively underscore the critical importance of frequency domain information in neural network models. However, a significant research gap remains in how to effectively combine dual-domain information to reconstruct high-quality images in the context of blind turbulence image restoration.

In the intersection of deep learning and computational optics, this paper introduces a novel neural network architecture, the Dual-Domain Removal Turbulence Network (DDRTNet). This model is inspired by the intricate relationship between spatial and frequency domain degradation in atmospheric turbulence imaging. DDRTNet is built upon cutting-edge neural network modules, with the primary goal of significantly enhancing the accuracy of image restoration and the quality of reconstructed images. At the heart of DDRTNet are its innovative multiscale spatial and frequency domain attention mechanisms, coupled with a dual-domain collaborative learning strategy. These features not only improve the model’s ability to capture and represent multiscale features but also effectively integrate degradation information from both the spatial and frequency domains, leading to precise restoration of turbulence-degraded images. To thoroughly assess DDRTNet’s performance, extensive simulations and real-world experiments were conducted. The results demonstrate that DDRTNet outperforms existing methods in image restoration tasks, highlighting its efficiency and practical value in blind turbulence image restoration. This research not only introduces new approaches for restoring turbulence-degraded images but also serves as a valuable reference for the application of deep learning in the broader field of image processing.

2. Materials and Methods

2.1. Atmospheric Turbulence Imaging Model

In this section, the properties and simplified models of atmospheric turbulence in imaging are discussed in detail. Figure 1 illustrates the process of how real image information is distorted by atmospheric turbulence and captured by a camera as a degraded image. In optical systems, the degradation function caused by turbulence can be considered as a spatially invariant function, thus the image degradation process can be modeled as a degradation system consisting of a degradation function and additive noise [47].

$$g\left(x,y\right)=h\left(x,y\right)\otimes o\left(x,y\right)+n\left(x,y\right)$$

(1)

$$\left\{\begin{array}{c}r=\sqrt{x^2+y^2}\\ \theta =arctan{\displaystyle \frac{y}{x}}\end{array}\right.$$

(2)

where $g\left(x,y\right)$ is a degraded image, $h\left(x,y\right)$ is a degenerate function, $o\left(x,y\right)$ is a real image, $n\left(x,y\right)$ is additive noise, and ⊗ is the convolution operation.

The commonly accepted degradation function for atmospheric turbulence is derived from the Kolmogorov spectrum [32], as in Equation (3):

$$h\left(r,\theta \right)={\left|F\left\{p\left(r,\theta \right)exp\left[i\phi \left(r,\theta \right)\right]\right\}\right|}^2$$

(3)

where $p\left(r,\theta \right)$ is an optical pupil function of the camera, $\phi \left(r,\theta \right)$ is the wavefront phase distribution, and $\mathit{F}$ represents the Fourier transform and the pupil function [5] as in Equation (4):

$$p\left(r,\theta \right)=\left\{\begin{array}{cc}0\hfill & other\\ {\displaystyle \frac{2}{\pi }}\left[arccos\left({\displaystyle \frac{\Omega }{{\Omega }_0}}\right)-{\displaystyle \frac{\Omega }{{\Omega }_0}}\sqrt{1-{\left({\displaystyle \frac{\Omega }{{\Omega }_0}}\right)}^2}\right]& \Omega \le {\Omega }_0\end{array}\right.$$

(4)

where $\Omega =2r/\lambda$ is the spatial angular frequency, ${\Omega }_0=D_0/\lambda$ is the cut-off spatial angular frequency of the optical system, $D_0$ is the diameter of the camera’s lens, and r is the polar coordinates radius.

The wavefront phase distribution function after degradation by atmospheric turbulence can be decomposed into orthogonal Zernike polynomials [48] as in Equation (5):

$$\phi \left(r,\theta \right)=\sum _{i=1}^{\infty }{a}_i{z}_i\left(r,\theta \right)$$

(5)

$$a_i=\int d\overrightarrow{r}p\left(r,\theta \right)\phi \left(r,\theta \right)z_i\left(r,\theta \right)$$

(6)

where $z_i\left(r,\theta \right)$ is the i-th term characterizing the Zernike polynomials and $a_i$ is the Zernike coefficient on the i-th term since $\phi \left(r,\theta \right)$ is consistent with a Gaussian distribution at each point in space and $a_i$ is also Gaussian. The Zernike coefficients can be calculated by Equation (6).

From the Kolmogorov turbulence theory, the relationship between the Zernike polynomial coefficients and the covariance matrix elements [48] is determined as follows:

$$〈a_i{a}_j〉=\left\{\begin{array}{c}\begin{array}{cc}{c}_{ij}{\left({\displaystyle \frac{D_0}{r_0}}\right)}^{5/3}& i-j=even\end{array}\\ \begin{array}{cccccc}0& & & & & i-j=odd\end{array}\end{array}\right.$$

(7)

where $c_{ij}$ is the covariance matrix element that can be calculated from the refractive fluctuation exponent characterized by the Kolmogorov power spectrum, $D_0$ is the diameter of the camera’s lens, and $r_0$ is the atmospheric coherence length. Because each term of the coefficients of the Zernike polynomials is related to $D_0/r_0$, it is often used to characterize the turbulence intensity.

2.2. The Structure of Network

2.2.1. Res FFT-GeLU Block

To integrate spatial and frequency domain information, we propose the Res FFT-GeLU Block, as illustrated in Figure 2. This structure consists of two parallel branches, a spatial domain branch and a frequency domain branch. The spatial domain branch focuses on extracting spatial domain features of the image to capture spatial degradation information. In the spatial domain, two 3 × 3 convolutions are equivalent to one 5 × 5 convolution. This is to expand the receptive field of the network while reducing the number of parameters so that the network can learn richer features. The 1 × 1 convolution kernel used in the frequency domain branch can be equivalent to a multiplication operation. Before inputting the Res FFT GeLU module, different texture information in the image has been extracted separately. Because different texture information is composed of different frequency domain information, we assume that the 1 × 1 convolution parameter can be approximated as the reciprocal of the degradation function value of different frequency information, thus alleviating turbulence interference in the frequency domain information of different channel dimensions. The residual connection in each branch serves to accelerate training, alleviate the vanishing gradient problem during training, and reduce the loss of feature information. The frequency domain branch acts like an adaptive filter, capable of filtering the frequency domain signals of different channels to adaptively process high- and low-frequency features. Through carefully designed filters, we can selectively enhance or suppress specific frequency components, effectively filtering and handling frequency domain degradation information. Finally, the outputs of the spatial domain branch and the frequency domain branch are merged using an additive fusion strategy, resulting in a feature representation that integrates spatial and frequency domain information. This dual-domain fusion approach leverages the complementary aspects of both domains, thereby enhancing the performance of the image restoration model.

Based on the atmospheric turbulence imaging model of Section 2.1, we will analyze the adaptive filtering capability of the frequency domain branch in detail. In order to process the frequency domain information, Equation (1) was Fourier transformed to the following:

$$G\left(u,v\right)=H\left(u,v\right)O\left(u,v\right)+N\left(u,v\right)$$

(8)

where $G\left(u,v\right)$ is a degraded image, $H\left(u,v\right)$ is a degenerate function, $O\left(u,v\right)$ is a real image, and $N\left(u,v\right)$ is additive noise. Compared to Equation (1), the neural network fits Equation (8) more easily.

The mathematical model of the frequency domain branch is shown in Equation (9) and the input feature of the branch are shown in Equation (10):

$$X_2^{\mathrm{C}\times H\times W}\left(u,v\right)={W_2}^{C\times 1\times 1}\otimes GeLU\left({W_1}^{C\times 1\times 1}\otimes X_1^{\mathrm{C}\times H\times W}\left(u,v\right)\right)+X_1^{\mathrm{C}\times H\times W}\left(u,v\right)$$

(9)

$${X_1}^{C\times H\times W}\left(u,v\right)=H^{\mathrm{C}\times H\times W}\left(u,v\right)O^{\mathrm{C}\times H\times W}\left(u,v\right)+N\left(u,v\right)$$

(10)

where $O^{\mathrm{C}\times H\times W}\left(u,v\right)$ is the frequency information of the real image texture features, ${X_1}^{C\times H\times W}\left(u,v\right)$ is the Spatial domain characteristics ${X_{in}}^{C\times H\times W}\left(x,y\right)$ after fast Fourier transform, and ${X_2}^{C\times H\times W}\left(u,v\right)$ is the output of the frequency domain branch.

Since the 1 × 1 convolution operation is equivalent to the multiplication operation, combining Equations (9) and (10) yields Equation (11), as follows:

$${W_1}^{C\times 1\times 1}\otimes X_1^{\mathrm{C}\times H\times W}=W_1^{\mathrm{C}\times 1\times 1}{H}^{\mathrm{C}\times H\times W}\left(u,v\right)O^{\mathrm{C}\times H\times W}\left(u,v\right)+W_1^{\mathrm{C}\times 1\times 1}N\left(u,v\right)$$

(11)

When u, v are in the tiny range, the $H^{\mathrm{C}\times H\times W}\left(u,v\right)$ can be seen as a constant. The weights ${W_1}^{C\times 1\times 1}$ learned adaptively through neural networks can approximate the specific frequencies $1/H^{\mathrm{C}\times H\times W}\left(u,v\right)$, which makes $W_1^{\mathrm{C}\times 1\times 1}{H}^{\mathrm{C}\times H\times W}\left(u,v\right)\approx 1$,$N^{\prime }\left(u,v\right)=W_1^{\mathrm{C}\times 1\times 1}N\left(u,v\right)$, and in this case, Equation (9) can be simplified to Equation (12):

$$X_2^{\mathrm{C}\times H\times W}(u,v)=W_2^{\mathrm{C}\times 1\times 1}GeLU\left(O^{\mathrm{C}\times H\times W}(u,v)+N^{\prime }(u,v)\right)+X_1^{\mathrm{C}\times H\times W}(u,v)$$

(12)

$$GeLU\left(x\right)=0.5x\left(1+tanh\left(\sqrt{2/\pi }\left(x+0.044715x^3\right)\right)\right)$$

(13)

From Equation (12), we can see that the feature $O^{\mathrm{C}\times H\times W}(u,v)+N^{\prime }(u,v)$ mainly contains the random noise interference of the degraded system, so we can carry out the filtering process to achieve noise reduction. Consequently, we can employ filtering techniques to achieve noise reduction. As depicted in Figure 3a,b, the GeLU [49] function expression, presented in Equation (13), and its derivative exhibit a striking resemblance to the Butterworth filter [36], and while the ReLU function possesses characteristics analogous to an ideal filter for superior noise suppression [50], in the context of neural networks, GeLU is deemed more suitable as the filter function for the Res FFT-GeLU Block. This is due to several reasons: First, ideal filters are physically unattainable, whereas Butterworth filters offer a more physically interpretable alternative [36]. Second, as we postulate that some noise is inherently coupled with the image frequencies, preserving relevant frequency information is conducive to image restoration. Last, the ReLU function is prone to the “dying ReLU” issue during network training [51], which GeLU effectively circumvents. After filtering and then integrating the frequency information by convolution ${W_2}^{C\times 1\times 1}$, added with the residual connection to improve the training speed and prevent the loss of key information [52], we finally obtain the frequency domain information ${X_2}^{C\times H\times W}\left(u,v\right)$ which is close to the real image.

2.2.2. Overall Architecture

To mitigate turbulence effects in images, we propose a deep learning model called the Dual-Domain Removal Turbulence Network (DDRTNet), illustrated in Figure 4. Inspired by state-of-the-art dual-domain neural network models, our model adopts the U-Net [53] encoder–decoder framework, which includes three scales in both the encoding and decoding stages. Research by Hamidreza Fazlali et al. [54] has shown that downsampling operations can alleviate image blur, and clearer small-scale outputs can help restore clearer large-scale images. Based on these findings, we construct corresponding inputs and outputs at three scales within the network. We hope to further enhance the network’s ability to restore different scale degradation areas by calculating the loss of downsampling output. The branches of different scales are all shaped like a variant of U-Net, which consists mainly of an encoder and a decoder for encoding feature information. After encoding features of different scales and dimensions, corresponding decoding processes are performed to obtain restored images of different scales, making the model capable of multi-scale restoration. This design can make fuller use of different scale feature information and enhance the model’s ability to represent different scale features. To enable the model to efficiently learn features from downsampled inputs, we use the SCM block, as shown in Figure 5a, which can rapidly increase the number of channels of the downsampled inputs to match the required channels for the corresponding scale of the model. As shown in Figure 5b, the ResGroup comprises a ResBlock, FFT-GeLU Block, and the Dual-Domain Strip Attention Mechanism (DSAM) module [55]. Initially, the ResBlock, consisting of two layers of 3 × 3 convolution and GeLU non-linear activation functions, is used to efficiently extract features with doubled or halved channels. Once features are extracted, they are quickly integrated with spatial and frequency domain information through the residual block composed of the FFT-GeLU Block and DSAM. The DSAM module, which efficiently refines spectra and integrates contextual information, is placed after the Res FFT-GeLU Block, with its structure depicted in Figure 5c. To reduce computational complexity while enhancing the network’s multiscale feature learning capability, we also embed the Omni-Kernel Module (OKM) [56] between the encoding and decoding stages, with the OKM structure shown in Figure 5d. The 3 × 3 convolution before the three scales output is to fuse the features extracted by ResGroup into a 3-dimensional restored image output, while the other 3 × 3 convolution is to increase or decrease the number of channels. These operations are crucial for adjusting the network structure, controlling model complexity, and improving model performance. Concatenation can help us connect multiple different features or multiple different network module outputs together by splicing the feature information extracted from the downsampled image and the original image in the channel dimension, thereby forming a richer feature representation. Then, through 3 × 3 convolution, we can fuse the features between different channels during the process of increasing or decreasing the channel, achieving interaction between feature information across channels. Finally, to minimize the loss of image information during the downsampling and upsampling processes, we design residual connections at each scale of the encoding and decoding stages, as well as in the output results.

Through the design of DDRTNet, the potential of frequency domain information in reconstructing high-quality images is fully exploited. Our model not only effectively reduces the number of ResBlock modules typically used in traditional spatial domain models, but also enhances the network’s nonlinear mapping capability. Aimed at reconstructing high-quality images from those degraded by atmospheric turbulence, our model design makes full use of the characteristics of the atmospheric turbulence imaging model.

2.2.3. The Implementation of DDRTNet

In this section, we will detail the construction of the training dataset and test samples. First, 1200 images were randomly selected from the ImageNet dataset [57] to serve as the original dataset. Degraded images were generated using the turbulence simulation algorithm proposed by Zhiyuan Mao et al. [58]. Relevant parameters of the simulation were $L=3000\mathrm{m},D_0=0.1\mathrm{m},r_0=0.1\mathrm{m}$. The ImageNet dataset [57] has been widely used in research on computer vision algorithms since it was introduced in 2010 for the ImageNet Large Scale Visual Recognition Challenge. The turbulence simulation algorithm proposed by Zhiyuan Mao et al. [58] has also gained widespread recognition and was adopted as the official simulation algorithm for the CVPR UG²⁺ Challenge Track 2.2-Coded Target Restoration through Atmospheric Turbulence. A subset of background images and their corresponding simulated degraded versions are shown in Figure 6. Visually, the results of the simulation closely resemble the actual effects of atmospheric turbulence, accurately replicating the distortions typically seen in real-world scenarios.

In our dataset, we used 1000 images for the training dataset, applying data augmentation techniques to expand the training set’s size. An additional 100 images were used as the validation dataset, and the remaining 100 images were designated as the test dataset. The network’s three input image sizes were the original size of 256 × 256, half-size downsampled to 128 × 128, and quarter-size downsampled to 64 × 64. To balance the representation learning of both the spatial and frequency domains, we employed the L1 loss function for both the spatial and frequency domain signals in the model’s restoration results. The loss functions for the spatial domain, frequency domain, and the overall loss are given by Equations (14), (15), and (16), respectively.

$$L_c={\displaystyle \frac{1}{E}}{∥I^{*}-Y∥}_1$$

(14)

$$L_f={\displaystyle \frac{1}{E}}{∥F\left(I^{*}\right)-F\left(Y\right)∥}_1$$

(15)

$$L_0=\left(L_c+L_{c,1/2}+L_{c,1/4}\right)+\lambda \left(L_f+L_{f,1/2}+L_{f,1/4}\right)$$

(16)

where $I^{*}$ and Y represent the predicted image and background reality, E indicates the number of output elements. $L_{c,1/2}$, $L_{c,1/4}$, $L_{f,1/2}$, $L_{f,1/4}$ denote the spatial and frequency domain loss corresponding to downsampling 1/2 and 1/4 times the predicted image and $\lambda$ represents the weighting factor, which is set to 0.01 here [50].

3. Results and Discussion

3.1. The Test of Image Restoration in Simulation

In this section, we will validate the performance and robustness of DDRTNet using a simulated dataset. The initial learning rate was set to $8\times {10}^{-4}$, and the training process took approximately 3 h. We employed a cosine annealing learning rate schedule, with the minimum learning rate set to $1\times {10}^{-6}$. The batch size was set to 20, using the Adam optimizer with a weight decay rate of $1\times {10}^{-8}$. Our training platform consisted of a Supermicro X11SCA-F server (Supermicro, Shenzhen, China), equipped with an Intel Xeon Silver 4210R CPU (Intel, Shenzhen, China) and an NVIDIA GeForce GTX 3090 GPU (Nvidia, Shenzhen, China). The model was trained and tested using PyTorch 1.8.1 and Python 3.7.3. The loss curves of DDRTNet are shown in Figure 7.

In Figure 8, we present the restoration results of DDRTNet on the test set. From the figure, it is evident that our model effectively suppresses the turbulence effects in the images, significantly enhancing their clarity and detail representation. To validate the superiority of DDRTNet, we compared it with several state-of-the-art deep learning-based image restoration methods, including frequency domain-based Frequency Selection Network (FSNet) [59], spatial domain-based MIRNet [60], and Nonlinear Activation Free Network (NAFNet) [61]. To ensure a fair comparison, all methods were trained using the same training protocol. The restoration results of these methods are also displayed in Figure 8. The comparison shows that DDRTNet not only preserves the texture of the images better but also more accurately corrects texture distortions and local blurring. From subjective evaluation, DDRTNet’s restoration results are the closest to the true background, demonstrating the best performance. Although other methods exhibit some restoration capabilities, they still leave certain areas blurry and some textures distorted. In contrast, DDRTNet shows higher accuracy and robustness in handling turbulence effects.

To objectively assess the quality of reconstructed images, we employed three widely used metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Normalized Root Mean Squared Error (NRMSE). These metrics measure the differences and similarities between the restored images and the original images from different perspectives. By calculating these metrics, we can further validate the superiority of DDRTNet.

$$PSNR=10\times {log}_{10}\left({\displaystyle \frac{max{\left(I^{*}\left(x,y\right)\right)}^2}{\frac{1}{HW}{\displaystyle \sum _{x=0}^{W-1}}{\displaystyle \sum _{y=0}^{H-1}}{\left[I^{*}\left(x,y\right)-Y\left(x,y\right)\right]}^2}}\right)$$

(17)

$$SSIM\left(I^{*},Y\right)={\displaystyle \frac{\left(2{\mu }_{I^{*}}{\mu }_Y+{\left(0.01R\right)}^2\right)\left(2{\sigma }_{I^{*}Y}+{\left(0.03R\right)}^2\right)}{\left({\mu }_{I^{*}}^2+{\mu }_Y^2+0.01R\right)\left({\sigma }_{I^{*}}^2+{\sigma }_Y^2+0.03R\right)}}$$

(18)

$$NRMSE={\displaystyle \frac{{\displaystyle \sum _{x=0}^{W-1}}{\displaystyle \sum _{y=0}^{H-1}}\left|I^{*}\left(x,y\right)-Y\left(x,y\right)\right|}{{\displaystyle \sum _{x=0}^{W-1}}{\displaystyle \sum _{y=0}^{H-1}}{\left(Y\left(x,y\right)\right)}^2}}$$

(19)

where H, W represent the height and width of the image; ${\mu }_{I^{*}}$, ${\sigma }_{I^{*}}$ represent the mean and variance of the recovered image $I^{*}$, and ${\mu }_Y$, ${\sigma }_Y$ represent the mean and variance of Y and R represents the range of pixel values. PSNR is an important objective metric for evaluating the fidelity of reconstructed images. Higher PSNR values indicate greater fidelity of the reconstructed images. SSIM, on the other hand, objectively measures the structural similarity between the reconstructed image and the original image. When the reconstructed image closely resembles the original, SSIM approaches 1. NRMSE quantifies the difference between the reconstructed image and the original image; a lower NRMSE indicates higher similarity, approaching 0 as the reconstructed image becomes more similar to the original. Additionally, the variances of PSNR (PSNR_td) and SSIM (SSIM_td) are employed to assess the stability of the model.

The results of the objective evaluation metrics are presented in

Table 1. Objective assessment of the above restoration methods.

	Degraded	MIRNet	NAFNet	FSNet	DDRTNet
PSNR ↑	24.54	25.26	24.98	25.23	26.28
SSIM ↑	0.7149	0.7654	0.7610	0.7723	0.8061
NRMSE ↓	0.1297	0.1240	0.1281	0.1207	0.1118
PSNR_td ↓	12.0471	13.2617	13.1166	13.5189	14.7575
SSIM_td ↓	0.01479	0.01141	0.01136	0.01031	0.00853

The best results are bolded. ↑ denotes that the bigger scores represent better quality of the images, ↓ indicates the opposite result.

. As shown, DDRTNet outperforms all other methods across all metrics. Specifically, DDRTNet achieves a 1.02 dB improvement in PSNR over the second-best method, indicating that our method reconstructs images with lower error and higher quality. In terms of SSIM, DDRTNet shows an increase of 0.0338, suggesting that our method better preserves the structural information of the images, making the restoration closer to the original. Additionally, DDRTNet reduces NRMSE by 0.0089, further demonstrating the superiority of our method in image restoration. Moreover, the standard deviations of PSNR (PSNR_td) and SSIM (SSIM_td) were calculated to assess the stability of different methods on the test data. As seen in Table 1, DDRTNet has lower SSIM_td values compared to other methods, indicating higher stability when handling various test data. In summary, DDRTNet exhibits exceptional performance in restoration tasks. Whether from the perspective of subjective perception or objective evaluation metrics, DDRTNet significantly outperforms other methods.

To comprehensively evaluate the performance of DDRTNet, we further compared DDRTNet with three other state-of-the-art image restoration methods in terms of algorithm complexity and runtime. As shown in

Table 2. Comparison of the average time, FLOPs, and Params consumed by methods to recover images.

Index/Unit	MIRNet	NAFNet	FSNet	DDRTNet
FLOPs/G	19.88	140.75	59.71	34.63
Params/MB	29.2	5.8	7.0	7.0
Time/ms	123.801	120.806	92.851	62.902

The best results are bolded.

, we recorded the total floating-point operations (FLOPs) and the number of parameters (Params) for each method, along with their average runtime on the test set. The total floating-point operations (FLOPs) indicate the time complexity of the algorithm, while the number of parameters (Params) represents the spatial complexity of the algorithm. These two metrics are crucial for evaluating the performance of the model, especially in real-time image restoration tasks. Smaller parameter counts and shorter runtimes imply better portability and real-time capability of the model.

From Table 2, although our method does not achieve the optimum in the comparison of computational complexity and space complexity, they are both sub-optimal results, achieving a balance between computational complexity and space complexity. Specifically, the parameter count of DDRTNet is relatively low, which helps in minimizing the storage requirements and enhancing its portability across different devices. Additionally, DDRTNet’s lower FLOPs indicate that it requires fewer computational resources to perform the restoration tasks, allowing for faster image data processing and meeting real-time requirements. In terms of average runtime, DDRTNet also excels. Compared to the other three methods, DDRTNet has a shorter average runtime, further demonstrating its superiority in real-time image restoration tasks. This advantage makes DDRTNet particularly suitable for applications requiring quick response times, such as autonomous driving and video surveillance.

In summary, DDRTNet not only excels in image restoration quality but also shows significant advantages in algorithm complexity and runtime. This makes DDRTNet an ideal choice for real-time image restoration tasks, offering broad application prospects and potential in various fields.

3.2. Ablation

To further investigate the impact of different components within DDRTNet on its performance, we conducted a series of detailed ablation experiments. In these experiments, we focused primarily on the activation functions within the Res FFT-GeLU Block and the specific modules within DDRTNet. Firstly, we replaced the GeLU function in the Res FFT-GeLU Block with a ReLU function to observe the effect of different activation functions on network performance. Additionally, we added or removed specific modules within DDRTNet (indicated by ✓/✗) to verify their contributions to the network’s ability to mitigate turbulence effects.

In the experiments, we ensured that each variant adopted the same hyperparameters and training strategies to ensure fairness and comparability of the results. We selected the model parameters that performed best on the validation set for testing and recorded the performance metrics of each variant. The experimental results are shown in

Table 3. The results of ablation experiments.

Method	ReLU	RFGB	DSAM	OKM	PSNR	SSIM
	✓	✓	✓	✓	25.92	0.7981
	✗	✗	✓	✓	25.97	0.7964
DDRTNet	✗	✓	✗	✓	25.87	0.7942
	✗	✓	✓	✗	25.34	0.7648
	✗	✓	✓	✓	26.28	0.8061

The best results are bolded.

, from which we can observe that both replacing the activation function and removing specific modules led to a performance decline in DDRTNet, resulting in a weakened ability to remove turbulence. This result clearly demonstrates the rationality of our selection of various modules in DDRTNet and the importance of these modules working together to enhance network performance.

Specifically, the GeLU function in the Res FFT-GeLU Block exhibited better performance than the ReLU function in the experiments, effectively validating the viewpoint we proposed when designing the Res FFT-GeLU Block. Additionally, each module within DDRTNet played a crucial role in the process of removing turbulence effects, and their combination enabled the network to better understand and handle complex degradation patterns in the images. Through this experiment, we not only gained a deeper understanding of the roles of each module in DDRTNet but also provided valuable insights for further optimizing the network structure in the future.

3.3. The Robustness of DDRTNet on Different Noises

To verify the robustness of DDRTNet against noise interference, we added four sets of Gaussian noise to the test set, with mean values of 0 and variances of 0.09, 0.16, 0.25, and 0.36, respectively. We directly input these noisy images into the trained model and obtained the corresponding restoration results as shown in Figure 9. The relevant objective evaluation metrics are presented in

Table 4. Comparison of the noise test sets.

		Var = 0	Var = 0.09	Var = 0.16	Var = 0.25	Var = 0.36
Noise test	PSNR	24.5861	24.5830	24.5794	24.5758	24.5721
sets	SSIM	0.7131	0.7125	0.7117	0.7109	0.7101
Recovery	PSNR	26.2832	26.1369	25.9011	25.7171	25.5145
results	SSIM	0.8061	0.8018	0.7942	0.7888	0.7817

.

In Figure 9, we can clearly observe the impact of additive noise on the quality of reconstructed images. Specifically, when the noise is weak, its influence on the reconstruction quality is relatively minor, and the image can still maintain a high level of clarity and detailed information. However, as the noise intensity gradually increases, the quality of the reconstructed image begins to decline, manifested by an increase in noise points in the image and a blurring of detailed information. It is worth noting that even with the increasing noise intensity, DDRTNet’s reconstruction results can still preserve a significant amount of detailed information. From a subjective visual perception standpoint, despite the image being subjected to a certain degree of noise interference, the target information can still be accurately identified and perceived. This indicates that DDRTNet exhibits strong robustness when handling noise interference. Additionally, the quantitative results in Table 4 further validate this observation. With the increase in noise variance, the PSNR and SSIM metrics of the reconstructed images do decrease, but the decrease is relatively small. This indicates that DDRTNet can maintain relatively stable performance when faced with noise interference, without causing a significant decline in image quality due to noise.

In conclusion, DDRTNet demonstrates strong robustness and excellent reconstruction capabilities when dealing with degraded images affected by additive noise interference. This characteristic makes DDRTNet highly promising for practical applications, especially in complex and dynamic natural scenes, where it can effectively address image degradation issues and provide high-quality image reconstruction results.

3.4. The Performance of DeturNet on Different Turbulences

To evaluate the ability of DDRTNet to remove turbulence of different intensities, we generated datasets using the same simulation method with varying turbulence strengths ($D_0/r_0=0.2$ and $D_0/r_0=1.3$). Utilizing these new datasets, we applied identical hyperparameters and training strategies to retrain the model. The experimental results are presented in Figure 10 and

Table 5. Comparison of the different turbulence intensity sets.

	${\mathit{D}}_0/{\mathit{r}}_0=0.2$		${\mathit{D}}_0/{\mathit{r}}_0=1.0$		${\mathit{D}}_0/{\mathit{r}}_0=1.3$
	Degraded	Our	Degraded	Our	Degraded	Our
PSNR	27.48	31.70	24.54	26.28	23.74	24.48
SSIM	0.8332	0.9366	0.7149	0.8061	0.6739	0.7405

.

When the turbulence intensity is $D_0/r_0=0.2$, the degradation of the images mainly manifests as atmospheric blur, causing partial loss of details in objects. However, DDRTNet demonstrates the ability to effectively reconstruct these lost image details. For example, in the test results shown in Figure 10, DDRTNet accurately restored the details of the billboard text, the small house behind the hot air balloon, and the background behind the plane. This indicates that DDRTNet exhibits good reconstruction capability when dealing with low-intensity turbulence degradation. As the turbulence intensity increases to $D_0/r_0=1.3$, the degradation of image quality becomes more severe. Key information such as text on billboards, images on hot air balloons, and aircraft outlines have suffered significant degradation. Nonetheless, DDRTNet is still able to reconstruct most of the texture information, although some details are lost. This indicates that DDRTNet still maintains a certain degree of robustness and reconstruction ability when faced with images with severe degradation.

From Table 5, it is evident that DDRTNet effectively enhances the quality of reconstructed images for different turbulence intensities. By comparing the PSNR and SSIM image quality evaluation metrics under different turbulence intensities, we can further validate DDRTNet’s reconstruction ability under varying turbulence strengths. The experimental results demonstrate that DDRTNet significantly improves the PSNR and SSIM values of reconstructed images under both turbulence intensities, thereby proving its excellent reconstruction performance under different turbulence strengths.

3.5. Outdoor Experiment Results and Discussions

In this chapter, the test dataset under real turbulence conditions is sourced from relevant videos on YouTube [62,63]. To verify the performance of DDRTNet under real turbulence conditions, we present the comparative experimental results and no-reference evaluation metrics (entropy) from the test set in Figure 11, Figure 11a shows the degraded image, while (b), (c), and (d) depict the restoration results of the mentioned comparative algorithms, and (e) showcases the restoration result of our proposed DDRTNet. In experiments under real turbulence conditions, due to the lack of real reference images, we utilize some no-reference evaluation metrics to assess the quality of the restored images.

Table 6. Objective assessment of the results of different methods in real scenarios.

	Degraded	NAFNet	MIRNet	FSNet	DDRTNet
Entropy ↑	6.0399	6.1034	6.1057	6.1139	6.1284
AG ↑	1.2478	1.7810	1.7149	1.7984	2.1132
NIQE ↓	9.1128	7.5898	7.7245	7.6107	7.0817
LG ↑	12.0088	45.0503	35.2508	30.7535	50.7452

The best results are bolded. ↑ denotes that the bigger scores represent better quality of the images, ↓ indicates the opposite result.

presents the average values of corresponding evaluation metrics for different restoration algorithms under real scenarios. Entropy effectively evaluates the texture details of images, calculated using Equation (20). Average Gradient (AG) measures the clarity of the restored images, with larger AG values indicating better image clarity and restoration quality, calculated using Equation (21). NIQE is a no-reference image quality evaluation metric based on statistical features of natural scenes [64], primarily used to assess image naturalness. Smaller NIQE values indicate images closer to background reality, calculated using Equation (22). Laplacian Gradient (LG) is a commonly used sharpness evaluation metric in image processing, where larger LG values indicate clearer images.

$$Entropy=-\sum _{i=1}^L{p}_{i}log\left(p_i\right)$$

(20)

where $p_i$ represents the ratio of the number of pixels with grey value i to the total number of pixels. In general, the entropy is an evaluation index of image texture details, and the larger its value is, the more complex the image texture is.

$$AG={\displaystyle \frac{1}{\left(H-1\right)\left(W-1\right)}}\sum _{x=1}^{W-1}\sum _{y=1}^{H-1}\sqrt{{\displaystyle \frac{{\left(I^{*}\left(x+1,y\right)-I^{*}\left(x,y\right)\right)}^2+{\left(I^{*}\left(x,y+1\right)-I^{*}\left(x,y\right)\right)}^2}{2}}}$$

(21)

where H, W represent the height and width of the image and $(x,y)$ represent the pixel coordinates. In general, the average gradient is an evaluation index of image clarity, the larger its value, the clearer the image, and the more detailed the information.

$$NIQE=\sqrt{{\left(V_1-V_2\right)}^T{\left({\displaystyle \frac{{\Sigma }_1+{\Sigma }_2}{2}}\right)}^{-1}\left(V_1-V_2\right)}$$

(22)

where $V_1$, $V_2$ and ${\Sigma }_1$, ${\Sigma }_2$ are the mean vector and covariance matrix of the multivariate Gaussian model for natural images and the multivariate Gaussian model for distorted images, respectively. In general, NIQE is an evaluation index of the naturalness of an image, and the larger its value, the more realistic the image is.

Based on the results in Table 6, our proposed DDRTNet achieved the best scores in various evaluation metrics for blind turbulence image restoration tasks. Despite being trained only on simulated data, DDRTNet demonstrated outstanding generalization ability when facing unknown natural scenes. This achievement significantly proves the effectiveness of DDRTNet in handling turbulence-degraded images. It is noteworthy that compared to traditional networks based solely on spatial domain mapping, DDRTNet exhibited higher reconstruction quality in restoring turbulence-degraded images. This is mainly attributed to the integration of frequency domain information in DDRTNet’s design, allowing it to capture image details and textures more comprehensively

From a visual perception perspective in Figure 11, while there are still some residual artifacts in the images reconstructed by DDRTNet that cannot be completely eliminated, compared to other methods, its reconstructed images are richer and clearer in texture information and image details. This is further corroborated by the quantified assessment of entropy, where DDRTNet’s reconstructed images exhibit higher entropy values, confirming its advantages in image texture restoration and detail preservation. Through both objective evaluation metrics and subjective visual perception judgments, we can confidently assert that DDRTNet not only achieves optimal performance in blind turbulence image restoration tasks but also demonstrates superior generalization capabilities.

4. Conclusions

In this paper, we introduce DDRTNet, an innovative deep learning network designed to leverage both spatial and frequency domain information for high-quality reconstruction of turbulence-degraded images. DDRTNet represents a pioneering approach to utilizing frequency domain information for blind turbulence image restoration tasks and is closely aligned with the atmospheric turbulence imaging model. Through extensive simulations and real-world tests, we demonstrate DDRTNet’s effectiveness and reliability. In simulations with additive noise, DDRTNet exhibits remarkable robustness to unknown noise interference, attributed to its filter-like structural design. In real-world scenarios, DDRTNet, trained exclusively on simulated data, successfully reconstructs degraded images under actual turbulence conditions, showcasing its impressive generalization capabilities. The network employs a dual-domain information fusion strategy, achieving superior restoration performance without increasing network depth by incorporating a modest number of frequency domain parameters at each feature scale. This design feature underscores DDRTNet’s potential for real-time applications. Furthermore, DDRTNet excels in image texture restoration and detail preservation due to its effective use of frequency domain information, surpassing existing methods in the quality of restored images. Overall, our experimental results affirm that DDRTNet offers a novel and effective solution for image restoration in the presence of atmospheric turbulence.

Despite these advancements, there is still room for improvement, particularly concerning residual artifacts in real-world restoration results. Future work will focus on optimizing the network architecture to mitigate these artifacts and refining network parameters by incorporating a larger dataset of real turbulence-degraded images.