Aircraft Target Detection in Low Signal-to-Noise Ratio Visible Remote Sensing Images

Abstract

With the increasing demand for the wide-area refined detection of aircraft targets, remote sensing cameras have adopted an ultra-large area-array detector as a new imaging mode to obtain broad width remote sensing images (RSIs) with higher resolution. However, this imaging technology introduces new special image degradation characteristics, especially the weak target energy and the low signal-to-noise ratio (SNR) of the image, which seriously affect the target detection capability. To address the aforementioned issues, we propose an aircraft detection method for RSIs with low SNR, termed L-SNR-YOLO. In particular, the backbone is built blending a swin-transformer and convolutional neural network (CNN), which obtains multiscale global and local RSI information to enhance the algorithm’s robustness. Moreover, we design an effective feature enhancement (EFE) block integrating the concept of nonlocal means filtering to make the aircraft features significant. In addition, we utilize a novel loss function to optimize the detection accuracy. The experimental results demonstrate that our L-SNR-YOLO achieves better detection performance in RSIs than several existing advanced methods.

1. Introduction

Accurate aircraft detection in remote sensing images (RSIs) remains challenging, which has considerable research significance for airport surveillance, aviation security, national defense construction, and other applications. With the rapid development of remote sensing techniques, in order to achieve the fine detection of small-sized targets represented by aircrafts, some brand new imaging technologies have emerged to improve the resolution of RSIs and broaden the detection range of the targets. An imaging system with an ultra-large area-array detector is one of the most effective ways to obtain broad width high-resolution RSIs. Its main implementation means is to obtain the focal plane by mechanical splicing of multiple small-scale detectors. Consequently, the ultra-large area-array detector imaging technology can simultaneously achieve high resolution and wide-area imaging, which lays the foundation for improving the target detection accuracy.

At present, many research institutions in the remote sensing field have carried out research on the imaging technology of an ultra-large area-array detector [1,2,3,4,5,6,7,8]. The study results indicate that this type of imaging mode introduces new imaging degradation characteristics in application. Limited by the primary lens aperture dimension of the space load optical system and the satellite resource constraints, the ultra-large area-array detectors apply small pixels. This structure reduces the energy received by the individual pixel. Relatively, the noise component in the remote sensing imaging chain enhances the interference with the imaging quality integrally, resulting in a significant decrease in the signal-to-noise ratio (SNR) of the imaging results compared with the ideal RSIs, as shown in Figure 1.

Meanwhile, the aircrafts are presented as small targets and sparse samples relative to the background in broad width high-resolution RSIs. In addition, under the condition of low SNR, the aircraft saliency is reduced, and the target details become fuzzier with the interference of high frequency noise, which greatly increases the difficulty of target detection. Nevertheless, the existing target detection algorithms in remote sensing are mainly aimed at general targets with sufficient sample size, which lack a treatment scheme for the small targets represented by aircrafts under the brand new imaging system with an ultra-large area-array detector.

In response to these problems, we present a low SNR visible RSI aircraft detection network named L-SNR-YOLO. The contributions of this work are summarized as follows:

• We propose a backbone combining the advantages of swin-transformer and traditional CNN, which extracts multiscale aircraft features and fuses the global–local information interactively.
• We present an effective feature enhancement (EFE) module, which suppresses the complex background interference in low SNR visible RSIs.
• We design a new loss function to promote the detection convergence performance.
• We carry out comparative experiments with a range of existing popular detectors on UCAS_AOD, DOTA, and Google Earth and verify the feasibility and algorithm boundary of the proposed architecture.

The remainder of this paper is organized as follows. Section 2 elaborates the development of the aircraft detection technology based on visible RSIs and analyzes the defects of the existing methods. Section 3 describes the details of the proposed network framework. Section 4 analyzes the detection performance of the proposed method compared with the existing typical algorithms. Section 5 summarizes the research and provides the conclusion.

2. Related Work

Currently, there exists a lack of a pertinent target detection scheme for the imaging results of a remote sensing camera with an ultra-large area-array detector. Template matching is one category of the earliest methods applied to target detection in RSLs. The templates are firstly generated based on the effective features that distinguish the targets from the background. Then, the similarities between the template and possible positions within the entire RSI are calculated to locate the targets. Xu et al. [9] proposed a shape-matching method for low-altitude aircraft detection applying the artificial bee colony algorithm with edge potential function. However, these approaches are sensitive to changes in the target shape and density. Therefore, they have great limitations in the target detection affected by the multiscale and multidirectional characteristics of aircrafts in RSIs.

Machine learning-based methods are significant approaches for target detection in RSIs. The core idea of machine learning is to extract the features of the possible target region and then input them into the corresponding classifier to determine whether the region contains targets [10,11]. Gao et al. [12] utilized the circle-frequency filter to extract the target candidate locations to detect aircrafts in high-resolution RSIs. Liu et al. [13] implemented airplane detection by extracting the rotation invariant feature combined with sparse coding and radial gradient transform. Sun et al. [14] extracted the image space sparse coded bag-of-words (BOW) information as the input of a support vector machine (SVM) classifier to achieve aircraft detection. Cheng et al. [15] trained the target mixed variability component model based on multiscale histogram of oriented gradients (HOG) features and then performed target detection by calculating the mixture model response. Bi et al. [16] proposed the local context deformable part model (DPM) by establishing the local context HOG feature pyramids to achieve target detection on the aircraft candidate regions.

With the increasingly wide application of convolutional neural network (CNN) architecture in the field of image processing [17,18,19,20], the approaches based on deep learning have greatly improved the target detection accuracy compared with the traditional methods. The CNNs train the image data by convolution, nonlinearity mapping, and pooling operations and obtain the feature extractors applied in the target detection task.

Two-stage and one-stage architectures are two common underlying frameworks of CNNs used in RSI target detection. Two-stage algorithms represented by regions with CNN features (R-CNN) [21], spatial pyramid pooling networks (SPP-Net) [22], Fast R-CNN [23], and Faster R-CNN [24] firstly generate candidate regions with potential targets on RSIs through CNNs and then perform classification and boundary regression on candidate regions. In these algorithms, the accuracy and quantity of the candidate regions directly influence the target detection performance [25]. Sequentially, one-stage algorithms omit the step of generating candidate regions and regard the classification and positioning in target detection as a regression problem [26], which can reduce the computational cost to enhance the target detection speed, including you only look once (YOLO) [27,28,29], single shot multi-box detector (SSD) [30], and anchor-free algorithms [31,32]. In particular, YOLO series algorithms have attracted much attention in RSI target detection by possessing high detection accuracy and detection speed simultaneously.

Meanwhile, considering the characteristics including the complex background and small target proportion in the RSIs, an increasing amount of research has introduced an attention mechanism [33,34,35,36] into CNNs to obtain target detection results with the interpretability of visual perception. Shi et al. [37] integrated contextual information into feature maps based on a spatial attention mechanism, which effectively improved the aircraft detection accuracy. Chen et al. [38] constructed a cascade attention network by combining self-attention and spatial attention modules to enhance the representation of the target characteristics. In particular, the transformer-based architectures apply the self-attention mechanisms and multilayer perceptron replacing the convolution operator to capture long-range dependencies, which have been gradually utilized for computer vision tasks based on RSIs to improve the algorithm performance [39].

Although there has been much research on RSI target detection, the influence of the brand new imaging system with an ultra-large area-array detector acting on the image quality restricts the application of the existing methods. Concretely, the low SNR of whole RSIs and fuzzy target details weaken the target saliency, which consequently decreases the target detection accuracy. For the above situation, the existing algorithms are difficult to directly apply, and it is urgent to put forward a targeted optimization and improvement scheme.

3. Proposed Method

We integrate the swin-transformer (ST) and YOLOv5 [40] as the backbone to obtain target multiscale information effectively and take global and local features into account. Concurrently, we enhance the target feature to inhibit the complex background interference in RSIs with a low SNR. In addition, we design a new loss function adding the angle between the prediction and the ground truth box as a constraint to promote the detection convergence performance.

3.1. Subsection Overall Architecture

The proposed entire network architecture is shown in Figure 2. Integrally, aiming at the aircraft target’s directional uncertainty, size inconsistency, and interference from a complex airport background, a backbone network is proposed to take the global/local feature into account in the low SNR visible RSIs. Convolution operations obtain local interactions among pixels in the RSIs, which makes the CNNs possess a generalizable and efficient framework but a lack of spatial connectivity. In addition, the vision transformers (ViTs) can extract the global interactions by their self-attention mechanisms and multilayer perceptron. We integrate the swin-transformer (ST) [41] and YOLOv5 in the proposed network, which optimizes the traditional CNN by combining the advantages of the ViTs. The former can further capture multiscale information, and the latter exhibits good performance on target detection speed and accuracy synthetically.

Specifically, we replace the last cross stage partial layer in the backbone of YOLOv5 with swin-transformer encoder blocks, which consist of multi-head self-attention (MSA) layers with “windows” and “shifted windows” followed by the multilayer perceptron (MLP) respectively, linked by the normalization layer (LN) and the residual connection processing.

Meanwhile, facing the target saliency reduction in RSIs with low SNR, we design an effective feature enhancement module to suppress the interference of high frequency noise on target details, which is elaborated in Section 3.2.

3.2. Effective Feature Enhancement Module

In order to enhance the learning ability, YOLOv5 adopts the simplified cross stage partial network (C3) for feature extraction. This structure splits the gradient flow and merges the output after different network paths to avoid the gradient disappearance caused by the deepening of network layers [42]. However, there exists a large amount of high-frequency noise in the deep features of an RSI with a low SNR caused by the small pixel structure in the ultra-large area-array detector, which affects the ability of the network to extract the target high-frequency features. Therefore, we design the WFE block based on C3 in combination with the idea of nonlocal means filtering.

Nonlocal means filtering can make full use of the feature map information and keep detailed features to the maximum extent while suppressing the noise. The core idea is to calculate the weighted sum of all pixels in the search window to a certain pixel as its output value, which can be expressed as

$$y_i={\displaystyle \sum }_{\forall j}w\left(x_i,x_j\right)\ast I\left(x_i\right)$$

(1)

where $x_i$ and $x_j$ are pixels in feature map, $I\left(x_i\right)$ is the pixel value of $x_i$, $w\left(x_i,x_j\right)$ represents the weight, and $y_i$ is the response value at the position of $i$ in the feature map after filtering. In addition, based on the principle of cosine similarity, we choose the Embedded Gaussian function to calculate the similarity of $x_i$ and $x_j$ in a feature map. It can be calculated as

$$w\left(x_i,x_j\right)=\frac{\exp \left({\left(\mathsf{\Psi }{x}_i\right)}^T\left(\mathsf{{\rm Z}}{x}_j\right)\right)}{{{\displaystyle \sum }}_{\forall j}\exp \left({\left(\mathsf{\Psi }{x}_i\right)}^T\left(\mathsf{{\rm Z}}{x}_j\right)\right)}=softmax\left({\left(\mathsf{\Psi }{x}_i\right)}^T\left(\mathsf{{\rm Z}}{x}_j\right)\right),$$

(2)

where$\mathsf{\Psi }$ and $\mathsf{{\rm Z}}$ are the linear mapping matrixes.

Based on the above formulas, the effective feature enhancement module is illustrated in Figure 3:

3.3. Loss Function

We improve the loss function in order to improve the target detection accuracy in RSIs with low SNR.

By analogy with YOLO networks, we jointly describe the loss function of the confidence loss, category loss, and location loss. First, we add the angle between the prediction box and the ground truth box as a constraint to design a new angular location loss. The distance loss ${\sigma }_{dis}$ is defined as

$${\sigma }_{dis}=\left[1-e^{-\left(2-\eta \right){\rho }_x}\right]+\left[1-e^{-\left(2-\eta \right){\rho }_y}\right]$$

(3)

$${\rho }_x={\left[\frac{x_{ct}-x_{ct}^{gt}}{\max \left(x_r-x_r^{gt}\right)-\min \left(x_l-x_l^{gt}\right)}\right]}^2$$

(4)

$${\rho }_y={\left[\frac{y_{ct}-y_{ct}^{gt}}{\max \left(y_r-y_r^{gt}\right)-\min \left(y_l-y_l^{gt}\right)}\right]}^2,$$

(5)

where $\left(x_r, y_r\right)$, $\left(x_l, y_l\right),$ and $\left(x_{ct}, y_{ct}\right)$ respectively represent the coordinates of the left, right, and center points of the prediction box. Analogously, $\left(x_r^{gt}, y_r^{gt}\right)$, $\left(x_l^{gt}, y_l^{gt}\right)$, and $\left(x_{ct}^{gt}, y_{ct}^{gt}\right)$ are the coordinates of the left, right, and center points of the ground truth box.

The angle parameter can be computed as

$$\eta =\sin \left(2\arcsin \left(\frac{d_{\mathrm{h}}}{\sqrt{d_{\mathrm{h}}^2+d_w^2}}\right)\right),$$

(6)

where $d_w$ and $d_{\mathrm{h}}$ are the distances in the $x,y$ directions between the center points of the prediction box and the ground truth box.

Meanwhile, we define a new angular intersection over union (AGIoU)

$$AGIoU=\frac{S{\displaystyle \cap }{S}^{gt}-\left(w_{dis}\times h_{dis}-S{\displaystyle \cap }{S}^{gt}\right)}{w_{dis}\times h_{dis}}$$

(7)

$$w_{dis}=\max \left(x_r-x_r^{gt}\right)-\min \left(x_l-x_l^{gt}\right)$$

(8)

$$h_{dis}=\max \left(y_r-y_r^{gt}\right)-\min \left(y_l-y_l^{gt}\right),$$

(9)

where $S, S^{gt}$ are the area of prediction box and ground truth box, respectively.

Then, the angular location loss ($L_{agloc}$) can be written as

$$L_{agloc}=1-AGIoU+{\sigma }_{dis}.$$

(10)

Parallelly, the confidence loss $L_{obj}$ and category loss $L_{aircraft}$ can be described separately by binary cross-entropy loss and cross-entropy loss, as

$$\begin{array}{cc}{L}_{obj}& ={\displaystyle \sum }_{i=0}^{M^2}{\displaystyle \sum }_{j=0}^B{l}_{ij}^{obj\_T}\left[C_{i}ln\left(C_i^{gt}\right)+\left(1-C_i\right)\ln \left(1-C_i^{gt}\right)\right]\hfill \\ & -{\displaystyle \sum }_{i=0}^{M^2}{\displaystyle \sum }_{j=0}^B{l}_{ij}^{obj\_F}\left[C_{i}ln\left(C_i^{gt}\right)+\left(1-C_i\right)\ln \left(1-C_i^{gt}\right)\right]\hfill \end{array}$$

(11)

$$L_{aircraft}={\displaystyle \sum }_{i=0}^{M^2}{l}_{ij}^{obj\_T}(-p_i^{gt}\_aircraft\log p_i\_aircraft),$$

(12)

where each input RSI is split into $M^2$ grids, each of which is distributed with $B$ bounding boxes, $l_{ij}^{obj}$ is used to determine whether the jth bounding box in the ith grid is responsible for predicting target, and $C_i^{gt}, C_i$ are the annotation value and the predicted value of bounding box in the ith grid containing target confidence, respectively. $p_i^{gt}, p_i$ are the annotation value and the predicted value of the prediction result belonging to an aircraft.

Furthermore, we set weighting factors to allocate the significance of each loss function reasonably. The loss function of L-SNR-YOLO can be obtained as:

$$L_{L-SNR-YOLO}={\lambda }_{obj}{L}_{obj}+{\lambda }_{aircraft}{L}_{aircraft}+{\lambda }_{agloc}{L}_{agloc},$$

(13)

where ${\lambda }_{obj}, {\lambda }_{aircraft}, {\lambda }_{agloc}$ are the weighting factors. After testing and contrasting, the most satisfactory detection result can be obtained when $({\lambda }_{obj},{\lambda }_{aircraft},{\lambda }_{agloc}$) are set to (1.0,0.5,0.05).

4. Discussion

4.1. Dataset

We carried out experiments based on RSIs from the University of Chinese Academy of Sciences-high resolution aerial object detection (UCAS_AOD) [43] dataset of object detection in aerial images (DOTA) [44] and Google Earth to analyze the application performance of the proposed algorithm. The dataset covers multi-sized aircraft targets in various parking directions along with complex backgrounds such as airstrips and airfield construction. Considering the new imaging characteristics caused by the ultra-large area-array detector integrated with small pixels in practical applications, we obtained the degraded RSIs based on the optical remote sensing chain [45], including atmospheric transmission, optical imaging system, platform vibration, detector, and other links, as shown in Figure 4. It is noteworthy that the RSI quality was seriously reduced by the comprehensive influence of the detector dark current noise, the shot noise, the readout noise, and the quantization noise generated by the ultra-large area-array detector and its electronic system.

Finally, we acquired 2439 RSIs containing aircrafts with a resolution of 640 × 640 pixels and randomly selected 80% of them as the training set (1951 images) and the rest as the testing set (488 images). Figure 5 shows some image slices of the employed dataset.

4.2. Evaluation Metrics

We selected the average precision (AP), which is defined as the area between the Precision–Recall curve and the coordinate axes, to evaluate the detection algorithm performance.

$$AP={{\displaystyle \int }}_0^{1}precision\left(recall\right)d\left(recall\right)$$

(14)

$$precision=\frac{TP}{TP+FP}$$

(15)

$$recall=\frac{TP}{TP+FN},$$

(16)

where $TP$ represents the correctly predicted positive samples, and $FP, FN$ represent the incorrectly predicted positive samples and negative samples by the detection algorithm, respectively.

Furthermore, the interference of the complex backgrounds with detection was enhanced in the low SNR RSIs, which increased the target misjudgment probability of the algorithm. Therefore, we introduced the false alarm rate (FAR) to further evaluate the algorithm detection effect.

$$FAR=\frac{\mathrm{number}\mathrm{of}\mathrm{detected}\mathrm{false}\mathrm{alarms}}{\mathrm{number}\mathrm{of}\mathrm{detected}\mathrm{candidates}}$$

(17)

Additionally, we calculated the SNR of the target region (TR-SNR) to measure the quality of the RSI imaging by the ultra-large area-array detector so as to determine the algorithm’s applicable range.

$$TR\_SNR=10lg\frac{{{\displaystyle \sum }}_{i=1}^M{{\displaystyle \sum }}_{j=1}^{N}g{\left(i,j\right)}^2}{{{\displaystyle \sum }}_{i=1}^M{{\displaystyle \sum }}_{j=1}^N{\left|g\left(i,j\right)-f\left(i,j\right)\right|}^2},$$

(18)

where $M, N$ are the length and width of in RSI target region, respectively, and $g\left(i,j\right)$, $f\left(i,j\right)$represent the gray values at $\left(i,j\right)$ of the original image and the processed image, respectively.

4.3. Results and Analysis

The experiments were carried out on a workstation with an NVIDIA GeForce RTX3080 Ti GPU (12 GB VRAM). Before network training, we applied the K-means clustering method to generate the anchor box dimensions based on prior geometric feature of the aircrafts in the dataset. In addition, we used the mosaic data augmentation to combine nine training samples into one to enrich the dataset and reduce the GPU video memory concurrently.

In the training parameter setting, we set the epoch to 220 and the batch size to eight. Combined with the aircraft target characteristics in our dataset, we used the stochastic gradient descent (SGD) to minimize the loss with momentum of 0.93 and weight decay of 0.0005 and used cosine annealing to adjust the learning rate with the periodic learning rate of 0.2. We conducted comparative experiments on the original RSIs and the degraded RSIs with low SNR to verify the effectiveness of the proposed method. Specifically, we tested the degraded RSIs with the model trained with the degraded RSIs and tested the original RSIs by the model trained with the original RSIs. Figure 6 shows several examples of the detection results under the condition of IoU ≥ 0.5 to illustrate this intuitively. The confidence labels in the detection results were removed for a concise and distinct display. Overall, the target detection effect on the degraded RSIs with low SNR dropped sharply compared with that of the original RSIs, indicating that the degradation characteristics caused by the ultra-large area-array detector seriously interfered with the target detection. In the original RSIs (the first, third, and fifth rows in Figure 6), under the condition that the targets could be detected, the prediction boxes of our proposed method had higher accuracy compared with the YOLOv5 baseline. In addition, in the degraded RSIs with low SNR (the second, fourth, and sixth rows in Figure 6), our method avoided false alarms effectively in the first scene. In contrast, the YOLOv5 network misjudged the airdrome facility as a target. For the second scene, our approach was capable of detecting targets with fuzzy details, while there were missing alarms in the detection results of the YOLOv5. Moreover, the prediction boxes of our algorithm were more accurate in the third scene. Consequently, our method obtained more satisfactory detection results.

Meanwhile, we analyzed the unsatisfactory detection results of the proposed algorithm in Figure 7. Our approach detected the target that the YOLOv5 baseline failed to detect, improving the target detection ability of the algorithm in the degraded RSIs. However, due to the excessive small scale of some targets and the texture features similar to noisy background, our method did not detect all targets in this scene.

We performed ablation experiments on the dataset to illustrate the effects of the proposed module and new loss function. Furthermore, we compared the proposed algorithm with several advanced methods containing Faster R-CNN [24], YOLOv3 [28], YOLOv4 [29], and RetinaNet [46].

Table 1. Ablation experiments of our proposed method. The bold highlights the best performance.

Method	Loss Function	AP	FAR
YOLOv5 [40]	The original function	78.6%	29.4%
YOLOv5+ST	The original function	80.3%	29.0%
YOLOv5+NLC3	The original function	81.0%	26.7%
YOLOv5+ST+WFE	The original function	81.8%	26.4%
YOLOv5+ST+WFE	The proposed function	82.2%	25.5%

and

Table 2. Results of the comparative experiment.

Methods	AP (Low SNR)	FAR (Low SNR)	AP (Original)	FAR (Original)
Faster R-CNN [24]	54.9%	49.8%	65.8%	25.4%
YOLOv3 [28]	62.5%	44.1%	70.2%	30.7%
YOLOv4 [29]	76.4%	32.1%	81.6%	29.2%
RetinaNet [46]	63.3%	42.8%	71.0%	33.2%
Proposed method	82.2%	25.5%	86.0%	25.2%

shows the detection results of the multi-scenario RSIs on the dataset. The proposed method outperformed the other compared approaches in detection accuracy. In particular, compared to the YOLOv5, the proposed algorithm achieved better detection results with the AP value increasing by 3.2% and the FAR value reducing by 3.0%. When constrained by the new loss function, the proposed algorithm performance was further improved, the AP value was increased by 3.6%, and the FAR value was reduced by 3.9%.

In addition, in order to analyze the proposed framework applicability, we plotted the AP and FAR variation curves of our detection method in the RSIs with different TR-SNR in Figure 8. The TR-SNR of the RSIs’ imaging by ultra-large area-array detector varied between 7 dB and 15 dB according to the existing prior cognition. When the TR-SNR was between 10 and 15 dB, our algorithm effectively suppressed the influence of the SNR reduction on the detection effect. While under the condition of the TR-SNR lower than 5 dB, the AP of the algorithm was approximately equal to the FAR, indicating that the detection results lost the reference significance. Accordingly, the TR-SNR of the RSIs should be considered in the imaging system design, by ensuring its value better than 10dB for subsequent image interpretation.

5. Conclusions

In this paper, facing the space application of the new imaging mode with ultra-large area-array detectors integrated with small pixels, the L-SNR-YOLO network is proposed for aircraft detection in degraded RSIs considering the new imaging characteristics. As opposed to the existing aircraft detection methods, the hybrid backbone is structured based on swin-transformer and CNN, which extracts the multiscale global and local RSI features to improve the algorithm robustness effectively. Moreover, an effective feature enhancement module is developed by the idea of nonlocal means filtering, which suppresses the background noise at the feature map level to enhance the saliency of the aircraft features. Additionally, a new loss function is presented to optimize the detection accuracy in RSIs with low SNR. Finally, we conducted experiments on the ordinary RSIs and the RSIs with low SNR to verify the impact of the new imaging degradation characteristics caused by the ultra-large area-array detector on the aircraft target accuracy. The experiments’ results indicate that the detection precision of the L-SNR-YOLO network reaches 82.2% in RSIs with low SNR, which has better accuracy effectiveness compared with other benchmarking frameworks. In addition, the proposed architecture can be applied to other targets under this new imaging mode. Finally, it can be extended to target detections of degraded images under harsh weather conditions or limited observation resource conditions. In further research, we plan to establish the aircraft detection method under the condition of target partial occlusion and other complex background distractions.