Side-Scan Sonar Image Fusion Based on Sum-Modified Laplacian Energy Filtering and Improved Dual-Channel Impulse Neural Network

Abstract

The operation mode of a single strip provides incomplete side-scan sonar image in a specific environment and range, resulting in the overlapping area between adjacent strips often with imperfect detection information or inaccurate target contour. In this paper, a sum-modified Laplacian energy filtering (SMLF) and improved dual-channel pulse coupled neural network (IDPCNN) are proposed for image fusion of side-scan sonar in the domain of nonsubsampled contourlet transform (NSCT). Among them, SMLF energy is applied to extract the fusion coefficients of the low frequency sub-band, which combines the characteristics of energy information, human visual contrast, and guided filtering to eliminate the pseudo contour effect of block flow. In addition, the IDPCNN model, which utilizes the average gradient, soft limit function, and novel sum-modified Laplacian (NSML) to adaptively represent the corresponding excitation parameters, is applied to improve the depth and activity of pulse ignition, so as to quickly and accurately select the image coefficients of the high frequency sub-band. The experimental results show that the proposed method displays fine geomorphic information and clear target contour in the overlapping area of adjacent strips. The objective index values are generally optimal, which reflect the information of image edge, clarity, and overall similarity.

1. Introduction

With the excellent propagation characteristics of sound waves in the water, side-scan sonar provides high-resolution and high-accuracy image information for underwater operations. Full sea area survey is general carried out by means of strip measurement, which requires the adjacent strip to have a certain width of overlapping area [1]. However, owing to the effect of imaging time, hull attitude, and ocean reverberation [2], underwater target and terrain texture in the overlapping area have a certain degree of distortion. The overlapping area of a single strip image has incomplete information detection, and the contours of underwater targets do not match. Therefore, the information fusion of overlapping area between adjacent strips is of great significance, which lays a foundation for the research of underwater target detection and distribution in public areas and classification of sediment types [3,4,5].

In recent years, the effective combination of multi-scale transform domain and other intelligent optimization methods has become a research hotspot of image fusion, especially the domain of nonsubsampled contourlet transform (NSCT) [6]. Anandhi used NSCT and statistical fusion rules for the information fusion of a multi-sensor image, which effectively preserved texture and image edge information [7]. Zhang et al. combined NSCT and texture information for forward-looking sonar image fusion, which weakened the effect of the registration error [8]. The image characteristics of side-scan sonar are similar to forward-looking sonar, and NSCT can be applied to the side-scan sonar image fusion of adjacent strips.

Generally, NSCT serves as a basis for multi-scale image fusion to decompose into low and high frequency sub-band images [9]. After that, it is very important for image fusion to make a reasonable fusion criterion for a low and high frequency image, respectively. For the sub-band coefficients of a low frequency image, the weighted average strategy was a conventional fusion criterion, which resulted in partial energy loss of the source image [10]. As for the fusion criterion of modified weighted saliency, the threshold selection of matching degree in this model is more skillful [11]. Moreover, sparse representation and its combination with dictionary learning were applied to select low frequency sub-band coefficients [12,13]. Their scheme effectively avoided smoothing the texture, edge, and other details of the original image, but the processing of the sliding window might interfere with the correlation between the images, and the latter model involved a lot of training images in the process of iterative learning. Hence, sum-modified Laplacian (SML) energy can be used to characterize the clarity of the sub-band image, and it also reflects the gradient information to a certain extent. Huang and Jing [14] introduced SML energy into the coefficient selection of a low frequency sub-band, which eliminated the interference of fuzzy background region availably. Yong et al. combined visual effects with SML to guide the selection of low frequency [15], which enhanced the judgment depth of multi focus image fusion. However, the overlapping area image of adjacent strips is relatively noisy, and the coefficient selection of a low frequency sub-band based on the above criteria can easily cause false judgment, as well as the pseudo-contour of block flow at the target edge.

For the sub-band coefficients of the high frequency image, the previous fusion criteria include maximum energy [16], regional variance [17], and direction contrast maximum [18], but they are easily susceptible to noise interference, making the edge contour of the target blurred. Pulse coupled neural network (PCNN) has been widely applied to guide the coefficient selection of high frequency [19]. Although the PCNN model has a good fusion effect, it contains a lot of non-linear parameters, and the region discrimination of image darkening is not very sensitive. Moreover, Wang and Ma [20] pioneered a dual-channel PCNN (DPCNN) model to enhance the selection and diffusion ability of feature information, which was faster and simpler than single channel PCNN coupling judgment. In the DPCNN model, some key parameters affected the quality and calculation efficiency of sub-band coefficients to a certain extent, such as link strength, external excitation, and ignition output value. So, the literature [21,22,23,24,25] has improved the model parameters of DPCNN. In essence, link strength was usually set as local standard deviation [21], and Yang et al. calculated the fuzzy membership of each pixel to adapt to the link strength of DPCNN, which obtained the fused image with high contrast [22]. El-taweel and Helmy adopted spatial frequency (SF) as the external excitation, which was capable of effectively overcoming the Gibbs phenomenon at the target boundary [23]. The morphological gradient of the sub-band was used as the external excitation value, which could availably judge the detailed edge information of the high frequency sub-band [24]. Xiang et al. took the average gradient as the link strength, and modified spatial frequency (MSF) as the external excitation value, which enabled the DPCNN to achieve the extraction of rich details [25].

The above DPCNN model achieves an excellent judgment of sub-band coefficients, but the traditional DPCNN model [20] must wait for all sub-band coefficients to be activated before judging, which may result in false fired pulses. Besides, the ignition output value was set to 1 or 0 in this model [21], which made the active judgment unable to reflect the grade difference. Hence, a method of selecting low and high frequency sub-band coefficients in NSCT domain is proposed for side-scan sonar image fusion of adjacent strips. The frame design is shown in Figure 1.

In this work, we have made the following progress: (1) SML energy filtering (SMLF) with multiple parameters and technologies, such as relating sum-modified Laplacian energy with visual contrast value (RLV), guide filter and multi-channel filter (MCF), is used to eliminate block flow in low frequency sub-band images; (2) Improved DPCNN (IDPCNN) model is employed to increase the depth and activity of pulse ignition, which can quickly and accurately select high frequency image coefficients; and (3) multiple fusion metrics are applied to evaluate the performance of single and combined fusion criteria.

The remaining of this paper is organized as follows. Section 2 introduces the proposed image fusion method and process in detail. The experimental results and analysis are illustrated in Section 3. Finally, the conclusions are presented in Section 4.

2. The Proposed Image Fusion Method

2.1. Fusion Pretreatment

The sonar image fusion in the overlapping area is to make full use of the complementary information from the source image to enhance the feature information, such as the size and edge outline of underwater target, the landform, and texture. The sonar image fusion pre-processing includes noise filtering, image registration, multi-scale decomposition, and reconstruction. In general, median filtering is used to de-noise the side-scan sonar image [26]. In addition, the method of speeding up robust features is registered to obtain images of the overlapping area. Moreover, in order to enhance the directional selectivity and obtain better spectral characteristics of image information, NSCT transformation is used to decompose the source image into low frequency and high frequency sub-band components. Then, different low frequency and high frequency fusion criteria are adopted to select and determine the signal source. Finally, a new fusion image is generated according to the combined signal reconstruction.

2.2. SML and SMLF Energy

2.2.1. SML Energy

The low frequency sub-band image retains most of the energy information from the source image, so the selection of its fusion criterion is crucial for the reconstructed fusion image. The work of [14] showed that SML can provide better performance than other focus functions for guiding the selection of low frequency sub-bands, such as variance, energy of image gradient (EOG) [27], energy of Laplacian (EOL), SF, and so on. SML represents the Laplacian energy in the horizontal and vertical directions of the image [28], and its parameter is defined as follows:

$$\begin{array}{l}\mathit{SML}\left(x,y\right)={\displaystyle \sum _{p=-1}^1{\displaystyle \sum _{q=-1}^1\mathit{EOL}{\left(x+p,y+q\right)}^2}}\\ \mathit{EOL}\left(x,y\right)=\left|2c\left(x,y\right)-c\left(x-1,y\right)-c\left(x+1,y\right)\right|+\\ \left|2c\left(x,y\right)-c\left(x,y-1\right)-c\left(x,y+1\right)\right|\end{array}$$

(1)

where EOL (x, y) is the energy of Laplacian. c (x, y) denotes as the low frequency sub-band coefficient.

For a pair of the low frequency sub-band images obtained by NSCT decomposition, the difference between the clear object and the fuzzy object in the image is relatively large, which reflects the feature information in the local neighborhood by calculating its Laplace energy. Therefore, SML energy is utilized to judge the low frequency sub-band coefficient, and the coefficient with larger SML energy is taken as the low frequency coefficient after fusion.

2.2.2. SMLF Energy

Side-scan sonar image is characterized by uneven brightness and severe speckle noise [29], which makes the SML energy unable to reflect the difference in image contrast and leads to false contours in the fused low frequency images. Therefore, SMLF energy is applied to improve the above problems. Its strategy can be described below. SML and the visual adjustment value α are nonlinearly combined into the RLV value. RLV(x,y) is defined as follows:

$$\mathit{RLV}\left(x,y\right)=\{\begin{array}{cc}\frac{\mathit{SML}\left(x,y\right)}{\overline{c}\left(x,y\right)\left(1+\alpha \right)}& \overline{c}\left(x,y\right)\ne 0\\ \mathit{SML}\left(x,y\right)& \overline{c}\left(x,y\right)=0\end{array}$$

(2)

where $\overline{c}$ (x, y) is the coefficient mean of centered on the pixels (x, y). α is the brightness value of the visual adjustment, and α ∈ (0.6, 0.7) [30] and α = 0.65 are used in this paper.

Taking RLV as the guide map, the guided image coefficients Gf_A and Gf_B are obtained by the guided filter (GF) [31]. The decision map can be obtained by judging the size of Gf_A and Gf_B and filtering twice with multi-channel filters. p × q is 3 × 3 window size.

$$\begin{array}{ccc}\mathit{map}=\{\begin{array}{cc}1& G{f}_A\ge G{f}_B\\ 0& \mathit{others}\end{array}& \mathit{and},& \mathit{map}1=\{\begin{array}{cc}1& {\displaystyle \sum _{p\ast q}\mathit{map}\left(i,j\right)\ge 6}\\ 0& \mathit{others}\end{array}\end{array}$$

(3)

According to the value of dp, the low frequency coefficient matrix of fused image c^F is finally generated. c^A and c^B are the low frequency sub-band coefficients, respectively.

$$c^F=map1\ast c^A+(1-map1)\ast c^B$$

(4)

2.3. DPCNN and Its Improved Model

2.3.1. DPCNN Model

Wang and Ma introduced a dual-channel PCNN (DPCNN) model to guide the fusion of high frequency sub-band images, which keeps the coupling characteristics of PCNN model, and simplifies parameter setting [20]. DPCNN model (Figure 2) is a simulation process in which cerebral cortex cells respond to visual pulse signals. Its bidirectional excitation and global characteristics can enhance the important information extraction from source images, which is conducive to the selection of detailed features [32]. Its mechanism is defined as shown below.

The neuron (i, j) receives external excitation value and coupling values of peripheral neurons L_ij in the receiving domain, and internal activity items U_ij^k are nonlinear modulated by link strength β_ij^k.

$$\begin{array}{c}{F}_{ij}^k\left(n\right)=\begin{array}{cc}\left|S_{ij}^k\left(n\right)\right|,& k=\left\{1,2\right\}\end{array}\\ L_{ij}\left(n\right)=e^{-{\alpha }_L}{L}_{ij}\left(n-1\right)+V_L{\displaystyle \sum _{p\ast q}{\omega }_{ij,pq}{Y}_{ij,pq}\left(n-1\right)}\\ U_{ij}^k\left(n\right)=\begin{array}{cc}{F}_{ij}^k\left(n\right)\left(CR+{\beta }_{ij}{}^k{L}_{ij}\left(n\right)\right),& k=\left\{1,2\right\}\end{array}\end{array}$$

(5)

where n is the iteration number. S_ij^k (k = 1, 2) is the external excitation. α_L is the attenuation constant of link input. V_L is the amplification factor of link input. p × q is the neighborhood range. ω_ij,pq is the connection weight. CR is the non-linear coefficient ratio between link input and external excitation. Y_ij,pq is pulse activation value (1 or 0).

The maximum active item U_ij is taken in the information fusion domain, and the dynamic threshold θ_ij is judged with it to generate a corresponding output activation signal.

$$\begin{array}{l}{U}_{ij}\left(n\right)=\max \{U_{ij}^1\left(n\right),U_{ij}^2\left(n\right)\}\\ Y_{ij}\left(n\right)=\{\begin{array}{cc}1& U_{ij}\left(n\right)\ge {\theta }_{ij}\left(n-1\right)\\ 0& \mathit{others}\end{array}\end{array}$$

(6)

where Y_ij is the output of neurons (1 or 0).

When the neuron is activated, the channel corresponding to the maximum activity item is selected as the fusion coefficient. Besides, dynamic threshold is updated to promote the activation of adjacent neurons, and the method of iterative updating is adopted until all neurons are ignited.

$${\theta }_{ij}\left(n\right)=e^{-{\alpha }_{\theta }}{\theta }_{ij}\left(n-1\right)+V_{\theta }{Y}_{ij}\left(n\right)$$

(7)

where α_θ is an attenuation constant of threshold. V_θ is a magnification factor of threshold.

2.3.2. Improved DPCNN Model

(1) Theory of improved DPCNN model

By analyzing the mechanism of the DPCNN model, it can be found that the setting of some default constants is very skillful for activating all neurons [33], which often affects the fusion performance and running time, such as maximum iteration Kn, α_θ, V_θ, and so on. Therefore, an improved DPCNN (IDPCNN) model is proposed to select the coefficients of high frequency sub-band images. The improved pulse activation process mainly occurs in the information judgment and pulse activation domain, as shown in Figure 3. The amplitude value Re_ij (k = 1, 2) is calculated by the soft limit function [34], which is in conjunction with the dynamic threshold to determine whether the pulse is activated and record each ignition time. After Kn iterations, the channel coefficients corresponding to the larger cumulative output time ∑T_ij^k are served as the fused high frequency sub-band coefficients.

$$\begin{array}{c}R{e}_{ij}^k\left(n\right)=1/\left(1+e^{\left({\theta }_{ij}\left(n-1\right)-U_{ij}{}^k\left(n\right)\right)}\right)\begin{array}{cc},& k=\left\{1,2\right\}\end{array}\\ Y_{ij}\left(n\right)=\{\begin{array}{cc}1& \max \left\{R{e}_{ij}^k\left(n\right)\right\}\ge 0.5\\ 0& \mathit{others}\end{array}\\ T_{ij}^k\left(n\right)=\begin{array}{cc}R{e}_{ij}^k\left(n\right)Y_{ij}\left(n\right),& k=\left\{1,2\right\}\end{array}\end{array}$$

(8)

where T_ij^k denotes as the pulse ignition time.

(2) Key parameter settings of IDPCNN model

In the traditional DPCNN model, the parameters mainly depend on experience and massive attempts [32]. A set of parameters that achieves good performance may be unsuitable for other data. Therefore, key points in the IDPCNN model are as follows for the image fusion of side-scan sonar, 1) pulse ignition time T_ij^k; 2) link strength β_ij^k; and 3) external excitation S_ij^k.

In this IDPCNN model, the soft limiting function is applied to reflect the amplitude difference of the total ignition time (see Equation (9)). Moreover, considering that local gradient energy is also one of image sharpness indicators, which can reflect the target edge and other detailed features, average gradient [35] is adapted to the characterization of link strength. The average gradient $\overline{d}$(x,y) is defined as follows:

$$\begin{array}{c}\overline{d}\left(x,y\right)=\frac{1}{9}{\displaystyle \sum _{p=-1}^1{\displaystyle \sum _{q=-1}^1\mathit{sqrt}\left(\left[g_1\left(x+p,y+q\right)+g_2\left(\left(x+p,y+q\right)\right)\right]/2\right)}}\\ \{\begin{array}{l}{g}_1\left(x,y\right)={\left[d\left(x,y\right)-d\left(x+1,y\right)\right]}^2\\ g_2\left(x,y\right)={\left[d\left(x,y\right)-d\left(x,y+1\right)\right]}^2\end{array}\end{array}$$

(9)

where d (x, y) is the high frequency sub-band coefficient. g₁ (x, y) and g₂ (x, y) represent the gradient difference in the horizontal and vertical directions, respectively.

Moreover, a novel sum-modified Laplacian (NSML) is set as external excitation, and denotes as:

$$\begin{array}{c}\begin{array}{ll}\hfill \mathit{nml}\left(x,y\right)=& \left|2\times d\left(x,y\right)-d\left(x-1,y\right)-d\left(x+1,y\right)\right|\\ & +\left|2\times d\left(x,y\right)-d\left(x,y-1\right)-d\left(x,y-1\right)\right|\\ & +\left|\sqrt{2}\times d\left(x,y\right)-\left(\sqrt{2}/2\right)\times d\left(x-1,y-1\right)-\left(\sqrt{2}/2\right)\times d\left(x+1,y+1\right)\right|\\ & +\left|\sqrt{2}\times d\left(x,y\right)-\left(\sqrt{2}/2\right)\times d\left(x-1,y+1\right)-\left(\sqrt{2}/2\right)\times d\left(x+1,y-1\right)\right|\end{array}\\ NSML\left(x,y\right)={\displaystyle \sum _{p=-1}^1{\displaystyle \sum _{q=-1}^{1}nml{\left(x+p,y+q\right)}^2}}\end{array}$$

(10)

2.4. Quality Evaluation of Fused Image

The evaluation of a fused image is usually performed by subjective visual and objective indexes. Subjective vision is mainly used to judge the fused effect of image contrast, ambiguity, noise elimination, target edge and texture features. Furthermore, objective indexes is able to accurately reflect the overall effect of the fused image and the visual information content of important targets, which is a complementary to the subjective visual effect. Therefore, some objective evaluation indexes of a fused image and its mathematical formula are listed in

Table 1. Objective index and mathematical description of the fused image. AG, average gradient; RCE, root mean square cross entropy; FD, figure definition; E, information entropy; MI, mutual information; Q^AB/F, edge-based similarity measure; SSIM, structural similarity index.

Objective Indexes	Mathematical Formulation
AG	$AG=\frac{1}{M\times N}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N\sqrt{\left({\left|f\left(x,y\right)-f\left(x-1,y\right)\right|}^2+{\left|f\left(x,y\right)-f\left(x,y-1\right)\right|}^2\right)/2}}}$
FD	$FD=\frac{1}{M\times N}{\displaystyle \sum _{x=1}^M{\displaystyle \sum _{y=1}^N\sqrt{\left({\left|f\left(x+1,y\right)-f\left(x,y\right)\right|}^2+{\left|f\left(x,y+1\right)-f\left(x,y\right)\right|}^2\right)/2}}}$
E	$E=-{\displaystyle \sum _{i=0}^{L-1}{p}_i{\log }_2{p}_i}$
RCE	$RCE=\sqrt{\left(C{E}_{A,F}^2+C{E}_{B,F}^2\right)/2}$, $C{E}_{A/B,F}={\displaystyle \sum _{i=0}^{L-1}{p}_{A/B}\left(i\right){\log }_2\frac{p_{A/B}\left(i\right)}{p_F\left(i\right)}}$
MI	$MI={\displaystyle \sum _{(a,b):f}{p}_{(A,B):F}\left(f,\left(a,b\right)\right){\log }_2\frac{p_{(A,B):F}\left(f,\left(a,b\right)\right)}{p_F\left(f\right)·p_{\left(A,B\right)}\left(a,b\right)}}$
QAB/F	$Q^{AB/F}=\frac{{\displaystyle {\sum }_{\forall m,n}\left(Q_{m,n}^{AF}{w}_{m,n}^A+Q_{m,n}^{BF}{w}_{m,n}^B\right)}}{{\displaystyle {\sum }_{\forall m,n}\left(w_{m,n}^A+w_{m,n}^B\right)}}$
SSIM	$\begin{array}{l}SSIM\left(A,B,F\right)=\left(SSIM\left(A,F\right)+SSIM\left(B,F\right)\right)/2\\ SSIM(A/B,F)=\frac{\left(2{\mu }_{A/B}{\mu }_F+C_1\right)\left(2{\sigma }_{A/B,F}+C_2\right)}{\left({\mu }_{A/B}^2+{\mu }_F^2+C_1\right)\left({\mu }_{A/B}^2+{\mu }_F^2+C_2\right)}\end{array}$

. Combining the characteristics of underwater sonar images, average gradient (AG) [36], figure definition (FD), information entropy (E) [37], root mean square cross entropy (RCE) [38], mutual information (MI) [9], edge-based similarity measure (Q^AB/F) [14], structural similarity index (SSIM) [39], and indexes of Piella [40] are applied to objectively evaluate the fused image.

2.5. Implementation of Fused Technique

The scheme process of the proposed fusion method is discussed in detail, as shown in Algorithm 1. The fusion framework based on the NSCT domain includes image preprocessing and registration, decomposed acquisition of the low and high frequency sub-band image, coefficient selection of the sub-band image, reconstruction of the fused image, and quality evaluation. First, the adjacent sonar images are registered using the speed up robust features (SURF) [41] method to obtain the information images of the overlapping area. NSCT is subsequently applied to decompose the pair of overlapping area images into low and high frequency sub-band images.

Moreover, the sub-band coefficients of low and high frequency images are determined by the SMLF and IDPCNN model, respectively. However, after the above fusion criteria are applied, there are some random discrete or isolated pixels in the small neighborhood of sub-band coefficient matrices, which are obviously different from the adjacent pixel source images [42]. Therefore, consistency verification (CV) [43] is carried out through a window size of 3 × 3 to ensure the consistency relationship between adjacent coefficient sources.

Finally, according to the selected sub-band images of low and high frequency, the fused image is reconstructed by inverse transformation of NSCT, and the effect of fused image is quantitatively analyzed and evaluated from subjective vision and multiple objective indexes.

Algorithm 1 Sonar Image Fusion Based on SMLF energy and IDPCNN Model

Input: Read in two sonar images of adjacent strip. Set some constant item values, Kn, αL, VL, αθ, Vθ, CR = 1. Initialize some parameters and matrices, Yij is zero matrix, ωij,pq is ones matrix, the iteration number n = 1.

Output: Fused image, difference image and objective evaluation indexes

1: Image registration to obtain overlapped area image I1, I2;

2: Acquisition of low and high frequency image matrix CA/B, dijA/B with NSCT transform;

3: Calculate the SML energy of low frequency sub-band coefficient by Equation (1);

4: Relate SML energy with visual adjustment value to obtain RLV parameter values based on Equation (2);

5: Decision map is obtained by the guided filter and multiple filters based on Equation (3);

6: Determine the coefficient source of low frequency sub-band by using Equation (4);

7: while (iteration number n ≤ maximum iteration Kn)

8: Average gradient $\overline{d}$ (x, y) and NSML (x, y) is obtained by Equations (9) and (10);

9: Get Fijk (n), Lij (n), Uijk (n) of each neuron in the receiving domain by Equation (5);

10: Judge whether the pulse signal is activated by Equation (8);

11: if Amplitude value Re (i, j) ≤ 0.5

12:  The neuron (i, j) is activated, Yij = 1. Record the ignition time of each iteration Tijk (n), and update dynamic threshold θij (n) by Equation (7);

13: end if

n = n + 1;

14: end while

15: After Kn iterations, high frequency sub-band coefficients are selected according to the total ignition time;

16: Sub-band coefficients of low and high frequency pass the window verification to ensure the consistency of adjacent coefficient sources;

17: The fused image of overlapping area is reconstructed by NSCT inverse transform;

18: return Fused image is subtracted from overlapping image to obtain the difference image, and the objective evaluation indexes of fused image are obtained by Table 1.

3. Experiments and Analysis

3.1. Data Description

To evaluate the proposed low and high frequency fusion criteria and their performance in the NSCT domain, three sets of side-scan sonar images are used to verify the performance of the fusion criteria. Data 1 is a small-scale image of aircraft debris with clear target profile features. Data 2 is a medium-scale pipeline detection image, which contains more information, such as oil pipeline targets, rich terrain texture, and different types of sediment characteristics. The above source images were subjected to median filtering, and local Gaussian blurring (sigma = 5) is performed on the left and right sides, as shown in Figure 4 and Figure 5. This procedure is an imitation process that the overlapping area image generated by the echo intensity of adjacent strips is not relatively clear in complex waters. The phenomenon of clear underwater topography and small noise interference exists in the right strip, but the image information of left strip is the opposite.

Data 3 is the sonar images with a high overlap rate, which is collected from the port area of Rhode Island, USA, in 2011. Two sonar images are processed by seabed tracking, time variable gain, image de-noising, and image registration to obtain the large-scale overlapping areas. Stacking the overlapped area image together reflects the information differences between two images (Figure 6). Partially enlarged areas of Figure 6 indicate that underwater protective embankment and the position of underwater target reef have basically overlapped after strict registration. In addition, image color values display that the amount of information is different with the overlapped area of adjacent strips, and local area information is missing. Moreover, the emphasis of the measured underwater target is different and the terrain relief texture details are diverse in the local area.

3.2. Comparison of Fusion Criteria

NSCT transform is adopted to decompose the images into low and high frequency sub-band images, and the information extraction influence of different fusion criteria is analyzed. Among them, the shared parameters of the PCNN or IDPCNN model are set as follows: max iteration Kn is 200, link input attenuation constant α_L =1, link input amplification factor V_L = 1, threshold attenuation constant α_θ = 0.2, threshold amplification factor V_θ = 20, and non-linear combination coefficient ratio between link input and external excitation CR = 1. The specific design schemes are as follows.

3.2.1. Low Frequency Fusion

The low frequency sub-band image inherits a large amount of energy information from the source image, so Data 1 is applied to analyze the influence of different fusion criteria on information extraction. Selection criteria include the following: (1) Mean; (2) Local_STD; (3) PCNN; (4) SML; (5) EOL Filtering; (6) SMLF. The parameter descriptions of low frequency fusion criteria are shown in

Table 2. Parameter description of low frequency fusion criteria. NSML, novel sum-modified Laplacian; SMLF, SML energy filtering; PCNN, pulse coupled neural network; EOL, energy of Laplacian; RLV, relating sum-modified Laplacian energy with visual contrast value; GF, guided filter; CV, consistency verification.

Criterion	Parameter Description
Mean	Average process
Local_STD	Maximum value of local standard deviation
PCNN	Tij: soft limiting function. βij: average gradient. Sij: NSML
SML	Maximum of SML value
EOL Filtering	Guidance map: EOL. Edge detection: GF. Source consistency: CV.
SMLF	Guidance map: RLV. Edge detection: GF. Source consistency: CV.

, and the experimental results are shown in Figure 7.

Figure 7a illustrates that the low frequency sub-band image processed by the Mean criterion is relatively blurred. The criterion of Mean smoothens the whole low frequency image and is not able to reflect the streamline profile of the aircraft wreck. Although the sub-band images (Figure 7b–d) maintain the advantages of high contrast and clear brightness, there is a pseudo-contour effect of block flow around the edges of the aircraft wreckage, especially, the sub-band image processed by the criterion of Local_STD leaves the most, followed by the result of the PCNN criterion. This is the reason that PCNN model is not adopted to select the sub-band coefficients of low frequency in the early processing of subsequent experiments, so as to avoid the interference of the analysis. Figure 7e shows that the fused image processed by the criterion of EOL filtering possesses distinct brightness on the whole aircraft (nose, wings, tail), but insufficient brightness exists in the shadow of the aircraft wreckage. This fusion criterion results in a reduced visual contrast of low frequency sub-band image. Figure 7f displays that the low frequency sub-band image processed by the SMLF energy criterion has a clear contour, relatively high contrast; the brightness is roughly the same as the source image; and no false fusion pseudo-contour effect is generated. In general, the visual effect of the sub-band image processed by the proposed low frequency fusion criterion is superior to other fusion strategies.

3.2.2. High Frequency Fusion

The high frequency sub-band image contains the detailed feature information from the source image, such as target contour and terrain texture. On the basis of the optimal low frequency fusion criterion of SMLF energy, experiments are conducted with Data 1 and 2, and the performance of high frequency fusion criteria is evaluated by the fused image and difference image (Figure 8 and Figure 9). The fusion criteria include the following: (1) Energy_Max; (2) Direction contrast maximum (Dire_Contrast_Max); (3) PCNN (NSML); (4) IDPCNN (MSF) [25]; (5) IDPCNN (NSML). Among the PCNN and IDPCNN model, ignition time and link strength are represented by the soft limiting function and AG value, respectively, which are the optimal choice directly. It mainly discusses the influence of external excitation value on these models, which are represented by MSF and NSML, respectively.

Figure 8b shows that the contrast of fused image is reduced and the feature of detail changes is smoothed, while other fused images are basically consistent with the source image, with good subjective visual effects. In addition, the difference image of Figure 8b displays that the remaining information leaves a lot on the left side, and the information extraction of the aircraft contour edge is not sufficient, which indirectly indicates that the Dire_Contrast_Max criterion is not applicable to sonar image fusion. The difference image of Figure 8a shows that there still exists an insufficient partial integration. The remaining information of Figure 8c–e is almost 0, which corresponds to the clear left area and blurred right area of source image. Better visual effects illustrate that the high frequency fusion model of the PCNN family (criterion 3, 4, 5) can fully extract the source image favorable information, which is more suitable for sonar image fusion.

Figure 9b shows that the information extraction of texture and features is not sufficient, but other fused images have good visual effects. In addition, the difference images display that the residual error in Figure 9b is the greatest, followed by the fusion criterion of Energy_Max, which fully reflects the poor ability of the two criteria in processing strip fusion. On the contrary, the residual errors on the left side of Figure 9c, d are almost 0. The terrain and geomorphology texture of the fused image are clear, and the peripheral contour of the petroleum pipeline has no false contour phenomenon, which basically inherits the detail information features from the source image.

From the perspective of subjective vision, the fused images of criteria 3, 4, and 5 meet the requirements of high information retention. In order to further evaluate the performance of the criterion, some objective indexes are analyzed quantitatively, as shown in

Table 3. Objective evaluation indexes of the fused aircraft debris image (Data 1).

Criterion	Energy_Max	Dire_Contrast_Max	PCNN (NSML)	IDPCNN (MSF)	IDPCNN (NSML)
AG	6.6622	3.9657	6.7027	6.7025	6.7031
E	6.9263	6.8291	6.9261	6.9261	6.9263
RCE	0.0895	0.0527	0.0825	0.0822	0.0822
FD	9.5084	5.3125	9.5364	9.5366	9.5373
QAB/F	0.6362	0.4134	0.6414	0.6417	0.6417
IFQI 1	0.7732	0.3769	0.7834	0.7826	0.7834
WFQI 2	0.9092	0.5074	0.9242	0.9247	0.9247
EFQI 3	0.4545	0.3793	0.4756	0.4793	0.4813

¹ IFQI denotes as the overall similarity with source image. ² WFQI denotes as local regional significance. ³ EFQI is described as edge image similarity. MSF, modified spatial frequency. The parameter settings of following categories are similar, and they are not repeated for comment.

and

Table 4. Objective evaluation indexes of the rich terrain fused image (Data 2).

Criterion	Energy_Max	Dire_Contrast_Max	PCNN (NSML)	IDPCNN (MSF)	IDPCNN (NSML)
AG	9.7983	5.6267	9.8573	9.8565	9.8580
E	7.0776	6.9663	7.0835	7.0835	7.0836
RCE	0.0409	0.0429	0.0438	0.0441	0.0439
FD	13.3184	7.3078	13.3596	13.3582	13.3608
QAB/F	0.6783	0.4031	0.6814	0.6817	0.6818
IFQI	0.7361	0.3585	0.7470	0.7474	0.7474
WFQI	0.8894	0.4604	0.9003	0.9003	0.9004
EFQI	0.0409	0.0429	0.0438	0.0441	0.0439

.

Table 3 and Table 4 show that each index value based on the criterion of Dire_Contrast_Max is the smallest. For example, the index values of AG, FD, and Q^AB/F are only 3.9657, 5.3125, and 0.4134 in Data 1, which indicates that the fused image deviates greatly from the source image and exhibits the phenomenon of distortion. The index value of maximum energy is not as good as that of the PCNN family. The quality of the fused image can be improved by using the PCNN family model to select the coefficients. Moreover, compared with the traditional PCNN model, the index values of the IDPCNN series are relatively larger. This is because the IDPCNN model combines the input of information from both source images, which reduces the ability of misjudging the information source. Compared with the IDPCNN model with the MSF parameter, most indicator values of the IDPCNN model with the NSML value are larger, which demonstrates that the IDPCNN model with the NSML value can better reflect the energy information of the sonar image, and conform to its own characteristics of the source image.

3.2.3. Combination of Low and High Frequency Fusion Criteria

In order to verify the combined performance of the superior low and high frequency fusion criteria mentioned above, experiments are performed using measured Data 3. The combined fusion criteria are shown in

Table 5. Combination method of low and high frequency fusion criteria.

Fusion Technique	Low Frequency	High Frequency
Technique 1	IDPCNN (NSML)	IDPCNN (NSML)
Technique 2	EOL Filtering	IDPCNN (MSF)
Technique 3	EOL Filtering	IDPCNN (NSML)
Technique 4	SMLF	IDPCNN (MSF)
Proposed Technique 5	SMLF	IDPCNN (NSML)

. The fused image (Figure 10) is highlighted by L1–L6 areas, and the characteristics of each region are used to analyze the performance of the combined fusion criteria. In addition, some objective indexes are applied to evaluate the algorithm’s ability, as shown in

Table 6. Objective evaluation indexes of fused underwater port image.

Indexes	Technique 1	Technique 2	Technique 3	Technique 4	Proposed Technique 5
AG	1.3041	1.3036	1.3045	1.2342	1.3042
E	3.9577	3.9626	4.0599	3.9346	4.0776
FD	1.4626	1.4614	1.4667	1.3894	1.4708
RCE	0.0641	0.0641	0.0730	0.1097	0.1290
MI	1.5594	1.5631	1.6243	1.5281	1.6661
QAB/F	0.4272	0.4259	0.4285	0.3922	0.4302
SSIM	0.8409	0.8390	0.8397	0.8313	0.8624
IFQI	0.1967	0.2001	0.2007	0.1606	0.2027
WFQI	0.6059	0.6100	0.6269	0.4535	0.5781
EFQI	0.2765	0.2760	0.2790	0.2649	0.2862

.

Compared with each area in Figure 10, different features and information are displayed comprehensively. In the image edge area L1 and the terrain fluctuation area L4, the phenomenon of incomplete fusion and discontinuous shadows are generated by the processing of combined technique 1, 2, and 3, and the combined technique 4 is unable to availably extract the information and topographic relief change characteristics of this area. For the L2 and L3 areas with incomplete detection information, the combined technique 4 and 5 fully absorb the complementary information of the two strips, showing a clear overview of the slope and boundary characteristics. In addition, the combined technique 1, 2, and 3 have no capacity for weakening the interference effect of residual error in the L5 area, and the fused image still retains the error traces after the subsea tracking processing. As for the edge contour area L6, the combined technique 5 proposed in this paper can effectively fuse the missing information of adjacent strips in the overlapping area, and the shadow trace of fused discontinuous discrimination is not produced.

Table 6 shows that many objective indexes of the proposed technique 5 are close to or better than other methods, for example, the indexes of AG, FD, RCE, and IFQI are 1.3042, 1.4708, 0.1290, and 0.2027, respectively, which demonstrate that the proposed method can better integrate the overlapping area information of adjacent strips. The larger value of AG and FD reflects the contour of reef, subtle features of geographic texture, and clarity of the fused image. Moreover, the image edge index Q^AB/F and EFQI reflect that the edge details of the fused image generated by the proposed method carry more information. Furthermore, the greater similarity indexes (MI, SSIM) demonstrate that the fused image processed by the proposed method fully combines the respective feature information of the multi-source image. Therefore, the evaluation results of multiple indexes objectively demonstrate that the proposed fusion criteria of SMLF energy and IDPCNN (NSML) is more suitable for the sonar image fusion processing of adjacent strips.

4. Conclusions

This paper details a method combining SMLF energy and the IDPCNN model for side-scan sonar image fusion in the NSCT domain. Compared with the common fusion criteria, such as mean, STD, SML, and PCNN, the optimal selection of low frequency fusion criteria is analyzed using the aircraft debris data. The experimental results demonstrate that SMLF energy can effectively eliminate the pseudo-contour effect of block flow from the target edge. In addition, in order to analyze the high frequency fusion criteria more comprehensively, experiments are carried out with aircraft debris and rich geomorphic data, and maximum energy, local directional contrast, and PCNN are utilized. It is revealed that the IDPCNN model can extract more detailed feature information from the source image and reduce the ability of misjudgment. Finally, the superiority of combined fusion criterion in the NSCT domain is demonstrated by using the measured sonar data of the port. Multiple areas display that the overall and detailed information processed by the proposed method were effectively integrated, which reflect the complete target contour and rich features of seabed topography fluctuation. In addition, the fused image clearly shows the stitching conversion line left by the image registration. In the future, in order to ensure the image brightness on both sides is the same, the image fusion of eliminating the stitching conversion line is the next topic to be further studied.