Autonomous Underwater Vehicle Based Chemical Plume Tracing via Deep Reinforcement Learning Methods

Abstract

This article presents two new chemical plume tracing (CPT) algorithms for using on autonomous underwater vehicles (AUVs) to locate hydrothermal vents. We aim to design effective CPT navigation algorithms that direct AUVs to trace emitted hydrothermal plumes to the hydrothermal vent. Traditional CPT algorithms can be grouped into two categories, including bio-inspired and engineering-based methods, but they are limited by either search inefficiency in turbulent flow environments or high computational costs. To approach this problem, we design a new CPT algorithm by fusing traditional CPT methods. Specifically, two deep reinforcement learning (RL) algorithms, including double deep Q-network (DDQN) and deep deterministic policy gradient (DDPG), are employed to train a customized deep neural network that dynamically combines two traditional CPT algorithms during the search process. Simulation experiments show that both DDQN- and DDPG-based CPT algorithms achieve a high success rate (>90%) in either laminar or turbulent flow environments. Moreover, compared to traditional moth-inspired method, the averaged search time is improved by 67% for the DDQN- and 44% for the DDPG-based CPT algorithms in turbulent flow environments.

1. Introduction

Autonomous underwater vehicles (AUVs) play a significant role in marine science surveys. One important application is using AUVs to find underwater hydrothermal vents. Hydrothermal vents, i.e., ocean volcanoes, are natural plumbing systems that transport heat and minerals from the interior of the Earth to the ocean [1]. Studying hydrothermal vents can help scientists investigate the origin and evolution of life on Earth [2]. Tracing the emitted hydrothermal plumes to find hydrothermal vents is the most effective method since the emitted chemical materials are distinct from the surrounding seawater and can disperse over a long distance driven by ocean currents [3]. Therefore, it is possible to deploy an AUV equipped with chemical sensors to detect emitted chemical plumes and trace them back to the source. Several successful AUV-based hydrothermal vent surveys were conducted in the Pacific [4] and Atlantic Oceans [5], which verified the feasibility of using AUVs in finding hydrothermal vents in real-world environments. However, existing AUV-based hydrothermal vent localization methods relied on a simple and fixed search trajectory, e.g., the lawn mower trajectory. Designing an effective, efficient, and intelligent hydrothermal vent search strategy requires further investigation.

The key to successfully finding hydrothermal vents is the design of an effective chemical plume tracing (CPT) algorithm [6]. CPT algorithms guide the search agent, i.e., an AUV, to detect chemical plumes as cues to find the source. A similar application is image-based navigation [7], which directs the search agent using the information extracted from captured images. In CPT algorithms, estimating emitted chemical plumes is the most challenging task since the plume distribution varies in different flow environments [8]. For instance, chemical plumes form a spatially coherent trajectory in laminar flow environments thanks to the stable flow advection, whereas in turbulent flow environments, the plume trajectory is an intermittent and patchy path since the turbulent flows stretch and twist chemical plumes in the environment.

Various CPT algorithms were developed for tracing chemical plumes and finding chemical sources. These CPT algorithms can be categorized into two groups, including bio-inspired and engineering-based methods. As suggested as its name, bio-inspired methods command a search agent to find the chemical source by mimicking animal olfactory behaviors. A typical example is the mate-seeking behaviors of male moths, which could locate a female moth by detecting emitted pheromone over a long distance. On the other hand, engineering-based methods (also called probabilistic methods) calculate the probability of a region containing the chemical source. These methods rely on mathematical plume propagation models to estimate the chemical source locations.

In recent years, thanks to the development of computational technologies, we have witnessed a surge in deep learning/machine learning technologies and applications. Among different deep learning algorithms, reinforcement learning (RL) [9] is suitable for implementing on robotic applications. The core of RL algorithms is to model the interaction between the agent and the environment. The agent can choose actions (either continuous or discrete actions) in the environment. In return, the agent will receive rewards and be transmitted to a new state in the environment. The goal of the agent is to obtain a maximal cumulative reward. Available RL algorithms can be categorized into two groups: model-based and model-free RL algorithms [10]. The difference between these two groups is whether the agent has access to a model of the environment. Having a model, e.g., model-based RL, allows the agent to plan and calculate future rewards. This ability enables the agent to predict what would happen when choosing an action. On the other hand, model-free RL (or deep RL) does not have access to the environment model. Instead, these RL algorithms focus on estimating the value functions directly from interactions with the environment. In practice, model-free RL has a broader application spectrum since the environment model is usually unavailable in many applications. Model-free RL algorithms have been implemented in many applications that require intelligent decision-making, such as playing Atari games [11] and playing the game of Go [12].

Considering the difficulty of modeling an unknown underwater environment, we leverage model-free RL algorithms (i.e., deep RL) to solve the problem of locating hydrothermal vents. The goal is to design an intelligent agent that can effectively fuse two traditional CPT algorithms from two categories, i.e., bio-inspired and engineering-based algorithms, to improve search efficiency. The core idea of our proposed method is to get a good search agent with a short training time and limited computational resources. Thus, instead of training a search agent from scratch, we pre-process sensor data using traditional CPT algorithms and let the RL models to choose a good search strategy during the search process. Compared to end-to-end training, this operation helps us reduce the training time and the querying time of neural networks [13,14]. The motivations for combining two traditional CPT methods are twofold. First, RL algorithms are notoriously in sample inefficiency [15]. Training a functional end-to-end deep neural network (DNN), i.e., a DNN that takes sensor measurements as inputs and calculates robot commands directly, using RL algorithms is time-consuming and requires intensive computational resources. Taking Alpha Go Zero [16] as an example, it takes 40 days on 4 Tensor Processing Units (TPUs) to train a superhuman level DNN to play the game of GO. Therefore, to accelerate the training process, we train a DNN to dynamically choose (or combine) provided CPT algorithms instead of training an end-to-end DNN that learns an effective navigation algorithm from scratch. Secondly, bio-inspired and engineering-based methods have distinct search performances in different flow environments. In laminar flow environments, a bio-inspired method is more efficient since it can command the robot to maintain inside plume trajectories, enabling the robot to quickly approach the odor source location. In turbulent flow environments, an engineering-based method is more effective since it can estimate possible odor source locations based on previous detection history. In contrast, a bio-inspired method must command the robot to perform a time-consuming cross-wind behavior to re-detect plumes, which has a very low success rate in turbulent flow environments.

Specifically, the moth-inspired method [17] (i.e., a typical bio-inspired CPT method) and the Bayesian-inference method [18] (i.e., an engineering-based CPT method) are selected in this work. Both methods were proven valid in real-world CPT experiments on an AUV. As for deep RL algorithms, we choose the double deep Q-network (DDQN) [19] and deep deterministic policy gradient (DDPG) [20] algorithms to train a deep neural network (DNN), respectively. Inputs of the DNN are AUV’s sensor measurements, and the output is a fusion coefficient that combines heading commands calculated from two traditional CPT methods. Depending on the implemented deep RL algorithm, the fusion coefficient could be a binary switch (for DDQN) or a variable between 0 to 1 (for DDPG). The training was conducted in a simulated plume tracing environment. After the training, we evaluated the search performance of two deep RL algorithms in repeated tests and compared them with traditional CPT methods.

The main contribution of this work can be listed as follows:

• Fuse two traditional CPT algorithms using deep RL methods to improve the search efficiency;
• Present two different fusion algorithms, including discrete and continuous fusion coefficients;
• Verify the effectiveness of the proposed methods in a realistic simulation program in repeated tests and compare the search results with traditional CPT algorithms without the fusion.

In summary, the remaining paper is organized as below. Section 2 reviews the recent related works. Section 3 demonstrates the flow diagram of the proposed algorithms, two traditional CPT algorithms, and the basic knowledge in RL algorithms. Section 4 presents the proposed deep RL-based CPT algorithm. Section 5 presents the simulation program and statistic results in repeated tests.

2. Related Works

In the early phase of the CPT studying, the intuitive chemotaxis CPT algorithm [21], i.e., the gradient following algorithm, was developed and implemented on robots to find chemical sources in a laminar flow environment. In chemotaxis CPT algorithms, the search agent was equipped with a pair of chemical sensors on two sides of the agent. The agent was commanded to move to the side with higher detected chemical concentration. This simple navigation algorithm performs well in laminar flow environments, as verified by these applications [22,23,24,25]. However, the plume trajectory becomes intermittent in turbulent flow environments, resulting in a discontinuous plume concentration distribution. Therefore, the search agent was always led in the wrong direction or trapped in a local maximal point with the chemotaxis CPT algorithms in turbulent flow environments.

A variant algorithm of chemotaxis is the lobster-inspired algorithm, which tries to solve the problem of tracing chemical plumes in turbulent flow environments. The lobster’s foraging behaviors inspired this kind of CPT algorithm: a lobster turns toward the side of higher odor concentration or goes straight forward if two antennules detect nearly the same concentration, and if neither of the antennules detects odor concentrations, the lobster moves backward. Compared to the original chemotaxis CPT algorithms, the lobster-inspired algorithms utilize both temporal and spatial information, helping improve the success rate in finding chemical sources in turbulent flow environments. According to [26], adding the second strategy, i.e., the search agent moves backward when neither chemical sensor detects chemical plume concentrations, improves the success rate by 33%. In addition, recent simulation studies [27,28] showed that the lobster-inspired algorithm achieves a higher success rate than the chemotaxis algorithm alone.

Another bio-inspired CPT algorithm is the moth-inspired algorithm. As suggested by its name, the moth-inspired method was derived based on the mate-seeking behaviors of moths. In nature, a male moth can locate a female moth over a long distance (several kilometers) by tracing the emitted pheromone plumes [29]. This search strategy can be summarized as two behaviors: (1) the upwind search when pheromone plumes were detected and (2) a local search after losing the plume contact [30]. Specifically, the male moth will fly upwind when it detects the emitted pheromone plumes and fly crosswind when it loses pheromone contact. The upwind search is also called the ‘surge’ behavior, and the crosswind search is called the ‘casting’ behavior. A male moth can find a female moth by iterating these two behaviors. There are many applications of moth-inspired methods on AUVs. For instance, Li et al. [31] and Pang et al. [32] applied the moth-inspired method on an AUV to find an underwater chemical source. In their methods, the traditional surge behavior was split into track-in and track-out behaviors, which helped the AUV maintain inside chemical plumes. Shigaki et al. [33] improved the traditional moth-inspired method by dynamically changing the duration of surge behavior. Their method calculated the surge time based on the moth’s muscle activities under the stimulation of chemical plumes. They also devised a fuzzy inference system to adaptively transit different search behaviors based on current search readings [34].

By contrast to bio-inspired methods, we can also use mathematical models of chemical plumes to estimate the possibility of plume distribution in the search area and use this probability distribution to direct the source search. Using mathematical models to direct the chemical source search is referred to as engineering-based algorithms (or probabilistic methods). The key in this kind of CPT algorithm is calculating a source probability map, which presents the probability of each region containing the chemical source in the search area. For instance, the Bayesian-inference method [18] was devised to estimate possible chemical source locations. The core of this method was to model plume movements as a Gaussian random process and inversely calculate the probability of a region containing the chemical source via a series of plume detection and non-detection events. Besides, hidden Markov model [35], particle filter [36], occupancy grid [37], and partially observable Markov decision process [38] can also be employed to design the source probability map. Once the source probability map was obtained, the search agent can move to the region with the highest probability of containing the chemical source by directing with the path planners. Possible path planning algorithms include artificial potential field [39] and A-star [40] path planning algorithms. Vergassola et al. [41] proposed a infotaxis algorithm, which employed the information entropy to direct the robot search for the chemical source. The agent will select the future movement that greatly reduces the information uncertainty of chemical source.

Multi-agent search is also popular in the recent CPT development. A heavily quoted method was particle swarm optimization (PSO) [42]. The core idea is to optimize the problem via iteratively improving candidate solutions with a fitness function. To apply PSO to CPT problems, [43,44] modeled the measured chemical concentration as the fitness function, and [45,46] modeled the source probability map, generated from engineering-based methods, as the fitness function. The best candidate can be treated as the chemical source estimate, and this best candidate will be evaluated based on the fitness function. The search agents’ positions will be updated to approach the best candidate.

Thanks to the recent development of high performance computing technologies, we have witnessed a shift of using deep learning and machine learning methods to design effective CPT algorithms. In [47], the plume tracing process is modeled as a model-based RL problem, where rewards are defined based on a combination of plume and odor source estimations. To dynamically combine plume and odor source estimations, a fuzzy inference system is designed to estimate the current search situation and balance the exploration and exploitation. In [48], two types of deep neural networks, including a feedforward and convolutional neural networks, are trained via the supervised learning method. Training data was obtained by implementing traditional CPT methods in a simulation program. Hu et al. [13] presents a deep RL-based CPT algorithm for using on an AUV for locating hydrothermal vents. In this work, a variant of deep deterministic policy gradient algorithm is adopted to train an end-to-end recurrent neural network (RNN), which takes AUV observations as inputs and calculates AUV heading commands. Chen et al. [49] presents a deep Q-network based CPT algorithm that takes a sensor measurement heating map as an input and calculates robot commands. To the best of authors’ knowledge, there is no available applications that use deep RL to combine multiple CPT algorithms to achieve a higher search efficiency.

3. Preliminaries

3.1. An Overview of the Proposed CPT Algorithms

In this work, we present two new CPT algorithms for using on AUVs to trace emitted chemical plumes and find hydrothermal vent locations. First of all, we model the plume tracing of hydrothermal vent as a 2-dimensional (2-D) search problem. This is because the emitted hydrothermal plumes is mainly diffused on a fixed depth in an underwater environment. As shown in Figure 1, the structure of the emitted hydrothermal plumes comprises two components, namely buoyant and non-buoyant plumes. The non-buoyant plumes could range from tens of hundreds of kilometers at a fixed altitude above the seafloor [17]. Therefore, we simplify the hydrothermal vent localization problem on a 2-D plane search problem, where the AUV aims to find the hydrothermal vent on a horizontal plane of non-buoyant plumes. This simplification can also be found in other CPT applications, such as [13,50,51].

Figure 2 shows the flow diagram of the proposed CPT algorithm. During the plume tracing process, onboard sensors on the AUV measure AUV’s positions $(x,y)$, flow speed and direction $(v,\phi )$, chemical concentration $\zeta$, and AUV’s current heading $\varphi$. Sensor measurements will be fed into two modules, including a DNN (trained by DDPG or DDQN) and traditional CPT methods. Specifically, two traditional CPT methods are implemented, including the moth-inspired method [17] and the Bayesian-inference method [18]. Two traditional methods will generate two heading commands, i.e., ${\varphi }_M$ and ${\varphi }_B$, based on the current sensor readings. On the other hand, the DNN also takes sensor measurements as inputs and calculates a fusion coefficient a. Here, a is a binary number if the DNN is trained by DDQN and a scalar if the DNN is trained by DDPG. Finally, the overall heading command is calculated by the superposition of ${\varphi }_M$ and ${\varphi }_B$, where the weights are a and $1-a$, respectively.

3.2. Moth-Inspired CPT Algorithm

The moth-inspired CPT algorithm is characterised in simplicity and high search efficiency in laminar flow environments. This section reviews the classic moth-inspired CPT method [17] that we implemented in this work.

The moth-inspired method can be summarized as the alternation of ‘surge’ and ‘casting’ behaviors. Figure 3 presents the implementation of moth-inspired CPT algorithm proposed in [17], which further splits the ‘surge’ behavior as ‘track-in’ and ‘track-out’ behaviors. Specifically, in the ‘track-in’ behavior (as shown in Figure 3a), the AUV detects chemical plumes and moves upwind to approach the chemical source. In the ‘track-out’ behavior (as shown in Figure 3b), the AUV moves outside of plume areas and tries to re-detect plumes in the nearby of last detection location by moving across the plume trajectory. If the AUV fails to re-detect plumes in this behavior, then the ‘casting’ behavior (in [17], this behavior was called ‘reacquire’ behavior) is activated, which commands the robot to perform crosswind excursions to detect chemical plumes over a large area. As shown in Figure 3c, the ‘casting’ behavior was modeled as a ‘bow-tie’ trajectory, where the AUV was commanded to move to four points, i.e., ${\mathbf{P}}_1$ to ${\mathbf{P}}_4$ around the last plume detection location, i.e., ${\mathbf{P}}_{ctr}$. The AUV will turn back to the ‘track-in’ behavior if it detects chemical plumes. The AUV repeats these search behaviors until it finds the chemical source.

The ‘track-in’ behavior in the moth-inspired method CPT method can quickly guide the AUV to approach the chemical source in a laminar flow environment. In turbulent flow environment, the AUV cannot continuously detect chemical plumes since plume trajectory is not continuous but a patchy and intermittent trajectory. Therefore, the moth-inspired method is not efficient in turbulent flow environments since the AUV can only relay on ‘casting’ behavior to re-detect plumes. To mitigate this problem, we also include an engineering-based method that can estimate possible chemical source locations when the AUV loses plume contact in our CPT algorithm’s design.

3.3. Bayesian-Inference CPT Algorithm

The Bayesian-inference method [18] models the plume movements as a Gaussian random process and estimates possible chemical source locations with sensed flow information. In this method, the search area was divided into multiple cells, where each cell has a probability of containing the chemical source. With the AUV maneuver, the probability of containing the chemical source in each cell was updating based on the plume detection and non-detection events. Figure 4 presents the source probability map in a search trial. In this diagram, the red cell indicates the estimate source location, i.e., the cell with the highest probability of containing the chemical source. In our previous simulation and real-world experiments [52], we found that the Bayesian-inference CPT algorithm is more effective than the moth-inspired CPT algorithm in a turbulent flow environment. This is because the source estimation in the Bayesian-inference method is calculated based on the previous plume information. When the AUV loses the plume contact, instead of randomly moving in a wide region and hoping to detect plumes by accident, the source estimation is more instructive to efficiently find the chemical source location since it pinpoints the possible source location.

3.4. RL Basics

The standard RL setting contains an agent and an environment, and the interaction between the agent and the environment is modeled as a Markov decision process (MDP)$\sim \{\mathcal{S},\mathcal{A},P,R,\gamma \}$, where $\mathcal{S}$ is a set of states, $\mathcal{A}$ is a set of actions, P is a transition probability, R is a reward function, and $\gamma$ is a discounting factor for future rewards. Figure 5 shows a MDP model within one time step. The agent is in a state ($S_t=s$, $s\in \mathcal{S}$) at time t, and after it takes an action ($A_t=a$, $a\in \mathcal{A}$) to arrive in the next state ($S_{t+1}=s^{\prime }$, $s^{\prime }\in \mathcal{S}$) with the state transition probability $P\left(s^{\prime }\right|s,a)$, the agent will obtain a reward ($R_{t+1}=r$, $r\in \mathcal{R}$).

An episode describes how the agent interacts with the environment. The policy ($\pi$) describes which action the agent will take in the state $S_t$. The policy can be considered as a mapping function from state $S_t$ to action $A_t$. The goal of the agent is to obtain a policy that maximizes the cumulative reward, which can be represented as: $G_t=R_{t+1}+\gamma R_{t+2}+\cdots ={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$.

The value function is a prediction of the future reward. There are two kinds of value functions: state-value function and action-value function. The state-value function (i.e., $V_{\pi }\left(s\right)$) evaluates the goodness of the agent being in a state s, and the action-value function (i.e., $Q_{\pi }(s,a)$) evaluates the goodness of the agent performs an action a in a state s. Mathematically, $V_{\pi }\left(s\right)$ is the expectation of the return (total future rewards) starting from the current state $S_t$, and then following the policy $\pi$: $V_{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[G_t|S_t=s\right]$; $Q_{\pi }(s,a)$ is the expectation of the return starting from state s, taking the action a, and then following the policy $\pi$: $Q_{\pi }(s,a)={\mathbb{E}}_{\pi }\left[G_t|S_t=s,A_t=a\right]$.

4. Deep RL-Based CPT Algorithms

Both DDQN and DDPG belong to model-free RL algorithms (i.e., deep RL), where the main difference is that DDQN has a discrete action space, and DDPG has a continuous action space. This section presents how to implement these two deep RL algorithms to train a DNN that can fuse heading commands of traditional CPT methods.

4.1. DDQN-Based CPT Method

The DDQN algorithm originates from the Q-learning and deep Q-network (DQN). In the Q-learning algorithm, every state-action pair has an action-value function, i.e., $Q(s,a)$. At every time step, the agent selects the action with the highest action-value function, i.e., $a=arg{max}_{a\in \mathcal{A}}Q(s,a)$. However, if the state space is large, iterating all state-action pairs becomes time-consuming and even impossible. To solve this problem, DQN [11] trained a DNN to calculate action-value functions for arbitrary state-action pairs, i.e., $Q(s,a)$. Thus, instead of calculating action-value functions for all state-action pairs in every time step, the agent relies on a DNN to estimate the action-value function for a given state-action pair. Two important ingredients are used in the DQN method to stabilize the training process, including the use of the target network and a experience replay.

Double Deep Q-Network [19] is a variant of DQN. In DQN, the target Q value is calculated based on the maximal action-value function, i.e., $ma{x}_{a^{\prime }}{\widehat{Q}}^{*}(s^{\prime },a^{\prime })$. This calculation introduces an overestimation in action-value functions and results in overoptimistic value estimates. Due to this reason, DDQN is proposed to reduce overestimations by decomposing the max operation in the target into two steps, i.e., action selection and action evaluation. In DDQN, the target is calculated as $\widehat{Q}(s^{\prime },arg{max}_{a}Q(s^{\prime },a))$. This equation contains two steps: (1) action selection: the agent chooses an action a that results in the maximal $Q(s^{\prime },a)$; (2) action evaluation: the target network $\widehat{Q}$ estimate the action-value function based on the proposed action a.

Algorithm 1 presents the DDQN training algorithm. To adapt the DDQN algorithm into the CPT problem, we select the state s as sensor measurements. Onboard sensor measurements include flow speed and direction (v, $\phi$), AUV’s speed and heading (u, $\varphi$), and chemical plume concentration $\zeta$. In choosing state variables, we convert angle-related variables, including flow direction ($\phi$) and AUV heading angles ($\varphi$) to vector forms, such as $v_x=vcos\phi$, $v_y=vsin\phi$, $u_x=ucos\varphi$, and $u_y=sin\varphi$. This is because angles do not make a good DNN input: one angle could refer to two different values, such as $-\pi$ and $\pi$, and angles should not matter if the corresponding speed is zero. Besides, we select the plume non-detection period $\delta t$ as one of the states since it can reflect whether the AUV is maintaining inside plumes ($\delta t=t-t_{last}$, where $t_{last}$ is the last plume detection time). In summary, the state s can be presented as

$$s=\{v_x,v_y,u_x,u_y,\delta t\}.$$

(1)

There are two possible actions in the action space, i.e., $\mathcal{A}=\{a_0,a_1\}$. When the action $a_0$ is activated, the AUV will select the moth-inspired method as the current CPT algorithm, and if $a_1$ is activated, the AUV will choose the Bayesian-inference method as the current CPT strategy. We define reward functions as follows:

$$R(s,a)=\left\{\begin{array}{cc}1\hfill & \mathrm{Concentration}\mathrm{above}\mathrm{threshold}\hfill \\ 0\hfill & \mathrm{Concentration}\mathrm{below}\mathrm{threshold}\hfill \\ 100\hfill & \mathrm{Find}\mathrm{the}\mathrm{source}\hfill \\ -100\hfill & \mathrm{Run}\mathrm{out}\mathrm{of}\mathrm{time}\mathrm{or}\mathrm{move}\mathrm{outside}.\hfill \end{array}\right.$$

(2)

Algorithm 1 DDQN Training Method

1: Inputs: sensor measurements $s=\{v_x,v_y,u_x,u_y,{\delta }_t\}$ 2: Outputs: fusion coefficient a 3: Initialize replay memory D to capacity N. 4: Initialize action-value function Q with random weights $\theta$. 5: Initialize target action-value function $\widehat{Q}$ with weights ${\theta }^{-}$. 6: for episode 1,2,3,…,M do 7:     for t = 1,2,3,…,T do 8:         With the probability $ϵ$ select a random action a; 9:         otherwise select $a=arg{max}_{a\in \mathcal{A}}Q(s,a;\theta )$. 10:         Execute the action a and observe reward r and $s^{\prime }$. 11:         Store transition $(s,a,r,s^{\prime })$ in D. 12:         Sample a mini-batch of L transitions ${\left\{(s_j,a_j,r_j,s_{j+1})\right\}}_{j=0}^L$ from D. 13:         Set $y_j=r_j$ if the episode ends at $j+1$; 14:         otherwise, set $y_j=r_j+\gamma \widehat{Q}(s_{j+1},arg{max}_{a}Q(s^{\prime },a;\theta );{\theta }^{-})$. 15:         Preform a gradient descent step on ${(y_j-Q(s_j,a_j;\theta ))}^2$ with respect to $\theta$. 16:         Every C steps, reset $\widehat{Q}=Q$. 17:     end for 18: end for

We consider that the AUV finds the source if the distance between the AUV and the source is less than a threshold (3 m in the simulation program). Hyper-parameters in Algorithm 1 are defined as follows: $N=\mathrm{50,000}$, $L=64$, $ϵ=0.05$, $\gamma =0.9$, and $C=20$, where N is the size of the replay buffer, L is the size of the mini-batch, $ϵ$ is the probability of choosing random actions, $\gamma$ is the discount factor, and C is the frequency of updating the target Q network. The architecture of the implemented DNN is presented in Figure 6, which is a 5-layer feedforward neural network with the ReLU activation function [54].

4.2. DDPG-Based CPT Method

The original DDPG algorithm employs the actor-critic architecture. There are two DNNs in the original DDPG algorithm, including an actor DNN and a critic DNN. The actor DNN calculates an action a (a continuous variable) based on the current state s, and the critic DNN calculates the action-value function $Q(s,a)$ to evaluate the actor’s action selection. The goal of the actor network is to choose an action that maximizes the action-value function, and the critic network aims to improve the accuracy of estimate the expected reward return. Similar to the DDQN algorithm, the DDPG algorithm also utilizes the target networks and experience replay to stabilize the training process. Specifically, both actor and critic networks have target networks. Target networks are initialized as clones of actor and critic networks and get updated gradually in a rate of $\tau$ during the training process. Transitions, i.e., $(s,a,s^{\prime },r)$, in an episode will be stored in a replay memory for repeatedly using during the training process.

Algorithm 2 presents the DDPG training algorithm. Specifically, the state s is identical to the one in the DDQN-based CPT algorithm, i.e., $s=\{v_x,v_y,u_x,u_y,\delta t\}$. The action calculated by the actor network is a scalar, ranging from 0 to 1. This scalar is the fusion coefficient that will be used to combine heading commands generated by two traditional CPT methods. Besides, the DDPG employs the same reward function as defined in the DDQN algorithm (see Equation (2)), i.e., the AUV receives positive rewards if it detects or find hydrothermal vent locations and negative rewards if it runs out of search time limit or moves out of the search area. The random noise $\mathcal{N}$ is modeled as a Gaussian white noise with zero mean and 10 variance. The purpose of adding random noise on the action allows the agent to explore more actions and states during the training.

Similar to the previous DDQN algorithm, we consider the AUV finds the hydrothermal vent if the AUV’s position is within a small region of the hydrothermal vent (3 m in the simulation program). The output of the critic network is the action-value function associated with the current state s and action a. Hyper-parameters in Algorithm 2 are defined as follows: $N=\mathrm{50,000}$, $L=64$, $\gamma =0.9$, and $\tau =0.005$. Figure 7 presents the employed actor and critic networks in the DDPG algorithm. Both DNNs contains 5 dense layers, i.e., fully connected neural networks. Specifically, inputs of the actor network is state s, defined based on AUV’s sensor measurements, and the output is an action a, i.e., a fusion coefficient that ranging from 0 to 1. The critic network’s inputs include state s and the ation a. Each input is processed by two dense layers and then concatenated together to feed into three dense layers. The output of the critic network is the action-value function $Q(s,a)$. The activation function in the actor and critic networks is ReLU.

Algorithm 2 DDPG Training Method

1: Inputs: sensor measurements $s=\{v_x,v_y,u_x,u_y,{\delta }_t\}$ 2: Outputs: fusion coefficient a 3: Initialize critic network $Q(s,a;{\theta }^Q)$ and actor network $\mu (s;{\theta }^{\mu })$ with weights ${\theta }^Q$ and ${\theta }^{\mu }$. 4: Initialize target network $\widehat{Q}$ and $\widehat{\mu }$ with weights ${\theta }^{\widehat{Q}}\leftarrow {\theta }^Q$; ${\theta }^{\widehat{\mu }}\leftarrow {\theta }^{\mu }$. 5: Initialize replay memory D to capacity N. 6: for episode = 1,2,3,…,M do 7:     Initialize a random noise $\mathcal{N}$ for exploration. 8:     Receive initial state s. 9:     for t = 1,2,3,…,T do 10:         Select an action $a=\mu (s;{\theta }^{\mu })+\mathcal{N}$. 11:         Execute the action a and observe reward r and a new state $s^{\prime }$. 12:         Store transition $(s,a,r,s^{\prime })$ in D. 13:         Sample a random mini-batch of L transitions ${\left\{(s_j,a_j,r_j,s_{j+1})\right\}}_{j=0}^L$ from D. 14:         Set $y_j=r_j$ if the episode ends at $j+1$; 15:         otherwise, set $y_j=r_j+\gamma \widehat{Q}(s_{j+1},\widehat{\mu }(s_{j+1};{\theta }^{\widehat{\mu }});{\theta }^{\widehat{Q}})$ 16:         Update the critic network by minimizing the loss: $Loss=\frac{1}{L}{\sum }_{j=0}^L{(y_j-Q(s_j,a_j;{\theta }^Q))}^2$ 17:         Update the actor network by using the sampled policy gradient: ${\nabla }_{{\theta }^{\mu }}J\approx \frac{1}{L}{\sum }_{j=0}^L{\nabla }_{a}Q(s,a;{\theta }^Q){{|}_{s=s_j,a=\mu \left(s_j\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu })|}_{s_j}$ 18:         Update the target networks: ${\theta }^{\widehat{Q}}\leftarrow {\theta }^{\widehat{Q}}(1-\tau )+{\theta }^Q\tau$ ${\theta }^{\widehat{\mu }}\leftarrow {\theta }^{\widehat{\mu }}(1-\tau )+{\theta }^{\mu }\tau$ 19:     end for 20: end for

5. Simulation and Results

In this section, we evaluate the effectiveness of the proposed CPT algorithms and compare the search results with traditional moth-inspired [17] and Bayesian-based [18] methods (without the fusion) in a simulation program.

5.1. Simulation Setup

5.1.1. Simulated Environment

The simulation program used in this work was built based on [8]. This simulation program can present graphical animation to show how chemical plumes develop and how the agent searches the chemical source in the search environment. The simulation includes a time-varying flow field, a coherent chemical plume trajectory, and a full six degree of freedom autonomous vehicle dynamics. This simulation program is also utilized by [55,56,57,58,59] to study CPT algorithms.

Figure 8 shows the search area in the simulation. The size of the search area is $100\times 100$ m${}^2$. A coordinate ($x-y$) is constructed to represent positions. Under the coordinate, the hydrothermal vent is located at $(20,0)$ m and releases 10 filament packages (i.e., plumes) per second. Grey dots in the center represent released plumes and form a curvy plume trajectory. Arrows in the background represent flow vectors, where an arrow points to the flow direction and the length of an arrow indicates the strength of flow velocity. In this simulator, flow vectors are calculated from time-varying boundary conditions that are generated by a mean flow (${\mathbf{U}}_0$) and a Gaussian white noise (zero mean and $\varsigma$ variance). By changing values of ${\mathbf{U}}_0$ and $\varsigma$, different amplitudes of airflow fields can be obtained. Figure 9 shows examples of two different flow environments: a laminar airflow environment was created with ${\mathbf{U}}_0=(1,0)$ m/s and $\varsigma =0$ in Figure 9a, and in Figure 9b, a turbulent airflow environment was generated with ${\mathbf{U}}_0=(2,0)$ m/s and $\varsigma =20$.

5.1.2. Vehicle Assumptions

In the simulation program, the vehicle dynamic is modeled based on a real-world AUV, i.e., the REMUS 100 AUV [60]. Comparing to the large scale of the search area, the size of the AUV is negligible. Therefore, the AUV is approximated as a single point in the simulation program. It is assumed that the AUV is equipped with a chemical sensor, a flow sensor, and a positioning sensor, which measure chemical plume concentrations at the AUV position, water flow speeds and directions in the inertial frame, and AUV positions in the inertial frame, respectively. To mimic the real-world applications, all sensors’ measurements are corrupted with Gaussian white noises. The noise parameters are listed in

Table 1. Gaussian Noises in the Simulated Sensor Measurements. (mmpv: million molecules per m${}^3$).

	Chemical Sensor	Flow Sensor	Positioning Sensor
Mean	0	0	0
Standard Deviation	0.05 mmpv	0.1 m/s and 1${}^{\circ }$	0.1 m

. The proposed CPT algorithms are implemented on an onboard computer to process sensor readings and calculate the heading commands, which is limited in a range of ${\varphi }_c=[-{90}^{\circ },{90}^{\circ }]$. The AUV’s velocity set as a constant to simplify the control problem, i.e., 1 m/s.

5.2. Training Results

The proposed DDQN- and DDPG-based CPT algorithms were trained in the aforementioned simulation program. Specifically, each algorithm was trained 4000 episodes. For each training episode, the AUV initial position $(x_0,y_0)$ was randomly sampled at the downflow area of the hydrothermal vent (i.e., $x_0\in [30,80]$ m and $y_0\in [-40,40]$ m), and the flow condition in the search environment varied. The training time for a single algorithm was around 4 h on a workstation computer with an Intel Core i9 CPU and Nvidia RTX 3090 GPU.

Figure 10 presents the averaged reward plots for the proposed DDQN- and DDPG-based CPT algorithms. We can see that both plots maintain at a high value after around 200 episodes, indicating the agent achieved a high success rate of finding hydrothermal vents. Besides, the averaged reward obtained by the agent trained by DDQN is higher than the DDPG counterpart. This is because the proposed DDQN-based CPT algorithm switches between moth-inspired and Bayesian-inference methods during the search, which is more instructive than the fusion of heading commands in the DDPG-based CPT algorithm. In the early training episodes, the fusion coefficient calculated by the DDPG’s actor network deteriorates the search efficiency since two traditional CPT methods may propose two opposite heading commands, and the fusion of two commands leads the AUV to an erroneous direction.

5.3. Simulation Results in Laminar Flow Environments

We first implement the proposed DDQN- and DDPG-based CPT algorithms in laminar flow environments where ${\mathbf{U}}_0=(1,0)$ m/s and $\varsigma =2$. We first present a sample search, performed by the proposed deep RL-based CPT algorithms. In the sample run, the AUV initial position is at $(60,-40)$ m, and the source location is at $(20,0)$ m.

5.3.1. Sample Run of the Proposed DDQN-Based Algorithm

Figure 11 presents the snapshots of search trajectories generated by the DDQN-based CPT algorithm. As we can see in Figure 11a, the AUV employed the ‘zigzag’ search trajectory to search plumes for the first time. Details of the ‘zigzag’ behavior can be found in [52], which can be summarized as follows: the AUV will head in a crosswind direction and bounce inward the search area when it arrives at the boundaries of the search area. At $t=48$ s, the AUV contacted plumes for the first time and switched to the proposed DDQN-based CPT algorithm. At this time, the DNN selected the Bayesian-inference method as the search algorithm. Thus, the AUV moves to the red region in the search area as presented in Figure 11b, which is the region with the highest probability of containing the hydrothermal vent. From $t=48$ s to $t=64$ s, the AUV was directed by the Bayesian-inference method to move upflow since the DNN’s output is 1. After $t=64$ s, the DNN’s output becomes $a_0$, commanding the AUV to follow the moth-inspired method to quickly approach the source location. As indicated in Figure 11d, the AUV correctly find the hydrothermal vent’s location at $t=78$ s.

5.3.2. Sample Run of the Proposed DDPG-Based Algorithm

Figure 12 presents the snapshots of search trajectories generated by the DDPG-based CPT algorithm. Similar to the DDQN-based CPT algorithm, the AUV successfully finds the hydrothermal vent’s location at the end of the search. Comparing to the DDQN counterpart, the DDPG commands the AUV to use moth-inspired CPT algorithm in the most of search time (i.e., the fusion coefficient is larger than 0.5). Specifically, the AUV first employed the ‘zigzag’ search trajectory to find the existence of plumes in the environment. At $t=45$ s, the AUV detected plumes for the first time and switched to the DDPG-based CPT algorithm. As indicated in Figure 12c, the AUV stayed inside plumes and moved upflow to approach the source location. At $t=71$ s, the AUV correctly found the source location.

5.3.3. Repeated Tests in Laminar Flows

We also implemented the proposed deep RL-based CPT algorithms in repeated tests and compared the statistic results with traditional CPT algorithms. Four CPT algorithms, including DDQN, DDPG, moth-inspired, and Bayesian-inference methods were implemented in the simulation program, and each method was repeatedly performed 100 times. In repeated tests, the AUV initial position $(x_0,y_0)$ was selected from four possible candidates, including $(60,-40)$ m, $(80,-30)$ m, $(60,40)$ m. and $(50,20)$ m. The AUV performed 25 trials at each initial position with four CPT algorithms, i.e., the proposed DDQN, DDPG, the traditional moth-inspired, and Bayesian-inference methods. The search time limit for a trial is 400 s. If the AUV cannot find the hydrothermal vent within the time limit, this trial will be considered as an unsuccessful trial in locating the hydrothermal vent.

Table 2. Statistic Results of Repeated Tests in Laminar Flow Environments.

	DDQN	DDPG	Moth-Inspired	Bayesian-Inference
Success Rate	100/100	98/100	99/100	90/100
Avg. Search Time (s)	88.5	105.8	97.7	108.8
Standard Deviation	42.6	86.9	54.2	98.3

presents the results of repeated tests.

From Table 2, we can see that the proposed DDQN-based CPT algorithm outperforms traditional moth-inspired and Bayesian-inference methods, where the success rate of the DDQN is 100 % versus 99% for the traditional moth-inspired and 90% for the Bayesian-inference method. The averaged search time for the DDQN is also shorter than two traditional methods. The proposed DDPG-based CPT algorithm achieves a better search performance than the Bayesian-inference method, where the success rate is 98% versus 90% for the Bayesian-inference method and the averaged search time is 3 s shorther than the Bayesian-inference counterpart. As for two traditional methods, we can see that the moth-inspired method achieves a shorter averaged search time than the Bayesian-inference method thanks to the effective ‘track-in’ behavior that can quickly lead the AUV to move upflow and approach the hydrothermal vent. The proposed DDQN-based CPT algorithm outperforms the moth-inspired method, indicating the effectiveness of the proposed DDQN-based CPT algorithm and the idea that combining bio-inspired and engineering-based methods benefits in improving the search performance.

5.4. Simulation Results in Turbulent Flow Environments

In this section, the proposed deep RL-based CPT algorithms will be implemented in turbulent flow environments, where ${\mathbf{U}}_0=(1,0)$ m/s and $\varsigma =10$. Similar to the previous group of tests, we first present sample runs of the proposed CPT algorithms in turbulent flow environments. The AUV initial position is at $(60,-40)$ m, and the source location is at $(20,0)$ m.

5.4.1. Sample Run of the Proposed DDQN-Based Algorithm

Figure 13 presents the snapshots of search trajectory generated by the DDQN-based CPT algorithm. Specifically, the AUV adapted the ‘zigzag’ search trajectory to find chemical plumes in the search environment and detected chemical plumes for the first time at $t=50$ s as indicated in Figure 13a. At $t=65$ s, the AUV adapted the Bayesian-inference method as commanded by the DDQN-based CPT algorithm and moved to the estimated source location (i.e., the red region in Figure 13b). At $t=113$ s, the AUV did not detect chemical plumes at the estimated source location and updated the source estimate as presented in Figure 13c. At $t=174$ s, i.e., Figure 13d, the AUV switched to the moth-inspired method and successfully detected the chemical plumes. At $t=244$ s, the AUV switched back into the Bayesian-inference method and correctly found the hydrothermal vent’s location, as presented in Figure 13e.

5.4.2. Sample Run of the Proposed DDPG-Based Algorithm

Figure 14 shows the snapshots of search trajectory produced by the DDPG-based CPT algorithm. From these diagrams, we can see that the plume trajectory (i.e., black particles) is very curvy due to the turbulent flows. Thus, it will be very inefficient if the AUV chooses the moth-inspired method since the AUV will spend a lot of time on the ‘casting’ behavior to re-detect plumes.

In this test, we observed that the DDPG’s output, i.e., the fusion coefficient, is 0 in most of the search time, indicating that the DDPG selected the Bayesian-inference method over the moth-inspired method, considering that the Bayesian-inference method is more effective than moth-inspired method in finding hydrothermal vents in turbulent flow environments. This observation indicates that the DDPG algorithm learns how to choose correct search strategy in different search environments. Specifically, the AUV detected the first plume at $t=87$ s after the ‘zigzag’ search strategy as presented in Figure 14a. From $t=87$ s to $t=117$ s, the AUV was directed by the Bayesian-inference method and detected chemical plumes several times in the nearby of the plume trajectory as indicated by Figure 14b,c. At $t=165$ s, the AUV was close to the source location thanks to the guidance of accurate source estimate as presented in Figure 14d. In Figure 14e, the AUV correctly found the source location at $t=213$ s.

5.4.3. Repeated Tests in Turbulent Flows

To verify the effectiveness and robustness of the proposed deep RL-based algorithms in turbulent flow environments, we implemented the proposed algorithms in repeated tests. Similar to the tests in laminar flow environments, the AUV started at four different initial positions, i.e., $(60,-40)$ m, $(80,-30)$ m, $(60,40)$ m, and $(50,20)$ m. Besides the proposed deep RL-based algorithms, the traditional moth-inspired and Bayesian-inference methods were also implemented in this group of tests.

Table 3. Statistic Results of Repeated Tests in Turbulent Flow Environments.

	DDQN	DDPG	Moth-Inspired	Bayesian-Inference
Success Rate	94/100	90/100	62/100	70/100
Avg. Search Time (s)	96.3	163.25	295.2	206.2
Standard Deviation	68.5	115.3	134.6	156.1

lists statistic results of the proposed deep RL-based CPT algorithms and traditional methods in repeated tests. As we can see, the DDQN-based CPT algorithm achieves the best performance in this group of tests, where the success rate is 94% in 100 tests and the averaged search time and standard deviation are 96.3 s and 68.5, respectively. The DDPG-based algorithm also outperforms traditional methods in turbulent flow environments, where the success rate is $90\%$, averaged search time is $163.25$ s, and standard deviation is 115.3. Statistic results in repeated tests show the effectiveness of the proposed deep RL-based algorithms in turbulent flow environments.

5.5. Discussion

We compared our proposed methods with an end-to-end RL algorithm proposed in [13]. According to the reported search performance (i.e., Table 1 in [13]), our proposed method achieved a comparable search performance in terms of the success rate. The highest success rate in [13] is 98.81% (the number of repeated tests is unspecified in [13]), and our double deep Q-network method achieved a 100% success rate in 100 repeated tests (see Table 2). It is also worth mentioning that our neural network structure is much simpler than the one proposed in [13], which needs less calculation time to process the input data.

We also evaluated the search performance of proposed methods in different source locations. Figure 15 shows snapshots of four trials with different source locations, including $(20,-10)$ m, $(10,20)$ m, $(40,10)$ m, and $(20,-20)$ m. We can see that the AUV successfully reached the hydrothermal vent location at the end of each trial. Simulation results reflect the effectiveness of the proposed CPT algorithms. In the future, we plan to implement the proposed CPT algorithms in real-world environments and study on more complicated search scenarios, such as searching with multiple AUVs and multiple odor sources.

6. Conclusions

In this paper, we present two deep RL-based CPT algorithms. Two deep RL algorithms, including DDQN and DDPG, are employed to design CPT algorithms for using on AUVs to locate underwater hydrothermal vents. Considering the low sample efficiency of deep RL algorithms, we employed deep RL algorithms to train DNNs that calculates a fusion coefficient to combine two traditional CPT algorithms, i.e., moth-inspired and Bayesian-inference methods. Specifically, the DDQN algorithm selects one search strategy from two traditional methods during the search process, while the DDPG algorithm calculates a fusion variable that combines two traditional methods in a superposition format. Based on simulation tests, we found that both deep RL-based algorithms are effective in finding hydrothermal vents in both laminar and turbulent flow environments. Besides, comparing to traditional CPT algorithms, i.e., without the fusion, the search performance is improved in terms of the averaged search time and success rate. In laminar flow environments, both DDQN- and DDPG-based CPT algorithms achieve a high success rate (>90%) in 100 repeated trials. In turbulent flow environments, our proposed DDQN achieved the best performance, which improves the search success rate by 67% compared to the moth-inspired method and 34% compared to the Bayesian-inference method. Simulation results indicate the effectiveness of the proposed CPT methods in both laminar and turbulent flow environments.