Energy-Efficient Driving for Adaptive Traffic Signal Control Environment via Explainable Reinforcement Learning

Abstract

Energy-efficient driving systems can effectively reduce energy consumption during vehicle operation. Most of the existing studies focus on the driving strategies in a fixed signal timing environment, whereas the standardized Signal Phase and Timing (SPaT) data can help the vehicle make the optimal decisions. However, with the development of artificial intelligence and communication techniques, the conventional fixed timing methods are gradually replaced by adaptive traffic signal control (ATSC) approaches. The previous studies utilized SPaT information that cannot be applied directly in the environment with ATSC. Thus, a framework is proposed to implement energy-efficient driving in the ATSC environment, while the ATSC is realized by the value-based reinforcement learning algorithm. After giving the optimal control model, the framework draws upon the Markov Decision Process (MDP) to make an approximation to the optimal control problem. The state sharing mechanism allows the vehicle to obtain the state information of the traffic signal agents. The reward function in MDP considers energy consumption, traffic mobility, and driving comfort. With the support of traffic simulation software SUMO, the vehicle agent is trained by Proximal Policy Optimization (PPO) algorithm, which enables the vehicle to select actions from continuous action space. The simulation results show that the energy consumption of the controlled vehicle can be reduced by 31.73%~45.90% with a different extent of mobility sacrifice compared with the manual driving model. Besides, we developed a module based on SHapley Additive exPlanations (SHAP) to explain the decision process in each timestep of the vehicle. That can make the strategy more reliable and credible.

1. Introduction

Fickle driving behaviors can easily lead to increased carbon emissions and energy consumption [1], which then becomes a source of global warming and the greenhouse effect. In order to build a eco-friendly transportation system, the application of electric vehicles and advanced driving strategies is necessary [2]. In this case, energy efficient driving, also known as “eco-driving”, is an important way to achieve energy conservation and emission reduction for vehicles [3]. It is usually implemented by avoiding sudden acceleration and reducing idling time [4], whereas frequent acceleration/deceleration, slow movements, and idling are the main factors affecting vehicular energy consumption. Generally, both freeway [5,6] and signalized intersections of urban roads are the primary scenarios for eco-driving [7,8], but the intersection scenario is more challenging because of the uncertainty of the changing signal phase and the evolving road traffic. Traffic signals and vehicle queues impose spatial and temporal constraints on the movement of the vehicle needed to be controlled (i.e., the ego vehicle) [9]. However, with the support of advanced vehicle-to-infrastructure (V2I) communication and vehicle-to-vehicle (V2V) communication, the uncertainty can be attenuated to help the vehicle make better decisions to cross the conflict area [10].

Scholars have presented many methods to realize eco-driving in the proximity of signalized intersections. More specifically, most of the existing works utilize signal phase and timing (SPaT) information so that the ego vehicle can achieve the energy-saving goal [11]. The SPaT data acquired by the ego vehicle through V2I communication includes the current phase and the phase changing time, which can guide the vehicle to adjust its acceleration to avoid a red light. Leveraging the SPaT data, the majority of previously proposed studies obtain the control law of the vehicle by formulating an optimal control problem [12,13,14]. More specifically, Asadi and Vahidi [15] reported a multi-objective adaptive cruise control (ACC) system to reduce idle time and fuel consumption by formulating an optimization-based control algorithm. Lin et al. [16] considered the powertrain dynamics of a CV and solved the eco-driving problem numerically by combining the multi-stage driving rule and the Dijkstra algorithm to get an approximated optimal trajectory. Wang et al. [17] introduced a connected and automated vehicle (CAV)-based method targeted for a signalized corridor scenario. The CAVs in their paper can receive SPaT and serve as leaders or followers in a string to achieve cooperative eco-driving. Mousa et al. [18] put forward a framework for CV passing through semi-actuated signalized intersections. They applied a simple algorithm to predict the signal changing time with the premise that the logic of signal timing is known, so the trajectory of the ego vehicle can be planned. Yang et al. [19] also proposed a SPaT-based trajectory planning approach, which was designed in a modular and scalable fashion. They carried out a series of comprehensive sensitivity analyses to explore the fuel-optimum configuration of the traffic phase and system penetrating rate. Moreover, Dong et al. [20] built a queue discharge prediction model and controlled the ego vehicle according to the queue estimation results. Dynamic programming (DP) and model predictive control (MPC) are used to design the general speed profile of the vehicle and compute the explicit control solution, respectively. Besides, some researchers extend the control framework to the platoon-based situations, which also utilized SPaT data to further reduce the energy consumption of the vehicles [21,22].

In addition to these optimization-based methods, some rule-based methods are also implemented, aiming at reducing the computation complexity. In general, the rule-based frameworks divided the running of CVs into several categories with regard to vehicular position and traffic signal status [23,24]. By acquiring the status of traffic lights, the controlled CVs know if it should speed up or slow down, judging whether the vehicle could reach the junction within the duration of the green light.

In fact, the works mentioned above are applicable to pre-timed or semi-actuated signal control environments. The traffic signal states are tractable for the CV to make decisions in these cases. Besides, some researches focus on situations with unknown traffic signal timing. For example, Mahler and Vahidi [25] developed a simple phase prediction module that uses historically-averaged data and real-time phase data. Then, the ego vehicle is controlled with probabilistic traffic signal circumstances by DP. Sun et al. [26], who formulated a random variable called effective red-light duration and appended a chance constraint to the optimal control problem. The control law is also determined by DP in their work. Nevertheless, the above-mentioned methods assume that the traffic light operates cyclically with a fixed cycle and duration. The situation with adaptive traffic signal control (ATSC) is ignored by scholars.

Meanwhile, the traditional traffic signal control methods will be gradually eliminated in some urban areas with the development and construction of smart cities. ATSC is becoming a promising way to alleviate mounting traffic congestion. As an important part of an intelligent traffic system (ITS), ATSC approaches change the signal state in real-time according to the traffic flow with the intent to reduce vehicle delay [27]. In particular, the reinforcement learning (RL) technique is one of the most up-and-coming approaches to implementing ATSC in terms of its learning-based mechanism. By combining with the deep learning method, deep reinforcement learning (DRL) shows great potential to solve decision-making problems like ATSC [28,29]. As far as the application is concerned, the DRL-based traffic signal control method is accomplished in the real world by fusing multi-source data. The application of the ATSC in Shenzhen, China, brings about more smooth traffic and less vehicle delay [30].

ATSC methods represented by DRL algorithms have two characteristics: (1) the change of the signal phase is determined by the current traffic state, where the state can be the average waiting time of vehicles, queue length, or some other numerical features; (2) the traffic lights controlled by ATSC methods do not have fixed cycles and phase plans. In this case, the standard SPaT information cannot be produced and obtained by CV, and this leads to more uncertainty regarding the control of the vehicle. More specifically, if the intelligent driving system of the vehicle cannot know when the phase would change, it is difficult to control the vehicle to cross the intersection during the green light, and this may raise the possibility that the vehicle performs a stop-and-go motion with increased energy consumption. Concurrently, considering the surrounding cars and the dynamics of the ego vehicle itself, the eco-driving task in an ATSC environment becomes a complex nonlinear optimal control problem, which is hard to be solved numerically. Therefore, the previous studies can no longer be applied to such situations. However, considering that the ATSC will replace the position of a traditional fixed timing signal control, the eco-driving strategy in such a situation is needed.

While model-based control strategies like DP and MPC are time-consuming, the model-free DRL algorithms are capable of controlling the vehicle in a real-time way with a “trial-and-error” mechanism. In DRL theory, the agent can learn a policy and carry out the action produced by the policy according to the observed state. Thus, the DRL algorithms are applied to the eco-driving domain to realize adaptive control of vehicles. In this case, Shi et al. [31] utilized Deep Q Network (DQN) to build an eco-driving system, but the action of the agent of the set is a discrete acceleration rate, since DQN cannot deal with continuous action space. Similarly, the framework proposed by Mousa et al. [32] encountered the same problem when the simplified action space leads to a local optimum solution. To address this issue, Guo et al. [33] introduced a deep deterministic policy gradient (DDPG) approach to control the ego agent with continuous acceleration. Furthermore, a twin delayed deep deterministic policy gradient (TD3) algorithm-based automated eco-driving method is reported by Wegener et al. [34], while TD3 is an improved version of DDPG. Although the adaptive control can be implemented in these DRL-based approaches, the accurate SPaT information is taken as part of the state of the agent. As these approaches have tested the effectiveness of the DRL-based control in a fixed timing situation [35,36], they cannot be directly applied in the ATSC environment. Furthermore, to the best of the authors’ knowledge, all of the previous DRL-based eco-driving system works lack explicability, which means that we cannot know why the ego vehicle accelerates or decelerates in every specific time step. It is obvious that the black-box logic may give rise to some safety-related issues and have a negative impact on reliable control.

In this paper, we focus on electric vehicles, which attract the public’s attention due to its eco-friendly feature, benefitting from the advancements of battery-related research [37,38]. We also develop a Markov Decision Process (MDP) model for the eco-driving problem in the ATSC traffic environment to fill the previous technical gap. The optimal speed profile is generated by the Proximal Policy Optimization (PPO) algorithm, which is capable of solving continuous action space problems. Meanwhile, SHapley Additive exPlanations (SHAP) [39] is used to explain the decision process of the agent (i.e., the ego vehicle) to make the DRL algorithm-based approach explainable.

The main contributions of our paper are summarized as follows:

(1)

A general MDP for eco-driving control is established, whereas the reward function of the system considers energy consumption, traffic mobility, and driving comfort;

(2)

Unlike operating in a fixed-timing traffic light environment, the system can be dedicated to a DRL-based ATSC environment, while the proposed state sharing mechanism allows the ego vehicle to take the state of the traffic light agent as part of the state itself;

(3)

Whereas conventional optimization-based methods suffer from computation complexity, a model-free DRL algorithm is adopted to generate the speed profile of the vehicle in real-time, and the SHAP approach is used to explain and visualize the decision of the ego vehicle, which is of help to reveal the black-box decision process of the DRL agent.

The remainder of this article is structured as follows. Section 2 introduces the preliminaries of DRL and DRL-based ATSC. Section “Problem formulation” provides the formulation of the eco-driving control problem with ATSC. Section 4 proposes the MDP framework and PPO algorithm with the intent to solve the problem. Section 5 reports a series of simulations carried out in the SUMO platform [40] to demonstrate the feasibility and effectiveness of the proposed method. Finally, some concluding remarks are presented in Section 6.

2. Preliminary

2.1. Deep Reinforcement Learning

Reinforcement learning is motivated by the MDP framework, which can be defined as a five-tuple $\left(S,A,R,P,\gamma \right)$. The process can be depicted by the agent-environment interface, which is shown in Figure 1. In a fully-observed MDP, the agent can observe the current state of the environment $s_t\in S$, while the environment can be a simulator or mathematical model. The agent will conduct action $a_t\in A$ and receive the reward $r_t\in R$. The actions of the agent directly influence its subsequent cumulative rewards $G_t$ for a ${\rm K}$-step control problem, which can be computed according to $G_t=r_{t+1}+\gamma r_{t+2}+\cdots +{\gamma }^{\tau -1}{r}_{t+\tau }={\displaystyle \sum }_{\tau =0}^{{\rm K}}{\gamma }^{\tau }{r}_{t+\tau +1}$. The parameter $\gamma$ is a discount factor to measure the weight of the current reward and future reward. Moreover, $P$ is a function specifying the transition probability for the agent conducting the action $a_t$ in state $s_t$.

The goal of the agent is to maximize the expected cumulative reward in each timestep $t$. Thus, an optimum policy $\pi \left(a|s\right):S\times A\to \left[0, 1\right]$ is supposed to be learned by the agent to guide the agent to select the optimal action $a_t$ with state $s_t$ to maximize its return. Following the policy $\pi$, the state-action (i.e., Q-function) can be given by:

$$Q^{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{\pi }\left({{\displaystyle \sum }}_{\tau =0}^{\infty }{\gamma }^{\tau }{r}_{t+\tau }\left(s_{t+\tau },a_{t+\tau }\right)|s_t,a_t\right).$$

(1)

In this case, an optimal policy for the agent is:

$${\pi }_{\ast }\left(s\right)=\underset{aϵA}{argmax}{Q}_{\ast }\left(s,a\right),s\in S,$$

(2)

which can generate the optimal action for the agent in each step.

However, the space of the state and action can be large, and the high-dimension data will lead to the “curse of dimensionality” in the process of learning. Deep neural networks are introduced in the RL domain to make an approximation as a result. We assume that there is a neural network with parameters $\theta$, and an approximate optimum policy ${\pi }_{\ast }\left(s,\theta \right)$ is established. Finally, the agent needs to learn in a data-driven way to update $\theta$ in order to make the equation ${\pi }_{\ast }\left(s,\theta \right)\approx {\pi }_{\ast }\left(s\right)$ come true.

2.2. Adaptive Traffic Signal Control

ATSC is a widely studied concept in the past few years. It is usually accomplished by fuzzy logic, swarm intelligence, or neural network [41]. As the traffic lights are supposed to respond to the changing traffic flow, the prevailing DRL framework brings about better solutions to the problem. In this paper, we implement a DRL-based ATSC system to demonstrate the feasibility of our framework, but it should be noted that the methodology is true for most of the state-dependent ATSC approaches.

The RL-based ATSC also follows the MDP model, and the corresponding definition is as follows:

(1)

State: The first part of the state for a traffic light is represented by the queue length ratio in each lane. We assume that the length of homogeneous vehicles is denoted as $l_v$, and the minimal gap between vehicles is $\varpi$. Then the time-dependent queue length ratio of traffic light $j$ can be defined as:

$${\Psi }_j\left(t\right)=min\left(1,\frac{n_i\left(t\right)\left(l_v+\varpi \right)}{L_i}\right),i\in I_j,$$

(3)

where, $I_j$ is the set of the lanes connected to the intersection area, while $n_i$, $L_i$ is the number of halting vehicles (i.e., the cars with a speed less than $0.1 \mathrm{m}/{\mathrm{s}}^2$) and the length of the lane, respectively.

The second part of the state for a traffic light is the one-hot encoding of its current phase with a predefined phase scheme. Let the encode vector at time $t$ be $E_j\left(t\right)$, and the state of traffic light $j$ is $s_j\left(t\right)=\left[{\Psi }_j\left(t\right),E_j\left(t\right)\right]$.

(2)

Action: The action in this paper is defined as $a=1$: switch to the next phase, and $a=0$: maintain the current phase. The action is carried out with the interval of $25$ s, while the specific definition of the signal phases is given in Figure 2. It should be noted that a 3 s yellow phase will be inserted if the phase changes.

(3)

Reward: The reward of the traffic signal agents is defined as the summation of the waiting time of vehicles on the lanes set $I_j$. The definition aims to reduce the average waiting time of vehicles to improve traffic efficiency.

We adopt the RAINBOW algorithm to control the traffic light, whereas the algorithm uses several tricks to improve the performance of the original DQN. The details of the algorithm can be found in [42]. In addition, a signalized corridor is taken as the research object, which means that the multi-agent system is built for the traffic lights. The training process is shown in Figure 3. The figure manifests the change in the total reward of the multi-traffic signal agents, and a distinct convergence trend is shown, which demonstrates the effectiveness of the training algorithm.

More details of the implementation of the DRL-based ATSC can be found in Appendix A, as it is not the core part of the presented paper.

3. Problem Formulation

The eco-driving problem can be formulated as an optimal control problem, whether in a fixed timing signal environment or an ATSC environment. The general scenario for the ego vehicle crossing a single intersection is shown in Figure 4.

Some assumptions are made in this paper for simplification without losing generality:

(1)

The ego vehicle knows its position on the lane, speed, and acceleration;

(2)

The data packets can be transmitted by the traffic light to the ego vehicle through V2I communication without delay or packet loss;

(3)

The acceleration/deceleration of the ego vehicle can be controlled accurately with the onboard control module;

(4)

The ego vehicle will not change the lane, which means that only longitudinal movement is considered.

With the intent to describe the problem, we also formulate the optimal control problem form for the task, while the control item is the acceleration of the ego vehicle.

Firstly, the main goal of the problem is to reduce the energy consumption in the whole process:

$$J_1={\displaystyle \sum }_{j=1}^m{{\displaystyle \int }}_{t_0^j}^{t_f^j}\varphi \left(v\left(t\right),a\left(t\right)\right)dt,$$

(4)

where $m$ is the total number of intersections; $t_0^j$ is the beginning time for the ego vehicle to enter the lane of intersection $j$, while $t_f^j$ refers to the time that the ego vehicle arrives at the stop line of the junction; $\varphi$ is the energy function that can produce instantaneous energy consumption of the vehicle. In this paper, the energy model for electric vehicles is embedded in SUMO with an energy brake-recovery mechanism, and the details can be found in [43]. Moreover, $v\left(t\right)$ and $a\left(t\right)$ represent the speed and acceleration of the vehicle in time $t$.

Secondly, mobility should also be considered, and we measure that by total travel time:

$$J_2={\displaystyle \sum }_{j=1}^m\frac{v_{max}\left(t_f^j-t_0^j\right)}{L_j},$$

(5)

where $v_{max}$ denotes the maximal road speed limit.

Finally, a soft constraint is imposed on the objective function, and the objective function is denoted as:

$$J\left(a\right)={\omega }_1{J}_1+{\omega }_2{J}_2+{\displaystyle \sum }_{j=1}^{m}c\left(j,t_f^j\right)\frac{1}{ϵ+p\left(j,t_f^j\right)},$$

(6)

where ${\omega }_1$ and ${\omega }_2$ are the weighting parameters; $c\left(j,t_f^j\right)$ is a boolean value: the value is set to 0 when the traffic light is green; the value is set to 1 if the vehicle is not allowed to cross the junction owing to the spatiotemporal interval produced by a traffic signal. $ϵ$ is a small number to serve as a penalty item and $p\left(j,t_f^j\right)$ measures the green probability for traffic light $j$ in time $t_f^j$. This soft constraint encourages the vehicle to pass the road crossing without halting in front of the stop line because of the red light, and the energy can be saved in this case.

Therefore, let $d_j\left(t\right)$ be the lane position of the vehicle approaching traffic light $j$, and $x\left(t\right)={\left[d_j\left(t\right), v\left(t\right)\right]}^T$. The whole control problem is formulated as:

$$minJ\left(a\right).$$

(7)

Subject to:

$$\dot{x}\left(t\right)={\left[v\left(t\right),a\left(t\right)\right]}^T,t\in \left[t_0^1,t_f^m\right],$$

(8)

$$0\le v\left(t\right)\le v_{max},t\in \left[t_0^1,t_f^m\right],$$

(9)

$$d_{max}\le a\left(t\right)\le a_{max},t\in \left[t_0^1,t_f^m\right],$$

(10)

$$a\left(t\right)\le a^{CF}\left(t\right),t\in \left[t_0^1,t_f^m\right],$$

(11)

$$x\left(t_0^1\right)=x_0,x\left(t_f^m\right)=x_f,$$

(12)

$$d_j\left(t_f^{j-1}\right)+D_j=d\left(t_0^j\right),i\in \left[2,m\right].$$

(13)

Equations $\left(9\right) \mathrm{and}\left(10\right)$ denote the dynamics constraints of the ego vehicle. $\left(11\right)$ denotes the safety consideration, while $a^{CF}\left(t\right)$ is the value of acceleration generated by the car-following model. The acceleration of the ego vehicle generated by the control model cannot go beyond the car-following model-based value to ensure that the ego vehicle will not collide with the potential leader cars. $\left(12\right)$ denotes the start state and the end state. In fact, the end speed can be any value. Finally, $\left(13\right)$ denotes the distance connection between multi road segments, while $D_j$ is the cover distance of the ego vehicle travel in inside junction $j$.

Obviously, we need to establish a function to fit $p\left(j,t_f^j\right)$ in real-time during the process of control. Then, some methods like DP can be applied to get the policy. However, it is difficult to finish that because the accurate prediction cannot be easily achieved in an ATSC environment. Meanwhile, the control requires high real-time performance in which the model-based DP cannot satisfy. Thus, we try to accomplish the adaptive optimal control through the DRL technique instead of some brute force ways.

4. Methodology

As shown in Figure 5, the sensed traffic state is taken as the input of traffic signal agents in an ATSC environment. Since the accurate change rule of the traffic signal phase is not clear, the vehicle agent is supposed to understand the dynamics of the traffic signal agents. From this point of view, the input state of the traffic signals can be part of the state of vehicle agents, utilizing the V2I communication to transmit the information. In addition, some other related states are collected to form the decision basis of the ego vehicle. The comprehensive design of the state vector of the CV, which is enhanced by the state sharing mechanism between the heterogeneous agents, is promising to help the CV catch the regular pattern of the adaptive traffic signals.

4.1. MDP Model

Similar to the RL-based ATSC, the MDP model is a prerequisite for applying the RL algorithm. Based on the assumptions we made in the previous section, we can specify the MDP according to the optimal control problem we build to make a mapping. The state, action, and reward of the MDP is defined as follows:

(1)

State: The ego vehicle selects action in terms of the state, so the definition of the state is crucial. The state of this problem should include its dynamics, the state of its surrounding vehicles, and the state of the traffic lights. Therefore, a vector with multi-elements is built to serve as the state of the ego vehicle:

$$s^v\left(t\right)={\left[d^s\left(t\right),v\left(t\right),\Delta d\left(t\right),\Delta v\left(t\right),\Delta a\left(t\right),{\xi }_j,t_j^c,s_j\left(t\right)\right]}^T,$$

(14)

where $d^s\left(t\right)$ measures the distance from the vehicle to the stop line, whereas the agent needs to know how far to enter the intersection; $\Delta d\left(t\right),\Delta v\left(t\right),$ and $\Delta a\left(t\right)$ denotes the distance difference, speed difference, and acceleration difference between the potential leading vehicle and the ego vehicle. This may help the ego vehicle keep a safe distance between itself and the front vehicle or prevent itself from entering the tail of a queue. More precisely, the front vehicle can be the tail of the queue, so the ego vehicle may enter the queue and keep the stop-go movement, which may increase the energy consumption significantly. However, a leading vehicle does not always exist, so we set the judgment threshold $\rho =300 \mathrm{m}$ to measure the relative distance between the front vehicle and the ego one. Let the absolute position of the front vehicle be $d^f\left(t\right)$, and the value of the ego vehicle be $d^e\left(t\right)$, then we have:

$$\Delta d=\left\{\begin{array}{c}{d}^f\left(t\right)-d^e\left(t\right), if d^f\left(t\right)-d^e\left(t\right)<\varrho \\ \rho , \mathrm{otherwise}\end{array}\right.,$$

(15)

when $d^f\left(t\right)-d^e\left(t\right)<\rho$, the value of $\Delta v\left(t\right)$ and $\Delta a\left(t\right)$ can be calculated in the same way. Otherwise, the two values will be set to the default values $\Delta v_d$ and $\Delta a_d$.

Moreover, ${\xi }_j$ in $\left(9\right)$ is a boolean value that represents whether the lane where the vehicle is located is opened with the green light in the current phase; $t_j^c$ denotes the time to act for the first downstream traffic signal agent. Finally, $s_j\left(t\right)$ is the state of traffic light $j$ (i.e., the traffic light can share its state with the ego vehicle through V2I communication). The three groups of the parameters here consist of the state related to the traffic signal agent, which may help the vehicle agent understand the performance of the traffic lights.

This state definition applies regardless of whether the vehicle is on a road segment or inside an intersection. Hence, the framework can be used for a multi-intersection scenario naturally.

(2)

Action: The action is consistent with the control item in the optimal control problem, that is, the acceleration of the ego vehicle. The action space should satisfy Equation (10) to generate an effective value. Meanwhile, the speed of the agent should also satisfy $\left(9\right)$, which means that the speed change of the vehicle is denoted as:

$$v\left(t\right)=max\left(min\left(v_{max},v\left(t-1\right)+{\widehat{a}}_t\right),0\right),$$

(16)

where $a_t$ is the output of the DRL algorithm. However, the value $a_t$ cannot be taken as the final acceleration to impose on the vehicle because $\left(11\right)$ should also be accomplished. Accordingly, the amended acceleration is:

$$a_t=min\left({\widehat{a}}_t,a^{CF}\left(t\right)\right),$$

(17)

where the $a^{CF}\left(t\right)$ is determined by IDM [44] in this paper, and the parameters of the car-following model are set to default values in SUMO to enhance traffic safety.

(3)

Reward: The definition of the reward function should be similar to the cost function $\left(6\right)$ to get the same control effect. However, the travel time item in $\left(6\right)$ is a Meyer item, which can only be calculated at the end of the process, in other words, the value can be obtained only if the ego vehicle reaches the stop line. In our RL framework, it is difficult to allocate the credit to every step because the contribution to the travel time of each step is hard to be measured. Thus, we take the travel distance in each step to replace that value, which is denoted as $l^d\left(t\right)$. Besides, we add jerk, which is defined as the differential of the acceleration, to the reward function with the intent to improve the driving comfort. The reward function is denoted as:

$$r^v\left(t\right)=-{\omega }_1\varphi \left(v\left(t\right),a\left(t\right)\right)+{\omega }_2{l}^d\left(t\right)-{\omega }_3\dot{a}\left(t\right)-\delta \frac{1}{ϵ},$$

(18)

where $\delta$ is a boolean flag to identify whether the ego vehicle is halting. In this case, a penalty value $\delta \frac{1}{ϵ}$ will be added if the vehicle encounters a red light and wait. Meanwhile, this item can encourage the agent not to join a queue. It should be noted that ${\omega }_1\varphi \left(v\left(t\right),a\left(t\right)\right)$ can be a negative value owing to the recycling mechanism of the braking energy of the electric vehicles. The reward function considers energy saving, traffic mobility, and comfort simultaneously, which is practical and economical.

4.2. Proximal Policy Optimization

PPO is an on-policy reinforcement learning algorithm, which can be used for problems with continuous action spaces [45]. Generally, the on-policy interaction takes the return in each step to estimate the long-term return ${\displaystyle \sum }_{\tau =0}^{{\rm K}}{\gamma }^{\tau }{r}_{t+\tau +1}$, and this will lead to large variance. Accordingly, some studies use an actor-critic framework with an advantage function to estimate the return, but it will increase the bias. Therefore, the paper applies generalized advantage estimation (GAE) to optimize the advantage function to get a trade-off between variance and bias.

The movement of the ego vehicle can be optimized when the parametric policy $\pi \left({\theta }^p\right)$ approaches the optimal policy. In this case, the main goal is to find the optimal parameter set ${\theta }_{\ast }^p$ of the neural network. That means the objective function should be maximized:

$$R\left({\theta }^p\right)={\mathbb{E}}_{\eta ~{\mathcal{D}}_k}\left[{{\displaystyle \sum }}_{\tau =0}^{\infty }{\gamma }^{\tau }{r}_{t+\tau }\left(s_{t+\tau },a_{t+\tau }\right)\right],$$

(19)

in which the ${\mathcal{D}}_k=\left\{{\eta }_i\right\}$ is a set of transitions collected from the interaction between the agent and the environment. The transition sequence ${\eta }_i$ herein refers to a set consisting of $\left\{s_1,a_1,r_2,s_2,a_2,r_3,\dots \right\}$, and a collection of transitions can be used in training.

The training process is shown in Algorithm 1. PPO makes use of a value function approximation (VFA) $V_{{\theta }^v}$, which also takes an artificial neural network (ANN) with a parameter set to ${\theta }^v$ to estimate the expected total reward in the future. Over each epoch $k=1,2,3,\dots K$, the parameters of the neural networks can be updated. The parameter $\theta$ is learned in the following way:

$${\theta }_{k+1}=arg\underset{\theta }{max}\underset{s,a~{\pi }_{{\theta }_k}}{E}\left[L\left(s,a,{\theta }_i,\theta \right)\right],$$

(20)

where $L\left(s,a,{\theta }_k,\theta \right)$ is given by:

$$L\left(s,a,{\theta }_k,\theta \right)=min\left({\beta }_k\left(\theta \right)A^{{\pi }_{{\theta }_k}}\left(s,a\right),g\left(\epsilon ,A^{{\pi }_{{\theta }_k}}\left(s_t,a_t\right)\right)\right),$$

(21)

where the definition of ${\beta }_k\left(\theta \right)$ and $g\left(\epsilon ,A^{{\pi }_{{\theta }_k}}\left(s_t,a_t\right)\right)$ is given by Equations (22) and (23):

$${\beta }_k\left(\theta \right)=\frac{{\pi }_{\theta }\left(a|s\right)}{{\pi }_{{\theta }_k}\left(a|s\right)},$$

(22)

$$g=\mathrm{clip}\left(\frac{{\pi }_{\theta }\left(a|s\right)}{{\pi }_{{\theta }_k}\left(a|s\right)},1-\epsilon ,1+\epsilon \right)A^{{\pi }_{{\theta }_k}}\left(s,a\right).$$

(23)

The parameter $\epsilon$ is preset to stipulate an interval $\left[1-\epsilon ,1+\epsilon \right]$ that defines the upper and lower bound of ${\beta }_k\left(\theta \right)$. The purpose of (23) is to restrict the search of ${\theta }_{k+1}$ within a predefined range that is close to ${\theta }_k$. The notation $A^{{\pi }_{{\theta }_k}}\in ℝ$ is the value produced by an advantage function reflecting the value of a selected action in accordance with current policy ${\pi }_{\theta }{}_k$ concerning other feasible decisions [46]. Moreover, the clip operation output $\left(1+\epsilon \right)A^{{\pi }_{{\theta }_k}}$ if $A^{{\pi }_{{\theta }_k}}\ge 0$, otherwise e $\left(1-\epsilon \right)A^{{\pi }_{{\theta }_k}}$ will be exported. In this paper, GAE is taken to calculate $A^{{\pi }_{{\theta }_k}}$:

$$A^{{\pi }_{{\theta }_k}}={\lambda }_k+\left(\gamma \chi \right){\lambda }_{k+1}+\cdots +\left(\gamma \chi \right){\lambda }_{K-1},$$

(24)

where the discount factor $\gamma \in \left[0,1\right]$ and the GAE parameter $\chi \in \left[0,1\right]$ are used to balance the estimated biases and variances. The notation ${\lambda }_k$ is the one-step temporal difference (TD), which is defined as:

$${\lambda }_k=\left\{\begin{array}{c}{r}_{k+1}-V_{{\theta }^v}\left(s_t\right), if k=K-1\\ r_{k+1}+\gamma V_{{\theta }^v}\left(s_{t+1}\right)-V_{{\theta }^v}\left(s_t\right), otherwise\end{array}\right..$$

(25)

During the training process, both ${\theta }^p$ and ${\theta }^v$ can be updated iteratively through the Adam optimization algorithm [47].

Algorithm 1 PPO for eco-driving training
1	Initialize policy parameters ${\theta }^p$ $\mathrm{and}\mathrm{value}\mathrm{function}\mathrm{parameters}{\theta }^v$ with random values
2	$\mathit{f}\mathit{o}\mathit{r}$ k $=1,2,3\dots k$ $\mathit{d}\mathit{o}$
3	Carry out the preloading process and insert the ego vehicle into the road network
4	Control the vehicle according to policy $\pi \left({\theta }^p\right)$ and collect the set of trajectories ${\mathcal{D}}_k=\left\{{\eta }_i\right\}$
5	Compute reward-to-go value $G_t$
6	Compute advantage estimates ${\widehat{A}}_t$ (optimized by GAE) based on the current value function $V_{{\theta }_v}$.
7	Update the policy by maximizing the objective via Adam optimizer:       ${\theta }_{k+1}^p=arg\underset{{\theta }^p}{max}\frac{1}{\left|{\mathcal{D}}_k\right|T}{\displaystyle {\displaystyle \sum }_{{\mathcal{D}}_k}}{\displaystyle {\displaystyle \sum }_{t=0}^T}min\left(\frac{{\pi }_{{\theta }^p}\left(a_t|s_t\right)}{{\pi }_{{\theta }_k^p}\left(a_t|s_t\right)}{A}^{{\pi }_{{\theta }_k^p}}\left(s_t,a_t\right),g\left(\epsilon ,A^{{\pi }_{{\theta }_k^p}}\left(s_t,a_t\right)\right)\right)$
8	Fit value function by regression to minimize mean-squared error via Adam optimizer:       ${\theta }_{k+1}^v=arg\underset{{\theta }^v}{min}\frac{1}{\left|{\mathcal{D}}_k\right|T}{\displaystyle {\displaystyle \sum }_{{\mathcal{D}}_k}}{\displaystyle {\displaystyle \sum }_{t=0}^T}{\left(V_{{\theta }^v}\left(s_t\right)-R_t\right)}^2$
9	$\mathit{e}\mathit{n}\mathit{d} \mathit{f}\mathit{o}\mathit{r}$

4.3. SHapley Additive exPlanations

We might want to know why the vehicle accelerates or decelerates under the driving of the PPO algorithm. In other words, the agent observes the state that consists of some elements. Some of the elements may spur the vehicle to accelerate, while others may slow down the vehicle. Meanwhile, the attribution of the features (i.e., the element in the state vector) is different for the agent to make decisions. Considering safety and reliability, we should build an explanation module to describe the behavior of the agent.

It is known that feature attribution is a common approach to analyzing a trained machine learning model, whereas the RL is a branch of machine learning. Conventionally, the machine learning community takes tree-based models as one kind of the interpretable methods [48]. Nevertheless, this paper deploys a DRL model, which can only be explained in a post-hoc manner. We assume that $F$ is a machine learning model that maps some input $x=\left(x_1,\dots ,x_n\right)\in R^n$ to the target values. The feature attribution of the prediction at $x$ relative to a baseline input $x_b$ can be denoted with a vector $A_F\left(x,x_b\right)=\left(f{a}_1,f{a}_2,\dots f{a}_n\right)\in R^n$ where $f{a}_i$ represents the contribution of $x_i$ to the prediction $F\left(x_i\right)$. In this case, the Shapley value becomes an effective method to distribute the total gains of a cooperative game to a coalition of cooperating players [49]. More specifically, taking a coalitional game with $n$ players and a function $F$ that maps the subset of players to the real numbers $f:2^N\to R$ as an example, the attribution amount of player $i$ to the game is:

$$A_f\left(i\right)={{\displaystyle \sum }}_{C\subseteq N\backslash \left\{i\right\}}\frac{\left|C\right|!\left(n-\left|C\right|-1\right)!}{n!}\left(f\left(C\cup \left\{i\right\}\right)-f\left(C\right)\right),$$

(26)

where $f\left(C\right)$ is the worth of coalition $C$, illustrating the total expected sum of payoffs the members of $C$ can derive from the cooperation. The gist of the equation is: the difference between value function with and without player $i$ is computed for each coalition $C$ without player $i$, so the contribution of player $i$ is equal to the weighted difference $\frac{\left|C\right|!\left(n-\left|C\right|-1\right)!}{n!}$. For the RL-based eco-driving problem, we formulate a game for the control state at each timestep. The “total gains” denote the acceleration of the vehicle, while the “players” is the set of the state in the MDP model. The Shapely-value-based approach tends to approximate Shapley values by examining the effect of removing an element under all possible combinations of the presence or absence of the other elements in the state vector.

However, computing the Shapely value for every feature is still challenging. On the other hand, LIME (Local Interpretable Model-Agnostic Explanations) [50] is proposed to be an additive feature attribution method that learns an explainable model like a decision tree or linear regression model $w\left(z^{\prime }\right)$ to approximate the target black-box model locally, where $z^{\prime }$ is a simplified binary input vector. Then the simplified features are transformed into the original algebraic space so that the corresponding target value is computed by $w\left(z^{\prime }\right)=f\left(h_x\left(z^{\prime }\right)\right)$, where $h_x$ is a function that maps the simplified input to the original input. The Kernel SHAP method draws lessons from LIME and computes the weight for each element in $z^{\prime }$. A SHAP value estimates the change in predicted output, from the base value $E\left[f\left(x\right)\right]$ to $f\left(x\right)$, attribute to every single feature $i$. More precisely, the SHAP value can be $E\left[f\left(z^{\prime }\right)|z_1^{\prime }=x_1\right]-E\left[f\left(z^{\prime }\right)\right]$ for feature with $i=1$.

5. Simulation Analysis

5.1. Simulation Configuration

The simulations are carried out on the SUMO platform. As an open-source software, SUMO provides higher privileges to do further development based on its framework. Meanwhile, SUMO can interact with Python programs through the Traci interface, whereas our algorithm and training procedure are coded in Python.

A signalized corridor scenario is established in SUMO, which can be shown in Figure 6. The lane configuration for each intersection is the same as Figure 4. For each episode of train or evaluation, the traffic will be loaded in the first $100$ s with a random arrival rate. Then the ego vehicle is inserted into the leftmost lane. The preloading procedure is to simulate the real traffic situation, that is, the ego vehicle can be influenced by some other vehicles. After that, the ego vehicle will cross the subsequent five signalized intersections with the support of the PPO algorithm. An episode of simulation will be terminated when the ego vehicle has crossed the last junction. Moreover, the transitions of each episode of simulation will be recorded and saved to prepare for the explanation process based on SHAP.

Table 1. General parameter settings for simulations.

Symbol	Description	Value
$v_{max}$	Road speed limit	$13.89\mathrm{m}/\mathrm{s}$
$a_{max}$	Maximal acceleration of vehicles	$3.0\mathrm{m}/{\mathrm{s}}^2$
$d_{max}$	Maximal deceleration of vehicles	$-4.5\mathrm{m}/{\mathrm{s}}^2$
$\Delta v_d$	Default value for speed difference	$-14\mathrm{m}/\mathrm{s}$
$\Delta a_d$	Default value for acceleration difference	$-7.2\mathrm{m}/{\mathrm{s}}^2$
$ϵ$	The value corresponds to the penalty item in Equation (18)	$0.01$
$\chi$	The GAE parameter	$1.0$
$-$	Learning rate of the Adam optimizer	$0.0001$
${\epsilon }_1$	Clip parameter for policy	$0.3$
${\epsilon }_2$	Clip parameter for value function	$10.0$

presents the general parameter settings for simulations. The parameters related to PPO are tuned manually through a series of simulations. We use a fully connected neural network architecture for both parametric policy and parametric value function. The units of the neural networks with three hidden layers are $\left[512,256,64\right]$. In addition, the default settings of weighting parameters ${\omega }_1$, ${\omega }_2$, and ${\omega }_3$ are set to 5, 3, and 7. The sensitivity analysis of these parameters is presented after the general analysis.

5.2. Simulation and Discussion

The simulations are built upon the environment described in the last subsection. We introduce the RAINBOW algorithm to be the baseline and make comparisons with PPO. The action space of the RAINBOW agent is set to a 16-dimension vector because the value-based algorithm can only be applied to problems with continuous action space. We divide the acceleration of the ego vehicle between $-4.5\mathrm{m}/{\mathrm{s}}^2$ and $3.0\mathrm{m}/{\mathrm{s}}^2$ in steps of $0.5\mathrm{m}/{\mathrm{s}}^2$; the agent selects an action from the 16-dimension action space, and we will see that the optimal control cannot be implemented in this discrete-action-space way.

The scatters in Figure 7 show the data point of different metrics during the training processes of PPO and RAINBOW. In addition, the line plots displayed above the scatters are the linear regression result of the corresponding data. It can be found that the energy consumption of the RAINBOW agent cannot be reduced significantly, despite the reward increases. Compared with RAINBOW, the energy consumption of the PPO agent converges to a nominal value, with extra sacrifice in terms of the travel time. The training profiles illuminate that the proposed PPO-based framework is capable of searching for a solution that enables the vehicle to run in a more energy-saving way.

After the training procedure, the well-trained agents are utilized to evaluate their performance. The results are collected from 20 independent simulations, and the random seed is set to the same value for the same round simulation of different control methods. Figure 8 illustrates the evaluation results of PPO, RAINBOW, and IDM, whereas the IDM can reflect the general performance of human drivers. According to Figure 8a, we can see that the RL-based approaches will reduce the mobility of the ego vehicle. However, PPO can save electric energy to a great extent compared with IDM, which can be seen in Figure 8b. On the other hand, although the RAINBOW sacrifices traffic mobility, the energy consumption still remains at a high level. The result proves that the DQN-based algorithm cannot deal with the eco-driving problem in an ATSC environment. Moreover, the mean jerk is improved slightly under the control of PPO compared with the situation of IDM, and this can be caused by some braking behaviors with the intent to recover the energy. Thus, the overall comfort of driving under RL-based control may decrease slightly. On average, PPO can save $41.06\%$ energy consumption compared with manual driving (i.e., IDM) in an ATSC environment.

Figure 9 presents the results in terms of electricity consumption for 20 groups of independent simulation-based evaluations in detail. The figure demonstrates that the proposed method lowers the consumed energy steadily compared with IDM, which is deemed as representative of normal human driving behavior. By contrast, the results produced by RAINBOW can be even worse than the baseline in most cases, further proving that the value-based algorithm with discrete action space cannot fit into the environment with high uncertainty.

We elicit the explain module in the methodology section because we are interested in the decision process of the agent. Here, we can analyze the feature attribution of the element in the state vector. Two samples in the evaluation results are selected randomly; the SHAP value is calculated for each simulation step. The trajectories of the ego vehicle are plotted in Figure 10 and Figure 11, and then we mark several trajectory parts with black square boxes. The SHAP force plot is provided for the boxes by using the tool developed by Ref. [51]. The features in Figure 10 and Figure 11 corresponds to the state definition $s^v\left(t\right)$: “dist2stop” corresponds to $d^s\left(t\right)$; “is_green” corresponds to $\xi$; “time_to_change_phase” represents $t^c$; the features started with “obs_ts” are part of the state of the traffic signal agent $s_j\left(t\right)$. The red part of the SHAP charts means that the corresponding features drive the vehicle to speed up, while the blue part drives the vehicle to slow down.

We can analyze the trajectories depicted in Figure 10 and Figure 11. It can be found that the agent may stop at the proximity of the intersection due to the uncertainty caused by the queue and the traffic light. This implies that the optimal control law cannot be given by the proposed algorithm. However, electrical energy can be saved significantly even in such an uncertain situation. The agent can make effective decisions through the use of the obtained information. For example, the upper box in Figure 10 manifests a slow-down decision of the ego vehicle. In this case, the feature $d^s\left(t\right)$ encourages the agent to accelerate, but the state of the traffic light lets the agent know that the traffic light will not turn green soon. Hence, the agent chooses to slow down to avoid the red light. The same process is true of the lower box in Figure 10; the traffic signal has 5 s to switch, and the ego vehicle judges that the green light will not continue according to the traffic light state. Therefore, the agent chooses to accelerate on the premise that there is no car ahead.

Taking Figure 11 as an example, more quantitative analysis can be conducted. The “base value” in the SHAP chart denotes the mean value of the acceleration in the whole episode of the simulation. The acceleration of the upper box in Figure 11 is $0.86 \mathrm{m}/{\mathrm{s}}^2$, which is the result of the incentive effect of the features. Similarly, almost all of the features prevent the agent from accelerating for the situation in the lower box, so the action of the agent is $-1.01 \mathrm{m}/{\mathrm{s}}^2$. Meanwhile, the base values in the sampled trajectories are small. It means that although the trajectory of the vehicle is not so smooth, the acceleration and deceleration range of the vehicle is relatively small. This can help the vehicle save energy and improve comfort.

5.3. Sensitivity Analysis

The sensitivity analysis is carried out by exploring the effect of the weighting parameters ${\omega }_1$, ${\omega }_2$, and ${\omega }_3$. Considering the absolute numerical value of each part of the reward, the alternative values of each parameter is set to $\left[1,3,5\right]$, $\left[1,2,3\right]$, and $\left[4,7,10\right]$. More precisely, a grid-based search can be programmed for the 27 situations. For each situation, an agent is trained in 300 episodes, and then 20 rounds of simulation are conducted to evaluate the performance of the agent.

Table 2. Results for different weighting parameters settings.

${\mathit{\omega }}_{\mathbf{1}}$	${\mathit{\omega }}_{\mathbf{2}}$	${\mathit{\omega }}_{\mathbf{3}}$	Energy Consumption (Wh)	Travel Time (s)	Mean Jerk (m/s3)
1	1	4	77.47	650.10	0.54
1	1	7	72.32	644.10	0.56
1	1	10	80.59	567.30	0.47
1	2	4	72.56	623.05	0.53
1	2	7	76.70	593.80	0.73
1	2	10	81.18	583.15	0.68
1	3	4	67.43	554.30	0.66
1	3	7	66.90	536.65	0.50
1	3	10	76.64	440.50	0.60
3	1	4	75.83	607.30	0.62
3	1	7	75.43	650.10	0.54
3	1	10	77.36	695.20	0.66
3	2	4	71.13	631.85	0.58
3	2	7	74.73	673.65	0.76
3	2	10	72.75	625.95	0.64
3	3	4	74.57	530.10	0.56
3	3	7	68.50	545.30	0.57
3	3	10	72.76	573.85	0.61
5	1	4	69.43	651.95	0.47
5	1	7	69.67	667.70	0.61
5	1	10	63.87	609.95	0.63
5	2	4	70.96	533.25	0.53
5	2	7	69.58	536.65	0.50
5	2	10	68.56	554.75	0.37
5	3	4	66.47	595.10	0.40
5	3	7	66.05	563.95	0.36
5	3	10	72.76	476.65	0.61

shows the metrics performance for different combinations of the parameters. It can be seen that an increase of ${\omega }_1$ and ${\omega }_2$ can improve the corresponding indicators to a certain extent in general, while the mean jerk is not sensitive to the change of ${\omega }_3$. Nevertheless, the increase of ${\omega }_2$ does not necessarily reduce the travel time on some occasions. We infer that the weighting parameter with a high value encourages the agent to choose a more aggressive and greedy strategy, and this kind of strategy may be limited by the uncertain environment. The same is true of ${\omega }_1$. Besides, the mutual restriction of the three parameters will also exert an unexpected influence on the vehicle. Therefore, a compromising scheme is selected in this study. Concurrently, this provides plasticity for the control law. We can choose the parameter configuration with low travel time to improve the traffic efficiency if the travel is sensitive to the travel time. Similarly, we can also choose the configuration with low energy consumption to save electricity to the greatest extent. From Table 2, we can conclude that the energy consumption can be reduced by $31.73\%~45.90\%$, while the sacrifice of travel time is $25\%~98\%$.

6. Conclusions

In this paper, we propose a reinforcement learning framework to implement eco-driving control for electric vehicles in the ATSC environment. The eco-driving problem is converted to an MDP because it is difficult to solve the optimal control problem directly. The MDP has been carefully designed: the state of the vehicle includes not only its dynamic parameters but also the shared state of the signal agents, while the state sharing mechanism can help the ego vehicle make decisions in an uncertain environment. The reward function considers mobility, energy consumption, and comfort to enhance the driving strategy. Besides, an explanation module is established to explain the decision process of the ego vehicle, which can make the black-box model-free method clear and intuitive. The simulations are carried out on a signalized corridor built in SUMO. Being compared with RAINBOW and IDM, the proposed PPO algorithm gives a much better energy-saving performance. Despite the traffic mobility being reduced, this method can be used in circumstances where the requirement of travel time is not high, or energy is extremely scarce. Moreover, the SHAP value explicates the decision basis of the ego vehicle well, and this can provide reliability and safety for the actual circumstances.

It should be pointed out that the proposed strategy Is still feasible for gasoline vehicles. Although we implement a RAINBOW-based ATSC, the whole framework can be applied to the ATSC environments controlled by some other algorithms.

This paper only considered an ideal V2I communication situation, but packet loss and communication delay are common in real world, so it is practical to study these conditions in future searches. Meanwhile, we only put forward the strategy for a single vehicle, while the corresponding methodology can be extended to platoon control, and this may be implemented in future research. Another research gap is that the interaction between vehicles with eco-driving systems and normal human-driven vehicles is not clear. Since the operation of eco-driving vehicles may have a direct impact on the holistic traffic flow, it is of value to explore the topic in a dedicated study. In addition, some data can be collected from a real-world vehicle operation system and used for training agents in an offline manner with offline DRL technique, which is helpful to formulate more applicable strategies to surmount the simulation-to-real transfer gap.