Applications of Agent-Based Methods in Multi-Energy Systems—A Systematic Literature Review

Abstract

The need for a greener and more sustainable energy system evokes a need for more extensive energy system transition research. The penetration of distributed energy resources and Internet of Things technologies facilitate energy system transition towards the next generation of energy system concepts. The next generation of energy system concepts include “integrated energy system”, “multi-energy system”, or “smart energy system”. These concepts reveal that future energy systems can integrate multiple energy carriers with autonomous intelligent decision making. There are noticeable trends in using the agent-based method in research of energy systems, including multi-energy system transition simulation with agent-based modeling (ABM) and multi-energy system management with multi-agent system (MAS) modeling. The need for a comprehensive review of the applications of the agent-based method motivates this review article. Thus, this article aims to systematically review the ABM and MAS applications in multi-energy systems with publications from 2007 to the end of 2021. The articles were sorted into MAS and ABM applications based on the details of agent implementations. MAS application papers in building energy systems, district energy systems, and regional energy systems are reviewed with regard to energy carriers, agent control architecture, optimization algorithms, and agent development environments. ABM application papers in behavior simulation and policy-making are reviewed with regard to the agent decision-making details and model objectives. In addition, the potential future research directions in reinforcement learning implementation and agent control synchronization are highlighted. The review shows that the agent-based method has great potential to contribute to energy transition studies with its plug-and-play ability and distributed decision-making process.

1. Introduction

In the context of global warming and fossil fuel depletion, energy system transitions play an important role in future sustainable societies [1]. Thus, energy systems are experiencing fundamental evolutions, including transitioning to multi-energy carrier integration and Internet of Things (IoT) penetrations. From the perspective of the organizational structure, planning and operations of energy sub-sectors are usually performed in an independent nature with consideration to only simple interactions [2]. In fact, technological advancements in energy generation, conversion, and storage lead to strong interactions with multiple energy sub-sectors. Some examples of such technologies include combined heat and power (CHP) [3] or heat pumps (HP) [4,5]. The increasingly tight interactions among energy subsectors sparked a novel method of integrating multiple energy sectors for more comprehensive planning and operation of energy systems. Such energy systems have also been termed as “multi-energy systems” in some studies [6,7]. A multi-energy system (MES) represents a single holistic system integrated with multi-energy-carrier production, conversion, and consumption. Figure 1 illustrates an example of a typical MES with key multi-energy technologies, including distributed generation units, CHP, and HP. In addition, the term “integrated energy system” is also noted in some research [8,9,10] to describe an energy system with multiple energy carriers.

In comparison to traditional single energy carrier systems, MESs have three advantages [7]: (1) encouraging renewable energy sources (RES) consumption with energy conversion and storage ability, where excess energy generated by renewable resources could be stored not only in electricity but also in other energy carriers; (2) accommodating centralized and distributed energy sources (DER) through effective market interactions; and (3) increasing system flexibility and resilience as it does not rely on a single energy carrier, with the ability to shift to other energy carriers to meet the demand in the case of an energy emergency.

Apart from integrating multiple energy carriers, there is also a trend of enabling the intelligence of energy systems by deploying IoT technologies. Subsequently, such energy systems are named “smart energy systems” [11,12]. IoT technologies enable fast and bidirectional communication between energy system components, such as DER producers, consumers, and storage units [13]. As a result, IoT technology penetration encourages the decentralization of the decision-making process [14]. In this context, agent-based methods, including multi-agent system (MAS) modeling and agent-based modeling (ABM), are essential modeling frameworks for decision-making decentralization, as both MAS and ABM utilize the concept of decentralized intelligent agents. The core concepts of intelligent agents are discussed in Section 2.

There are existing articles published on the MAS applications in micro-grid control, such as Tanjimuddin et al. [15], Priyadarshana et al. [16], and Coelho et al. [17], as well as ABM application in the socio-technical transition of energy systems by Hansen et al. [18]. Nevertheless, there is a gap in comprehensively examining both MAS and ABM applications from the viewpoint of MES transitions. Thus, this gap motivates the need for a systematic literature review. This article is particularly interested in the current applications of agent-based methods in the energy transition process toward a multi-energy system. This review has adopted a systematic literature review method, which utilizes predefined protocols to systematically search literature in predefined databases with the help of predefined inclusion and exclusion criteria [19,20,21]. The review question to be answered include:

• What is an intelligent agent?
• What are the state-of-the-art applications of agent-based methods in MES transitions?
• What are the sources, temporal, and thematic distributions of current research?
• What are the key research topics?
• How do agents make decisions?
• How did current studies implement agent-based decision making?
• What are future research directions?

To answer the above four review questions, this review will be organized into several sections. Section 2 will discuss the core concepts of intelligent agents to answer question 1. Section 3 will illustrate the systematic literature review methodology, including the inclusion and exclusion criteria protocols. Section 4 and Section 5 will demonstrate literature distribution in journal source, publication date, and thematic distribution to answer question 2 to 5. Section 6 will evaluate the agent-based methods in MES research to answer question 6. Section 7 will discuss future research trends to answer question 7. Finally, Section 8 will offer a summary and concluding remarks.

2. Background on Intelligent Agent Concepts

The intelligent agent concept emerged from artificial intelligence research [22,23,24]. In the book, Artificial Intelligence: A Modern Approach, Russel and Norvig [23] (p. 38, third edition) outlined the definition of an agent: “An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators”. The environment is where an agent exists, containing information external to the agent. The agent is able to observe the environment (perceiving) and alter the environment (acting). The reasoning process after an agent perceives the environmental information is the key part of artificial intelligence creation. Russell and Norvig named such a process “rationality”. A rational agent should be able to maximize its performance by selecting the appropriate actions based on its knowledge. Similarly, another valuable publication in agent theory is by Wooldridge and Jennings [22], where an intelligent agent possesses the following four attributes:

(1)

Reactivity: agents are able to collect external environmental information or datasets and execute timely responses to environmental changes with predefined actions.

(2)

Autonomy: agents can operate on their own without direct human intervention.

(3)

Pro-activeness: agents have an objective function or goal to guide their actions. Such objective-oriented behavior indicates the dynamic behavior of an intelligent agent to fulfill the objective in a dynamic environment.

(4)

Social ability: agents are able to communicate with other agents through agent communication languages such as Knowledge Query and Manipulation Language [25] and Foundation for Intelligent Physical Agent—Agent Communication Language [26].

For the sake of consistency, this review has adopted the definition of intelligent agents by Wooldridge and Jennings [22] to describe the agents with rigorous reasoning ability. The structure of such intelligence is shown in Figure 2. An intelligent agent comprises of six modules: a data collection module, a goal module, a knowledge base module, a communication module, a decision-making module, and an implementation module. The decision-making module is the core module, as it is responsible for processing the information provided by the data collection module (information from the environment), the knowledge-based module (prior precept information), and the communication module (information from other agents).

The definition of a multi-agent system is straightforward: a system that consists of multiple intelligent agents. The intelligent agents are distributed in the environment, working on their individual objectives with predefined behavior programs. The distributed nature of an MAS reveals its potential to solve complex problems by dividing global problems into local problems. Intelligent agents will communicate with other agents and use their respective knowledge bases to solve local problems. The division process will result in a lower-cost solution than a centralized power entity solving the complex global problem [27].

The distributed nature of MASs drew the attention of practitioners and researchers in a wide range of disciplines. For example, MASs have been used in cloud computing [28,29,30,31] and robotics [32,33,34]. Considering MAS applications in power systems, the IEEE Power Engineering Society’s Intelligent System Subcommittee published a two-part publication that evaluated the potential of MAS applications in power systems and outlined the general guidance of MAS implementation [35,36] in the first part. In the second part, McArthur et al. emphasized the importance of the Foundation for Intelligent Physical Agents (FIPA) specification [37], as it could enhance MAS interoperability with other systems or technological tools [38].

Similar to MASs, agent-based modeling (ABM) also utilizes the concept of the intelligent agent with the decision-making ability to simulate the system. Bale, Varga, and Foxon [39] examined energy systems through the lens of complex adaptive system (CAS) theories and highlighted the great potential of using ABM in energy system research. Another review which was written by Hansen, Liu, and Morrison [18] investigated the socio-technical interaction with ABM in the energy transition process. The literature mentioned above has demonstrated the great value of ABM in energy system research. In the literature, the term “agent-based modeling” and “multi-agent system” are sometimes used interchangeably, as shown in reference [40]. Ringler, Keles, and Fichtner [41] discussed the terminologies of agent-based methods in their review on ABM in smart electricity grids and markets. They highlighted that ABM research focuses on understanding the interactions within a CAS and the emergence pattern of the CAS. Therefore, the term “agent-based modeling” and “multi-agent system” cannot be considered interchangeable in this review article.

3. Review Methodology

This review has evaluated articles on multi-energy system research using an agent-based approach, including MASs and ABM. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) review protocol has been followed, with inclusion of a checklist and a flow diagram [42]. Figure 3 shows the identification, screening, and inclusion process of this review.

3.1. Literature Identification

A preliminary review of related literature determined keywords used in the search query. The search query had the form:

Boolean expression: A and {B and C or D}

where A, B, and C are search terms shown in

Table 1. Search query keywords.

A	B	C	D
Modeling method	Primary energy carrier	Other energy carrier	System description
agent-based	electricity	Heat *	multi-energy
multi-agent	power	gas	integrated energy

“Heat *” indicate that it is a wildcard word for searching. Scopus and WoS search tool will return records with the word “heat” and “heating”.

.

The Boolean expression was applied to Scopus and Web of Science (WoS), as these two databases are comprehensive and multi-disciplinary. The search request was performed until the end of 2021 on both database websites. The query was conducted in the TITLE-ABS-KEY search in Scopus and Topic Search (TS) in WoS, where records that match the search query in title, abstract, and keywords were returned. The article search returned 807 records, as shown in

Table 2. Articles search results.

	Web of Science	Scopus
Search results	353	642
Records after removing duplicates	590

. Records with data entries authors, title, year, source, publisher, article URL, type, and abstract were exported to EXCEL. After removing duplicate records, there were 590 records that were passed to screening.

3.2. Literature Screening

The first step of screening removed conference proceedings, conference reviews, letters, and book chapters, leaving peer-reviewed studies in the review database. The second step was to remove records not written in English. The inclusion and exclusion criteria are shown in

Table 3. Inclusion and exclusion criteria for records type and language.

	Inclusion	Exclusion
Records type	Peer-reviewed journal articles	Conference review, conference proceedings, letter, book chapter
Language	English	Non-English

In the next step, an eligibility check is done. A total of 4 records were removed due to paywalls, and 173 records were removed due to being off-topic after full-text review—the inclusion and exclusion criteria in off-topic screening is shown in

Table 4. Inclusion and exclusion criteria for topic screening.

	Inclusion	Exclusion
Topic screening	subject: multi-energy system or integrated energy system method: multi-agent methodology	subject: energy systems with only one energy carrier methods: methods other than the multi-agent methodology

. Six studies were added to the review database as they were identified in the references of review database articles. The final database contained 80 articles.

4. Article Source and Temporal Distributions

Section 4 reports the source and temporal distributions of the selected articles. The sources revealed the key journals that could be considered publication outlets for future research. The temporal distribution demonstrated the development of this research topic.

4.1. Article Source Distribution

The selected 80 studies were sourced from 36 journals, as shown in

Table 5. Selected article source distribution.

	2007	08	09	10		11	12	13	14	15	16	17	18	19	20	21	Total
AIMS Energy														1			1
Applied Energy								1			2	2	3	1	1		10
Applied Science (Switzerland)											1						1
Applied Thermal Engineering														2			2
Building and Environment										1							1
Building Services Engineering Research and Technology														1			1
Building Simulation											1				1	1	3
Computers and Chemical Engineering							1										1
Computers Environment and Urban Systems								1									1
Ecological Economics									1								1
Electric Power Components and Systems									1								1
Electric Power Systems Research												1			1		2
Energies												1		2	3		6
Energy														1	1	1	3
Energy and Buildings											1		1			1	3
Energy Conversion and Management									1	1					1		3
Energy Policy	1			1													2
Environmental Impact Assessment Review										1							1
Futures									1								1
IEEE Journal of Emerging and Selected Topics in Power Electronics													1				1
IEEE Transactions on Control Systems Technology																1	1
IEEE Transactions on Industrial Informatics																1	1
IEEE Transactions on Industry Applications								1									1
IEEE Transactions on Smart Grid																1	1
IEEE Transactions on Sustainable Energy										1					1		2
IET Generation, Transmission and Distribution													1				1
IET SMART GRID															1		1
International Journal of Electrical Power and Energy Systems													1	1	1	3	6
Journal of Cleaner Production															2	1	3
Journal of Physical Agents			1														1
Proceedings of the IEEE						1											1
Renewable and Sustainable Energy Reviews										1		2			1		4
Renewable Energy									1								1
Sensors													1				1
Sustainability (Switzerland)															2		2
Sustainable Cities and Society												1					1

. The major publication source was Applied Energy (10), Energies (6), the International Journal of Electrical Power & Energy Systems (6), and Renewable and Sustainable Energy Reviews (4). The reminder of the journals were sources for fewer than three selected studies.

4.2. Article Temporal Distribution

The earliest article was from Jackson in 2007 [43] in Energy Policy. Figure 4 shows a noticeable trend in the publication of ABM and MAS applications in MESs, where the number of publications has increased significantly in recent years. Due to the recent advancements in IoT technologies, the distributed system is gaining popularity among researchers. MAS is a valuable method for distributed energy system management. The years 2020 and 2021 were notable for the agent-based method publications, with 16 and 20 articles published, respectively.

5. Article Thematic Distribution

The energy system is a hierarchical system, where an energy system could be part of a bigger energy system [44]. In the system hierarchy definition of Skyttner [45], parts assemble into units that form components. The components further create a sub-system which is a part of a system that is a portion of a macro-system. Figure 5 shows the hierarchical nature of energy systems. The components, such as space heating and lighting, build a system (building energy system) that is a part of a macro-system (a district energy system). To extend this concept further, the district energy system could be a part of a larger macro-system (a regional energy system). In this review, system boundaries are based on spatial boundaries. The building energy system contains a single building, such as a residential or commercial building, where the system boundaries stop at the building’s exterior walls. The boundary of a district energy system or regional energy system stops at the administrative district or regional boundaries. The MAS can be applied to manage every level of the MES, such as home energy system automation, micro-grid design and scheduling, and coordination of regional multi-energy networks. The difference is that the agents represent different system components. Thus, the studies have been categorized based on the level of the model’s spatial resolution, as it is systematic to report study themes and model characteristics.

As discussed in Section 2, the agent-based methods considered in this review include MAS and ABM. To reiterate, although both MAS and ABM research utilize intelligent agent concepts, the respective research objectives are different. MAS research aims to realizing distributed decision making. ABM research aims to study the emergence pattern of complex adaptive systems. Therefore, the selected studies were allocated to one of the two main categories: MAS applications or ABM applications, based on how authors implement the agent-based method. There are two sub-categories for ABM: (1) behavior simulation and (2) policymaking; and three sub-categories for MASs: (1) building energy system management, (2) district energy system management, and (3) regional energy system management. Figure 6 shows the theme classification process, in which each selected study should be categorized with one of the sub-category topics. For instance, if a study uses MAS to investigate the multi-energy micro-grid control, this study should be allocated to multi-agent system applications (main category)—district energy management (sub-category). Figure 7 shows the temporal distribution of the themes from 2007 to 2021. MAS-based energy system management in all three spatial resolution levels is gaining popularity among the academic community.

The review articles in the database are shown in

Table 6. Review articles in the database.

Source	Year	Review Focus
Labeodan et al. [38]	2015	MAS applications in the building energy management system
Ahmad et al. [46]	2016	Computational intelligence in HVAC systems
Howell et al. [47]	2017	Energy system transitions and multi-agent management
Khan and Wang [48]	2017	MAS control and optimization in micro-grids
Coelho et al. [17]	2017	MAS applications in micro-grids
Vázquez-Canteli and Nagy [49]	2019	Reinforcement learning applications in demand response program
Priyadarshana et al. [16]	2019	MAS applications in micro-grids
Ma et al. [50]	2019	The MAS application of ontologies in the energy system

. These review articles focus on MAS applications in specific energy system spatial resolutions, such as building energy systems (HVAC systems) or district energy systems (micro-grid systems). Thus, there is a need for a comprehensive review of agent-based methods, including both MASs and ABM, in various energy system spatial resolutions.

5.1. Multi-Agent System Applications

Section 5.1 is organized as follows: Section 5.1.1 reviews the applications of MASs in building energy system management; Section 5.1.2 reviews applications of MASs in district energy systems management; Section 5.1.3 reviews the applications of MASs in regional system management. The term management is used as an umbrella term to describe the systems’ design, plan, and control of the systems.

5.1.1. Building Energy System Management

The building energy system is a multi-energy system, as occupants demand electricity, heating, and cooling services. The key components of a building energy system usually include appliance components (such as washing machines, dishwashers, and refrigerators), HVAC components, and a gas boiler for domestic hot water usage. The smart building or smart home concepts promote IoT adoption for intelligent control of distributed energy resources and household appliances.

The earliest published article on building energy system management was by Conte et al. [51] in 2009. Conte et al. proposed a framework for the cognitive construction of agents in the building energy management system. In 2013, Zhao, Suryanarayanan, and Simoes [52] designed a MAS control strategy with three agents: an electricity agent, a heating agent, and a cooling agent responsible for optimizing each energy carrier service, respectively. The decision making of the heating agent and the cooling agent is based on energy cost minimization with the responding energy carrier. The decision making of electricity agents includes two objectives: minimizing the peak load and communicating with grids (electricity grid and natural gas grid) for price updating. The agents’ optimizations are achieved by the CAPLEX solver [53]. Wang et al. [54] proposed a MAS-based control method for a building energy system with a combined cooling, heating, and power (CCHP) system. The CCHP system contains thermal energy storage (chilled water storage tank) to balance supply and demand mismatches. Ahrens, Kern, and Schmeck [55] adopted the MAS-based building energy management system to improve the resilience of the distribution grid by utilizing the multi-energy flexibility offered by smart home appliances.

Demand response (DR) or demand-side management (DSM) is a popular research field in energy system management. DR or DSM is a management strategy to manage the demand load based on the availability of the electricity supply. The selected articles show that an MAS is a valuable approach to implementing the DR or DSM. Devia, Agbossou, and Cardenas [56] adopted distributed co-evolutionary optimization algorithms with agent-based architectures to reduce consumption profiles. The control strategy is divided into two phases. The first phase is to extract and store energy use information in every room along with temperature variance. The second phase is to control every heating device in the house with distributed optimization. Other than cost reduction, the results also show that the inclusion of thermal storage units could increase the overall system’s efficiency. Vanhoudt et al. [57] conducted a lab test on the MAS-controlled residential heat pump. This lab test aimed to examine the performance of heat pumps on peak load shaving and self-consumption of renewable energy. The lab test results showed that the peak load was reduced by 2% to 5%. However, due to more installation of switching, the electricity consumption increased by 8% to 12%. In addition, the MAS control strategy did not encourage renewable energy consumption, which requires further investigation. Franceschelli, Pilloni, and Gasparri [58] proposed a MAS control architecture DSM on thermostatically controlled loads (TCLs). The MAS enabled anonymous communication of TCL agents via a network to retain the privacy of TCL internal temperatures. The MAS optimization was based on a dynamic consensus algorithm.

The HVAC control strategy is one of the key research areas in building energy system control. In the selected articles, a number of distributed control strategies were proposed based on MASs. Yu et al. [59] developed a multi-agent deep reinforcement learning HVAC control system for multi-zone commercial buildings to control the total energy cost with consideration for random zone occupancy, thermal comfort, and indoor air quality. This energy cost control problem was reformulated to a Markov problem which was solved by multi-agent deep reinforcement learning with a multi-actor–attention–critic approach. Yu et al. remarked that the simulation with real-world data demonstrated this algorithm’s effectiveness, robustness, and scalability. González-Briones et al. [60] proposed a multi-agent building temperature management system to increase energy usage efficiency. On the physical side, the authors deployed temperature monitoring and occupancy data sensors with wireless sensor networks to gather data for system optimization. On the logic side, the multi-agent system processes collected data and returns an optimized HVAC system’s control strategy. The case study showed that such a control strategy could achieve an average energy savings of 41%. Cai et al. [61] proposed a general multi-agent control approach for the HVAC system. This approach consists of an agent definition framework and control optimization procedure. The agents are required to define a collection of objective functions and constraints based on the framework guidelines. The framework then formulates the optimization problem based on the consensus algorithms, including the sub-gradient method and alternating direction multiplier method (ADMM) [62]. Cai et al. [61] also pointed out a potential issue in implementing the consensus-based algorithm, in which the intermediate iterations could be con-consensus. Thus, the algorithm could not reach a consensus point when the decision time had been reached. Li, Li, and Wang [63] developed a multi-agent control scheme for HVAC system on an IoT-based wireless sensor network. This scheme balances the HVAC optimization performance and power consumption of battery-powered sensors by solving the multi-objective optimization problem with ADMM. The power consumption of the sensors is reduced by using the event-driven control approach rather than the time-driven optimization approach.

5.1.2. District Energy System Management

The district energy system consists of distributed energy generation, energy conversion, and energy storage systems of a district or a community (building complexes, universities) [7]. For example, the multi-energy micro-grid [64,65,66] is a type of district energy system that has gained increasing attention from researchers.

• District energy system design

The studies in district energy system design focus on the design optimization of district energy systems. In 2013, Kyriakarakos et al. [67] developed an energy management system for optimal component sizing of poly-generation micro-grids, which can meet the consumers’ needs in remote areas with potable water, hydrogen, space heating, space cooling, and electricity. Multi-agent-based DSM is embedded in the management system for load shedding when generation capacities cannot meet the demands. The MAS is designed in a hierarchical manner. The components in a building, such as lighting and refrigeration, are controlled by the intelligent agents which are supervised by an upper-level building control agent. The component control agents are responsible for disconnecting the virtual power lines when load shedding is required. In 2015, Karavas et al. [68] further developed the agent-based poly-generation micro-grid management system from a hierarchical architecture to a decentralized architecture. Decentralization means that the optimization does not require a supervisory agent for central optimization. The component agents communicate with each other to update the system parameters, such as surplus power, consumed power, or remaining capacity (battery agent). The energy management system aims to minimize net present cost (optimal design) and optimize fuzzy cognitive map weights (optimal control). The system optimization results compared with a centralized management system, showing that the decentralized approach presented a 2% lower net present cost than the centralized approach. Later in 2017, Karavas, Arvanitis, and Papadakis [69] investigated the multi-agent decentralized management of the poly-generation micro-grid with game theory. The previous decentralized management system is a cooperative case where agents work cooperatively to minimize the global cost function. However, the agents could have conflicting interests, so agents interact with each other in a non-cooperative way. Thus, Karavas, Arvanitis, and Papadakis used a non-cooperative game model with Nash equilibrium [70] to simulate the competitiveness between electrolyze agent and desalination agent as both of them compete to consume more power. Another cooperative game is modeled with fuel cell and battery agents since these two agents aim to meet corporations’ load demand. Compared to the previous purely cooperative management system, the results showed that the game theory-based approach lowered the cost by 1.62%.

Wang et al. [71] proposed a game theory-based optimal component sizing method for the multi-energy district energy system. In the district energy system configuration, Wang et al. used compressed air energy storage (CAES) for electricity and thermal storage instead of using batteries and thermal tanks. Each component agent’s utility function is defined by the net present value function in both cooperative games and non-cooperative games. The Shapley value [72] and Nash equilibrium were used to solve the cooperative and non-cooperative game models, respectively. The results showed that coalition formation results in better economic outcomes for individual agents and the system. Jin et al. [73] proposed a game theory-based component optimization method for a multi-energy micro-grid. The main contribution of the proposed method is the incorporation of uncertainties in the renewable energy generation with a probability density function. Moreover, the utility function of agents was set as annualized economic profit during the whole life cycle. It has also been noted that future research could focus on incomplete information games when information asymmetry occurs.

• Fully decentralized district energy system control

The fully decentralized control approach does not require a central agent for optimization or management. The optimization work is achieved by each agent solving their objective function. In 2015, Harb et al. [74] introduced a decentralized day-ahead scheduling strategy for a multi-energy micro-grid for cost minimization with Mixed-Integer Linear Programming (MILP). The global MILP problem is decomposed into a series of local problems with Dantzig–Wolfe decomposition [75]. The local problems are, in fact, the local objective function for each agent, which are solved by the iterative column generation procedure [76]. After comparing the decentralized control with a centralized scheduling approach, it has been showed that the centralized approach offers a better solution than the decentralized approach. Nonetheless, the centralized approach computation time increases exponentially with the increasing number of agents. Thus, the decentralized approach has distinctive advantages in system scalability as the decentralization reduces the computation time required. Blaauwbroek et al. [77] proposed a multi-agent-based decentralized algorithm with mixed-integer quadratic programming (MIQP) to balance the distributed energy resources and flexible appliances, such as HP and CHP. Li et al. [78] proposed a decentralized control method for coupled heat and power systems. This control method involves two first-order consensus protocols [79] for heat supply optimization and electricity supply optimization. The optimization is carried out in an alternating iterative way so that electricity supply and heat supply converge to optimal solution alternatingly. The decentralized method results in a better solution than the centralized method with Langrangian relaxation [80].

Nguyen and Ishihara [81] proposed decentralized management for household clusters with fuel cells (FCs) and CHP with peer-to-peer trading architecture. The non-convexity problem of fuel cell operation is addressed by linearization of the FC consumption and production. As a result, the peer-to-peer trading problem could be solved with the ADMM algorithm. Alishavandi and Moghaddas-Tafreshi [82] presented a decentralized operation strategy for multi-energy micro-grids with interactive clearing energy prices. The clearing price is determined by agent communication between generation agents and consumption agents. In each time step, the clearing price is used in calculating each agent’s profit function and the system’s social welfare. In addition, the gradient projection method [83] was used for profit function and social welfare optimization. The results showed that this decentralized method had slightly better performance in cost reduction than the centralized method as PV agents tended to maximize their own profit. However, social welfare decreased when an individual agent’s profit increased. Shabani and Moghaddas-Tafreshi [84] presented a similar decentralized approach with interactive clearing prices to optimize a multi-energy system micro-grid. The agents were programmed to optimize the social welfare at the first stage, then to optimize their own profit. The results agreed with Ref. [82] in that the fully decentralized approach had lower social welfare but higher individual profits than the centralized approach. Shabani and Moghaddas-Tafreshi [84] also observed that the demand response program was able to increase both system social welfare and agent profit. Moreover, an investigation of the peer-to-peer trading scheme in a decentralized model was suggested for future research.

Samadi, Badri, and Ebrahimpour [85] proposed a decentralized management strategy with reinforcement learning (RL). The optimal agent behavior policy was evaluated with the action–value function (Q-function) [86,87]. The agents are able to find the optimal behavior by interacting with each other in a competitive environment. The trade-off between exploration and exploitation is an important consideration for an agent to determine the best action. Thus, Samadi, Badri, and Ebrahimpour [85] compared three action selection methods, including soft-max, epsilon-greedy, and upper confidence bound methods [87]. They showed that the soft-max method had the best performance over the other methods. Kumari and Tanwar [88] extended the Q-leaning-based RL management on micro-grids with blockchain communication. The communication and agreement among the stakeholders are achieved based on blockchain encrypted smart contracts to ensure privacy. Instead of using the discrete Q-learning method, Dong et al. [89] proposed multi-energy micro-grid management with the asynchronous advantage actor–critic algorithm based on deep RL, which was first introduced by Mnih et al. [90]. The result shows that the asynchronous algorithm could shorten the training time by 30% compared with the deep Q network.

• Multi-energy micro-grid management

The multi-energy micro-grid management is a trendy research field where researchers investigate the optimal management strategies. Anvari-Moghaddam et al. [91] proposed an optimal control scheme for a building-integrated micro-grid with distributed generation units and demand response. Micro-CHP, building energy storage, and radiant floor heating/cooling systems are the constituents of building energy systems. A mixed objective function was formulated to optimize the energy operation cost and convenience level by building agents. At the micro-grid level, a central grid battery is responsible for compensating for the energy mismatch. The central battery agent was optimized with the Bayesian reinforcement learning (BRL) algorithm [92]. The proposed BRL method was compared with Q learning [93] and time-based reinforcement learning [94]. The results showed that the proposed BRL method leads to faster learning and higher reward than the other methods. Kolen et al. [95] also proposed a control scheme with the building and micro-grid combined bi-level optimization. The building level optimization aims to minimize the total number of switch events. The operation functions (peak-to-valley distance) on the grid level are optimized with decentralized agent interactions by updating the local energy fluctuation function. The optimization work is carried out by the CPLEX optimization studio from IBM [53]. The performance of the proposed bi-level scheme is a trade-off between building-level optimization and grid-level optimization due to the narrowed search space. Hutty, Dong, and Brown [96] investigated the feasibility of reversible solid oxide cells in the microgrid operation with multi-agent simulation in the UK and Texas. The authors suggested that, in future research, simulated battery and reversible solid oxide cells as a hybrid energy storage system could be considered and also how hybrid storage systems perform compared to traditional battery storage systems could be investigated.

Moghaddas-Tafreshi et al. [97] proposed a multi-energy micro-grid optimal operation scheme considering uncertainties in renewable energy generation and energy demand, as well as a demand response program. The uncertainties in renewable energy generation were modeled with a Weibull distribution [98] of wind speed in wind turbine agents. The load agents simulated uncertainties of the electrical and thermal load with the normal distribution function. Both wind turbine and load agents generate 1000 scenarios in a Monte Carlo simulation each hour to evaluate the micro-grid performance. The demand response program from [99] is implemented in the load agent program. Li et al. [100] established a three-layer control model for microgrid management with the improved particle swarm optimization algorithm. Li et al. combined the adaptive weight and chaotic search into the PSO algorithm to avoid the optimal local solution. The results showed that the proposed algorithm had a much smaller computation standard deviation than the original PSO and chaotic search PSO. Liu, Li, and Ge [101] proposed a hierarchical control scheme for a multi-energy micro-grid considering the multi-agent game. In the first layer (decision layer), the generation agents participate in a static cooperative game with complete information to maximize their own profit. The Nash equilibrium point of the cooperative game is solved by the evolutionary game theory combined Q-learning method [102]. Khan, Wang, and Xiong [103] proposed a hierarchical multi-agent control architecture on the multi-energy micro-grid with three layers. In formulating the optimization problem, harmful gas emissions and multi-energy generation costs were considered in the objective function. Farinis and Kanellos [104] proposed a micro-grid management system with a building system operator with consideration of plug-in EVs as energy storage components.

Two of the selected articles investigated the operation strategy of an energy hub. Lin et al. [105] proposed a multi-agent energy hub operation control considering EV penetration rate and EV behavior simulation. EV agents simulated the travel patterns and charging patterns, including the uncontrolled, rapid patterns. The behavior simulation result then was passed to the energy hub control system as electricity demands. The vehicle-to-grid (V2G) technology was considered in operation optimization by adding the V2G cost function to the global optimization function. The results showed that the demand brought by increased EV penetration could be met with gas turbines. Moreover, the electricity and cooling prices were lower because of the V2G technology in the reference case. Zeng et al. [106] proposed an optimal dispatch scheme of an energy hub considering the integrated demand response program. The demand response program is achieved by designing the user agents’ objective function, which minimizes the cost of electric, thermal, cooling load, and EV cost. The EV charging and discharging behaviors are considered with a random variable to indicate the charging state of the EV. The optimal dispatch of the energy hub was formulated into a multi-objective optimization problem which includes minimization of user cost, maximization of generator profit, and maximization of operating income. Zeng et al. used the non-dominated sorting genetic algorithm-III as elaborated in Ref. [107], to find the Pareto frontier. The optimal solution of each agent in the Pareto frontier was obtained with the technology for order preference a, similar to the ideal solution method [108].

• District energy management with district heating system and heating clusters

The following two paragraphs cover the selected articles which focused on studying the district management with district heating systems and clusters of TCLs. In 2017, Haque et al. [109] proposed a unified multi-agent control strategy to manage the district electricity distribution network congestion and voltage limit with PVs and HPs. Bünning et al. [110] proposed a distributed control method for bidirectional low-temperature networks (BLTN). BLTN is a new district heating and cooling network concept that promises more network efficiency. The temperature set point is optimized by the simplex Nelder-Mead method [111] in Python. The consumer and producer agents are distributed in the network, tracking the local set point. Based on the difference between the optimized set point, the agents calculate the local cost functions and make proposals to the markets. The centralized broker evaluates all the proposals and chooses the most cost-efficient combination to implement. The results showed that the BLTN with the proposed control method could reduce energy consumption by more than 50% in comparison to conventional district heating systems.

Claessens et al. [112] proposed an optimization control approach for TCL-connected district heating systems. This approach combined reinforcement learning and a market-based multi-agent system. After aggregating the TCL cluster state information, the TCLs select the optimal action under the action policy with the Fitted Q-Iteration batch reinforcement learning algorithm [113]. Then, the optimized actions were dispatched to the cluster of TCLs with a market-based multi-agent system. Claessens et al. [112] highlighted a future research direction to investigate autonomous feature extraction techniques. Behboodi [114] introduced a transactive load control scheme for TCLs in real-time retail market energy prices. Each TCL is aggregated with an agent to bid in the retail market based on temperature and anticipated energy price state information. This scheme requires less accuracy in price forecasting than a demand profile scheme since the proposed scheme only requires the mean and volatility of energy price in a specific time window. The summary of the district energy management is shown in

Table 7. Summary of characteristics of district energy system studies.

Author	Year	Energy Carrier						Storage Type	Focused Topic	Approach	Platform
		Electricity	Heating	Cooling	Gas	Hydrogen	EV
District level design
Kyriakarakos et al. [67]	2013	✓	✓	✓		✓		Electric storage and hydrogen storage	Optimal design of multi-energy micro-grid with demand-side management	PSO	TRNSYS, MATLAB, and GenOpt
Karavas et al. [68]	2015	✓				✓		Electric storage and hydrogen storage	Decentralized energy management and component sizing of multi-energy micro-grid	PSO	TRNSYS, MATLAB and GenOpt
Karavas et al. [69]	2017	✓				✓		Electric storage and hydrogen storage	Game theory-based multi-energy micro-grid optimal component sizing	PSO	TRNSYS, MATLAB and GenOpt
Wang et al. [71]	2021	✓	✓					Electric storage and thermal storage (CAES)	Game theory-based capacity optimization of multi-energy district system with CAES	PSO	Not mentioned
Jin et al. [73]	2021	✓			✓			None	Game theory-based component optimization method for multi-energy micro-grid	PSO	Not mentioned
Decentralized control
Blaauwbroek et al. [77]	2015	✓	✓		✓			Electric storage and thermal storage	Decentralized multi-energy microgrid control	Mixed-integer quadratic programming	JADE, MATLAB
Harb et al. [74]	2015	✓	✓					Thermal storage	Decentralized control of multi-energy micro-grid	Gurobi optimizer	JADE
Li et al. [78]	2016	✓	✓					Electric storage and thermal storage	Decentralized control of electricity and heating coupled system	Consensus theory	Not mentioned
Alishavandi and Moghaddas-Tafreshi [82]	2019	✓	✓	✓	✓			Electric storage and thermal storage	Decentralized multi-energy micro-grid management for cost and emission minimization	Gradient projection	Anylogic
Shabani and Moghaddas-Tafreshi [84]	2019	✓	✓			✓		Electric storage, thermal storage, and hydrogen storage	Fully decentralized multi-energy micro-grid control with an interactive clearing price	Gradient projection algorithm	Anylogic
Samadi et al. [85]	2020	✓	✓					Electric storage and thermal storage	The decentralized control of multi-energy micro-grid with reinforcement learning	Q-learning	MATLAB
Nguyen and Ishihara [81]	2021	✓	✓		✓	✓		Hydrogen storage	Distributed P2P trading with fuel cells	ADMM	MATLAB
Kumari and Tanwar [88]	2021	✓	✓	✓	✓			None	Multi-energy micro-grid management with blockchain-based communication	Q-Learning	Not mentioned
Dong et al. [89]	2021	✓	✓		✓			Electric storage, thermal storage, and gas storage	Multi-energy micro-grid optimization	Asynchronous advantage actor-critic algorithm	Open AI
Micro-grid management
Anvari-Moghaddam et al. [91]	2017	✓	✓					Electric storage and thermal storage	Optimal management of building integrated micro-grid	BRL	JADE and MATLAB
Kolen et al. [95]	2017	✓	✓		✓			Thermal storage	Decentralized control for clusters of electro-thermal heating devices for switch event and peak-to-valley distance optimization	CPLEX	MESCOS
Yang et al. [115]	2018	✓	✓					Thermal storage	Optimal dispatch of CHP units	Newton–Raphson	MATLAB
Lin et al. [105]	2018	✓	✓	✓	✓		✓	Electric storage	EV impact on EH management	Taboo search	Anylogic
Moghaddas-Tafreshi et al. [97]	2019	✓	✓		✓	✓		Electric storage, thermal storage, and hydrogen storage	Multi-energy micro-grid optimization	PSO	MATLAB
Zeng et al. [106]	2019	✓	✓	✓				Electric storage and thermal storage	Optimal dispatch scheme of an energy hub with integrated demand response	NSGA-III	Not mentioned
Li et al. [100]	2020	✓	✓					Electric storage and thermal storage	Multi-energy micro-grid optimization	Chaotic search PSO	JADE and MATLAB
Liu et al. [101]	2020	✓	✓	✓	✓		✓	Electric storage and thermal storage	Hierarchical control of multi-energy micro-grid with RL	Q-learning	Not mentioned
Hutty et al. [96]	2020	✓				✓		Hydrogen storage	Feasibility study with reversible solid oxide cells	Greedy algorithm	Anylogic
Khan et al. [103]	2021	✓	✓		✓	✓	✓	Electric Storage	Multi-energy micro-grid optimization	Generalized pattern search algorithm	JADE and MATLAB
Farinis and Kanellos [104]	2021	✓	✓	✓			✓	Electric storage and EV	Multi-energy micro-grid optimization	PSO	Not mentioned
With heating network and clusters
Haque et al. [109]	2017	✓	✓					None	Network congestion and voltage control	Active power curtailment mechanism	JADE and MATLAB
Claessens et al. [112]	2018	✓	✓					None	Optimal control of TCL and district heating network with RL	Batch reinforcement learning	Not mentioned
Behboodi et al. [114]	2018	✓	✓					None	Transactive control of TCL with DR	Market bidding	Not mentioned
Bünning et al. [110]	2018	✓	✓	✓				None	Distributed control of bidirectional low-temperature network	Simplex Nelder–Mead method and market bidding	Python and Modelica

.

5.1.3. Regional Energy System Management

The regional energy system is a macro-system to the district energy systems, which consists of multiple district energy systems or clusters of micro-grids. Gao and Ai [116] proposed a three-layer control scheme for MESs with the integration of microgrids. In this hierarchical scheme, agents include regional system layer agents (top), micro-grid cluster layer agents (middle), and component layer agents (bottom). Top-layer agents are responsible for regional energy network optimization. Upon receiving top-layer optimization results, middle-layer agents will check with network limits and coordinate micro-grid clusters. The bottom components agents are responsible for each unit’s voltage and frequency control. Zhang and Yu [117] introduced a real-time control strategy for multi-area MESs. The control strategy is based on the Stackelberg game, where a global IES behaves as a leader and the rest local MES are followers. The top layer global MES is responsible for improving the response performance of the entire system, whereas the bottom layer local IES is responsible for cost optimization. The Stackelberg game is solved with Q-learning. The results showed that the proposed learning methods computed faster than common heuristic algorithms, including genetic algorithms, particle swarm optimization, and differential evolution.

In 2007, Geidl and Andersson [118,119] introduced the concept of the energy hub (EH) to model the multi-energy flow on the regional level energy network. The EH concept was soon welcomed by academics. The following selected articles adopted the MAS to manage EHs. Durana et al. [120] developed a multi-agent, multi-energy flow calculation framework for the EH network. The other energy carriers’ flow, such as natural gas and water, mimic the electricity power flow calculation. The multi-energy flow problem was solved with the classic Gauss–Seidel algorithm [121]. Loose et al. [122] used a similar concept with unified agents for both district heating and electricity network simulation, while the calculation was done using the Newton–Raphson algorithm. Skarvelis-Kazakos et al. [123] proposed hierarchical management of networked EHs. Each EH agent is responsible for optimizing internally, then participating in the energy market via a centralized commercial agent. The authors also conducted a lab experiment to evaluate the technical feasibility of agent-based control with a micro-CHP rig. The experiment showed that the agent-based control was technically feasible with cost-efficient equipment, such as a personal laptop. Zhang et al. [124] proposed a consensus-based control on the EH network with adaptive dual control to ensure operational security and cost minimization. Farshidian, Rajabi-Ghahnavieh, and Haghi [125] formulated the multi-EH planning problem as a competitive game between the hubs. Mohamed, Jin, and Su [126] provide a distributed energy management scheme for smart islands, consisting of networked multi-energy micro-grids, EHs, and plug-in EVs. The optimization is achieved with the primal-dual method of multipliers, which showed a better accuracy and convergence time than the ADMM method [127].

Xi et al. [128] proposed an automated generation control scheme combined with the double deep Q network and action discovery algorithm. The proposed scheme showed a faster convergence rate than the traditional Q-learning method. Wang and Zhang [129] proposed a two-layer multi-EH coordination strategy with micro-grid clusters. Each layer was formulated with the cooperated game. The two-layer optimization problem was solved by the deep deterministic policy gradient algorithm. Li et al. [130] proposed a multi-agent reliability evaluation method for the multi-energy network considering the uncertainties of wind generation. This method adopted the Smart Agent Communication algorithm [131] to achieve system reconstruction automation. Kou et al. [132] proposed a reliability evaluation model that considers the data privacy of each energy sub-system. The reliability evaluation model was designed in a distributed architecture with agent communication. Kou et al. [133] also proposed a multi-energy network coordination method with distributed accelerated descent algorithm. One of the selected studies proposed a novel multi-agent simulation framework for the multi-energy economy system. Zhu et al. [134] introduced a novel energy–economy system simulation approach based on the Java Agent Development Framework (JADE). This simulation framework enables the modeler to define the simulation time step based on the demand with the compatibility of different simulation time steps. The summary of regional-level energy system management is shown in

Table 8. Review summary of articles in regional energy system.

Source	Year	Energy Carrier						Storage Type	Focused Topic	Approach	Platform
		Electricity	Heating	Cooling	Gas	Hydrogen	EV
Durana et al. [120]	2014	✓	✓		✓			None	Multi-energy flow calculation	Gauss–Seidel algorithm	Anylogic
Li et al. [130]	2016	✓	✓	✓	✓			Electricity storage	IES reliability evaluation with agent-based modeling	K-1 algorithm to evaluate fault occurrence	Anylogic
Skarvelis-Kazakos et al. [123]	2016	✓	✓		✓	✓		Electric storage	Energy hub network optimization	Java optimization modeler	JADE and JOM
Gao and Ai [116]	2018	✓	✓		✓			Electricity storage and thermal storage	Multi-level hierarchical control of IES with multiple micro-grids	Multi-energy network control	PSCAD and EMTDC
Zhang and Yu [117]	2019	✓	✓		✓			Gas storage	Real-time coordinated control of multi-area IES	Solve fast Stackelberg equilibrium with Q-learning	MATLAB
Zhu et al. [134]	2020	✓	✓		✓			None	Novel simulation framework with multi-energy economy coupled system	Linear programming	JADE
Loose et al. [122]	2020	✓	✓					None	Unified multi-energy network simulation	Newton–Raphson	Agent.Workbench and JADE
Mohamed et al. [126]	2020	✓	✓		✓		✓	Electric storage	Smart island management	PDMM	Not mentioned
Xi et al. [128]	2020	✓	✓		✓		✓	Fly wheel storage	Multi-energy network automatic generation control	DDQN-AD	Not mentioned
Zhang et al. [124]	2021	✓	✓		✓			None	Multi-energy network control	Adaptive dual and consensus algorithm	Not mentioned
Farshidian and Ghahnavieh [125]	2021	✓	✓		✓			Electric storage	Multi-EH planning	GAMS	GAMS
Kou et al. [132]	2021	✓	✓		✓			Electric storage and gas storage	Reliability evaluation of multi-energy network	ADMM	MATLAB and MOSEK
Kou et al. [133]	2021	✓	✓		✓			None	Multi-energy network coordination	Distributed accelerated descent	MATLAB
Wang and Zhang [129]	2021	✓	✓		✓			Electric storage	Multi-EH coordination	DDPG	Python TensorFlow

.

5.2. Agent-Based Modeling Applications

ABM is a growing research field to study the complex adaptive system by simulating the behavior and co-evolution of system components (agents). The characteristics of agents in ABM are similar to that of MASs, as agents in both systems are able to behave based on rationality. The difference between ABM and MASs is the modeling objective. The modeling objective of MASs focus on the technological implementation of distributed information processing, such as distributed system control. On the other hand, the modeling objective of ABM primarily focuses on studying the interactions of agents and the emergence patterns. Therefore, ABM is a valuable method for studying the energy system social-technology transition. The ABM has been used to simulate stochastic consumer behaviors and policy-making research in selected studies.

5.2.1. Behavior Simulations

Energy demands are usually estimated from historical data and computer modeling [135]. When historical data is unavailable, demand estimation relies on computer simulation. However, the computer-simulated demand may significantly differ from actual consumption. Hong et al. [136] argued that this mismatch resulted from uncertainties of human behaviors. Turner and Frankel [137] reported that estimated energy usage in the design stage has a root-mean-square error of 18% compared to actual measured energy usage in a 62-building cluster. Moreover, Eguaras-Martínez et al. [138] showed that simulations including consumer behaviors might differ up to 30% from simulations excluding consumer behaviors. Thus, human behavior integration is an important aspect of energy system modeling. Pfenninger et al. [139] regarded human behavior integration as one of the four challenges of energy system modeling in the 21st century. Thus, agent-based modeling shows a promising way to simulate random occupant behavior. Chen et al. [140] developed an agent-based simulation model for stochastic heat pump usage in a residential area. Such a model utilizes technical information (household layout, family members, as well as heat pump location and quantities) and social information (occupant profiles and usage rules, and interaction rules of heat pumps) for stochastic behavior simulation. Case study results showed that the simulated results were consistent with actual usage data (obtained from surveys), and simulation accuracy can be improved significantly with a larger sample size. Chingcuanco and Miller [141] developed an urban energy consumption model integrated with agent-based modeling. It was shown that agent-based modeling is able to capture both short-term and long-term decisions of firms and households. Tian et al. [142] combined the deep learning model and ABM for household energy consumption simulation, considering spatial and temporal aggregation. The deep learning model was pre-trained with household behavioral data, such as appliance usage patterns, dwelling type, and energy bills. The deep learning model generated the agent behavioral probability distributions based on the input variables. Individual household agents’ behaviors during the day were further aggregated at the regional level and in monthly and yearly time intervals.

Furthermore, researchers combined the behavioral simulations in ABM with the optimization model as a holistic modeling method. Zhang et al. [143] proposed a holistic combined optimization approach for integrated energy systems, including demand simulation with ABM and system optimization with the MILP model. The novelty of this approach is the inclusion of occupant behavior simulations, where random behaviors of heterogeneous energy consumers and interactions among them are considered. ABM is used for modeling end-users and energy consumption appliances by considering different consumption activities, behavioral rules, and user preferences. Zhang et al. conclude that occupant behaviors contribute to demand uncertainties considerably, in which the cooling and heating demands showed great variance. Chakrabarti et al. [144] combined agent-based modeling for EV demand simulation and district energy system optimization with MILP. Each EV was considered an agent traveling around to fulfill its objective.

5.2.2. Policymaking

The studies in policymaking adopt the ABM to simulate the co-evolution of the social-technological systems, examining the exogenous factors (policy, regulation, energy prices) that affect the systems. Since this review article focuses on agent-based method applications in MESs, Section 5.2.2 covers how researchers implement ABM in the energy system transitions towards MESs by answering the following questions. Readers could refer to the reference for detailed policy implications.

Jackson [43] analyzed the utility stand-by rate effect on adopting micro-combined heat and power (micro-CHP) generation units in the US. The utility standby rate is the infrastructure cost for the utility grid to offer backup services when a micro-CHP outage occurs. The agents (building owners) consider investing in micro-CHP when the knowledge about micro-CHP reaches the threshold level. The knowledge will increase with the installation of micro-CHP in the neighborhood. Faber, Valente, and Janssen [145] also explore households’ adoption of micro-CHP in the Netherlands with ABM. The model specifically investigated the competition of the market share of micro-CHP and incumbent condensing boilers. The market share dynamics was simulated with supply side: micro-CHP agents and incumbent condensing boiler agents; and demand side: individual consumers agents. The consumer agents will make a decision to adopt either type of technology with minimal cost. In addition, the exogenous effect, such as natural gas prices, electricity prices, and policy interventions on micro-CHP adoption, were explored. Grubic et al. [146] investigated how the adoption of multiple micro-generation technologies affects the consumption of water, gas, gasoline, electricity, CO2 emissions, and electricity generation cost. The micro-generation technologies investigated in this study included battery EV, PV, solar thermal water heating, rainwater harvesting, greywater recycling, and waste heat recovery. The behaviors of household agents were modeled with state charts, in which agent states transit from one to another. The quasi-rational approach was adopted to determine whether agents transition to another state.

Sorda, Sunak, and Madlener [147] investigated the spatial-temporal diffusion of agricultural biogas-fueled CHP plants in Germany with Geographic Information Systems (GISs). The spatial ABM includes two types of agents. The first type of agent is organizations: federal government agents, bank agents, electricity utility agents, and plant manufacturer agents, which only provide information to the system. The second type of agent performs the investment calculation that results in investment decisions, including substrate supplier, heat consumer, decision maker, and district. The substrate supplier agents calculate the availability of biogas production in communities. The heat agents calculate the heat potential of communities and determine the earnings and costs with heat demand. The calculated results from the substrate supplier agent and heat consumer agent are sent to the decision-making agents. The decision-making agents communicate with the organization agents to obtain financial information such as bank loan availability, interest rates, and the bio-gas plant cost. The decision-making agent then calculates the net present value (NPV) of the associated investment and decides to invest if the NPV is positive. The district agents rank the decision-making agents of communities based on the availability of substrate potentials. Schnuelle et al. [148] also modeled the energy system transition in Germany with a focus on the adoption of power-to-fuel technologies. The power-to-fuel technologies refer to the conversion of electricity to hydrogen for hydrogen-based synthetic fuels. The ABM model treated the German energy system as a single MES where each energy carrier was simulated as a sub-system which has its own supply, price, and trade volume. This model included three agent classes: customers, operators, and subsidizers. The operator agents are the key agent class, as this agent class drives the investment and installation of power-to-fuel technologies. The decision-making process of operator agents was based on rational choice theory [149].

Allen and Liz [150] modeled the UK energy system with ABM to explore the possible pathways to achieve an 80% reduction of CO₂ emissions by 2050. The agents decide to adopt generation technologies based on calculating the attractivity index, which considers the relative importance of carbon reduction and the financial cost. Walzberg et al. [151] investigated how smart home appliances could contribute to the rebound effect with ABM. The rebound effect is the phenomenon that the introduction of cost-efficient technology could lead to an increase in the demand. The ABM model simulated the household agents’ behavioral changes in the adoption of smart home appliances with stochastic electricity load profiles. The decision-making process of household agents in this ABM was based on the social-psychological model by Byrka et al. [152]. Then et al. [153] investigated the effect of building owners’ retrofitting on the gas and electricity demand and the district network owners’ investment decisions on grids. The household agents make the retrofitting decision to minimize the life-cycle cost of total expenditure with MILP. The district network owners plan the closure, reactivation, construction, and reinforcement of gas or electricity grids based on cash flow calculations.

Hodge et al. [154] proposed a multi-paradigm modeling framework to study the new technologies’ impact on an energy system. The multi-paradigm framework was based on the modulation of energy systems by dividing the energy system into a series of sub-systems. The sub-systems could be modeled with various simulation techniques depending on the sub-systems’ geographical aggregation level or time resolution level. The modulation enables modelers to incorporate the multi-energy system with energy carriers as sub-systems. In the case study of Hodge et al., California’s combined power generation and natural gas model was studied to investigate the effects of natural gas price fluctuation on gas-fired power generation plant operations. Agent-based modeling is used to simulate the electricity generation units in California as the generation units are heterogeneous. The simulation results showed that the framework provides valuable information to policymakers to study the interaction of energy sub-systems, especially when investigating the effect of new technologies.

6. Discussion of Agent-Based Method Applications

Section 5 has reviewed the research themes and characters of the selected studies. Section 6 discusses how agent-based methods have been implemented to study MES transitions, including how MASs are implemented in MES optimization, how ABM contributes to the study of social-technological co-evolution of energy systems, and what are agent development environments. Figure 8 provides a schematic overview of agent-based method applications in the MES modeling and simulation in terms of spatial and temporal resolutions, which demonstrates the applicability of agent-based methods to resolve a variety of MES research questions. The ABM models on social-technical systems tend to have the largest temporal resolution that span decades to capture the system co-evolution process. The spatial resolution of the ABM varies depending on the system boundaries. Noticeably, ABM can simulate the system on the national level, as shown by Allen and Varga [150].

6.1. Agent-Based Optimization of MESs

This section discusses the agent-based optimization of MESs from an economic perspective, such as optimal dispatch, optimal design planning, and optimal multi-energy power flow. The classic economic optimization problem can be solved with convex optimization techniques. Convex optimization techniques transfer the primal problem to the Langrangian dual problem, then solve Karush–Kuhn–Tucker (KKT) optimality conditions. [155]. However, four paradigm shifts are reshaping energy system optimization. First, the dataset size is expanding significantly due to the increasing number of DER units. Second, the data are becoming extremely high-dimensional since the IoT technologies, such as smart meters, are able to measure detailed information about each data point. Third, the dataset tends to be stored in a distributed manner, as the DER owners are distributed geographically. Fourth, the liberalization of the electricity market enables competitive bidding and asking activities for energy production companies, retailers, and households in day-ahead markets [62,156]. Decentralized optimization could be an optimization framework that is suitable for future MESs.

The early systematic research in decentralized optimization can be dated back to the 1980s by two MIT professors, Bertseksas and Tsitsiklis, in their early publication [157] and their book on numerical methods in distributed computing [158]. After nearly four decades of development, MAS optimization is emerging as a framework to formulate and solve optimization problems. In MAS formulation, each decision-making agent is responsible for optimizing the local objective function, and the global objective function is the summation of local objective functions. Moreover, the distributed optimization problem is also subject to a list of constraints that depends on the scope of the energy system. An example of decentralized optimization problem formulation with a multi-energy micro-grid focusing on generation cost minimization is shown as follows:

$$\min {\displaystyle \sum }_{i=1}^N{\displaystyle \sum }_{j=1}^M{\mathrm{C}}_{i,j}\left(g_{i,j}\right)$$

(1)

$$\forall Energycarrie{r}_i,{\displaystyle \sum }_{j=1}^M{g}_{i,j}=d_i$$

(2)

$$\forall g_{i,j}:g_{i,j}^{min}\le g_{i,j}\le g_{i,j}^{max}$$

(3)

In the distributed optimization problem, $g_{i,j}$ represents the quantity of $energy carrie{r}_i$ generated by $agen{t}_j$ where $g_{i,j}$ belongs to the set $\left\{g_{i,j}|g\in ℝ, i=1,2\dots M;j=1,2,\dots N \right\}$ such that the system contains $M$number of energy carriers and $N$ number of generation agents. The multi-energy generation quantity is a real value matrix denoted as $\mathit{G}$, where $\mathit{G}\in {ℝ}^{M\times N}$. For instance, if the electricity carrier is denoted as $g_{1,*}$, the generated electricity vector in the system is the first-row vector of $\mathit{G}$, $\left[g_{1,1}, g_{1,2}, \dots g_{1,M}\right]$. As a result, the total electricity generated in the system is the element-wise summation of the electricity vector $\sum _{j=1}^M{g}_{1,j}$. ${\mathrm{C}}_{i,j}\left(·\right): {ℝ}^{+}\to {ℝ}^{+}$ is the local objective function of $energy carrie{r}_i$ generating $agen{t}_j$. The global objective function is the summation of local objective functions, as shown in $\left(1\right)$. Depending on the nature of the problems, the objective function could be an operation cost function [84,85,103,159] or investment cost function [68,69,73]. $\mathit{D}$ is the total system demand of energy carriers, where $\mathit{D}={\left[d_1, d_2, \dots d_M\right]}^T$ corresponding to the $M$number of energy carriers in the system.

For a multi-energy system consisting of $M$number of energy carriers, the energy balance constraint $\left(2\right)$should always be satisfied. The number of energy carriers considered in the optimization problem depends on the configuration of the target energy system. For example, Refs. [68,69] considered electricity and hydrogen carriers, and Refs. [86,130] considered electricity and heating carriers in the optimization problem. Constraint $\left(3\right)$ indicates the capacity constraint of energy generation agents. For studies interested in optimizing regional network systems, the network constraints, such as transmission line capacity, should be considered accordingly [156].

To solve the optimization problem, various optimization algorithms, such as Particle Swarm Optimization (PSO) and Genetic Algorithm (GA) were used by researchers, as shown in Table 7. PSO is the most popular optimization technique in the review database. PSO is a population-based heuristic search method invented by Kennedy and Eberhart [160]. The mathematical description of the particle movement is shown in Equations $\left(4\right)$ and $\left(5\right)$.

$$v_{i,t+1}=w{v}_{i,t}+r_1{C}_1\left[p_{i,t}-x_{i,t}\right]+r_2{C}_2\left[g_t-x_{i,t}\right]$$

(4)

$$x_{i,t+1}=x_i+v_{i,t+1}$$

(5)

where $r_1$ and $r_2$ in $\left(4\right)$ are two random variables, ranging from 0 to 1; $C_1$ and $C_2$ in $\left(5\right)$ are the acceleration constants; $v_{i,t}$ and $x_{i,t}$ are the velocity and position of particle I at time t; $p_{i,t}$ is the previous personal best position of particle i; and $g_t$is the best position of all particles.

A particle is a solution in the search space. On initialization, the PSO algorithm randomly generates a group of particles containing individual position and velocity information. Then, particles evaluate their own best position with the all-particle best position with a fitness function. After evaluation, each particle will move to its previous personal best position ($p_{i,t}$) or the global best position ($g_t$) [161]. Readers could refer to Zhang, Wang, and Ji [162] for state-of-the-art PSO studies, such as modification of PSO and hybridization of PSO with other heuristic algorithms. The standard application of PSO with MASs in multi-energy systems can be found in [67,68,69,73,97]. However, the standard PSO tends to fall into local optimality due to insufficient search spaces. Li et al. [100] used the adaptive weight chaotic PSO (ACPSO) to optimize the micro-grid. The idea of a chaotic search is to iteratively generate a chaotic sequence if the solution did not change significantly under a pre-defined number of iterations. Moreover, the adaptive weight technique balances the global and local search abilities. The ACPSO shows better convergence than the standard PSO. Another popular algorithm to solve the MES distributed optimization problem is multi-agent reinforcement learning which will be discussed in Section 7.1.

With the decentralized optimization problem formulation and algorithms to solve the problems, MASs are able to solve the MES optimization problem in a decentralized way. Figure 9 shows three types of optimization architecture: centralized, decentralized, and hybrid. The conventional centralized optimization architecture is shown on the left of Figure 9, where a central computation agent is responsible for collecting the systemwide information and performing optimization tasks. The advantage of using centralized control is the comprehensive multi-objective optimization ability with available information [163]. However, the major shortcoming of centralized decision-making is its susceptibility to single-point failure.

The decentralized optimization utilizes local agents to assess the operation signal of individual DER units [103]. The IoT technologies equip local agents with computation and communication abilities. The local agent performs the optimization of the local objective and then updates the operation parameters with other local agents. The global optimum is reached once all agents reach a consensus. Therefore, decentralized optimization architecture does not require a central agent for optimization. As a result, the system becomes more robust by reducing the risk of a single-point failure. Another advantage of the decentralized architecture is the plug-and-play ability. Therefore, the system could scale smoothly by adding more agents to the system. Moreover, decentralized control could protect local privacy as the local agent will not share detailed information with the central agent.

In hybrid optimization schemes, an intermediary layer is introduced, where a group of local agents reports state information to the respective intermediary agent. The hierarchical organization of hybrid optimization facilitates the bidirectional information exchange of micro-grid operation and upstream grids as well as the aggregation of micro-grid-generated energy to upstream grids. The top layer central agent is responsible for updating energy price and operation information from the upstream grid to the intermediary agents. Based on the central agent information, the intermediary and local agent perform the optimization and send the optimization results to the top layer central agent. With the micro-grid optimization results, the central agent will evaluate the feasibility of aggregating the locally generated energy to the upstream grid, informing the subsidiary agents if such a decision is made. Therefore, MAS-based management has the following four advantages:

(1)

Flexibility: Intelligence and rationality enable the agent to learn and adapt to system changes.

(2)

Reliability (fault-tolerance): The decentralization brought by MAS makes the system less susceptible to a single-point failure. If one agent fails, the communications among agents are still intact.

(3)

Scalability (extensibility): Due to the plug-and-play attribute of MAS, the system could be extended with ease. Instead of redesigning the whole system, the modeler can only design a class of agents and embed such agents into the system.

(4)

Privacy protection: The fully decentralized control does not require an agent to collect systemwide information. As a result, each agent can preserve the privacy of classified information.

6.2. MES Social-Technological Simulation with ABM

The previous Section 6.1 discussed the methodology to apply MASs in MES optimization. This section discusses how the ABM could contribute to studying the multi-energy system social-technical transitions. Figure 10 demonstrates the critical concept of co-evolution of the social-technical system. The social entities, such as households and energy network operators, were modeled as agents. The agents will decide whether to adopt a particular technology based on rationality. Agents’ actions will alter the environment (technical systems), resulting in the system’s transition to the next state. ABM is a systematic approach to investigating how exogenous factors (policies, regulations, energy prices) change the system evolution in states. Based on the simulation results, the modeler could provide valuation implications for designing the appropriate energy policies for policymakers.

6.3. Agent Development Environment

The agent-based method, in nature, is agent-oriented, which echoes the object-oriented programming (OOP) languages. In OOP language, the computation work is carried out with a series of objects with pre-defined attributes (instance parameters) and behaviors (methods). Thus, the agent-based methods could be carried out in various OOP languages, including Java, C, C++, and Python.

Java Agent Development Framework (JADE) is a popular development environment for building MASs that follow Foundation for Intelligent Physical Agents (FIPA) specification with Agent Communication Language (ACL) [26]. JADE offers developers a series of agent services such as an agent management system, directory facilitator, and agent communication channel [164]. To develop a MAS application with JADE, developers can define the subclass agent attributes by extending the superclass Agent in the jade.core package and agent behaviors by extending the superclass Behaviour in the jade.core.behaviours package. Agents communicate with each other on the java runtime environment through sending/receiving the JAVA objects named ACLMessage. The complete guide for developing MAS with JADE can be found in the JADE handbook [165].

Anylogic is another popular platform where the model is Java-based [166]. Anylogic provides a convenient modeling environment for agent-based modeling due to its Java-based GUI. Python also offers a ABM modeling library called Mesa [167]. MATLAB Simulink is a powerful software environment for carrying out heavy computational simulations. The Simulink S-function [168] enables developers to write Simulink blocks in an object-orient way with C or C++. MACSimJX [169] is an open-source package in MAS-enabled smart energy system research. MACSimJX serves as a wrapper between MATLAB Simulink and JADE, integrating the high computation capability of MATLAB and the multi-threading of JADE. For a comprehensive review of agent development tools, readers could refer to the review paper on agent-based modeling and simulation tools by Abar et al. [170].

7. Future Research Directions

Section 7 highlights two key future research directions after evaluating the existing literature: multi-agent reinforcement learning implementation for MES management and synchronization among agents in agent-based control.

7.1. Multi-Agent Reinforcement Learning Applications

Reinforcement learning is a machine learning domain that focuses on optimal sequential decision making, akin to dynamic programming, by Bellman [171]. The building block of reinforcement learning is the Markov Decision Process (MDP). The theory of MDP was developed in the optimal control field to solve the problem of sequential decision making under uncertainty [172,173,174,175]. The MDP is established on the decision-making agent and environment, which are exceptionally similar to the concepts discussed in Section 2. Agents and environment interact over a sequence of discrete time steps. Agents take actions upon prior knowledge and environmental information. Then, the environment gives numerical feedback, named rewards, to agents based on the actions that agents took [88]. Mathematically speaking, at each time step $t$, an agent receives the description of the environment information state $S_t\in \mathcal{S}$. Based on the state information received, the agent selects an action $A_t\in \mathcal{A}\left(\mathcal{S}\right)$. In the next time step $t+1$, the agent receives a numerical reward $R_{t+1}\in \mathcal{R}\subset ℝ$ in new environment state $S_{t+1}$. Then, the sequence of an MDP is $S_0, A_0, R_1,S_1,A_1,R_2,S_2,A_2, \dots$ The reward $R_t$ and environment state $S_t$ are random variables that depend on the previous state and agent action. The objective of agents is to formulate the optimal policy to maximize the total reward within the considered time period. The Q-learning method is the most widely used value-based method to obtain the optimal policy because of its simplicity [176,177]. The main idea of Q-learning is to use the previous action value (Q-value) as the input for the subsequent iteration.

Game theory is an effective method to solve the problem with energy market trading. In 2020, He et al. [178] reviewed the application of game theory in the integrated energy system, in which multi-energy commodity trading was considered. The implementation of game theory-based multi-agent systems included optimal component sizing [71,73] and optimal operation [69,101,117]. In these game models, the agents are designed with complete information so that they are aware of systemwide information. Thus, Liu, Li, and Ge [101] highlighted the future research direction to study game models with incomplete information for MAS applications in multi-energy trading. In MAS applications in MES optimization, Zhang et al. [117], Smadi et al. [85], and Liu, Li, and Ge [101] adopted the Q-learning technique in their studies. However, the Q-learning method is criticized as having slow convergence [179] and the curse of dimensionality [180]. Thus, Claessens et al. [112] used batch reinforcement learning with fitted Q-iteration [181] in combination with an extremely randomized tree algorithm [182]. In future research, more advanced learning algorithms (such as deep learning [128,183]) are expected to be incorporated into energy system optimization.

7.2. Synchronization in Agent-Based Control

Synchronization is an active research topic in MAS control, where agents send and receive control signals with minimal time-varying delays. In the context of MES control, renewable DER production, in nature, is stochastic. The stochastic distributions of solar radiation and wind profiles lead to an intermittent production signal. If the DER production fails to synchronize, it will cause disturbances to the electric network [184]. With a systematic review of the existing literature, there is a very limited discussion on the synchronization coordination of agents in the MES systems. Therefore, this research gap demonstrates good potential for future research.

8. Conclusions

The next generation of energy systems are distributed, multi-energy carriers integrated and smart due to the penetration of DER and IoT technologies. The agent-based methods, including MASs and ABM, could potentially facilitate the energy system transformation to the next generation. This article aimed to systematically review the MAS and ABM applications in multi-energy system transitions. This review showed that the number of publications related to agent-based methods has been increasing in recent years, indicating that both MASs and ABM are gaining popularity among researchers.

This review summarizes and discusses the key research themes identified in the selected articles. In energy system management, MASs have been applied in operation control, economic dispatch, and optimal planning in different spatial resolution levels, including building, district, and regional energy systems. The distributed decision-making ability of MASs provides an energy system with a high level of flexibility, scalability, and reliability. As a result, MASs are a viable method to improve system performance. Moreover, the energy system is becoming increasingly complex due to the increased penetration of new energy generation, conversion, and storage technologies. ABM was a promising method to study the system emergence of social-technical systems. The various research themes demonstrated the agility of agent-based methods as a research methodology.

There is potential for future research to investigate the multi-agent reinforcement learning applications in energy systems. This rapidly developing artificial intelligence/machine learning technique has promising implementation in system control and design optimization. Implementing and improving such techniques in energy systems could lead to valuable results. Furthermore, future research could investigate the synchronization of the agent-based control for MES. However, there are limitations to this review. Agent-based methods are gaining popularity among researchers so articles have continued to be published on related topics. Thus, this review has not been able to cover the publications on the topic after 2021. Additionally, this review highlighted the energy system optimizations from the economic perspective in Section 6. Other aspects of energy system control, such as frequency control and fault analysis are beyond the scope of the discussion section of this review.