A Multi-Provider End-to-End Dynamic Orchestration Architecture Approach for 5G and Future Communication Systems

Abstract

Network segregation is the solution adopted in the IMT-2020 standardization of the International Telecommunications Union (ITU), better known as 5G networks (Fifth Generation Mobile Networks), under development to meet the requirements of performance, reliability, energy, and economic efficiency required by applications in the various verticals of current and near-future economic activities. The philosophy adopted for the IMT-2020 standardization relies on the use of Software-Defined Networking (SDN), Network Function Virtualization (NFV), and Software-Defined Radio (SDR), i.e., the softwarization of the network. Softwarization allows network segregation through its slicing, which is discussed herein this work. Network slicing is performed by a novel Orchestrator, as provided in IMT-2020, which maintains the end-to-end network slices independent of each other and performs horizontal handover when the possibility of a loss of Quality of Service (QoS) is predictively detected by monitoring quality parameters during operation. Therefore, the Orchestrator is dynamic, operates in uptime, and allows horizontal handover. Hence, it chooses the most appropriate telecommunication infrastructure provider and network operator to guarantee QoS and Quality of Experience (QoE) to end-users in each network segment. These features make this work modern and keep it aligned with the actions being carried out by ITU. Based on this objective, as the main result of this paper, we propose an effective architecture for implementing the Orchestrator, not only to contribute to the state of the art for 5G and beyond communication systems but also to generate economic, technological, and social impacts.

1. Introduction

The advent of the Internet of Things (IoT) [1,2,3] and the possibility of processing in the entire network cloud (Distributed Cloud Computing, from the core to the edges of the network [4]) are some of the numerous projections to be intertwined with Fifth Generation Mobile Networks (5G) and next-generation wireless communication systems (6G and beyond), allowing them to offer faster mobile services under higher frequencies waves and thus offer new applications [5,6,7,8]. Thus, 5G and beyond will be a mix of efforts in computing and telecommunications technologies, as well as computer and communications networks.

Fifth Generation Mobile Networks are not just a simple evolution of mobile telecommunications technologies but a true revolution, in which computing and telecommunications technologies are present in the same architecture, aiming to solve the problem of connectivity for any class of service, regardless of its non-functional requirements [9,10].

Fifth Generation Mobile Networks promote rapid and massive adoption of new solutions [11], as they allow using legacy networks and, when required, among others, very high broadband Internet speeds (greater than 1 Gbps), low latency (1 ms) [12,13], and high reliability (99.9999%) [13], providing profound transformations in several industries, in addition to enabling the emergence of business models that would currently be unfeasible at the hospital, logistical, vehicle traffic levels, and so on [14].

Among the applications of great relevance to 5G, we can mention: autonomous driving by vehicles connected virtually and logically (Vehicle-to-Everything (V2X)) by tactile Internet. That is, in the broadest sense, 5G will provide better and greater connectivity between mobile and/or even fixed devices [15]. However, to use of all the network resources according to the end-users’ demands, network slicing and orchestration are essential [16].

The new scenarios have strict and heterogeneous requirements to be achieved by improvements to the Radio Access Network (RAN) and a collection of innovative wireless technologies [17]. Software improvements, such as Software-Defined Radio (SDR), Software-Defined Networking (SDN), and Network Function Virtualization (NFV), will play a vital role in the integration of these different technologies [18,19]. In addition, the access networks and 5G radio core will be based on a virtualized SDN/NFV infrastructure, which will orchestrate resources and control the network, to provide network services in an efficient, flexible, and scalable manner.

A Network Slice (NS) is an important complex attribute of 5G networks [20]. According to NGMN [21], an NS is defined as a set of network functions and resources to run them, forming a complete instantiated logical network to meet specific network characteristics required by the Service Instance(s).

This work presents the fundamentals of network slicing and orchestration, highlighting the challenges for slice management. The objective is to present and discuss a conceptual model and a horizontal (multi-domain) End-To-End (E2E) dynamic multi-provider Orchestrator architecture approach to reduce regulatory barriers to the expansion and improvement of the performance of 5G and future communication systems, as well as to preliminarily prove its efficiency in a V2X scenario under a higher degree of automation for an imminent collision scenario.

The remainder of this paper is organized as follows: Section 2 contextualizes the problem in question; Section 3 exposes the related work; Section 4 presents the proposed Orchestrator architecture by reporting, analyzing, and discussing it; Section 5 presents the experimental tests conducted; Section 6 reveals the results and discusses them. Finally, Section 7 concludes the paper and outlines the future perspectives from this work.

2. Problem Statement

The 5G Infrastructure Public Private Partnership (5G-PPP) proposes addressing all aspects of segregation or slicing of the 5G network. A Fifth Generation Mobile Network is expected to be a multi-service network that supports a wide range of verticals with a diverse set of performance and service requirements based on the division of a single physical network into several isolated logical networks [22]. Therefore, the slicing of the network appears as an economic solution for implementing the diverse requirements of 5G and their respective verticals. Thus, 5G-PPP divides its architecture into different layers [23].

By selecting NSs, the Network Service Orchestrator, or simply the Orchestrator, is able to create a logical network with different characteristics customized to the needs of each user, ensuring parameters such as latency, bandwidth, transfer rate, security, availability, etc. [24]. Thus, the Orchestrator selects the best logical and physical networks to serve the end-user. The purpose of network slicing is to provide logical isolation and independent operations under all parts and layers of the network.

An E2E NS is deployed across one, two, or several networks stretching across the RAN, transport, and core network, belonging to the same or different administrative domains. The process of establishing a multi-domain Network Slice Instance (NSI) [25] leverages the benefits of recursive virtualization, as described in [26].

The process of building the NS in 5G networks will be carried out under three main virtualized layers, namely: (i) the Service Instance Layer, (ii) the Network Slice Instance Layer, and (iii) the Resource Layer, as illustrated in Figure 1.

A Service Instance characterizes each service, that is, an operating time construction of an end-user service or a commercial service performed within or by an NS [27]. Each service instance reflects a service provided by a vertical segment via the service provider application.

The NSI represents a set of resources customized to accommodate the performance requirements of a specific service and can contain none, one, or several different Sub-Network Slice Instances (SNSIs), being isolated or shared. An SNSI can be a network function, a subset of network functions, or network resources that realize part of an instance of the NS. Each instance of the NS is established E2E and can contain distinct sub-networks—domains that are logically and/or physically isolated from another instance of the NS.

In particular, resources associated with a sub-network can be used in isolation, disjunctive, or shared, following the specific policies and configuration arrangements of the NSI. An NSI, in turn, can be used exclusively by a service instance or shared between different service instances, usually of the same type. Common abstractions of relevant resources and open Application Programming Interfaces (APIs) allow dynamic control and automation of instances of Network Slices that reflect dynamic demands of services [22].

In the process of orchestrating or managing services and controlling network slicing, the NS provides E2E connectivity, allowing different network technologies to coexist in a common infrastructure and under a continuous closed-loop process that analyzes service requirements to ensure desired performance. NSs consist of Virtual Network Functions (VNFs), Physical Network Functions (PNFs), value-added services, cloud network resources with dedicated and shared software and hardware in the RAN, transport, and core networks, thus combining different technologies [22,28].

Nevertheless, implementing E2E network slicing is a challenging objective that requires well-developed enabling technologies, global standards, and a mature ecosystem [29].

Such a paradigm can facilitate the composition of NS across different administrative borders, efficiently and flexibly combining different types of resources. Thus, the main challenge is the deployment and runtime management since the involved domains may not only be geographically far from data centers interconnected over a Wide Area Network (WAN) infrastructure, but they may belong to different administrative domains (multi-domain) [28].

Note that the term multi-domain may be applied to the same provider or different providers, but the term multi-provider necessarily refers to multi-domain.

Current providers can implement slicing without 5G, but this is likely to become much more prevalent with 5G and its emerging specifications, which require partitioning of data, control, and management planes to separate the environments to be created. Thus, serving individual customers or providing specific services, giving Internet Service Providers the opportunity to more easily support multi-tenancy, specific customers, and use cases to meet each slice’s exclusive Service Level Agreements (SLAs).

Additional efforts are needed to develop algorithms and methods capable of managing resources and of truly providing advanced, efficient, and reliable 5G services with appropriate Quality of Service (QoS) for end-users [30]. In 5G, QoS must be E2E, i.e., established between a sole Service Provider (SP) and the end-user [23,31].

Regarding the concept of QoS, there is also that of experience (QoE), which is related to users’ perceptions of their experience of a given service. The better the compliance with the QoS parameters, the better the perception of QoE. [32].

It will also be a challenge to meet the E2E response time for slicing the network and establishing the connection according to the needs demanded by each user. Thus, our proposed Orchestrator, called 5G-Horizontal (5G-H), is horizontal since it must establish and maintain the NS that meets the requirements of the target service at the lowest cost during the service’s life cycle while considering all the infrastructure available in the region, regardless of the infrastructure and networks (specific providers), working in operating time, and aiming solely at the requirements of the services demanded.

Further research should mitigate the challenges in Computer Engineering about aspects of dynamic orchestration in operating time, Distributed Cloud Computing, Distributed Big Data, and management algorithms with Artificial Intelligence (AI), which are all vitally related to orchestration in 5G [33,34].

3. 5G-Related Work

This section provides an overview of the development of 5G network service orchestration.

Currently, in addition to International Telecommunications Union (ITU), 5G standardization is being detailed by other entities, such as 5G-PPP, 3rd Generation Partnership Project (3GPP), European Telecommunications Standards Institute (ETSI), and 5G Brasil, in collaboration with several international research projects [35,36,37,38,39,40].

A number of companies and entities integrate testing projects in 5G transmissions to overcome deficiency problems in mobile communications coverage [41]. Nevertheless, studies on 5G technology [39] are not yet entirely conclusive, and there are some researchers in partnerships with companies that have worked on the topic in Asia, Europe, and the USA, among them [42,43,44]; however, little exploring the layers above the network layer of the 5G architecture [31].

Thus, based on the elements presented, the performance evaluation model and Orchestrator, an entity that implements the NSs for each service and guarantees the QoS E2E, the 5G network must concomitantly and transparently consider and meet all bands present in the characteristics of the frequency bands below 1 GHz, between 1 GHz and 6 GHz, and above 6 GHz [45,46].

An issue that has arisen in some countries is whether the slicing of the 5G network will be consistent with the network neutrality regulation. Some say that the practical implications for current open Internet rules are speculative at this stage concerning 5G. This is because the evolution of the different 5G elements, such as network slicing, depends not only on the occasional technological capabilities but also on the market demand, the degree of competition, the commercial strategies, and so on [14].

The design of a 5G Orchestrator must consider vertical applications under the service requirements appropriate to 5G use cases and indicators defined by the regulator. In this sense, the ITU specified several challenges at IMT-2020. According to the preliminary report on Minimum requirements related to technical performance for IMT-2020 radio interfaces, performance indicators must be obtained in each 5G use case [47,48].

Thus, studies and developments at a global level have been carried out for vertical (single-domain) Orchestrators, that is, those who attend to a Telecommunications Infrastructure Provider (TIP) or Mobile Network Operator (MNO) specifically, by entities and researchers, such as Ericsson, Nokia, ETSI, and others [40,49,50,51,52,53,54,55].

TIP is a provider of wireless communications infrastructures that owns all the elements necessary to sell and deliver services to an end-user, including radio spectrum allocation, as well as wireless and backhaul network infrastructures. MNO is a provider of wireless communications services that operates or controls all those elements to offer services to an end-user, including wireless and backhaul network infrastructures, customer care, provisioning computer systems, and repair organizations.

3GPP introduced an orchestration and management architecture in [25], composed of a service management function that analyzes incoming slice requests. Thereby, converting service requirements into networking ones and an NS management function, which performs the mapping onto network resources and takes care of the Life-Cycle Management (LCM). Although the resource mapping process is carried out across different technology domains, including the RAN, transport, and core, the current 3GPP efforts concentrate only on NSIs deployed and managed by a single administrative entity. The authors in [56] present the status of the access networks and 5G radio core defined by software and a wide range of future research challenges in terms of orchestration and control. The concept of virtualization brings the need for production, control, and management of virtual machines performed by the hypervisor or Virtual Machine Monitor (VMM), a firmware capable of providing a virtual platform for operating systems, allowing the execution of applications or other services, such as the selection of NSs. In addition, hypervisors allow supervising the sharing of hardware resources between instances of the NSs [22].

However, although several proposals point to promising paths, it is not yet possible to aggregate the various features in a unique and fully functional approach, which defines the operation and management mechanisms of each slice, in addition to providing subsidies for scalability, orchestration and support decision-making, in domains involving heterogeneous technologies and access methods (e.g., 5G, LTE, Wi-Fi, Wireline) [57,58].

A preliminary study toward a framework for virtualization in a multi-domain environment is introduced in [59], elaborating on the main concepts of isolation, programmability, and performance maintenance, as well as the fundamental functional components. Logical resources from different administrative domains are collected by a virtualization resource manager, which runs as a broker, allowing third parties to establish a virtual network optimized for supporting main services. In [28], a proposal is explored for a multi-domain orchestration and management framework to address the service challenges of NS when utilizing federated (E2E multi-domain) resources.

Another federated slicing solution is presented in [60] introducing the notion of a multi-domain Orchestrator, which handles slice requests for resources beyond its domain. The proposed multi-domain Orchestrator analyzes the related service requirements and then directly contacts the appropriate neighboring domains performing resource negotiation. Once a slice is established, a peer-to-peer management plane is responsible for handling the LCM considering service-oriented key performance indicators, while closely coordinating with domain-specific Orchestrators [28].

A hierarchical multi-domain orchestration architecture is introduced in [61] based on the concept of recursive abstraction and resource aggregation that stitches NSI heterogeneous resources initially on a per-domain level and then across federated domains.

A similar concept is presented in [62], wherein an overarching Inter-slice Resource Broker functional element is proposed to manage and orchestrate resources for E2E slices across multiple technology domains. Each domain facilitates a local instance of the standard ETSI NFV-MANO [19] interacting with the broker. Although different technology domains may belong to a distinct administration, the solution assumes a unified orchestration and management provided by only one administrative domain. Such unified orchestration and management act as an aggregator without supporting service federation to form an E2E multi-domain NSI [28].

Furthermore, some relevant research projects have been developed worldwide in the orchestration of mobile networks, whose items of scope of the desired works, as well as features to be resolved, are illustrated and compared with our 5G-H proposal in

Table 1. Scopes of work on research projects in the mobile networks´s orchestration. Based on [37,54].

Class	Feature	5G-H*	VITAL-	5G	T	5GEx	NECOS	5G	5G-T	MATILDA	5G!	SliceNet
			5G [63]	NORMA [64]	NOVA [65]	[66]	[67]	TANGO [68]	[69]	[70]	Pagoda [71]	[72]
Network	Access	√	√	√	X	√	√	√	√	√	√	√
	Transport	√	√	X	√	√	√	√	√	√	√	√
	Core	√	√	X	X	√	√	√	√	√	√	√
	Data Center	√	X	√	√	√	√	√	√	X	!	!
Technology	Cloud	√	√	√	√	√	√	√	√	√	√	√
	SDN	√	√	√	√	√	√	√	√	√	√	√
	NFV	√	√	√	√	√	√	√	√	√	√	√
	Legacy	√	X	!	√	√	X	⨀	!	!	!	!
Domain – Provider(s)	1. Intra	√	√	!	√	X	√	√	√	√	√	√
1. Single	1. Inter	√	√	√	√	√	√	√	√	√	√	√
2. Multiple	2. Multiple	√	⨀	!	X	√	√	X	X	X	√	√
Orchestration Functions	Services	√	√	!	√	√	√	√	√	√	X	√
	Resources	√	⨀	√	√	√	√	√	√	√	√	√
	Life Cycle	√	√	!	√	√	⨀	√	√	√	√	√
Open Source		√	√	√	√	⨀	√	√	⨀	√	√	!
AI in Orchestration		√	X	X	X	!	⨀	X	!	X	!	√
E2E QoS		√	!	X	X	√	⨀	√	!	!	!	√
Regulatory Framework		√	X	X	X	⨀	X	X	X	X	X	X

* 5G-H: 5G-Horizontal, according to the further proposal of this work which is explained in Section 4. Fully; Partially; Out of Scope; Undefined.

.

4. Proposed Orchestrator

The general conception of our proposed Orchestrator (5G-H) is depicted in Figure 2. Thus, it presents an overview of our Orchestrator—Operating Time Dynamic Multi-Provider Orchestrator for 5G and Future Generations Mobile Networks (5G-Horizontal)—model, which highlights the same delimited in purple by the “box” identified by Orchestrator.

It allows inferring that the workflow (tasks) to be considered in the Orchestrator’s coordination resources must be as follows: manage and orchestrate the different SDN and NFV technologies implemented; implement a horizontal network division scheme that allows for the efficient realization of the different expected 5G verticals; allocate the necessary wireless resources; finally, monitor the different components of the 5G network. Note that 5G-Horizontal should be in sync with local Orchestrators across the networks (access, transport, and core) [73].

Figure 2 shows this proposed conceptual Orchestrator model. It will work horizontally in operating time, assessing users’ needs and directing their requests to the network infrastructure with the best execution routes and the lowest costs. The Orchestrator may know the data being obtained, at all times, from the network operators.

From now on, it is emphasized that for the whole work process (extraction of application requirements, definition and negotiation of NSs and guarantee of services) to occur in operation time at 5G-Horizontal, it will be necessary and important to use Big Data, AI, and their respective Machine Learning (ML) techniques to predict results. In this sense, state-of-the-art methods will be advanced following the trend of [74,75,76,77,78], within the context of a more ambitious proposal because it deals with a more complex problem: the horizontal handover.

Furthermore, vital is the support of Distributed Cloud Computing towards meeting latency [4], which may be low at the millisecond level [79]. Unlike 4G orchestration, which is vertical, the 5G Orchestrator proposed here must comply with the principle of the automated and horizontal orchestration, making it possible to perform measurements to verify whether QoS, QoE, and SLAs are being met.

Therefore, efforts are needed to develop algorithms and methods capable of managing resources and providing advanced and reliable 5G services with appropriate QoS for end-users and generating an economic system [56]. For all these reasons, the proposed Orchestrator will be extremely important in the regulation models since it may offer data for newer regulation definitions.

Therefore, for composing the system, concepts and tools of AI, Distributed Cloud Computing, Big Data, Isomorphic Cryptography, and Prediction shall be used under a private computing cloud environment, where the system components are software elements that will operate in an integrated manner.

The implementation has been conducted through open source solutions and tools, such as the Open Network Automation Platform (ONAP) [80], so the final cost is as low as possible and updates and maintenance are facilitated. Whenever there is no ready-made block of free access to be used and integrated, it will be developed. Thus, the intention is to use the maximum of existing components for the system implementation.

Digging deeper, Figure 3 shows the process for establishing and managing the NS in terms of the Orchestrator’s view, from which the workflow should be executed as follows:

1.

First, the 5G-Horizontal receives the description of the service request to be provided (represented by the arrow Service Request) in a standardized format from the Business Function layer [23,31], with the definitions established in [81]. Moreover, for executing QoS parameters and SLA analysis (QoE), an instance that will be hosted at the edge of the network, as described in Section 4.1.4, which uses multicriteria decision-making methods, will analyze the parameters at run time, such as mapping (labeling) from the radio base station (gNodeB). As observed, the Service is set up based on historical service data stored in a Big Data structure, in which AI algorithms are used to estimate service resource requirements according to operation history to fit network slicing parameters to packages flow needs.

2.

The Service Request is received by the Application Requirements block, whose function is to generate the information for defining the NS. This follows the description of the services, from which the network performance requirements necessary for providing the service with the quality requirement contracted by the end-user to the SP are extracted, that is, a table of QoS parameters is defined from generic templates existing in the database of the block and filled with the respective values necessary to serve the end-user. AI tools and service history databases will be used to implement this block.

3.

The data tables generated by the Application Requirements block are received by the Network Slice Definition block, which accounts for generating the NS through a series of generic SLAs (types of NSs), which will be sent to the Network Slice Negotiation block so that the NS can be negotiated with the TIPs and MNOs.

The parameters from SLAs are generated using the tables received and information about the locations where the service will be provided, including mobility, communication infrastructure available from TIPs in the areas in question, and information of operation related to MNOs that serve specific areas. Bearing in mind that, as the orchestration is horizontal, all the existing infrastructure must be considered regardless of which TIP or MNO the technology used belongs to.

The infrastructures that provide coverage in the target locations of the services will be the candidates, with the rest being discarded. It is worth noting that, in the case of mobility, it may be necessary to carry out a horizontal handover procedure when a candidate does not cover all areas. Nevertheless, it will not be discarded, as it may still be chosen to provide connectivity in the area it serves.

With the infrastructures chosen by the criterion of physical coverage, an analysis will be made that lists which infrastructures meet the requirements present in the tables. After that, a list ordered by degree of adherence to the required QoS parameters will be generated and horizontal NSs will be defined using, if applicable, different MNOs for each telecommunications service (for example, core and access).

4.

To generate the list of candidate NSs, the Network Slice Definition block simulates its operation for each of the NSs present in the list, considering the data present in the operational databases.

5.

After the simulation, the candidates will be sent to the Network Slice Negotiation block through a set of SLAs.

The use of AI tools is essential for operating the Network Slice Negotiation block due to the need for negotiation processing time. Note that the infrastructure database related to the use of the network must be populated and maintained with the support of the TIPs and that the operation database can be negotiated with the MNOs so that their data can be used, ensuring security and privacy with the use of isomorphic encryption. This is a sore point. If it is not possible to obtain this information, it will be sought from open sources or created from the previous work experience of our own Orchestrator. Thus, if it is not possible to use the MNOs operating databases, they can be created with the information collected by 5G-Horizontal during its operation.

6.

Then, in the opposite direction, the Network Slice Definition block receives QoS data from the services and draws up a new list of candidate NSs if it is not feasible to establish the NSs for the services in progress and whose requirements are not being met (impossibility of horizontal handover). This feedback is part of the proposed Orchestrator’s dynamic and operating time specification, and also to popularize the operation database, which has a Big Data structure.

7.

Candidate NSs are sent to the Network Slice Negotiation block, which accounts for negotiating and establishing NSs through the negotiation of SLAs with the MNOs involved in the slicing in question (horizontal orchestration).

At this point, only MNOs with the possibility of negotiating SLAs online are observed to be in the candidate NSs, which, in the future, will encourage the softwarization and virtualization of communication infrastructures, in compliance with the 5G philosophy.

The negotiation will take into account the operational parameters of the MNO, as well as its statistical behavior, seeking to predict the behavior of the infrastructure in question during the provision of the service. The ideal, in this case, is to have access to the operational information of the MNO in question, which would be accomplished through agreements, and the use of isomorphic encryption is foreseen to guarantee the security and privacy of the data from User Equipment (UE) to gNodeB at the application layer, but it can also be employed in the aggregation and transport networks.

If it is not possible to use this information, a predictive algorithm may be used based on data from operations already performed and present in the Orchestrator’s Big Data and AI techniques, which will calculate the risks of not meeting the requirements during the provision of the service to the end-user. In other words, the handover risk must be calculated.

Again, simulation tools should be used to simulate the operation of the NSs at this point to estimate the prices offered by the MNOs involved in the candidate NS. This simulation will be carried out for each of the candidates, and the lowest price will be listed. The others are on the waiting list so that in case there is a need for handover, it is not necessary to repeat the whole process but just confirm it.

8.

In the next step, the Network Slice Negotiation block sends the SLAs to MNOs, and then negotiates and accepts them by those MNOs.

This is a sensitive point of the proposed solution because it is currently necessary to have specific solutions for each set of network equipment until there is a standardization of the interfaces. It is noteworthy that 5G-Horizontal will not act directly on the hardware, but rather, it should invoke the Orchestrators/local solutions in their respective domains, that is, invoke the APIs of each local solution to establish the NSs E2E.

For the design and implementation of the Network Slice Negotiation block, it will be necessary to have tested the APIs from network equipment manufacturers [34,82].

9.

Conversely, to close the control loop, the Network Slice Negotiation block will collect QoS information from 5G QoS Identifier (5QI) or even, if possible, QoS offered by the networks that make up each NS in operation to perform simulations and predict the future situation of each. If there is a probability above a particular critical value (which will be defined and adjusted throughout the operation of 5G-Horizontal, using AI and Big Data tools), a handover procedure will be performed to avoid discontinuity of service or loss of quality.

The collected QoS data will be stored in the Orchestrator’s Big Data structure and will be used by the Network Slice Negotiation and Network Slice Definition blocks to define the NSs when it is not possible to use the operation databases of MNOs. The feedback of the QoS values, associated with the possibility of horizontal handover oriented to the end-user during the execution of the service, provides the dynamic characteristic of the 5G-Horizontal.

10.

Finally, the existence of the Big Data structure is essential, where information about services and operation is stored, from which data parameters are collected using NetOps and DevOps techniques, populating relational or non-relational bases. Hence, the AI algorithms will consume the dump and make the prediction analysis based on the defined periodicity or business analysis (operation costs), defined periods (historical analysis), and future operations. It is expected that the more the Orchestrator is used, the more precisely and faster it operates. The private distributed cloud computing environment adopted will allow the Orchestrator, which is implemented in software, to operate adequately and safely.

4.1. Description of the Architecture

In addition to the conceptual model (Figure 2), we also propose the Orchestrator architecture. To understand the inherent complexity, this architecture of the 5G-Horizontal is illustrated in Figure 4. Its main components and modules are distributed in large blocks, which contain functionalities that express the proposals of the standardization entities, particularly the ETSI MANO framework architecture [40,83].

The network services are implemented from several layers of virtualization, constituting integrated blocks from the use of REST APIs. A description of the large blocks and the functioning of their internal structures are detailed below.

Similar to others [28,83,84,85,86], this work follows the technical reference of the standardization bodies [35,37,40]. However, the architecture proposed herein presents an interesting combination between the features provided by Edge and Cloud Computing, thus enabling a promising differential for an efficient orchestration service.

Several architectural proposals are found in the literature on vertical models, in which SLA guarantee mechanisms are proposed for specific applications through well-defined templates in an E2E architecture, such as 5GTANGO, 5GEx, 5G-Transformer, 5G EVE, 5G-VINNI, 5GENESIS, 5GROWTH, and 5G-VICTORI [66,68,69,87,88,89,90,91]. Nevertheless, a strategy that has shown good results is to decouple and distribute computational resources between the edge and the cloud. For these cases, the integration between specific network functions (edge and cloud) helps to support different types of applications and services, mainly satisfying the requirements of QoS/QoE. The NECOS architecture, as well as 5G!PAGODA, 5G NORMA, MATILDA, 5G-Crosshaul, and 5GUK architectures, shows this integration in detail [83,92,93,94].

Our proposal brings together the integration characteristics reported in the aforementioned projects, with a greater focus on the selection of slices in the Edge cloud. This path has proven to be quite interesting in reducing latency and ensuring a better user experience, recovering the concept of Always Best Connected, in environments of heterogeneous and multi-provider technologies [95].

In this article, we only highlight these modules since the functionality of all other entities is similar to those stipulated in the ETSI MANO framework. As illustrated in Figure 4, the architecture consists of the following functional blocks: Multi-Provider Service Leader Plane, E2E Network Slice Orchestration Plane, and Logical Multi-Provider Network Slices. The last block mentioned consists of network functions that comprise aspects of infrastructure management, the separation between control and data plane, and edge functions, especially the slice selection service.

The mapping between components and modules from the Orchestrator architecture (Figure 4) to the conceptual model (Figure 2) is according to

Table 2. Mapping between components and modules from the Orchestrator architecture to the conceptual model.

Proposed Orchestrator Architecture	Conceptual Model
Intelligent Service Leader	Application Requirements
Mobility Manager	Application Requirements
Multi-Provider Network Slice Functions	Infrastructure Layer, Network Function Layer
Multi-Provider Network Slice Selector	Application Requirements
Multi-Provider Slice Manager	Application Requirements, Network Slice Definition, Network Slice Negotiation
Multi-Tenants	Services
Physical Resources	Infrastructure Layer
Service Manager	Network Slice Definition, Network Slice Negotiation
Slice Life-Cycle Manager	Network Slice Definition, Network Slice Negotiation
Specific-Provider Connectivity Control	Network Slice Definition, Network Slice Negotiation
Specific-Provider NFV-MANO	Network Slice Definition, Network Slice Negotiation
Virtual Resources	Network Function Layer
Virtualization Layer	Network Function Layer
United Connectivity Resource Manager	Network Slice Definition
United Distributed Cloud Mediator	Network Slice Negotiation

.

4.1.1. Multi-Provider Service Leader Plane

It performs service orchestration and management across federated resources from successfully admitted slice requests. In this article, we propose a set of functional blocks that integrate the physical and logical infrastructure of mobile operators and SPs, aligned with a horizontal orchestration service.

To that end, network functions have been proposed that implement 5G network slicing using Fog/Edge and cloud computing (Figure 4). In general, integration with the orchestration service takes place based on the perception, definition, selection, or creation of the best slice in a given coverage area. In this case, the slice selection service using computational intelligence techniques and, using SDN-based traffic management, selects the slice that has the best requirements (QoS parameters) to meet a given user. Alternatively, from the negotiation of an SLA, it defines the necessary metrics for the slice, serving as input to the Mobility Manager module.

The Mobility Manager, in turn, makes use of traffic prediction techniques and consults a database of available network services (BD Services/Network module), which act as a kind of catalog that shows the health of the network in real-time, using monitoring tools (e.g., Prometheus ecosystem (https://prometheus.io/docs/introduction/overview/ (accessed on 7 December 2021)). It then checks the network panorama and executes the mobile operator’s handover to satisfy the requested requirements.

However, for the handover decision to take place efficiently, the prediction and heuristics information provided by the Intelligent Service Leader module is used, which contains a set of data analytics algorithms and techniques to make traffic prediction, that is, to ensure that the user has the SLA met while avoiding the ping-pong effect, in which the user is switched between the slices available in short periods, thus reducing their QoE [96].

The signaling performed by the Mobility Manager module is then received by the Multi-Provider Slice Manager module, which sends the resource allocation model, verifying the connectivity services and computational resource capacity that should be made available. This request (e.g., TOSCA or YAML template) is then sent for provisioning in the E2E Network Slice Orchestration Plane block, which will be detailed later.

It is important to note that unlike the 5G NORMA project [64], our slice selection service is shared between the edge and the cloud. Another relevant aspect regarding the participation of the Multi-Provider Slice Manager module consists of verifying the prediction models previously provided by the Intelligent Service Leader proactively scaling the VMs and/or containers for the orchestration service, which uses specific VNFs to allocate these resources in the Core Network.

The United Connectivity Resource Manager negotiates connectivity across different administrative domains. The United Distributed Cloud Mediator guides and interprets the slice requirements related to VNFs and value-added services across heterogeneous platforms.

Together, the United Connectivity Resource Manager and United Distributed Cloud Mediator map the resources needed for the slice, following the specification defined by the previous modules (e.g., Mobility Manager and Intelligent Service Leader).

4.1.2. E2E Network Slice Orchestration Plane

The E2E Network Slice Orchestration Plane block is under the ETSI MANO framework [40], and thus, the modules that compose it implement the framework functionalities. In general, an NS is negotiated directly between the end-user (slices are dedicated per UE) and the network operator, as described in the previous section. The end-user requests using its consumption profile (QoS requirements), and the slice is allocated according to the defined SLA with the operator [97].

In this sense, several platforms present features and functionalities that provide the orchestration service, highlighting here the Open Source Mano (OSM) (https://osm.etsi.org/ accessed on 7 December 2021) based on ETSI-NFV Management and Orchestration (MANO), Open Baton (https://openbaton.github.io/ accessed on 7 December 2021), and ONAP (https://www.onap.org/ accessed on 7 December 2021). Using these platforms as a base, the other orchestration solutions implement an upper layer of functionality, proposing standardization interfaces, integration, and regulatory models, which is the case of the solution proposed in this work and also in [28,74,84,85,86,98].

Here, the additional modules are being tested on ONAP and OSM due to the acceptance by the technical-scientific community, good documentation of these solutions, and the adoption by major players in the market. The comparison between these platforms is outside the scope of this work but can be obtained from [83,99,100].

The operation of this block is defined as follows: The Service Manager module receives multiple requests from slice templates and dynamic resource allocation. Our Orchestrator considers 3GPP REST as per TS. 32.158 (https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=3396 accessed on 7 December 2021), which will facilitate the integration with regional orchestrators implemented on the ONAP and OSM platforms.

The Service Manager module then processes the template files and starts the necessary actions for provisioning the requested resources. Note that the standard components of the ETSI framework are maintained through the sub-modules: Slice Life-Cycle Manager, Virtualized Infrastructure Manager (VIM), VNF Manager (VNFM), and NFV Orchestrator (NFVO) (see Figure 4). These modules together implement the reference architecture and provide the necessary virtualization technologies for implementing the orchestration service. In addition to the NFV MANO specification module, the block integrates the connectivity verification and provisioning network resources using the SDN Controller module, responsible for managing and executing the necessary controls to establish the transport layer of the service slice requested [100,101].

Finally, a set of REST APIs (southbound clients) connect to the virtual resources of the Cloud, Multi-access Edge Computing, and NFV architecture in multiple providers in order to subsidize the provisioning and delivery of the E2E slice. In this context, E2E slices creation and quality assurance involve several virtualization technologies, access networks, transports, and core networks. It also needs different virtualization technologies (e.g., containers or VMs) and different types of Orchestrators (e.g., vertical and horizontal) [85,99].

4.1.3. Logical Multi-Provider Network Slices

This block consists of two modules: the Multi-Provider Network Slice Selector (please refer to Section 4.1.4) and Multi-Provider Network Slice Functions. The Multi-Provider Network Slice Functions makes the interface between the Multi-Provider Network Slice Selector and the multiple providers in terms of control plane (VNFs) and data plane (PNFs and VNFs).

The infrastructure management performed by each SP uses several layers of virtualization. That is, the TIPs (composed of Core and Edge data centers) must make compute and network resources available from a multi-tier structure.

Hence, several tools and platforms have been used to provide the softwarization layers. In [99], a set of open source tools is presented for each of the layers that enable implementing 5G networks. In this work, in addition to the ONAP and OSM orchestration platfforms, and the SDN controllers OpenDayLight (https://www.opendaylight.org/ accessed on 7 December 2021) and ONOS (https://opennetworking.org/onos/ accessed on 7 December 2021), other tools and platforms are being evaluated. Additional modules implemented have the functionality of providing E2E horizontal orchestration, the specification of interface partners, in addition to providing traffic prediction strategies and SLA assurance using data analysis.

4.1.4. Multi-Provider Network Slice Selector

Selecting the so-called RAN networks in an environment of heterogeneous technologies is a complex problem since operators can provide specific types of slices directly to meet the requirements of an application or multiple slices to meet different requirements of the same user. Therefore, there is still no solution or technique that understands all aspects and mechanisms of access to these technologies [102,103,104]. Furthermore, the implementation of new selection techniques becomes necessary due to the demand in the growing use in vehicular networks, patient monitoring, smart cities, IoT, among other technologies and scenarios involving network convergence, mobility management, and service continuity in 5G networks [95,105].

In general, the literature suggests approaches that consider the following situation: Given a set of criteria or network parameters, verify at any given time and among the available slices, which better fits the user needs, supporting the network exchange (handover process) for the mobile [95,102,106]. In this case, the process of choosing the slice is subject to some criteria [83,107,108].

Our work presents a new approach that employs several techniques aiming at the integration and interoperability between RAN networks and the core of the proposed orchestration architecture based on an efficient and robust Slice Selection Service (SSS) that provides compatibility with the specification standards in progress (e.g., 3GPP, ETSI NFVI, and 5G-PPP).

An overview of the proposed framework to NS selection is shown in Figure 5. The Multi-Provider Network Slice Selector Framework consists of a solution, where a part is executed in the user’s equipment (e.g., smartphones, vehicles, IoT brokers), running as a transparent service, and another part runs at the edge of the network operator.

The framework has three modules that can be configured according to the context of applications, geographic location, scenarios of mobility, strategies of slices selection, and others. For energy saving, the user equipment only signals its consumption profile or user application preference for the framework hosted at the edge of the architecture. That is, no processing is conducted in the mobile device or the IoT broker.

According to Figure 5, the framework proposed is divided into three main blocks: data collection, processing, and classification of NSs. Fundamentally, the criteria for network selection are closely related to the demands or applications in use. Thus, parameters that measure the application QoS, as well as objective quality metrics, for specific applications, such as QoV (Quality of Video), and subjective metrics, such as indicators based on the user experience (QoE) and MOS (Mean Opinion Score), may be considered [109,110].

The Collector Module focuses on the assessment and dynamic mapping of the appropriate QoS requirements for each type of service, in addition to considering the signaling of the User Profile (e.g., V2X, Virtual Reality—VR, Augmented Reality—AR, Video on Demand—VoD, Video Stream), monetization, and geographic location [109,111,112]. The Multi-Provider Network Slice Selector is indifferent to the technique used to mark packets in gNodeB. That is, our architecture assumes that a software instance based on widely used solutions, such as Segment Routing, Multi-Protocol Label Switching, and/or definitions of IPv6 classes of service, have already marked (labeled) the packets [113].

Therefore, our framework only identifies and collects the QoS, QoE, and MOS parameters, processes them, and, from there, defines which NS best meets those requirements or in case of non-existence, signals to the Network Slice Definition block, the parameters for instancing the slice at run time (already in the cloud).

Regarding the Processor Module, several strategies can be used. The most common methods reported in the literature for solving the problem of NS selection include the use of fuzzy logic, Genetic Algorithms, Artificial Neural Networks, and Multiple Attribute Decision Making (MADM) [104,107,108,114,115,116,117]. Among the MADM methods used, there is TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), a widely accepted decision-making tool, considering its understandable logic and algorithmic logic, and mathematical form [95].

The Processor Module preferably uses models that consider hybrid solutions; a feature that has achieved significant results are techniques that include the use of fuzzy logic and MADM methods [95,115,118,119]. In general, these models work in a similar way, that is, after the process of data collection, according to the criteria considered in the Collector Module, the fuzzy logic processing occurs, followed by the classification method via a decision strategy. In this case, each criterion is given a certain weight to prioritize some services over others, guiding the choice of the new slice per the application in use. The definition of weights also takes into account the user’s profile.

For storing the data collected in the measurement processes and the persistence of results related to the processing module, a NoSQL database is used. Data manipulation and analysis can be performed through computational intelligence algorithms, using APIs, and consuming data directly from the Network Data Analytics module, which is based on the Hadoop (https://hadoop.apache.org/ accessed on 7 December 2021) and Spark (https://spark.apache.org/ accessed on 7 December 2021) ecosystem. Note that the framework supports any relational database.

The Decision Maker Module selects the slice that best suits the user’s profile and, therefore, serves as a trigger for the slice change functions (Roaming Functions) in the Mobility Manager block of the Orchestrator architecture. All the operations between the modules and blocks of the framework are carried out via APIs, allowing external access by other entities through RSA key pairs (SSH Keys).

Information about traffic conditions on the slices and extra options for configuring the framework is available on a dashboard in the Graphical User Interface (GUI) or via Command Line Interface (CLI) when accessing the slice selection service. For example, combining more than one strategy or selection method, such as genetic algorithms, fuzzy logic, or multi-criteria decision methods. It is also possible to define which QoS parameters should be considered in the selection process.

All computational methods are implemented using VNFs running in an environment based on OpenStack (https://www.openstack.org/ accessed on 7 December 2021) and Kubernetes (https://kubernetes.io/ accessed on 7 December 2021) [99].

4.2. Solution Considerations

To further explain the orchestration of multi-provider network slicing (i.e., orchestration and management procedures) from the point of view of the Orchestrator architecture, a series of operational procedures are elaborated considering slice configuration and updating.

4.2.1. Multi-Provider Network Slice Configuration

A multi-provider NSI slice is instantiated following the procedure illustrated in Figure 6.

Figure 6 shows the flow of requests, definitions, modifications, as well as the establishment of slices. It appears that the initial negotiation occurs between the equipment of users and the operators’ offers of various slices within a given geographical area. The offer of slices from predefined templates does not inhibit the requests for custom slices, which is one of the main functions of the Orchestrator [28,85,92,97]. This function of mapping and evaluating the best slice is performed by the Multi-Provider Network Slices, as described in Section 4.1.4.

Hence, after the perception of the slice by the end-user suggested by the Multi-Provider Network Slices, the Mobility Manager module requests the Intelligent Service Leader module, which MNO complies with the requested SLA requirements, then initiating the provisioning of the slice, whereby the computational and network resources are verified and allocated for establishing the requested slice.

The proposed orchestrator solution can be deployed in the Core and Edge scenario, assuming scalability according to available computational (pod) resources. The scalability threshold will be marked according to the resources mapped in the VIM (OpenStack or Kubernetes NFVI). This part is not inherent in the proposed solution, which only makes use of resources from the consumption of APIs from the Multiple Providers block (as shown in Figure 6).

The multi-provider slice provisioning follows the ETSI MANO framework and occurs through cooperation between the United Connectivity Resource Manager and the United Distributed Cloud Mediator.

After mapping the resources, the Service Manager module triggers the defined templates, which is a common resource used mainly in the provisioning of vertical applications and/or modifies the template to accommodate the allocation of resources, providing the negotiated SLA requirements. Issues related to allocation time, besides the compliance with QoS metrics, are maintained by the Slice Life-Cycle Manager module, which implements the ETSI MANO specifications. The whole part of softwarization is implemented from the treatment of data flows in the SDN Controller, in addition to the instantiation of the network functions (VNFs) necessary for the services that the slice will serve.

The operation described above occurs in parallel in multi-providers, multi-operators, multi-domains, and under the management and orchestration of 5G-Horizontal. The initial tests have shown good performance in meeting the latency and jitter requirements. However, it is outside the scope of this article.

4.2.2. Multi-Provider Network Slice Modification

Once slices are created, they may be modified. Figure 7 shows the sequence of requests and interactions for NS modification.

Once the Multi-Provider Slice Manager is configured, the Intelligent Service Leader provides the corresponding service decomposition details of the slice request. The Multi-Provider Slice Manager relies on jobs from the United Distributed Cloud Mediator for guidance on heterogeneous platforms. Cross-domain connectivity is established through the United Connectivity Resource Manager. After that, the Multi-Provider Slice Manager establishes secure communication with each Service Manager in the relative administrative domain. It then provides service-type specifics (e.g., SLA and policy) related to the corresponding slice request.

The Service Manager, in turn, performs a mapping analysis to identify the network resources, that is, network functions, value-added service, and connectivity, that correspond to certain technology subdomains, and then informs the Slice Life-Cycle Manager.

The Slice Life-Cycle Manager selects the appropriate slice template and creates the desired slice resource graph. It then carries out the resource configuration toward the corresponding subdomain by issuing a request toward the respective Specific-provider NFV-MANO and/or Specific-provider SDN Controller, which, in turn, needs to create the desired NFV, computing, and connectivity slate. There are two major options when configuring an NFV or computing slate: the Specific-provider NFVO forwards the request directly to the corresponding VIM, or it communicates the request to the relevant VNFM.

When the request directly reaches the VIM, it represents a situation of resource scaling related to a shared VNF resource. However, requests for instantiating VNFs are handled by the Subdomain VNFM. For the connectivity slate, the Specific-provider SDN Controller performs the necessary network configurations to establish the transport layer and related service chain. A multi-domain NSI becomes operational when all domain-specific NS Subnet Instances (NSSIs) and cross-domain connectivity are configured successfully. Once the resources are granted, an Acknowledgment is returned to the tenant, which also updates the Multi-Provider Network Slice Selector.

4.3. Preliminary Contributions

Regarding the end-user, the philosophy adopted for the 5G architecture provides a fundamental change in contracting digital services, whereby the end-user of the service will directly contract the SP for its provision and will no longer need to contract an infrastructure service provider or telecommunications operation in parallel. In this sense, the role of a horizontal Orchestrator makes sense, that is, it chooses the TIP and the MNO best-suited to offer the service (with the guarantee of QoS and QoE) to the end-user in each segment of the network.

In this sense, 5G-Horizontal proposes an advancement in the state of the art. The implementation of NSs must be carried out horizontally, that is, considering all TIPs and MNOs present in the area service provision, regardless of the technology used, but that meet the requirements demanded by the application. Among those that meet the requirements, the lowest price will be chosen.

The NS is established through the automatic negotiation of SLAs, and the connection is established and the service is started. The 5G-Horizontal proposed here then starts to monitor the parameters in a predictive way and, if necessary, to perform a handover, establishing a new NS to guarantee continuity of service. This function must be carried out within operating time and up to the limit of the local Orchestrators [73] of the providers or network operators.

5. Experimental Evaluation

The vast quantity of new technologies introduced simultaneously presents a strong research challenge in defining the method to accurately assess the E2E performance of the network when all technologies are interconnected. The combination of new frequencies, formats, and physical layer codes, edge computing, and virtualized network functions (VNFs/NFV) create an end-chain for potentially unpredictable interactions, which will require an efficient research methodology.

To start proving the operational efficiency of our Orchestrator, we focused some experimental tests on the Multi-Provider Network Slice Selector, which are related to its Collector, Processor, and Decision Maker modules.

First, we defined the service demand scenario worked on in our experiment, according to the specification 3GPP TS 22.186; R.5.4-006 (Performance requirements: extended sensors information sharing between UEs supporting V2X application under a higher degree of automation for an imminent collision scenario) in terms of Maximum E2E Latency, Reliability, Data rate, and Minimum required communication range, as described in

Table 3. Performance requirements: extended sensors information sharing between UEs supporting V2X application under a higher degree of automation for an imminent collision scenario. Based on [120].

Max E2E Latency (ms)	Reliability (%)	Data Rate (Mbps)	Min Required Communication Range (m)
10	99.99	1000	50

[120].

The implementation of the simulation scenario was based on the OMNeT++ 6.0 (pre10/pre11) simulator 11, INET v4.3.2 [121], and on the Simu5G framework v1.2.0 [122]. The choice was due to simulating data forwarding using five UEs, two gNodeBs (5G RAN), 5G Core (5GC), User Plane Function (UPF), Router, 5G-H Orchestrator, and services (Internet or public cloud), in addition to other features of the simulator and framework that could guarantee the detailed implementation in Figure 8. It should be noted that the mobility of nodes (UEs) was considered within the range intervals predefined by the 3GPP TS 22.186; R.5.4-006 specification (Table 3), but that the experiment did not consider issues related to mobility management and handover. It was also considered that the traffic generated by the UEs was of the Constant Bit Rate (CBR) type and, therefore, subject to variations, inferred latency, and reliability due to packet loss.

The simulation parameters have been defined according to

Table 4. Parameters used in each one of 100 simulations per test.

Parameter Description	Specified Value
Each Simulation Time	50 s
Play Ground Size	50 m, 800 m, and 1100 m
gnodeB Broadcast Message Interval	0.5 s
FbPeriod	40
AmcType	NRAmc
Pilot Mode	ROBUST_CQI
Target Bler	0.01
Bler Shift	5
Num Components Carriers	1
Num Bands	25
Mobility UE	0 m (static)
Service Hosts Max Apps	100
Service Hosts Max Ram	32 GB
Service Hosts Max Disk	100 TB
Service Hosts Max Cpu Speed	400,000
UE Start Time	1 s
UE Stop Time	35 s
UE Num Apps	1

.

By applying fuzzy logic, the mathematical technique that works with the theory of sets, motivated by the accurate output supplied from the raw data input, we calculated the degree of membership of the data for each linguistic input variable in each set using the triangular and trapezoidal membership functions with the inference of the Mamdani method over the generated result, as illustrated in Figure 9.

The second step was establishing fuzzy rules for all cases and the application of the fuzzy inference system. As there are four input variables with three sets (Low, Medium, and High), there are 81 fuzzy rules. For the output, as seen in Figure 10, the output fuzzy sets are five: bad, close to good, good, close to great, and great.

In Appendix A, Table A1 and Table A2 show the set of rules used in the experiment. The application is based on the fuzzy logic rule in which the fuzzy operator AND equals the minimum operator and the operator OR equals the maximum operator. Thus, it is possible to calculate the membership function of the latter. For example, apply the rules to the bad output set.

• If E2E Latency is High and Reliability is Low and Data Rate is Low and Communication Range is Low, then discourse universe (mos) is bad;
• If E2E Latency is High and Reliability is Low and Data Rate is Low and Communication Range is Medium, mos is bad;
• If E2E Latency is High and Reliability is Low and Data Rate is Medium and Communication Range is Low, mos is bad;
• If E2E Latency is High and Reliability is Low and Data Rate is Medium and Communication Range is Medium, mos is bad;
• If E2E Latency is High and Reliability is Medium and Data Rate is Low and Communication Range is Low, mos is bad;
• If E2E Latency is High and Reliability is Medium and Data Rate is Low and Communication Range is Medium, mos is bad;
• If E2E Latency is High and Reliability is Medium and Data Rate is Medium and Communication Range is Low, mos is bad;
• If E2E Latency is Medium and Reliability is Low and Data Rate is Low and Communication Range is Low, mos is bad.

Thus, as there is an alternation between these eight cases, the diffuse operator OR must be used, that is, making the maximum of the minimum values already acquired. Finally, there is the value of the degree of membership of the result (in this case, of the bad set). For the other sets, the method is the same; the fuzzy rules of each are checked, the necessary membership degrees are calculated and the fuzzy operators AND and OR are applied.

Finally, the defuzzification process transforms the result obtained from the fuzzy system, the fuzzy aggregated set, into a crisp number from 0 to 5—the linguistic variable called mos, as illustrated in Figure 10. The one chosen in this experiment was the maximum value, which can be applied through Equation (1):

$$x_0={\displaystyle \frac{{\sum }_{i=1}^m{h}_i{x}_i}{{\sum }_{i=1}^m{h}_i}}$$

(1)

where $x_0$ is the output crisp value, m is the number of output fuzzy sets, $h_i$ is the membership degree of each calculated output fuzzy set, and $x_i$ is the center point of the set. The sets bad and good were considered in the interval [0, 2] and [4, 5], respectively.

Once the fuzzy system was built, a simulation was carried out in which the best slice was selected in each of the 10 tests. In each test, there are predefined intervals for the variation of each slice in each criterion. For example, the E2E Latency of slice 01 has the range of 1 ms to 13 ms for the first test, whereas the Reliability of slice 02 has the range of 99,985% to 99,930% for the second test. Details about the simulation parameters and all four criteria for the three slices of the ten tests have a predefined and in a modifiable range, as shown in

Table 5. Set-up of the QoS parameters used in testing.

Test	E2E Latency (ms)			Reliability (%)			Comunication Range (m)			Data Rate (Gbps)
	Slice 1	Slice 2	Slice 3	Slice 1	Slice 2	Slice 3	Slice 1	Slice 2	Slice 3	Slice 1	Slice 2	Slice 3
Test 1	[1, 13]	[3, 14]	[18, 20]	[99.989, 99.999]	[99.986, 99.992]	[99.996, 99.998]	[22, 88]	[6, 34]	[35, 74]	[0.4, 0.8]	[0.8, 1.4]	[0.1, 0.6]
Test 2	[2, 6]	[2, 9]	[5, 11]	[99.991, 99.992]	[99.985, 99.993]	[99.989, 99.995]	[60, 82]	[49, 71]	[43, 75]	[0.5, 0.9]	[0.4, 1.4]	[1.1, 1.4]
Test 3	[3, 5]	[5, 13]	[5, 9]	[99.986, 99.994]	[99.995, 99.999]	[99.986, 99.994]	[50, 96]	[51, 85]	[89, 98]	[1, 1.5]	[1.4, 1.7]	[0.9, 1.2]
Test 4	[1, 12]	[10, 11]	[6, 17]	[99.986, 99.996]	[99.988, 99.998]	[99.990, 99.995]	[12, 38]	[45, 53]	[26, 61]	[0.9, 1.5]	[0.7, 1.3]	[1.3, 2.0]
Test 5	[6, 13]	[1, 14]	[9, 15]	[99.997, 99.998]	[99.990, 99.995]	[99.993, 99.999]	[21, 69]	[40, 86]	[70, 92]	[0.6, 1.6]	[0.7, 1.5]	[1.1, 1.3]
Test 6	[4, 20]	[3, 11]	[6, 11]	[99.988, 99.989]	[99.987, 99.991]	[99.989, 99.999]	[40, 90]	[20, 77]	[34, 85]	[1.3, 1.8]	[0.3, 1.3]	[0.3, 1.2]
Test 7	[5, 7]	[3, 10]	[1, 15]	[99.994, 99.998]	[99.990, 99.993]	[99.985, 99.994]	[23, 51]	[50, 87]	[65, 72]	[0.8, 1.0]	[0.2, 1.0]	[1.0, 1.5]
Test 8	[3, 15]	[4, 11]	[5, 9]	[99.986, 99.987]	[99.994, 99.996]	[99.989, 99.993]	[72, 89]	[10, 33]	[46, 54]	[0.5, 1.5]	[0.9, 1.6]	[0.9, 1.3]
Test 9	[8, 19]	[11, 16]	[3, 17]	[99.987, 99.997]	[99.989, 99.999]	[99.984, 99.986]	[5, 55]	[50, 55]	[50, 58]	[0.6, 0.8]	[0.5, 0.9]	[0.6, 1.7]
Test 10	[5, 16]	[9, 18]	[5, 19]	[99.994, 99.996]	[99.990, 99.997]	[99.991, 99.994]	[21, 26]	[20, 66]	[20, 29]	[0.2, 1.2]	[0.8, 1.4]	[0.7, 1.0]

.

As shown in Figure 8, the UPF segments the traffic (traffic steering) coming from the UEs into three slices according to the interval of the evaluated QoS variables (vide Table 5). Then, the traffic already marked (target) is forwarded by the vrouter to the slice selector, which performs the fuzzification and defuzzification processes on the attributes and, finally, the slice selection. From there, the 5G-H Orchestrator reserves and allocates the necessary resources defined in the slice and forwards the packets to the respective service providers, which, in this scenario, are represented by hosts Service 1 and Service 2.

In each test, there are 100 (one hundred) simulations. In each simulation, values and criteria of each slice are drawn according to the pre-established intervals. Thus, in that simulation, fuzzy logic was applied to each slice, inserting the randomly selected inputs into the system and obtaining the crisp output of each. By comparing the crisp values of the outputs of each slice, it was possible to determine which was considered better. Finally, a new simulation is started, or if a hundred simulations have already occurred, another test is started.

Our experimental strategy presented here can be integrated with orchestration platforms since the Multi-Provider Network Slice Selector may send a JSON array to OSM or ONAP [51,80].

6. Results and Discussion

From the Collector and Processor modules of the Multi-Provider Network Slice Selector, Figure 11 presents the results from the variation of the Data Rate, Communication Range, E2E Latency, and Reliability for each test performed to support the decision of NS selection.

To support the data analysis, there is the percentage of iterations in which some slices were considered better than the others.

Table 6. Percentage of choice for each of the slices evaluated in each test performed.

Test	Slice 01 (%)	Slice 02 (%)	Slice 03 (%)
1	73	23	4
2	78	6	16
3	64	24	12
4	53	11	36
5	22	51	27
6	31	20	49
7	20	35	45
8	26	46	28
9	23	24	53
10	42	30	28
Mean	43.2	27	29.8
VAR	496.62	198.88	262.62
SD	22.28	14.10	16.20
CI	27.25–59.14	16.91–37.08	18.20–41.39

shows this percentage of choice for each of the slices evaluated in each test performed. For example, in the first test, slice 01 obtained 73%, slice 02 23%, and slice 03 04%; thus, in this test, slice 1 was considered the best since it obtained the highest percentage.

For the set of 10 tests performed, the mean, standard deviation (SD), variance (VAR), and confidence interval for the mean (CI) are calculated at a significance level of 95%. Note that each test consists of 100 iterations and follows the defined configurations used in the set of rules for the fuzzification process.

After obtaining the data resulting from the fuzzy selection process, a descriptive analysis was performed in order to verify if there are significant differences in the performance of the slices for the set of tests performed. The experiment primarily consists of a comparative analysis using Tukey’s test of multiple comparisons of means from the VAR analysis. In addition, the Shapiro–Wilk normality, Durbin–Watson independence, and Fligner–Killeen homoscedasticity tests were applied.

From the analysis of the test results and the Tukey test, it was observed that the selection means between the slices do not differ significantly, as illustrated in Figure 12. Another point refers to the results from the Shapiro–Wilk normalization test that describes a normal distribution of the samples. By the Durbin–Watson test, it can be stated with 95% confidence that the residuals are not independent. For the Fligner–Killeen test, it was found that the samples have homoscedasticity of variances.

Figure 13 consolidates the results presented in Table 6. Note that, from test 5, slice 01 has its proportion of choice lower than the other slices. In general, this behavior occurs due to the increase in E2E Latency and the consequent decrease in slice reliability. Anyway, from the processing performed, the best choice is slice 01, slice 03, and slice 02, respectively. However, for better understanding the slice selector decision process, it is necessary to verify the criteria values for each of the slices throughout the tests performed using fuzzy logic.

7. Conclusions

Orchestration and E2E control of 5G systems will be vital to effectively coordinate and explore the full potential of 5G technology. This work has elaborated a multi-provider orchestration and management architecture and framework to address the service challenges of network slicing when utilizing federated resources. In particular, a Multi-Provider Service Leader plane is introduced considering:

• Its main functional components, including the Multi-Provider Slice Manager, United Connectivity Resource Manager, and United Cloud Mediator elements.
• Interworking issues with the conventional single administrator Fully Fledged network domain, wherein NSSIs are established by combining computing, storage, and network slates with RAN, transport, and core network capabilities.

The main operations are explained considering a multi-provider NSI instantiation and management, also providing insight into the further architectural and operational challenges.

A Multi-Provider Network Slice Selector is also introduced and tested to solve issues in the RAN and edge of the cloud.

The final result of this work aims to achieve the following contributions worldwide:

1.

To guarantee the fulfillment of the requirements of each application and, therefore, better QoS and experience to users;

2.

Ensure flexibility for new business models, which will imply new services and better prices for users;

3.

Enable the improvement of competition, which will also imply better prices for users;

4.

Facilitate the establishment of regulatory models and, consequently, greater control and organization of the regulatory body;

5.

Improve sustainability, as all the resources of 5G networks will be better utilized and intelligent applications may effectively and simultaneously exist, minimizing the consumption of, for example, electricity, fossil fuels, and so on.

Further research is essential to bring a pioneering study of computational processing and orchestration structures towards the requirements of better performance, resilience, and international standardization of 5G and next-generation mobile networks. In this sense, our future work will focus on integrating and implementing the whole 5G-horizontal Orchestrator from the definition of the conceptual model to all the blocks proposed in its architecture to have a complete framework and then reduce regulatory barriers as well as improve business models for expanding and improving the performance of 5G networks and beyond.