A Human-Centric Model for Sustainable Asset Management in Railway: A Case Study

Abstract

The ongoing digital transformation is changing asset management in the railway industry. Emerging digital technologies and Artificial Intelligence is expected to facilitate decision-making in management, operation, and maintenance of railway by providing an integrated data-driven and model-driven solution. An important aspect when developing decision-support solutions based on AI and digital technology is the users’ experience. User experience design process aims to create relevance, context-awareness, and meaningfulness for the end-user. In railway contexts, it is believed that applying a human-centric design model in the development of AI-based artefacts, will enhance the usability of the solution, which will have a positive impact on the decision-making processes. In this research, the applicability of such advanced technologies i.e., Virtual Reality, Mixed Reality, and AI have been reviewed for the railway asset management. To carry out this research work, literature review has been conducted related to available Virtual Reality/Augmented Reality/Mixed Reality technologies and their applications within railway industry. It has been found that these technologies are available, but not applied in railway asset management. Thus, the aim of this paper is to propose a human-centric design model for the enhancement of railway asset management using Artificial Intelligence, Virtual Reality, and Mixed Reality technologies. The practical implication of the findings from this work will benefit in increased efficiency and effectiveness of the operation and maintenance processes in railway.

1. Introduction

The digital transformation within various industrial domains leverages the use of various technologies. These new technologies encourage research in the domain of scientific development such as application of Virtual Reality/Augmented Reality/Mixed Reality (VR/AR/MR) or XR and Artificial Intelligence (AI) in various industrial domains. Some of these industrial domains that have implemented XR technologies include health [1,2,3], aviation [4,5,6], manufacturing [7,8,9], mining [10,11,12], and railway [13,14,15]. VR immerses users in a fully virtual environment by using head mounted devices and hand controllers [16]. In AR, users can see and interact with the real world while digital contents are added/overlaid to it [16]. In MR, a user remains in the real-world environment while digital content is added to it; moreover, a user can interact with virtual objects [16].

Researchers are active in providing virtual training using XR technology in the railway domain. The ARIMA (Augmented Reality and Image Processing in Maintenance Application) platform allows technicians to intervene in a machine breakdown using a distant expert [17]. Researchers have also investigated challenges and applications of AR technologies and their feasibility in maintenance operations [18]. In addition to this, authors have looked into the application of AR in industrial environments particular to maintenance sector and development of framework for AR integration in maintenance systems respectively [19,20].

Based on the initial idea of Granström [21] and a literature review, various XR technologies are available but not applied in railway for Asset Management (AM). Thus, there is a need of a holistic human-centric design (HCD) model that provides the railway organisations with a walkthrough how to implement these technologies for sustainable AM by considering User Experience (UX) discussed at Section 3.1. Generally, in AM, an asset is considered an item, thing, or entity with potential or actual value to an organisation [22]. The assets included in this research are railway infrastructure as well as natural assets. Trafikverket (Swedish Transport Administration) is actively monitoring tall trees that can have a risk of falling and creating accidents. The proximity of water bodies has a significant effect on track degradation. This research work is restricted to geo located assets, meaning that they have a fixed position or that its position changes marginally (e.g., grows and shrinks). Rolling stock is, therefore, out of the scope of this research study. The reason is due to the nature of the dataset (Light Detection and Ranging (LiDAR) and 360° images). Currently, railway assets such as mast, catenary, and beam bridge are the assets being considered for this preliminary research work.

To extract all these railway assets from the dataset Artificial Intelligence (AI) has been applied. AI is understood here as any technology which, in the presence of input data, can perform tasks which require complex reasoning. More particularly, technologies which automate the generation of MR environments and the interaction with humans will now be discussed. These are divided here in three main areas: computer vision (CV), speech recognition, and LiDAR data processing.

CV algorithms can be used for identifying and localizing assets of relevance in 360° images-Such images could form the background of VR environments where asset information can be accessed upon request from the user. Simple and intuitive AIs for CV are often preferred when the subject, objects and images have low complexity and variability. Simple CV AIs such as Expert Systems have been proven effective in a variety of tasks, such as the steering of drones inside tunnels [23,24] or the detection of cracks in railway tracks [25]. Object detection and localization are nowadays dominated by Deep Learning (DL) techniques in the cases where images and objects have high complexity and variability. This dominance over Expert Systems is only recent, due to advances such as the crowdsourcing of images [26], the larger explainability of convolutional Networks [27] or the more efficient training of networks [28,29].

Two traditional Deep Learning CV problems provide with value in the context of this paper: (i) object detection and (ii) instance segmentation. In the object detection problem, images are analysed to detect multiple instances of assets of interest as well as to return bounding boxes indicating the location of such assets [30]. Instance segmentation also targets the detection of multiple assets, but the difference is an enhanced localization by identifying all the pixels in the image which belong to each specific asset.

Additionally, there is a potential to exploit the 360° images to detect faults which could be communicated to the user of the MR environment. CV methods for inspection of assets in the catenary system often include two steps: component localization and defect detection. Previous work on CV applications in Overhead Catenary Systems Include: the detection of bird nests reported in [30], a system has been introduced in [31] to detect obstacles between the train and the contact wire, the automatic extraction of images of droppers [32], detection of split pins [33].

Audio Speech Recognition (ASR) may be used for Human Computer Interaction (HCI) in order to e.g., vocalize instructions to access or store asset information. ASR systems are often composed by a Voice Activity Detection (VAD) and an Audio Feature Extraction (AFE) [34]. The goal of the VAD is to extract audio segments containing speech [35,36], which will be further processed for interpretation with support of the extracted features from the Audio Spectrum Flatness (ASF). A widely used ASF method is the Mel Frequency Cepstral Coefficients (MFCC) [37]. The achievement of a semantic interpretation enables the HCI to place simple requests. However, the ubiquity of assets will impose additional challenges related to pervasive interaction.

LiDAR data processing consists of point cloud generation because of 360° or limited angle scans. The point cloud data consists of position (X, Y, Z) and intensity data, and some may have color (R, G, B) data. This data has been pre-processed and used for clustering and segmentation to generate 3D models of the railway assets for XR environments in this research work.

Creating content for MR representations requires that geometries of arbitrary complexity need to be approximated by a number of convex primitives such as trihedra and triangles. The hardware capabilities will place limitations on the number of primitives to be used [38]. Delaunay Triangulations [39] are particularly well suited for connecting LiDAR point clouds with trihedra due to the geometric properties which imply that points are connected in a nearest neighbor manner and that triangles with large internal angles are favoured.

The goal of this research work is to enhance the railway AM by proposing a holistic HCD model using AI and XR technologies. The proposed model will help the railway stakeholders in their decision support systems who work in different life cycle stages. The proposed model will facilitate the following possibilities to enhance the railway AM by:

• providing the possibilities of using XR technologies to create virtual environment to visualise the condition of the assets (e.g., railway assets lie within the middle of a forest, remote area, or during COVID-19 pandemic-like conditions) remotely on one’s desk
• visualising the areas where maintenance has been done and needs to be done by matching the real coordinates of the real assets
• entering direct remarks in the maintenance register from visual inspections of various railway assets
• automating the matching of railway asset registry in Computerised Maintenance Management System (CMMS) with real visualised objects through VR environment
• Visualising natural assets, for example, monitoring trees that have a risk of falling and creating accidents and water bodies that have a significant effect on track degradation
• visualising and identifying the leaning of railway assets
• making practical use of point cloud for example, by constructing 3D models of railway assets that can be used during the design phase of the life cycle of an asset
• finding objects with fault in the surroundings from one’s desk

The above-mentioned possibilities will consider HCD to understand the context, needs and requirements of the railway stakeholders to augment the Decision-Making Process (DMP) within operation and maintenance. This will further enhance the overall maintenance process as described at Section 4 by reducing the time to decision-making. In addition, the above-mentioned points also show some of the possibilities how the railway stakeholders’ make decisions in real time based on asset’s condition.

In addition, to evaluate the developed prototypes, a study group has been tested within a laboratory setting, involving experts from railways, researchers, and students. These researchers and students are from operation and maintenance background. The whole evaluation process is discussed at Section 5.

2. Related Work

To obtain initial estimates of the scale of XR work in railway sector, a web-based search was conducted. Articles related to XR in railway sector have been explored. The popular databases used in this study are Scopus, Google Scholar, Web of Science, and the Institute of Electrical and Electronics Engineers (IEEEXplore) Digital Library. Keywords used for the search are “Virtual Reality”, “Augmented Reality”, and “Mixed Reality” along with Railway. Example of search string in Google scholar is: “allintitle: Railway AND “Virtual reality” OR “augmented reality” OR “mixed reality””.

Table 1. Comparison of existing solutions specific to railway.

Authors	Years	Railway Assets Involved	Main Points
Hermant and Herrmann [40]	2011	Railway station (RS)	PTV VISSIM Traffic simulation software 3D Studio-Max for object creation
Guan et al. [41]	2013	Traction and braking control	3D computer graphics technology Direct3D, a graphics application programming interface (API)
Egger [15]	2016	RS	Using Definitely Affordable Virtual Environment (DAVE) projector with lightweight LCD shutter glasses
Wei et al. [42]	2018	RS	3D models of railway buildings 3D software, like,3DS Max and Maya
Xu et al. [43]	2018	Simulated 3D railway accidents scenes involving train, catenary, and railway crane	Using scale-invariant feature transform (SIFT) CV algorithm HTC Vive glasses
Liu et al. [13]	2019	Catenary	Based on convolutional neural networks (CNNs)

shows the comparison of the existing XR solutions within railways.

To design the RS, researchers have created a 3D VR simulation model by using PTV VISSIM traffic simulation for pedestrian behaviour and 3D Studio-Max software for 3D object creation [40]. To show the performance of the traction and braking for the train running process, researchers have simulated the train running in the virtual reality system by using Direct3D, a graphics application programming interface (API) [41]. Researchers have developed a guidance system for the RS in a fully immersive virtual environment [15,42]. Researchers have proposed a VR demonstrator for the training of people to learn the railway accident rescue procedure [43]. In addition, researchers have proposed a methodology for defect detection in railway catenary support using CNNs within a VR environment [13].

From the literature review, we can conclude that most of the solutions are using simulation and software to create VR environment and 3D models, but no solution is directly representing 3D models of the real physical assets of the railway using AI. For example, this paper clearly specifies the usage of LiDAR technology for data acquisition and application of Al for extracting 3D models of individual railway assets. The novelty of this research lies in the usage of combined technology, i.e., AI for extracting 3D models and XR technologies for visualisation of inspection information to augment the DMP within maintenance process.

In addition, the search string TITLE: (“Virtual reality” OR “augmented reality” OR “mixed reality”) AND TITLE: (“Asset Management”) has also been searched and only 3 articles have been found in all the above-mentioned databases. These articles are in the field of construction warehousing facility, offshore oil and gas operations, and bio-manufacturing industry [44,45,46]. This means that much work has not been done in the railway AM using XR technologies. Therefore, it is a research gap and there is a need to extend this research work

3. Materials and Methods

To conduct this research work, Trafikverket has provided us data that have been collected through LIDAR scanner mounted over the train [21,47]. These data are transferred to eMaintenance cloud in the form of point cloud. Figure 1 shows research methodology used within this research work starting from data collection to its analysis to generate 3D models (See, Section 4.2), creating XR environments for interaction with digital railway assets (See, Section 4.3), and finally laboratory tests for UX (See, Section 5). The hardware and software used in this research are described in Section 2. The configuration of the computer system used within this research work is:

• OS Name: Microsoft Windows 10 Business
• Processor: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz, 3600 Mhz, 8 Core(s), 16 Loical Processor(s)
• RAM: 64 GB
• Graphics card: NVIDIA GeForce RTX 2080 SUPER

3.1. User Experience (UX)

There are various definitions for UX defined within standards and by researchers. The definition of UX in standard ISO FDIS 9241-210 is “a person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service”. According to Alben [48] UX is defined as “all the aspects of how people use an interactive product: the way it feels in their hands, how well they understand how it works, how they feel about it while they’re using it, how well it serves their purposes, and how well it fits into the entire context in which they are using it”. Thus, it is very much necessary to consider UX that can enhance the usability of the XR solution, which will have positive impact on the DMP process within railway AM. Based on the UX and laboratory tests conducted on various users, the process of evaluation is continuously improving the XR prototypes.

3.2. Human-Centric Design (HCD) Models

According to ISO 9241-210:2019, human-centric design is “approach to systems design and development that aims to make interactive systems more usable by focusing on the use of the system and applying human factors/ergonomics and usability knowledge and techniques” [49]. HCD models are based on the requirements of the human to have great experience, relevance, and usefulness of the application or solution designed and visualised for the virtual and real environments. On the one hand, XR technologies have gone from emerging technologies to established technologies in recent years [50]. But on the other hand, still there are some challenges in using these technologies regarding the UX. One of them is occurrence of Motion Sickness in VR environment that has been tested and validated [51]. In addition, the user inside a pleasant virtual environment endures less motion sickness than in a horror environment [51]. There are also differences in results between the genders, where males were less prone to Motion Sickness than females [51].

The ergonomics of VR has been evaluated by using RULA score to assess interaction methods [52] which can be feedback to HCD models. These models should consider the user’s posture, Field of Vision (FOV), viewing distances and resolution. According to Chang [53] UX is defined by two features i.e., comfort and immersion. Comfort relates to wearability, vestibular sense, visual perception, and social ability, while immersion includes the field of vision, resolution, and gestures and haptic interaction [53].

Therefore, HCD models for XR should be designed by considering UX attributes/features, as some of them are presented in Figure 2 because it’s only the human who must interact with the digital assets within XR environments.

3.3. Wearables

Wearables are the physical devices that can be worn by an individual to track, visualise, and interact with digital assets in virtual or real environment. The weight of these wearables can affect the amount of strain on user’s body [52].

Table 2. List of XR wearables used in research.

	Wearabless	Google Cardboard [55]	Oculus Quest2 [56]	HTC Vive [57]	Samsung HMD Odyssey (WMR) [58]	HoloLens2 [59]
Characteristic
Company name		Google	Facebook	HTC	Samsung	Microsoft
Initial cost		$20	$299	$799	$499	$3500
Type		With Mobile Phone	PC-independent	Headset with a PC	Headset with a PC	Standalone
Platform		Android, iOS	Oculus Mobile, based on Android 10	SteamVR, VivePort	Windows Mixed Reality	Windows 10
Resolution		Smartphone Resolution	1832 × 1920	2160 × 1200	2880 × 1600	2048 × 1080
Display type		Smartphone Display	IPS LCD	OLED	AMOLED	See-through holographic lenses
Field of view		90°	90°	110°	110°	52°
Multiple concurrent users		No	Yes	Yes	Yes	Yes
Controller		Magnet	IR LED-based tracking	Vivecontroller, PC compatible gamepad	Samsung HMD Odyssey	Gaze, Gesture, Voice
Primary input device		No	Controllers	Controllers	Controllers	Gaze & Gesture
Portability and setup		Easy	Medium	Hard	Medium	Easy

presents the general characteristics of XR wearables that are commercially available and were used in research study [54].

The Virtual Reality set-up within the eMaintenance LAB [60] has been shown in Figure 3 that includes an edge computer, HTC Vive, hand controllers, and two lighthouses. To visualize and interact with railway assets, HTC Vive (See, Figure 3, Right Bottom) has been used as a Head-mounted display (HMD) along with two hand-held controllers and two lighthouse base stations in this research. Researchers are active in using HTC Vive devices for VR [61,62,63,64]. Authors have evaluated ten HMDs and their results indicate that HTC Vive performs better with regards to comfort, display quality and compatibility with glasses [65].

HTC utilizes two passive laser-emitting “lighthouses” that are attached to the ceiling in two opposing corners of the eMaintenance Lab (shown in Figure 3 with red circles) to transfer the player’s head movements into the virtual reality. To use the HTC Vive and the HTC Setup Software, an account at the online gaming platform Steam is necessary. This requires a stable internet connection, as both Steam and the HTC Setup software is free to use. Since this device is one of the expensive ones on the market, it is used mainly for academic or industrial research rather than private gaming [66].

For Mixed reality, HoloLens 2 wearable has been used within the eMaintenance LAB. HoloLens 2 is the most advanced MR device available [67]. By wearing this device, we can visualize the extracted 3D models of railway assets merged within the reality. We can also interact with these extracted 3D models of railway using HoloLens 2.

3.4. Gaming Platform

In this research work, Unity^® [68] version 2019.4.18f1, a game engine platform by Unity^® Technologies has been preferred due to many available application programming interfaces (APIs) and good compatibility with a variety of VR headsets [69]. Unity^® runs on Windows and Mac and a Unity^®-built project can be run on almost all common platforms including mobile devices like tablets or smartphones. Unity comes with many benefits; it is an easy and simple platform to learn game development and is a powerful tool that is used by many researchers and professionals [70]. The results of a survey where the top ten most utilized gaming engines and frameworks were listed shows that 23.3% of the games were developed with Unity, 13.3% with Unreal, and 8.9% were developed using Torque [71]. Therefore, the gaming environment for AM within railway is built in Unity^® engine [68]. This engine is used together with a graphical user interface (GUI) called the Unity^® editor, which supports 2D and 3D graphics as well as scripting in JavaScript and C# to create dynamic objects inside a simulation. Visual studio has been used for scripting in C# in this research work.

For VR configuration within UNITY, SteamVR plugin has been installed from the Unity^® asset store. The modern SteamVR Unity Plugin manages three main things for developers: loading 3D models for VR controllers, handling input from those controllers, and estimating what the hand looks like while using those controllers [68].

4. Description of Case Study and Results

Railway is a part of transportation and is complex in nature because of its inherently widely distributed and networked nature with the long lifetime of its sub-systems and components. It is one of the major contributors towards the elevation of the economy of a country. In Sweden, Trafikverket is the government authority responsible for railway infrastructure administration as well as the development of the railway sectors. Railway infrastructure consists of tracks, catenary/carrier wires, wayside monitoring systems, switches and crossings, bridges and tunnels, and railway platforms. In addition, railway is one of the time-critical systems that follow strict timetables for maintenance activities. Maintenance may be seen as a process that consists of combination of all technical, administrative and managerial actions during the life cycle of an item intended to retain it in, or restore it to, a state in which it can perform the required function [72]. The purpose of the maintenance process [72] is to sustain the capability of a system to provide a service [72,73]. The maintenance process consists of number of activities required for managing, support planning, preparing, executing, assessing, and improving maintenance [72,74].

The maintenance process [72] is the context for the users in this research work and we are going to augment the DMP for the users interacting at each step in this process. This is an industrial context and these kinds of technologies have not been commonly used and, therefore, there is less experience when it comes to HCD in the context of operation and maintenance in industry. Therefore, this research considers UX that can enhance the usability of the XR solution, which will have positive impact on the DMP process within railway AM.

In addition, it is sometimes difficult to inspect the physical railway assets (e.g., tracks, switches, mast, catenaries, etc.) in remote areas, under harsh weather conditions and during pandemic situations like COVID-19 [75]. Thus, under such situations, use of advanced technologies like VR [18] integrated with AI have been tested to accelerate the procedure of maintenance activities to save time, cost, and energy. With these technologies, railway personnel can visualize the assets’ condition or inspection information remotely on a screen using VR devices. Visual inspection is an important part of the maintenance process which makes management of assets easier and results in improved railway efficiency. This will further improve the overall efficiency of DMP within maintenance process.

This research work has proposed a holistic HCD model that will enhance the railway AM by using AI and XR. This will further help the railway stakeholders to augment the DMP within all the stages of maintenance process. This model for railway AM has been adapted from conceptual model of E365 Analytics^® and eMaintenance solution for Maintenance Analytics [76]. The 3D Analytics and Immersive Experience provides services for the whole DMP, such as data fusion and integration, data modelling and analysis, and context sensing and adaptation. 3D Analytics and Immersive Experience provides a set of interconnected, loosely coupled services, which can be orchestrated to fulfil users’ demands on decision support. These services are built upon technologies such as big data, data analytics, CV, speech recognition, DL, context awareness, cloud computing, and edge computing. When user interacts with such end solutions his/her experience is very important to consider and, therefore, this model considers the human perspective for the navigability, visibility, satisfactions, motivation, and usability of the solution. The AI integrated human-centric design model is depicted in Figure 4.

This model consists of three components starting with various data sources involving assets’ health data, maintenance data, and point cloud data for constructing 3D models; 3D Analytics and Immersive Experience described in Section 3.1, Section 3.2 and Section 3.3; and decision makers who will be benefited with these technological advancements to manage their assets.

4.1. Data Fusion and Integration

The purpose of this step is to integrate data from various sources i.e., condition monitoring data (Optram), LiDAR data and 3D images of railway infrastructure, and maintenance data into eMaintenance cloud to construct 3D models and access assets information in the later stages of the proposed XR model. LiDAR data are transferred to eMaintenance cloud in the form of point cloud. Most of the data are stored into SQL server database and retrieved later during data modelling and analysis and data visualization steps.

4.2. Data Modelling and Analysis

The purpose of this step is to generate 3D models of the railway assets using AI technology. LiDAR is one of the main data sources of this research work.

LiDAR generates point cloud data having (X Y Z) coordinates, by scanning a laser beam over the area. The reflected beams are used to compute the distance through time-of-flight calculations. The scan results are stored in standard formats with each point having its absolute GPS coordinates or other coordinate systems along with its altitude, intensity, colour information, classification codes etc. In contrast to standard image formats where the stored data of pixels location are relative to real world view, the point cloud data generated by the LiDAR does not have any such relationship. A single LiDAR scan may contain millions of points of which generally 10% fall in the region of interest i.e., railways specific assets. Following Section 4.2.1, Section 4.2.2, Section 4.2.3, Section 4.2.4 and Section 4.2.5 shows various steps involved in the generation of 3D models from point cloud. Figure 5a shows a view of a point cloud containing 8.5 million points.

4.2.1. Point Cloud Pre-processing

Pre-processing of point cloud involves decimation of points, a random selection of 25–50% of points reduces the time and space impact while computations are performed. The final extraction of features for creation of models is taken from the original point cloud. The LiDAR scan in the example data has a range of about 100 m forward but much lesser in other dimensions. (X Y Z) coordinates in the point cloud can be stored in different formats and may have large values if absolute values are used for representation. During computation, such large values can cause integer overflow and generate computing error, hence transformation of data is required. Preservation of relative information is required to maintain the aspect ratio along the (X Y Z) axis.

In addition, the Random Sample Consensus (RANSAC) method has been applied for plane and lines detection. RANSAC is an iterative method to estimate parameters of a mathematical model from a set of data containing outliers [77].

4.2.2. Classification

For clustering and segmentation, density-based spatial clustering of applications with noise (DBSCAN) algorithm [78] has been used in this study. DBSCAN can find arbitrarily shaped clusters and, therefore, very much suitable for LiDAR point cloud data. Thus, point cloud clustering has been performed by applying DBSCAN algorithm as presented in Expression (1).

$$DBSCAN\left(DB, distFunc, eps, minPts\right)$$

(1)

where, DB is the database consisting of dataset to be scanned,

• eps is epsilon that defines the radius of neighborhood around a point x. It’s called the ϵ-neighborhood of x,
• minPts is the minimum number of neighbors within “eps” radius, and
• distFunc computes distance and check epsilon.

This algorithm groups points based on how closely packed the points are while rejecting the points away from the cluster (Figure 5b). However, the algorithm is sensitive to the parameter values and needs to be set based on the application and dataset. After the classification is performed height of clusters and position relative to scan are used to classify the clusters. Figure 5c,d show segmentation for the overhead catenary system and track respectively.

4.2.3. Labelling

Once segmentation is completed through clustering, labelling of assets is performed based on their features. Clusters of point clouds like masts, catenary/cables, beams etc., are extracted and information about their centroid, bounding box as coordinates is extracted and stored in database for further processing. The mast locations are used to form a convex hull for next stage of processing.

4.2.4. Object Extraction

This stage involves extracting objects of interest such as railway masts (Figure 5e). Authors have used voxelisation for the extraction of these assets. Extraction of individual objects involves filtering all the points from the point cloud fitting inside a cylinder or a cuboid at the location of the object within known asset dimensions and its geometry is defined by length (l), width (w), and height (h). The index (i, j, k) of each point in the point cloud is calculated using Expressions (2)–(4).

$$i=floor\left(X-Xmin\right)/l$$

(2)

$$j=floor\left(Y-Ymin\right)/w$$

(3)

$$k=floor\left(Z-Zmin\right)/h$$

(4)

where, (Xmin, Ymin, Zmin) are the minimum coordinate.

The extracted set of points corresponding to individual railway assets are stored as separate files for further processing.

4.2.5. Model Creation

Generation of 3D model (Figure 5f–h) involves generation of a surface mesh of triangles from the point cloud, discarding points not required for surface formation. For 3D model creation, this research has used ball-pivoting algorithm [79].

4.3. Context Sensing and Adaptation

The purpose of this step is to create and adapt visualization models based on UX. In this research work, railway has been considered as a case and three prototypes have been developed based on user’s requirements for visualizing and interacting with railway assets using XR technologies. The support of these XR technologies could improve efficiencies in railway AM by augmenting DMP in the operation and maintenance that will further enhance the maintenance process as discussed at Section 4. For example, fast transfer of asset information via handheld mobile such as smart phones and tablets as well as directly to railway personnel’s desk makes it easier and more efficient to manage assets remotely for inspection and maintenance.

To investigate application of XR technology in the railway, especially in AM, three prototypes as XR proof-of-concept (POC) application for railway AM are developed. These three XR prototypes for AM contains a railway track section (approx. 200 m) as one complete game scene. Once all the 3D models are ready (extracted from point cloud), they were imported into the Unity environment for interaction with them. C# script has been used to create dynamic objects inside a simulation. All associated assets, including mast, catenaries, and beam bridge are included and offer a prominent level of detail. It also contains a relevant code that executes the interactions of the objects mentioned. Thus, users have all the desirable components for an experiment in one consistent package. These three prototypes are developed as a GUI to visualize and interact with railway assets using XR technologies.

Prototype 1 (Figure 6a) uses 3D models extracted from point cloud data (Figure 5f–h) during the second step of the proposed model using AI algorithms. These models consist of railway mast, catenaries, and beam bridge. The present prototype has two views to visualize and interact with railway assets i.e., front view and top view. Within prototype 1 for railway AM, each individual asset is intractable. By using VR laser pointer of hand-held controllers, information can be displayed related to inspection/condition monitoring of particular assets (mast, catenary, and beam bridge) within the VR environment. A person wearing VR glasses can move forward, backward, left, or right in this VR environment. Thus, instead of going out in the field, the 3D models of railway assets are constructed from the point cloud within the eMaintenance LAB to overlay asset’s inspection information as presented in Figure 6a. This information will help the railway stakeholder to visualize health information of the assets and the next due time of the inspection.

Prototype 2 has been developed by using 360° images Figure 6b. By using gaming technology, this 360° VR environment is user interactive. By using VR laser pointer, the condition of the present railway asset can be displayed within the VR environment using hand-held controllers. In prototype 1, three traffic light buttons have been used to show the condition of railway assets (Figure 6b). To visualise 360° images within Unity environment, these images were imported from eMaintenance cloud data source. To play these 360° images with VR glasses following steps within the Unity are required:

• In the Inspector window set Texture Shape of 360° image to Cube and apply the changes and this will create a cubemap of the 360° image.
• Next step is to create a material. In the inspector window of created material, set Shader to Skybox/cubemap.
• Next, drag and drop the created cubemap of 360° image in the material window.
• In the last step, drag this created cubemap material to the scene window.

To interact within this 360° image environment various interactive buttons have been created. On the click of these buttons, visual inspection/condition monitoring information is displayed in the VR environment. The initial idea is to feel how the mixed reality environment will look like when used within the physical real railway environment.

Prototype 3 also uses 3D models extracted from point cloud data (Figure 5f–h) using AI. However, in the mixed reality these digital objects are merged within the real world as shown in Figure 6c. The person wearing HoloLens 2 (Figure 6c) can interact with these 3D models in the real world to see the overlaid inspection or condition monitoring information of the railway asset. The 3D models used for HoloLens 2 are optimized using AI technology.

All three prototypes have been tested on a group of experts from industry and academia. Their evaluation results from interviews, workshops, seminars, and questionnaire show that the proposed model will help the railway stakeholders in augmenting their DMP within operation and maintenance.

5. Evaluations

To evaluate the usage of XR for AM within railway, this study has used qualitative data within laboratory-based settings with a population sample of about 50 users. These users involve railway stakeholders, academic researchers, and a group of students. Some of these railway stakeholders include project manager, investigation coordinator, fleet controller, and business developer IT. Other end-users are mostly from operation and maintenance background.

The whole evaluation process is summarised as:

• Verbal interviews were conducted with railway stakeholders to investigate if the XR solution design meets their requirements and expectations.
• Two workshops and one seminar were conducted with railway stakeholders to get feedback and UX for the developed prototypes. The data collected is usually verbal data.
• Questionnaire has been sent to railway stakeholders, academic researchers, and group of students to get feedback on the developed prototypes. We got responses from 15 users (three female and 12 males). The average result of XR user experience is shown in Figure 7.
• This evaluation process is iterative in nature and provide us continuous feedback for further improvements.

The above-mentioned questionnaire has been developed using XR user experience parameters i.e., learnability, navigability, visibility, satisfaction, usability, and motivation [80,81]. Learnability considers how easy it is for the end-users to interact within XR environment for the first time to become efficient after repetitions. Navigability considers user interactions and movements within XR environment. Visibility considers the display of the information within XR environment. Satisfaction considers meeting the expectations of the end-users. Usability considers how well end-user can use the XR solution to achieve a defined goal effectively and efficiently. The motivation attribute will stimulate the end-users to act i.e., to use these technologies for Railway AM.

Figure 7 shows evaluation results for the above-mentioned parameters. Out of these parameters, usability, visibility, satisfaction, and motivation were rated higher by many end-users. This can be due to their higher interest in using these technologies for augmenting DMP in operation and maintenance. Parameters such as, learnability and navigability were on lower priority for the end-users. This can be due to the completely different interface presented while using the HoloLens device i.e., gesture and haptic interface as compared to traditional keyboard, mouse, and joystick. Since most of the users are experiencing haptic interface for the first time, therefore, navigability becomes an issue which has been observed in the usage pattern. However, users who have spent more than 1–2 h. find no problems during learnability and navigability.

Continuous evaluation of these experiments is important to determine the optimums and the limits of the implemented solutions. The qualitative data used in the evaluation showed a holistic view of the developed prototypes for further upgrades. The whole process of evaluation is iterative to meet the requirements and expectations of the railway stakeholders.

6. Discussions and Future Directions

Digitalisation in railway is bringing positive changes in the operation and maintenance of railway systems. The use of AI and XR technologies have proved to benefit many industries to accelerate the procedure of maintenance activities to save time, cost, and energy. Generated 3D models of the railway assets can be used during the design phase of the life cycle of an asset. These technologies can benefit the railway industry for example, railway personnel can visualize the assets’ condition or inspection information remotely on a screen using VR devices and save considerable amount of time and money. The use of these technologies within railway enhances the UX in visualising railway assets either virtually or in reality. With the use of these technologies, visual inspection data are either overlaid or merged on the railway assets on reality. This will help the railway personnel to directly interact with the railway assets either virtually or in reality. In future, the railway personnel can directly enter remarks in the maintenance register from visual inspections of various railway assets through graphical user interface.

Literature supports that, previously, these technologies were used in the railway for employees’ training and maintenance using a distant expert. There is still a gap in utilizing these technologies for the AM in railway. Thus, this research is an attempt for the applications of AI and XR as supporting technologies and their feasibility for the management of railway assets by proposing a holistic human-centric design model. The results from this research work can be used by railway organisations to augment their decision-making process within operation and maintenance using AI and XR technologies. The results from the three prototypes also show the possibilities of interacting with individual digital railway assets (mast, catenaries, and beam bridge). By using virtual laser pointer of hand controller devices in VR environment, inspection or condition monitoring information are overlaid on individual railway assets.

In future, this research will use Building Information Model (BIM) based on cloud technology and combining BIM with MR. Combining BIM and MR with cloud technology will realise real-time data correlation and automate the matching of railway asset registry in Computerised Maintenance Management System (CMMS) with real visualised objects through MR environment using GPS location data. This will help to upload the results in real time directly into the registry system through the mobile phone, tab, or other devices remotely. In addition, 3D LiDAR scanner combined with BIM in operation and maintenance can realize the monitoring of railway and natural assets deformation. The extracted 3D models will also help in the designing of the railway assets in future. We will also investigate more into latency, movement, speed, and environment optimisation within VR environment.

7. Conclusions

A human-centric design model has been proposed that will help the railways to enhance AM using AI and XR. This model will provide the opportunity to railway stakeholders to interact with digital assets of the railway virtually and in reality. It will further help them in augmenting their DSP within operation and maintenance to further enhance the maintenance process. Based on the results, it has been concluded that the proposed model will benefit in increased efficiency of the operation and maintenance processes in railway. In addition, the constructed 3D models of railway assets within this work can be used during the design phase of the life cycle of an asset.

The resulted prototypes have been evaluated within laboratory-based settings and it has been found that end-users and railway stakeholders are highly satisfied and motivated with these prototypes. Thus, the developed prototypes within the railway context provide the feasibility of using AI and XR technologies for railway AM. This will further lead to augment the DMP within operation and maintenance processes in railways.

In this research work rolling stock is out of the scope and the reason is due to the nature of the dataset (LiDAR and 360° images). Currently, railway assets such as mast, catenary, and beam bridge are the assets being considered for this preliminary research work.