Immersive Safe Oceans Technology: Developing Virtual Onboard Training Episodes for Maritime Safety

Abstract

This paper introduces four safety training episodes and virtual training technology called Immersive Safe Oceans which can be used in further education of professionals in immersive training scenarios. These episodes were developed for maritime safety and are under testing. Immersive Safe Oceans Technology is a cost-effective, portable technology that can be used on board just in time or in maritime training centers. Four introduced episodes, namely, (1) command bridge, (2) machine room, (3) crane, and (4) fire safety, illustrate how Immersive Safe Oceans technology can be used in various professional training scenarios. These episodes also emphasize the growing need for virtual reality training in the shipping industry. As a result, next generation learning will happen onboard in sophisticated virtual training centers.

1. Introduction

The global virtual reality (VR) and augmented reality (AR) market is growing at an impressive rate. In 2018, the virtual reality market was expected to reach a value of 12.1 billion US dollars. The financial institution Goldman Sachs estimates VR and AR to grow into a $95 billion market by 2025 [1]. The continuous demand for VR and AR technologies derives primarily from the creative industries [2], but it is used now effectively in many business sectors such as the engineering, education and training, medicine, logistics and transport, and others. Virtual reality is defined as a method of interacting with a computer-simulated environment [3]. Augmented reality, in turn, is a technology which allows computer generated virtual imagery to exactly overlay physical objects in real time [4]. Statistics shows that both VR and AR gaming have not achieved a dominant role in entertaining business, but they underline the readiness level of VR and AR technologies to be used in the various domains listed above. Google’s ARCore and Apple’s ARKit have been available for developers since 2017. ARCore and ARKit are frameworks for building AR applications. Despite technological advances, Ingress and Pokemon GO are still the biggest and only success stories in AR gaming. According to Google Play statistics, Ingress has at the moment more than 10 million downloads. AppStore, in turn, has reported Pokemon GO achieved 1 billion downloads. At the same time, one of the most successful VR games, Beat Saber, has surpassed 2 million copies sold [5].

Both AR and VR technologies were initially designed for the gaming markets. The statistics above show that gamers are not yet widely using these technologies. At the same time, various other domains, such as the technology industry, started utilizing these affordable technologies that were designed originally for consumer markets. Particularly in education, the future of VR seems to be very promising. Due to the interactive content accessible by anyone, anytime, and anywhere, VR transforms the industry effectively and impressively. Allied Market Research [6] has predicted that the global virtual training and simulation market size will grow to USD 329 billion by 2022, and its compound annual growth rate in the period 2016–2022 will be approximately 16.8%.

One industry that indicates impressive adaptation to VR is the maritime sector, where technology and VR, in particular, impacts significant investments, decisions, and organizational structures on maritime training. There are many examples of initiatives by leading organizations in the maritime sector that approached safety training with VR and other related technologies [7]. Some of them are the following:

• Lloyd’s Register showcased during the last SPE Offshore Europe its VR Safety Simulator and gaming technology in order to raise awareness about the critical area of safety and risk in the oil and gas industry;
• Korean Register intends to develop a VR training simulator using a realistic ship environment to provide surveyor training on the classification rules and inspection procedures;
• Mitsui O.S.K. Lines, Ltd. (Tokyo, Japan) developed a mariner safety education tool goggle that uses VR technology created by Tsumiki Seisaku Co. Ltd. (Tokyo, Japan);
• Immerse, a London startup with leading role in the VR learning movement, developed a platform to create bespoke live learning experiences;
• Kongsberg Digital provides VR solutions for enhanced training experiences. The company innovates by integrating mixed reality (MR), VR, and AR;
• Propel developed SAYFR, a 3D simulation model to allow people interact in different scenarios onboard and ashore;
• K Line LNG Shipping (UK) Ltd. (London, UK) uses the Propel 3D simulation tool to train crew onboard.
• Carnival opened a Center of Simulator Maritime Training (CSMART) in Almere, Netherlands, which is a simulator and maritime training tool, to provide maritime training through technology solutions from Transas.

Due to the importance and expectations for continuous training in the maritime sector the development of immersive and engaging training technologies within effective knowledge transfer environments, AR and VR technologies will have a significant share of this growth. According to Goldman Sachs [1] knowledge transfer is the biggest business area for AR and VR technologies. Virtual reality, as an interactive technology, offers training centers a way to educate their students in immersive learning environments. However, and despite the projected and expected effectiveness of AR and VR in the global markets, today education holds the smallest market share (2%) with only 0.7 billions USD, indicating the potential growth in the AR and VR sector (Figure 1) [8].

In this paper, we will introduce our virtual training technology called Immersive Safe Oceans Technology and four safety training episodes, namely, (1) command bridge, (2) machine room, (3) crane, and (4) fire safety training episodes to illustrate how virtual training technology can be used in various professional training scenarios on board and just in time. We believe that this can lead into a blue ocean by creating a new and uncontested marketspace. Blue ocean is led by open innovation initiatives to create and capture new demands while making competition irrelevant [9]. These episodes also emphasize the need for VR training in the shipping industry. In the next section, the research design of this paper is introduced. The third section the research questions for virtual training in maritime safety training are identified. In the fourth section, maritime safety training learning episodes and virtual training technology are introduced, and in the fifth section, services, especially the VR training center concept, are presented. Finally, future research areas and concluding remarks will conclude this paper.

2. Research Design

As previously stated, AR and VR are promising technologies in knowledge transfer. In fact, in this paper, our focus on virtual training is on a sophisticated method of knowledge transfer in organizations with wider market share, as shown in Figure 1, compared to pure education. The aim of this paper was to describe the VR development process in maritime safety training as a next generation learning environment. This process presented in Figure 2 was designed according to Hevner’s [10] Information Systems Research framework in which both frequent industrial cooperation and rigorous academic research activities are included. Later in this paper, we will introduce four VR training episodes which were developed in test–generate cycles, where feedback was received both from industry and academia. In practice, industrial feedback was collected based on human-centric design principles [11] to identify requirements and needs and using agile software development methods [12] (especially scrum) to enable frequent communication and steady progress in the development. At the same time, from a scientific point of view, the research focus has so far been mainly on usability and user experience studies [13]. A combination of human-centric design and academic experiments provided valuable feedback to achieve a situation where training episodes will be ready for long-term use and effectiveness studies. These studies are definitely the most interesting part for both industrial and academic users. In addition, if scientists are able to find evidence that VR training is a more efficient training method than online training or hands-on-training, there is no question whether industrial partners would start establishing VR training centers (this concept will be presented in detail later in this paper). In this phase, we can say that, for example, four VR training episodes are not enough to establish VR training centers, but it can be enough for on board training if evidence is clear that these VR training episodes can improve maritime safety, for example, in navigation, energy efficiency, human errors, or fire safety.

3. Identifying the Research Challenges in Virtual Training in Maritime Safety

The research design presented above shows that developing VR training episodes for maritime safety requires practical but innovative solutions. In order to identify research challenges, the Stanford Research Institute’s NABC method (Needs, Approach, Benefits, Competition) was used as shown in

Table 1. Summary of the NABC analysis.

Needs	Approach	Benefits	Competition
Growing demand No onboard solutions Lack of simulator centers Lack of time Learning analytics needed Lack of motivation Immediate feedback needed Need for proving the effectiveness	Various episodes developed Industry-driven research Multidisciplinary research team International visibility and recognition Immersive and interactive technologies Evidence-based solutions Realism with advanced technological solutions	Engaging and effective VR training Bringing training onboard Portable and cost-effective solution Learning analytics to improve maritime safety New industrial sector growing	No particular suites available for onboard training No comprehensive measurement or assessment tools available

, used to identify, and refine a potential business idea [14].

3.1. Needs

Growing demand for onboard training in maritime safety is a recognized need all over the world. Various shipping companies are not able to train their seafarers in sophisticated simulation centers due to the lack of centers availability and time. To offer onboard VR training opportunities, there is a lack of novel VR training episodes collections to be used in maritime safety. The use of learning analytics in onboard training will be needed to give immediate feedback with highly motivated learning content. In addition, there is a need for proving the effectiveness of VR training in onboard environments, and the shipping industry needs to learn how to utilize VR training efficiently in their business to improve maritime safety.

3.2. Approach

We developed various VR training episodes in close cooperation with the maritime industry leading organizations such as Wärtsilä and Aboa Mare. Our episodes, which were developed in a multidisciplinary research team, have already achieved international recognition [15]. Based on experiences collected in these industry-driven research projects and on the state-of-the-art maritime safety training, we utilized innovative, modern, and immersive interactive technologies to create practical solutions. Our solutions still require further studies in order to be able to create evidence-based solutions with as realistic technical solutions as possible, covering high-resolution hand or even finger recognition hydrodynamic models and machine learning algorithms to track and trace human behavior during the training sessions.

3.3. Benefits

Engaging and effective VR training in maritime safety onboard can be seen as a complementary, or maybe in the future a compensatory training method, which should be a portable and a cost-effective solution. Learning analytics will produce interesting data for researchers, maritime professionals, and shipping companies. In addition, maritime safety covering various areas, such as carriage of cargoes and containers, fire protection, navigation, or radio communication and search and rescue, shows that a totally new industrial sector will be growing around next generation learning facilitation in maritime safety.

3.4. Competition

Based on the state-of-the-art technology listed in Section 1, no particular suites of VR training episodes designed for maritime safety to be used for onboard conditions exist. This also means that comprehensive measurement or assessment tools integrated into VR training in this context are not yet available in the market.

4. Immersive Safe Oceans Technology with Virtual Maritime Training Episodes

Immersive Safe Oceans Technology was designed based on experiences collected from various VR training experiments at Turku University of Applied Sciences from 2017–2020. It is also based on our previous studies in which we focused on virtual training and pedagogy [6], driving perception with neuroscientists [16,17] and human behavior under stressed conditions [12]. Thus, this technology was developed for years in several Research, Development and Innovation (RDI) projects for academic and industrial purposes. We have to admit that our research focus has been so far on learning episode pilots and not yet on the back-end system development. Therefore, we have to emphasize that this technology is still in a non-commercial phase, but it can be utilized in the future both to train maritime trainees and professionals. Compared to traditional maritime training methods such as simulators, developers utilizing this technology have no limitations while designing the learning content. Immersive Safe Oceans Technology adopts the same design principles as the gaming industry to implement immersive training scenarios that assure the engagement of the user to understand the right decision-making processes in a gamified environment.

4.1. MarSEVR

Description: The MarSEVR (Maritime Safety Education with VR Technology) was developed at Turku University of Applied Sciences and in close cooperation with Aboa Mare maritime specialists to practice situational awareness and decision making by replicating a ship bridge environment. In previous studies, we presented in detail how this replication was developed and how to use virtual training technology in maritime safety training [18]. At this moment, we are focusing on usability and user experience studies before creating different training scenarios with various difficulty levels.

Training Objectives: Our first training scenario was designed to be a watch change and collision avoidance situation, where the learning goals are for the trainee to practice paying attention to the equipment’s status and settings and to take actions when required for collision avoidance.

Experiences: This training episode consists of tutorial and command bridge stages. All controllers to be used in the command bridge stage are first introduced. At the moment, trainees must use VR controllers which are especially challenging for unexperienced users without any gaming experience. Therefore, in this MarSEVR training episode, we minimized the interaction methods by using only a trackpad button which is highlighted in all interactions. In addition, interactive objects are also highlighted to help users familiarize themselves in this immersive training environment (Figure 3). Based on these first experiences, it seems that high resolution with Varjo VR devices will be needed (e.g., recognizing an approaching fishing ship on the horizon). In addition, these devices will also provide eye tracking information to be used in human behavior observation. To increase the usability and to avoid the use of VR controllers, for example, the use of a Leap Motion sensors would offer finger recognition.

4.2. ShipSEVR

Description: The ShipSEVR (Ship Engine Safety Education Virtual Reality) training episode focuses specifically on ship engines and engine rooms safety procedures [19]. Wärtsilä has years of experience and focus now on the proof of virtual training’s potential [20]. The ShipSEVR training episode has been designed and developed in a joint project between Wärtsilä Land and Sea Academy, Turku University of Applied Sciences, and Ade Ltd. The delivered learning episode consists of a 3D ship engine room space where trainees are expected to find certain devices and equipment by utilizing the available technical drawings.

Training Objectives: The ShipSEVR episode provides trainees accessibility to engine and engineer room schematics (Figure 4) and technical diagrams and documentation through the trainee eyes as a portable digital technical library. The easiness to access, read, analyze the details (zooming), search, and store technical documentation is a significant contribution to shipping engine maintenance, as most of the physical technical documentation is not easily accessible when needed and not in good conditions due to its frequent use.

Experiences: In the marine industry specifically, various professions contain work routines in harsh or even hazardous conditions where technical documentation is difficult to read. Especially for novices, a three-dimensional understanding of technical drawings can be challenging. To train these skills with traditional methods is also challenging. In the classroom it is easy to read technical documentation but difficult to understand the meaning, while on the vessel it is easy to understand technical documentation but difficult to teach it. This solution seems to provide an easy way to reach the required documentation just in time. One of the future directions can be to utilize AR and some of the developed training content for onsite training. Also, in this training episode, high resolution would be beneficial (e.g., while reading schematics). When focusing on more hands-on-training activities in the future, finger recognition would also be required.

4.3. Crane Simulator

Description: This Crane Simulator training episode designed for maritime safety and developed in close cooperation with VR specialists from Ade Ltd. (Pontefract, UK) and with maritime specialists from ESL Shipping. The Crane Simulator training contains a crane model in a harbor environment and a crane cabin from the first person’s perspective with main functionalities on steering the crane’s tower (left joystick) and the crane’s grab (right joystick).

Training Objectives: The trainee should learn how to unload the carbon bulk vessel without mistakes. The crane’s grab should not collide with the silo or the vessel body. In addition, the pier environment should be kept clean with a maximal unloading speed (Figure 5).

Experiences: The development of this work specifically, required effort in the crane’s grab physics to ensure it behaved as realistic as possible both when it was loaded and empty. In this case, the developers did not receive any 3D models, so a 360 video camera was used to receive enough reference material and to model this crane realistically. Haptic technology was used to provide the trainee the possibility to feel the touch of the joysticks. This could also be used to increase the immersiveness to touch and to feel the grab physics and behavior while unloading the vessel and loading the silo. This new training episode is almost ready for pilot tests in one vessel. Before starting these tests, we will replace the VR controllers with joysticks to increase the usability and user experience.

4.4. Fire Safety Onboard

Description: Turku University of Applied Sciences developed the Electric Cabin Fire Simulation introduced in Reference [21]. This original version contains a simple room with a metal electric cabinet on fire. This episode was tested at the Oman National Training Institute and received quite promising results. Now, we modified this training episode based on the feedback received from the Meyer Turku shipyard. This new version contains a ship corridor with laundry on fire (Figure 6).

Training Objectives: At the very beginning, after a short tutorial, the trainee’s behavior will be tracked and traced, and this behavioral information will be analyzed and visualized for the trainee at the end of this episode. In this episode, the trainee should press the fire alarm button, pickup the fire extinguisher, release the pin, check whether the door is hot, squat down to avoid breathing smoke, extinguish the fire efficiently, close the sliding fire door, and move to the exit immediately.

Experiences: This modification proves that our Immersive Safe Oceans Technology training episodes can be relatively easily scaled in other environments. In practice, once a training scenario has been designed and game mechanics implemented, 3D assets can be replaced with a reasonable amount of work.

5. Immersive Safe Oceans Technology Services

Safety at sea technologies and strategies contribute to maritime operations safety with a positive impact on the environment due to the prevention of maritime accidents and the reduction of sea pollution. Such technologies can create Blue Oceans strategies for organizational sustainability and profitability. The immersive element of these technologies through VR and AR extend such Blue Ocean Strategies into Green Ocean Strategies where operational effectiveness promotes environmental sustainability. Green Ocean Strategies shift innovation towards sustainability, aiming to address social, corporate, and customer needs and expectations [22]. Organizations adapting Green Ocean Strategies can achieve high ESG (environmental, social, and governance) ratings that impact significantly on their reputation to clients and investors [23]. Immersive Safe Oceans Technologies bridge Blue Ocean Strategies with Green Ocean Strategies for maritime safety and clean oceans.

5.1. Training Library

Immersive Safe Oceans Technology will operate in the future as a training jukebox with each scenario to be executed upon schedule or demand (Figure 7) [17]. As shown in this paper, various VR training episodes are already available in maritime safety training. This technology can be brought onboard in any type of vessel. With minimal set-up, it can be used in offline mode and an onboard server will collect and analyze training data. Online mode also enables training from various locations (not yet in the same session). In the future, multiple trainees will also be able to participate in real-time simultaneous training sessions.

5.2. Training Center with Research Activities

Virtual training technology provides tools to develop next generation learning environments. Due to the nature of maritime onboard training, it seems to be the most promising way to make next generation learning widely available for seafarers. In some circumstances, special training centers, as part of traditional simulator centers or separately, VR can offer a better solution for maritime safety training. This way it is possible by using multiple monitoring screens to arrange several training sessions at the same time under the control of an instructor or a specialized trainer available on the same premises (Figure 8). Based on our experiences, the maritime and marine industries are intensively testing VR training. It is not fully clear whether VR training is a more effective training method compared to traditional methods. In order to prove this, a multidisciplinary team is required in which specialists in behavioral science should also be involved. We have various research methods available to arrange field experiments in these training centers.

In our previous studies [24], we presented a two-fold structure for developing virtual learning environments utilizing serious games design principles. This two-fold structure, including internal and external learning environments with characteristics of physical, psychological, and social learning, represents next generation learning allowing the absorption of knowledge, skills, and attitudes. Virtual training, as an immersive technology, can be utilized to train trainees and professionals on board or in training centers. Such a training requires 5G with latency free and wide bandwidth connectivity to enable rich and immersive user interaction. According to the European 5G Observatory [25], various VR pilots utilizing 5G connectivity were conducted during 2018 and 2019. These pilots have so far been only technology demonstrations that proved 5G technology to be reliable and robust technology for next generation media such as VR and AR.

6. Areas of Further Research

The research conducted on this topic indicated the critical roles of immersive futuristic technologies in sectors highly dependent on human expertise while operating in spaces where no help can be provided. A ship on the ocean or a plane on the sky stand alone with no or limited assistance to be provided when an accident occurs. The time to respond and the conditions to offer help can be undefined and unpredictable. The Japan Transport and Safety Board recorded 199 marine incidents from 1 January to 30 November 2019 (Figure 9) with 89% of them to be engineering and navigation related in the engine room and on the bridge [26].

Furthermore, the Maritime Injury Guide notes that the lack of maintenance and lack of proper training need to be corrected to reduce human error in most maritime accidents [27]. This is related to the high turnover rate indicated in the seafarers (officers and unlicensed mariners), as it is challenging to recruit well trained human resources and retain existing ones [28].

The test runs conducted during the development period of this research, especially with the MarSEVR technology where we had the opportunity to test it with experienced ship captains in a major shipping company, indicated not only the need and the market for such technologies but also further areas to extend this research related with the psychology of the captains and the engineers while they are on duty. We identified that it is not just the level of knowledge or expertise that contributes to the safe execution of a task but also the emotional and mental stage of the person at that specific period or during the training session. These findings will lead our future research in this area towards emphasizing more on the behavioral elements of the technology in order to provide a holistic learning experience that will capture knowledge, skill, feelings, and emotions.

This behavioral dimension that we already tapped into will be further extended to behavioral biometrics analysis, as people have unique ways of expressing themselves with their personal behavioral characteristics such as attention, body movement, and self-coordination [29]. With behavioral biometrics, we will be able to record, analyze, and understand the psychology of the trainee when executing a task. Today, the distance between VR and psychology has been shortened due to the commercial availability of the technology which can be acquired and adopted in many disciplines [30].

Furthermore, we intend to extend our work on behavioral biometrics with the utilization of data analytics related to the trainee behavior during one or more training sessions. This will allow us to deliver reliable predictions on the performance of the crew, the safety of the vessel, and the crew on it and also the staffing process prior to the sailing of the vessel. Data analytics and data science are important factors often neglected in training.

7. Discussion and Conclusions

The paper presented four training episodes developed for maritime safety training and our Immersive Safe Oceans Technology. Emphasis was given to the concept of a portable training system which should be cost effective and able to be used in onboard training centers or even office environments. The technology and the developed training episodes are now at the testing phase and several research activities are planned such as usability tests, user and game experience evaluations, and finally effectiveness studies. It must be noted that just four training sections have been introduced in this paper.

For commercial use, a wide range of training sections (with various episodes) either as individual training scenarios or as continuous episodes as part of a training curriculum/program are needed. The future of maritime training with the help of VR training moves on board, far from the physical training centers that require time and expensive training periods. The training episodes presented in this paper attempted to address the critical maritime training challenges and contribute to the effective training standardization of the maritime industry.

Immersive Safe Oceans Technology is still at its very early stages, while our focus has so far been on learning episode pilots and not yet on the backend system development. Significant advances have been designed and expected to be implemented on both the hardware and software level, aligned with the VR technologies progression but also with the maritime sector maturity to adopt VR education especially on safety issues.

Hardware wise, Immersive Safe Oceans Technology intends to integrate one of the most promising new VR technologies, namely, Varjo’s VR-2 glasses with high resolution (over 60 pixels per degree) [31]. We have already made small prototype versions of MarSEVR and ShipSEVR training episodes for the Varjo’s VR-1 glasses with promising results. In MarSEVR, these glasses already showed its potential to show small detailed objects such as a fishing boat far in the horizon. In ShipSEVR, in turn, the VR-1 glasses improved significantly the readability of technical diagrams and documentation for the trainee’s eyes. Based on these tests, it seems that this type of glasses can offer opportunities to focus on solutions where detailed textual or other visual information is a crucial part of learning aspects in maritime education. Software wise, Immersive Safe Ocean Technology will expand its safety training episodes beyond these examples (four presented training episodes) to cover hundreds of critical safety cases related with different processes inside a ship where continuous training onboard and ashore is expected to be repeated to all seafarers.

The continuous changes on the crew and the fast operations cycle does not allow time for effective safety training prior boarding. Immersive Safe Ocean Technology will primarily operate as an onboard safety training virtual classroom assuring optimal training to all, anytime and anywhere. The approach presented in this paper can be scaled to other fields and industries.