Mixed Reality in an Operating Room Using Hololens 2—The Use of the Remote Assistance from Manufacturers Techinicians during the Surgeries ^†

Abstract

The aim of this work is that the participants, using HoloLens 2 and Dynamics 365 Remote Assistance, can receive all the training and information necessary for the correct application of prosthesis and medical devices remotely, from a support center of the manufacturers, avoiding the displacement and presence of these technicians during surgeries. After implementing this method, an analysis will be made on its impact, avoiding displacement and the presence of technicians during surgery, in terms of increasing satisfaction and improving the experience of the participants, reduction of various risks (including the risk of infection) and on reduction of some economic and environmental costs.

1. Introduction

Mixed reality (MR) was first mentioned in 1994 by Paul Milgram, and is a blend of physical and digital worlds, unlocking the links between human, computer, and environment interaction, based on advancements in computer vision, graphical processing power, display technology, and input systems [1]. By using holographic devices, such as Microsoft HoloLens 2, the participants could take advantage of the ability to place digital content in the real world as if it were there. This technology allows participants to see through display and see the physical environment while wearing the headset and allows a full six-degrees-of-freedom movement, both rotation and translation. The participants can hold “hands-free” and “heads-up” teams video calls with experts anywhere in the world, with all of benefits in this kind of experience [1,2].

Recent research has shown that with the use of advanced technological solutions such as MR, the spaces of the operating rooms tend to decrease and the number of professionals present during a surgery too [3], improving the efficiency in the use of resources in a hospital, whether they are human, of space, or technicians and materials. MR is a concept which provide an “ideal virtual space with [sufficient] reality essential for communication” [4]. The combination of computer processing, human input, and environmental input sets the stage for creating true MR experiences. Movement through the physical world translates to movement in the digital world and improves the experience and better outcomes of the participants and tasks [5], once it did not blind doctors’ original view of the real world, showing a new vision with mutual correction function, which improved the safety of surgery [6].

Currently, surgeons and nurses focus their attention on operating room, both on the patient’s body and on surgical monitors, to get all the information needed [7]. To help them in their procedures, MR starting been in operating rooms all over the world [8]. This kind of technology is also a powerful for better training and improving education on surgical tactics and methods [9]. During the last few decades, changes in surgical materials are constantly being developed and incremental to improve better outcomes [10], most of them needs accomplishment by the manufacturer’s technicians during the surgeries. Their presence in the operating room increases the risk of infection [11,12], the delays in surgeries, and the logistic costs of the prosthesis, particularly, in orthopedic field.

Every year in Portugal, orthopedic surgeons perform around 5000 arthroplasties (surgical replacements of necrotic or fractured with a prosthesis), mostly hip and knees arthroplasties [13]. Through displaying specific images in the HoloLens 2 and using in the Dynamics 365 Remote Assistance application, all the assistance could be virtual with a better accuracy of the patient instrumentation, scaling up the expert support all over the world with less costs. There are numerous complex surgeries that require the physical presence of technicians from the prosthesis manufacturers during the surgery to help the participants with their correct application. These technicians, despite being trained for the surgical environment, increase the number of professionals within the operating room, with all the associated risks, namely, the risk of infection.

The objectives of this study are: Test the effectiveness use of MR technology, in the surgical environment as a pilot example for future projects, as a tool for aiding participants during arthroscopies (position and collocation of the prosthesis) without the presence of the manufacturer’s technicians, firstly from the identification of the main requirements for this process: analyzing its impact of avoiding displacement and the presence of technicians during surgery, reducing the risk of infection, increasing satisfaction and improving the experience of the participants, the eventual reduction of surgery time, increasing flexibility in scheduling surgeries, increasing the profitability of the technicians’ time to perform assistance to a greater number of surgeries and finally, with no physical visits to hospitals, reducing the CO${}_2$ emissions.

2. Methods

Preliminary systematic review, document analysis, analysis of similar previous cases in the industry, in particular the automotive industry using HoloLens 2 and Dynamics 365 Remote Assistance [14], identification of requirements to implement with professionals involved in surgeries with prosthesis, and initial exploratory interviews with orthopaedic surgeons.

3. Results

The development of a system that allows the use of HoloLens 2 as a tool for aiding participants during arthroscopies, and analyze its impact on several factors such as reducing the risk of infection; increasing satisfaction and improve the experience of the participants, using the video recording feature for future training; improve patient experience, sharing some previous videos of the procedure collected from the HoloLens 2; reducing surgery time increasing flexibility in scheduling surgeries; increasing the profitability of the technicians’ time to perform assistance to a greater number of surgeries; reducing the CO${}_2$ emissions; reducing of the global costs of the prosthesis.

4. Conclusions

Using HoloLens 2 to support the surgical team in the operating rooms during the arthroplasty’s surgeries, participants will be able to share, in real-time, their vision of the patient’s specific location where the prosthesis will be applied. While communicating with the manufacturer’s specialist technician via Dynamics 365 Remote Assistance, to monitor the correct sequence of application of the parts, maintaining constant visual and voice interaction with technicians, they can be watched continuing its activity in a concentrated and focused way with the patient and the rest of the team. The collected data have shown that the use of HoloLens 2, while an immersive technology of mixed reality, could be useful to eliminate the presence of the manufacturer’s technicians during these surgeries, improving participants and patient experience, reducing the risk of infection and the duration of the surgeries while reducing the price of the prosthesis in a future negotiation with the manufacturers.