Smart Facility Management System Based on Open BIM and Augmented Reality Technology

Abstract

With the wave of the Fourth Industrial Revolution, the construction industry is also witnessing the application of numerous state-of-the-art technologies. Among these, augmented reality (AR) technology has the advantage of utilizing existing 3D models and BIM data and is thus an area of active research. However, the main area of research to date has either been in visualizing information during the design phase, where architects and project stakeholders can share viewings, or in confirming the required information for construction management through visualization during the construction phase. As such, more research is required in the application of AR during the facility management (FM) phase. Research utilizing BIM in the FM phase, which constitutes the longest period during the lifecycle of a building, has been continuously carried out but has faced challenges with regard to on-site application. The reason for this is that information required for BIM during the design, construction and FM phases is different, and the reproduced information is vast, so identifying the required BIM data for FM and interfacing with other systems is difficult. As a measure to overcome this limitation, advanced countries such as the US and UK have developed and are using Construction Operations Building information exchange (COBie), which is an open-source BIM-based information exchange system. In order to effectively convert open-source BIM data to AR data, this research defined COBie data for windows and doors, converted them to a system and validated that it could actually be applied for on-site FM. The results of this system’s creation and validation showed that the proposed AR-based smart FMS demonstrated faster and easier access to information compared with existing 2D blueprint-based FM work, while information obtained through AR allowed for immediate, more visual and easier means to express the information when integrated with actual objects.

1. Introduction

1.1. Background and Purpose of the Study

Facility management (FM) is the process of sustaining the initial expected asset value of the facility through maintenance and improvement in its quality, acquired during the design and construction phases. The FM phase constitutes the longest part of the building’s lifecycle, and the costs incurred during this phase can take up to 85% of the life cycle cost (LCC) [1]. For effective and systematic operations and maintenance of the facility, the information on the facility generated during the design and construction phases forms and is used as the basic data for the facility management system (FMS). However, defining and verifying the required information for the operations and maintenance of the facility, thereby using them in building baseline data for the FMS, requires much time and associated cost. To alleviate these difficulties in the current FMS set-up process, building information modeling (BIM) technology is being adopted and is seen as an alternative for effective and systematic facility operations and maintenance. The BIM model is made up of objects, properties and relationships, and as the object defines all properties and relationships, it can be used to manage the required information for FM based on function, space and purpose.

Therefore, the BIM information can be utilized as the initial data for the FMS without the need for additional modification and is expected to be applied in the operations and maintenance of the facility, along with the space and energy management, as visual and multi-approach analyses become feasible. Through utilizing the BIM-based information, building and facility owners or facility managers can reduce maintenance costs during the FM phase, which is much longer compared with the design or construction phase. However, despite the many advantages and possibilities of utilizing BIM information in the FM phase, this has yet to be fully applied to on-site FM work. The reason for this is that the wide-ranging information required for FM has yet to be defined and is thus often not included in the BIM-based design information.

As a measure to resolve these issues, research is conducted on the development of systems to transfer information produced during the design and construction phases to the FM phase. This type of research has been conducted by public institutions carrying out contracting work for major facilities within the US, and the result has been the development of a data-converting format called Construction Operation Building information exchange (COBie) [2].

Meanwhile, the Fourth Industrial Revolution brought forth technology advancement in various industries, and the construction industry has also advanced through the application of many of these developments. Of these, augmented reality (AR) technology in particular enables adding additional information (e.g., images, text and 3D models) to the actual environment obtained from camera images so as to provide users with better information sharing and task execution through an increased sense of reality and immersion. In addition, AR technology is in use for product quality evaluation, manufacturing systems design and various other areas, while recent advances in smart devices have made it possible for users to more readily use AR technology. The construction industry is also actively seeking the application of AR technology through visualization of on-site facility data and the adoption of real-time coordination functions. However, from an LCC point of view, and compared with the design and construction phases, research on applying AR has been lacking in the area of FM, which is expected to bring the most economic benefit.

Therefore, in order to suggest a more efficient FM approach and contribute to expanding the body of research related to automated management for smart FM, this research investigates recent trends in technology related to BIM and AR along with measures to connect information from both and proposes a smart FM system based on AR. As for the development of a system prototype, this research defines the required information system benchmarking after the COBie system and designs a user interface (UI) for its implementation. Lastly, the proposed system is validated through the case study and surveys that were conducted to verify its effectiveness.

1.2. Study Scope and Method

The main purpose of this study is to improve the application of BIM to the existing FM work and to increase work productivity through the application of AR. To this end, the research proceeded by focusing on operations and maintenance work carried out by the on-site FM crew. In order to apply efficient AR to tasks carried out by the on-site FM crew, the existing maintenance procedures and work processes were analyzed, and a new work process was proposed by separating the task categories to which AR technology could be applied and implemented. Within the proposed process, we defined the property information (attribute data) category required for FM and established COBie datasheet parameters for each category to enable interfacing with BIM data. Through this, it was possible to convert BIM data to AR visualization data without further modification, while manual input for some additionally required information was made available. Such converted AR data was used to build AR-based smart FMS, which supports the on-site FM crew and automates history track record management. Finally, the developed system was evaluated and validated through survey instruments.

This research adopted a methodology, as outlined in Figure 1, to develop a smart FMS by applying BIM and AR for use in the management of facilities in the public sector. The research procedure followed five major steps: (1) analysis of preceding studies on existing BIM-based FMS and AR technology. At this stage, we w derived ICT technology applicable to existing maintenance processes through a literature review and investigated the current problems and issues of FM; (2) categorization of smart FM work processes to which a COBie system based on BIM and AR would be applied, followed by the definition of COBie data required at each work stage. Information that required AR visualization was mapped from the BIM attribute information to connect BIM and AR; (3) development of a scenario-based Business Process Modeling Notification (BPMN), which is a segmented data exchange process system necessary for the smart FMS prototype. The period proposal process was subdivided, and the information exchange system was schematized; (4) designing the UI for the smart FMS functional implementation method. At this stage, a web-based maintenance AR information DB was constructed, and the UI functions were defined; and (5) development of the AR-based smart FMS prototype, prototype implementation and testing and verification. An experimental control group study was designed and evaluated to implement and verify the prototype system.

2. Literature Review

2.1. Building Facility Management Utilizing BIM

One of the advantages of applying BIM in the FM phase includes its utilization as an analytical tool for purchasing machinery, equipment, control systems and other necessary procurements. Moreover, BIM can be used for checking the proper functioning of all systems following the completion of building construction. By providing accurate space and system information, BIM can be used for real-time monitoring of control systems and for remote operations and management of a facility while functioning as an interface for the sensors [3]. BIM for FM is the only information base that provides manuals to the FM crew, since it allows the visual recognition of locations for equipment, attachments, furniture, etc. and provides support to emergency responses, security management and scenario planning [4]. As such, research is being actively conducted to bring forth the maximum advantage when applying BIM to FM.

To solve the issues with data integration between BIM and FMS, research up to now has proposed two types of open-source standards, namely COBie and Industry Foundation Classes (IFC). COBie is an information exchange system that captures and delivers the lifecycle of information needed by facility managers. Stakeholders use the COBie standard to share information from BIM to FMS, which may include some graphic and non-graphic information but does not include the complete information set for FM activities. The BIM framework, with regard to the attribute data category, is still unable to completely integrate the required information for FMS [5]. In addition, information format and data exchange policies that have yet to mature result in lengthy processing times when sending out the data [6]. Some research was carried out that proposed an IFC-based data model for an integrated FMS when expanding BIM to include FMS data [7]. However, the possibility of visualizing the BIM to improve FM activities was not considered in this research [8].

Yu et al. developed a data model for computer-integrated FM that could extract the required information from IFC for data compatibility when converting BIM to FM data [9]. A study by Mendez proposed functions that utilize BIM in the FM phase and developed a web-based prototype that could improve the utilization of BIM information [10]. East and Brodt developed COBie, a spreadsheet-based data exchange format, to solve the inefficiencies due to information generated during the design and construction phases that were not required in the FM phase [11]. A study by Becerik-Gerber et al., through expert interviews, defined and proposed the BIM recognition level during FM, areas for possible application and data and processes required for successful implementation of BIM [12]. Lee et al. benchmarked the COBie system to improve FMS for facilities [13], while Choi et al. studied the establishment of an FMS based on COBie for sewage work [14].

Studies on the aspect of integrating the BIM data with FM include research proposing software architecture for effectively integrating BIM with the FMS based on the geological information system (GIS), as well as research proposing methodologies for integrating BIM-based mechanical, electrical and plumbing (MEP) data during operations and FM phases [5,15]. Studies by Gao and Pishdad-Borgozi clearly defined what BIM needs to support the FM phase for the successful implementation of FM-supported BIM and proposed realistic processes while defining interoperability plans for data between FMSs, such as the BIM tool and the Computerized Maintenance Management System (CMMS) [16]. Other research includes studies on Symantec web-based BIM data utilization frameworks for utilizing the BIM data from existing structures for FM and using laser scans and generating COBie data through ontology concepts to augment the existing insufficient BIM data from structures [17].

However, in most cases, the application of BIM for facility operations and maintenance was limited to preliminary studies [12,18]. It can be noted that the majority of the preceding research was directed into either (1) studies for interfacing data produced during the design and construction phases and on interoperability with FMSs, or (2) on commercialization through application to FMSs.

2.2. Utilizing AR Technology within the Construction Industry

A review of recent trends with respect to AR technology utilization within the construction industry revealed that research was initially focused on the feasibility of applying visualization technology. Hammad proposed the feasibility of AR technology for on-site tasks with regard to infrastructure [19], while Shin analyzed the display, tracking and server aspects of developing a system using AR based on wireless technology at construction sites and proposed an augmented reality system prototype for the inspection of steel pole construction [20].

Studies regarding AR authoring tools include research for developing a 3D modeling tool called ‘Tinmith’ that allows modeling of AR through real-time capturing of external structures and rendering, allowing users to directly interact using augmented reality [21]. More recently, prototype developments that allow users to directly apply them at construction sites are actively carried out, with Dunston conducting a study on AR-CAD that overlays 3D CAD models on top of actual background images using a head-mounted display (HMD), allowing the discovery of spatial interference on blueprints during the design phase [22]. In their study, Moon et al. demonstrated an augmented reality prototype for selecting the appropriate location for tower crane installation at high-rise construction sites and carried out testing procedures based on an actual construction blueprint [23].

Research into applying AR for FM has also been active recently. Bae et al. conducted a study that proposed a novel method to support site construction and FM activities by allowing site workers to visually document construction issues or progress status through automatic access to the latest project information with AR overlays [24]. There have also been studies that proposed a natural marker-based AR framework that can digitally support facility managers when searching for FM items or when actually performing maintenance and repair work [25]. Kwon et al. proposed a smart FM system that improved existing maintenance processes and studied a new maintenance work process and implementation measures for the application of AR and VR technology [26].

A study by Abramovici et al. proposed an augmented reality-based support system for collaboration among working-level stakeholders involved in the FM process, whereby AR was used not only as a tool for visualization of maintenance data but also as a tool for communication and alerts or displaying other coordination-related aspects for the whole team [27]. There have been studies that developed, tested and evaluated mobile BIM AR systems (Artifact) with cloud-based storage capabilities aimed at integrating BIM and AR systems to improve information retrieving process as well as operational efficiency during construction [28]. Research by Liu and Seipel suggested a framework for visualization guidelines to implement objects such as pipes hidden behind walls or ceilings into AR, similar to X-rays [29]. There were also studies that used BIM to derive categories for fire safety equipment inspections as well as the development and validation of systems that can configure compiled data for visualization through AR [30].

In essence, visualization of non-visual data can be a very useful cognitive aid for identifying the information needed for decision-making in FM. To improve the efficiency of BIM applications, some studies have shown that the integration of AR technologies will help improve the usability and accessibility of BIM information [31,32]. However, research demonstrating the efficiency or usefulness of AR from a scientific point of view was still lacking [33]. Chu et al. found that the practical benefits of integrating AR and BIM models are somewhat limited and mentioned the importance of building a viable information search mechanism that meets user information needs in order to maximize the benefits [28].

From analysis of the preceding research results, the utilization of AR technology in FM work was found to enable efficient construction management by allowing the user to intuitively recognize the architecture blueprint or paper documents. However, it is important to define the information required for FM and to interface with the BIM data, along with effective visualization. Since the 4D BIM based on an existing virtual environment does not effectively reflect actual construction site conditions, the value and utility of AR technology with reduced cognitive resistance when applied to FM is expected to be significant in terms of productivity improvement.

3. Smart FM and Information Exchange Systems

3.1. Smart FM Work Process

While virtual reality (VR) allows the user to interact with an immersive and entirely virtual application, AR can present a layer of virtual content over a layer of real-world images captured in real time. The latter can help users improve their performance in many tasks by providing pertinent information about the pictured scenario. VR, AR and mixed reality (MR) are represented by Milgram and Kishino through a virtuality continuum diagram [34], presented in Figure 2.

While VR, at the rightmost end, is represented by a fully virtual environment, AR remains closer to the experiences comprising a real environment. Additionally, MR brings the two ends of the continuum together (i.e., reality and virtuality). The three concepts are grouped in the general concept of extended reality (XR), which brings them all together [35]. The smart FMS proposed in this study aims to support the FM crews and managers by introducing AR technology into existing on-site FM tasks.

The baseline data for incorporating this new technology is BIM. Analysis of the existing FM work process based on preceding studies and interviews with experts revealed the following sequence: (1) establish an inspection plan; (2) conduct the inspection; (3) inspect the site conditions and identify defective areas; (4) check the maintenance history track record; (5) perform the site inspection through the management company; and (6) approve the execution of maintenance work and its procedure. In this process, a number of simple tasks between facility managers, inspectors and the FM company repeat frequently, which greatly reduces the productivity of FM work. To solve this problem, AR technology is applied to existing FM tasks to propose a process that improves the work sequence and reduces repetition while defining the data required for each task. The proposed process, as shown in Figure 3, is to apply AR technology during the preliminary review, inspection and maintenance work process to support real-time collaboration and information visualization in support of each task.

In the proposed process, the data such as drawings, history track record and related documents, which were stored in a scattered manner, are integrated and stored in the BIM data, thus overcoming the limitations due to repetition of simple tasks and inefficient data utilization. To this end, the necessary information for the FM work is processed through the hub-like AR-based smart FM platform, which manages the central facility information database that collects various scattered data and processes them upon request. In this paper, the research was conducted by limiting the scope of the FM work to windows and doors among a building’s interior management work. The BIM attribute data or history track record data could be different for each required task within the FM work. In order utilize clearly defined information, which forms the database into AR visualization, it is necessary to refine the FM work scenarios and categorize each work task based on them.

First, within the daily or routine inspection scenario, the on-site inspector determines the location of the facility that requires inspection and identifies the AR marker attached to the target inspection space. Through this marker, the existence of the object needing inspection within the target space and the accompanying checklist are visualized, through which the inspector performs visual inspection and takes photo records. In this process, the images captured by the inspector’s camera are shared in real time to allow all stakeholders to determine whether there is a defect. If it is determined that there is a defect through the inspection, the historical track record of the company responsible for the installation, warranty and maintenance is visualized and transmitted directly on site to the responsible company with the defective condition. If the FM manager subsequently determines that a repair is required and carries out the repair work, he or she ensures that the related guidance and progress are monitored throughout the repair process, and related information is stored in the FMS. The daily or routine maintenance work scenario is shown in Figure 4, with items applying AR technology shown in red.

Another scenario is shown in Figure 5, where a defect occurs in the building, the repair request is made, the FM manager confirms the defect through the image information of the site, and the relevant historical track record is visualized through AR and maintenance work before proceeding. The key technologies required in this process for AR visualization include (1) technology to support AR-based surveillance tasks; (2) 3D information + AR overlaying technology; and (3) 3D data transformation technology for AR and VR.

3.2. Defining the COBie System

The information exchange system for implementing an AR-based smart FM process proposed in this paper uses COBie as a benchmark. As a substitute to the vast amount of documentation related to FM at the end of each construction, COBie is a spreadsheet-based information exchange data format jointly developed by public authorities, led by the US Army Corps of Engineers, who are responsible for the contracting of major US facilities. Among the BIM information generated at each phase of a construction project, including planning, design, construction, operation and maintenance, COBie includes the definition and formatting of information necessary during the FM phase.

As shown in

Table 1. COBie datasheets and contents for each construction phase.

Phase	Sheet	Contents
All	Contact	People and companies
	Document	All applicable document references
	Attribute	Properties of referenced item
	Coordinate	Spatial locations in box, line or point format
	Issue	Other issues remaining at handover
Early Design	Facility	Information on facilities and standards
	Floor	Vertical levels and exterior areas
	Space	Spaces
	Zone	Sets of spaces sharing a specific attribute
	Type	Types of equipment, products and materials
Detailed Design	Component	Individually named or schedule items
	System	Sets of components providing a service
	Assembly	Constituents for types, components and others
	Connection	Logical connections between components
	Impact	Economic, environmental and social impacts at various stages in the life cycle
O&M	Spare	On-site and replacement parts
	Resource	Required materials, tools and training
	Job	PM, safety and other job plans

, the COBie datasheets consist of a total of 18 data sheets, and the relevant details are included in expandable templates. First, the data sheets that can be used during the entire construction project phases include information on contract-related matters, drawings, coordinate systems and construction-related issues for each project participant. During the early design phase, information on facility details, floors, space, material etc. are included, while during the detailed design phase, additional information related to the connection and assembly of different systems, system configuration, schedule and cost are added. During the operation and maintenance (O&M) phase, information on replacement and repair, required resources and supported work are included. In addition, COBie is configured to allow continuous expansion of the required attribute data.

The COBie system definitions required for detailed scenario-specific maintenance tasks proposed above do not require all of these datasheets, and it is important to lighten the data load and take advantage of the inter-connectivity of the attribute data to utilize only the minimum amount of data essential for AR visualization. Therefore, mapping of the required information necessary for the aforementioned scenario-specific FM tasks and COBie datasheets was conducted based on a literature review as well as through interviews and surveys of actual on-site FM workers, as shown in Figure 6.

The mapping of task categories within an FM work requiring AR technology with the COBie datasheets showed that only five datasheets (i.e., Document, Issue, Space, Type and Job) included all the attribute data required for on-site FM work. For analysis of each datasheet, the ‘Document’ sheet includes image data acquired from the site, along with contract documents, 3D shape information, shop drawings, warranty documents and other documents, as shown in Figure A1. It consists of seven data fields: ‘Document Name’, ‘Document Type’, ‘Sheet Name’, ‘Row Name’, ‘Storage Location’, ‘File Name’ and ‘Description’. All of these data fields are not included in the BIM data at the time of handover and must be populated through additional work. The ‘Issue’ sheet has a dataset to record issue-specific items and is composed of six data fields, consisting of ‘Writer’, ‘Date of Writing’, ‘Issue Type’, ‘Space Name’, ‘Component Type’ and ‘Description’, as shown in Figure A2. The ‘Space’ sheet, which includes information such as the space name, building name, floor, ID for spatial identification and installed sensor information, automatically generates all this information through mapping with the BIM data (Figure A3). The ‘Space’ sheet data fields include ‘Description’, ‘Facility Name’, ‘Floor Name’, ‘Room Tag’ and ‘Sensor ID’. As shown in Figure A4, the ‘Type’ sheet includes five data fields: ‘Component Name’, ‘Space Name’, ‘Category’, ‘Manufacturer’ and ‘Warranty Description’. Each data field includes detailed information, such as the type of construction materials installed, their model names, manufacturers and warranty information, which is generated through interfacing with the BIM data. Finally, the ‘Job’ sheet contains the overall data on maintenance progress and completion, warranty period and history track record information (Figure A5). The data fields that make up the ‘Job’ sheet include ‘Job Name’, ‘Status’, ‘Start Date’, ‘Duration’, ‘Warranty Period’ and ‘Description’. The standard COBie datasheets consist of 277 data fields in 18 datasheets, but the COBie information exchange system proposed in our study consists of 29 data fields in 5 datasheets.

4. Design of an AR-Based Smart FMS

4.1. Smart Facility Management BPMN

To devise a prototype for an AR-based field support system for smart FMS based on the maintenance scenarios proposed in this paper, a BPMN with the input/output (I/O) status of the data within the process is required. BPMN is a visualized notification method for representing a business process, created to show the process time flow and the flow of messages between parts of different activities.

Figure 7 shows the BPMN with the I/O of the COBie data required for the task categories of the aforementioned AR technology applied to daily or routine inspection scenario-based FM work progress. First, the inspector generates outputs and reviews the 3D information of the inspection site from the ‘Document’ sheet. The inspector then visits the inspection site, identifies the AR marker and generates the input/output attribute data of the member to be inspected from the ‘Type’ sheet. Site photographic information gathered from the camera module of the mobile device is configured so as to be stored and retrieved from the ‘Document’ sheet. Once the site inspection is completed and it has been determined that there is indeed a defect, the relevant details are recorded on the ‘Issue’ sheet, and the maintenance history track record of the defected member is reviewed from the ‘Job’ sheet. Details on the warranty are reviewed from the ‘Document’ sheet, and repair work then proceeds.

The BMPN at the time of defect occurrence and receipt of a defect repair report is shown in Figure 8. In this case, as with the case of daily or routine inspections, image data from the site is stored in the ‘Document’ sheet, while the information on the defect area is recorded on the ‘Issue’ sheet. The ‘Job’ sheet is used to confirm the maintenance history track record, and the warranty-related information is viewed from the ‘Document’ sheet. The COBie datasheet is configured to be mapped to each repair work item, including the maintenance repair work that occurs afterward.

Most of the 3D models and attribute data of the member subject to inspection are extracted from the BIM information and stored in the COBie Excel sheet database. At this point, the additional attribute data required includes the AR marker information, inspection date, inspection cycle and warranty duration, among other factors, which are manually input by the FM inspector after receiving relevant information from the responsible supply company. Clarifying the definition of required information and the use case is the most important aspect when applying BIM information for FM. Therefore, the proposed BPMN allows for the specific implementation of the use case for smart FM.

4.2. System Function Display Configuration and UI

In essence, AR systems consist of cameras, image converters, tracking, and video background renderers. The camera component ensures that the camera frame is delivered efficiently and automatically in accordance with the image format and size. When an AR marker is recognized by the smartphone camera, the user defines the object to be viewed through AR by means of the pixel format conversion. The target defined within the database configured with the COBie system detects the object through a tracking module. After identifying the condition of the target object, a 3D virtual object is modeled, and the information is provided on the display. Auxiliary information on the augmented 3D virtual object in the display is managed by web-based FMS and is augmented through extensible markup language (XML). The web service application program consists of a management system and a two-way communication system for auxiliary information on the augmented object. The auxiliary information management system resolves the issue of an existing 3D object shown with simple augmentation and is configured to provide various information to the user rapidly, while the two-way communication system is configured to provide information in real time that is not provided in the support information management system through bidirectional communication between the user of the AR system and the manager.

The UI of an AR application that emphasizes interaction function with the real world requires a different design than the UI of traditional mobile applications. Mobile devices used on site are mostly smartphones and tablets that provide touch-based interfaces. The on-site FM support application proposed in this study targets the use of these mobile devices and takes advantage of existing touch operation methods. Therefore, operations using simple controls are necessary to allow ease of operations, and this in turn requires a more user-friendly and intuitive UI. Due to this, the menu button was placed on the left, and required attribute data was placed at the bottom of the screen. Because AR interacts with the real world, transparency control is provided to support the projection of the visualized information to real-world imagery.

This prevents the possibility of sight obstruction when the operator is either moving or carrying out tasks, and a navigation map of the building interior was placed on the upper right corner of the screen to allow the operator to verify his or her location. Figure 9 shows an example of a prototype implementation screen in which the site inspector has visualized the specifications for a door. The temperature and humidity sensory information of the space, as well as a checklist document for the interior finish inspection, are provided at the bottom of the screen. The inspector can view the door, windows and other target objects in this manner and input the inspection results for each inspection category through the checklist.

5. System Development and Validation

5.1. Development of AR-Based Smart FMS

To validate the proposed AR-based smart FMS, the system was developed according to the aforementioned BPMN and functional display UI designs. The generation of AR visualized data and the simulation of data transmission were configured according to the maintenance scenario previously mentioned. AR-related software development kits (SDKs) and 3D rendering tools were needed to develop the AR application proposed in this study. The development environment for the application prototype was based on C# and utilized Unity and PTC Vuforia. In addition, Android-based mobile devices (e.g., smartphones and tablets) were adopted to run the developed application, and the IFC version of the open BIM model used in the system was IFC2 × 4. The development process of the proposed AR-based smart FMS is shown in Figure 10. The BIM modeling platform was Autodesk Revit, and COBie data was automatically extracted using Revit add-ons. For AR visualization, marker-based technology was utilized, and QR codes were used to recognize spatial information. The information in this QR code was interfaced with the COBie datasheet to display maintenance attribute data to the user.

As shown in Figure 11, when the field inspector visits the maintenance site and runs the application, he or she receives the spatial information and BIM data along with the IoT sensor information via a QR code. In addition, the attribute data as well as the maintenance history track record for the member being inspected is reviewed through web-based COBie data, and the inspector performs the inspection. In this process, the application manager is able to store and synchronize the inspector’s work status and updated information on a real-time basis.

The information exchange process created and transmitted during this maintenance work is shown in Figure 12. The system’s underlying data utilizes BIM data at the time of handover. The BIM data in this step includes more detailed information about the space and interior finishes, and these spatial and attribute data are used through interfacing with COBie. The 3D shape information from the information required for AR visualization requires a separate extraction process. In addition, the modules required to recognize AR marker information and to visualize the information are tracking, rendering and measurement modules that are implemented through separate application programming interfaces (APIs). In addition to the AR visualization function, interior navigation, data transmission, voice recognition, video streaming, auto-measurement and photography functions are required. When the inspection is completed, the resulting information is stored in COBie’s ‘Issue’ sheet, while the progress and completion of the maintenance work is automatically updated with the relevant information on the ‘Job’ sheet.

5.2. Case Study Using a Prototype AR-Based Smart FMS

A demonstration of the AR-based smart FMS was conducted after specifying a target building. To evaluate the viability of the proposed AR-based smart FMS, validation work took place using a real-world case project. The BIM data at the time of handover for an actual building (shown in Figure 13) was obtained, and the COBie data was automatically extracted via Revit add-ons. Then, the additionally required dataset items were configured separately and utilized for web-based AR data.

Each menu option of the web-based AR data was made visible when the marker was recognized through the camera module of a smart device. The prototype consisted of a functional UI required for actual maintenance work in order to increase practical availability. The on-site inspector selects the inspection type from the inspection menu and confirms the sensor information of the space and the checklist of the inspection target. Then, the attribute information of the member to be inspected can be confirmed. When reporting a defect, we implemented a photography function, a BIM model confirmation function and a contact function with the person in charge. In the history information view menu, the maintenance history can be viewed, and warranty-related matters can be checked. When repair work is carried out, the contract document can be viewed, and the work status can be selected to record the recorded items, and when completed, the photo data is saved. Figure 14 shows the screenshots of the prototype AR-based smart FMS actually running. As shown in Figure 14a, the starting menu consists of four options—‘Inspection’, ‘Report defects’, ‘Check history’ and ‘Repair work’—and each menu has submenu options. Within the ‘Inspection’ submenu, ‘Inspection type’ allows selection of the type of inspection from daily, periodic and report reception of the defect (Figure 14b). The ‘Sensor’ submenu visualizes the sensor information within the space being inspected (Figure 14c). In the ‘Check list’ submenu, a checklist entry for the member to be inspected appears, while clicking on the ‘Property’ submenu shows the attribute data for the member being inspected (Figure 14d,e). The ‘Report defects’ option allows for taking and transmitting image information, checking the BIM model information and real-time connection with the staff, and it includes three submenu options: ‘Photographing’, ‘Check the BIM model’ and ‘Contact with staff’ (Figure 14f–h). The ‘Maintenance history’ submenu under the ‘Check history’ option displays the maintenance history track record information on the screen, while the ‘Department/Company’ submenu provides information for the responsible department and company (Figure 14i,j). The ‘Warranty Check’ feature also provides a menu that allows the viewing of warranty-related documents and contents (Figure 14k). The ‘Repair work’ option consists of four submenus that include ‘Contract document’, ‘Work status’, ‘Records’ and ‘Complete inspection’. The ‘Contract document’ function provides access to contract-related documents for the repair work, and ‘Work status’ provides the option to select the progress (e.g., scheduled, progressed and completed) of the repair work (Figure 14l,m). The necessary records for repair work progress and completion can be stored in the ‘Records’ submenu, while ‘Completion inspection’ allows for the storage and transmission of information regarding the condition at the time of completion in an image format (Figure 14n,o).

5.3. System Demonstration of an AR-Based Smart FMS and Validation

After the system was built, an experimental control group study was designed and conducted to test the developed AR-based smart FMS. Specifically, an experimental group was set up to compare the maintenance work efficiency between the traditional paper-based (control group) approach and the AR-based smart FMS (non-control group) approach. The experiment involved 20 graduate and undergraduate architectural engineering students. Each participant was randomly assigned to conduct an experiment using paper-based media (control group) or an AR-based smart FMS (non-control group) to perform maintenance tasks.

The inspection work on the condition of the doors, windows and inspection space was divided into 14 maintenance tasks (

Table 2. Maintenance tasks for testing.

Member Subject to Inspection	Operation and Maintenance Tasks
Door	(1) Check open or closed status
	(2) Check model name and warranty condition
	(3) Check the doorknob condition
	(4) Check for any serious damage externally
	(5) Verify history track record
	(6) Record inspection result
Window	(1) Check open or closed status
	(2) Check model name and warranty condition
	(3) Check the window knob condition
	(4) Check for any serious damage externally
	(5) Verify history track record
	(6) Record inspection result
Inspection Space	(1) Measure and record temperature
	(2) Measure and record humidity

) which were randomly assigned to the study participants, and their performance was measured and evaluated. The experimental site measured approximately 200 m² and had 10 windows with 3 doors. Following an introduction to the experiment, and after becoming familiar with the relevant devices and documents, the participants began the maintenance work. The start and finish times of the work were recorded for each individual, and a brief post-assessment questionnaire was requested to be completed after the maintenance work. The post-experiment questionnaire was adapted from Chen et al. [32], and it focused on the quality of human interaction with technology.

Most student participants took BIM and AR courses and therefore possessed mastery of the subject matter needed for the experiment. However, most participants still preferred to use traditional methods and techniques for data searching. Compared with the control group, the non-control group participants thought it was easier to use an AR-based smart FMS to find information during maintenance tasks. Moreover, the non-control group participants responded that the system provided sufficient functionality to complete the task. Although the control group participants were used to the paper-based approach, they felt it was difficult to determine the location of the member subject to inspection and to find the relevant maintenance information for the member. On the other hand, the non-control group participants answered that using the smart FMS had the advantages of location confirmation, rapid information access and ease of use. A summary of the survey statistics of these participants is shown in

Table 3. Survey summary of test participants.

Summary of Participants’ Experiences
Experience with BIM and AR	Control Group	Non-Control Group	Experience of Data Retrieval	Control Group	Non-Control Group
Level of BIM Understanding (Proficiency)			Preferred Data Retrieval Method (Multiple Responses Allowed)
Very proficient	2	1	Electronic 2D or 3D CAD and PDFs	8	6
Proficient	3	3	Native models (.rvt, .dwg, etc.)	5	6
Somewhat proficient	4	3	Printed drawings or documents	2	4
Not very proficient	1	2
No proficiency at all	-	1
Level of AR Understanding (Proficiency)			Preferred Technologies for Data Retrieval (Multiple Responses Allowed)
Very proficient	-	-	Printed drawings	5	8
Proficient	3	5	Tablet	2	1
Somewhat proficient	5	4	Smartphone	6	5
Not very proficient	1	1	Laptop PC	1	3
No proficiency at all	1	-	Desktop PC	2	4
Participants’ Opinion of Using the Tools (Paper-Based vs. AR-Based Smart FMS)
How difficult was it to find and retrieve information?	Control Group	Non-Control Group	Did you feel that the tool was sufficient to complete the required tasks?	Control Group	Non-Control Group
Easy	1	6	Yes	2	9
Medium	2	3	No	8	1
Hard	6	1
Could not locate it	1	-
Participants’ Opinion of the Quality of Interaction
			Control Group	Non-Control Group
Please comment on which part of the tool (paper-based or AR-based smart FMS) operation was difficult?			Viewing of attribute data Location confirmation Viewing of sensor information	Small display Inconvenient controls
Please comment on which part of the tool (paper-based or AR-based smart FMS) was helpful to you?			Intuitive information viewing	Location confirmation Rapid information access Portability

.

To compare the performance of maintenance tasks using traditional 2D blueprint methods with the AR-based smart FMS, the time it took for the two groups to complete the task was analyzed. The group statistics and independent sample test for task completion time are shown in

Table 4. Group statistics and independent sample test for task completion times.

Group Statistics
Participant	1	2	3		4	5	6	7	8	9	10
Control Group	13.04′	15.11′	13.96′		15.03′	18.74′	16.39′	17.72′	15.66′	19.63′	18.42′
Non-Control Group	9.85′	11.63′	12.39′		8.06′	10.59′	7.79′	11.88′	10.21′	11.49′	9.51′
Group		N		Mean		Std. Deviation				Std. Error Mean
Control Group		10		16.37		2.18719				0.69165
Non-Control Group		10		10.34		1.57303				0.49744
		Levene’s test for equality of variances				T-test for equality of means
		F		Sig.		t	df	Sig. (2-tailed)		Mean Difference
		1.64538		0.21586		7.07785	18	0.00001		6.03

. The experimental results for the non-control group (AR-based smart FMS) show that the participants were able to complete the maintenance work in an average of 10 min and 34 s, faster than the average of 16 min and 37 s took for the control group (using 2D blueprints) participants. This showed that the smart FMS developed in this paper allowed faster completion of tasks than the traditional method. In addition, the outcome of the maintenance work performed by the participants using the smart FMS achieved 100% accuracy, while the results recorded by the participants engaged in the 2D-based maintenance work achieved 95.7% accuracy. Participants using the AR-based smart FMS received information on each task directly through their smartphones. Therefore, the display of information was realized in an immediate, visual and convenient way.

In order for the FM crew on site to immediately obtain the necessary information through AR, this study demonstrated the use of the AR data conversion approach, which converts BIM data through a COBie-based information exchange system. In addition, spatial information can be visualized through the on-site AR marker, which provides web-based AR space information, while the inspection results can be recorded using the screen capture function of the mobile device. Instead of traditional 2D methods, the web-based auto-update technology, which automatically updates the completed maintenance work information, enables the real-time communication between the staff and FM crew, thereby assisting the decision-making process. However, some issues were observed during the experiment that require further attention. First, due to the limitations of marker-based AR technology, maintenance work was possible only in areas where the markers were displayed. Second, the auxiliary functions of the AR image were limited in tight spaces, which implies that the AR system is more suitable in large, open spaces. Third, the small display size of a smartphone was noted as a discomfort during operations and lacking in providing intuition when images were displayed.

6. Conclusions

This study presents the system design and development of the BIM-based smart FMS carried out by configuring data categories and defining the functions needed for applying AR technology to on-site maintenance work. To this end, existing studies on BIM-based maintenance as well as AR applications in construction were analyzed, and on-site FM tasks that can benefit from applying AR technology were identified. Lastly, the required information system for the AR-applied smart maintenance work process was defined based on COBie, which was then used in proposing prototype development. From the proposed prototype, the AR-based smart FMS was developed, which was then tested and validated through a case study.

The study results show that interoperability of various software can be attained when maintenance information is configured based on the COBie system. Moreover, utilizing AR technology that visualizes the maintenance information through an overlay with actual on-site conditions will enable speedy information sharing and decision making regarding the worksite. The fact that the BIM dataset generated during the design and construction phases is being utilized signifies the importance of life cycle application of BIM data. Most participants participating in the case study conducted for verification replied that complete acquisition of the needed information for maintenance tasks was possible on-site through the use of the proposed smart FMS. Moreover, they were able to complete actual maintenance tasks faster and more accurately compared with using the existing approach. The sharing of real-time site conditions through smart devices and collaboration functions minimized the intervention of an individual’s subjective judgment, which led to more accurate and enhanced reliability of the maintenance work results.

However, since the study scope was limited to on-site maintenance work only involving building interior finishes, our proposed system may not be applied to every FM task, which is the limit of this research. In addition, future research should consider getting feedback from actual users when developing a proposed system and configuring the UI. To overcome the limit of maker-based AR, a study on makerless AR systems should be conducted. Expanding from mobile devices, head-mounted displays (HMDs), such as smart glasses, should be explored and utilized in future research. Finally, future research should expand from interior finishes to mechanical, electrical and plumbing (MEP) applications and develop technology that applies AR to visualize information on complicated equipment, which is difficult to distinguish using the naked eye.

In this study, a COBie datasheet-based information exchange system was adopted to improve the usability of BIM in the maintenance phase. However, the BIM data at the time of handover lacked the information necessary for the maintenance stage, so a lot of time and effort was required to manually input the necessary information. The COBie dataset, which could be automatically extracted from the BIM data, consisted of spatial information and model names. This was a very insufficient result to improve the usability of BIM data. In a future study, it seems necessary to define the BIM data required in this maintenance step separately and make a guideline so that it can be written in the handover BIM data.