Enhancing the Thermal Inspection of Buildings Using Texture Analysis ^†

Abstract

The thermographic inspection of buildings is a powerful and non-invasive method for monitoring and diagnosing building performance and structural integrity. It can effectively detect moisture, evaluate heat loss, and assess building’s roofs. The early detection of problems in a building allows owners to fix issues before they become more severe and costly. One of the challenges in automating thermal analysis for building inspections comes in the form of distinguishing between different surfaces. This paper presents an automated process pipeline using coupled thermal and visible images for building inspections to assist inspectors in adapting strategies and methods to discriminate the thermal signatures between different kinds of surfaces. A deep learning method is employed to segment visible images texturally. Thermal images are then analyzed based on the resulting segmentation. Moreover, a multi-modal dataset is introduced, presenting coupled thermal and visible images acquired during multiple building inspections.

1. Introduction

The building industry contributes significantly to global energy consumption and greenhouse gas emissions due to various inefficiencies in structures [1]. Regular inspections are critical but face scalability issues due to their time-consuming nature. The adoption of technologies like thermal cameras and laser scanners can help to automate inspections, but they present limitations [2]. Multi-modal data acquisition strategies that combine different forms of sensor data have been introduced to overcome these challenges [3]. Despite their potential, the construction industry’s adoption of these multi-modal platforms has been limited [4]. This paper proposes a solution by integrating multi-modal data acquisition strategies with texture analysis for energy management-focused building inspections. We hope that this paper can help to enhance building inspection processes and reduce the industry’s energy footprint [5].

2. Background

Thermographic inspection, a fast and non-invasive methodology, has been instrumental in monitoring and diagnosing building performance, significantly aiding the early detection and rectification of structural issues. However, automating thermal analysis for building inspections presents distinct challenges, primarily with respect to differentiating between various surfaces. Research by Pozzer et al. and Mirzabeigi et al. has advanced the field, offering promising methods for enhancing our understanding of building energy performance and detecting structural anomalies using integrated data, infrared thermographic images, and drone thermography [6]. Further expanding the spectrum of techniques, Singh et al. provided an in-depth discussion on multimodal inspection in industrial settings, underscoring the importance of such techniques in ensuring equipment safety and reliability. Novel contributions like the multi-modal mapping unit by Mascarich et al. for underground tunnel exploration and the thermal–optical imaging and texture analysis algorithm by Nooralishahi et al. exemplify the ongoing efforts to innovate within the field [7,8,9]. In alignment with these innovative approaches, this paper presents an automated process pipeline that employs deep learning for the textural segmentation of visible images, followed by thermal analysis of the segmented regions. Furthermore, this paper introduces a novel multi-modal dataset composed of coupled thermal and visible images collected over several building inspections, marking a significant contribution to the automation and optimization of building inspections.

3. Methodology

3.1. Image Registration

This study employs a two-stage approach, with image registration forming the crucial first stage. Image registration aligns two or more images of the same scene, facilitating an effective analysis of the corresponding regions across these images. This section explores our usage of both manual and automatic image registration techniques.

3.1.1. Manual Image Registration

Manual image registration, or point-to-point registration, necessitates the manual identification of the corresponding points across the images that need alignment. A geometric transformation is applied to one image to align it with the other. Utilized as an initial benchmark for the automatic methods, this approach provides flexibility in handling diverse image scenarios despite being time-consuming and error-prone.

3.1.2. Automatic Image Registration

To address the limitations of the manual method, automatic image registration is preposed herein and specifically leverages homography matrices. Homography involves transformation mapping the points in one image to corresponding points in another while accommodating for changes in scale, rotation, translation, and perspective. A homography matrix based on the feature matching between images was employed to warp and align the images. This method provided a rapid, accurate alignment process, essential for efficiently processing large multi-modal datasets. In the subsequent section, the second stage of the methodology, which involved the application of deep learning techniques to the registered images for segmentation and analysis, is discussed.

3.2. Image Segmentation

3.2.1. Segmentation of Visible Images

For visible image segmentation, a deep learning approach was used. Specifically, the UNET++ architecture, an advanced variant of the classic UNET, was employed, along with the Resnet 152 as the backbone. This combination delivered an effective model capable of predicting and outputting a mask that highlighted different building components. Each segment of the mask corresponded to a different part of the building, such as windows, walls, or roofs. This mask served as an invaluable guide to the exact location and context of various building components, paving the way for an efficient analysis of the corresponding thermal images.

3.2.2. Segmentation of Thermal Images

The segmentation of thermal images leveraged an unsupervised image segmentation approach based on the method developed by Kanezaki [10]. As thermal images do not readily lend themselves to traditional segmentation techniques due to the nature of the data they contain, this unsupervised method proved particularly useful. It allowed for the segmentation of thermal images without prior training on specific building components. The segmented thermal images, when combined with the segmented visible images, provided a comprehensive understanding of the building’s thermal properties, thus enabling effective defect detection and energy performance analysis.

4. Results and Discussion

4.1. Image Registration

The results confirmed that automatic image registration outperforms the manual approach. Its capacity to quickly and accurately align thermal and visible images was integral to establishing an efficient and scalable automated pipeline for building inspections.

4.2. Segmentation of Visible Images

The effectiveness of UNet ++ with Resnet 152 for visible image segmentation was validated by the obtained results. As displayed in Figure 1, the deep learning model predicted a mask that closely mirrored the ground truth, effectively pinpointing the location of various building components. Whether for cement walls, windows, window frames, or metal parts, the method showed the potential to segment and identify different building parts accurately.

4.3. Segmentation of Thermal Images

Similarly, the unsupervised image segmentation method produced promising results on thermal images. It effectively highlighted regions with elevated thermal leakage, underscoring the efficacy of this approach for analyzing thermal properties. Figure 2b shows the results of unsupervised image segmentation on the thermal images. The averages for the IoU scores and F1-scores achieved by our model across various building components were 0.83 and 0.89, respectively.

4.4. Fusion of Results

Figure 2c shows the fusion results regarding thermal image segmentation and visible image segmentation. Our methodology’s core strength lies in merging segmented visible and thermal images, leading to the superior prediction of thermal leakage locations across different building parts. This integrated approach, which involves utilizing various image processing and machine learning methods, offers an efficient, automated pipeline for enhanced building inspections. Our promising results suggest the technique’s vast potential for broader applications in support of energy-efficient building management.

5. Conclusions

This paper introduced a pioneering method that combines multi-modal data acquisition, deep learning, and unsupervised image segmentation for enhanced building inspection, focusing on thermal leakage detection. The implemented methodology yielded promising results, displaying its capacity to automate and optimize building inspections. Key advancements include efficient automatic image registration and the successful segmentation of visible and thermal images, leading to accurate thermal leakage localization on specific building components. Despite its achievements, further research is required to refine this methodology and explore its potential limitations. Our future studies will incorporate varied deep learning architectures and segmentation methods, along with diverse building types and environmental conditions, to bolster the model’s robustness. In conclusion, this study signifies a substantial stride towards an energy-efficient future by revolutionizing building inspections through precise, automated, and data-driven techniques, holding extensive implications for both the building industry and broader global energy framework.