Multivariate Analysis of Transient State Infrared Images in Production Line Quality Control Systems

Abstract

Manufacturers would like to increase production volumes while preserving the high quality of their products. The long testing times can cause a bottleneck of production processes especially taking into account the observed tendency for testing all produced devices. The main aim of this work consists in the analysis of time changes of features extracted from thermal images using the multivariate approach. The paper shows that if the principal component analysis (PCA), belonging to multivariate methods, is applied for quality control based on infrared images, then the minimum testing times can be estimated. In order to draw the final conclusions regarding testing times and, what is also very important, which principal components should be selected for classification, a detailed temporal analysis for an exemplary production line has been carried out. The future impact of the results includes higher productivity and cost-effectiveness due to the determination of an optimal decision time in production line quality control systems using the proposed approach.

1. Introduction

The constant development of measuring techniques and equipment results in a huge amount of data available for quality control tasks. Thermal imaging and thermal cameras are very good examples of this trend [1]. However, the progress in hardware capabilities would be useless if not followed by the progress in algorithms and software [2,3]. Of course, new algorithms force hardware development, too. In general, both key players in quality control, i.e., hardware and software, influence each other.

A pattern recognition system used for industrial quality control based on infrared images has to be highly reliable and fast. While the reliability is a self-evident feature of any system aimed at industry, the acceptable time for testing a single device is highly dependent on the application. Sometimes the algorithm that is too slow for the production line application gives the most accurate results. Therefore, especially in the case of very high-volume production lines, the choice of the algorithm should be a compromise between the classification reliability and speed of calculation.

This paper is devoted to the application of a well-known multivariate algorithm, i.e., PCA, to analysis of data acquired with the help of thermal cameras. The successful application of such algorithms in other fields allows us to expect improvement of production line quality control results. Multivariate algorithms enable reduction of the dimension of input space in classification problems and so they make them easier to solve [4]. It is especially important in the case of n × m images for which each pixel can be treated as a separate piece of information, and so the dimension of the input space can be as high as n × m [5]. Depending on the resolution, information included in adjacent pixels can be more or less correlated and feature extraction is indispensable [6].

Quality control can be based on single thermal images (2D data) but the output signal of a thermal camera is a 3D signal. In this paper, the importance of the third dimension through detailed analysis of the time evolution of features extracted using a multivariate algorithm has been shown. Such an approach, i.e., temporal analysis of PCA results aimed at acquisition time selection, has not been presented in the literature. The current state of the art of applications of PCA in infrared imaging has been presented in Section 3. So far, the testing time has been selected on the base of temporal thermal contrast evolution [7]. Some rough estimations have been also used to set the acquisition time which should be optimal to perform quality control [8,9,10]. However, such estimations usually suffer from overestimating the required time which is especially undesirable in production line applications. Moreover, this research has revealed that in some cases waiting too long can decrease the efficiency of the quality control based on thermal images.

Transient state analysis of thermal images has been considered in many papers. In the simplest case the feature extraction has not been applied and each pixel of the thermal image has been used for classification [5,10,11]—the comparison of heating and cooling patterns for each pixel or the number of pixels above a threshold value allows for the detection of elements or regions with defects. Sometimes it is enough to monitor time changes of maximum temperature. Advanced temporal analysis, but not based on a multivariate approach, can be found in [2]—the classification is based on time waveforms representing statistical features extracted from the fused infrared and vision images. The other application of infrared temporal analysis consists in the calculation of energy balance and heating curves of electrical machines which afterwards can be used for defect detection [12]. The temperature time history of the thermographic data is also used to extract the region of interest before applying post-processing and classification methods [13].

This paper aims at bringing advanced data analysis techniques based on multivariate approach to thermal image processing. The novelty of the paper consists in the detailed analysis of principal components temporal behavior aimed at the selection of the components which ensure the shortest time for performing production line quality control. The main conclusion is that it is possible to minimize the time required for quality control and obtain satisfactory quality control results but the principal components selection must be made individually for each problem. These factors, i.e., time and quality, are crucial in production line applications of infrared imaging.

The PCA, as a well-established method, has been briefly described in the following section and next its application to analysis of infrared images has been given.

2. Fundamentals of Principal Component Analysis

The contemporary real-world systems become more and more complicated so one needs more and more variables to describe it. A set of these variables is just a point in the measurement space. If the dimension of this space is high then methods of its reduction become very important as they simplify the analysis, help to uncover the relationship between observations and variables revealing groups or trends and enable visualization of the data. PCA is one of these methods. It consists in the projection of the data into space in which the first axis of the new coordinate system is aligned with the direction where the data has the largest variance. The next axes are ordered according to the value of variance of the data for the direction they represent. Moreover, this principal coordinate system is orthogonal. From the mathematical point of view the original coordinate system and principal coordinate system are related by an affine transform, i.e., translation and rotation. So PCA is a statistical procedure that performs a linear subspace projection. It enables data compression while preserving the major features of the data. PCA involves minimization of the mean square error between the projected data (onto an m-dimensional subspace) and the original data (given for the n-dimensional space, n > m).

The PCA is based on the following theorem of linear algebra [4]. If an n-dimensional vector x has the covariance matrix C_x, then there exists an orthogonal transform:

$$y=Q^{\mathrm{T}}x,$$

(1)

such that the covariance matrix C_y of the vector y is:

$$C_y=\left[\begin{array}{cccc}{\lambda }_1& 0& \dots & 0\\ 0& {\lambda }_2& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & {\lambda }_n\end{array}\right],$$

(2)

where λ₁ ≥ λ₂ ≥ … ≥ λ_n are the eigenvalues of the matrix C_x and the columns of Q are the corresponding eigenvectors.

The components of y are statistically uncorrelated. Moreover, each component has the maximum variance among all normalized linear combinations that are uncorrelated with the other components. It can therefore be said that y is the vector of principal components of x and the method is called PCA.

The projection of the data onto a smaller dimensionality space m (m < n) is commonly called feature extraction. What is important, such projection always results in data distortion. The error of approximation of the data set by only m features is equal to the sum of the eigenvalues that have not been taken into account:

$$\delta ={{\displaystyle \sum }}_{i=m+1}^n{\lambda }_i.$$

(3)

Principal components (PCs) can be calculated not only with the help of classical algorithms based on eigenvector calculation, but also by means of more sophisticated ones like Karhunen–Loeve transform or neural networks. One can use also Singular Value Decomposition (SVD) which allows us to express a given m-by-n real matrix A, with m > n, as:

$$A=U{S}_{\mathrm{o}}{V}^{\mathrm{T}},$$

(4)

where U and V^T are orthogonal matrices, S_o is n-by-n diagonal matrix with elements on the diagonal in decreasing order. These elements are called singular values of the matrix A. The time of calculation is the main drawback of using SVD to calculate PCs in the case of high dimensional spaces. Moreover, usually only the first few PCs are important and SVD calculates all n PCs. A method that enables us to determine only PCs that are needed is called the deflation method. It resembles the well-known recursive orthogonalization procedure by Gram–Schmidt.

3. Application of PCA to Infrared Image Analysis in Production Line Quality Control Systems

Data mining that leads to information extracting is one of the biggest challenges of contemporary science and one of the main needs of modern industry. In many present-day applications the problem is not to get the data but to analyze a huge amount of easily accessible information in order to make some decisions. Multivariate methods are a tool that can be used to solve this problem. Thermal image analysis belongs to this group of applications for which multivariate methods, including PCA, can be very useful. They become more and more popular in other fields and for sure they deserve also more attention from the infrared community.

It must be stressed that the effectiveness of PCA in the solution of a particular problem should always be verified. There is no guarantee that classifiers based on PCA features will work well, because PCA determines the projection of the data that enables the best reconstruction and not necessarily the best classification. So far PCA of infrared images has been used for non-destructive testing of materials [14,15,16,17]. It is especially useful for data compression in non-destructive examination and an IR-video sequence and what follows the reduction of processing times required for image classification [18,19,20]. However, there are almost no references on the application of PCA approach to high volume production line quality control [6]. Recently, application of the Candid Covariance-Free Incremental Principal Component Thermography (CCIPCT) as well as the Sparse Principal Component Analysis (SPCA) based K-means clustering for automatic defect segmentation in infrared non-destructive testing have been proposed [21,22]. The CCIPCT method is an extension of Principal Component Thermography (PCT) which has been proposed as a method for defect depth characterization [23,24]. The other relatively new PCA application consists in the enhancement of contrast [25] and removal of the noise and information redundancy of the infrared image sequences [26].

The transformation of the original thermal images into principal components applied in production line quality control has been illustrated in Figure 1. The detailed description of the procedure and the classification algorithm have been given in [6]. In this paper, analysis and discussion of time evolution and selection of principal components is a key problem.

The analysis of temporal evolution of principal components aimed at the optimization of production line quality control for home appliances has been carried out taking refrigerators as a device under test (DUT). They can be regarded as complex units exhibiting a dynamic thermal process after switching on. Although the first question was how to set the time of inspection in order to get the best quality control results, the performed analysis have led us to answers regarding some other questions and more general recommendations valid when dealing with PCA in such application [6]. A typical layout of the production line quality control system based on infrared imaging has been presented in Figure 2.

If infrared images have the resolution of 320 × 240 pixels then the input feature space is 76800-dimensional assuming that each pixel is an input variable. The reduction of the dimension of the feature space to 2D presented by Grabowski and Cristalli [6] allowed for fast classification and simple interpretation and visualization of classification results. Of course, it brings about loss of information, which can be accepted as long as the correct classification is possible.

4. Temporal Evolution of Principal Components in Exemplary Production Line Application

The first question that must be addressed is how to set the shortest image acquisition time which ensures a good discrimination between fully functional and faulty devices. In the case of refrigerators, the transient thermal state which takes place after switching on takes a few minutes, usually not more than 10–15 min. The heating process has been illustrated in Figure 3 which shows typical time changes of evaporator inlet and outlet average temperatures. What is the shortest time after which quality control can be carried out using PCA approach?

Let us conduct PCA analysis for 10 devices including 5 defective devices belonging to class B (red and empty point curves in Figure 4, Figure 5 and Figure 6) and 5 non-defective devices belonging to class G (green and filled point curves in Figure 4, Figure 5 and Figure 6). The time dependence of the first 2 PCs has been shown in Figure 4 and Figure 5. The answer to the question asked in the beginning of this section can be given. In this example the minimum image acquisition time should be set to 5 min.

It must be noticed that behavior of PCs is different—the discrimination between classes for PC1 remains, while for PC2 it reduces significantly after 10 min and practically declines after 12 min. It is also visible in Figure 6 which includes data clusters obtained after 17 min. In order to get the best separation between classes the acquisition time should be set to 8 min.

The results presented in Figure 4, Figure 5 and Figure 6 could suggest that the first two PCs, representing the most of the total variance of the data, should be used for quality control. Unfortunately, such theorem is not true. The following example concerning the same type of device but a different model shows that selection of PCs must be made very carefully and separately for each model. In this case, there are 21 devices under consideration including:

• 14 non-defective devices belonging to class G with an exemplary infrared image presented in Figure 7a (these devices are represented by green points in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12),
• 4 defective devices belonging to class E with an exemplary infrared image presented in Figure 7b (these devices are represented by blue points in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12),
• 3 defective devices belonging to class B with an exemplary infrared image presented in Figure 7c (these devices are represented by red points in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12).

For a defective refrigerator of class E (Figure 7b), the evaporator outlet temperature is higher than in the case of non-defective devices (Figure 7a). It means that the evaporator is not working properly. For a defective refrigerator of class B (Figure 7c), the temperature difference between the evaporator inlet and outlet is higher than usual. It can be caused by an occlusion, an overcharge or undercharge of liquid refrigerant. Generally, the correct classification of devices that belong to class E is not an easy task but it is usually possible as it has been shown below.

The analysis of Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12 leads to a conclusion that the classification using PC1 and PC2 is possible in this case but it requires much longer acquisition time (Figure 12), which in the production line environment can be unacceptable. Moreover, the comparison between PCs presented in Figure 11 results in a conclusion that for this acquisition time it is better to choose PC2 and PC5 in order to obtain higher class separation. Successive plots in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12 have axes being successive PCs. Of course, the best PCs for classification are not always consecutive, e.g., PC2 and PC5 for the time of acquisition equal to 15 min.

It is evident that in this example PC3 and PC4 are the only components that enable quick classification after just 4 min (Figure 8). On the other hand, the separation between classes for these PCs decreases with time (Figure 9, Figure 10 and Figure 11) and they do not allow us to perform reliable classification for times greater than 10 min (Figure 12).

The classification can be made after about 5 min depending on the device model. It is two times less than the time indicated in some other publications [8,9]. Moreover, tests proved that it is even easier to detect some faults, e.g., evaporator defects (class E, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12), on the base of infrared images taken during the transient state (for times less than 10 min).

As for the classification efficiency, the analysis of Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12 reveals that one of the faulty devices belonging to class E cannot be correctly classified using PCA approach. It must be noticed that its infrared image, presented in Figure 7b, is very similar to the image of a non-defective device presented in Figure 7a. Therefore, the final efficiency is equal to 95.2%.

In Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12, it can be noticed that class B can be separated for almost all PCs and all acquisition times—in 18 considered cases out of 20 comparing to 6 for the class G and E assuming acceptable level of classification efficiency to be equal at least 95%.

As it has been already suggested, it is not possible to point out the best PCs before performing a preliminary analysis of the data.

Table 1. Percentage of total variance of data covered by principal components calculated for different measurement times.

Principal	Percentage of Total Variance of Data Covered by Principal Components (%)
Component	t = 240 s	t = 300 s	t = 450 s	t = 600 s	t = 900 s
PC1	75.4	73.6	70.6	70.0	68.1
PC2	6.5	6.8	8.5	8.8	9.9
PC3	5.0	5.6	6.6	7.3	7.1
PC4	4.2	4.5	4.1	4.3	4.2
PC5	3.2	3.2	3.6	3.2	3.6
PC6	2.2	2.6	2.7	2.8	3.0
PC7	2.1	2.2	2.2	1.8	2.4
PC8	1.4	1.5	1.8	1.7	1.8
PC9-768	0.0	0.0	0.0	0.0	0.0

shows the time dependence of the total variance of data covered by each PC. Can it be helpful to choose the best PCs?

In order to answer this question, the relative changes of total variance values covered by each PC have been presented in Figure 13. It has been assumed that the PC total variance values for t = 240 s are reference values. Two groups of PCs can be distinguished. The first group, for which the relative value changes are less than ±15%, includes PC1, PC4, PC5 and PC7. The second group, for which the relative value changes are higher than +25% and less than +55%, includes PC2, PC3, PC6 and PC8. The variance for the PC1 is decreasing with time when the other values are non-monotonically increasing (PC2–PC3 and PC5–PC8) or stay at the same level (PC4). Two PCs with the best classification power for each acquisition time have been indicated by yellow background in Figure 13. It seems that there is no simple rule that could be used to select PCs for classification on the base of the time dependence of the variance for each PC.

5. Conclusions

It is possible to reduce testing times and so the unit costs of production line quality control, which are proportional to the testing times, by application of the infrared imaging and the principal component analysis. Tests performed during the thermal transient state can simplify classification of devices with defects difficult or even impossible to be detected in the steady state using PCA approach. As for the transient state dynamics, the time constants as well as initial and steady state values are different for each PC—the waveforms describing the time behavior of the PCs can be approximated by solutions of ordinary linear differential equations of the second order with constant coefficients. However, there is no algorithm which allows us to select the best PCs in advance—each case must be considered individually.

Future research directions may include application of self-organizing neural networks for clustering the PCA results especially if more than two principal components must be chosen to reach the required classification efficiency level. Such an approach will enable easy and understandable presentation of the data clusters as it is possible to visualize high-dimensional input spaces in two dimensions of the network topology. The performance of some other classifiers, such as support vector machines or decision trees combined with the AdaBoost algorithm, can be also verified and compared.