Review of Recent Automated Pothole-Detection Methods

Abstract

Potholes, a kind of road defect, can damage vehicles and negatively affect drivers’ safe driving, and in severe cases can lead to traffic accidents. Efficient and preventive management of potholes in a complex road environment plays an important role in securing driver safety. It is also expected to contribute to the prevention of traffic accidents and the smooth flow of traffic. In the past, pothole detection was mainly performed via visual inspection by human experts. Recently, automated pothole-detection methods apply various technologies that converge basic technologies such as sensors and signal processing. The automated pothole-detection methods can be classified into three types according to the technology used in the pothole-recognition process: a vision-based method, a vibration-based method, and a 3D reconstruction-based method. In this paper, three methods are compared, and the strengths and weaknesses of each method are summarized. The detection process and technology proposed in the latest research related to automated pothole detection are described for each method. The development plans of future technology that is connected with those studies are also presented in this paper.

1. Introduction

A pothole is a hole in a road surface that results from gradual damage caused by a traffic and weather conditions. Potholes, a kind of road defect, can damage vehicles and negatively affect drivers’ safe driving, and in severe cases can lead to traffic accidents.

According to the Korea Express Corporation, the number of potholes identified on Korean expressways from 2015 to 2019 was approximately 47,200. Considering the total distance of 4767 km of expressways in South Korea, the average number of potholes per kilometer of expressways is about 10. Additionally, 221 pothole-related traffic accidents were reported officially from 2017 to 2019 [1]. Identifying and managing potholes in advance plays an important role in securing driver safety and preventing traffic accidents.

The general process of pothole detection consists of four steps: data acquisition, data preprocessing, feature extraction, and pothole classification. The data-acquisition step is the process of building a dataset for learning or analyzing potholes by acquiring raw data such as various sensor data or images. In the data-acquisition step, it is important to obtain sufficient data to learn and analyze the pothole. The data-preprocessing step uses the dataset built in the previous step to refine the data for easy learning or analysis of potholes. In the data-preprocessing step, a variety of signal-processing techniques such as filtering and masking are applied to refine the raw data. The feature-extraction step is a process of finding a factor to distinguish between potholes and non-potholes in the preprocessed data. In the feature-extraction step, it is important to consider the type and characteristics of the preprocessed data when looking for factors for pothole classification. The pothole-classification step determines the existence of potholes by applying a pothole-detection algorithm based on the features. In the pothole-classification step, various algorithms are applied to improve the accuracy of pothole classification and an optimal method is derived in the process. In the pothole-detection process, data sensing and processing is an important factor that determines the type of pothole detection and affects its performance. Therefore, the pothole-detection technique based on data sensing and processing is considered to be of great value compared to other possible techniques. The general process of pothole detection and considerations in each step are summarized in Figure 1.

In the past, pothole detection was mainly performed by visual inspection by human experts. Recently, automated pothole-detection methods using various techniques have been applied [2,3,4,5,6,7,8]. Automated pothole-detection methods can be classified into three types according to the technology used in the pothole-recognition process. The three automated pothole-detection methods are a vision-based method, a vibration-based method, and a 3D reconstruction-based method. The three methods summarized in this paper were selected based on the most widely applied method through the results of recent research trend analysis.

This paper is structured as follows. Three automated pothole-detection methods were compared, and the strengths and weaknesses of each method are summarized in Section 2. The detection process and technology proposed in the latest research related to three types of automated pothole detection are described for each method in Section 3. The performance and characteristics of recent automated pothole detection research are also summarized in Section 3. Conclusions and future work are presented in Section 4.

2. Comparison of Automated Pothole-Detection Methods

A vision-based method uses images or videos as input data and determines the presence of potholes on the road surface by applying image-processing and deep-learning technology. The vision-based method is more cost-effective than the 3D reconstruction-based method and is also suitable for determining the number and approximate shape of potholes. However, the vision-based method has a limit in measuring the volume and depth of potholes because it utilizes two-dimensional information. It is also affected by lighting and shadow conditions [2,3,4,5,6].

A vibration-based method judges the existence of potholes and predicts the depth of potholes based on the data acquired from the acceleration sensor in the vehicle. The vibration-based method is most cost-effective among the three methods. It requires small storage in the data-acquisition process and real-time data processing can be applied. The vibration-based method has a limit in providing the exact shape of potholes because it analyzes only the vibration information of the acceleration sensors. It is also affected by the sensor and vehicle applied in the data-acquisition process [6,7,8].

A 3D reconstruction-based method predicts the shape of potholes and measures the volume of them based on stereo-vision technology. The 3D reconstruction-based method predicts the shape of potholes and measures the volume of them most accurately among the three methods. It has a disadvantage in that it is expensive to detect potholes compared to other methods and hard to recognize when the potholes filled with water or dirt [6,8].

The characteristics of automated pothole-detection methods are presented in Figure 2. The strengths and weaknesses of each automated pothole-detection method are also summarized in

Table 1. The strengths and weaknesses of automated pothole-detection methods.

Methods	Strengths	Weaknesses
Vision-based method	It is more cost-effective than the 3D reconstruction-based method It is suitable for determining the number and approximate shape of potholes	It has limitations in measuring information such as volume and depth of potholes It is affected by lighting and shadow condition
Vibration-based method	It is the most cost-effective among the three methods It requires small storage Real-time data processing can be applied	It has limitations in providing the exact shape of potholes It is affected by the sensor and vehicle applied in the data-acquisition process
3D reconstruction-based method	It measures the shape of potholes most accurately among the three methods	It is the most expensive among the three methods

.

3. The Three Types of Recent Automated Pothole-Detection Methods

3.1. A Vision-Based Method (Type A)

Lim et al. [9] proposed automated pothole-detection methods based on deep-learning techniques using images as input data. Authors of [9] collected pothole images through public database sites such as Flicker, Google Images, and Pixabay. They labeled and annotated a bounding box representing the size and position of the pothole in each image. The dataset consists of 996 training images containing 1796 potholes and 203 testing images. The dataset on which labeling and annotation was performed based on pothole images collected by the authors of the paper was not open to the public. The proposed methods for training and testing the pothole dataset in the paper were two modified models based on YOLOv2 (You Only Look Once version 2). In the design of the two modified models, the creation of an anchor box suitable for pothole detection and the application of the denser grid suitable for input-image resolution were considered. The proposed models are an anchor-box model based on YOLOv2 and a denser-grid model based on YOLOv2. Authors evaluated the performance of two proposed models and original YOLOv2 using average precision, recall and frames per second (FPS) as evaluation indicators. They also explained that the proposed models were superior to YOLOv2 through the performance evaluation result. The two proposed models were able to exhibit superior performance compared to original YOLOv2 by reducing the layers of the YOLOv2 and adjusting the size of the anchor box. However, it was difficult to apply it directly to the pothole-detection process of a vehicle since this study was trained using a limited number of pothole data. In this respect, authors proposed real-time pothole detection as a future work by training more pothole data.

Baek et al. [10] proposed a pothole-detection and -classification method based on edge detection using pavement images as input data. The proposed method consists of three phases: image preprocessing, feature extraction of road damage, and road-damage classification. In the process of image preprocessing, RGB image data were converted to gray-scale image data, and the objects in images except potholes were detected via object-detection algorithm. The contour of the pothole in the preprocessed images was extracted via edge-detection algorithm for feature extraction. Potholes were detected and classified via YOLO algorithm in the road-damage-classification phase. The performance of the proposed method was evaluated by distortion rate and restoration rate of the image, and the accuracy of the classification. The dataset used in the performance evaluation process is the Global Road Damage Detection Challenge 2020 dataset [11]. The experimental results showed that the mean-squared error (MSE) of the distortion rate and restoration rate of the proposed method had errors of 0.2–0.44. The average of the classification accuracy and precision of the proposed method were 0.7786 and 0.8345. The accuracy and precision of pothole detection presented as experimental results of this study are restricted to one pothole in the image data. On the other hand, it is confirmed that the accuracy of pothole detection was relatively low when there were multiple potholes in one image datum. In addition, it is possible to detect the shape of a pothole by applying the proposed method, but there are limitations when estimating the actual size of a pothole. Authors presented a method of extracting detailed features to predict the actual size of a pothole as a future work.

Park et al. [12] presented a method for automated pothole detection that applied different YOLO models using images as input data. Three YOLO models such as YOLOv4, YOLOv4-tiny, and YOLOv5 were applied in the process of training and testing. The dataset of [12], which is found in [13], was composed of 665 pothole images and was divided into training, validation, and testing subsets. First of all, the images in the training subset were converted to be suitable for the various YOLO models. The models were trained and validated until the loss function reached a steady-state line. Next, the performance of three YOLO models was evaluated using mean average precision at 50% intersection-over union threshold ([email protected]). The experimental results showed that the mean average precision as 0.5 ([email protected]) of YOLOv4, YOLOv4-tiny, and YOLOv5s were 0.777, 0.787, and 0.748, respectively. It showed that the performance of YOLOv4-tiny was better than the performance of YOLOv4 and YOLOv5s for pothole detection. The low accuracy when detecting small potholes located at a long distance is a limitation of this study. In addition, it is judged as a limitation that the study was not carried out in bad weather conditions and under insufficient light conditions.

Wanli Ye et al. [14] presented a method for automated pothole detection that applied prepooling CNN using images as input data. The 400 raw pothole images were collected from different pavements under different light conditions. All images were cropped into 96,000 small images and they were used as a training or testing dataset of the proposed method. The dataset built on the pothole images collected by the authors of [14] was not open to the public. Authors explained the characteristics of prepooling CNN compared to the conventional CNN. The prepooling CNN is characterized by inserting the prepooling layer as an input before the first convolution layer of the conventional CNN. The advantages of the proposed method were analyzed in three aspects. It was demonstrated that the prepooling layer can improve the precision of pothole detection by comparing the average precision of the prepooling CNN and the conventional CNN. It was confirmed that the outputs of the prepooling CNN were little affected by different light and different pavement material conditions through MSE. It was also demonstrated that the CNN-based methods are more suitable for pothole detection than other conventional methods such as Sobel edge detection and K-means clustering analysis.

Hanshen Chen et al. [15] proposed a pothole-detection method based on location-aware convolutional neural networks using images as input data. The proposed method consists of two main subnetworks such as a localization network (LCNN) and a part-based classification network (PCNN). The localization subnetwork employs a high recall network to find as many candidate regions as possible, and the part-based subnetwork performs classification on the candidates on which the network is expected to focus. Authors used two different sizes of image as input the network to achieve accuracy and efficiency of the proposed method. The performance of the proposed method was evaluated by accuracy, precision, recall, and F1 score using the public pothole dataset [16,17]. The experimental results showed that the accuracy, precision, recall, and F1 score of the proposed method were 0.950, 0.952, 0.920, and 0.936, respectively. Authors explained that the proposed method had better performance than the existing methods [17,18,19,20] through the experimental results. The method proposed in this paper has scalability in detecting not only potholes, but also bumps, cracks, or rutting. However, the proposed method has a disadvantage in that performance is degraded when the input data is low-resolution. The authors proposed exploiting scale selections and cascade techniques to improve the performance of pothole detection as a future work. The authors also plan to conduct additional experiments on various types of roads such as concrete roads and gravel roads, as well as asphalt roads.

Deepak Kumar Dewangan et al. [21] proposed a pothole-detection method based on CNN with an embedded vehicle prototype. The proposed model utilized a monocular camera unit and Raspberry Pi computing device for pothole detection. The system consisted of three modules: a pothole-detection module, a data-processing module, and an embedded autonomous-vehicle-system (AVS) module. The dataset that was used in the process of training in a pothole-detection module was composed of 3915 images. The dataset size was increased from 783 to 3915 through standard data-augmentation techniques such as flipping, blurring, warping, and rotation techniques. The performance of the proposed method was evaluated by accuracy, precision, recall, and F1 score using the public pothole dataset [16]. The experimental results showed that the accuracy, precision, recall, and F1 score of the proposed method were 0.9902, 0.9903, 0.9903, and 0.9833, respectively. The authors explained that the proposed method had better performance than the existing methods [16,22,23,24,25] through the experimental results. The proposed method showed good performance by optimizing the CNN architecture according to the hardware requirements and the purpose of pothole detection. It is judged that a real-time pothole-detection system using low-cost edge devices such as Jetson Nano can be implemented based on the research results of this paper.

Nhat-Duc Hoang [22] proposed pothole-detection methods based on a machine-learning algorithm using images as input data. The proposed approach for pothole detection consists of three phases: image acquisition and feature extraction; dataset construction; and artificial-intelligence (AI) model training and prediction. The 200 asphalt pavement images were prepared and classified into 100 pothole images and 100 non-pothole images in the image-acquisition phase. Image-processing techniques, including the Gaussian filter, a steerable filter, and integral projection were utilized for extracting features of images in the image-acquisition and feature-extraction phase. In the phase of AI model training and prediction, the least-squares support-vector machine (LS-SVM) and the artificial neural network (ANN) were applied. The performance of two proposed models was evaluated by the classification accuracy rate (CAR) and the area under the curve (AUC). The experimental results showed that the CAR of the LS-SVM and ANN were 88.75% and 85.25%, respectively. The authors explained that both LS-SVM and ANN are applicable methods for pothole detection through experimental results. They also showed that the performance of LS-SVM was higher than that of ANN through performance comparison of the two models. The authors proposed improving the accuracy of pothole detection as a future work by applying other advanced AI models in the training and prediction phases. The authors also suggested estimating the size of potholes as a future work by applying advanced image-signal-processing techniques in the feature-extraction phase. The pothole dataset of [22] is available through e-mail contact with the author of this paper if readers need it.

Penghui Wang et al. [26] proposed an asphalt pavement pothole-detection and -segmentation method based on a wavelet energy field. The proposed method consists of two processes: pothole detection and pothole segmentation. In the process of pothole detection, the wavelet energy field of the asphalt image was constructed by morphological processing and geometric criteria. The detected pothole was segmented via Markov random-field model and the pothole edge was extracted accurately. The proposed method was trained and tested on 120 pavement images. The experimental results showed that the overall accuracy, precision, and recall of the proposed method were 0.867, 0.833, and 0.875, respectively. The authors also showed that the performance of the proposed method is better than that the of existing other methods [25,27] in the aspects of pothole detection and segmentation through the performance comparison. The authors proposed applying a more effective mathematic model as a future work to improve the accuracy of pothole detection.

Muhammad Haroon Yousaf et al. [28] proposed a pothole-detection and -localization method using pavement images as input data. The proposed method consists of two processes: pavement-image classification and pothole localization. The pavement images were classified as two groups: pothole and non-pothole images, using a bag-of-words (BoW) approach. The authors utilized scale-invariant feature transform (SIFT) features to establish the visual vocabulary of words. The support vector machine (SVM) was applied for training and testing of histograms of words about pavement images. A graph-cut segmentation scheme was also used for localizing the potholes in the labeled pothole images. The proposed method utilized a dataset of 120 pavement images created by Koch et al. [27]. The dataset of [28] was not open to the public. The experimental results showed that the accuracy, precision, and recall of the proposed method were 0.957, 0.970, and 0.941, respectively. The authors explained that the performance of a proposed method is better than the performance of existing other methods [22,25,27,29] through the experimental results. The authors suggested applying the state-of-the-art texture-based techniques to improve the performance of pothole detection as a future work. The authors also proposed conducting additional experiments to detect other distresses such as cracks, depressions, rutting, and patching as a future work.

Pothole-detection techniques of vision-based methods are summarized as shown in

Table 2. Detection techniques and characteristics of vision-based pothole-detection method.

No.	Authors	Detection Techniques	Dataset	Performance	Experimental Circumstance
A1	Lim et al. [9]	Training and testing: Two modified models based on YOLOv2 - An anchor-box model based on YOLOv2 (A1-1) - A denser-grid model based on YOLOv2 (A1-2) - YOLOv2 (A1-3)	Total dataset: 1199 images - Training dataset: 996 images - Testing dataset: 203 images	Average precision A1-1: 0.6774, A1-2: 0.8243, A1-3: 0.6014 Recall A1-1: 0.7493, A1-2: 0.8372, A1-3: 0.6561 Frames Per Second (FPS) A1-1: 32, A1-2: 21, A1-3: 23	A GeForce GTX 1080 with 10 GB RAM
A2	Baek et al. [10]	Feature extraction: edge detection Training and testing: YOLO	Total dataset: 13,376 images - Training dataset: 70% - Validation dataset: 10% - Testing dataset: 20%	Mean-square error: 0.2–0.44 Average classification accuracy: 0.7786 Average classification precision: 0.8345	Windows 10, Intel Core i5-8440, 16 GB RAM, and 2070 super GPU
A3	Park et al. [12]	Training and testing - YOLOv4 (A3-1) - YOLOv4-tiny (A3-2) - YOLOv5 (A3-3)	Total dataset: 665 images - Training dataset: 70% - Validation dataset: 10% - Testing dataset: 20%	Mean average precision as 0.5 ([email protected]) - A3-1: 0.777 - A3-2: 0.787 - A3-3: 0.748	Tesla K80 GPU with 12 GB RAM Google Colab virtual machine
A4	Wanli Ye et al. [14]	Training and testing: Prepooling CNN	Total dataset: 96,000 images - Training dataset: 72,000 images - Testing dataset: 24,000 images	Average precision: higher than 0.80 * The precision of the optimized prepooling CNN: 0.9895	Intel Core i7-8750H CPU, 32 GB RAM, and a GeForce GTX 1080 8GB GPU
A5	Hanshen Chen et al. [15]	Training and testing: Location-aware CNN - A localization network (LCNN) - A part-based Classification network (PCNN)	Total dataset: 5676 images - Training dataset: 4026 images - Testing dataset: 1650 images	Accuracy: 0.950 Precision: 0.952 Recall: 0.920 F1 score: 0.936	A GeForce GTX 1080Ti GPU with 11 GB RAM
A6	Deepak Kumar Dewangan et al. [21]	Training and testing: CNN	Total dataset - Training dataset: 3915 images - Testing dataset: public dataset named as ‘Nienaber Potholes’	Accuracy: 0.9902 Precision: 0.9903 Recall: 0.9903 F1 score: 0.9833	an embedded vehicle prototype - monocular camera unit, Raspberry Pi unit
A7	Nhat-Duc Hoang [22]	Feature extraction: the Gaussian filter, a steerable filter and integral projection Training and testing: Two machine-learning algorithms - LS-SVM (A7-1) - ANN (A7-2)	Total dataset: 200 images - Training dataset: 160 images - Testing dataset: 40 images	Classification accuracy rate (CAR) - A7-1: 88.75%, A7-2: 85.25% Area under the curve (AUC) - A7-1: 0.96, A7-2: 0.92	N/A
A8	Penghui Wang et al. [26]	Pothole detection: the wavelet energy field is constructed by morphological processing, geometric criterions Pothole segmentation: Markov random field model	Total dataset: 120 images - Training dataset: 30 images - Testing dataset: 90 images	Accuracy: 0.867 Precision: 0.833 Recall: 0.875	MATLAB prototype
A9	Muhammad Haroon Yousaf et al. [28]	Image classification: a bag-of-words (BoW) approach Pothole localization - Feature extraction: Scale-Invariant Feature Transform (SIFT) - Training and testing: Support Vector Machine (SVM)	Total dataset: 120 images - Training dataset: 50 images - Testing dataset: 70 images	Accuracy: 0.957 Precision: 0.970 Recall: 0.941	a Core i3 computer having 4GB RAM using MATLAB 2015

.

3.2. Vibration-Based Method (Type B)

Ronghua Du et al. [30] proposed a road-surface-recognition method based on smartphone acceleration sensor when the vehicle passes through an abnormal road surface. The proposed method consists of three phases: data acquisition and preprocessing, abnormal-road-surface recognition, and abnormal-surface classification. The vehicle speed, acceleration, and position information were collected by the smartphone’s built-in accelerometer and Global Positioning Navigation system. The raw data were preprocessed using a Butterworth filter. The improved Gaussian model was used to recognize the abnormal road surface using the z-axis acceleration threshold condition. The training samples and the test samples were collected on two different roads for maintaining independence. The k-nearest neighbor (kNN) algorithm was used to classify the abnormal pavement types, including potholes and bumps. The authors measured the accuracy by comparing the field-measurement results for abnormal road surfaces such as potholes and bumps with the identification results of the proposed method. The test result shows that the accuracy of the recognition of the road-surface pothole is 0.9603, and the accuracy of the road-surface bump is 0.9412.

Azza Allouch et al. [31] proposed an automated pothole-detection method based on accelerometer and gyroscope built in a smartphone using the machine-learning technique. The proposed method consists of two main phases: a training phase and a prediction phase. The training phase consists of four steps: data collection, data preprocessing, features extraction and selection, and training a classifier. The raw data were collected using an accelerometer and gyroscope sensor built in the smartphone, and they were preprocessed by applying a low pass filter. The preprocessed data in the time domain were transformed the Fourier coefficients in the frequency domain using Fourier transform for feature extraction. The effective feature data were selected from the frequency and time-domain feature data using a correlation-based feature selection. The selected feature data were used to train a classifier model for identifying road conditions. A machine-learning technique, such as C4.5 decision tree, support vector machine (SVM), and Naive Bayes was used as classification algorithm. The subprocesses of the prediction phase are the same as the subprocesses of the training phase. The prediction of the road quality was performed by loading the classifier model, which was acquired in the training phase. The performance of the three algorithms used as classifier in the proposed method was compared using classification accuracy, precision, and recall. The experimental results showed that the performance of the C4.5 decision tree was better than the performance of support vector machine (SVM) and Naive Bayes.

Chao Wu et al. [32] proposed an automated pothole-detection method based on accelerometer and Global Positioning System receivers embedded in smartphones. The proposed method consists of four phases: data acquisition, data processing, feature extraction, and classification. The data-acquisition process was conducted online using a smartphone; other processes were conducted offline using a laptop. The vibration information was collected by using accelerometer and Global Positioning System of a smartphone in the vehicle. Data processing was performed to obtain useful features for pothole detection using sliding windows with simple threshold methods. Features were extracted from potential pothole windows in the time and frequency domain. Finally, machine-learning classifiers such as linear regression (LR), support vector machine (SVM), and random forest (RF) were applied for training and testing the dataset. The performance of the proposed method was evaluated by precision, recall, F1-score, and accuracy. The experimental results showed that the features from the time and frequency domain were better than other features for detecting a pothole. The random forest (RF) has the best performance among the testing classifiers, with a precision of 0.885 and recall of 0.750.

Pothole-detection techniques of vibration-based methods can be summarized as shown in

Table 3. Detection techniques and characteristics of vibration-based pothole-detection method.

No.	Authors	Detection Techniques	Dataset	Performance	Experimental Circumstance
B1	Ronghua Du et al. [30]	Data preprocessing: Butterworth filter Road-surface recognition: improved Gaussian model Road-surface classification: k-nearest neighbor	Training data sample - Bump: 118, Flat: 174, Pothole: 103 Testing data sample - Bump: 151, Flat: 583, Pothole: 68	Accuracy rate - Pothole: 0.9603 - Bump: 0.9412	Experimental vehicle: A-class vehicle (Cavalier) and SUV (Qoros 5) The smartphone is fixed on the handrail of the driver’s seat An Android-based app working on a smartphone (Redmi Note 8 Pro) Sampling frequency: 400 (Hz)
B2	Azza Allouch et al. [31]	Data preprocessing: low pass filter Feature extraction and selection: Fourier transform, correlation Classification - C4.5 decision tree (B2-1) - Support vector machine (B2-2) - Naïve Bayes (B2-3)	Total data sample: 2000	Classification accuracy - B2-1: 0.9860, B2-2: 0.9525, B2-3: 0.9690 Precision - B2-1: 0.985, B2-2: 0.951, B2-3: 0.972 Recall - B2-1: 0.985, B2-2: 0.953, B2-3: 0.969	The smartphone is mounted on the car dashboard An Android-based app working on a smartphone (Galaxy alpha) Sensor sampling rate: 50 (Hz)
B3	Chao Wu et al. [32]	Data preprocessing: sliding window Feature extraction and selection: Fourier transform, correlation Training and testing - Linear regression (B3-1) - Support vector machine (B3-2) - Random forest (B3-3)	Total dataset: 4088 potential windows - Normal windows: 3061 - Pothole windows: 474 - Transverse windows: 13	Accuracy - B3-1: 0.952, B3-2: 0.948, B3-3: 0.957 Precision - B3-1: 0.851, B3-2: 0.908, B3-3: 0.885 Recall - B2-1: 0.734, B3-2: 0.642, B3-3: 0.750	Experimental vehicle: A five-seater Shanghai-Volkswagen Lavida Sedan The smartphone is placed on the back seat without fixing An Android-based app working on a mobile network-connected smartphone Sensor sampling rate: 50 (Hz)

.

3.3. 3D Reconstruction-Based Method (Type C)

Amita Dhiman et al. [33] proposed four pothole-detection methods using computer vision and learning. Two methods are based on stereo-vision analysis of road environments ahead of the vehicle: the single-frame stereo-vision-based method and the multiframe fusion-based method. The other two methods are designed by using deep-learning techniques such as transfer learning with Mask R-CNN and transfer learning with YOLOv2. The authors experimented with the proposed four methods using the existing datasets [11,17,34,35,36] on websites and evaluated their performance in the aspects of precision, recall, and IoU. The experimental result showed that each method has its own benefits and can provide different pathways to a number of applications. The authors explained that the multiframe fusion-based method was suitable for identifying potholes and road manifolds with high accuracy. They also explained that transfer learning with YOLOv2 was capable of real-time pothole identification.

Muhammad Uzair Ul Haq et al. [37] proposed an automated pothole-detection method based on stereo imaging on roads using a hybrid-matching scheme that combines keypoint-matching and block-matching techniques. The proposed method consists of six phases: data acquisition, data preprocessing, stereo matching, 3D data generation, 3D data postprocessing, and pothole metrology. Static and dynamic images were collected from left and right cameras of the moving vehicle in the data-acquisition process. The dynamic images were filtered as a data preprocessing task before performing stereo matching. The hybrid-matching scheme was applied in the stereo-matching process using static and dynamic images. 3D data were generated by applying the stereo-triangulation process to the result of stereo matching. The 3D data was postprocessed for identifying the pothole and estimating the volume calculation. The performance of the proposed method was evaluated by accuracy qualification, stereo reconstruction, volume calculation, etc. The experimental result showed that the 3D measurements are within 3 mm for the static situation and 5 mm for the dynamic situation. It also showed that potholes are imaged to accuracies of −20% for volume, −15% for area, and −4% for depth at 10 km/h vehicle speed when compared to static imagery.

Jinchao Guan et al. [38] proposed an automated pixel-level pavement-distress-detection method based on stereo vision and deep learning. The proposed method consists of five phases: data acquisition, 3D reconstruction, 3D data processing, pixel-level pavement detection, pothole segmentation, and volume measurement. Parallel and oblique photos to be used as input data were obtained using the vehicle-mounted photography system. The 3D pavement point-cloud model was generated by applying stereo-vision technology to the input images in the 3D reconstruction phase. Point-cloud calibration was based on a principal-component-analysis (PCA) algorithm, and orthoimages such as color image, depth image, and overlapped image were generated in the 3D data-processing phase. A deep-learning technology such as a modified U-net was applied to segment a pavement crack and a pothole in orthoimages. Furthermore, the volume of a pothole was measured by integrating pothole segmentation and the proposed volume-measurement algorithm. The performance of the proposed method was evaluated by precision, recall, and F1-score. The experimental results showed that the precision, recall, and F1-score of the proposed method were 0.9632, 0.9552, and 0.9592. The authors explained that the performance of their proposed method is better than the performance of other existing methods [39,40] in the aspects of pothole segmentation through the performance comparison. The experimental results also showed that the overall relative error was 4.62% when comparing the manual volume measurements and automated volume measurements. The proposed method also has high accuracy in measuring the volume of a pothole through the experimental results.

Pothole-detection techniques of 3D reconstruction-based methods can be summarized as shown in

Table 4. Detection techniques and characteristics of 3D reconstruction-based pothole-detection method.

No.	Authors	Detection Techniques	Performance	Experimental Circumstance
C1	Amita Dhiman et al. [33]	Training and testing - Single-frame stereo-vision-based method (C1-1) - Multiframe fusion-based method (C1-2) - Transfer learning with Mask R-CNN (C1-3) - Transfer learning with YOLOv2 (C1-4)	Precision - C1-1: 0.458, C1-2: 0.674, C1-3: 0.898 Recall - C1-1: 0.458, C1-2: 0.512, C1-3: 0.928 Intersection over Union (IoU) - C1-4: 0.69	Training - a GeForce GTX 1080 GPU (C1-3) - TESLA K80 GPU (C1-4)
C2	Muhammad Uzair Ul Haq et al. [37]	Data preprocessing: filtering - high pass filter, histogram equalization Stereo matching: hybrid matching - keypoint matching, block matching - 3D data generation: stereo triangulation	Accuracy (Comparing the static imagery at 10 km/h vehicle speed) - volume: −20% - area: −15% - depth: −4%	A stereo pair of two A4Tech PKS-732 g webcams - be mounted on a tripod (for static imaging) - be mounted to the back of a car (for dynamic imaging) Image acquisition and processing: Window PC
C3	Jinchao Guan et al. [38]	3D data processing: principal component analysis (PCA) Pixel-level pavement detection: a modified U-net	Precision: 0.9632 Recall: 0.9552 F1-score: 0.9592 The overall relative error: 4.62%	A hatchback vehicle equipped with several GoPro HERO8 Three GoPRO HERO8 were fixed on the horizontal support in the rear of the vehicle The resolution of GoPro HERO8: 12 Megapixels

.

4. Conclusions

Automated pothole-detection technologies will reduce the labor for managing potholes and make a great contribution to planned road management. They are identified as a key element of autonomous technology, and research on this technology is expected to continue. According to the development trend of technology, it is considered that research on automated pothole detection should be continued. In this paper, the methods proposed in the latest research for each of the three types of automated pothole detection are analyzed and their characteristics are described. The process and results of recent automated pothole-detection research based on each method are also summarized. Feature extraction and training and testing play an important role in vision-based methods. Image-processing technologies such as edge detection and SIFT are applied in the process of feature extraction in those methods. Deep-learning technologies such as CNN, YOLO, and SVM are used in the process of training and testing in those methods. Vibration-based methods generally consist of three steps, namely data preprocessing, feature extraction, and classification. Signal-processing techniques such as filtering, Fourier transform, and correlation are applied in the process of data preprocessing and feature extraction. Machine-learning techniques such as k-nearest neighbor, linear regression, and random forest are used in the process of classification. 3D reconstruction-based methods that are presented in this paper include the process such as data processing and training and testing. Signal-processing and deep-learning techniques such as filtering, PCA, and a modified U-Net are applied in those methods. The performance of the proposed models based on the three methods is commonly evaluated for accuracy, precision, and recall. The weblink about the publicly available pothole datasets is provided for reader’s future research.

The automated pothole-detection methods presented in this paper are expected to be applicable in various fields such as road management in connection with the intelligent transportation system. It can be effectively used in the establishment of preventive road-maintenance policies by sharing the various feature information obtained in the pothole-detection process and performing big-data analysis. It is expected to make a great contribution to improving the performance of vehicle suspension systems by analyzing the pothole-detection results and the acceleration-sensor data. Furthermore, it is expected to greatly contribute to improving the performance of autonomous-driving technology by combining the results of pothole detection with real-time traffic information, which is provided by the intelligent transportation system.

In the course of the development of automated pothole detection, the application of deep-learning and machine-learning technology is gradually expanding. In particular, many researchers are working to improve the accuracy of pothole detection while applying various algorithms based on deep learning and machine learning [41]. In addition, research on edge devices capable of real-time pothole detection is being actively conducted. It is judged that improvements in pothole-detection accuracy and real-time pothole detection will become an important axis in future research trends. The application of common indicators such as IoU, mAP is required for objective accuracy evaluation when deep-learning and machine-learning techniques are applied in the pothole-detection process [42,43]. It is judged that the development and application of these common indicators will become the other important axis in future research trends. We plan to implement an edge device capable of real-time pothole detection using a public pothole dataset in the future. We also plan to secure the objectivity of the research by performing an accuracy evaluation using common indicators for the implemented system.