Assisting Visually Impaired People Using Deep Learning-based Anomaly Detection in Pedestrian Walkways for Intelligent Transportation Systems on Remote Sensing Images

INTRODUCTION

Currently, in metropolitan areas, intelligent transportation systems were devised to help decrease traffic, the rate of injuries, accidents, and deaths, diminish environmental pollution, and reduce fuel consumption, etc. (Khan et al., 2022). Such structures use diverse technologies [deep learning (DL), Internet of Things, image processing, and neural network (NN)] for many applications. Alternatively, technology companies (namely Tesla and Google) are attempting to develop self-driving cars that could offer safe travel for persons while the driver is sleepy, which could save his life along with other travelers’ lives, with automatic control cars to avoid accidents (Pustokhina et al., 2021). Such automobiles should be fortified with sensors for sensing the atmosphere, finding objects close to the car, and notifying the driver, and have actuators to do real-time processes while the driver is sleepy or fails to pay attention to hazards to avoid accidents (Mansour et al., 2021). Intelligent transportation systems were advanced for real-time operations and automatic road management in the situation of a natural accident (like an icy road, a mountain fall, and an avalanche) or unnatural disasters (namely car traffic accidents and road repairs) (Chang et al., 2022). Such systems utilize sensors as a tool to understand and find the state of travel atmosphere. The data gathered by such sensors by means of communication technologies [dedicated short-range communication (DSRC and WiFi)] to other vehicles on the route are transferred to control centers (Khan et al., 2022).

The captured visual data have details that are precise when compared with substitute data sources namely radar signals, global positioning system (GPS), mobile transmission, etc. (Avola et al., 2022). In addition, it serves a main role in predicting accidents and other abnormal actions by collecting particulars concerning road traffic circumstances. The usage of closed-circuit television (CCTV) is useful in different real-time applications like accidental falls and grade crossing trespassing (Sabuhi et al., 2021). Several computer vision (CV) relies on works were modeled by focusing processes namely behavioral learning, data acquisition, and feature extraction. The intention of studies is to calculate processes like vehicle prediction and observation, scene detection, anomaly prediction approaches, activity examination, multi camera-relied schemes and difficulties, video processing models, traffic observation, and human behavior learning (Dey et al., 2019). Now, anomalous prediction is a sub-field of behavior learning from captured visual scenes. Machine learning (ML) refers to a domain of artificial intelligence (AI) that utilizes statistical approaches to learn hidden patterns from prevailing data and to make decision regarding unobserved records (Abdullah and Jalal, 2023). The core task of machine learners is constructing a general model on the possible dispersal of training samples and generalizing experience to unseen instances. The learning procedure relies upon the data quality exhibited (Aldayri and Albattah, 2022). An instance is presented in a dataset with various properties. Inappropriately, mining proficient features seems to be hard for a few tasks.

This manuscript presents a new sine cosine algorithm with deep learning-based anomaly detection in pedestrian walkways (SCADL-ADPW) method. The SCADL-ADPW techniques focus on the identification and classification of anomalies, i.e. vehicles in the pedestrian walkways of visually impaired people (VIP). To accomplish this, the SCADL-ADPW technique uses the VGG-16 model for feature vector generation. In addition, the SCA approach is designed for the optimal hyperparameter tuning process. For anomaly detection, long short-term memory (LSTM) method can be exploited. The experimental results of the SCADL-ADPW technique are studied on the UCSD anomaly detection dataset.

RELATED WORKS

Yang et al. (2018) devised to seize convolutional neural network (CNN)-related per-pixel semantic segmenters to cover navigational perception necessities in a unified way. It is not critical to observe crosswalk location or traffic light signals, but analyze the states of other vehicles and pedestrians. At the centroid of unification proposal is a DL structure, aspired to achieve robust and potential semantic understanding. Shimakawa et al. (2019) deal with obstacle recognition to support VIP walks. Previous studies had adopted a method of utilizing some image processing methods for finding the steps of staircase on depth imageries by an RGB-D camera. Still, it was hard to carry the RGB-D camera. This study implemented CNN, a type of DL approach to hindrance detection, and advanced an application program functioning on smartphones.

Lin et al. (2018) devised a visual localizer that has global optimization and ConvNet descriptor for attaining strong visual localization for aided navigation. Through concatenation of two compressed convolution layers of GoogLeNet, thousands of bytes are utilized for denoting image proficiently. Hsieh et al. (2021) developed an assistive mechanism to help VIP walk outdoors. Depending on the depth map acquired by ZED 2 and CNN-NN-FAST-SCNN, the image of the environment in front of the VIP can be devised into seven equal divisions.

Bhattacharya and Asari (2021) devised a method that incorporated wearable devices and Android devices to support VIP and make it simpler to navigate safely. The glove sync and the Android device are connected through Bluetooth connection, and the vibration actuator needs haptic feedback from the user based on the instruction from the paired devices. Le et al. (2022) recommend a novel BGN for camera-related recognition of pedestrian lanes in unstructured scene. For the VIP safety, along with an output segmentation map, network provided two complete-resolution maps of epistemic and aleatoric uncertainty in addition to well-calibrated confidence measure.

PROPOSED ANOMALY DETECTION MODEL

In this manuscript, we have devised a new SCADL-ADPW technique for anomaly detection purposes. The proposed SCADL-ADPW technique identifies the occurrence of anomalies in the pedestrian walkways on remote sensing images. The SCADL-ADPW techniques focus on the identification and classification of anomalies, i.e. vehicles in the pedestrian walkways of VIP. Figure 1 represents the workflow of the SCADL-ADPW algorithm.

Figure 1:

Workflow of the SCADL-ADPW approach. Abbreviation: SCADL-ADPW, sine cosine algorithm with deep learning-based anomaly detection in pedestrian walkways.

Feature extraction module

The SCADL-ADPW techniques use the VGG-16 model for feature vector generation. The VGG-16 is the essential basis of CNN (Deepa and Chokkalingam, 2022). VGG designs and trains object detection where the most important CNN is VGG-16. In 2015, Google researchers introduce the InceptionV3 network. The high computation cost related to CNN can be overcome by the pretrained structure namely VGG-16. Figure 2 illustrates the infrastructure of VGG-16. In the presented method, the pooling, convolution, output and fully connected layers are the four main elements included in CNNs.

Figure 2:

Structure of VGG16.

Convolutional layer

The building block in CNNs are a convolution layer. The convolution-related method convert inputs into outputs. The fixed height and width with plane composes the input volume. The depth D can be determined by number of planes. The amount of M planes with the volume structures presented in the convolution layer. The amount of filters or kernels with a small size (K) determines all the planes. The weight with filter size is 3 × 3. Y_K shows the matrix value in terms of k^th input plane.

(1) $$X_l=F\left(b_l+{\displaystyle \sum _{k=1}^D{w}_{kl}\ast y_k}\right);l=1,\dots ,M,$$

where w_kl denotes the parametric weight. Over kth input planes, the 2D matrix w_kl * Y_k is generated by means of convolution of these filters. The F activation function transform the outcome and bias function b_l .

The feature map resultant matrix X_l . The feature map denotes M. In this work, the activation function was ReLU. While keeping all the positive input, it changes every value of input to 0.

(2) $$F\mathrm{(}y)=\text{Max}(0,y)$$

The convolution layer output employs the ReLU function. The network learning speed and classification accuracy improved.

Pooling layer

The feature map size can be decreased by means of the pooling layer after all the blocks of a layer or convolution layer. The crucial data are preserved while avoiding inappropriate details. The squared section and the stride (s) of feature map size (F) are the two crucial parameter to determine pooling.

The s = 2 and F = 2 with five pooling layers existing in VGG-16. The block organizes 13 convolution layers which comprise the VGG-16.

Fully connected and output layer

Attain feature map set afterwards the successive and pooling convolution layer. The input–output function implemented by all the neurons of the fully connected (FC) layer is represented as follows:

(3) $$Z=F\left(b+{\displaystyle \sum _j^n{w}_j{Y}_j}\right)$$

Therefore, the w_j and Y_j neuron denote input features and neuron weight. The FC layer output is Z:

(4) $$F\mathrm{(}{x}_j)=Softmax\mathrm{(}{x}_l)=\frac{e^{\mathrm{(}{x}_l)}}{{{\displaystyle \sum }}_{k=1}^8{e}^{\mathrm{(}{x}_k)}},for\hspace{0.17em}l=1,\dots \mathrm{..},8$$

The probability distribution outcome can be attained by using the output layer of all the node’s outcomes.

EXPERIMENTAL EVALUATION

The proposed model is simulated using Python 3.6.5 tool on PC i5-8600k, GeForce 1050Ti 4GB, 16GB RAM, 250GB SSD, and 1TB HDD. The parameter settings are given as follows: learning rate: 0.01, dropout: 0.5, batch size: 5, epoch count: 50, and activation: ReLU.

In this section, the anomaly recognition in pedestrian walkways results is examined on the UCSD anomaly detection dataset. Figure 3 illustrates the sample images.

Figure 3:

Sample images.

In Table 1 and Figure 4, the recognition accuracy of the SCADL-ADPW approach is examined with current methods on the Test004 dataset. The outcomes guaranteed the improvement of the SCADL-ADPW technique on the detection process. S-40 sequence had an increased detection accuracy of 99.11%, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methods have reported decreased detection accuracy of 96, 92.51, 80.41, and 78.21% respectively. Likewise, on S-46 sequence, an increased detection accuracy of 99.85% was observed, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methodologies have reported decreased detection accuracy of 98.14, 96.60, 85.17, and 75.48% respectively. Similarly, on S-106 sequence, an increased detection accuracy of 99.23% was observed, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methodologies have reported decreased detection accuracy of 97.62, 100.55, 90.77, and 82.73% respectively. Eventually, S-137 sequence had an increased detection accuracy of 99.37%, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methods have reported decreased detection accuracy of 98.04, 99.49, 90.81, and 84.76%, respectively. Finally, on S-180 sequence, an increased detection accuracy of 99.10% was observed, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methods had decreased detection accuracy of 99.39, 99.42, 85.96, and 84.24%, respectively.

Figure 4:

Accu_y outcome of the SCADL-ADPW approach on Test004 dataset. Abbreviation: SCADL-ADPW, sine cosine algorithm with deep learning-based anomaly detection in pedestrian walkways.

In Table 2 and Figure 5, the recognition accuracy of the SCADL-ADPW approach is inspected with current approaches on the Test004 dataset. The results guaranteed the improvement of the SCADL-ADPW technique on the detection process. On S-40 sequence, an increased detection accuracy of 96.78% was observed, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methods had decreased detection accuracy of 99.36, 97.44, 92.26, and 87.73%, respectively. Similarly, on S-46 sequence, an increased detection accuracy of 96.97% was observed, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methodologies have reported decreased detection accuracy of 99.89, 99.91, 95.76, and 89.31%, correspondingly. Likewise, on S-106 sequence, an increased detection accuracy of 96.35% was observed, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT methods had decreased detection accuracy of 89.42, 86.97, 85.98, and 85.46%, correspondingly. Eventually, S-137 sequence had an increased detection accuracy of 96.80%, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT algorithms had decreased detection accuracy of 88.96, 85.52, 84.48, and 83%, respectively. Finally, S-180 sequence had an increased detection accuracy of 96.47%, while the DLADT-PW, RS-CNN, Fast R-CNN, and MDT systems had decreased detection accuracy of 90.05, 86.15, 83.47, and 83.78%, correspondingly.

Figure 5:

Accu_y outcome of the SCADL-ADPW approach on Test007 dataset. Abbreviation: SCADL-ADPW, sine cosine algorithm with deep learning-based anomaly detection in pedestrian walkways.

The overall anomaly detection results of the SCADL-ADPW technique have been compared with other DL models in Table 3 and Figure 6 (Pustokhina et al., 2021). The outcomes notify the improved average accuracy results over other models. On the Test004 dataset, the SCADL-ADPW technique achieves improved average accu_y of 99.24% while the RS-CNN, Fast R-CNN, MDT, and MPPCA techniques attain decreased average accu_y of 98.07, 98.03, 84.86, and 81.33%, respectively. Besides, on the Test007 dataset, the SCADL-ADPW method achieves improved average accu_y of 96.59%, while the RS-CNN, Fast R-CNN, MDT, and MPPCA approaches decreased average accu_y of 92.36, 89.84, 85.39, and 80.93%, respectively.

Figure 6:

Average accu_y outcome of the SCADL-ADPW approach on two datasets. Abbreviation: SCADL-ADPW, sine cosine algorithm with deep learning-based anomaly detection in pedestrian walkways.

These results ensured the effectual anomaly detection performance of the SCADL-ADPW technique.

CONCLUSION

In this manuscript, we have designed a new SCADL-ADPW technique for anomaly detection purposes. The proposed SCADL-ADPW technique identifies the occurrence of anomalies in the pedestrian walkways on remote sensing images. The SCADL-ADPW techniques focus on the identification and classification of anomalies, i.e. vehicles in the pedestrian walkways of VIP. To accomplish this, the SCADL-ADPW technique uses the VGG-16 model for feature vector generation. In addition, the SCA approach is designed for the optimal hyperparameter tuning process. For anomaly detection, the LSTM model can be exploited. The experimental results of the SCADL-ADPW technique are studied on the UCSD anomaly detection dataset. The comparative outcomes stated the improved anomaly detection results of the SCADL-ADPW technique.

The authors declare no conflicts of interest in association with the present study.