Road surface real-time detection based on Raspberry Pi and recurrent neural networks

This paper focuses on road surface real-time detection by using tripod dolly equipped with Raspberry Pi 3 B+, MPU 9250, which is convenient to collect road surface data and realize real-time road surface detection. Firstly, six kinds of road surfaces data are collected by utilizing Raspberry Pi 3 B+ and MPU 9250. Secondly, the classifiers can be obtained by adopting several machine learning algorithms, recurrent neural networks (RNN) and long short-term memory (LSTM) neural networks. Among the machine learning classifiers, gradient boosting decision tree has the highest accuracy rate of 97.92%, which improves by 29.52% compared with KNN with the lowest accuracy rate of 75.60%. The accuracy rate of LSTM neural networks is 95.31%, which improves by 2.79% compared with RNN with the accuracy rate of 92.52%. Finally, the classifiers are embedded into the Raspberry Pi to detect the road surface in real time, and the detection time is about one second. This road surface detection system could be used in wheeled robot-car and guiding the robot-car to move smoothly.


Introduction
The detection of road surface conditions -for example, flat road, gravel road, wavy road, grass road, bumpy road, upslope and downslope road -is essential to ensure the motion stability of the wheeled robot-car. Knowing the road surface condition and sending a correct command to the motors of the wheel are useful for the robot-car to move quickly and fluently. In addition, the road surface detection can provide great help for advanced driver-assistance systems or autonomous navigation systems in choosing a better driving way and making the users safe and comfortable.
A large number of recent papers in road surface detections clearly demonstrate an increasing interest in this research area (Alqudah and Sababha, 2017;Ng et al., 2018;Nomura and Shiraishi, 2015). Currently, since most smart mobile phones are installed with accelerometers and gyroscopes, the road surface detection was investigated in some papers by utilizing smart phones. In Nomura and Shiraishi (2015), a method for estimating and detecting changes of road surface conditions based on a smart mobile phone was investigated, and road surface conditions were classified into three levels. By calculating the second moment of sensor data in the mobile phone, the analysis of road surface conditions was studied (Alqudah and Sababha, 2017). Based on Alqudah and Sababha (2017), the identification of road surface conditions was investigated in view of internet-of-things (IoT) and machine learning (Ng et al., 2018). Being different from utilizing smart mobile phone, several road monitoring methods were proposed based on wireless sensor networks and vehicular sensor networks in Alvarez et al. (2014). The camera-based and forward looking classification of the road-surface was presented in Nolte et al. (2018), which is helpful for the early parametrization of vehicle control algorithms.
For designing vehicle control stability algorithms or trajectory control, the road-tyre friction coefficient is an essential model parameter for the control performance (Raj et al., 2012). One major impact factor of the friction coefficient is the road surface condition. A central challenge for the implementation of well-performing control algorithms of vehicle dynamics control systems needs to estimate the friction coefficient between the vehicles tyres and the road surface. Ambrozě t al. (2019) designed a low-cost device for measuring acceleration based on the Raspberry Pi and estimating friction between the vehicle tyres and the road. In order to monitor the degradation of road conditions, computer vision technology was used to detect the road crack and types. In Zhang et al. (2016), the automatic detection methods based on camera and supervised deep convolutional neural networks were studied for road surface crack detection. After that, the road crack surface condition detection was executed by utilizing the three-dimensional reconstruction method in Fan and Dahnoun (2018). In Carr et al. (2018), the road and pavement surface cracks detection was researched by utilizing the residual neural network. In addition, based on the novel data fusion scheme, the cracks detection from inspection videos of nuclear power plants has been studied by using convolutional neural network in Chen and Jahanshahi (2018). Deep learning is one of the key technologies for road surface detection and classification. Long short-term memory (LSTM) could be adopted for processing road surface data because of its time series prediction. Due to the complexity of the road scene and the illumination change, the road surface detection based on machine vision has the problems of weak adaptability, the poor robustness of illumination, and low accuracy of detection. Although a lot of efforts have been done to detect road conditions, a highly reliable method is yet to be built.
Raspberry Pi was developed in the laboratory of the University of Cambridge, and the first generation was released by Raspberry Pi foundation in February 2012 (Brock et al., 2013). As the single board computer, Raspberry Pi became affordable and gained popularity in the scientific community (Ambrozˇ, 2017). Hence, a large number of of highly cited academic papers utilizing Raspberry Pi in solving scientific and engineering challenges were investigated in Ambrozˇ(2017), Anwaar and Shah (2015), Leccese et al. (2014), Fletcher and Malalasekera (2016), Lewis et al. (2016), Go´mez et al. (2015), Vujovic´and Maksimovic´(2015. By utilizing Raspberry Pi, the construction and evaluation of a lowcost digital environmental monitoring system were studied in Lewis et al. (2016). Since Raspberry Pi is a programmable small computer and supports a large number of peripherals and network communication, an implementation of sensor web node as a part of IoT based on Raspberry Pi was proposed in Vujovic´and Maksimovic´ (2015). To the best of our knowledge, there are few road surface detection systems that make use of the cheap device Raspberry Pi and artificial intelligence method.
This paper utilizes Raspberry Pi, machine learning and RNN to detect road surface conditions, which is more effective to collect training data and realize real-time detection in low cost. Firstly, this hardware system mainly includes Raspberry Pi 3 B + , MPU 9250 and portable power at the bottom of the tripod dolly. The Raspberry Pi 3 B + and MPU 9250 are connected with jumper cables, and the portable power is used for the power supply of the Raspberry Pi. Secondly, this system collects several different kinds of labeled road surface data, such as flat road, gravel road, wavy road, grass road, bumpy road, upslope and downslope as training dataset. After that, the detection system adopts several machine learning algorithms and deep learning algorithms to train these data respectively and obtain several kinds of classifiers. Finally, these classifiers are embedded into the Raspberry Pi and used for realizing real-time road surface detection. The main advantages of this road surface detection system are listed as follows: Compared with the road surface detection system based on smart mobile phone, the price of this system based on Raspberry Pi 3 B + and MPU 9250 is cheaper than smart mobile phone. In addition, our road surface detection system is based on Scikit-Learn and Keras library built in Raspberry Pi, and it is efficient to implement the data collection, data training and real-time detection. Compared with the road surface condition inspection based on computer vision, our road surface detection system will not be affected by sunlight, which can be executed at any time. Compared with the existing road surface detection system, Raspberry Pi and MPU can be easily connected by using jumper cables, and the machine learning and deep learning classifiers based on python can be effectively embedded into Raspberry Pi and robot-car.

Data collection
This road surface detection system adopts Raspberry Pi 3 B + and MPU 9250 with a 3-axis accelerometer and a 3-axis gyroscope to collect road surface data. The MPU 9250 is installed at the middle of the tripod dolly pedestal, which gives the MPU 9250 have the equal chance to feel the vibration of the wheels. The MPU 9250 can also be installed in the other part of the tripod dolly pedestal. It is better to keep the MPU 9250 in the same location for collecting training data and detecting the road surface, which is helpful to get a high detection accuracy rate.
To preprocess the gyroscope and accelerometer signals, it is necessary to identify its noise components. When MPU 9250 collects data, deterministic error and random error will occur. In order to obtain accurate data, the above two kinds of errors need to be calibrated. The deterministic errors mainly include bias, scale, misalignment and so on, which are usually calibrated by the six-sided static method of accelerometers and gyroscops. The random errors mainly include gaussian white noise and bias instability. Allan variance method is usually used to calibrate the random errors of accelerometers and gyroscopes. Since Allan variance analysis could finely segment the power spectrum by adjusting the bandwidth of the Allan variance filter, it has obvious advantages to identify many different types of random process errors and quantitatively separate various error coefficients (El-Sheimy et al., 2008). This paper uses Allan variance method for analyzing IMU performance to calibrate white noise and bias instability of the accelerometers and gyroscopes. The noise is usually high frequency signal, and the low-pass filter with less delay is utilized to process the data collected by MPU 9250.
The hardware system can be seen in Figures 1 and 2. In addition, Figure 2 also shows the orientation of the accelerometers relative to the tripod dolly pedestal. When the Raspberry Pi and MPU 9250 are installed on the tripod dolly, the road surface condition detection system begins to collect the accelerometer and gyroscope data for several road surface conditions. The road surface conditions include flat road, gravel road, wavy road, grass road, bumpy road, upslope and downslope, which can be seen in Figures 3-9. For each road surface, the 5600 rows labeled raw data were collected by using tripod dolly in Figure 1.

Machine learning algorithms
In order to realize the real-time detection of the road surface conditions, the system should obtain the classifiers in advance.
In this section, the collected 5600 rows of labeled data for each kind of road surface are utilized to train classifiers. Each row of these training data is a 1 3 6 dimensional vector. The first three elements represent x-axis, y-axis and z-axis accelerometer, respectively. Similarly, the second group elements represent x-axis, y-axis and z-axis gyroscope, respectively. These training data are then processed to extract more useful statistical features such as, median, mean, variance, skewness, kurtosis, minimum, maximum, and so forth. The calculation methods for a part of extracted features are shown in Table 1. Based on Raspberry Pi 3 B + and Python 3.5, the system is built with Scikit-Learn software machine learning library. The hyperparameters tunning of machine learning classifiers are given as follows: (1) K-nearest neighbours (KNN): Number of neighbors is 5. The weight function used in prediction is uniform, and the power parameter for Minkowski metric is 2.    (2) Support vector machine (SVM): The kernel type is rbf. The tolerance for stopping criterion is 0.001, and the regularization parameter is 1.0. (3) Logistic regression (LG): L 2 is used to specify the norm used in the penalization. The tolerance for stopping criteria is 0.0001. (4) Decision tree (DT): Criteria that measure the quality of a split are Gini. (5) Support vector machine cross validation (SVMCV): The 3-fold cross-validation process is adopted. (6) Gradient boosting decision tree (GBDT): The number of boosting stages to perform is 300. The learning rate that shrinks the contribution of each tree is 0.1. The maximum depth of the individual regression estimator is 3.
The paper uses these machine learning methods to train the training dataset and obtain several different types of machine learning algorithms classifiers. After that, these classifiers are embedded into Raspberry Pi, and the system utilizes one of these classifiers as the road surface detection algorithm.
The detection system collects 2400 rows of data for each kind of road surface and makes a comparison among these classifiers. All the machine learning classifiers are trained based on Raspberry Pi. In this road surface real-time detection system, the detection time is about one second.

Recurrent neural networks classifiers
Neural networks that compose a large number of highly interconnected processing neurons are designed to emulate human brain (Aalizadeh, 2019;Han et al., 2018;Jiang et al., 2018;      Mao et al., 2018;Shao et al., 2018;Su et al., 2020;Wang et al., 2017;Yi et al., 2019). Recurrent neural networks (RNNs) are a class of artificial neural networks and a powerful model for sequential data since their hidden state is a function of all previous hidden states ( Graves et al., 2013). The time dependency of RNNs is achieved by recurrently connecting hidden layers at different time stamp, which is shown in Figure 10. RNNs have good performance in modeling the nonlinear time series data in an effective way . However, the traditional RNNs always are used to deal with short-term dependency, and they cannot well solve the long-time lags time series (Hochreiter and Schmidhuber, 1997;Elsheikh et al., 2019;Liu et al., 2018;Ma et al., 2015;Zhao et al., 2017). As a special class of RNNs, LSTM neural networks are specifically arisen to deal with the restrictions of RNNs in modeling the long-term dependencies. In the past few years, some representative results have been successfully achieved by using LSTM neural networks. In Zhao et al. (2017), a novel traffic forecast model based on LSTM network was proposed, and the proposed network could achieve a better performance in the vehicle prediction. A large-scale financial market prediction task was studied by using LSTM in Fischer and Krauss (2018). In addition, there were also many results based on LSTM, such as vehicle classification (Simoncini et al., 2018), speech recognition (Graves et al., 2013), online multi-target tracking (Milan et al., 2017), and so forth.
The main feature of LSTM is that the memory cells are contained in the hidden layers. In each memory cell, there are three gates including a forget gate f t , an input gate i t , and an output gate o t , which are used to maintain and adjust the cell state c t . The memory cell and network structure of LSTM neural networks are shown in Figures 11 and 12  .
The forget gate f t (1) defines the information that needs to be removed from the previous cell state c tÀ1 . The sigmoid (2) is used as the activation function of the forget gate. The activation value at time step t is computed based on the current input x t , the output h tÀ1 of the memory cells at the previous time step (t À 1), and the bias terms b f of the forget gates. The equations for these steps are as follows (Fischer and Krauss, 2018;Zhao et al., 2017) f t is the activation value of forget gate. W fx , W ft are the weight matrices. b f is the bias vector. x t is the time-series data at current time interval t. h tÀ1 is the output of the memory cell at the previous time interval (t À 1).
The input gate i t (3) defines the information that needs to be added to the cell state. This process includes two stages. Firstly, tanh (4) layer generates candidate values c t (5) that could be added to the cell states. Secondly, the activation values i t of the input gates are based on current input data x t and the output h tÀ1 of the memory cell at the previous time step (t À 1). The equations of these two steps are given in the following Zhao et al., 2017) tanh(x) = e x À e Àx e x + e Àx ð4Þ W cx , W ch , W ix and W ih are the weight matrices. b c and b i are the bias vectors. tanh represents the activation function. After that, the new cell states c t (6) are calculated based on the results of the previous two steps (Fischer et al., 2018) c t = f t 8 c tÀ1 + i t 8 c t s , a is mean value  ''8'' represents the Hadamard (elementwise) product. The equation represents that the current cell state c t is based on long term memory c tÀ1 and the current memory c t . The amount of information from long term memory c tÀ1 stored in the current cell state depends on the forget gate f t , and the amount of information from the current memory c t that is stored in the current cell state depends on input gate i t .
The output gate o t (7) defines the information that comes from the cell state generates output h t (8). The final output h t of the memory cell is computed in the following equations (Fischer and Krauss, 2018) W gx and W gh are the weight matrices. b g is the bias vectors, and h t is the output vector of the LSTM neural networks layer. Based on Raspberry Pi 3 B + , this paper builds Keras neural networks library to operate RNN and LSTM with Python 3.5. The system utilizes three layers of LSTM neural networks with 100 memory cells in each layer to train the road surface detection classifier based on Raspberry Pi and training data before the real-time detection is realized. When training the LSTM neural network, it is necessary to randomly initialise weight matrices W fx , W fh , W cx , W ch , W ix , W ih , W gx , W gx and W gh , the bias vectors b f , b c , b i , b g Zhao et al., 2017). RMSprop optimizer is used to train LSTM, and it restricts the oscillations in the vertical direction. Therefore, the learning rate is increased and our algorithm could take larger steps in the horizontal direction to converge faster. Based on the RMSprop optimizer, the weights and bias terms are trained to minimize the loss of objective function. The developed LSTM neural networks can be used to detect the road surface based on Raspberry Pi and MPU. In this road surface realtime detection system, the detection time is about one second based on RNN or LSTM classifiers.

Results and discussions
Raspberry Pi 3 B + is with a 64-bit quad core processor running at 1.4 GHz, dual-band 2.4 GHz and 5GHz wireless LAN, Bluetooth 4.2/BLE, faster Ethernet. The paper has built the machine leaning and deep learning library on the Raspberry Pi 3 B + and MPU 9250. In order to realize the road surface real-time detection system, there are three steps.
Firstly, based on machine learning and deep learning library built in the Raspberry Pi, the system collects several kinds of road surface (flat road, gravel road, wavy road, grass road, bumpy road, upslope and downslope) as the training dataset. For each road surface, 5600 rows of labeled data have been collected. Secondly, 100 rows of data are select as a window. The data in each window is a 100 3 6 dimensional vector, and this paper extracts some useful statistical features such as, median, mean, variance, skewness, kurtosis, minimum, maximum, and so forth, and connect all these statistical features to compose a big row vector. After that, the system trains these data based on Raspberry Pi 3 B + by utilizing machine learning and deep learning algorithms to obtain several kinds of classifiers. Thirdly, these classifiers are embedded into the Raspberry Pi to realize the real-time road surface detection. The detection time is about one second. When the system utilizes another 2400 rows of data as the testing dataset, the accuracy rates based on machine learning classifiers can be obtained, which is shown in Table 2. In the LSTM neural networks, this paper adopts three layers hidden neural networks, and each layer contains 100 memory cells. Based on simple RNN that contains 1 layer networks with 100 neural cells and LSTM neural networks, this paper can get the corresponding accuracy rates, which is shown in Table 3.
Among the machine learning classifiers, GBDT has the highest accuracy rate of 97:92%, which improves by 29:52% compared with KNN with the lowest accuracy rate of  75:60%. In RNN and LSTM neural networks classifiers, the accuracy rate of LSTM neural networks is 95:31%, which improves by 2:79% compared with RNN with the accuracy rate of 92:52%. The accuracy rates of road surface detection and classification based on other methods and classifiers (Bystrov et al., 2016;Raj et al., 2012;Rateke et al., 2019) are shown in Tables 4-6. In Table 4, based on different methods, the highest accuracy rate of classification using automotive ultrasonic sensor is 85:8%. In Table 5, based on traditional image-based approaches like the Canny edge algorithm, intensity histogram and so on, the average accuracy rate of six types of road surface detection is 90:35%. In Table 6, based on RTK and KITTI datasets, the accuracy rates of road surface detection using CNNs are 95:72% and 93:13%, respectively. In addition, the accelerometers and gyroscopes states for these several kinds of road surface are shown in Figures 13-26, and the differences between the 3-axis accelerometer and 3-axis gyroscope readings for different types of road surface can be observed. In this paper, the system mainly focuses on real-time road surface detection by using tripod dolly installed with Raspberry Pi 3 B + , MPU 9250. In future work, the road surface real-time detection system will be integrated into the robot-car and conduct the robot-car move smoothly, and the general flow charts can be seen in Figure 27. It is of essential importance for knowing road surface conditions and friction coefficient between the road surface and the tyres of the Table 4. The accuracy rates of classification using automotive ultrasonic sensor (Bystrov et al., 2016).

Methods
Accuracy  vehicles, which could ensure robot-car to run smoothly and safely. Since different types of road conditions have different frictions, in order to guide the robot-car to move smoothly under gravel road surface, grass road surface and other road surfaces, the control strategy must be changed accordingly. For example, when the road surface real-time detection system detects the flat road, the detection result will be sent to the processor. After that, the processor will send a command to the motors of wheels to make the robot-car move efficiently and quickly, which is similar to a moving car with high speed and less power. When the road surface detection system detects the bumpy road, the processor will send a corresponding command to make the robot-car move slowly and give more power to pass through the bumpy road, which is similar to a moving car with low speed but enough power.

Conclusion
In the road surface detection system, Raspberry Pi and MPU were illustrated to be affordable and effective in data collection and real-time detection of the road surface conditions. This system based on Scikit-Learn and Keras libraries has collected several kinds of road surface data including flat road, gravel road, wavy road, grass road, bumpy road, upslope and downslope by utilizing the tripod dolly equipped with Raspberry Pi 3 B + and MPU 9250, and several classifiers have been obtained based on training dataset. After that, machine learning classifiers and RNN classifiers are embedded to the Raspberry Pi, and the system adopts these classifiers to realize the road surface real-time detection. In the future, multi-sensor fusion method based on IMU and camera will be utilized to improve the accuracy and reliability of detection.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.