1. Introduction

Protective coatings are the main route used to prevent corrosion of marine and offshore structures. Checking the protective coating is a key issue for asset management, but traditional visual inspection methods are time-consuming and labor-intensive. The project aims to develop an automatic coating inspection system for corrosion management applications. The developed system will be able to quickly and comprehensively screen and evaluate coating conditions and can be used as a scanning tool to help investigate the identification and classification of coating failures.

In order to achieve a comprehensive collection of data for coating condition assessment, a micro-aerial vehicle (MAV) assisted by the latest technological innovations, will be used to facilitate data collection in inaccessible locations. An image-based coating breakdown and corrosion (CBC) assessment system has been developed to provide objective assessment of the severity of coating failure. After inspection is completed, the automatic CBC assessment system (A-CAS) is able to generate an inspection report. The inspection method developed and the test data obtained in this paper can assist the inspection and maintenance team to improve efficiency and productivity. It reduces maintenance cost, improves structural integrity and minimizes risk of failure. Compared to conventional inspection, ACAS can reduce coating failure evaluation time and manpower.

The structure of the paper is as follows. Section 2 lists related works, Section 3, Section 4 and Section 5 present innovative detection solutions for CBC evaluation, introduces image-based CBC evaluation methods, and explains non-destructive hidden corrosion assessment method. Finally, conclusions are drawn in Section 6.

2. Background

This paper is an extension of work originally presented in the 13th IEEE conference on industrial electronics

Figure 1: UAS for automated CBC assessment

and applications (ICIEA), 2018 [1]. Referring to automated vision-based coating corrosion detection, to the best of the authors’ knowledge, In [2], Jahanshahi and Masri utilize color wavelet-based texture analysis for corrosion detection. Ji et al. [3] apply watershed transform over the gradient of gray images. Siegel et al. [4] choose wavelets for characterizing and detecting corrosion texture in airplanes, Zaidan et al. [5] focus on corrosion texture using standard deviation and entropy as discriminating features. Last but not the least, Ortiz et al. [6] present a solution for coating breakdown and corrosion detection which adopts a semi-autonomous MAV, and an artificial neural network (ANN) to discriminate between pixels suspected/not suspected of corresponding to coating breakdown and corrosion areas through sufficient color and texture descriptors.

2.1. UAS Facilitated Data Collection

Nowadays, visual inspection is widely used in both industry and daily life for security or maintenance checking purposes. Sometimes, visual inspections take place in areas that put the inspector in highly risky situations, hence extra precautions have to be taken, resulting in increasing inspection costs and turnaround time. All these negative factors create difficulties in inspection and would delay the subsequent processes. To meet the needs of inspection, different types of unmanned vehicles were suggested and introduced to replace manual inspection. In coming years, as the technology in aircraft flight control and video processing and transmission systems becomes mature, unmanned arial system (UAS) would be a solution. It is imperative to use MAVs to replace in-situ human inspection in order to save the cost of preparing the vessel for inspection. Indoor vessel inspection requires the aerial platform to be capable of stable, low speed flight and have anti-collision functions. For research purposes, we customized a MAV with a protective cage and a RGB-D camera for indoor anti-collision coating failure inspection to facilitate our data collection.

In this study, we propose a UAS facilitated CBC assessment system as shown in Figure 1. First of all, a customized autonomous AeroLion drone equipped with necessary inspection tools conducts vessel condition inspection; semi-autonomous navigation technology fulfills the flight capability requirement in indoor environments. The data collected from the drone are then transferred to web-server for post-image analysis including CBC detection and localization in the reconstructed 3D model. Image mapping and image indexing technologies are being developed for traceability of the inspection. The customized Aerolion drone is integrated with a Nvidia Jetson TX2 embedded processor and a ZED RGB-D stereo camera for visual-based simultaneous localization and mapping (vSLAM) mapping, navigation and data collection. Due to the drone’s computational power limitation, post-image processing on a separate platform is required for detailed CBC assessment. Figure 1 visualizes an integrated postprocessing system. First, a web-server will integrate CBC analysis, image mapping and the CBC visualization model. Then, the HTTP-based client application can be accessed through mobile tablets. Finally, AIbased autonomous CBC inspection results are sent to the surveyor for result verification and report generation. In this paper, we will focus on the automated CBC assessment system.

Figure 2: A-CAS system process

Instead of image data analysis by certified surveyors, this paper presents one deep learning based visual CBC detection system. The system can automatically recognize and analyze the coating condition by classifying and determining the type of CBC. Corrosion includes but is not limited to CBC on edges, CBC on welds, and CBC on surfaces including hard rust and pitting. Due to the limited size of the available dataset for system development, a transfer learning network was applied to learn the model for CBC detection. An AI-assisted CBC assessment system was successfully developed for the maritime industry. In addition to the visual camera, an infrared/thermal camera will be used for inspection of corrosion behind coating. Thermal radiation from areas of the ship’s surface with CBC will be different from that without CBC, and this difference can be identified and visualized in some cases. A technology of active infrared thermography to detect the hidden CBC in the vessel structures by combining an external heating source and infrared thermography is developed during the study. The thermal images collected by the drone will be also analyzed by the CBC detection system.

3. Autonomous CBC Assessment System

A vision-based protective coating inspection system was developed for automatic CBC assessment. With this program, even a relatively inexperienced surveyor will be able to make an objective judgment on the severity of CBC. This would improve the efficiency and reliability of coating inspection while reducing the time and manpower cost required. Algorithm development is divided into five phases. They are: region of interest (ROI) proposals, CBC feature extraction, CBC feature learning and prediction, CBC detection and CBC measurements.

In this paper, CBC detection and segmentation are combined for CBC instance aware segmentation. Instance aware segmentation is challenging because it requires proper detection of all CBCs in the image. In practice, due to the limited data set, transfer learning on convolutional activation feature (TLCAF) network [7] is used for CBC feature extraction and feature learning. Models are generated for CBC prediction, segmentation and measurements.

As shown in Figure 2 (b), labeled features are visualized in 3D space and the three categories including surface CBC, non-coating-failure and edge CBC can be differentiated in hyperplane. Figure 2 shows a typical example of the instance-aware semantic segmentation algorithm for coating failure assessment to speed up the inspection process. The predicted CBC ROIs are reconstructed by background removal. The ROIs are then sent to the next step for active segmentation [8] and CBC measurement. Figure 2 (e, f) demonstrates edge detection and active segmentation, where the background has been masked by active segmentation. The CBC assessment results are more accurate and constant than the ground truth images indicated in [9].

3.1. Feature Extraction and Visualization

Figure 3 lists the five classes of CBC. Different ROIs are manually labeled as different categories. The three categories used in this algorithm are surface CBC, edge CBC and non-coating-failure. Surface CBC can be further categorized into hard rust and pitting. Edge CBC can be divided into CBC on edges and CBC on welds. In order to balance dataset, the three major categories CBC including surface CBC, edge CBC and non-coating-failure are used for prediction for this work.

Figure 3: CBC classification

During the study, TLCAF network [7] is used to learn the region of interest for different types of coating failure including surface CBC and edge CBC. Bounding box is used for region of interest proposal, Faster RCNN [10, 11] framework and vgg19 model [10] are used for convolutional neural network (CNN) feature extraction, and a linear classifier is used for feature classification. Then, the ROIs are reconstructed for different types of coating failure for instance-aware semantic segmentation. Here, active segmentation[8] is used for background removal. The seeds generated in the previous step are used for auto-guided active selective propagation. Ten iterations of propagation can generate the optimal result which matches with the ground truth which is the value provided by human surveyors’ assessment.

2000 randomly selected CBC image features were chosen for the t-distributed stochastic neighbor embedding (t-SNE) [12] high-dimension feature visualization experiment. Figure 4 represents the distribution of three types of CBC features through t-SNE. High dimension CNN features of the three types of CBC including surface-based CBC (yellow), edge-based CBC (green) and non-coating-failure (blue) can be easily distinguished in hyper-plane for feature prediction.

Figure 4: Feature visualization in t-SNE

Figure 5 represents a high-level features learnt from the three categories of extracted CBC features using deep dream. It shows that features for CBC on surfaces are some red dots with sharp gradient decent, while features for CBC on edges/welds are red dots along lines which means the corrosion happens on edges/welds. The non CBC features do not show any red corrosion color or texture. After extracting the R-CNN features, softmax [13] is used for multi-class feature classification and prediction.

Figure 5: Feature visualization through deep dream

3.2. Feature Learning and Prediction

In the study, 1900 images with CBC were labeled for feature extraction; 12,184 features were extracted and divided into three broad categories, namely edge CBC, non-CBC and surface CBC. A random selection of 2437 features (20% of total features) was used for verification. In the experiment, vgg19 was chosen for network training. Stochastic gradient descent with momentum was used for loss function optimization; the learning rate was set to 1e-6; the maximum number of epochs for training was 20; and a mini-batch with 487 observations was used at each iteration. The max iteration was 9740. Validation frequency was set to 487 iterations duration. Validation patience value, the number of times that the loss on validation set can be larger than or equal to the previously smallest loss before network training stops, was set to five. During the experiment, the training process stopped when it reached the final iteration. The confusion matrix in Figure 7 demonstrates the verification accuracy. The total recognition accuracy rate observed was 89.54%.

The recognition accuracy can be improved by increasing the dataset. Continuous data augmentation will benefit the development of the CBC inspection system, improve CBC generalization and increase accuracy.

Figure 6: Detected CBC

Figure 7: Feature prediction result

3.3. Feature Detection

Randomly generated bounding boxes are used to propose ROIs for different categories of CBC prediction. The predicted CBC ROIs are reconstructed with background removal. Instance-aware segmentation was developed for CBC instances’ background removal.

The predicted CBC ROIs are reconstructed with background removal. Then, the reconstructed image is sent to the next step for active segmentation [8] and CBC measurement. A database with thousands of photos related to coating failure and corrosion was built up, which is expanding continuously as additional photos are collected from inspections. The system accurately identifies the main types of features, including surface CBC and edge CBC. Figure 6 shows classic examples of ROIs for CBC evaluation to speed up the inspection process. The CBC measurement is more precise and consistent than the ground truth images indicated in [9]. The ground truth images’ assessment results are visually estimated by human surveyors and there is bias between different surveyors.

3.4. CBC Measurement

For CBC measurements, hue, saturation and value

(HSV) color space is chosen to represent CBC color features. The extracted HSV data are used to identify possible CBC pixels to generate seeds for CBC detection. The pre-fetched HSV data contain most of the necessary information for pixel-wise CBC measurement. Our system gives users the flexibility to choose how they want to obtain the image that requires analysis. They can choose to either “Upload” or “Capture” an image with the device’s camera. The algorithm 1 describes the detailed work flow. The system accurately identified the main types of CBC characteristics, including surface CBC and edge CBC. The study provides a comprehensive A-CAS system for marine and offshore industries. After post-analysis of the RGB image data (448 * 448 bytes), the system was able to promptly generate the inspection report containing field measurement data and the analysis result, as shown in Figure 8. A web-server for data communication will be developed to integrate the local server and client in autonomous on-line process. With the integration of all the techniques developed in this study, the AI-facilitated drone inspection system is able to assist surveyors to increase their efficiency during vessel condition inspection.

Figure 8: Report generation

Figure 9: Image Enhancement

Algorithm 1: CBC Assessment System

Input: The original image

Output: CBC assessment result

• Input an image and convolve network through different CNN layers;
• Propose relative ROIs;
• ROIs pooling;
• Prediction through fully connected layer with softmax;
• Morphological methods for edge CBC analysis, output processed image;
• If an edge is discovered Edge CBC ROIs reconstruction;

Hough transform finds lines;

Active selection for propagation;

Edge CBC quantitative measurement;

Endif;

• If surface CBC is detected Surface CBC ROIs reconstruction;

Active semantic segmentation;

Surface CBC quantitative measurement;

End-if;

• Adjust the weight for CBC on welds, if any;
• Overall condition grading;
• Generate report;
• The end

The application program can be installed and operated on portable tablets and laptops. The CBC assessment report is automatically generated according to international standards such as IACS Recommendation 87 and IMO recommendations [14] (such as IMO MSC.1/Circ.1330 and IMO MSC.1/Circ.1399). Our program’s results are consistent with the ground truth measurements of the surveyors. Our system continues to collect field data to optimize our algorithms and correlate the results with the ground truth results obtained by traditional methods.

4. Active Human Intervention to Improve Accuracy

The section above introduced a fully automated CBC assessment system (A-CAS) for effective coating failure inspection. This method is more suitable for inspecting large areas by capturing and analyzing images of the target area than the surveyor’s existing manual inspection solution. However, the machine generated result is not 100 percent correct, and active human intervention and identification can correct and re-adjust these imperfect results. The following section describes how human intervention can fine-tune the result with the help of relevant advanced image processing methods.

4.1. Image Enhancement

Under normal circumstances, lighting will not affect the CBC assessment results. However, non-ideal lighting conditions, such as too high or too low light intensity, can affect the CBC assessment. So a function for adjusting brightness is implemented to facilitate CBC inspection. As shown in Figure 9 (a1) and (a2), we noticed that this function is helpful in certain cases.

Another affiliate function is Gamma correction. By exploiting the non-linear approach of human perception of light and color, Gamma correction works by optimizing the use of bits when encoding images, or transferring the bandwidth of an image. Figure 9 (b1) and (b2) show an example of before and after the Gamma correction process. It shows that this feature is also useful.

Figure 10: Method I: Auto-report generation

The third image enhancement function we implemented is histogram equalization. Histogram equalization was developed to increase contrast. The effect of histogram equalization is demonstrated in the two sets of examples in Figure 9 (c1) and (c2), (d1) and (d2). For CBC, whether histogram equalization helps to improve accuracy is situational dependent. For example, Figure 9 (c1) and (c2) shows that histogram equalization is not necessary, as the dust in Figure 9 (c1) is converted to the CBC color dataset, and the image looks more corroded after the histogram is equalized. For Figure 9 (d1) and (d2), the image before processing, Figure 9 (d1) has low brightness contrast, while Figure 9 (d2) shows that the brightness contrast has increased after processing, and CBC colors in the image better correspond to corrosion colors in the CBC color dataset. Therefore, histogram equalization is necessary for case (d1).

4.2. Ground Truth Adjustment

For special CBC color dataset adjustment, the reassign/reset ground truth function was developed as shown in Figure 11 for result fine-tuning. New special type of HSV values for CBC color can be added to CBC color dataset for CBC measurements.

Figure 11: Ground truth adjustment

4.3. Active ROI Selection

There are two ways the system can measure edge CBC. Method I is the system default and the result is autogenerated by the system after input of images through the “Upload and Process”, “Capture and Process” or “Batch Processing” buttons as shown in Figure 12. Figure 10 demonstrates a typical result based on method I. It shows the edge CBC is detected with 22.54%. The ROIs of CBC on edges which are auto-proposed by the CBC detection system is not perfect.

Figure 12: Method I: Auto report generation

Figure 13: Method II: active ROI selection

Figure 14: Method II result

Figure 15: Typical results based on Method II

Therefore, in order to align with human experience, active ROI selection method, i.e. method II, is developed to let machine interact with human and learn from human. It involves hard assignment of ROI especially for CBC on edges. As shown in Figure 13, the user needs to choose the “Tools” drop-down menu, followed by “Development Tools” and then click “Region of Interest” for ROI selection and edge CBC measurement. After that, a new figure pops out to show the original image, where the user can draw a rectangle box on the image to select a ROI, and then right-click to “crop image”. Afterwards, the edge CBC measurement result will be shown in the main GUI in Figure 14. This edge CBC result is 8.21%, which is more accurate than method I. Figure 15 displays two typical results by using method II. The results are more in line with human surveyors’ experience.

5. Non-destructive Hidden Corrosion Assessment

Hidden corrosion behind coated surfaces is a problem that could cause dangerous failure, causing shut down of production processes and high cost to the marine and offshore industry [15, 16]. At present, hidden corrosion can only be visually observed by human eyesight after cutting and/or coating removal. Compared to existing solutions, active infrared thermography (IRT) is relatively faster and non-destructive. Combined with algorithms for auto CBC detection, condition monitoring with active IRT can maintain higher precision and quality of assessment for consistent defect detection and measurement. While ultrasonic methods are limited to inspection of small areas, active IRT is fit for inspecting large areas, producing thermal images of the target. Active IRT can be used as an auxiliary tool for rough defect area positioning, followed by ultrasonic inspection for precise defect depth measurement. It can also be applied to risk management for cost savings and improving safety in the maritime industry. In the last 10 years, different research institutes have studied active IRT, but this technique has not been widely taken up by industry. The major problem is the long heating duration, making it unfit for on-site assessment. But with advances in heating technology and sensitivity of thermal cameras, active IRT has potential to “see through” more advanced materials [17, 18, 19].

In this project, a mid-wave thermal imaging camera (FLIR a6702sc) and a halogen heater are chosen for instant heat generation to collect data in near real time for post image processing and analyzing [20]. By using a mid-wave thermal imaging camera FLIR A6703sc in combination with a 1500W halogen heater, the corrosion area could be imaged after only 5-10 seconds of heating. After removing the heat source, a temperature difference develops between the corroded and noncorroded areas, and the thermal contrasts as shown in Figure 16 (a) are captured by the thermal camera. In addition, the video captured is passed through machine learning algorithms for corrosion prediction and assessment. Figure 16 (b) is the processed result of hidden corrosion behind the coating for assessment.

Active infrared thermal imaging developed to detect hidden CBCs can be used to examine areas that are not identifiable by visual inspection. RBG imaged based CBC detection is used for 1st round screening test. Thermal camera is used for 2nd round hidden corrosion assessment once necessary. This will greatly increase the inspection effectiveness and help to predict and reduce the potential risk caused by the hidden corrosion.

6. Summary

The CNN-based automated CBC assessment system developed includes new coating breakdown and corrosion assessment algorithms with higher accuracy to facilitate quantitative data analysis for effective coating failure inspection. Compared to existing manual inspection solutions by surveyors, this method is more suitable for inspecting large areas by means of capturing and analyzing pictures/videos of the target areas.

The system accurately identified the main types of CBC characteristics, including surface CBC and edge CBC. The study provides a comprehensive ACAS system for marine and offshore industries. A coating condition evaluation report is automatically generated according to international standards such as IACS Recommendation 87 and IMO recommendations. The detection system uses deep TLCAF technology to automate CBC assessment, and an instance-aware semantic segmentation method for CBC measurement and grading was developed. The three main types of CBCs can be distinguished and used for CBC prediction through the last fully connected layer. A model for coating failure prediction is generated, followed by CBC segmentation, CBC quantitative measurement and grading. The developed A-CAS system is going to improve maritime coating inspection works faster and smarter. With the assistance of IRT, the presence of corrosion behind protective coatings will be identifiable. Additionally, the non-destructive approach eliminates the need to remove the coating system.

Figure 16: Hidden corrosion

The developed A-CAS system will make maritime coating inspection safer and more efficient. Shipping companies will be able to save on docking costs as well as eliminate losses incurred from goods left on the dock. The process of maritime coating inspection is set to have a makeover in the near future.

The days of cumbersome CBC inspections are gone forever as surveyors’ jobs become highly automated. This automated corrosion assessment system can also be implemented in other industries that also face the problem of inefficient coating inspection. Such industries include the aviation, railway and building industries.

Conflict of Interest

The authors declare no conflict of interest.

Acknowledgment

The author would like to thank Dr. Hai Gu of the American Bureau of Shipping (ABS) for his technical advice and data support. In addition, we are very grateful to Mr. Kelvin Lim Chee Quan, Mr. Ma Yiheng, Mr. Donvis Nguyen, Mr. Yeo Eng Hoe Jason and Mr. Duncan Goh Yitang for their contributions to the study. This work was funded by the Singapore Maritime Institute (grant no. SMI-2015-OF-05) and conducted at SP.

1.  L. Liu, E. Tan, Y. Zhen, X. J. Yin, Z. Q. Cai, Ai-facilitated coating corrosion assessment system for productivity enhancement, in: 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE, 2018, pp. 606–610.
2.  M. Jahanshahi, S. Masri, Effect of color space, color channels, and sub-image block size on the performance of wavelet-based texture analysis algorithms: An application to corrosion detection on steel structures, in: Computing in Civil Engineering (2013), 2013, pp. 685–692.
3.  G. Ji, Y. Zhu, Y. Zhang, The corroded defect rating system of coating material based on computer vision, in: Transactions on Edutainment VIII, Springer, 2012, pp. 210–220.
4.  M. Siegel, P. Gunatilake, G. Podnar, Robotic assistants for aircraft inspectors, Industrial Robot: An International Journal 25 (6) (1998) 389–400.
5.  B. Zaidan, A. Zaidan, H. O. Alanazi, R. Alnaqeib, Towards corrosion detection system, International Journal of Computer Science Issues 7 (3) (2010) 33–36.
6.  A. Ortiz, F. Bonnin-Pascual, E. Garcia-Fidalgo, et al., Visionbased corrosion detection assisted by a micro-aerial vehicle in a vessel inspection application, Sensors 16 (12) (2016) 2118.
7.  L. Liu, R.-J. Yan, V. Maruvanchery, E. Kayacan, I.-M. Chen, L. K. Tiong, Transfer learning on convolutional activation feature as applied to a building quality assessment robot, International Journal of Advanced Robotic Systems 14 (3) (2017) 1729881417712620.
8.  S. Dutt Jain, K. Grauman, Active image segmentation propagation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2864–2873.
9.  A. American Buireau of Shipping, Guidance notes on the inspection maintenance and application of marine coating system third edition, Book, 2007.
10.  S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards realtime object detection with region proposal networks, in: Advances in neural information processing systems, 2015, pp. 91–99.
11.  J. Schmidhuber, Deep learning in neural networks: An overview, Neural Networks 61 (2015) 85–117.
12.  L. V. D. Maaten, G. Hinton, Visualizing data using t-sne, Journal of Machine Learning Research 9 (Nov) (2008) 2579–2605.
13.  G. Hinton, R. Salakhutdinov, Replicated softmax: an undirected topic model, in: Advances in neural information processing systems, 2009, pp. 1607–1614.
14.  M. Resolution, 215 (82),“performance standard for protective coatings for dedicated seawater ballast tanks in all types of ships and double-side skin spaces of bulk carriers”, IMO, London, UK.
15.  P. Traverso, E. Canepa, A review of studies on corrosion of metals and alloys in deep-sea environment, Ocean Engineering 87 (2014) 10–15.
16.  Z.Wang, Y. Cong, T. Zhang, et al., Effect of hydrostatic pressure on the pitting corrosion behavior of 316l stainless steel, Int J Electrochem Sci 9 (2014) 778–798.
17.  Y. Hung, Y. S. Chen, S. Ng, L. Liu, Y. Huang, B. Luk, R. Ip, C. Wu, P. Chung, Review and comparison of shearography and active thermography for nondestructive evaluation, Materials Science and Engineering: R: Reports 64 (5) (2009) 73–112.
18.  J. R. Brown, H. Hamilton, Quantitative infrared thermography inspection for frp applied to concrete using single pixel analysis, Construction and Building Materials 38 (2013) 1292–1302.
19.  L. Liu, I.-M. Chen, E. Kayacan, L. K. Tiong, V. Maruvanchery, Automated construction quality assessment: A review, in: Mechatronics and its Applications (ISMA), 2015 10th International Symposium on, IEEE, 2015, pp. 1–6.
20.  C. Duberstein, D. Virden, S. Matzner, J. Myers, V. Cullinan, A. Maxwell, Automated thermal image processing for detection and classification of birds and bats, Offshore Wind Technology Assessment.

(Click Here)

(Click Here)