Multi-modal robotic visual-tactile localisation and detection of surface cracks.

We present and validate a method to detect surface cracks with visual and tactile sensing. The proposed algorithm localises cracks in remote environments through videos/photos taken by an on-board robot camera. The identified areas of interest are then explored by a robot with a tactile sensor. Faster R-CNN object detection is used for identifying the location of potential cracks. Random forest classifier is used for tactile identification of the cracks to confirm their presences. Offline and online experiments to compare vision only and combined vision and tactile based crack detection are demonstrated. Two experiments are developed to test the efficiency of the multi-modal approach: online accuracy detection and time required to explore a surface and localise a crack. Exploring a cracked surface using combined visual and tactile modalities required four times less time than using the tactile modality only. The accuracy of detection was also improved with the combination of the two modalities. This approach may be implemented also in extreme environments since gamma radiation does not interfere with the sensing mechanism of fibre optic-based sensors.