Multi-modal deep fusion for bridge condition assessment

Bridge condition rating is a challenging task as it largely depends on the experience-level of the manual inspection and therefore is prone to human errors. The inspection report often consists of a collection of images and sequences of sentences (text) explaining the condition of the considered bridge. In a routine manual bridge inspection, an inspector collects a set of images and textual descriptions of bridge components and assigns an overall condition rating (ranging between 0 and 9) based on the collected information. Unfortunately, this method of bridge inspection has been shown to yield inconsistent condition ratings that correlate with inspector experience. To improve the consistency among image-text inspection data and further predict the accordant condition ratings, this study first provides a collective image-text dataset, extracted from the collection of bridge inspection reports from the Virginia Department of Transportation. Using this dataset, we have developed novel deep learning-base methods for an automatic bridge condition rating prediction based on data fusion between the textual and visual data from the collected report sets.