Deep learning-based augmented reality work instruction assistance system for complex manual assembly

The manual assembly process of complex products is lengthy, the assembly requirements are difficult to recall, and the assembly quality requirements are high. The separation of the operation guidance information from the real physical object in the traditional paper manual easily distracts the operator, increasing their cognitive burden. To address this issue, we integrate augmented reality (AR) and artificial intelligence (AI) technologies for manual assembly assistance. A novel encoding and decoding convolutional neural network (CNN) is built to realize accurate AR registration under mark-less assembly environment. An assembly quality inspection method integrating a neural network and virtual model matching was proposed for AR systems. The specific process of the proposed method includes two stages: offline and online. In the first stage, a monocular RGB image dataset was built for assembling object keypoints and assembly object detection. CNN models were designed and trained for deployment in an AR-assisted assembly system. Second, the deployed CNN models were used to perform AR registration and assembly quality inspection. Experimental results demonstrate that the proposed method can accurately present AR work instruction guidance and assembly quality inspection for manual assembly. The proposed AR registration method can achieve a frame rate of 25FPS, which satisfies the timeliness requirements of the AR system. The accuracy rate of the proposed assembly quality inspection method for the MONA component assembly reached 92.5%.