Multi-Modal Autonomous Ultrasound Scanning for Efficient HumanCMachine Fusion Interaction

Robotic autonomous ultrasound imaging is a challenging task as robots require strong analytical capabilities to make sound decisions in complex spatial relationships. In this paper, we integrate visual and tactile information into the ultrasound robotic system drawing inspiration from the process of human doctors conducting ultrasound scans, and explore the impact of different modalities of information on our task. The proposed multimodal deep reinforcement learning (DRL) framework can integrate real-time visual feedback and tactile perception, and directly output 6D pose decisions to control the ultrasound probe, thereby achieving fully autonomous ultrasound imaging of soft, movable, and unmarked targets. We demonstrate the feasibility of our method on a simulation platform and propose an effective model transfer learning method. Subsequently, we conducted further evaluations of the approach in a real-world environment. The results indicate that our approach effectively enhances the performance of autonomous ultrasound scanning and manual adjustments further optimize the outcomes.