Self-Recalibrating Micromanipulator System for Resilient Robotic Vision-Based Control

A self-recalibrating micromanipulator system is developed in this work to perform vision-based control under unpredictable scenes and poor imaging conditions. To ensure resilient vision-based control in such challenging conditions, we proposed a self-recalibrating mechanism that combines the estimates from both the robot manipulator and microscope vision. Our method is demonstrated for micropipette cell aspiration. In the initialization stage, the system automatically locates the tip and selects the templates for visual tracking. At the self-calibration phase, accurate tracking of the tip is achieved by the background subtraction template matching (BSTM) method in order to estimate the calibration matrix in complex environments. During the operation, the control error is detected according to the visual information and the self-recalibration is executed when the error exceeds the threshold value. Experimental analysis shows that the BSTM method for self-calibration achieves accurate tracking with an average error of less than 1 pixel (0.18 μm). And the system has an average control error of 4.18 pixels (0.75 μm) under complex calibration backgrounds, satisfying the accuracy requirements for cell aspiration. Moreover, the system achieved stable tracking in complex operating environments, including cell interaction, occlusion, tip leaving the focal plane or beyond the field-of-view. By designing a self-recalibrating micromanipulator equipped with resilient vision-based control system, we hope to robotize the procedures for the study of cell mechanics during micropipette aspiration and pave the path for the advancement of robotic micromanipulation in biomechanical research.