Dynamic Projection Mapping of Deformable Stretchable Materials

ABSTRACTWe present a method for dynamic projection mapping on deformable, stretchable and elastic materials (e.g. cloth) using a time of flight (ToF) depth camera (e.g. Azure Kinect or Pico-Flexx) that come equipped with an IR camera. We use Bezier surfaces to model the projection surface without explicitly modeling the deformation. We devise an efficient tracking method that tracks the boundary of the surface material using the IR-Depth camera. This achieves realistic mapping even in the interior of the surface, with simple markers (e.g. black dots or squares) or without markers entirely, such that the projection appears to be printed on the material. The surface representation is updated in real-time using GPU based computations. Further, we also show that the speed of these updates is limited by the camera frame rate and therefore can be adopted for higher speed cameras as well. This technique can be used to project on several stretchable moving materials to change their appearance.