Image-Based 3D Reconstruction of Cleft Lip and Palate Using a Learned Shape Prior

We present a novel pipeline that takes smartphone videos of the intraoral region of newborn cleft patients as input and produces a 3D mesh. The mesh can be used to facilitate the plate treatment of the cleft and support surgery planning. A retrained LoFTR-based method creates an initial sparse point cloud. Next, we utilize our collection of existing scans of previous patients to train an implicit shape model. The shape model allows for refined denoising of the initial sparse point cloud and; therefore, enhances the camera pose estimation. Finally, we complete the model with a dense reconstruction based on multi-view stereo. With Moving Least Squares and Poisson reconstruction we convert the point cloud into a mesh. This method is low-cost in hardware acquisition and supports minimal training time for a user to utilize it.