IMMAT: Mesh Reconstruction from Single View Images by Medial Axis Transform Prediction

The representation of a 3D shape is a key element for capturing the overall structure as well as the local details. In this paper, we propose to predict a mesh representation of the Medial Axis Transform (called medial mesh) as an intermediate representation with our IMMAT framework, to reconstruct the 3D shape from a single view image. Because the MAT contains the skeleton topology and local thickness information, it not only enhances the ability to reconstruct topologically complex shapes but also better preserves the local details with its thickness control. The framework consists of three modules. The Image2Sphere module predicts the medial spheres inside the shape surface and the Topology Prediction module predicts the topological relationship (skeleton) between the predicted spheres. Then the MAT Smoothing module smooths the predicted MAT and fine-tunes the coordinates and radii of the spheres by graph convolution. The final 3D surface can be reconstructed by converting the predicted MAT to an implicit surface through CSG operation and then extracting the boundary surface through Marching Cubes. Experimental results show that our method outperforms the state-of-the-art methods both quantitatively and qualitatively on the reconstruction task. • We introduce MAT as the underlying representation for shape reconstruction from a single view image and propose a novel framework for MAT prediction. We have created a MAT dataset that will be open source and used for deep learning research. • We propose the Image2Sphere module, the first learning-based method for predicting medial spheres from a single view image, to simultaneously predict the spatial distribution and volume information of 3D shapes. • We propose a deep learning-based method to predict the topology relationships of 3D spheres and achieve high-quality reconstruction results with the generated MAT.