Faithful Face Image Completion for HMD Occlusion Removal

Head-mounted-displays (HMDs) provide immersive experiences of virtual content. While being flexible, HMDs could be a hindrance for Virtual Reality (VR) applications such as VR teleconference where facial components and expressions of the user are partially occluded thus cannot be seen by others. We present an automatic face image completion solution that treats the occluded region as a hole and completes the hole with the help of an occlusion-free reference image of the same person. Given the occluded input image and an occlusion-free reference image, our method first computes head pose features from estimated facial landmarks. The head pose features, as well as images, are then fed into a generative adversarial network (GAN) to synthesize the output image. Our method can generate faithful results from various input cases and outperforms other face completion methods. It provides a light-weighted solution to HMD occlusion removal and has the potential to benefit VR applications.