HiStyle: Reinventing Historic Portrait for 3D Colorization and Zero-shot Stylization

Restoring and reinventing historical portraits has long been a challenging task in computer vision. In order to faithfully explore the portrait images, it is necessary to restore the color and reconstruct the 3D geometry. Furthermore, the ability to stylize historic portraits is crucial for extending their use to different forms of media. Existing methods for each specific task make huge progress. However, they struggle with conflicts of multi-tasks, which hinders their ability to meet the requirement in a unified model. To achieve this goal, we propose HiStyle, a generative model for generally reinventing historic portraits, which simultaneously realizes 2D-to-3D reconstruction, gray-to-RGB restoration, and photo-to-style image translation. We introduce a GAN inversion technique to transfer a gray historical portrait into the latent space of a 3D generator, restoring the lost color information and meanwhile lifting the 2D image to 3D representation. Besides, we incorporate the power of CLIP model with 3D-aware GANs to achieve zero-shot text-driven style transfer. The results demonstrate the superior of HiStyle in the quality and diversity of synthesized images and also highlight the potential of 3D-aware GANs for preserving cultural heritage.