Lighting Estimation Via Differentiable Screen-Space Rendering

In this paper, we present a method that estimates the real-world lighting condition from a single RGB image of an indoor scene, with information of support plane provided by commercial Augmented Reality (AR) frameworks (e.g., ARCore, ARKit, etc.). First, a Deep Neural Network (DNN) is used to segment the foreground. We only focus on the foreground objects to reduce computation complexity. Then we introduce Differentiable Screen-Space Rendering (DSSR), a novel approach for estimating the normal and lighting condition jointly. We recover the most plausible lighting condition using spherical harmonics. Our approach provides plausible results and considerably enhances the visual realism in AR applications.