Differentiable Surface Splatting for Point-based Geometry Processing

Felice Serena
Felice Serena
Cengiz Öztireli
Cengiz Öztireli

ACM Transactions on Graphics (TOG), pp. 230-14, 2019.

Cited by: 13|Bibtex|Views74|Links
EI
Keywords:
deep learning differentiable renderer neural renderer
Weibo:
We showed how a high-quality splat based differentiable renderer could be developed in this paper

Abstract:

We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We de...More

Code:

Data:

0
Introduction
  • Differentiable processing of scene-level information in the image formation process is emerging as a fundamental component for both 3D scene and 2D image and video modeling.
  • Assuming a differentiable loss function L(I) = L(R(θ )) on a rendered image I, the authors can update the parameters θ with the gradient ∂L ∂I ∂I ∂θ.
  • This view provides a generic and powerful shape-from-rendering framework where the authors can exploit vast image datasets available, deep learning architectures and computational frameworks, as well as pre-trained models.
  • The challenge, is being able to compute the gradient
Highlights
  • Differentiable processing of scene-level information in the image formation process is emerging as a fundamental component for both 3D scene and 2D image and video modeling
  • The challenge of developing a differentiable renderer lies at the intersection of computer graphics, vision, and machine learning, and has recently attracted a lot of attention from all communities due to its potential to revolutionize digital visual data processing and high relevance for a wide range of applications, especially when combined with the contemporary neural network architectures [Kato et al 2018; Liu et al 2018; Loper and Black 2014; Petersen et al 2019; Yao et al 2018]
  • We introduce Differentiable Surface Splatting (DSS), the first high fidelity point based differentiable renderer
  • We evaluate the performance of DSS by comparing it to state-of-theart differentiable renderer (DR), and demonstrate its applications in point-based geometry editing and filtering
  • We showed how a high-quality splat based differentiable renderer could be developed in this paper
  • We showcased a few applications of how such a renderer can be utilized for image-based geometry processing
Methods
  • A differentiable renderer R is designed to propagate image-level changes to scene-level parameters θ.
  • This information can be used to optimize the parameters so that the rendered image I = R (θ ) matches a reference image I∗.
  • Θ includes the coordinates, normals and colors of the points, camera position and orientation, as well as lighting
  • This can be formulated as an optimization problem θ ∗ = arg min L R (θ ) ∗ I (1).
Results
  • The authors evaluate the performance of DSS by comparing it to state-of-theart DRs, and demonstrate its applications in point-based geometry editing and filtering.

    The authors' method is implemented in Pytorch [Paszke et al 2017], the authors use stochastic gradient descent with Nesterov momentum [Sutskever et al 2013] for optimization.
  • Ours for points and normals, respectively.
  • The authors render in back-face culling mode with 256 × 256 resolution and diffuse shading, using RGB sun lights fixed relative to the camera position.
  • The authors optimize for up to 16 cycles of Tn and Tp optimization steps for point normal and position.
  • 12 randomly sampled views are used simultaneously for an optimization step.
  • The authors refer to Appendix A for a detailed discussion of parameter settings
Conclusion
  • CONCLUSION AND FUTURE WORKS

    The authors showed how a high-quality splat based differentiable renderer could be developed in this paper.
  • DSS inherits the flexibility of pointbased representations, can propagate gradients to point positions and normals, and produces accurate geometries and topologies.
  • These were possible due to the careful handling of gradients and regularization.
  • Developing DSS is the first step of transferring these techniques from image to geometry domain
  • Another fundamental application of DSS is in inverse rendering, where the authors try to infer scene-level information such as geometry, motion, materials, and lighting from images or video.
  • The authors believe DSS will be instrumental in inferring dynamic scene geometries in multi-modal capture setups
Summary
  • Introduction:

    Differentiable processing of scene-level information in the image formation process is emerging as a fundamental component for both 3D scene and 2D image and video modeling.
  • Assuming a differentiable loss function L(I) = L(R(θ )) on a rendered image I, the authors can update the parameters θ with the gradient ∂L ∂I ∂I ∂θ.
  • This view provides a generic and powerful shape-from-rendering framework where the authors can exploit vast image datasets available, deep learning architectures and computational frameworks, as well as pre-trained models.
  • The challenge, is being able to compute the gradient
  • Objectives:

    The authors aim to move points lying on a 2D grid to match the silhouette of a 3D teapot. The authors aim to cover all possible angles by sampling camera positions from a hulling sphere using farthest point sampling.
  • Methods:

    A differentiable renderer R is designed to propagate image-level changes to scene-level parameters θ.
  • This information can be used to optimize the parameters so that the rendered image I = R (θ ) matches a reference image I∗.
  • Θ includes the coordinates, normals and colors of the points, camera position and orientation, as well as lighting
  • This can be formulated as an optimization problem θ ∗ = arg min L R (θ ) ∗ I (1).
  • Results:

    The authors evaluate the performance of DSS by comparing it to state-of-theart DRs, and demonstrate its applications in point-based geometry editing and filtering.

    The authors' method is implemented in Pytorch [Paszke et al 2017], the authors use stochastic gradient descent with Nesterov momentum [Sutskever et al 2013] for optimization.
  • Ours for points and normals, respectively.
  • The authors render in back-face culling mode with 256 × 256 resolution and diffuse shading, using RGB sun lights fixed relative to the camera position.
  • The authors optimize for up to 16 cycles of Tn and Tp optimization steps for point normal and position.
  • 12 randomly sampled views are used simultaneously for an optimization step.
  • The authors refer to Appendix A for a detailed discussion of parameter settings
  • Conclusion:

    CONCLUSION AND FUTURE WORKS

    The authors showed how a high-quality splat based differentiable renderer could be developed in this paper.
  • DSS inherits the flexibility of pointbased representations, can propagate gradients to point positions and normals, and produces accurate geometries and topologies.
  • These were possible due to the careful handling of gradients and regularization.
  • Developing DSS is the first step of transferring these techniques from image to geometry domain
  • Another fundamental application of DSS is in inverse rendering, where the authors try to infer scene-level information such as geometry, motion, materials, and lighting from images or video.
  • The authors believe DSS will be instrumental in inferring dynamic scene geometries in multi-modal capture setups
Tables
  • Table1: Comparison of generic differential renderers. By design, OpenDR [Loper and Black 2014] and NMR [Kato et al 2018] do not propagate gradients to depth; Paparazzi [Liu et al 2018] has limitation in updating the vertex positions in directions orthogonal their face normals, thus can not alter the silhouette of shapes; Soft Rasterizer [Liu et al 2019] and Pix2Vex [Petersen et al 2019] can pass gradient to occluded vertices, through blurred edges and transparent faces. All mesh renderers do not consider the normal field directly and cannot modify mesh topology. Our method uses a point cloud representation, updates point position and normals jointly, considers the occluded points and visibility changes and enables large deformation including topology changes
  • Table2: Runtime and GPU memory demand for exemplar models in different applications. The images are rendered with 256 × 256 resolution and 12 views are used per optimization step
Download tables as Excel
Related work
  • In this section we provide some background and review the state of the art in differentiable rendering and point based processing.

    2.1 Differentiable rendering

    We first discuss general DR frameworks, followed by DRs for specific purposes.

    Loper and Black [2014] developped a differentiable renderer framework called OpenDR that approximates a primary renderer and computes the gradients via automatic differentiation. Neural mesh renderer (NMR) [Kato et al 2018] approximates the backward gradient for the rasterization operation using a handcrafted function for visibility changes. Liu et al [2018] propose Paparazzi, an analytic DR for mesh geometry processing using image filters. In concurrent work, Petersen et al [2019] present Pix2Vex, a C∞ differentiable renderer via soft blending schemes of nearby triangles, and Liu et al [2019] introduce Soft Rasterizer, which renders and aggregates the probabilistic maps of mesh triangles, allowing flowing gradients from the rendered pixels to the occluded and far-range vertices. All these generic DR frameworks rely on mesh representation of the scene geometry. We summarize the properties of these renderers in Table 1 and discuss them in greater detail in Sec. 3.2.
Funding
  • This work was supported in part by gifts from Adobe, Facebook and Snap, Inc
Study subjects and analysis
cases: 3
An illustration of the artificial gradient in two 1D scenarios: the ellipse centered at pk,0 is invisible (Fig. 4a) and visible (Fig. 4b) at pixel x. Φx,k is the pixel intensity Ix as a function of point position pk , qx is the coordinates of the pixel x back-projected to world coordinates. Notice the ellipse has constant pixel intensity after normalization (Eq (6)). We approximate the discontinuous Φx,k as a linear function defined by the change of pixel intensity ∆Ix and the movement of the ∆pk during a visibility switch. As pk moves toward (∆pk+ ) or away (∆pk− ) from the pixel, we obtain two different gradient values. We define the final gradient as their sum. Illustration of the 3 cases for evaluating Eq (10) for 3D point clouds. Comparison between RBF-based gradient and our gradient approximation in terms of the gradient value at pixel x and residual in image space x − xk as we optimize the point position pk in the initial rendered image to match the target image. While our approximation (blue) is invariant under the choice of the hyper-parameter σk , the RBF-based gradient (purple, orange and the dashed pink curves) is highly sensitive to its value. Small variations of σk can severely impact the convergence rate

Reference
  • Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34, 11 (2012), 2274–2282.
    Google ScholarLocate open access versionFindings
  • Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Claudio T Silva. 2003. Computing and rendering point set surfaces. IEEE Trans. Visualization & Computer Graphics 9, 1 (2003), 3–15.
    Google ScholarLocate open access versionFindings
  • Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. arXiv preprint arXiv:1903.07145 (2019).
    Findings
  • Matthew Berger, Andrea Tagliasacchi, Lee M Seversky, Pierre Alliez, Gael Guennebaud, Joshua A Levine, Andrei Sharf, and Claudio T Silva. 2017. A survey of surface reconstruction from point clouds. In Computer Graphics Forum, Vol. 36. 301–329.
    Google ScholarLocate open access versionFindings
  • Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. 2008. MeshLab: an Open-Source Mesh Processing Tool.
    Google ScholarFindings
Your rating :
0

 

Tags
Comments