PDF | SemaSch

Notes

(2024-06-02)

Abstract

• Task: Display point set surface.

• Solution:

  1. Define a smooth manifold surface based on local maps and MLS.

     • local map from differential geometry:
     • “local”: neighbors

  2. Upsampling and downsampling to fit the points spacing with the screen space resolution.

  3. The approximation error is bounded.

• Advantage: Applicable for any point set

Introduction

• Task: Use point set as the shape representation

• Why matter?

  1. Point sets data are getting popular.

  2. Representing a highly detailed surface with primitives requires substantial small unit that’s smaller than a single pixel.

• Problem:

  1. Although point set is an economical representation, point sets contain noisy and redundant points.

  2. Point cloud is discrete without explicit surface.

• Solution:

  1. Adjust the density of points to reconstruct a smooth surface.

  2. Use polynomials to approximate surface with MLS

     • Polynomials: What are basis functions are used to fit the surface function?
       • What does the surface function look like?

     • Projection: weighted sum / linear combination of x,y coordinate

       $$ \begin{bmatrix} a_1 \\ a_2 \\ a_3 \end{bmatrix} [1 \quad x \quad x^2] [^x_y] $$

     • MLS: The partial derivative of error w.r.t. coefficient matrix 𝐀 equals 0.

• Explain idea:

  • Generating new points is a sampling process on the hidden surface.

    • Can generation model be applied? as this is a sampling process.
    • The error has a limited bound.

  • Rendering a point set is achieved by up-sampling and down-sampling the point set.

    • The sampling error is bounded. How to prove?

  • Reduce the input point set $P=\{𝐩ᵢ\}$ to “representation points” $R=\{𝐫ᵢ\}$ defining an MLS surface $S_R$, which approximates the original surface $S_P$.

    • Sounds like Variational Inference: Let an assumed posterior distribution of z $q(z|x)$ with learnable params (μ,σ) to approximate the true intractable posterior distribution of z $P(z|x)$, through minimizing the KL-divergence.

      On the other hand, if analyzing ELBO, the goal of maximizing the log-likelihood of x is transformed into maximizing the variational lower bound: ELBO.

    • Figuratively, use a proxy object to approximate the true object.

    

Related Work

(2024-06-07)

1. Consolidation

   • A practical point cloud includes multiple scans to capture the complete geometry for a nontrivial object.

   • Weighted sum based on implicit function

   • Weighted sum based on points directly

     • Trianglulation

   • Physically-based

   • MLS + Projection + Iterations

2. Point Sample Rendering

• time-critical rendering

• Global illumination effects -> ray tracing technique

• Resampling the surface in object space is better than interpolating in the screen space.

• Hybrid triangle-point approaches

Method

Define the Surface

• Basic assumption: The input point set represent a surface implicitly.

• Main idea:

1. The Projection Procedure: 2 steps

   1. Step Ⅰ: Approximate a Reference domain (plane) for a point 𝐫:

      $H =\{𝐱 | ⟨𝐧,𝐱⟩-D = 0, 𝐱∈ ℝ³\}, 𝐧∈ ℝ³,‖𝐧‖=1$

      The plane H is determined by MLS, i.e., minimizing a weighted sum of squared errors (distances) from each neighboring point 𝐩ᵢ of the point 𝐫 to the approximate plane H.

      The weight is a function θ of the distance between the projection 𝐪 of 𝐫 on the plane to each of its neighboring point 𝐩ᵢ.

      $$ ∑ᵢ₌₁ᴺ \underset{error}{(⟨𝐧,𝐩ᵢ⟩-D)²} ⋅ \underset{weights}{θ(‖𝐩ᵢ-𝐪‖)} $$

      • N is the number of neighbors of 𝐫.

      • A plane in 3D is defined with its normal vector 𝐧 and a point 𝐱 within the plane, that deviates from the origin by a distance D along the direction of normal: $⟨𝐧,𝐱⟩=D$

      Further, represent the projection 𝐪 with a parameter t:

      $$𝐪 = 𝐫 + t𝐧$$

      The loss function: encouraging neighboring points 𝐩ᵢ close to the H, becomes:

      $$∑ᵢ₌₁ᴺ ⟨𝐧,𝐩ᵢ - 𝐫 - t𝐧⟩^2 ⋅ θ(\| 𝐩ᵢ - 𝐫 - t𝐧 \|)$$

      • As the projection 𝐪 is on the surface, the distance from 𝐩ᵢ to the surface is written as the inner product of 𝐧 and 𝐩ᵢ.

      Denote the target plane H with the set of projections:

      $$Q(t) = 𝐪 = 𝐫 + t𝐧$$

      • The t reminds me the sampling in NeRF: $𝐨 + t𝐝$. After rescaling the range of [near,far] to [-1,1], the t ranges from [0,1] to sample points starting from the near plane.

        t is not the depth z, but steps that are compatible for LLFF scene (infinite boundary) and Blender scene (bounded).

      • Similarly, Point-MVSNet used steps to refine the depth of the point along a ray.

      The plane H will serve as a “ground” for measuring the height of each 𝐩ᵢ, using the coordinates (x,y) on the plane H for indexing.

   2. Step Ⅱ: approximate surface with Local map:

      

      The projeciton 𝓟 of 𝐫 onto the original surface $S_P$ is a sum of the projection of 𝐫 on the plane H and the “residual” approximed with a bivariate polynomial: g(x,y),

      $$ \begin{aligned} \cal{P}(𝐫) &= 𝐪 + g(0,0)𝐧 \\ &= 𝐫 + (t+g(0,0))𝐧 \end{aligned} $$

      • Similarly, PointFlow in Point-MVSNet also predicts the depth residual to tweak a point.

      The g(x,y) is optimized through minimizing the weighted sum of neigboring pionts 𝐩ᵢ height error:

      $$∑ᵢ₌₁ᴺ (g(xᵢ,yᵢ) - fᵢ)² ⋅ θ(\| 𝐩ᵢ - 𝐪 \|)$$

      • fᵢ is the height of the point 𝐩ᵢ w.r.t. the plane H:

        $$fᵢ= 𝐧⋅(𝐩ᵢ - 𝐪)$$

      • The projection 𝐪 of 𝐫 is the origin of the plane-H coordinate system.

        The projection of 𝐫 on the original surface can be represented as: $𝐪+g(0,0)⋅𝐧$

2. Properties of the Projection Procedure

   (2024-06-09)

   1. The equation holds, when the plane H is the original surface $S_P$:

      $$\cal{P}(\cal{P}(𝐫)) = \cal{P}(𝐫)$$

3. Computing the Projection

4. Data Structures and Tradeoffs

5. Results

Play