Arxiv

Summary

(2024-08-01)

• Novel view synthesis is a solution for image-based rendering task.

• NeRF leverages volumetric representation and volume rendering for novel view synthesis. While LLFF uses light field representation and MPI for NVS.

  As Ben quoted Kajiya: “… in 10 years, all rendering will be volume rendering.” on ECCV2022Tutorial

(2022-01-08)

1. The input and output are the same as a volumetric model, but the loss function is based on the renderer error.
2. A volumetric model maps something to the attributes of every volumetric grid.
3. The output of a model is determined by the loss function. In NeRF, the loss is the rendering error between ground truth RGB image and the rendered image that is generated according to density (alpha channel) and RGB color at each point on the path of ray ¹. Therefore, the output of the network should be a filed of density and color.

Overview

diagram

Optimize Tricks

1. Positional encoding: mapping the inputs to a higher dimensional space so that MLP could converge to a high-frequency function.

   (2023-10-27) PE is Fourier transform and then MLP re-composes Fourier coefficicent to the final value.

   • will. - QQ group:

     傅里叶级数是对三角函数项的线性加权；神经网络的线性层是对输入的线性加权。 density 网络就是用傅里叶级数拟合了 sigma 在三维空间中的分布 用优化器优化了傅里叶级数，然后通过体渲染约束体密度的性质。

   • 赤子泛舟：

     我理解也是等效做了一次傅里叶变换和反变换，mlp其实在拟合频域特征。 虽然声称没用卷积，但其实整个过程相当于做了个大的卷积。

   • 群除我佬：就是一个更吊的3d输入。

2. Hierarchical volume sampling: allocating more samples to where it is expected to have a significant contribution on the final rendering.

viewdir is represented by Cartesian (x,y,z), rather than spherical coordinates (φ,θ,r)

σ is output of linear layer, so it lies in (-∞,+∞); but volume density is [0~∞), so σ(x) is activated by ReLU. Then it converts to alpha (the probability of terminating) by $(1-e^{-σ⋅δ})$, that is when σ=0, alpha=0 meaning no volume no block, while if σ→∞, the alpha→1 meaning this volume block light entirely. Therefore, the transmittance is (1-alpha)=$e^{-σδ}$, so the decay coefficient (cumulative transimitance) of a RGB color is $Π^{i-1}e^{-σδ}$

Math derivation 2.3 Integration along Ray Segmetns

Second sampling is based on “color decay coeff (weights)”, rather than volume density. The bigger color weights means more volume here. σ(x) is proportional to alpha and weights

(2024-07-20)

• Ben has an derivation in his ECCV2022 report

  • Opacity is alpha. If so, the opacity and alpha in 3DGS may should be understood as: opacity is an attribute of the scene elements, while alpha is an attribute of the filter contributing to pixels.

  • Density is used in the continuous volume render function, whereas Alpha is used the discrete alpha compositing.

• Challenge: Indirect illumination is even more computationally-expensive than rendering with direct lighting. Angjoo

(2024-07-22)

• NeRF 是在各射线上采样数据点，用MLP拟合 整个三维空间的从坐标到 RGBA 的函数； MLS 是用多项式拟合各点 局部的曲面函数，然后在各点的 local plane 采样以实现 resampling。

Code

1. ImportError: No module named ‘_pywrap_tensorflow_internal’

   Create environment: conda env create -f environment.yml, and this error occured when executing import tensorflow.

   Solution: I didn’t creat the envrionment from the yml file but directly specify packages and version. conda create -n nerf python=3.7 cudatoolkit=10.0 tensorflow-gpu=1.15 -c conda-forge

Reference