Arxiv

Summary

(2022-01-08)

1. The input and output are the same as a volumetric model, but the loss function is based on the renderer error.
2. A volumetric model maps something to the attributes of every volumetric grid.
3. The output of a model is determined by the loss function. In NeRF, the loss is the rendering error between ground truth RGB image and the rendered image that is generated according to density (alpha channel) and RGB color at each point on the path of ray ¹. Therefore, the output of the network should be a filed of density and color.

Overview

diagram

Optimize Tricks

1. Positional encoding: mapping the inputs to a higher dimensional space so that MLP could converge to a high-frequency function.

   (2023-10-27) PE is Fourier transform and then MLP re-composes Fourier coefficicent to the final value.

   • will. - QQ group:

     傅里叶级数是对三角函数项的线性加权；神经网络的线性层是对输入的线性加权。 density 网络就是用傅里叶级数拟合了 sigma 在三维空间中的分布 用优化器优化了傅里叶级数，然后通过体渲染约束体密度的性质。

   • 赤子泛舟：

     我理解也是等效做了一次傅里叶变换和反变换，mlp其实在拟合频域特征。 虽然声称没用卷积，但其实整个过程相当于做了个大的卷积。

   • 群除我佬：就是一个更吊的3d输入。

2. Hierarchical volume sampling: allocating more samples to where it is expected to have a significant contribution on the final rendering.

viewdir is represented by Cartesian (x,y,z), rather than spherical coordinates (φ,θ,r)

σ is output of linear layer, so it lies in (-∞,+∞); but volume density is [0~∞), so σ(x) is activated by ReLU. Then it converts to alpha (the probability of terminating) by $(1-e^{-σ⋅δ})$, that is when σ=0, alpha=0 meaning no volume no block, while if σ→∞, the alpha→1 meaning this volume block light entirely. Therefore, the transmittance is (1-alpha)=$e^{-σδ}$, so the decay coefficient (cumulative transimitance) of a RGB color is $Π^{i-1}e^{-σδ}$

Math derivation 2.3 Integration along Ray Segmetns

Second sampling is based on “color decay coeff (weights)”, rather than volume density. The bigger color weights means more volume here. σ(x) is proportional to alpha and weights

Code

1. ImportError: No module named ‘_pywrap_tensorflow_internal’

   Create environment: conda env create -f environment.yml, and this error occured when executing import tensorflow.

   Solution: I didn’t creat the envrionment from the yml file but directly specify packages and version. conda create -n nerf python=3.7 cudatoolkit=10.0 tensorflow-gpu=1.15 -c conda-forge

Reference