read: Fourier PlenOctrees for Dynamic

3. Generalized PlenOctree Fusion

generalized NeRF Ψ: averaging the density σ and
Ψ are queried at every position with several view directions to predicted density and color
PlenOctree stores the averaged density and color
Filter out leaves having low density ($σ < 1e-3$) by “averaging” the queried results of 100 rendered views at the points, where the (cumulated) transmittance $T > 1e-3$ (close to the camera?).
“Coarse” means sparse adjacent views of a target view, while “fine” means dense adjacent views

“adopt PlenOctree to dynamic scenes by compressing time-variant information in the frequency domain”
4D Scene Representation: position and time (x,y,z,t)
high dimensional frequency domain: Mapping the position to Fourier Transform coefficients of the density σ(t) and each SH coefficient 𝐳(t)

$$Φ(x,y,z) = 𝐤^σ, 𝐤^𝐳$$

where $𝐤^σ ∈ ℝ^{n₁}$ (a sigma corresponds to n₁ DFT coefficients),

and $𝐤^𝐳 ∈ ℝ^{n₂ × (l_{max} + 1)^2 × 3 }$ (a point has $(l_{max} + 1)^2 × 3$ SH coefficients and each SH coefficient has n₂ DFT coefficients.)
Reconstruct density σ at time t by summing n₁ DFT coefficients of the Fourier PlenOctree with orthogonal basis:

$$\rm σ(t; 𝐤^σ) = ∑ᵢ₌₀^{n₁-1} 𝐤^σᵢ ⋅ IDFTᵢ(t)$$

where IDFTᵢ(t) = $\{^{cos(\frac{i\pi}{T} t) \quad \text{if i is even}} _{sin(\frac{(i+1)\pi}{T} t) \quad \text{if i is odd}}$
Reconstruct each SH coefficient at time t by summing n₂ DFT coefficients of the Fourier PlenOctree:

$$\rm z_{m, l}(t; 𝐤^𝐳) = ∑ᵢ₌₀^{n₂-1} 𝐤^𝐳_{m,l,i} ⋅ IDFTᵢ(t)$$

Aggregating the PlenOctrees of different frames at T times.

Leaves of different PlenOctrees at the same position are stacked, so the density σ and SH coefficients 𝐳(t) are stacked along the time axis.

The stacked vector performs DFT becoming Fourier coefficients $𝐤^σ, 𝐤^𝐳$, which are stored in Fourier PlenOctree.

Based on PlenOctree fusion “training”, the Fourier PlenOctree can be continuous optimizing via gradient descent.