read: NVS - NeRF | MVS-NeRF

(2023-08-16) Re-read

Experiments

Train model only on (MVSNet) DTU dataset, where the objects are partitioned the same as PixelNeRF (imgsize: 300x400).

Datasets (Download):

Stage	Data	Contents	Resolution	N_views	Size
Train	DTU	88 scenes	512x640		19G
Test	DTU	16 scenes		3/20
Test	LLFF
Test	Blender

Feature map: 32 channels
Cost volume: 128 planes
MLP: 6 layers
Coarse-to-fine: One field and fine-tune
Ray pts: 128
Device: 2080Ti
Batch size: 1024 rays
Optimizer: Adam, lr=5e-4

Old Notes

Old notes on 2022-12-06

Abstract

3 nearby input views
plane-swept cost volumes
geometry-aware scene reasoning
generalize across scenes

1 Introduction

Topic: Novel view synthesis
Recent progress: neural rendering
Former solutino and drawbacks:
1. NeRF and its following works can produce photo-realistic novel view synthesis results. But they need a long-time per-scene optimization
Own Solution:
- Goal: use … to …
- 1 sentences introduce the name and functionality, properties.
- Analysis:
  1. generalizability -> avoid tedious per-scene optimization and regress directly novel viewpoints
  2. quantitative outcomes
Tech stack
1. MVSNet -> generalizable net of 3D reconstruction
  - Cost volume is built by warping 2D img features of src views onto sweeping planes
2. Regress geometry and appearance from a cost volume (per-voxel neural features)
  - 3D cnn aggregates the cost volumes to a neural scene encoding volume

(2023-08-16)

Play

Environment

inplace-abn needs to be installed from source, as issue#36. But I cannot install it:

1
2
3


RuntimeError: 
The detected CUDA version (10.2) mismatches the version that was used to compile
PyTorch (11.3). Please make sure to use the same CUDA versions.

My cuda version is 11.3 as shown by:

1
2


import torch
print(torch.version.cuda)

Then I specified the version the same as the pl-tutorial of AIkui:
1 2 3 4 5 6

- pip - pip: - torch==1.11.0 - torchvision==0.12.0 - pytorch_lightning==1.6.0 - inplace_abn
With that, inplace_abn has installed.

But the API has change since v1.5 (as issue) and resulted in: Exception has occurred: TypeError __init__() got an unexpected keyword argument 'distributed_backend'

Try this setting:

1
2
3
4


   - torch==1.10.1
   - torchvision==0.11.2
   - pytorch_lightning==1.3.5
   - inplace_abn

Cannot import inplace_abn:

1
2


Exception has occurred: ImportError
/home/zichen/anaconda3/envs/mvsnerf/lib/python3.8/site-packages/inplace_abn/_backend.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPvmm

I checked my cuda version, which is 10.2:

1
2
3
4
5


(mvsnerf) zichen@lambda-server:~/Downloads/mvsnerf-comments$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

According to Previous PyTorch Versions | PyTorch, I install the versions compiled with cuda10.2. Then print(torch.version.cuda) returned 10.2 instead.

1
2
3
4
5
6
7
8


- pip
- pip:
   - torch==1.10.1+cu102 
   - torchvision==0.11.2+cu102 
   - torchaudio==0.10.1 
   - -f https://download.pytorch.org/whl/cu102/torch_stable.html
   - pytorch_lightning==1.3.5
   - inplace_abn

Still cannot import inplace_abn with the same ImportError as above:

Then Install from source, and inplace_abn can be installed:

Another Error:

1
2
3
4


Exception has occurred: ImportError
cannot import name 'get_num_classes' from 'torchmetrics.utilities.data' (/home/zichen/anaconda3/envs/mvsnerf/lib/python3.8/site-packages/torchmetrics/utilities/data.py)
  File "/home/zichen/Downloads/mvsnerf-comments/train_mvs_nerf_pl.py", line 17, in <module>
    from pytorch_lightning.callbacks import ModelCheckpoint

Change to specific versions, as summary:

1
2


pip install torchmetrics==0.5.0
pip install setuptools==59.5.0

Numpy version incompatible issue#77

1

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

File “/home/zichen/Downloads/mvsnerf-comments/data/dtu.py”, line 98, in build_proj_mats self.proj_mats, self.intrinsics = np.stack(proj_mats), np.stack(intrinsics)