Featured image of post read: Render - NVS | MVSNeRF
read

read: Render - NVS | MVSNeRF

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo (ICCV 2021)

Code | Arxiv (2103) | ProjPage

(2023-08-16) Re-read

Experiments

Train model only on (MVSNet) DTU dataset, where the objects are partitioned the same as PixelNeRF (imgsize: 300x400).

Datasets (Download):

Stage Data Contents Resolution N_views Size
Train DTU 88 scenes 512x640 19G
Test DTU 16 scenes 3/20
Test LLFF
Test Blender
  • Feature map: 32 channels
  • Cost volume: 128 planes
  • MLP: 6 layers
  • Coarse-to-fine: One field and fine-tune
  • Ray pts: 128
  • Device: 2080Ti
  • Batch size: 1024 rays
  • Optimizer: Adam, lr=5e-4

Old Notes

Old notes on 2022-12-06

Abstract

  • 3 nearby input views
  • plane-swept cost volumes
  • geometry-aware scene reasoning
  • generalize across scenes

1 Introduction

  • Topic: Novel view synthesis

  • Recent progress: neural rendering

  • Former solutino and drawbacks:

    1. NeRF and its following works can produce photo-realistic novel view synthesis results. But they need a long-time per-scene optimization

  • Own Solution:

    • Goal: use … to …
    • 1 sentences introduce the name and functionality, properties.
    • Analysis:
      1. generalizability -> avoid tedious per-scene optimization and regress directly novel viewpoints
      2. quantitative outcomes

  • Tech stack

    1. MVSNet -> generalizable net of 3D reconstruction
      • Cost volume is built by warping 2D img features of src views onto sweeping planes
    2. Regress geometry and appearance from a cost volume (per-voxel neural features)
      • 3D cnn aggregates the cost volumes to a neural scene encoding volume

(2023-08-16)

Play

Environment

  1. inplace-abn needs to be installed from source, as issue#36. But I cannot install it:

    1
2
3

    RuntimeError: 
The detected CUDA version (10.2) mismatches the version that was used to compile
PyTorch (11.3). Please make sure to use the same CUDA versions.

    My cuda version is 11.3 as shown by:

    1
2

    import torch
print(torch.version.cuda)

  2. Then I specified the version the same as the pl-tutorial of AIkui:

    1
2
3
4
5
6

    - pip
- pip:
    - torch==1.11.0
    - torchvision==0.12.0
    - pytorch_lightning==1.6.0
    - inplace_abn

    With that, inplace_abn has installed.

    But the API has change since v1.5 (as issue) and resulted in: Exception has occurred: TypeError __init__() got an unexpected keyword argument 'distributed_backend'

  3. Try this setting:

    1
2
3
4

       - torch==1.10.1
   - torchvision==0.11.2
   - pytorch_lightning==1.3.5
   - inplace_abn

    Cannot import inplace_abn:

    1
2

    Exception has occurred: ImportError
/home/zichen/anaconda3/envs/mvsnerf/lib/python3.8/site-packages/inplace_abn/_backend.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPvmm

  4. I checked my cuda version, which is 10.2:

    1
2
3
4
5

    (mvsnerf) zichen@lambda-server:~/Downloads/mvsnerf-comments$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

    According to Previous PyTorch Versions | PyTorch, I install the versions compiled with cuda10.2. Then print(torch.version.cuda) returned 10.2 instead.

    1
2
3
4
5
6
7
8

    - pip
- pip:
   - torch==1.10.1+cu102 
   - torchvision==0.11.2+cu102 
   - torchaudio==0.10.1 
   - -f https://download.pytorch.org/whl/cu102/torch_stable.html
   - pytorch_lightning==1.3.5
   - inplace_abn

    Still cannot import inplace_abn with the same ImportError as above:

    Then Install from source, and inplace_abn can be installed:

  5. Another Error:

    1
2
3
4

    Exception has occurred: ImportError
cannot import name 'get_num_classes' from 'torchmetrics.utilities.data' (/home/zichen/anaconda3/envs/mvsnerf/lib/python3.8/site-packages/torchmetrics/utilities/data.py)
  File "/home/zichen/Downloads/mvsnerf-comments/train_mvs_nerf_pl.py", line 17, in <module>
    from pytorch_lightning.callbacks import ModelCheckpoint

    Change to specific versions, as summary:

    1
2

    pip install torchmetrics==0.5.0
pip install setuptools==59.5.0

  6. Numpy version incompatible issue#77

    1

    ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

File “/home/zichen/Downloads/mvsnerf-comments/data/dtu.py”, line 98, in build_proj_mats self.proj_mats, self.intrinsics = np.stack(proj_mats), np.stack(intrinsics)