Memo: Vis - 3D | Visualize Camera Poses

camtools

Plot the 49 Camera Poses in DTU

References:

(2024-03-28)
1. yxlao/camtools - Github
Supports:

Actions:

Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


import numpy as np
import os
def read_cam_file(filename):
    with open(filename) as f:
        lines = [line.rstrip() for line in f.readlines()]
    # extrinsics: line [1,5), 4x4 matrix
    extrinsics = np.fromstring(' '.join(lines[1:5]), dtype=np.float32, sep=' ')
    extrinsics = extrinsics.reshape((4, 4))
    # intrinsics: line [7-10), 3x3 matrix
    intrinsics = np.fromstring(' '.join(lines[7:10]), dtype=np.float32, sep=' ')
    intrinsics = intrinsics.reshape((3, 3))
    # depth_min & depth_interval: line 11
    depth_min = float(lines[11].split()[0])
    return intrinsics, extrinsics, depth_min

import camtools as ct
import open3d as o3d

Ks, Ts= [], []
for i in range(49):
    intrinsics, extrinsics, _ = read_cam_file(os.path.join('/mnt/data2_z/MVSNet_testing/dtu','scan1', f'cams/{i:08d}_cam.txt'))
    Ks.append(intrinsics)
    Ts.append(extrinsics)

cameras = ct.camera.create_camera_frames(Ks, Ts)
o3d.visualization.draw_geometries([cameras])

Results:

Result	Demo in README

Don’t know how to add camera frames like his.

1
2
3
4
5
6


          0 1 2 3 4
         10 9 8 7 6 5
   11 12 13 14 15 16 17 18
  27 26 25 24 23 22 21 20 19
 28 29 30 31 32 33 34 35 36 37
48 47 46 45 44 43 42 41 40 39 38

CameraViewer

(2024-03-28)

xt4d/CameraViewer found by DDG when searching “how to visualize camera pose”

It uses plotly to visualize cameras internally.

(2024-03-28)

Test the poses in DTU.

The extrinsics (w2c) don’t appear on the canvas if keeping the translation vectors.

I set the translation (camera pose), i.e., 4th column in extrinsics to all 0, then the rotations are shown.

1
2
3
4
5


# /mnt/data2_z/MVSNet_testing/dtu/scan1/cams/{0:08d}_cam.txt
array([[ 0.970263  ,  0.00747983,  0.241939  ,  0.        ],
       [-0.0147429 ,  0.999493  ,  0.0282234 ,  0.        ],
       [-0.241605  , -0.030951  ,  0.969881  ,  0.        ]],
      dtype=float32)

The reason could be that the translation vector is too large: [-191.02, 3.28832, 22.5401].

By reducing it by 1/100 times (extrinsics[:,3] = extrinsics[:,3]/100): [-1.9102, 0.0328832, 0.225401], the camera appears.

The w2c (extrinsics) can be prepared as npy files:

1
2
3
4
5


for i in range(49):
    intrinsics, extrinsics, _ = read_cam_file(os.path.join('/mnt/data2_z/MVSNet_testing/dtu','scan1', f'cams/{i:08d}_cam.txt'))
    extrinsics = extrinsics[:3]
    extrinsics[:,3] = extrinsics[:,3]/100
    np.save(f'/mnt/data2_z/Poses_CamViewer/obj/poses/{i:03d}', extrinsics)

BTW, writing json file manually is a time black hole.

As long as the filenames of poses and images are the same, it’s okay. The indexing doesn’t matter.

1

~/Downloads/CameraViewer$ python app.py --root /mnt/data2_z/Poses_CamViewer/obj/ --type w2c --image_size 128

If omitting the argument --type, the program will use poses.json. Otherwise, the program will read directories: poses/ and images/.

49 cameras for scan1
- The above figure shows the original poses. And the principal axis is facing away from the object.

pytransform3d

(2024-03-29)

Plot mesh and cameras:

Visualizing camera trajectory in Open3D #148 (Found when searching “open3d visualize camera poses” DDG)

The mesh in the image is produced by Meshroom. And then use Figure.plot_camera()

pytransform3d.camera.plot_camera() Example

Code for plotting pose 1 of DTU

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


import numpy as np
import matplotlib.pyplot as plt
import pytransform3d.camera as pc
import pytransform3d.transformations as pt

w2c = np.array([[0.970263, 0.00747983, 0.241939, -191.02],
                [-0.0147429, 0.999493, 0.0282234, 3.28832],
                [-0.241605, -0.030951, 0.969881, 22.5401],
                [0.0, 0.0, 0.0, 1.0] ])
c2w = np.linalg.inv(w2c)

intrinsics = np.array([
       [ 2.89233051e+03, -2.48063349e-04,  8.23205273e+02],
       [ 0.00000000e+00,  2.88317528e+03,  6.19070918e+02],
       [ 0.00000000e+00,  0.00000000e+00,  1.00000000e+00]])
sensor_size = np.array([1600, 1200])  # image size
virtual_image_distance = 1

ax = pt.plot_transform(A2B=c2w, s=0.2)
ax.set_xlim(186, 188)
ax.set_ylim(3, 5)
ax.set_zlim(20, 22)

pc.plot_camera(
    ax, cam2world=c2w, M=intrinsics, sensor_size=sensor_size,
    virtual_image_distance=virtual_image_distance)
plt.show()

pytransform3d	matplotlib

In the matplotlib code, I have corrected the camera position to -extrinsics[:-1][:,:-1].T @ extrinsics[:,-1][:-1]. The 2 results are the same.

Plot basis and camera plane:

Camera Extrinsic Matrix with Example in Python - Part2

1
2
3
4
5
6
7


# plot the global basis and the transformed camera basis
ax = pr.plot_basis(ax)
ax = pr.plot_basis(ax, R, offset)

# plot the original and transformed image plane
ax.plot_surface(xx, yy, Z, alpha=0.75)
ax.plot_surface(xxt, yyt, Zt, alpha=0.75)

Matplotlib

(2024-03-30)

Code

{{{

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110


%matplotlib widget 
import os
import numpy as np

def read_cam_file(filename):
    with open(filename) as f:
        lines = [line.rstrip() for line in f.readlines()]
    # extrinsics: line [1,5), 4x4 matrix
    extrinsics = np.fromstring(' '.join(lines[1:5]), dtype=np.float32, sep=' ')
    extrinsics = extrinsics.reshape((4, 4))
    # intrinsics: line [7-10), 3x3 matrix
    intrinsics = np.fromstring(' '.join(lines[7:10]), dtype=np.float32, sep=' ')
    intrinsics = intrinsics.reshape((3, 3))
    # depth_min & depth_interval: line 11
    depth_min = float(lines[11].split()[0])
    return intrinsics, extrinsics, depth_min

poses_list = []
for i in range(49):
    _, extrinsics, _ = read_cam_file(os.path.join('/mnt/data2_z/MVSNet_testing/dtu','scan23', f'cams/{i:08d}_cam.txt'))
    poses_list.append({
        # "position": extrinsics[:,-1][:-1],    # Wrong
        "position": - extrinsics[:-1][:,:-1].T @ extrinsics[:,-1][:-1],
        "rotation": extrinsics[:-1][:,:-1],
        })

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d.art3d import Poly3DCollection, Line3DCollection

def plot_camera_poses(poses, axis_length=0.1):
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d') 

    # Plot each camera pose
    for pose in poses:
        # Extract camera position and orientation
        cam_position = pose['position']
        cam_orientation = pose['rotation']
        # Plot camera position
        ax.scatter(cam_position[0], cam_position[1], cam_position[2], c='r', marker='.')

        axes_endpoints = cam_position + axis_length * cam_orientation
        # Plot camera canvas determined by 4 corners (forming 2 triangles)
        corner_back_1 = axes_endpoints[2] + 0.3*axis_length * cam_orientation[1] + 0.3*axis_length * cam_orientation[0]
        corner_back_2 = axes_endpoints[2] - 0.3*axis_length * cam_orientation[1] + 0.3*axis_length * cam_orientation[0]
        corner_back_3 = axes_endpoints[2] + 0.3*axis_length * cam_orientation[1] - 0.3*axis_length * cam_orientation[0]
        corner_back_4 = axes_endpoints[2] - 0.3*axis_length * cam_orientation[1] - 0.3*axis_length * cam_orientation[0]
        verts = np.array([[ corner_back_1, corner_back_2, corner_back_4]])
        tri = Poly3DCollection(verts, alpha=0.3, facecolors='cyan',)
        ax.add_collection3d(tri)

        verts = np.array([[ corner_back_1, corner_back_3, corner_back_4]])
        tri = Poly3DCollection(verts, alpha=0.3, facecolors='cyan',)
        ax.add_collection3d(tri)

        # Hull of the camera 
        ax.plot3D([cam_position[0], corner_back_1[0]], 
                  [cam_position[1], corner_back_1[1]], 
                  [cam_position[2], corner_back_1[2]], 'gray')
        ax.plot3D([cam_position[0], corner_back_2[0]], 
                  [cam_position[1], corner_back_2[1]], 
                  [cam_position[2], corner_back_2[2]], 'gray')
        ax.plot3D([cam_position[0], corner_back_3[0]], 
                  [cam_position[1], corner_back_3[1]], 
                  [cam_position[2], corner_back_3[2]], 'gray')
        ax.plot3D([cam_position[0], corner_back_4[0]], 
                  [cam_position[1], corner_back_4[1]], 
                  [cam_position[2], corner_back_4[2]], 'gray')
        # Camera plane edges
        ax.plot3D([corner_back_1[0], corner_back_2[0]], 
                  [corner_back_1[1], corner_back_2[1]], 
                  [corner_back_1[2], corner_back_2[2]], 'gray')
        ax.plot3D([corner_back_3[0], corner_back_1[0]], 
                  [corner_back_3[1], corner_back_1[1]], 
                  [corner_back_3[2], corner_back_1[2]], 'gray')
        ax.plot3D([corner_back_3[0], corner_back_4[0]], 
                  [corner_back_3[1], corner_back_4[1]], 
                  [corner_back_3[2], corner_back_4[2]], 'gray')
        ax.plot3D([corner_back_2[0], corner_back_4[0]], 
                  [corner_back_2[1], corner_back_4[1]], 
                  [corner_back_2[2], corner_back_4[2]], 'gray')

    lines = []
    for idx in range(len(poses)-1):
        lines.append([poses[idx]['position'], poses[idx+1]['position']])
    # Create a line collection
    lc = Line3DCollection(lines, colors='b', linewidths=1, label='Camera Trajectory')
    ax.add_collection3d(lc)

    # Set plot limits and labels
    ax.set_box_aspect([1, 1, 1])
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel('Z')

    plt.show()
    return ax

# Plot camera poses
ax_prev = plot_camera_poses(poses_list, axis_length=60)

# Plot point cloud
import open3d as o3d
pcd = o3d.io.read_point_cloud("/mnt/data2_z/SampleSet/MVS Data/Points/stl/stl023_total.ply", format='ply')
vs = np.asarray(pcd.points)
samples = vs[np.random.choice(vs.shape[0],100)]
x = samples[:,0]
y = samples[:,1]
z = samples[:,2]
ax_prev.scatter(x, y, z, color='gray', alpha=0.4)

}}}

The Line3DCollection usage is seen from an chatGPT-generated anwser: How to visualize colmap export that Camera parameters -SO

open3d

camera hull

(2024-03-29)

create_camera_visualization()

Sample code: Is there a way to draw a camera in a visualizer? #3876
(Found when searching “open3d draw cameras” DDG)

camera moves

Iterate Multiple Camera Poses (Extrinsics)

References:

(2024-03-31)
1. Setting the extrinsic matrix for ViewControl #2121 - Open3D
  - Searched by “open3d camera extrinsic set_extrinsic” at DDG
Supports:

Actions:

Iterate multiple camera poses (extrinsics):

Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


import numpy as np
import os

def read_cam_file(filename):
    with open(filename) as f:
        lines = [line.rstrip() for line in f.readlines()]

    # extrinsics: line [1,5), 4x4 matrix
    extrinsics = np.fromstring(' '.join(lines[1:5]), dtype=np.float32, sep=' ')
    extrinsics = extrinsics.reshape((4, 4))

    # intrinsics: line [7-10), 3x3 matrix
    intrinsics = np.fromstring(' '.join(lines[7:10]), dtype=np.float32, sep=' ')
    intrinsics = intrinsics.reshape((3, 3))

    # depth_min & depth_interval: line 11
    depth_min = float(lines[11].split()[0])
    return intrinsics, extrinsics, depth_min

poses_list = []
for i in range(49):
    _, extrinsics, _ = read_cam_file(os.path.join('/mnt/data2_z/MVSNet_testing/dtu','scan23', f'cams/{i:08d}_cam.txt'))
    poses_list.append({
        "position": - extrinsics[:-1][:,:-1].T @ extrinsics[:,-1][:-1],
        "rotation": extrinsics[:-1][:,:-1],
        "extrinsics": extrinsics,
        })

import time
import itertools
import open3d as o3d

pcd = o3d.io.read_point_cloud("/home/yi/Downloads/DTU_SampleSet/MVS Data/Points/stl/stl001_total.ply", format='ply')
vis = o3d.visualization.VisualizerWithKeyCallback()
vis.create_window(window_name="Playback", visible=True)
vis.get_render_option().background_color = np.asarray([0, 0, 0])
vis.add_geometry(pcd)
ctr = vis.get_view_control()
cam = ctr.convert_to_pinhole_camera_parameters()

for view_idx, camParams in zip(itertools.count(), poses_list):
    cam.extrinsic = camParams['extrinsics']
    ctr.convert_from_pinhole_camera_parameters(cam, True)
    vis.poll_events()
    vis.update_renderer()
    time.sleep(0.1)
vis.destroy_window()

Note: The argument allow_arbitrary=True is required in convert_from_pinhole_camera_parameters(cam, True) (using 0.18.0),

Custom Animation

Customized visualization - Open3D Docs

Code from View Control No Effect in 0.17 #6098

1
2
3
4
5
6
7
8
9


import open3d as o3d

def rotate(vis):
    ctr = vis.get_view_control()
    ctr.rotate(5, 0)
    return False

frame = o3d.geometry.TriangleMesh.create_coordinate_frame()
o3d.visualization.draw_geometries_with_animation_callback([frame], rotate)

Others

demul/extrinsic2pyramid
OpenCV has example code. How to plot the camera and image positions from camera calibration data?
pytorch3d/docs/tutorials/utils/camera_visualization.
sxyu/nerfvis: NeRF visualization library under construction
Blender add-on: Photogrammetry-Importer How to visualize colmap export ‘images.txt’ in blender? -SO
kaolin.render.camera — Kaolin documentation - Read the Docs
WebGL Visualizing the Camera

OpenCV

(2024-04-01)

OpenCV: cv::viz::WCameraPosition Class Reference

Draw coordinates axes

How to draw 3D Coordinate Axes with OpenCV for face pose estimation? - SO

1
2


scale = 0.1 
img = cv2.drawFrameAxes(img, K, distortion, rotation_vec, translation_vec, scale)

cvtkit

nburgdorfer/cvtkit

(Found: nburgdorfer/confidence-based-fusion -> github.io -> cvtkit)

(2024-07-18)

Functions: Point cloud ➡ GIF, Video; Mesh ➡ GIF, Video; Depth map ➡ Visibility map. Tutorials

1

pip install cvt imageio-ffmpeg

ply to mp4

(2024-07-19)

The input camera poses can use the ones of DTU dataset.

1
2
3
4


python script_ply2gif_cvtkit.py \
  -p /home/yi/Downloads/CasMVSNet_pl-comments/results/dtu/points/scan001_l3.ply \
  -c /mnt/data2_z/MVSNet_testing/dtu/scan1/cams \
  -n 48  # Use the 49 poses

The default value for the argument --video_file is None. There will be an error if without setting it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


(AIkui) yi@yi-Alienware:~/Experiments/PointCloud/log$ python script_ply2gif_cvtkit.py -p /home/yi/Downloads/CasMVSNet_pl-comments/results/dtu/points/scan001_l3.ply -c /mnt/data2_z/MVSNet_testing/dtu/scan1/cams
[Open3D INFO] EGL headless mode enabled.
FEngine (64 bits) created at 0x55882c2a12c0 (threading is enabled)
EGL(1.5)
OpenGL(4.1)
Traceback (most recent call last):
  File "/home/yi/OneDrive/Exercises/Experiments/PointCloud/log/script_ply2gif_cvtkit.py", line 72, in <module>
    main()
  File "/home/yi/OneDrive/Exercises/Experiments/PointCloud/log/script_ply2gif_cvtkit.py", line 59, in main
    with imageio.get_writer(video_file, mode="I", fps=ARGS.fps) as writer:
  File "/home/yi/anaconda3/envs/AIkui/lib/python3.10/site-packages/imageio/v2.py", line 163, in get_writer
    image_file = imopen(uri, "w" + mode, **imopen_args)
  File "/home/yi/anaconda3/envs/AIkui/lib/python3.10/site-packages/imageio/core/imopen.py", line 240, in imopen
    raise err_type(err_msg)
ValueError: ImageIO does not generally support reading folders. Limited support may be available via specific plugins. Specify the plugin explicitly using the `plugin` kwarg, e.g. `plugin='DICOM'`

Similar problem discussed in: One-Shot_Free-View_Neural_Talking_Head_Synthesis - Issue#41

Table of contents

camtools

CameraViewer

pytransform3d

Matplotlib

open3d

camera hull

camera moves

Custom Animation

Others

OpenCV

cvtkit

ply to mp4