memo: CMake | Include 3rd-Party Library

Example

Source video: 【cmake教程】为你的项目引入外部第三方库(以真实项目partio为例) - 只喝白开水

Prepare: Download the code of partio.

1

git clone https://github.com/wdas/partio.git --depth=1

  • Directory structure:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

    (base) yi@yi:~/Downloads/partio$ tree -L 2
.
├── build_wheels.sh
├── CMakeLists.txt
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── setup.cfg
├── setup.py
└── src
    ├── data
    ├── doc
    ├── lib
    ├── Makefile
    ├── py
    ├── tests
    └── tools

7 directories, 9 files

Four steps for testing the introduced library:

  1. Copy codes

    Only the codes partio/src/lib/ are needed in this example., so copy it to your own project, and place it under the directory example_cmake/external/partio/ as shown below:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    (base) yi@yi:~/Downloads/example_cmake$ tree
.
├── CMakeLists.txt
├── external
│   └── partio
│       ├── CMakeLists.txt   # partio/src/lib/CMakeLists.txt
│       ├── core
│       ├── io
│       ├── PartioAttribute.h
│       ├── Partio.h
│       ├── PartioIterator.h
│       ├── PartioVec3.h
│       └── test
│           └── test_helloWorld.cpp
└── src
    ├── CMakeLists.txt
    └── main.cpp

  2. Inherit the original CMakeListst settings:

    Copy all contents of the top-level CMakeLists.txt of the original partio project to the front of the copied lib’s external/partio/CMakeLists.txt, i.e., the previous partio/src/lib/CMakeLists.txt, as it’s included first.

    So as to retain some macro settings and variables for compiling. And then delete unnecessary settings.

  3. The top-level CMakeLists.txt of your own project is:

    1
2
3
4
5
6
7
8
9

    cmake_minimum_required(VERSION 3.20)
project(learn-cmake LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# each sub-directory has its own CMakeLists.txt:
add_subdirectory(external/partio)
add_subdirectory(src)

    The src/ folder stores your own code, which will be built to an executable program. So, the executable in the src/CMakeLists.txt is your own main.cpp:

    1

    add_executable(main main.cpp)

    An example of src/main.cpp:

    1
2
3

    int main(int argc, char const *argv[]){
    return 0;
}

  4. Write an test executable program (test_helloWorld.cpp) inside external/partio.

    In this case, test_helloWorld.cpp calls functions in external/partio “natively”. However, this isn’t the case for calling its functions from another project.

    1
2
3
4

    #include "Partio.h"
int main(){
  Partio::ParticlesDataMutable* data = Partio::read("bgeoFile");
  std::cout << "Number of " << std::endl;}

    Set the external/partio/CMakeLists.txt to compile it as an executable application:

    1
2
3
4

    add_library(partio ${PARTIO_LIBRARY_TYPE} ${io_cpp} ${core_cpp})
# Setting target_include_directories ....
add_executable(test_helloWorld ${CMAKE_CURRENT_LIST_DIR}/test/test_helloWorld.cpp)
target_link_libraries(test_helloWorld PRIVATE partio)

    After compiling, a binary file test_helloWorld will be generated under “external/partio”.

    This test code has nothing to do with the other subdirectories’ “src/”, where the main application will be build based on its own CMakeLists.txt.

    The CMakeLists.txt of different projects are independent to each other.

    The most commonly used rule is “One CMakeLists.txt per target”. cmake: add_subdirectory() vs include() - SO

Test DiffRast

I want to debug the library diff-gaussian-rasterization.

  1. I can create a project, and make the diffRast as an external library to call its methods.

    1
2
3
4

    mkdir debug_diff_rast && cd debug_diff_rast
git init
mkdir external && cd external
git submodule add https://github.com/graphdeco-inria/diff-gaussian-rasterization

    Directory structure:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

    (base) yi@yi:~/Downloads/debug_diff_rast$ tree
.
├── CMakeLists.txt
├── main.cpp
├── README.md
└── external
    └── diff-gaussian-rasterization
        ├── CMakeLists.txt
        ├── cuda_rasterizer
        │   ├── auxiliary.h
        │   ├── backward.cu
        │   ├── backward.h
        │   ├── config.h
        │   ├── forward.cu
        │   ├── forward.h
        │   ├── rasterizer.h
        │   ├── rasterizer_impl.cu
        │   └── rasterizer_impl.h
        ├── diff_gaussian_rasterization
        │   └── __init__.py
        ├── ext.cpp
        ├── LICENSE.md
        ├── rasterize_points.cu
        ├── rasterize_points.h
        ├── README.md
        ├── setup.py
        └── third_party
            ├── glm
            └── stbi_image_write.h

  2. The original diff-gaussian-rasterization/CMakeLists.txt would build the project to a library (CudaRasterizer), not an executable application.

    The command add_subdirectory() is used to add an external project, which will automatically build according to its own CMakeLists.txt when the root CMakeLists.txt starts building.

    • If without add_subdirectory, an error would occur: CMake Error at CMakeLists.txt:17 (target_include_directories): Cannot specify include directories for target which is not built by this project.

    Hence, the library CudaRasterizer is linked as a static library through target_link_libraries() to an executable program MyApp based on the main.cpp, and debug it.

    In summary, the top-level (root) CMakeLists.txt needs to link 2 libraries: libtorch and CudaRasterizer, as follows.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    cmake_minimum_required(VERSION 3.20 FATAL_ERROR)
project(MyApp)    # ${PROJECT_NAME}

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_subdirectory(external/diff-gaussian-rasterization)

add_executable(MyApp main.cpp)
set_property(TARGET MyApp PROPERTY CXX_STANDARD 17)

target_link_libraries(MyApp "${TORCH_LIBRARIES}")
target_link_libraries(MyApp CudaRasterizer)

# rasterization_points.cu and ~.h aren't in their CMakeLists.txt, so need:
target_sources(MyApp PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/external/diff-gaussian-rasterization/rasterize_points.cu)
target_include_directories(MyApp PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/external/diff-gaussian-rasterization)

    In the diff-gaussian-rasterization/CMakeLists.txt, the header files inside the folder cuda_rasterizer/ are included and set to PUBLIC, such that when the library is linked to the target, those headers will be appended to the target’s INCLUDE_DIRECTORIES. So they can be included via their filenames, for example #include "forward.h", without relative path. (Not sure. 2023-11-16)

    However, the rasterize_point.h isn’t included into the library (CudaRasterizer), so it has to be included using its relative path.

    Specifically, in the main.cpp:

    1
2

    #include <torch/torch.h>
#include "external/diff-gaussian-rasterization/rasterize_points.h"

    With adding that header file (declaration) to executable target, the compilation of main.cpp can pass. However, if the function body rasterize_points.cu (function implementation) isn’t added to the executable in CMakeLists.txt, there will be a linking error:

    1
2

    /usr/bin/ld: CMakeFiles/MyApp.dir/main.cpp.o: in function `main':
main.cpp:(.text.startup+0x383): undefined reference to `RasterizeGaussiansCUDA(at::Tensor const&, ...)'

  3. Two modifications for source code of diff-gaussian-rasterization:

    1. rasterize_point.cu isn’t included into the CudaRasterizer library, because it’s not specified within the add_library() function in the diff-gaussian-rasterization/CMakeLists.txt.

      Therefore, when the project diff-gaussian-rasterization is built by executing cmake -B ./build, the .cu file rasterize_point.cu isn’t compiled.

      Thus, the following error of missing Python.h won’t be tiggered.

    2. rasterize_point.cu includes header <torch/extension.h>, which is used for interacting with Python. Based on that, the library CudaRasterizer can be called in Python scripts as a PyTorch extension.

      Therefore, the Python development headers must be present to compile it. Otherwise, an error would occur:

      1
2
3
4
5
6
7
8
9

      In file included from /usr/local/libtorch/include/torch/csrc/Device.h:3,
                 from /usr/local/libtorch/include/torch/csrc/api/include/torch/python.h:8,
                 from /usr/local/libtorch/include/torch/extension.h:6,
                 from /home/yi/Downloads/diff-gaussian-rasterization/rasterize_points.h:13,
                 from /home/yi/Downloads//diff-gaussian-rasterization/test_DiffRast.cpp:2:
/usr/local/libtorch/include/torch/csrc/python_headers.h:10:10: fatal error: Python.h: No such file or directory
   10 | #include <Python.h>
      |          ^~-~~-~~-~~-~~
compilation terminated.

      Previously, my trivial experiments only used LibTorch tensors without interacting with python. Therefore, no error was reported when building without specifying the path of Python.

      But the Python development headers is already installed:

      1
2

      sudo apt-get install python3-dev
apt list python3-dev

      A possible solution is to include python in CMakeLists.txt: Similar issure

    3. Since here I just want to debug it as a C++ project, instead of for python, and I have LibTorch installed.

      So I changed #include <torch/extension.h> to #include <torch/torch.h> in the 2 files: “rasterize_points.cu” and “rasterize_points.h”.

  4. Build the top-level project “debug_diff_rast”:

    1
2
3
4
5
6

    cd ~/Downloads/debug_diff_rast
# configure:
cmake -B ./build -DCMAKE_PREFIX_PATH=/usr/local/libtorch -GNinja
# build: 
cmake --build ./build
./build/MyApp

Ref:

【公开课】现代CMake高级教程（持续更新中）- 双笙子佯谬 - bilibili

Include Python

cmake

(2024-01-29)

The direct solution for Python.h: No such file or directory could be include Python in the CMakeLists.txt,

g++ CLI

(2024-01-31)

  • Check if the Python Development Libraries has been installed: dpkg -l | grep python3-dev. devicetests

  1. Install it: sudo apt-get install python3-dev

  2. Find where it is: find / -type f -iname 'python.h' 2>/dev/null Debian / Ubuntu: Fatal error: Python.h: No such file or Directory

  3. The directory of Python is also required to be included: g++ -I/usr/include/python3.8 main.cpp. GeeksforGeeks

    A complete example command is:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    (AIkui) yi@yi-Alien:~/Downloads/CppCudaExt_PT_Tut_AIkui$ g++ \
-g -std=c++17 \
-I/usr/local/libtorch/include \
-I/usr/local/libtorch/include/torch/csrc/api/include \
-I/usr/local/libtorch/include/torch \
-I/home/yi/anaconda3/envs/AIkui/include/python3.10 \
-I./include  \        # Custom functions declarations
-L/usr/local/libtorch/lib \
-Wl,-rpath,/usr/local/libtorch/lib \
test_None_tensor.cpp \    # main function definition
interpolation.cpp \   # Custom functions definitions
-ltorch -ltorch_cpu -lc10
  • A demo

    Refer to Nathan Sebhastian

    1
2
3
4
5
6
7
8

    #include<Python.h>
#include<stdio.h>

int main()
{
    printf("Hello World!\n");
    return 0;
}

    Compile: g++ test_python.cpp, an error occurs:

    1
2
3
4
5

    (base) yi@yi-Alienware-Aurora-R8:~/Downloads/Cpp_Study$ g++ test_python.cpp
test_python.cpp:1:9: fatal error: Python.h: No such file or directory
    1 | #include<Python.h>
      |         ^~~~~~~~~~
compilation terminated.

    Locate header file:

    1
2
3

    sudo apt install mlocate
sudo updatedb
locate Python.h

    Include Python dir: gcc test_python.cpp -I/usr/include/python3.8