Torch Points3D：用于在点云上进行深度学习的Pytorch框架

Torch Points3D：用于在点云上进行深度学习的Pytorch框架

This is a framework for running common deep learning models for point cloud analysis tasks against classic benchmark. It heavily relies on Pytorch Geometric and Facebook Hydra.

The framework allows lean and yet complex model to be built with minimum effort and great reproducibility. It also provide a high level API to democratize deep learning on pointclouds.

Requirements

CUDA 10 or higher (if you want GPU version)Python 3.6 or higher + headers (python-dev)PyTorch 1.5 or higher (1.4 and 1.3.1 should also be working but are not actively supported moving forward)MinkowskiEngine (optional) see here for installation instructions

Install with

pip install torchpip install torch-points3d

Project structure

├─ benchmark # Output from various benchmark runs├─ conf # All configurations for training nad evaluation leave there├─ notebooks # A collection of notebooks that allow result exploration and network debugging├─ docker # Docker image that can be used for inference or training├─ docs # All the doc├─ eval.py # Eval script├─ find_neighbour_dist.py # Script to find optimal #neighbours within neighbour search operations├─ forward_scripts # Script that runs a forward pass on possibly non annotated data├─ outputs # All outputs from your runs sorted by date├─ scripts # Some scripts to help manage the project├─ torch_points3d ├─ core # Core components ├─ datasets # All code related to datasets ├─ metrics # All metrics and trackers ├─ models # All models ├─ modules # Basic modules that can be used in a modular way ├─ utils # Various utils └─ visualization # Visualization├─ test└─ train.py # Main script to launch a training

As a general philosophy we have split datasets and models by task. For example, datasets has three subfolders:

segmentationclassificationregistrationobject_detection

where each folder contains the dataset related to each task.

Methods currently implemented

PointNet from Charles R. Qi et al.: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (CVPR 2017)PointNet++ from Charles from Charles R. Qi et al.: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpaceRSConv from Yongcheng Liu et al.: Relation-Shape Convolutional Neural Network for Point Cloud Analysis (CVPR 2019)RandLA-Net from Qingyong Hu et al.: RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point CloudsPointCNN from Yangyan Li et al.: PointCNN: Convolution On X-Transformed Points (NIPS 2018)KPConv from Hugues Thomas et al.: KPConv: Flexible and Deformable Convolution for Point Clouds (ICCV 2019)MinkowskiEngine from Christopher Choy et al.: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks (CVPR19)VoteNet from Charles R. Qi et al.: Deep Hough Voting for 3D Object Detection in Point Clouds (ICCV 19)

FCGF from Christopher Choy et al.: Fully Convolutional Geometric Features (ICCV'19)

Available datasets

Segmentation

Scannet from Angela Dai et al.: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes S3DIS from Iro Armeni et al.: Joint 2D-3D-Semantic Data for Indoor Scene Understanding

* S3DIS 1x1* S3DIS Room* S3DIS Fused

Shapenet from Angel X. Chang et al.: ShapeNet: An Information-Rich 3D Model Repository

Object detection and panoptic

Scannet from Angela Dai et al.: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes

Registration

3DMatch from Andy Zeng et al.: 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Classification

ModelNet from Zhirong Wu et al.: 3D ShapeNets: A Deep Representation for Volumetric Shapes

API

KPConv

from torch_points3d.applications.kpconv import KPConvmodel = KPConv( architecture="unet", input_nc=3, output_nc=5, in_feat=32, num_layers=4,)

PointNet2

from torch_points3d.applications.pointnet2 import PointNet2model = PointNet2( architecture="unet", input_nc=3, output_nc=5, num_layers=3, multiscale=True, config=None,)

RSConv

from torch_points3d.applications.rsconv import RSConvmodel = RSConv( architecture="unet", input_nc=3, output_nc=5, num_layers=3, multiscale=True, config=None,)

VoteNet

from torch_points3d.applications.votenet import VoteNetmodel = VoteNet( original=False, # Wether to create official VoteNet or use custom backbone. backbone="kpconv", input_nc=3, num_classes=20, mean_size_arr=[], # (num_classes, 3) If available, prior mean box sizes for each class compute_loss=True, in_feat=64, # Used for the bakcbone)

Developer guidelines

Setup repo

We use Poetry for managing our packages. In order to get started, clone this repositories and run the following command from the root of the repo

poetry install --no-root

This will install all required dependencies in a new virtual environment.

Activate the environment

poetry shell

You can check that the install has been successful by running

python -m unittest -v

Minkowski Engine

The repository is supporting Minkowski Engine which requires openblas-dev and nvcc if you have a CUDA device on your machine. First install openblas

sudo apt install libopenblas-dev

then make sure that nvcc is in your path:

nvcc -V

If it's not then locate it (locate nvcc) and add its location to your PATH variable. On my machine:

export PATH="/usr/local/cuda-10.2/bin:$PATH"

You are now in a position to install MinkowskiEngine with GPU support:

poetry run pip install git+git://github.com/StanfordVL/MinkowskiEngine.git#7087411269605b8cb5b9f61246d474f507ca3781

Pycuda

pip install pycuda

Train pointnet++ on part segmentation task for dataset shapenet

poetry run python train.py task=segmentation model_type=pointnet2 model_name=pointnet2_charlesssg dataset=shapenet-fixed

And you should see something like that

The config for pointnet++ is a good example of how to define a model and is as follow:

# PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (https://arxiv.org/abs/1706.02413)# Credit Charles R. Qi: https://github.com/charlesq34/pointnet2/blob/master/models/pointnet2_part_seg_msg_one_hot.pypointnet2_onehot: architecture: pointnet2.PointNet2_D conv_type: 'DENSE' use_category: True down_conv: module_name: PointNetMSGDown npoint: [1024, 256, 64, 16] radii: [[0.05, 0.1], [0.1, 0.2], [0.2, 0.4], [0.4, 0.8]] nsamples: [[16, 32], [16, 32], [16, 32], [16, 32]] down_conv_nn: [ [[FEAT, 16, 16, 32], [FEAT, 32, 32, 64]], [[32 + 64, 64, 64, 128], [32 + 64, 64, 96, 128]], [[128 + 128, 128, 196, 256], [128 + 128, 128, 196, 256]], [[256 + 256, 256, 256, 512], [256 + 256, 256, 384, 512]], ] up_conv: module_name: DenseFPModule up_conv_nn: [ [512 + 512 + 256 + 256, 512, 512], [512 + 128 + 128, 512, 512], [512 + 64 + 32, 256, 256], [256 + FEAT, 128, 128], ] skip: True mlp_cls: nn: [128, 128] dropout: 0.5

Benchmark

S3DIS 1x1

Shapenet part segmentation

The data reported below correspond to the part segmentation problem for Shapenet for all categories. We report against mean instance IoU and mean class IoU (average of the mean instance IoU per class)

Explore your experiments

We provide a notebook based pyvista and panel that allows you to explore your past experiments visually. When using jupyter lab you will have to install an extension:

jupyter labextension install @pyviz/jupyterlab_pyviz

Run through the notebook and you should see a dashboard starting that looks like the following:

Inference

Inference script

We provide a script for running a given pre trained model on custom data that may not be annotated. You will find an example of this for the part segmentation task on Shapenet. Just like for the rest of the codebase most of the customization happens through config files and the provided example can be extended to other datasets. You can also easily create your own from there. Going back to the part segmentation task, say you have a folder full of point clouds that you know are Airplanes, and you have the checkpoint of a model trained on Airplanes and potentially other classes, simply edit the config.yaml and shapenet.yaml and run the following command:

python forward_scripts/forward.py

The result of the forward run will be placed in the specified output_folder and you can use the notebook provided to explore the results. Below is an example of the outcome of using a model trained on caps only to find the parts of airplanes and caps.

Containerize your model with Docker

Finally, for people interested in deploying their models to production environments, we provide a Dockerfile as well as a build script. Say you have trained a network for semantic segmentation that gave the weight , the following command will build a docker image for you:

cd docker./build.sh outputfolder/weights.pt

You can then use it to run a forward pass on a all the point clouds in input_path and generate the results in output_path

docker run -v /test_data:/in -v /test_data/out:/out pointnet2_charlesssg:latest python3 forward_scripts/forward.py dataset=shapenet data.forward_category=Cap input_path="/in" output_path="/out"

The -v option mounts a local directory to the container's file system. For example in the command line above, /test_data/out will be mounted at the location /out. As a consequence, all files written in /out will be available in the folder /test_data/out on your machine.

Profiling

We advice to use snakeviz and cProfile

Use cProfile to profile your code

poetry run python -m cProfile -o {your_name}.prof train.py ... debugging.profiling=True

And visualize results using snakeviz.

snakeviz {your_name}.prof

It is also possible to use torch.utils.bottleneck

python -m torch.utils.bottleneck /path/to/source/script.py [args]

Troubleshooting

Cannot compile certain CUDA Kernels or seg faults while running the tests

Ensure that at least PyTorch 1.4.0 is installed and verify that cuda/bin and cuda/include are in your $PATH and $CPATH respectively, e.g.:

$ python -c "import torch; print(torch.__version__)">>> 1.4.0$ python -c "import torch; print(torch.__version__)">>> 1.1.0$ echo $PATH>>> /usr/local/cuda/bin:...$ echo $CPATH>>> /usr/local/cuda/include:...

Undefined symbol / Updating Pytorch

When we update the version of Pytorch that is used, the compiled packages need to be reinstalled, otherwise you will run into an error that looks like this:

... scatter_cpu.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c1012CUDATensorIdEv

This can happen for the following libraries:

torch-points-kernelstorch-scattertorch-clustertorch-sparse

An easy way to fix this is to run the following command with the virtual env activated:

pip uninstall torch-scatter torch-sparse torch-cluster torch-points-kernels -yrm -rf ~/.cache/pippoetry install

Contributing

Contributions are welcome! The only asks are that you stick to the styling and that you add tests as you add more features!

For styling you can use pre-commit hooks to help you:

pre-commit install

A sequence of checks will be run for you and you may have to add the fixed files again to the stashed files.

When it comes to docstrings we use numpy style docstrings, for those who use Visual Studio Code, there is a great extension that can help with that. Install it and set the format to numpy and you should be good to go!

Finaly, if you want to have a direct chat with us feel free to join our slack, just shoot us an email and we'll add you.

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。