Inferencing UniDet3D on ARKitScenes

[aayush-shah14]

Hi,

Thank you for this wonderful work! I am trying to run the model solely on ARKitScenes dataset and I had some questions.

1. I was successfully able to run data_prepare_offline.py and create the offline_prepared_data folder. However, while running this command:

python tools/create_data.py arkitscenes --root-path ./data/arkitscenes --out-dir ./data/arkitscenes --extra-tag arkitscenes-offline

I get this issue:

Traceback (most recent call last):
  File "/app/unidet3d/tools/create_data.py", line 57, in <module>
    raise NotImplementedError(f'Don\'t support {args.dataset} dataset.')
NotImplementedError: Don't support arkitscenes dataset.

I wanted to ask if I am supposed to be using some other script to process ARKitScenes since this script does not support it?

As a result of this, the .pkl files for arkitscenes are not created, resulting in an error while running test.py

2. The weights file mentioned in the read me is unidet3d_1xb8_scannet_s3dis_multiscan_3rscan_scannetpp_arkitscenes/epoch_1024.pth, however the checkpoint link given downloads unidet3d.pth. Are these the same files?

Thank you again for open sourcing the model, I look forward to implementing it in my research.

[filaPro]

1. Yes, looks like something is missing in our preprocessing code. Did you try to download our preprocessed data from huggingface?
2. Yes, should be the same.

[aayush-shah14]

Thank you for the reply! I was able to progress with the preprocessed data from huggingface, however I am facing some CUDA errors.

I got a warning saying there was a mismatch between model and the loaded state dict.

I am using the Docker provided by the repo and my CUDA version is 12.2

WARNING - The prefix is not set in metric class IndoorMetric_.
Loads checkpoint by local backend from path: work_dirs/tmp/unidet3d.pth
The model and loaded state dict do not match exactly

unexpected key in source state_dict: decoder.self_attn_layers.0.pe.1.weight, decoder.self_attn_layers.0.pe.1.bias, decoder.self_attn_layers.1.pe.1.weight, decoder.self_attn_layers.1.pe.1.bias, decoder.self_attn_layers.2.pe.1.weight, decoder.self_attn_layers.2.pe.1.bias, decoder.self_attn_layers.3.pe.1.weight, decoder.self_attn_layers.3.pe.1.bias, decoder.self_attn_layers.4.pe.1.weight, decoder.self_attn_layers.4.pe.1.bias, decoder.self_attn_layers.5.pe.1.weight, decoder.self_attn_layers.5.pe.1.bias

The script test.py exited with the following error:

Traceback (most recent call last):
  File "/app/unidet3d/tools/test.py", line 150, in <module>
    main()
  File "/app/unidet3d/tools/test.py", line 146, in main
    runner.test()
  File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1823, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/loops.py", line 435, in run
    self.run_iter(idx, data_batch)
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/loops.py", line 454, in run_iter
    outputs = self.runner.model.test_step(data_batch)
  File "/opt/conda/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
    return self._run_forward(data, mode='predict')  # type: ignore
  File "/opt/conda/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 346, in _run_forward
    results = self(**data, mode=mode)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/mmdet3d/models/detectors/base.py", line 86, in forward
    return self.predict(inputs, data_samples, **kwargs)
  File "/app/unidet3d/unidet3d/unidet3d.py", line 464, in predict
    results_list = self.predict_by_feat(x, sp_pts_masks_src,
  File "/app/unidet3d/unidet3d/unidet3d.py", line 522, in predict_by_feat
    self._single_scene_multiclass_nms(pred_bboxes,
  File "/app/unidet3d/unidet3d/unidet3d.py", line 626, in _single_scene_multiclass_nms
    nms_ids = nms3d(class_bboxes, class_scores, iou_thr)
  File "/workspace/mmcv/mmcv/ops/iou3d.py", line 87, in nms3d
    ext_module.iou3d_nms3d_forward(
RuntimeError: iou3d_nms3d_forward_impl: implementation for device cuda:0 not found.

Could you guide me regarding the reason for this error, and if I am implementing something incorrectly? Thank you!

[filaPro]

This means mmcv is not properly built for your GPU. Can you check if TORCH_CUDA_ARCH_LIST here needs to be updated for your GPU?

[aayush-shah14]

Thank you for the advice, I was able to run the script by changing the TORCH_CUDA_ARCH_LIST to include only 8.6 (for A6000 GPU). Changing this allowed the script to run, however the previous warning of size mismatches persisted. I also got poor mAP values on the dataset, as mentioned in #25. However, I am using the Dockerfile provided by the repo (with modifications to the cuda arch list, addition of ENV MMCV_WITH_OPS=1 and RUN apt-get update && apt-get install -y python3-dev). Am I missing something in the implementation? Your guidance has been very helpful, thank you!

Here is the log after running the command for scannet dataset: python tools/test.py configs/unidet3d_1xb8_scannet.py work_dirs/tmp/unidet3d.pth

NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
07/02 19:03:09 - mmengine - WARNING - The prefix is not set in metric class IndoorMetric_.
Loads checkpoint by local backend from path: work_dirs/tmp/unidet3d.pth
The model and loaded state dict do not match exactly

size mismatch for decoder.outs_cls.2.weight: copying a param with shape torch.Size([100, 256]) from checkpoint, the shape in current model is torch.Size([19, 256]).
size mismatch for decoder.outs_cls.2.bias: copying a param with shape torch.Size([100]) from checkpoint, the shape in current model is torch.Size([19]).
unexpected key in source state_dict: decoder.self_attn_layers.0.pe.1.weight, decoder.self_attn_layers.0.pe.1.bias, decoder.self_attn_layers.1.pe.1.weight, decoder.self_attn_layers.1.pe.1.bias, decoder.self_attn_layers.2.pe.1.weight, decoder.self_attn_layers.2.pe.1.bias, decoder.self_attn_layers.3.pe.1.weight, decoder.self_attn_layers.3.pe.1.bias, decoder.self_attn_layers.4.pe.1.weight, decoder.self_attn_layers.4.pe.1.bias, decoder.self_attn_layers.5.pe.1.weight, decoder.self_attn_layers.5.pe.1.bias

07/02 19:03:11 - mmengine - INFO - Load checkpoint from work_dirs/tmp/unidet3d.pth
/opt/conda/lib/python3.10/site-packages/mmdet3d/structures/points/base_points.py:136: UserWarning: point got color value beyond [0, 255]
  warnings.warn('point got color value beyond [0, 255]')
07/02 19:03:22 - mmengine - INFO - Epoch(test) [ 50/312]    eta: 0:00:54  time: 0.2069  data_time: 0.0405  memory: 4298  
07/02 19:03:30 - mmengine - INFO - Epoch(test) [100/312]    eta: 0:00:39  time: 0.1686  data_time: 0.0358  memory: 7075  
07/02 19:03:38 - mmengine - INFO - Epoch(test) [150/312]    eta: 0:00:29  time: 0.1616  data_time: 0.0334  memory: 6833  
07/02 19:03:47 - mmengine - INFO - Epoch(test) [200/312]    eta: 0:00:20  time: 0.1879  data_time: 0.0398  memory: 5745  
07/02 19:03:57 - mmengine - INFO - Epoch(test) [250/312]    eta: 0:00:11  time: 0.1831  data_time: 0.0373  memory: 7897  
07/02 19:04:05 - mmengine - INFO - Epoch(test) [300/312]    eta: 0:00:02  time: 0.1725  data_time: 0.0354  memory: 6366  
07/02 19:04:10 - mmengine - INFO - 
+----------------+---------+---------+---------+---------+
| classes        | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 |
+----------------+---------+---------+---------+---------+
| bookshelf      | 0.0000  | 0.0519  | 0.0000  | 0.0130  |
| picture        | 0.0006  | 0.1622  | 0.0001  | 0.1036  |
| desk           | 0.0019  | 0.2283  | 0.0004  | 0.1181  |
| refrigerator   | 0.0025  | 0.1930  | 0.0013  | 0.1404  |
| showercurtrain | 0.0007  | 0.6429  | 0.0006  | 0.5714  |
| toilet         | 0.0036  | 0.7931  | 0.0027  | 0.6897  |
| chair          | 0.0000  | 0.0000  | 0.0000  | 0.0000  |
| sofa           | 0.0000  | 0.0103  | 0.0000  | 0.0000  |
| table          | 0.0003  | 0.0229  | 0.0002  | 0.0171  |
| otherfurniture | 0.0006  | 0.0302  | 0.0003  | 0.0226  |
| curtain        | 0.0000  | 0.0000  | 0.0000  | 0.0000  |
| sink           | 0.0000  | 0.0000  | 0.0000  | 0.0000  |
| door           | 0.0000  | 0.0021  | 0.0000  | 0.0000  |
| counter        | 0.0000  | 0.0000  | 0.0000  | 0.0000  |
| bathtub        | 0.0128  | 0.3548  | 0.0067  | 0.2581  |
| cabinet        | 0.0002  | 0.0081  | 0.0001  | 0.0054  |
| bed            | 0.0000  | 0.0000  | 0.0000  | 0.0000  |
| window         | 0.0000  | 0.0000  | 0.0000  | 0.0000  |
+----------------+---------+---------+---------+---------+
| Overall        | 0.0013  | 0.1389  | 0.0007  | 0.1077  |
+----------------+---------+---------+---------+---------+
07/02 19:04:10 - mmengine - INFO - Epoch(test) [312/312]    scannet: {'bookshelf_AP_0.25': 4.1685088945087045e-05, 'picture_AP_0.25': 0.0006383808213286102, 'desk_AP_0.25': 0.0019036437151953578, 'refrigerator_AP_0.25': 0.002545332070440054, 'showercurtrain_AP_0.25': 0.0007062208605930209, 'toilet_AP_0.25': 0.003634118940681219, 'chair_AP_0.25': 0.0, 'sofa_AP_0.25': 4.09098356612958e-05, 'table_AP_0.25': 0.00025577080668881536, 'otherfurniture_AP_0.25': 0.0006191321881487966, 'curtain_AP_0.25': 0.0, 'sink_AP_0.25': 0.0, 'door_AP_0.25': 1.9993722162325867e-06, 'counter_AP_0.25': 0.0, 'bathtub_AP_0.25': 0.012755639851093292, 'cabinet_AP_0.25': 0.0001524976105429232, 'bed_AP_0.25': 0.0, 'window_AP_0.25': 0.0, 'mAP_0.25': 0.0012941850645297058, 'bookshelf_rec_0.25': 0.05194805194805195, 'picture_rec_0.25': 0.16216216216216217, 'desk_rec_0.25': 0.2283464566929134, 'refrigerator_rec_0.25': 0.19298245614035087, 'showercurtrain_rec_0.25': 0.6428571428571429, 'toilet_rec_0.25': 0.7931034482758621, 'chair_rec_0.25': 0.0, 'sofa_rec_0.25': 0.010309278350515464, 'table_rec_0.25': 0.022857142857142857, 'otherfurniture_rec_0.25': 0.03018867924528302, 'curtain_rec_0.25': 0.0, 'sink_rec_0.25': 0.0, 'door_rec_0.25': 0.0021413276231263384, 'counter_rec_0.25': 0.0, 'bathtub_rec_0.25': 0.3548387096774194, 'cabinet_rec_0.25': 0.008064516129032258, 'bed_rec_0.25': 0.0, 'window_rec_0.25': 0.0, 'mAR_0.25': 0.13887774288661126, 'bookshelf_AP_0.50': 2.4526937068003463e-06, 'picture_AP_0.50': 0.00014436915807891637, 'desk_AP_0.50': 0.0003835152310784906, 'refrigerator_AP_0.50': 0.0013462913921102881, 'showercurtrain_AP_0.50': 0.0005580016295425594, 'toilet_AP_0.50': 0.002663275459781289, 'chair_AP_0.50': 0.0, 'sofa_AP_0.50': 0.0, 'table_AP_0.50': 0.00017386814579367638, 'otherfurniture_AP_0.50': 0.00032002132502384484, 'curtain_AP_0.50': 0.0, 'sink_AP_0.50': 0.0, 'door_AP_0.50': 0.0, 'counter_AP_0.50': 0.0, 'bathtub_AP_0.50': 0.006746784783899784, 'cabinet_AP_0.50': 5.571341171162203e-05, 'bed_AP_0.50': 0.0, 'window_AP_0.50': 0.0, 'mAP_0.50': 0.0006885718461515151, 'bookshelf_rec_0.50': 0.012987012987012988, 'picture_rec_0.50': 0.1036036036036036, 'desk_rec_0.50': 0.11811023622047244, 'refrigerator_rec_0.50': 0.14035087719298245, 'showercurtrain_rec_0.50': 0.5714285714285714, 'toilet_rec_0.50': 0.6896551724137931, 'chair_rec_0.50': 0.0, 'sofa_rec_0.50': 0.0, 'table_rec_0.50': 0.017142857142857144, 'otherfurniture_rec_0.50': 0.022641509433962263, 'curtain_rec_0.50': 0.0, 'sink_rec_0.50': 0.0, 'door_rec_0.50': 0.0, 'counter_rec_0.50': 0.0, 'bathtub_rec_0.50': 0.25806451612903225, 'cabinet_rec_0.50': 0.005376344086021506, 'bed_rec_0.50': 0.0, 'window_rec_0.50': 0.0, 'mAR_0.50': 0.10774226114657273}  data_time: 0.0368  time: 0.1793

Here is my modified Dockerfile:

FROM pytorch/pytorch:2.1.2-cuda12.1-cudnn8-devel

# Install base system packages
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    ffmpeg libsm6 libxext6 git ninja-build libglib2.0-0 libxrender-dev libopenblas-dev

# Set environment variables for CUDA extension builds
ENV MMCV_WITH_OPS=1
ENV FORCE_CUDA=1
ENV TORCH_CUDA_ARCH_LIST="8.6"

# Install MinkowskiEngine
RUN pip install git+https://github.com/daizhirui/MinkowskiEngine.git@ce930eeb403a8e3f99693662ec5ce329a0ab3528 -v --no-deps \
    --global-option="--blas=openblas" \
    --global-option="--force_cuda"

# Install OpenMMLab packages (without deps to speed up and prevent conflict)
RUN pip install --no-deps \
    mmengine==0.9.0 \
    mmdet==3.3.0 \
    mmsegmentation==1.2.0 \
    mmdet3d==1.4.0 \
    mmpretrain==1.2.0

# Install and build mmcv with CUDA ops
RUN git clone https://github.com/open-mmlab/mmcv.git /workspace/mmcv && \
    cd /workspace/mmcv && \
    git reset --hard 780ffed9f3736fedadf18b51266ecbf521e64cf6 && \
    python setup.py clean && \
    rm -rf build && \
    find . -name "*.so" -delete && \
    pip install -v -e . --no-deps

# Install torch-scatter (with CUDA 12.1)
RUN pip install torch-scatter==2.1.2 -f https://data.pyg.org/whl/torch-2.1.0+cu121.html --no-deps

# Install system Python dev headers
RUN apt-get update && apt-get install -y python3-dev

# Build segmentator (superpoint backend for ScanNet)
RUN git clone https://github.com/Karbo123/segmentator.git /workspace/segmentator && \
    cd /workspace/segmentator/csrc && \
    git reset --hard 76efe46d03dd27afa78df972b17d07f2c6cfb696 && \
    sed -i "s/set(CMAKE_CXX_STANDARD 14)/set(CMAKE_CXX_STANDARD 17)/g" CMakeLists.txt && \
    mkdir build && cd build && \
    cmake .. \
        -DCMAKE_PREFIX_PATH=$(python -c 'import torch;print(torch.utils.cmake_prefix_path)') \
        -DPYTHON_INCLUDE_DIR=$(python -c "import sysconfig; print(sysconfig.get_path('include'))") \
        -DPYTHON_LIBRARY=$(python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR') + '/libpython' + sysconfig.get_config_var('VERSION') + '.so')") \
        -DCMAKE_INSTALL_PREFIX=$(python -c 'from distutils.sysconfig import get_python_lib') && \
    make && make install

# Install Python packages (no system deps)
RUN pip install --no-deps \
    spconv-cu120==2.3.6 \
    addict==2.4.0 \
    yapf==0.33.0 \
    termcolor==2.3.0 \
    packaging==23.1 \
    numpy==1.24.1 \
    rich==13.3.5 \
    opencv-python==4.7.0.72 \
    pycocotools==2.0.6 \
    Shapely==1.8.5 \
    scipy==1.10.1 \
    terminaltables==3.1.10 \
    numba==0.57.0 \
    llvmlite==0.40.0 \
    pccm==0.4.7 \
    ccimport==0.4.2 \
    pybind11==2.10.4 \
    ninja==1.11.1 \
    lark==1.1.5 \
    cumm-cu120==0.5.1 \
    pyquaternion==0.9.9 \
    lyft-dataset-sdk==0.0.8 \
    pandas==2.0.1 \
    python-dateutil==2.8.2 \
    matplotlib==3.5.2 \
    pyparsing==3.0.9 \
    cycler==0.11.0 \
    kiwisolver==1.4.4 \
    scikit-learn==1.2.2 \
    joblib==1.2.0 \
    threadpoolctl==3.1.0 \
    cachetools==5.3.0 \
    nuscenes-devkit==1.1.10 \
    trimesh==3.21.6 \
    open3d==0.17.0 \
    plotly==5.18.0 \
    dash==2.14.2 \
    plyfile==1.0.2 \
    flask==3.0.0 \
    werkzeug==3.0.1 \
    click==8.1.7 \
    blinker==1.7.0 \
    itsdangerous==2.1.2 \
    importlib_metadata==2.1.2 \
    zipp==3.17.0 \
    natsort==8.4.0 \
    timm==0.9.16 \
    imageio==2.34.0 \
    portalocker==2.8.2 \
    ftfy==6.2.0 \
    regex==2024.4.16

[filaPro]

Our checkpoint in corresponding to the 6-dataset config. Please use 6-dataset config and comment non-necessary datasets from validation.

[aayush-shah14]

Thank you so much, I was able to reproduce the results using that script. I also had a query regarding running the model on my custom point cloud data. If I format the data in Scannet format, and run the create_data script, will it preprocess the data accordingly to be used directly? Presently I have a .ply file, and I was trying to figure out how to inference using UniDet3D. I checked the other threads mentioning this issue, however they seem to be following the approach of doing superpoint clustering manually, whereas I was wondering if directly using create_data.py on a properly formatted dataset could work.