exiawsh/streampetr

[ICCV 2023] StreamPETR: Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection

streampetr's Issues

why all the ground truth 3dbbox and intrinsics extrinsics all changed after train_pipeline process？

installing flash-attn

Hi everyone
I tried to install the libraries on colab, but I had a problem with one of these libraries and could not install them:

Result mAP

Hi, I hope you are well
According to #23 (comment) I did 20,000 iterations but the results were not good and it didn't work at all! What should I do? (because I want it to be based on my own training)

ViT-Large Backbone in StreamPETR-large

Hi, i'm very interested in your work and its very helpful for me. I have a few questions about the config of StreamPETR-large:

1. MIM-pretrained VIT-Large backbone ？ Is this the same as MMPretrain's ?
2. The patch size is set to 16, right?
3. What about the img_neck? Does it need to combine features from different layers in ViT? if so, which layers?

Thanks very much!

No module named 'flash_attn'

Hi, thanks for sharing your wonderful work!
I met a problem when I was trying to run PETRv1 testing using the following code on Tesla V100-PCIE-32GB:

tools/dist_test.sh projects/configs/PETRv1/petrv1_r50_flash_704_24e.py ckpts/fcos3d_vovnet_imgbackbone-remapped.pth 4 --eval bbox

I have read the docs and a relevant issue, and have changed type='PETRMultiheadFlashAttention' in petrv1_r50_flash_704_24e.py line 80 to type='PETRMultiheadAttention' , but I still got the following error:

ModuleNotFoundError: No module named 'flash_attn'

For more detailed infomation:

pip list:

Package                  Version      Editable project location
------------------------ ------------ ------------------------------------------------------
absl-py                  1.4.0
addict                   2.4.0
anyio                    3.7.0
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
asttokens                2.2.1
attrs                    23.1.0
backcall                 0.2.0
beautifulsoup4           4.12.2
black                    23.3.0
bleach                   6.0.0
cachetools               5.3.1
certifi                  2023.5.7
cffi                     1.15.1
charset-normalizer       3.1.0
click                    8.1.3
colorama                 0.4.6
comm                     0.1.3
contourpy                1.0.7
cycler                   0.11.0
debugpy                  1.6.7
decorator                5.1.1
defusedxml               0.7.1
descartes                1.1.0
einops                   0.6.1
exceptiongroup           1.1.1
executing                1.2.0
fastjsonschema           2.17.1
fire                     0.5.0
flake8                   6.0.0
fonttools                4.39.4
fqdn                     1.5.1
google-auth              2.19.0
google-auth-oauthlib     1.0.0
grpcio                   1.54.2
idna                     3.4
imageio                  2.29.0
importlib-metadata       6.6.0
importlib-resources      5.12.0
iniconfig                2.0.0
ipykernel                6.23.1
ipython                  8.12.2
ipython-genutils         0.2.0
ipywidgets               8.0.6
isoduration              20.11.0
jedi                     0.18.2
Jinja2                   3.1.2
joblib                   1.2.0
jsonpointer              2.3
jsonschema               4.17.3
jupyter                  1.0.0
jupyter_client           8.2.0
jupyter-console          6.6.3
jupyter_core             5.3.0
jupyter-events           0.6.3
jupyter_server           2.6.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments      0.2.2
jupyterlab-widgets       3.0.7
kiwisolver               1.4.4
llvmlite                 0.36.0
lyft-dataset-sdk         0.0.8
Markdown                 3.4.3
markdown-it-py           2.2.0
MarkupSafe               2.1.2
matplotlib               3.5.2
matplotlib-inline        0.1.6
mccabe                   0.7.0
mdurl                    0.1.2
mistune                  2.0.5
mmcls                    0.25.0
mmcv-full                1.6.0
mmdet                    2.28.2
mmdet3d                  1.0.0rc6     /home/q.du/workspace/3d_object_detection/mmdetection3d
mmsegmentation           0.30.0
model-index              0.1.11
mypy-extensions          1.0.0
nbclassic                1.0.0
nbclient                 0.8.0
nbconvert                7.4.0
nbformat                 5.8.0
nest-asyncio             1.5.6
networkx                 2.2
notebook                 6.5.4
notebook_shim            0.2.3
numba                    0.53.0
numpy                    1.23.4
nuscenes-devkit          1.1.10
oauthlib                 3.2.2
opencv-python            4.7.0.72
openmim                  0.3.7
ordered-set              4.1.0
overrides                7.3.1
packaging                23.1
pandas                   2.0.1
pandocfilters            1.5.0
parso                    0.8.3
pathspec                 0.11.1
pexpect                  4.8.0
pickleshare              0.7.5
Pillow                   9.5.0
pip                      23.0.1
pkgutil_resolve_name     1.3.10
platformdirs             3.5.1
plotly                   5.14.1
pluggy                   1.0.0
plyfile                  0.9
prettytable              3.7.0
prometheus-client        0.17.0
prompt-toolkit           3.0.38
protobuf                 4.23.2
psutil                   5.9.5
ptyprocess               0.7.0
pure-eval                0.2.2
pyasn1                   0.5.0
pyasn1-modules           0.3.0
pycocotools              2.0.6
pycodestyle              2.10.0
pycparser                2.21
pyflakes                 3.0.1
Pygments                 2.15.1
pyparsing                3.0.9
pyquaternion             0.9.9
pyrsistent               0.19.3
pytest                   7.3.1
python-dateutil          2.8.2
python-json-logger       2.0.7
pytz                     2023.3
PyWavelets               1.4.1
PyYAML                   6.0
pyzmq                    25.1.0
qtconsole                5.4.3
QtPy                     2.3.1
requests                 2.31.0
requests-oauthlib        1.3.1
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
rich                     13.3.5
rsa                      4.9
scikit-image             0.19.3
scikit-learn             1.2.2
scipy                    1.10.1
Send2Trash               1.8.2
setuptools               66.0.0
Shapely                  1.8.5
six                      1.16.0
sniffio                  1.3.0
soupsieve                2.4.1
stack-data               0.6.2
tabulate                 0.9.0
tenacity                 8.2.2
tensorboard              2.13.0
tensorboard-data-server  0.7.0
termcolor                2.3.0
terminado                0.17.1
terminaltables           3.1.10
threadpoolctl            3.1.0
tifffile                 2023.4.12
tinycss2                 1.2.1
tomli                    2.0.1
torch                    1.9.0+cu111
torchaudio               0.9.0
torchvision              0.10.0+cu111
tornado                  6.3.2
tqdm                     4.65.0
traitlets                5.9.0
trimesh                  2.35.39
typing_extensions        4.6.2
tzdata                   2023.3
uri-template             1.2.0
urllib3                  1.26.16
wcwidth                  0.2.6
webcolors                1.13
webencodings             0.5.1
websocket-client         1.5.2
Werkzeug                 2.3.4
wheel                    0.38.4
widgetsnbextension       4.0.7
yapf                     0.33.0
zipp                     3.15.0

Any ideas about this problem?

Ablation study about img_roi_head

Thanks for your great work!
I found that most camera-based 3D detection methods had poor performance in predicting depth, which is the key to increasing mAP and NDS.
As mentioned in BEVFormerV2, the perspective 2D supervision is important for predicting depth and obtaining more accurate image features.
So I'm interested in the ablation study about img_roi_head if you have done it.

dataset problem

Hello, is the model trained by keframe or full of nuscence dataset?

How to change multi-GPU training to single GPU training

I noticed that this amazing project uses 8 GPU for training in setting, and I would like to ask how to conduct a preliminary training if I only have one GPU

请问下tracking相关模块预计什么时候释放？

您好，请问下tracking相关模块预计什么时候释放？

Segmentation task will be supported in the near future？

How would I perform inference on a single jpg image?

Dear StreamPETR team,

I have managed to reproduce your results under your paper using the stream_petr_vov_flash_800_bs2_seq_24e.py config file and latest checkpoint. I am wondering how I may modify test.py to load my own image and run inference on the model. Thank you!

Questions about queue_length in stream training and prev_exists flag ?

1. Hi, after reading the code implementation, I'm curious about why you set queue_length = 1 in your config when you used stream training. Wouldn't the performance be better if your increase queue_length ?
2. If the queue_length is set to 1, what's is the difference between prev_exists flag is True or False ?

How about the performance with Streaming Video Training.

Hi, thanks for your great work. I wonder about the performance of streaming video training. Does it achieve the same mAP/NDS as the random shuffle one?

How to eval model every epoch?

Hello, I change the interval in config to num_iters_per_epoch, but i got error after first eval.
evaluation = dict(interval=num_iters_per_epoch, pipeline=test_pipeline)

[                                                  ] 2/6019, 0.6 task/s, elapsed: 3s, ETA:  9675s/app/StreamPETR/projects/mmdet3d_plugin/core/bbox/coders/nms_free_coder.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.post_center_range = torch.tensor(self.post_center_range, device=scores.device)
[                                                  ] 6/6019, 1.7 task/s, elapsed: 3s, ETA:  3456s/app/StreamPETR/projects/mmdet3d_plugin/core/bbox/coders/nms_free_coder.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.post_center_range = torch.tensor(self.post_center_range, device=scores.device)
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6020/6019, 16.8 task/s, elapsed: 358s, ETA:     0sTraceback (most recent call last):
  File "tools/train.py", line 263, in <module>
    main()
  File "tools/train.py", line 251, in main
    custom_train_model(
  File "/app/StreamPETR/projects/mmdet3d_plugin/core/apis/train.py", line 30, in custom_train_model
    custom_train_detector(
  File "/app/StreamPETR/projects/mmdet3d_plugin/core/apis/mmdet_train.py", line 203, in custom_train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/app/conda/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 144, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "/app/conda/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 64, in train
    outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
  File "/app/conda/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 63, in train_step
    output = self.module.train_step(*inputs[0], **kwargs[0])
  File "/app/conda/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/app/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/app/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 233, in new_func
    output = old_func(*new_args, **new_kwargs)
  File "/app/StreamPETR/projects/mmdet3d_plugin/models/detectors/petr3d.py", line 217, in forward
    return self.forward_train(**data)
  File "/app/StreamPETR/projects/mmdet3d_plugin/models/detectors/petr3d.py", line 269, in forward_train
    losses = self.obtain_history_memory(gt_bboxes_3d,
  File "/app/StreamPETR/projects/mmdet3d_plugin/models/detectors/petr3d.py", line 130, in obtain_history_memory
    loss = self.forward_pts_train(gt_bboxes_3d[i],
  File "/app/StreamPETR/projects/mmdet3d_plugin/models/detectors/petr3d.py", line 189, in forward_pts_train
    outs = self.pts_bbox_head(location, img_metas, topk_indexes, **data)
  File "/app/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/app/StreamPETR/projects/mmdet3d_plugin/models/dense_heads/streampetr_head.py", line 581, in forward
    self.pre_update_memory(data)
  File "/app/StreamPETR/projects/mmdet3d_plugin/models/dense_heads/streampetr_head.py", line 330, in pre_update_memory
    self.memory_timestamp += data['timestamp'].unsqueeze(-1).unsqueeze(-1)
RuntimeError: output with shape [1, 1280, 1] doesn't match the broadcast shape [4, 1280, 1]

inference with nuScence mini dataset- Failed!!!!

Hi,

Great work, much appreciated :)

i am facing issue with the inference of this model,

• config : https://github.com/exiawsh/StreamPETR/blob/main/projects/configs/StreamPETR/stream_petr_vov_flash_800_bs2_seq_24e.py
• model : https://github.com/exiawsh/storage/releases/download/v1.0/stream_petr_vov_flash_800_bs2_seq_24e.pth

my package are as below :-
flash-attn 0.2.8
torch 1.12.1+cu116
torchaudio 0.12.1+cu116
torchvision 0.13.1+cu116

i am running on GPU 1080Ti, with cuda 11.6.

Error as below:-

File "/media/ubuntu/data/git_downloads/StreamPETR/projects/mmdet3d_plugin/models/utils/attention.py", line 74, in forward
output = flash_attn_unpadded_kvpacked_func(
File "/media/ubuntu/data/envs/streampetr/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 299, in flash_attn_unpadded_kvpacked_func
return FlashAttnKVPackedFunc.apply(q, kv, cu_seqlens_q, cu_seqlens_k,
File "/media/ubuntu/data/envs/streampetr/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 96, in forward
out, softmax_lse, S_dmask = _flash_attn_forward(
File "/media/ubuntu/data/envs/streampetr/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 21, in _flash_attn_forward
softmax_lse, *rest = flash_attn_cuda.fwd(
RuntimeError: Expected is_sm8x || is_sm75 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

kindly let me know if any where am i doing wrong, correct me if so.

Thanking you in advance :)

Inquiry on config parameters

May I check what is position_range, post_centre_range and point_cloud_range used for in the model?

Taken from cfg file

...
bbox_coder=dict(
            type='NMSFreeCoder',
            # post_center_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
            post_center_range=[0, 0, 0, 61.2, 61.2, 10.0],
            pc_range=point_cloud_range,
            max_num=300,
            voxel_size=voxel_size,
            num_classes=10), 
...

...
pts_bbox_head=dict(
        type='StreamPETRHead',
        num_classes=10,
        in_channels=256,
        num_query=644,
        memory_len=1024,
        topk_proposals=256,
        num_propagated=256,
        with_ego_pos=True,
        match_with_velo=False,
        scalar=10, ##noise groups
        noise_scale = 1.0, 
        dn_weight= 1.0, ##dn loss weight
        split = 0.75, ###positive rate
        LID=True,
        with_position=True,
        position_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
        code_weights = [2.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
        .....

Ego-pose is False then results are too poor!!! Please guide me

Hi,

much appreciated with the work with good FPS and performance.

my set up is same - #1

i am making false for this field with_ego_pose(https://github.com/exiawsh/StreamPETR/blob/main/projects/configs/StreamPETR/stream_petr_vov_flash_800_bs2_seq_24e.py#L82) as mention in paper - tab5 with out Ego pose, time and velocity. Below are the metrics.

please suggested me, how to disable (ego-pose).

thanking you in advance :)

Problem with data preparation （关于数据准备的问题）

Hi, I've found that our project's data preparation is different from others. The problem is if I creates 2D annotations and temporal information follow data_preparatio.md in our project, will it effect other projects like BEVDet, SOLOFusion which use the same data root(nuscenes).
Thank you.

作者你好，我发现我们的data preparation和现有的工作不太一致，我想问下，如果按照我们的pipeline对nuscenes数据集进行了处理，对于我其他使用nuscnenes数据集的project（例如BEVDet，SOLOFusion那些完全follow mmdet3d）的工作是否会有影响？谢谢

3D photo target coordinates

Hello again
Thank you very much for the previous tips
I got the final result.
I'm going to give it a picture and give me the 3D box with the coordinates, what should I do?

Ask about motion-aware layer normalization

Thanks for sharing this nice work!

I'm very interested in the motion-aware layer normalization module. In Tab. 5, the performance does not improve when adopting explicit motion compensation (MC), this conclusion is a bit counter-intuitive. I would like to ask about the details of the explicit implementation. For example, is the position of the memory query updated by ego pose translation first, seems like p = E^(t)_inv @ E^(t-1) @ p. Then it's updated by motion, p = p + vt. Do you use the updated position to generate position encoding? How about the context embedding, did you update it in some way, or just keep it in the explicit MC implementation?

about TensorRT inference

Hi, I see you list "TensorRT inference" on the checklist, are you still working on the TensorRT inference? I'm trying to use mmdeploy on this project but things not go well. Do you have some suggestions about it?
Btw, when will this function updated on this project? Thank you.

How to obtain the result on Nuscenes test set?

I trained the model with DD3D pre-trained weights, and obtained the results of 51.1 mAP and 59.5 NDS on Nuscenes validation set. And I want to further reproduce the results on Nuscenes test set, how can I achieve this? Thanks very much.

It seems that there is no label on test set, and simply replace 'nuscenes2d_temporal_infos_val.pkl' with 'nuscenes2d_temporal_infos_test.pkl' in config file doesn't work.

question about tracking

First, thank for your great work.
However, I have a question about tracking, you have mentioned to use
'python nusc_tracking/pub_test --version v1.0-trainval --checkpoint {PATH_RESULTS.JSON} --data_root {PATH_NUSCENES}'
to evaluate the tracking.
Could you plz tell me how to get the {PATH_RESULTS.JSON}?

Loading Custom Data to perform inference with StreamPETR

Dear StreamPETR team,

We are following this documentation to try to use our custom set of images from a video stream. We want to preliminarily perform inference on the custom set of images. However, we are having trouble with creating a new dataset.

Instead, we decided to manually create our own data loader for our custom dataset. We thus want to clarify the proper way to load the image tensor from the image file path.

We will further modify the following:

• the camera extrinsic and intrinsics
• point cloud range

We will disable the following in config:

• LiDAR and ego_pose (However, the model still requires these keys, so I believe that we need to feed in random data for these keys) (Do correct me if I am wrong).

We have also compiled further questions in this document. We hope for your reply, thank you!

can not find the dist_params in config file

/StreamPETR-main/tools/train.py", line 165, in main
init_dist(args.launcher, **cfg.dist_params)
/python3.9/site-packages/mmcv/utils/config.py", line 50, in getattr
raise ex
AttributeError: 'ConfigDict' object has no attribute 'dist_params'
raise ex
AttributeError: 'ConfigDict' object has no attribute 'dist_params'

sweep data of nuscenes

May I ask if the sweep data of nuScenes is necessary in training or testing？
Thank you very much for your help！

Small bug

Hi exiawsh,

Here is a small bug, all configs provided in "projects/configs/StreamPETR/*.py", the val&test of data config didn't clarified the data_root, only train clarified. Maybe you can fixed this bug in the next version~

Still about TensorRT inference

@exiawsh Hi, thank you for your suggestion last time. After that, I have tried with https://github.com/woquyuna/StreamPETR/blob/deployV1.0/docs/deploy.md, but fialed at last.
Two problems happened here.

1. I tried to run this command: tools/dist_test_pth.sh projects/configs/StreamPETR/stream_petr_r50_704_bs2_seq_428q_nui_60e_pth.py ckpts/stream_petr_r50_flash_704_bs2_seq_428q_nui_60e.pth 1 --eval=bbox
   , and it reported with this error:
   
   Is it because stream_petr_r50_704_bs2_seq_428q_nui_60e_pth.py differing from stream_petr_r50_flash_704_bs2_seq_428q_nui_60e.pth and this author had another pth file. So I try to train with stream_petr_r50_flash_704_bs2_seq_428q_nui_60e.pth config file.
2. tools/dist_train.sh projects/configs/StreamPETR/stream_petr_r50_704_bs2_seq_428q_nui_60e_pth.py 8 --work-dir work_dirs/stream_petr_r50_704_bs2_seq_428q_nui_60e/
   Then I run this command, and it report with this error:
   
   It seems that this author had changed something with models/dense_heads/streampetr_head_pth.py or petr3d_pth

P.S.: I tried to run this training command: "tools/dist_train.sh projects/configs/StreamPETR/stream_petr_r50_flash_704_bs2_seq_24e.py 8 --work-dir work_dirs/stream_petr_r50_flash_704_bs2_seq_24e/", and it runs well.

Map Mask nuscenes

Hello again
I got to the test section, but I run into a problem.
According to issue 23, I don't have a part.

I downloaded the folder map from nuscenes, but it doesn't seem to be part of it

dataset queue length conflict

Hi, Thank you for your sharing in the first place. I was using the default streaming config with queue length = 1. and over here when preparing the samples,

https://github.com/exiawsh/StreamPETR/blob/1111ffaa685068059fb112e58c6c4bb9d9c418f6/projects/mmdet3d_plugin/datasets/nuscenes_dataset.py#L91

random length is 0 by default, and queue_length equals to 1 and index_list will only contains one sample in the end but generally it should contains two samples since it was used at here later on
https://github.com/exiawsh/StreamPETR/blob/1111ffaa685068059fb112e58c6c4bb9d9c418f6/projects/mmdet3d_plugin/models/detectors/petr3d.py#L255

Could you please point out where am I wrong about this? do we need 2 samples each iter all the time? since I saw there's one example of [batchsize, 1, 6, 3, h, w] in the issues before. Thank you!

config of vit large

Thanks for sharing the great work!

I was wondering, it would be great if you could also share the training config files and checkpoints for nuscenes leaderboard version which achieves 0.62 map and 0.676 nds.

Looking forward to your reply

Can not reproduce the results of R50 nuimage config.

I used 8*V100 to train the stream_petr_r50_flash_704_bs2_seq_428q_nui_60e config, but I got the worse performance than the repo.
53.45 NDS, 43.4 mAP

When I used the SyncBN and set requires_grad=True, norm_eval=False, I got even worse results. 😭
53.07 NDS, 43.03 mAP

BTW, I didn't change anything in the config while I was not using SyncBN.

pip issue with MMCV

Hi I installed required MMCV-full == 1.6.0 but when I run the train.py in tools folder I run into this issue. Is there a way to fix it? thanks!

Traceback (most recent call last):
File "/home/Desktop/Git/StreamPETR/tools/train.py", line 21, in
from mmdet3d import version as mmdet3d_version
File "/home/anaconda3/envs/streampetr/lib/python3.8/site-packages/mmdet3d/init.py", line 21, in
assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.6.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.1.0.

Warning of writing non-writable tensors

Hi,

thank you for yr excellent work! I have a warning when retraining with config stream_petr_r50_flash_704_bs2_seq_24e as below:

StreamPETR/projects/mmdet3d_plugin/models/utils/grid_mask.py:113: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:199.)
mask = torch.from_numpy(mask).float().cuda()

is this my own case problem or anyone has the same problem with me? Does it affect the final result?

Thank you!

Are there any difficulties in implementing the solution in actual scenarios?

Error in training

Hello everyone
Thank you very much for your attractive project.
As you said in the instructions, I went and installed the libraries and made the folders and put the pkl related to the V1.0-mini version that I asked you about in the previous question in the data/nuscenes folder, but for train I had a problem and it doesn't start working.
At first I will send the folder and after that I will send the error. Thank you for your help.

├── ckpts
├── data
│   └── nuscenes
│       ├── maps
│       │   ├── basemap
│       │   ├── expansion
│       │   └── prediction
│       ├── samples
│       │   ├── CAM_BACK
│       │   ├── CAM_BACK_LEFT
│       │   ├── CAM_BACK_RIGHT
│       │   ├── CAM_FRONT
│       │   ├── CAM_FRONT_LEFT
│       │   ├── CAM_FRONT_RIGHT
│       │   ├── LIDAR_TOP
│       │   ├── RADAR_BACK_LEFT
│       │   ├── RADAR_BACK_RIGHT
│       │   ├── RADAR_FRONT
│       │   ├── RADAR_FRONT_LEFT
│       │   └── RADAR_FRONT_RIGHT
│       ├── sweeps
│       │   ├── CAM_BACK
│       │   ├── CAM_BACK_LEFT
│       │   ├── CAM_BACK_RIGHT
│       │   ├── CAM_FRONT
│       │   ├── CAM_FRONT_LEFT
│       │   ├── CAM_FRONT_RIGHT
│       │   ├── LIDAR_TOP
│       │   ├── RADAR_BACK_LEFT
│       │   ├── RADAR_BACK_RIGHT
│       │   ├── RADAR_FRONT
│       │   ├── RADAR_FRONT_LEFT
│       │   └── RADAR_FRONT_RIGHT
│       └── v1.0-mini
├── docs
├── figs
├── mmdetection3d
│   ├── configs
│   │   ├── 3dssd
│   │   ├── _base_
│   │   │   ├── datasets
│   │   │   ├── models
│   │   │   └── schedules
│   │   ├── benchmark
│   │   ├── centerpoint
│   │   ├── dgcnn
│   │   ├── dynamic_voxelization
│   │   ├── fcaf3d
│   │   ├── fcos3d
│   │   ├── free_anchor
│   │   ├── groupfree3d
│   │   ├── h3dnet
│   │   ├── imvotenet
│   │   ├── imvoxelnet
│   │   ├── monoflex
│   │   ├── mvxnet
│   │   ├── nuimages
│   │   ├── paconv
│   │   ├── parta2
│   │   ├── pgd
│   │   ├── pointnet2
│   │   ├── pointpillars
│   │   ├── point_rcnn
│   │   ├── regnet
│   │   ├── sassd
│   │   ├── second
│   │   ├── smoke
│   │   ├── ssn
│   │   └── votenet
│   ├── data
│   │   ├── lyft
│   │   ├── nuscenes
│   │   │   ├── maps
│   │   │   │   ├── basemap
│   │   │   │   ├── expansion
│   │   │   │   └── prediction
│   │   │   ├── samples
│   │   │   │   ├── CAM_BACK
│   │   │   │   ├── CAM_BACK_LEFT
│   │   │   │   ├── CAM_BACK_RIGHT
│   │   │   │   ├── CAM_FRONT
│   │   │   │   ├── CAM_FRONT_LEFT
│   │   │   │   ├── CAM_FRONT_RIGHT
│   │   │   │   ├── LIDAR_TOP
│   │   │   │   ├── RADAR_BACK_LEFT
│   │   │   │   ├── RADAR_BACK_RIGHT
│   │   │   │   ├── RADAR_FRONT
│   │   │   │   ├── RADAR_FRONT_LEFT
│   │   │   │   └── RADAR_FRONT_RIGHT
│   │   │   ├── sweeps
│   │   │   │   ├── CAM_BACK
│   │   │   │   ├── CAM_BACK_LEFT
│   │   │   │   ├── CAM_BACK_RIGHT
│   │   │   │   ├── CAM_FRONT
│   │   │   │   ├── CAM_FRONT_LEFT
│   │   │   │   ├── CAM_FRONT_RIGHT
│   │   │   │   ├── LIDAR_TOP
│   │   │   │   ├── RADAR_BACK_LEFT
│   │   │   │   ├── RADAR_BACK_RIGHT
│   │   │   │   ├── RADAR_FRONT
│   │   │   │   ├── RADAR_FRONT_LEFT
│   │   │   │   └── RADAR_FRONT_RIGHT
│   │   │   └── v1.0-mini
│   │   ├── s3dis
│   │   │   └── meta_data
│   │   ├── scannet
│   │   │   └── meta_data
│   │   └── sunrgbd
│   │       └── matlab
│   ├── demo
│   │   └── data
│   │       ├── kitti
│   │       ├── nuscenes
│   │       ├── scannet
│   │       └── sunrgbd
│   ├── docker
│   │   └── serve
│   ├── docs
│   │   ├── en
│   │   │   ├── datasets
│   │   │   ├── _static
│   │   │   │   ├── css
│   │   │   │   └── image
│   │   │   ├── supported_tasks
│   │   │   └── tutorials
│   │   └── zh_cn
│   │       ├── datasets
│   │       ├── _static
│   │       │   ├── css
│   │       │   └── image
│   │       ├── supported_tasks
│   │       └── tutorials
│   ├── mmdet3d
│   │   ├── apis
│   │   ├── core
│   │   │   ├── anchor
│   │   │   ├── bbox
│   │   │   │   ├── assigners
│   │   │   │   ├── coders
│   │   │   │   ├── iou_calculators
│   │   │   │   ├── samplers
│   │   │   │   └── structures
│   │   │   ├── evaluation
│   │   │   │   ├── kitti_utils
│   │   │   │   ├── scannet_utils
│   │   │   │   └── waymo_utils
│   │   │   ├── points
│   │   │   ├── post_processing
│   │   │   ├── utils
│   │   │   ├── visualizer
│   │   │   └── voxel
│   │   ├── datasets
│   │   │   └── pipelines
│   │   ├── models
│   │   │   ├── backbones
│   │   │   ├── decode_heads
│   │   │   ├── dense_heads
│   │   │   ├── detectors
│   │   │   ├── fusion_layers
│   │   │   ├── losses
│   │   │   ├── middle_encoders
│   │   │   ├── model_utils
│   │   │   ├── necks
│   │   │   ├── roi_heads
│   │   │   │   ├── bbox_heads
│   │   │   │   ├── mask_heads
│   │   │   │   └── roi_extractors
│   │   │   ├── segmentors
│   │   │   ├── utils
│   │   │   └── voxel_encoders
│   │   ├── ops
│   │   │   ├── dgcnn_modules
│   │   │   ├── paconv
│   │   │   ├── pointnet_modules
│   │   │   └── spconv
│   │   │       └── overwrite_spconv
│   │   └── utils
│   ├── mmdet3d.egg-info
│   ├── projects
│   │   └── example_project
│   │       ├── configs
│   │       └── dummy
│   ├── requirements
│   ├── resources
│   ├── tests
│   │   ├── data
│   │   │   ├── kitti
│   │   │   │   ├── kitti_gt_database
│   │   │   │   └── training
│   │   │   │       ├── image_2
│   │   │   │       ├── velodyne
│   │   │   │       └── velodyne_reduced
│   │   │   ├── lyft
│   │   │   │   ├── lidar
│   │   │   │   └── v1.01-train
│   │   │   │       ├── maps
│   │   │   │       └── v1.01-train
│   │   │   ├── nuscenes
│   │   │   │   ├── samples
│   │   │   │   │   ├── CAM_BACK_LEFT
│   │   │   │   │   └── LIDAR_TOP
│   │   │   │   └── sweeps
│   │   │   │       └── LIDAR_TOP
│   │   │   ├── ops
│   │   │   ├── s3dis
│   │   │   │   ├── instance_mask
│   │   │   │   ├── points
│   │   │   │   └── semantic_mask
│   │   │   ├── scannet
│   │   │   │   ├── instance_mask
│   │   │   │   ├── points
│   │   │   │   └── semantic_mask
│   │   │   ├── semantickitti
│   │   │   │   └── sequences
│   │   │   │       └── 00
│   │   │   │           ├── labels
│   │   │   │           └── velodyne
│   │   │   ├── sunrgbd
│   │   │   │   ├── points
│   │   │   │   └── sunrgbd_trainval
│   │   │   │       └── image
│   │   │   └── waymo
│   │   │       ├── kitti_format
│   │   │       │   ├── training
│   │   │       │   │   ├── image_0
│   │   │       │   │   └── velodyne
│   │   │       │   └── waymo_gt_database
│   │   │       └── waymo_format
│   │   │           └── validation
│   │   ├── test_data
│   │   │   ├── test_datasets
│   │   │   └── test_pipelines
│   │   │       ├── test_augmentations
│   │   │       └── test_loadings
│   │   ├── test_metrics
│   │   ├── test_models
│   │   │   ├── test_common_modules
│   │   │   ├── test_fusion
│   │   │   ├── test_heads
│   │   │   ├── test_necks
│   │   │   └── test_voxel_encoder
│   │   ├── test_runtime
│   │   ├── test_samples
│   │   └── test_utils
│   └── tools
│       ├── analysis_tools
│       ├── data_converter
│       ├── deployment
│       ├── misc
│       └── model_converters
├── nusc_tracking
├── projects
│   ├── configs
│   │   ├── PETRv1
│   │   ├── StreamPETR
│   │   └── test_speed
│   └── mmdet3d_plugin
│       ├── core
│       │   ├── apis
│       │   ├── bbox
│       │   │   ├── assigners
│       │   │   ├── coders
│       │   │   └── match_costs
│       │   └── evaluation
│       ├── datasets
│       │   ├── pipelines
│       │   └── samplers
│       └── models
│           ├── backbones
│           │   └── __pycache__
│           ├── dense_heads
│           ├── detectors
│           ├── necks
│           └── utils
└── tools
    └── data_converter
        └── __pycache__

.
.
.

Does cls_branches/reg_branches share the weights?

Hello, from the denfinition, cls branches seems share the weights with fc_cls, but in forward function, outputs_class = self.cls_branches[lvl](outs_dec[lvl]), which looks like every decoders' output intends to use different classifier.

StreamPETR/projects/mmdet3d_plugin/models/dense_heads/streampetr_head.py

Lines 254 to 257 in 1111ffa

	self.cls_branches = nn.ModuleList(
	[fc_cls for _ in range(self.num_pred)])
	self.reg_branches = nn.ModuleList(
	[reg_branch for _ in range(self.num_pred)])

Whether cls branchs ModuleList shares the weights or not? And will they differ in metrics? Thank you.

What does this sentence mean cone = torch.cat([intrinsic, coords3d[..., -3:], coords3d[..., -90:-87]], dim=-1)？

https://github.com/exiawsh/StreamPETR/blob/main/projects/mmdet3d_plugin/models/dense_heads/streampetr_head.py#L416

coords3d[..., -3:], coords3d[..., -90:-87], what do they represent?

dataset preparation question

Hi, I found that nuscenes web has many different versions of map and data. Which one should I download? Also I wonder what should I put in sweep and samples folder?

Thanks!

Checkpoint of PETRv1?

Hi,
Could you please provide a checkpoint of PETRv1?
Really need that! Thanks!
(Since the env of StreamPETR is much easier to setup than the original PETR, as the author of PETR mentioned many times in the issues under PETR)

latest.pth for nuscense dataset is not available anywhere

Hi, I am quite interested in this work, especially for its comparable results to the lidar object detection. I was trying to regenerate your visualizations without training it. However, I am not able to find a good "latest.pth" for the model. I tried to use dd3d_det_final.pth as below.

./tools/dist_test.sh projects/configs/StreamPETR/stream_petr_vov_flash_800_bs2_seq_24e.py ckpts/dd3d_det_final.pth 1 --format-only --eval-options 'jsonfile_prefix=results/bevfusion-det-val-predictions

However, the "results_nusc.json" does not seem to be correct? Other than trained by myself, can you offer a good latest.pth for me to test on Nuscense dataset

grad_norm:nan

hello, thanks for your work~
when i train using "stream_petr_r50_flash_704_bs1_8key_2grad_24e.py", grad_norm is nan.
Is this normal?

问题

可以提供在ViT-Large-900q上的训练log文件吗

data preparation question

Hi i have some questions towards data preparation

1. Do I need to create a folder called data and manually put everything in that folder like the structure you showed us ?
2. How can I creating infos file, where should I execute the command python tools/create_data_nusc.py --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes2d --version v1.0 ?

Evaluation Error

Hi,
I'm trying to reproduce your great project. I want to get the result first on Mini dataset. So here is what I did.
I followed the environment setup and data preparation steps. Instead of downloading the full-dataset, I replaced them with mini dataset. so here is what my data folder looks like:

I didn't generate the pkl file myself, I found you uploaded mini pkl before, so I downloaded them and renamed them to nuscenes2d_temporal_infos_train.pkl and nuscenes2d_temporal_infos_val.pkl.

Then I went to the third step and want to do the Evaluation:
tools/dist_test.sh projects/configs/StreamPETR/stream_petr_vov_flash_800_bs2_seq_24e.py work_dirs/stream_petr_vov_flash_800_bs2_seq_24e/latest.pth 8 --eval bbox

Since work_dirs/stream_petr_vov_flash_800_bs2_seq_24e/latest.pth is the path for checkpoint so I renamed it to ckpts/stream_petr_vov_flash_800_bs2_seq_24e.pth

And I am using an one gpu 4060 8GB laptop, so I switched 8 to 1. So my running command for Evaluation is :
tools/dist_test.sh projects/configs/StreamPETR/stream_petr_vov_flash_800_bs2_seq_24e.py ckpts/stream_petr_vov_flash_800_bs2_seq_24e.pth 1 --eval bbox

Then running it I got error :
RuntimeError: CUDA error: operation not supported when calling cusparseCreate(handle)

I wonder which step I am missing? I saw that you said in previous issue that a laptop with a gpu that has higher than 6GB could run it. So I want to reproduce your work, I am looking forward to hearing from you, thanks!

will open the big model config of ViT-Large-900q ?

Mini-version of Nuscenes Dataset

Hello everyone!

Tnx for sharing your helpful project.

I have started working on your Github. However, I want to run the code with mini-version of the Nuscenes dataset. In all part of your code, only the full dataset is used. Since, I have three questions!

1- I want to know that, is it possible to run the code with mini-version?
2- Also, is it possible to generate the .pkl files with mmdetection? Or, can you provide it on your Github?
3- Are sweeps data (I mean non-annotated frames) necessary for your training algorithm? Or, just by the keyframes we can run the training code.

Thank you.

focalhead initialization

when i train the model, it throws an error, as follows:
File "/home/xxxx/anaconda3/lib/python3.6/site-packages/mmcv/utils/registry.py", line 55, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
TypeError: FocalHead: Can't instantiate abstract class FocalHead with abstract methods get_bboxes

After train nuscence 6000 iterations with batchsize of 16 , loss and grad_norm become nan. use conf stream_petr_r50_flash_704_bs2_seq_90e.py

2023-05-22 21:39:18,645 - mmdet - INFO - Iter [7050/79110] lr: 7.844e-04, eta: 2 days, 10:28:21, time: 1.844, data_time: 0.164, memory: 21441, frame_0_loss_cls: 1.2109, frame_0_loss_bbox: 2.0770, frame_0_d0.loss_cls: 1.2623, frame_0_d0.loss_bbox: 2.0982, frame_0_d1.loss_cls: 1.2512, frame_0_d1.loss_bbox: 2.0969, frame_0_d2.loss_cls: 1.2396, frame_0_d2.loss_bbox: 2.0894, frame_0_d3.loss_cls: 1.2348, frame_0_d3.loss_bbox: 2.0902, frame_0_d4.loss_cls: 1.2180, frame_0_d4.loss_bbox: 2.0775, frame_0_dn_loss_cls: 0.9295, frame_0_dn_loss_bbox: 1.7723, frame_0_d0.dn_loss_cls: 0.9603, frame_0_d0.dn_loss_bbox: 1.7837, frame_0_d1.dn_loss_cls: 0.9425, frame_0_d1.dn_loss_bbox: 1.7803, frame_0_d2.dn_loss_cls: 0.9305, frame_0_d2.dn_loss_bbox: 1.7719, frame_0_d3.dn_loss_cls: 0.9298, frame_0_d3.dn_loss_bbox: 1.7708, frame_0_d4.dn_loss_cls: 0.9286, frame_0_d4.dn_loss_bbox: 1.7733, frame_0_enc_loss_cls: nan, frame_0_enc_loss_bbox: nan, frame_0_enc_loss_iou: nan, frame_0_centers2d_losses: nan, frame_0_centerness_losses: nan, loss: nan, grad_norm: nan

How to turn off distributed training?

I'm trying your project with a single Nvidia graphics card, but it seems that the train.sh file is using distributed training, I tried changing it to

#!/usr/bin/env bash

CONFIG=$1

PYTHONPATH="$(dirname $0)/..":$PYTHONPATH
python $(dirname "$0")/train.py
$CONFIG
--seed 0

but I got an error

Traceback (most recent call last):
File "tools/train.py", line 263, in
main()
File "tools/train.py", line 251, in main
custom_train_model(
File "/home/Desktop/Git/StreamPETR/projects/mmdet3d_plugin/core/apis/train.py", line 30, in custom_train_model
custom_train_detector(
File "/home/Desktop/Git/StreamPETR/projects/mmdet3d_plugin/core/apis/mmdet_train.py", line 203, in custom_train_detector
runner.run(data_loaders, cfg.workflow)
File "/home/anaconda3/envs/streampetr/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 126, in run
self.call_hook('before_run')
File "/home/anaconda3/envs/streampetr/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 317, in call_hook
getattr(hook, fn_name)(self)
File "/home/anaconda3/envs/streampetr/lib/python3.8/site-packages/mmcv/runner/dist_utils.py", line 135, in wrapper
return func(*args, **kwargs)
File "/home/anaconda3/envs/streampetr/lib/python3.8/site-packages/mmcv/runner/hooks/logger/tensorboard.py", line 47, in before_run
from torch.utils.tensorboard import SummaryWriter
File "/home/anaconda3/envs/streampetr/lib/python3.8/site-packages/torch/utils/tensorboard/init.py", line 4, in
LooseVersion = distutils.version.LooseVersion
AttributeError: module 'distutils' has no attribute 'version'

I have put all required files into ckpt and data already, but still could not train or test visualize. I might need your help, thanks!