dvl-tum/sushi

Official PyTorch implementation of Unifying Short and Long-Term Tracking with Graph Hierarchies (CVPR 2023)

sushi's Issues

threshold detections?

Do you threshold the yolox detections before feeding to SUSHI?

Release 4 level SUSHI trained models

Hi, Ocetintas:
Because of my Nvidia card's memory, I can't train SUSHI model with 9 level. Can you release your 4 level SUHSI pretrained models for me to reproduce and compare?
Many thx!

Weight sharing between levels

Hi Orcun,
I have a question about weight sharing between levels. You said "SUSHI blocks use the same GNN architecture at every hierarchy level. We observe that we can share parameters and learnable weights among the GNNs used for each SUSHI block." in paper. But why train the different layers independent and use different params in code ?

Can a V100 be used for training?

Can a V100 be used for training?

Edge Loss on Tracklets

Hi,

Thanks for sharing the code, amazing work. I would like to ask something because as I read the paper again and again it is still unclear to me. For the edges that connect tracklets, how do you compute the loss. Because the binary labeling while connecting detections is very intuitive, but the tracklets formed in the lower levels of the hierarchy may not contain detections belonging to the same individual. So in that case, what is the true label for the edges connecting the tracklets?

Thanks.

Tracking in a "live" scenario

Hi there,

I was wondering if this approach would work for a "live"/online approach. Such that there is a buffer of X past frames and then the tracker outputs a stable result for each subsequent frame with the buffer working in a FIFO method. Do you believe this would work or does this method require future information to work effectively?

Thanks!

Code to run the demo

Hello authors,

Is there any plan or code already available to run the demo of the tracker on a video?

list index out of range

I'm getting an error when running SUSHI:

Traceback (most recent call last):
  File "/home/atao/devel/unified-dev/SUSHI/scripts/main.py", line 65, in <module>
    epoch_val_logs, epoc_val_logs_per_depth = hicl_tracker.track(dataset=hicl_tracker.test_dataset, output_path=osp.join(hicl_tracker.config.experiment_path, 'test'),
  File "/home/atao/devel/unified-dev/SUSHI/src/tracker/hicl_tracker.py", line 551, in track
    hicl_graphs, _, logs_all = self.hicl_forward(hicl_graphs = hicl_graphs, 
  File "/home/atao/devel/unified-dev/SUSHI/src/tracker/hicl_tracker.py", line 333, in hicl_forward
    curr_batch, _ = self._hicl_to_curr(hicl_graphs=hicl_graphs)  # Create curr_graphs from hierarachical graphs            
  File "/home/atao/devel/unified-dev/SUSHI/src/tracker/hicl_tracker.py", line 195, in _hicl_to_curr
    curr_graph_batch = Batch.from_data_list([hicl_graph.add_edges_to_curr_graph(self.config, curr_graph)
  File "/home/atao/.miniconda/envs/SUSHI/lib/python3.9/site-packages/torch_geometric/data/batch.py", line 97, in from_data_list
    add_batch=not isinstance(data_list[0], Batch),
IndexError: list index out of range

I'm attaching my MOT detection file in case it might be useful for a repro.

yolox.txt

Unable to compute embeddings

I am trying to run the given test code but am unable to compute the embeddings.

Could you identify the issue? is it because of system limitations?
Thank you!

Logs:

(SUSHI) asimzahid@Asim:~/diddo/prototype/SUSHI$  cd /home/asimzahid/diddo/prototype/SUSHI ; /usr/bin/env /home/asimzahid/anaconda3/envs/SUSHI/bin/python /home/asimzahid/.vscode-server/extensions/ms-python.python-2023.10.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 54403 -- main.py --experiment_mode test --cuda --test_splits mot17-test-all --run_id example_mot17_test_fastreid_msmt_BOT_R50_ibn --interpolate_motion --linear_center_only --det_file byte065 --data_path assets/ --reid_embeddings_dir reid_fastreid_msmt_BOT_R50_ibn --node_embeddings_dir node_fastreid_msmt_BOT_R50_ibn --zero_nodes --reid_arch fastreid_msmt_BOT_R50_ibn --edge_level_embed --save_cp --hicl_model_path assets/models/mot17private.pth --output_path output 
Experiment ID: output/experiments/example_mot17_test_fastreid_msmt_BOT_R50_ibn_06-16_21:35:52.788910
Experiment Mode: test
----- CONFIG -----
experiment_mode : test
run_id : example_mot17_test_fastreid_msmt_BOT_R50_ibn
connectivity : chunk
data_path : assets/
output_path : output
seed : 0
cuda : True
num_workers : 4
train_splits : [None]
val_splits : [None]
test_splits : ['mot17-test-all']
cval_seqs : mot17-train-all
det_file : byte065
save_cp : True
feature_embedding_model_path : 
hicl_model_path : assets/models/mot17private.pth
load_train_ckpt : False
share_weights : all_but_first
node_level_embed : False
edge_level_embed : True
reid_arch : fastreid_msmt_BOT_R50_ibn
reid_sim_fn : l2
edge_sim_fn : l2
node_dim : 2048
l2_norm_reid : False
reid_embeddings_dir : reid_fastreid_msmt_BOT_R50_ibn
node_embeddings_dir : node_fastreid_msmt_BOT_R50_ibn
num_epoch : 250
num_batch : 8
lr : 0.0003
gamma : 1
weight_decay : 0.0001
augmentation : False
train_dataset_frame_overlap : 20
start_eval : 100
no_fp_loss : False
min_track_len : 2
rounding_method : exact
no_interpolate : False
top_k_nns : 10
frames_per_graph : 512
frames_per_level : [2, 4, 8, 16, 32, 64, 128, 256, 512]
pruning_method : ['geometry', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005']
hicl_depth : 9
node_id_min_ratio : 0.5
depth_pretrain_iteration : 500
motion_max_length : [2, 4, 8, 16, 32, 64, 128, 256]
motion_pred_length : [2, 4, 8, 16, 32, 64, 128, 256]
interpolate_motion : True
linear_center_only : True
do_hicl_feats : False
hicl_feats_args : {}
reid_img_h : 256
reid_img_w : 128
zero_nodes : True
mpn_use_motion : [False, True, True, True, True, True, True, True, True]
mpn_use_reid_edge : [True, True, True, True, True, True, True, True, True]
mpn_use_pos_edge : [True, True, True, True, True, True, True, True, True]
old_experiments : [None]
mot_sub_folder : mot_files
gt_training_min_vis : 0.0
gt_assign_min_iou : 0.5
min_iou_bb_wiggling : 0.85
min_ids_to_drop_perc : 0
max_ids_to_drop_perc : 0.1
min_detects_to_drop_perc : 0
max_detects_to_drop_perc : 0.2
evaluation_graph_overlap_ratio : 0.5
solver_backend : pulp
symmetric_edges : True
verbose_iteration : 10
device : cuda:0
experiment_path : output/experiments/example_mot17_test_fastreid_msmt_BOT_R50_ibn_06-16_21:35:52.788910
VIDEO_COLUMNS : ['frame_path', 'frame', 'ped_id', 'bb_left', 'bb_top', 'bb_width', 'bb_height', 'bb_right', 'bb_bot']
TRACKING_OUT_COLS : ['frame', 'ped_id', 'bb_left', 'bb_top', 'bb_width', 'bb_height', 'conf', 'x', 'y', 'z']
do_motion : True
------------------
Backend Qt5Agg is interactive backend. Turning interactive mode on.
/home/asimzahid/anaconda3/envs/SUSHI/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning: 
NVIDIA GeForce RTX 3050 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the NVIDIA GeForce RTX 3050 Laptop GPU GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Detections for sequence MOT17-01-SDP need to be processed. Starting processing
REID ARCH??
FASTREID MODEL!!
[06/16 22:15:02 fastreid]: Rank of current process: 0. World size: 1
[06/16 22:15:04 fastreid]: Environment info:
----------------------  -----------------------------------------------------------------------------------
sys.platform            linux
Python                  3.8.11 (default, Aug  3 2021, 15:09:35) [GCC 7.5.0]
numpy                   1.23.1
fastreid                1.3 @/home/asimzahid/diddo/prototype/SUSHI/fast-reid/fastreid
FASTREID_ENV_MODULE     <not set>
PyTorch                 1.8.2 @/home/asimzahid/anaconda3/envs/SUSHI/lib/python3.8/site-packages/torch
PyTorch debug build     False
GPU available           True
GPU 0                   NVIDIA GeForce RTX 3050 Laptop GPU
CUDA_HOME               None
Pillow                  8.3.1
torchvision             0.9.2 @/home/asimzahid/anaconda3/envs/SUSHI/lib/python3.8/site-packages/torchvision
torchvision arch flags  /home/asimzahid/anaconda3/envs/SUSHI/lib/python3.8/site-packages/torchvision/_C.so
cv2                     4.7.0
----------------------  -----------------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.3-Product Build 20210617 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.2
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

[06/16 22:15:04 fastreid]: Command line arguments: Namespace(config_file='/home/asimzahid/diddo/prototype/SUSHI/fast-reid/configs/MSMT17/bagtricks_R50-ibn.yml', dist_url='tcp://127.0.0.1:50152', eval_only=False, machine_rank=0, num_gpus=1, num_machines=1, opts=['MODEL.WEIGHTS', '/home/asimzahid/diddo/prototype/SUSHI/fastreid-models/model_weights/msmt_bot_R50-ibn.pth'], resume=False)
[06/16 22:15:04 fastreid]: Contents of args.config_file=/home/asimzahid/diddo/prototype/SUSHI/fast-reid/configs/MSMT17/bagtricks_R50-ibn.yml:
_BASE_: ../Base-bagtricks.yml

MODEL:
  BACKBONE:
    WITH_IBN: True

DATASETS:
  NAMES: ("MSMT17",)
  TESTS: ("MSMT17",)

OUTPUT_DIR: logs/msmt17/bagtricks_R50-ibn


[06/16 22:15:04 fastreid]: Running with full config:
CUDNN_BENCHMARK: True
DATALOADER:
  NUM_INSTANCE: 4
  NUM_WORKERS: 8
  SAMPLER_TRAIN: NaiveIdentitySampler
  SET_WEIGHT: []
DATASETS:
  COMBINEALL: False
  NAMES: ('MSMT17',)
  TESTS: ('MSMT17',)
INPUT:
  AFFINE:
    ENABLED: False
  AUGMIX:
    ENABLED: False
    PROB: 0.0
  AUTOAUG:
    ENABLED: False
    PROB: 0.0
  CJ:
    BRIGHTNESS: 0.15
    CONTRAST: 0.15
    ENABLED: False
    HUE: 0.1
    PROB: 0.5
    SATURATION: 0.1
  CROP:
    ENABLED: False
    RATIO: [0.75, 1.3333333333333333]
    SCALE: [0.16, 1]
    SIZE: [224, 224]
  FLIP:
    ENABLED: True
    PROB: 0.5
  PADDING:
    ENABLED: True
    MODE: constant
    SIZE: 10
  REA:
    ENABLED: True
    PROB: 0.5
    VALUE: [123.675, 116.28, 103.53]
  RPT:
    ENABLED: False
    PROB: 0.5
  SIZE_TEST: [256, 128]
  SIZE_TRAIN: [256, 128]
KD:
  EMA:
    ENABLED: False
    MOMENTUM: 0.999
  MODEL_CONFIG: []
  MODEL_WEIGHTS: []
MODEL:
  BACKBONE:
    ATT_DROP_RATE: 0.0
    DEPTH: 50x
    DROP_PATH_RATIO: 0.1
    DROP_RATIO: 0.0
    FEAT_DIM: 2048
    LAST_STRIDE: 1
    NAME: build_resnet_backbone
    NORM: BN
    PRETRAIN: True
    PRETRAIN_PATH: 
    SIE_COE: 3.0
    STRIDE_SIZE: (16, 16)
    WITH_IBN: True
    WITH_NL: False
    WITH_SE: False
  DEVICE: cuda
  FREEZE_LAYERS: []
  HEADS:
    CLS_LAYER: Linear
    EMBEDDING_DIM: 0
    MARGIN: 0.0
    NAME: EmbeddingHead
    NECK_FEAT: before
    NORM: BN
    NUM_CLASSES: 0
    POOL_LAYER: GlobalAvgPool
    SCALE: 1
    WITH_BNNECK: True
  LOSSES:
    CE:
      ALPHA: 0.2
      EPSILON: 0.1
      SCALE: 1.0
    CIRCLE:
      GAMMA: 128
      MARGIN: 0.25
      SCALE: 1.0
    COSFACE:
      GAMMA: 128
      MARGIN: 0.25
      SCALE: 1.0
    FL:
      ALPHA: 0.25
      GAMMA: 2
      SCALE: 1.0
    NAME: ('CrossEntropyLoss', 'TripletLoss')
    TRI:
      HARD_MINING: True
      MARGIN: 0.3
      NORM_FEAT: False
      SCALE: 1.0
  META_ARCHITECTURE: Baseline
  PIXEL_MEAN: [123.675, 116.28, 103.53]
  PIXEL_STD: [58.395, 57.120000000000005, 57.375]
  QUEUE_SIZE: 8192
  WEIGHTS: /home/asimzahid/diddo/prototype/SUSHI/fastreid-models/model_weights/msmt_bot_R50-ibn.pth
OUTPUT_DIR: logs/msmt17/bagtricks_R50-ibn
SOLVER:
  AMP:
    ENABLED: True
  BASE_LR: 0.00035
  BIAS_LR_FACTOR: 1.0
  CHECKPOINT_PERIOD: 30
  CLIP_GRADIENTS:
    CLIP_TYPE: norm
    CLIP_VALUE: 5.0
    ENABLED: False
    NORM_TYPE: 2.0
  DELAY_EPOCHS: 0
  ETA_MIN_LR: 1e-07
  FREEZE_ITERS: 0
  GAMMA: 0.1
  HEADS_LR_FACTOR: 1.0
  IMS_PER_BATCH: 64
  MAX_EPOCH: 120
  MOMENTUM: 0.9
  NESTEROV: False
  OPT: Adam
  SCHED: MultiStepLR
  STEPS: [40, 90]
  WARMUP_FACTOR: 0.1
  WARMUP_ITERS: 2000
  WARMUP_METHOD: linear
  WEIGHT_DECAY: 0.0005
  WEIGHT_DECAY_BIAS: 0.0005
  WEIGHT_DECAY_NORM: 0.0005
TEST:
  AQE:
    ALPHA: 3.0
    ENABLED: False
    QE_K: 5
    QE_TIME: 1
  EVAL_PERIOD: 30
  FLIP:
    ENABLED: False
  IMS_PER_BATCH: 128
  METRIC: cosine
  PRECISE_BN:
    DATASET: Market1501
    ENABLED: False
    NUM_ITER: 300
  RERANK:
    ENABLED: False
    K1: 20
    K2: 6
    LAMBDA: 0.3
  ROC:
    ENABLED: False
[06/16 22:15:04 fastreid]: Full config saved to /home/asimzahid/diddo/prototype/SUSHI/logs/msmt17/bagtricks_R50-ibn/config.yaml
[06/16 22:15:04 fastreid.utils.env]: Using a generated random seed 5745143
[06/16 22:15:04 fastreid.engine.defaults]: Model:
Baseline(
  (backbone): ResNet(
    (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (bn1): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=True)
    (layer1): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
        (downsample): Sequential(
          (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (2): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
    )
    (layer2): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn2): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
        (downsample): Sequential(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (2): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (3): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
    )
    (layer3): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn2): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
        (downsample): Sequential(
          (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (2): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (3): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (4): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (5): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): IBN(
          (IN): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (BN): BatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
    )
    (layer4): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
        (downsample): Sequential(
          (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
      (2): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (se): Identity()
      )
    )
  )
  (heads): EmbeddingHead(
    (pool_layer): GlobalAvgPool(output_size=1)
    (bottleneck): Sequential(
      (0): BatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (cls_layer): Linear(num_classes=0, scale=1, margin=0.0)
  )
)
[06/16 22:15:07 fastreid.utils.checkpoint]: Loading checkpoint from /home/asimzahid/diddo/prototype/SUSHI/fastreid-models/model_weights/msmt_bot_R50-ibn.pth
[06/16 22:15:08 fastreid.utils.checkpoint]: Some model parameters or buffers are not found in the checkpoint:
  heads.weight
[06/16 22:15:08 fastreid.utils.checkpoint]: The checkpoint state_dict contains keys that are not used by the model:
  pixel_mean
  pixel_std
  heads.classifier.weight
/home/asimzahid/anaconda3/envs/SUSHI/lib/python3.8/site-packages/torchvision/transforms/transforms.py:257: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
  warnings.warn(
assets/MOT17/test/MOT17-01-SDP/det/byte065.txt
Computing embeddings for 4854 detections

There are 4 levels in paper while 9 levels in code.

Hi,
The SUSHI levels in paper are 4 like [5,25,75,150] and the frames range 150.
But in the released code, there are 9 levels and the frames range 512.
So, how many levels are the models trained indeedly in the paper?

Multi-GPU support

Hi,

Thank you so much for sharing your great work!

I would like to ask if you are planning to add multi-gpu support for training? I keep running out of memory for the default settings, so I think this would be a good option :)

Thanks again!
Joanna

bdd100k

Hi, first of all thank you for this repo.You mentioned bdd_to_motcha.py and bdd_submission.py in assets ,but i can not find them in scripts.

frame-by-frame GNN

Just curious. Are you working on any real-time frame-by-frame GNN approach?

Choose specific CUDA for training or inference

Hi, i have multi GPU and i want to choose specific GPU, how can i do that.

I only see --cuda parameter in training script

Cuda out of Memory

/truba/home/hbilgi/miniconda3/envs/py_cuda/lib/python3.8/site-packages/torch_geometric/deprecation.py:22: UserWarning: 'data.DataLoader' is deprecated, use 'loader.DataLoader' instead
warnings.warn(out)
Experiment ID: /truba/home/hbilgi/dev/SUSHI/outputs/experiments/example_mot17_training_fastreid_msmt_BOT_R50_ibn_09-05_14:24:29.837579
Experiment Mode: train
----- CONFIG -----
experiment_mode : train
run_id : example_mot17_training_fastreid_msmt_BOT_R50_ibn
connectivity : chunk
data_path : /truba/home/hbilgi/dev/thesis/mot_data
output_path : /truba/home/hbilgi/dev/SUSHI/outputs
seed : 0
cuda : True
num_workers : 4
train_splits : ['mot17-train-all']
val_splits : ['mot17-train-all']
test_splits : [None]
cval_seqs : mot17-train-all
det_file : det
save_cp : True
feature_embedding_model_path :
hicl_model_path :
load_train_ckpt : False
share_weights : all_but_first
node_level_embed : False
edge_level_embed : True
reid_arch : fastreid_msmt_BOT_R50_ibn
reid_sim_fn : l2
edge_sim_fn : l2
node_dim : 2048
l2_norm_reid : False
reid_embeddings_dir : reid_fastreid_msmt_BOT_R50_ibn
node_embeddings_dir : node_fastreid_msmt_BOT_R50_ibn
num_epoch : 250
num_batch : 8
lr : 0.0003
gamma : 1
weight_decay : 0.0001
augmentation : False
train_dataset_frame_overlap : 20
start_eval : 100
no_fp_loss : False
min_track_len : 2
rounding_method : exact
no_interpolate : False
top_k_nns : 10
frames_per_graph : 512
frames_per_level : [2, 4, 8, 16, 32, 64, 128, 256, 512]
pruning_method : ['geometry', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005', 'motion_005']
hicl_depth : 9
node_id_min_ratio : 0.5
depth_pretrain_iteration : 500
motion_max_length : [2, 4, 8, 16, 32, 64, 128, 256]
motion_pred_length : [2, 4, 8, 16, 32, 64, 128, 256]
interpolate_motion : True
linear_center_only : True
do_hicl_feats : False
hicl_feats_args : {}
reid_img_h : 256
reid_img_w : 128
zero_nodes : True
mpn_use_motion : [False, True, True, True, True, True, True, True, True]
mpn_use_reid_edge : [True, True, True, True, True, True, True, True, True]
mpn_use_pos_edge : [True, True, True, True, True, True, True, True, True]
old_experiments : [None]
mot_sub_folder : mot_files
gt_training_min_vis : 0.0
gt_assign_min_iou : 0.5
min_iou_bb_wiggling : 0.85
min_ids_to_drop_perc : 0
max_ids_to_drop_perc : 0.1
min_detects_to_drop_perc : 0
max_detects_to_drop_perc : 0.2
evaluation_graph_overlap_ratio : 0.5
solver_backend : pulp
symmetric_edges : True
verbose_iteration : 10
device : cuda:0
experiment_path : /truba/home/hbilgi/dev/SUSHI/outputs/experiments/example_mot17_training_fastreid_msmt_BOT_R50_ibn_09-05_14:24:29.837579
VIDEO_COLUMNS : ['frame_path', 'frame', 'ped_id', 'bb_left', 'bb_top', 'bb_width', 'bb_height', 'bb_right', 'bb_bot']
TRACKING_OUT_COLS : ['frame', 'ped_id', 'bb_left', 'bb_top', 'bb_width', 'bb_height', 'conf', 'x', 'y', 'z']
do_motion : True

---

Loading processed dets for sequence MOT17-02-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-02-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-02-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-02-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-02-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-02-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-04-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-04-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-04-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-04-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-04-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-04-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-05-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-05-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-05-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-05-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-05-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-05-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-09-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-09-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-09-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-09-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-09-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-09-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-10-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-10-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-10-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-10-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-10-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-10-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-11-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-11-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-11-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-11-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-11-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-11-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-13-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-13-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-13-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-13-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-13-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-13-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-02-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-02-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-02-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-02-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-02-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-02-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-04-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-04-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-04-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-04-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-04-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-04-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-05-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-05-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-05-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-05-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-05-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-05-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-09-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-09-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-09-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-09-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-09-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-09-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-10-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-10-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-10-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-10-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-10-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-10-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-11-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-11-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-11-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-11-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-11-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-11-DPM/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-13-SDP from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-13-SDP/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-13-FRCNN from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-13-FRCNN/processed_data/det/det.pkl
Loading processed dets for sequence MOT17-13-DPM from /truba/home/hbilgi/dev/thesis/mot_data/MOT17/train/MOT17-13-DPM/processed_data/det/det.pkl
Tracking MOT17-02-SDP
Tracking MOT17-02-FRCNN
Tracking MOT17-02-DPM
Tracking MOT17-04-SDP
Tracking MOT17-04-FRCNN
Tracking MOT17-04-DPM
Traceback (most recent call last):
File "main.py", line 34, in
hicl_tracker.train()
File "/truba/home/hbilgi/dev/SUSHI/src/tracker/hicl_tracker.py", line 456, in train
_, _ = self.track(self.val_dataset, output_path=osp.join(self.config.experiment_path, 'oracle'), mode='val', oracle=True)
File "/truba/home/hbilgi/dev/SUSHI/src/tracker/hicl_tracker.py", line 548, in track
hicl_graphs, _, logs_all = self.hicl_forward(hicl_graphs = hicl_graphs,
File "/truba/home/hbilgi/dev/SUSHI/src/tracker/hicl_tracker.py", line 333, in hicl_forward
curr_batch, _ = self._hicl_to_curr(hicl_graphs=hicl_graphs) # Create curr_graphs from hierarachical graphs
File "/truba/home/hbilgi/dev/SUSHI/src/tracker/hicl_tracker.py", line 185, in _hicl_to_curr
batch.pruning_score = compute_giou_fwrd_bwrd_motion_sim(batch, motion_pred)
File "/truba/home/hbilgi/dev/SUSHI/src/utils/motion_utils.py", line 90, in compute_giou_fwrd_bwrd_motion_sim
fwrd_start_box = fwrd_pred[row].gather(1, t_diff_fwrd[:,None, None].expand(-1, -1, 4))[:, 0]
RuntimeError: CUDA out of memory. Tried to allocate 121.04 GiB (GPU 0; 79.35 GiB total capacity; 2.57 GiB already allocated; 74.24 GiB free; 4.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hi, I tried to run the code with default configurations and started training MOT17 public detections. However, I get the error above. Is there a problem with my system or does this method require more than 120 GB of GPU memory.

Thanks.

How to obtain the file 'aplift.txt'?

Hi,

Thank you for sharing the code. I cann't find the file 'aplift.txt' in MOT17 or MOT20 dataset. How can I obtain this file?
Thank you.

RuntimeError: CUDA out of memory. Tried to allocate 121.04 GiB (GPU 0; 47.54 GiB total capacity; 2.57 GiB already allocated; 13.33 GiB free; 32.50 GiB reserved in total by PyTorch)

Hello,when I training use config like this

I use RTX 6000 GPU and torch=1.8.1+cu111

The error "CUDA out of memory" always occurs! It didn't work even if I set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:32

Do you know if it related to code contents or not? Or how can I fix it?

...
Tracking MOT17-02-SDP
Tracking MOT17-02-FRCNN
Tracking MOT17-02-DPM
Tracking MOT17-04-SDP
Tracking MOT17-04-FRCNN
Tracking MOT17-04-DPM
Traceback (most recent call last):
File "/home/hisham/anaconda3/envs/SUSHI/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/hisham/anaconda3/envs/SUSHI/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/hisham/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in
cli.main()
File "/home/hisham/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/hisham/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/home/hisham/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/hisham/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/hisham/.vscode-server/extensions/ms-python.python-2023.8.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "scripts/main.py", line 42, in
hicl_tracker.train()
File "/home/hisham/yuhaoli/code/SUSHI/./src/tracker/hicl_tracker.py", line 456, in train
_, _ = self.track(self.val_dataset, output_path=osp.join(self.config.experiment_path, 'oracle'), mode='val', oracle=True)
File "/home/hisham/yuhaoli/code/SUSHI/./src/tracker/hicl_tracker.py", line 548, in track
hicl_graphs, _, logs_all = self.hicl_forward(hicl_graphs = hicl_graphs,
File "/home/hisham/yuhaoli/code/SUSHI/./src/tracker/hicl_tracker.py", line 333, in hicl_forward
curr_batch, _ = self._hicl_to_curr(hicl_graphs=hicl_graphs) # Create curr_graphs from hierarachical graphs
File "/home/hisham/yuhaoli/code/SUSHI/./src/tracker/hicl_tracker.py", line 185, in _hicl_to_curr
batch.pruning_score = compute_giou_fwrd_bwrd_motion_sim(batch, motion_pred)
File "/home/hisham/yuhaoli/code/SUSHI/./src/utils/motion_utils.py", line 90, in compute_giou_fwrd_bwrd_motion_sim
fwrd_start_box = fwrd_pred[row].gather(1, t_diff_fwrd[:,None, None].expand(-1, -1, 4))[:, 0]
RuntimeError: CUDA out of memory. Tried to allocate 121.04 GiB (GPU 0; 47.54 GiB total capacity; 2.57 GiB already allocated; 13.33 GiB free; 32.50 GiB reserved in total by PyTorch)

support single object tracking?

i noticed that your evaluation all in multi-objects tracking datasets, i want to know this algorithm can use in single object tracking? please help me?

What is my pretrained model?

Thank you for sharing your code and work.

I trained the model with a 7-layer hierarchical structure using the provided code.

However, I'm unsure about the 'PRETRAINED_MODEL_PATH=your_pretrained_model_path' part in the test execution command.

Could you please provide guidance on this?

Below is the command I used for training with 7 layers. Should I also include this setting in the test execution command?

Oline or Offline

Dear author,

Is your algorithm for online multi-target tracking? If so, how to generate the tracking results of each frame in real time?

Looking forward to your reply, thanks!

Training Command Correction

According to the Readme, there's a mistake in the command line for the Training part. Instead of using "byte065", it should be corrected to "det".

--det_file byte065 -> --det_file det

Online version

Are you planing to release an online version of SUSHI. Runnable on video sources for example?

How can I reproduce the detection results

Hi,
Sorry to disturb you again.
Does 'a YOLOX detector [13] trained following [77].' means you directly consider the bytetrack results as the input of SUSHI?

Pretrained model weights for 7 level / 128 frames config

Hi,

I am trying to fine-tune the model on a different dataset but running into CUDA memory issues that others have reported. Switching to 7 levels / 128 frames configuration resolves the memory issue but prevents me from using the pretrained model. Would you happen to have pretrained model weights for 128 frame configuration that you can share?

Thank you.

how to adjust the dimension of the feature vector output by the ReID module?

Hello, I am applying SUSHI to a custom MOT dataset I created, with the dataset format following MOT17. Considering that the target size in my dataset is not large (I resized the ReID box images to 48*96), and there are minimal appearance changes during movement, I wanted to reduce the dimension of the feature vector output by the ReID module. So I modified the "--node_dim" in SUSHI-main/configs/config.py from 2048 to 256.

However, after running it, an error occurred:

RuntimeError
mat1 dim 1 must match mat2 dim 0
File "/workspace/lianghaochen/SUSHI-main/src/models/mpntrack.py", line 32, in forward return self.fc_layers(input) File "/workspace/lianghaochen/SUSHI-main/src/models/mpntrack.py", line 153, in forward out_node_feats = self.node_mlp(nodes_feats) File "/workspace/lianghaochen/SUSHI-main/src/models/mpntrack.py", line 368, in forward latent_edge_feats, latent_node_feats = self.encoder(edge_attr, x) File "/workspace/lianghaochen/SUSHI-main/src/models/hiclnet.py", line 59, in forward return self.layers[ix_layer](data, node_level_embed=node_level_embed, edge_level_embed=edge_level_embed) File "/workspace/lianghaochen/SUSHI-main/src/tracker/hicl_tracker.py", line 354, in hicl_forward outputs = self.model(curr_batch, curr_depth) # Forward pass for this specific depth File "/workspace/lianghaochen/SUSHI-main/src/tracker/hicl_tracker.py", line 271, in _train_epoch _, loss, logs = self.hicl_forward(hicl_graphs = hicl_graphs, File "/workspace/lianghaochen/SUSHI-main/src/tracker/hicl_tracker.py", line 479, in train epoch_train_logs = self._train_epoch() File "/workspace/lianghaochen/SUSHI-main/scripts/main.py", line 39, in hicl_tracker.train() RuntimeError: mat1 dim 1 must match mat2 dim 0

So, I modified node_in_dim in SUSHI-main/configs/mpntrack_cfg.yaml from 2048 to 256, but the error remains unchanged.
thank you

Zero_nodes option

Hi
I see that if zero_nodes option is used you ignore the precomputed node embeddings.
Is it correct? Is there a reason to use the pre computed node embeddings? Are they equal to the Reid embeddings?