syscv/sam-pt

SAM-PT: Extending SAM to zero-shot video segmentation with point-based tracking.

sam-pt's Issues

want to get image mask, for use in bundlesdf project, just run demo line,

sam 社群 群聊 主要是解决开发的时候不交叉/重复 大家都玩sam 坐下来谈谈 里面有中科院自动化大佬 fastsam官方git群

HQ-SAM predictor issue

Hi, there's a problem when I try to use HQ-SAM-Light-VIT-T model

In sam_pt.yaml I set
- sam@sam_predictor.sam_model: samhq_light_vit_tiny

and when running demo I get

Traceback (most recent call last):
  File "C:\Users\Alina\Desktop\Workplace\sam-pt\demo\demo.py", line 55, in main
    logits, trajectories, visibilities, scores = run_inference(model, rgbs, query_points, target_hw)
  File "C:\Users\Alina\Desktop\Workplace\sam-pt\demo\demo.py", line 131, in run_inference
    outputs = model(video)
  File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 176, in forward
    query_masks = self.extract_query_masks(images, query_points)
  File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 324, in extract_query_masks
    _, query_masks_logits, _ = self._apply_sam_to_trajectories(
  File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 843, in _apply_sam_to_trajectories
    mask_frame_logits, iou_prediction_score = predict_mask(visible_point_coords, visible_point_labels)
  File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 783, in predict_mask
    _, _, low_res_masks = self.sam_predictor.predict_torch(
  File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\segment_anything\predictor.py", line 229, in predict_torch
    low_res_masks, iou_predictions = self.model.mask_decoder(
  File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)

TypeError: forward() missing 2 required positional arguments: 'hq_token_only' and 'interm_embeddings'

How to generate a mask file with a demo program

problem when installing requirements.txt

hello this work is this repository is very nice,
however i got some issue when i try to install the requirements.txt
its report this error
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for detectron2 Running setup.py clean for detectron2 Failed to build detectron2 ERROR: Could not build wheels for detectron2, which is required to install pyproject.toml-based projects

I think that the problem is on the git+https://github.com/facebookresearch/[email protected]
i try to install manually but its doesn't work either

I'd be grateful for your assistance if you could lend a hand.

Error running the demo

Hello, thanks for your work.
I installed the checkpoints and setup the environment but I am not able to run the demo.

[...]
$ cd sam-pt/
$ python -m demo.demo frames_path=~/Documents/CoTracker-SAM/samples/horsejump-stick/ query_points_path=null
Error while finding module specification for 'demo.demo' (ModuleNotFoundError: __path__ attribute not found on 'demo' while trying to find 'demo.demo')

Is there any Demo Notebook to test on One Image with Positive, negative and occluded points being passed

First of all thanks for this amazing work..

I am looking for a demo notebook/example to test it out on individual image/s, is there any available?

how much gpu and how much memory are required for training and testing?

Thanks for your work, how much gpu and how much memory are required for training and testing?

An error occurred while running the interactive demo, how to solve it

(sam) D:\computer_vision\sam-pt>python -m demo.demo frames_path='sam-pt/data/demo_data/demo/' query_points_path=null longest_side_length=1024 frame_stride=1 max_frames=-1
E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Setting seed to 72
model:
point_tracker:
target: sam_pt.point_tracker.pips.PipsPointTracker
checkpoint_path: ${hydra:runtime.cwd}/models/pips_ckpts/reference_model
stride: 4
s: 8
initial_next_frame_visibility_threshold: 0.9
sam_predictor:
sam_model:
image_encoder:
target: segment_anything.modeling.image_encoder.ImageEncoderViT
depth: 32
embed_dim: 1280
img_size: ${ ..image_size }
mlp_ratio: 4
norm_layer:
partial: true
target: torch.nn.LayerNorm
eps: 1.0e-06
num_heads: 16
patch_size: ${ ..vit_patch_size }
qkv_bias: true
use_rel_pos: true
global_attn_indexes:
- 7
- 15
- 23
- 31
window_size: 14
out_chans: ${ ..prompt_embed_dim }
prompt_encoder:
target: segment_anything.modeling.prompt_encoder.PromptEncoder
embed_dim: ${ ..prompt_embed_dim }
image_embedding_size:
- ${ ...image_embedding_size }
- ${ ...image_embedding_size }
input_image_size:
- ${ ...image_size }
- ${ ...image_size }
mask_in_chans: 16
mask_decoder:
target: segment_anything.modeling.mask_decoder.MaskDecoder
num_multimask_outputs: 3
transformer:
target: segment_anything.modeling.transformer.TwoWayTransformer
depth: 2
embedding_dim: ${ ...prompt_embed_dim }
mlp_dim: 2048
num_heads: 8
transformer_dim: ${ ..prompt_embed_dim }
iou_head_depth: 3
iou_head_hidden_dim: 256
target: sam_pt.modeling.sam.SamHydra
checkpoint: ${hydra:runtime.cwd}/models/sam_ckpts/sam_vit_h_4b8939.pth
prompt_embed_dim: 256
image_size: 1024
vit_patch_size: 16
image_embedding_size: 64
pixel_mean:
- 123.675
- 116.28
- 103.53
pixel_std:
- 58.395
- 57.12
- 57.375
target: segment_anything.predictor.SamPredictor
target: sam_pt.modeling.sam_pt.SamPt
sam_iou_threshold: 0.7
iterative_refinement_iterations: 12
positive_point_selection_method: kmedoids
negative_point_selection_method: mixed
positive_points_per_mask: -1
negative_points_per_mask: -1
add_other_objects_positive_points_as_negative_points: true
point_tracker_mask_batch_size: 5
use_patch_matching_filtering: false
patch_size: 3
patch_similarity_threshold: 0.01
use_point_reinit: false
reinit_point_tracker_horizon: 8
reinit_horizon: 8
reinit_variant: reinit-on-horizon-and-sync-masks
logging:
wandb:
entity: null
project: demo
tensorboard: true
log_code_path: ${hydra:runtime.cwd}/sam_pt
debug: false
exp_id: debug_72_2023.08.12_14.24.36
frames_path: sam-pt/data/demo_data/demo/
query_points_path: null
longest_side_length: 1024
frame_stride: 1
max_frames: -1
seed: 72
annot_size: 16
annot_line_width: 6

wandb: W&B API key is configured. Use wandb login --relogin to force relogin
wandb: Network error (SSLError), entering retry loop.
wandb: Network error (SSLError), entering retry loop.
[2023-08-12 14:25:45,470][urllib3.connectionpool][WARNING] - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/store/
[2023-08-12 14:25:46,266][urllib3.connectionpool][WARNING] - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/store/
Problem at: D:\computer_vision\sam-pt\demo\demo.py 75 setup_logging
Error executing job with overrides: ["frames_path='sam-pt/data/demo_data/demo/'", 'query_points_path=null', 'longest_side_length=1024', 'frame_stride=1', 'max_frames=-1']
wandb: ERROR Run initialization has timed out after 60.0 sec.
wandb: ERROR Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-
Traceback (most recent call last):
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\computer_vision\sam-pt\demo\demo.py", line 372, in
main()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self._return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "D:\computer_vision\sam-pt\demo\demo.py", line 42, in main
setup_logging(cfg)
File "D:\computer_vision\sam-pt\demo\demo.py", line 75, in setup_logging
wandb.init(
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_init.py", line 1169, in init
raise e
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_init.py", line 1150, in init
run = wi.init()
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_init.py", line 769, in init
raise error
wandb.errors.CommError: Run initialization has timed out after 60.0 sec.
Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-
[2023-08-12 14:25:47,013][urllib3.connectionpool][WARNING] - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/store/
[2023-08-12 14:25:48,531][urllib3.connectionpool][WARNING] - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/envelope/
[2023-08-12 14:25:49,424][urllib3.connectionpool][WARNING] - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/envelope/
[2023-08-12 14:25:50,050][urllib3.connectionpool][WARNING] - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/envelope/

Disappearing tracked objects

Hello,

First of all, congrats! Great work.

I run a few test and it has remarkable precision. Except one case: disappearing-reappearing objects, works really poorly (almost every case the tracking was lost) . I'm trying to track different, moving objects with moving camera. I haven't dug into your work in detail yet, but do you have any suggestions for that scenario? (more positive points for example?).
I tested TAM also and it's working really well in this case(because of XMem), but has worse precision with small, blurry objects.

Confused about the key to enter when running demo or test

Hi, thanks for your efforts and novel ideas.
However, as I am running demo or experiments, I was asked to either enter API or API key not found. (3 options)
Would you kindly provide more detailed tutorial video or instructions?
Thanks again for your time.

Fail to download UVO dense dataset

Hello, as describe in the dataset prepare part, the second google drive path is no longer available, how can i get the uvo dense and sparse dataset?
"
cd data

pip install gdown
gdown --no-check-certificate https://drive.google.com/uc?id=1AGu4BL-i_vDCMNtwsoSuo5wIyDVd5dRf
unzip UVOv1.0.zip
rm UVOv1.0.zip

Download the preprocessed videos

gdown --no-check-certificate --folder https://drive.google.com/drive/folders/1fOhEdHqrp_6D_tBsrR9hazDLYV2Sw1XC
unzip UVO_Videos/uvo_videos_dense.zip
unzip UVO_Videos/uvo_videos_sparse.zip
mv uvo_videos_dense/ UVOv1.0/
mv uvo_videos_sparse/ UVOv1.0/
rm -rf UVO_Videos/
rm -rf __MACOSX/"

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 2.00 GiB total capacity; 1.69 GiB already allocated; 0 bytes free; 1.74 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

Error executing job with overrides: ["frames_path='D:\computer_vision\sam-pt\data\demo_data\demo'", 'query_points_path=null', 'longest_side_length=1024', 'frame_stride=1', 'max_fr
ames=-1']
Traceback (most recent call last):
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\computer_vision\sam-pt\demo\demo.py", line 372, in
main()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self._return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "D:\computer_vision\sam-pt\demo\demo.py", line 50, in main
model = load_model(cfg, positive_points_per_mask, negative_points_per_mask)
File "D:\computer_vision\sam-pt\demo\demo.py", line 111, in load_model
return model.to("cuda" if torch.cuda.is_available() else "cpu").eval()
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 927, in to
return self._apply(convert)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
module._apply(fn)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
module._apply(fn)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 602, in _apply
param_applied = fn(param)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 925, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 2.00 GiB total capacity; 1.69 GiB already allocated; 0 bytes free; 1.74 GiB reserved in total by PyTorch) If reser
ved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1). Press Ctrl-C to abort syncing.
wandb: / 0.021 MB of 0.021 MB uploaded (0.000 MB deduped)
wandb: Run summary:
wandb: work_dir D:\computer_vision\s...
wandb:
wandb: View run debug_72_2023.08.12_15.26.10 at: https://wandb.ai/fangyuanguyue38/demo/runs/xodo3pvq
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: .\wandb\run-20230812_152614-xodo3pvq\logs
Exception in thread ChkStopThr:
Traceback (most recent call last):
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 278, in check_stop_status
self._loop_check_status(
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 216, in _loop_check_status
local_handle = request()
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\interface\interface.py", line 787, in deliver_stop_status
Fatal Python error: could not acquire lock for <_io.BufferedWriter name=''> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=000001B9DBB4C980)

Thread 0x0000524c (most recent call first):
File "E:\Anaconda3\envs\sam\lib\threading.py", line 302 in wait
File "E:\Anaconda3\envs\sam\lib\site-packages\sentry_sdk_queue.py", line 240 in get
File "E:\Anaconda3\envs\sam\lib\site-packages\sentry_sdk\worker.py", line 127 in _target
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870 in run
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap

Thread 0x000053b0 (most recent call first):
File "E:\Anaconda3\envs\sam\lib\site-packages\sentry_sdk\sessions.py", line 117 in _thread
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870 in run
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap

Thread 0x0000157c (most recent call first):
File "E:\Anaconda3\envs\sam\lib\threading.py", line 306 in wait
File "E:\Anaconda3\envs\sam\lib\threading.py", line 558 in wait
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\mailbox.py", line 126 in _wait
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\mailbox.py", line 130 in _get_and_clear
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\mailbox.py", line 283 in wait
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 224 in _loop_check_status
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 260 in check_network_status
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870 in run
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap

Thread 0x00003f8c (most recent call first):
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\redirect.py", line 640 in write
File "E:\Anaconda3\envs\sam\lib\threading.py", line 1202 in invoke_excepthook
File "E:\Anaconda3\envs\sam\lib\threading.py", line 934 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap

Current thread 0x00005b04 (most recent call first):

Error to load model when running the demo

Hi! I'm trying to test with the demo code. I've installed the detectron2 and segment-anything pkgs, but when running the demo, there seems to be something went run when it's loading the sam_pt model...

(sam) D:\computer_vision\sam-pt>python -m demo.demo \frames_path='sam-pt/data/demo_data/bees/' \query_points_path=null \ longest_side_length=1024 frame_stride=1 max_frames=-1 LexerNoViableAltException: \frames_path='sam-pt/data/demo_data/bees/' ^ See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Error running the Non-interactivte Demo

Thanks for your great work, but I meet some errors when I running the demo. I tried to change the PYTHONPATH and check the init.py files, but I still have no idea about how to solve it.

Error executing job with overrides: ['frames_path=${hydra:runtime.cwd}/data/demo_data/bees/', 'query_points_path=${hydra:runtime.cwd}/data/demo_data/query_points__bees.txt', 'longest_side_length=1024', 'frame_stride=1', 'max_frames=-1', 'model.use_point_reinit=true']
Traceback (most recent call last):
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 644, in _locate
    obj = getattr(obj, part)
AttributeError: module 'sam_pt.modeling' has no attribute 'sam_pt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 650, in _locate
    obj = import_module(mod)
  File "/conda/envs/sampt/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/sam-pt/sam_pt/modeling/sam_pt.py", line 15, in <module>
    from sam_pt.point_tracker import PointTracker, SuperGluePointTracker
  File "/sam-pt/sam_pt/point_tracker/__init__.py", line 7, in <module>
    from .cotracker import CoTrackerPointTracker
  File "/sam-pt/sam_pt/point_tracker/cotracker/__init__.py", line 1, in <module>
    from .tracker import CoTrackerPointTracker
  File "/sam-pt/sam_pt/point_tracker/cotracker/tracker.py", line 6, in <module>
    from cotracker.models.core.cotracker.cotracker import CoTracker
ImportError: cannot import name 'CoTracker' from 'cotracker.models.core.cotracker.cotracker' (/sam-pt/co-tracker/cotracker/models/core/cotracker/cotracker.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
    target = _locate(target)
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 658, in _locate
    raise ImportError(
ImportError: Error loading 'sam_pt.modeling.sam_pt.SamPt':
ImportError("cannot import name 'CoTracker' from 'cotracker.models.core.cotracker.cotracker' (/sam-pt/co-tracker/cotracker/models/core/cotracker/cotracker.py)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/conda/envs/sampt/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/conda/envs/sampt/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/sam-pt/demo/demo.py", line 371, in <module>
    main()
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/sam-pt/demo/demo.py", line 49, in main
    model = load_model(cfg, positive_points_per_mask, negative_points_per_mask)
  File "/sam-pt/demo/demo.py", line 109, in load_model
    model = instantiate(cfg.model)
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node
    _target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
  File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error locating target 'sam_pt.modeling.sam_pt.SamPt', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: model

Training on Custom Dataset

Can you provide a guideline for training on a custom dataset?

How to use mask sampling for tracing?

我阅读了你们的论文，其中提到了通过提供初始帧掩码的方式进行采样，然后进行跟踪的方法。我想要知道你们提供了相关的测试代码吗？如果有提供就太好了
I read your paper which mentions a method of sampling by providing an initial frame mask and then tracking it. I would like to know if you have provided any test code for this? It would be great if you do.

Add feature points in real time

Thanks for your great work！
But when i run sam-pt,I found that sometimes it divided the ground into ranges.
Have you tried adding feature points in subsequent frames?

AssertionError: No frames found in /content/sam-pt/data/demo_data/bees/

Different number of positive points

Hi,
Thanks for your work again.

I have two questions about the code.

1. For the non-interactive demo, passing a different number of positive points for each object is not possible. Is there any way that I can pass the different number of positive points?

2. The other question is about "mask_n_timestep" which you mentioned here

Can you explain how does it work?

I thought "mask_n_timestep" means the frame number where an object appears in the frame for the first time.

But I experimented on the same sequence and the results were like these:

If we only have three objects in the "Query Points" text file and they all appear in the first frame:

If we have the same objects in the "Query Points" text file + three other points that appear in the next frames for the first time (Like frames 234, 520, 640)] :

If we have the same objects in the "Query Points" text file + only one other point that appears in the next frames for the first time (Like frames 234)] :

I don't why there are some random points appear in the first frame although I mentioned the right frame number for them and I followed your format.

Thanks for your answers in advance.

How to get evaluation results for DAVIS2016, YouTube2018 and MOSE?

Thank you for your great work! Would you mind sharing the code to get the evaluation results for DAVIS2016, YouTube2018 and MOSE? Thanks a lot!

Tracking ability with a small number of frames

Hi again :)
Your work is truly amazing, although, I wonder If model won't be able to work correctly with, for example, 3-5 frame video?

For now, I have an issue when for short sequences I get zero masks and trajectories, scores are -inf.

Could It be a consequence of no object motion?

Error in call to target 'sam_pt.point_tracker.pips.tracker.PipsPointTracker'

Hi! I'm trying to replicate the demo, and I'm running into an issue. I have CUDA 11.8 installed. TensorRT is also installed and can be loaded into python, so I'm not sure what the issue is there.

I tried fresh installs of the environment, CUDA, Tensorflow, Pythorch and TensorRT.

This is the relevant log:

Loading PIPS model from /mnt/sam-pt/models/pips_ckpts/reference_model

reading ckpt from /mnt/sam-pt/models/pips_ckpts/reference_model

Error executing job with overrides: ['frames_path=${hydra:runtime.cwd}/data/demo_data/bees/', 'query_points_path=null', 'longest_side_length=1024', 'frame_stride=1', 'max_frames=-1']

Error in call to target 'sam_pt.point_tracker.pips.tracker.PipsPointTracker':

AssertionError()

full_key: model.point_tracker

sam_pt.yaml config issue

I've already change the 'use_point_reinit' to true, but it doesn't change at all,still remain false.

About training spec

Hi,

Has this model been implemented without any training process?
And is the only required training process the training of HQ-SAM independently?
Alternatively, please provide the GPT/memory usage details for training this model.

Thank you.

Toolbar not rendering in on MacOS Ventura via X forwarding

Can someone post a screenshot of what the toolbar is supposed to contain?

Depth information

can I use it for 3d. can you please share some information

Error when run interactive demo

Hello, thanks for your work.
I installed the checkpoints and set up the environment but cannot run the demo.
python -m demo.demo frames_path='/home/bbnclxy/ext16t/lyx-star/sam-pt-main/data/demo_data/bees/' query_points_path=null longest_side_length=1024 frame_stride=1 max_frames=-1
it reports error:

I would appreciate it if you could help me.

Cab sam-pt automatically track?

Hi, Can I ask if sam-pt can automatically track?

detectron2 fails installation

the latest detectron2 requires pytorch>=1.18 but since the instructions tells us to do conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch it always fails.

Installing detectron2 with a torch>=1.8 fails the demo. Instead using python -m pip install detectron2 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html to install detectron2 solves this issue.

Real-time integration

You are not mentioning any time performance results in your paper. Can we use it for segmenting real time ? Are there any fps rates results of your runs or how much time does it takes per frame with your setup ?

Thanks for your great work,
best wishes,

Code Release

Do you plan to release the code?
if so, do you know when?

Thanks, this work looks great!

Model creation outside repo directory

Hi, I need to create model outside sam-pt directory

For now I get the following error
error locating target sam_pt.modelling.sam_pt.SamPt

How do I change _target_ in config in order this to work?