xl-sr/thor

[BMVC'19] Tracking Holistic Object Representations

thor's Issues

video input from sjCam in IP Adress format

what is command to takevideo input from sjcam mounted on my drone to run siamMask teracking

Notice: please do not divide the jogging dataset and skating2 dataset of OTB2015 into two parts

do as the describe as title, or you will get error while executing the eval.py.

Why use Gram matrix, not Covariance matrix in allocation strategy?

About VOT setup protocol

I've noticed that in THOR/benchmark/vot.py, Line 28, you pass frame number into the initilaization function as follows:

state = tracker.setup(im, target_pos, target_sz, f)

Then in THOR/trackers/THOR_modules/wrapper.py, Line 56-63, you use the information whether f is 0 ("not f" in code), which indicates if the given frame is the first frame of a video(f=0) or the reinitialization frame after drift(f!=0), to determine whether the long term module(as well as its list of LT templates) should be cleaned&reinitialized or just updated(with the template information before drift).

        if not f or self._cfg.vanilla:
            self.lt_module = LT_Module(K=self._cfg.K_lt, template_keys=self.template_keys,
                                       lb=self._cfg.lb, lb_type=self._cfg.lb_type,
                                       verbose=self._cfg.verbose, viz=self._cfg.viz)
            self.lt_module.fill(temp)
        else:
            # reinitialize long term only at the beginning of the episode
            self.lt_module.update(temp, div_scale=0)

In my opinion, the latter would generate a data leak since trackers should not have the such information (f=0 or not) during (re)initialization phase according to VOT protocol.

Wish to hear your explanation. Thanks in advance.

custem object tracking, and old object initlization

Thanks for publishing the source code for the paper.
After reading the paper I know that I can put your work on top of any siamse network such as MASK or RPN++ ...

As the title implies I have two issues the first is how to initialize the object to be tracked.
I edited the file webcam_demo.py to run on video an existing video. I added an argument parser the video link and added to the function show web_cam the video passed and edited the line to

parser.add_argument('--video_name', type=str, required=True,help='Name of The video to be processed')



def show_webcam(tracker, args, mirror=False, viz=False):
    global initialize
    ## get path of video from user
    video_path = args.video_name
    #vs = cv2.VideoCapture(0) 
    vs = cv2.VideoCapture(video_path)

when I run the script the video open and starts without the ability to lock down the tracker on an object and the video plated without responding to the mouse too. any hints?
Moreover Iam trying to track a custom object. on the siamse rpn++ network. if I train the siamse network is their any additional step to add your work on top of siamse rpn++

Adding new Tracker

As mentioned in the paper your work can be applied on top of any tracker. In other words we can transform any offline tracker to online learning.
can you please provide some details on adding new tracker to the project, such as SIAMESE RPN++ or Trackers not based on SIAMESE.
Regards.

installing on jetson machines

i am trying to implement this tracker on the jetson nano and NX for a project im working on, usual opencv trackers dont work on GPU and i found this tracker to work on GPU, but when im trying to build the environment i have a lot of missing dependencies and im wondering if im even able to make this tracker work on jetson and if yes, how to do i deal with missing dependencies.

i am missing these dependencies(even though i've checked and i do have a few of them it just doesn't detect them for some reason):
Solving environment: failed
ResolvePackageNotFound:

• libstdcxx-ng==8.2.0=hdf63c60_1
• mkl_fft==1.0.12=py37ha843d7b_0
• six==1.12.0=py37_0
• blas==1.0=mkl
• mkl_random==1.0.2=py37hd81dba3_0
• numpy==1.16.4=py37h7e9f1db_0
• cudatoolkit==10.0.130=0
• pytorch==1.1.0=py3.7_cuda10.0.130_cudnn7.5.1_0
• zlib==1.2.11=h7b6447c_3
• pillow==6.0.0=py37h34e0f95_0
• ca-certificates==2019.1.23=0
• libpng==1.6.37=hbc83047_0
• libtiff==4.0.10=h2733197_2
• zstd==1.3.7=h0b5b093_0
• jpeg==9b=h024ee3a_2
• readline==7.0=h7b6447c_5
• cffi==1.12.3=py37h2e261b9_0
• torchvision==0.3.0=py37_cu10.0.130_1
• libedit==3.1.20181209=hc058e9b_0
• freetype==2.9.1=h8a8886c_1
• mkl==2019.4=243
• libgcc-ng==8.2.0=hdf63c60_1
• openssl==1.1.1c=h7b6447c_1
• ncurses==6.1=he6710b0_1
• libgfortran-ng==7.3.0=hdf63c60_0
• numpy-base==1.16.4=py37hde5b4d6_0
• sqlite==3.28.0=h7b6447c_0
• olefile==0.46=py37_0
• ninja==1.9.0=py37hfd86e86_0
• libffi==3.2.1=hd88cf55_4
• xz==5.2.4=h14c3975_4
• python==3.7.3=h0371630_0
• intel-openmp==2019.4=243

any help would be greatly appreciated

reinitialize tracker for same object

I want to reinitialize the tracker which completely scan the image for the same object. If object is lost then tracker stays at the position where it lost, but if enters from some another side then tracker doesn't identify it.

Is there any solution for that?

Installation of oxuva==0.0.1

Hello,

Quick questions:
The Oxuva package in the file: environment.yml is supposed to be the long term tracking benchmark? Or is there another package with the same name?
Also when executing the cmd pip install oxuva==0.0.1, I get an ERROR indicating that there is no matching distribution for oxuva. Do you have any clues on how to install the package and is the package mandatory for THOR?

Thank you for your time and consideration!

a little different with paper

hi, @xl-sr thanks for your clean codes!
I have reviewd the codes, and find when lower bound type is dynamic, the implement is a little different with formula in paper:

THOR/trackers/THOR_modules/modules.py

Lines 185 to 188 in 0a36b56

	elif self._lb_type=='dynamic':
	lb = self._lb - (1 - div_scale)
	lb = np.clip(lb, 0.0, 1.0)
	reject = (curr_sims[0] < lb*base_sim)

Well, it is fc*f1>l*G11 - gamma(sorry for no latex support in github) in paper, however, as above views, it is fc*f1>(l - gamma)*G11?

And another question is about idea. How do you come up with the fomula gamma in paper3.2 to represent diversity of STM templates?
Thank you!Regards!

Is it possible for multi-object tracking?

Thank you for your implementation.
Can this tracking track multiple object?
Thank you

Error on a CPU-only machine

RuntimeError: Attempting to deserialize object on a CUDA device but  torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use  torch.load with map_location='cpu' to map your storages to the CPU.  

According to the torch doc it is needed to provide a map_location when a gpu saved model is loaded on a cpu.

Changing line 76 in trackers/tracker.py solves the problem :

    model.load_state_dict(torch.load(model_path, map_location=('cpu'
                   if str(self.device) == 'cpu' else None)))

Integration with other trackers

Hi, is it possible to integrate THOR with other trackers like the improved SiamRPN++ and SiamMask_E?

About VOT setup issue (#2)

The former issue has been created here: (#6) and has been closed by the repo collaborator.

The original code, which uses the frame number (f) that is not provided to tracker's init function according to VOT official toolkit, is pasted as follows

        # initialize the long term module
        if not f or self._cfg.vanilla:
            self.lt_module = LT_Module(K=self._cfg.K_lt, template_keys=self.template_keys,
                                       lb=self._cfg.lb, lb_type=self._cfg.lb_type,
                                       verbose=self._cfg.verbose, viz=self._cfg.viz)
            self.lt_module.fill(temp)
        else:
            # reinitialize long term only at the beginning of the episode
            self.lt_module.update(temp, div_scale=0)

With this version of code, I successfully reproduced the reported result(EAO=0.416).

{'acc': 0.5818219747855319, 'robustness': 0.2341262408690766, 'eao': 0.4160014947564685, 'mean_fps': 94.913035446227}

However, as I commented the conditional branch which uses f:

        # initialize the long term module
        # if not f or self._cfg.vanilla:
        self.lt_module = LT_Module(K=self._cfg.K_lt, template_keys=self.template_keys,
                                   lb=self._cfg.lb, lb_type=self._cfg.lb_type,
                                   verbose=self._cfg.verbose, viz=self._cfg.viz)
        self.lt_module.fill(temp)
        # else:
            # reinitialize long term only at the beginning of the episode
            # self.lt_module.update(temp, div_scale=0)

, I got this result:

{'acc': 0.5797615533198762, 'robustness': 0.28563401386027343, 'eao': 0.368673347545505, 'mean_fps': 102.34960671562767}

Honestly, with an EAO of 0.369, the method did bring improvement w.r.t. to SiamRPN baseline(EAO=0.322), but not as much as reported while being removed the conditional branch that uses the frame number information. The latter indicates if an initialization is due to the beginning of new test video or a reinitialization after a drift. This information should not been provided to trackers according to VOT official toolkit.

I wish that a reasonable justification can be provided w.r.t. to this question before this issue is closed. Thanks.

get_test_data.sh is not found

https://github.com/xl-sr/THOR/blob/master/README.md#get-the-datasets

$ cd data
-bash: cd: data: No such file or directory

get_test_data.sh is not found.

Training code

How can I train a new dataset?