huanglianghua/siamfc-pytorch

A clean PyTorch implementation of SiamFC tracking/training, evaluated on 7 datasets.

SiamFC - PyTorch

Highlights of this update:

• Higher scores with more stable training performance.
• Faster training (~11 minutes to train one epoch on GOT-10k on a single GPU).
• Added MIT LICENSE.
• Organized code.
• Uploaded pretrained weights. (Google Drive or Baidu Yun (password: wbek))

A clean PyTorch implementation of SiamFC tracker described in paper Fully-Convolutional Siamese Networks for Object Tracking. The code is evaluated on 7 tracking datasets (OTB (2013/2015), VOT (2018), DTB70, TColor128, NfS and UAV123), using the GOT-10k toolkit.

Performance (the scores are not updated yet)

GOT-10k

Dataset	AO	SR0.50	SR0.75
GOT-10k	0.355	0.390	0.118

The scores are comparable with state-of-the-art results on GOT-10k leaderboard.

OTB / UAV123 / DTB70 / TColor128 / NfS

Dataset	Success Score	Precision Score
OTB2013	0.589	0.781
OTB2015	0.578	0.765
UAV123	0.523	0.731
UAV20L	0.423	0.572
DTB70	0.493	0.731
TColor128	0.510	0.691
NfS (30 fps)	-	-
NfS (240 fps)	0.520	0.624

VOT2018

Dataset	Accuracy	Robustness (unnormalized)
VOT2018	0.502	37.25

Installation

Install Anaconda, then install dependencies:

# install PyTorch >= 1.0
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
# intall OpenCV using menpo channel (otherwise the read data could be inaccurate)
conda install -c menpo opencv
# install GOT-10k toolkit
pip install got10k

GOT-10k toolkit is a visual tracking toolkit that implements evaluation metrics and tracking pipelines for 9 popular tracking datasets.

Training the tracker

1. Setup the training dataset in tools/train.py. Default is the GOT-10k dataset located at ~/data/GOT-10k.

2. Run:

python tools/train.py

Evaluate the tracker

1. Setup the tracking dataset in tools/test.py. Default is the OTB dataset located at ~/data/OTB.

2. Setup the checkpoint path of your pretrained model. Default is pretrained/siamfc_alexnet_e50.pth.

3. Run:

python tools/test.py

Running the demo

1. Setup the sequence path in tools/demo.py. Default is ~/data/OTB/Crossing.

2. Setup the checkpoint path of your pretrained model. Default is pretrained/siamfc_alexnet_e50.pth.

3. Run:

python tools/demo.py

siamfc-pytorch's Issues

more details about requirements.txt

python 3.7
shapely 1.7.1
I can run train.py
when i run test.py
File "G:\practice\SiamFC\siamfc-pytorch-master\tools\test.py", line 8, in
from got10k.experiments import ExperimentGOT10k

File "E:\anaconda\1\lib\site-packages\got10k\experiments_init_.py", line 3, in
from .got10k import ExperimentGOT10k

File "E:\anaconda\1\lib\site-packages\got10k\experiments\got10k.py", line 14, in
from ..utils.metrics import rect_iou

File "E:\anaconda\1\lib\site-packages\got10k\utils\metrics.py", line 4, in
from shapely.geometry import box, Polygon

File "E:\anaconda\1\lib\site-packages\shapely\geometry_init_.py", line 23, in
import shapely.speedups

File "E:\anaconda\1\lib\site-packages\shapely\speedups_init_.py", line 99, in
enable()

File "E:\anaconda\1\lib\site-packages\shapely\speedups_init_.py", line 74, in enable
affine_transform.doc = shapely.affinity.affine_transform.doc

AttributeError: module 'shapely' has no attribute 'affinity'

CPU usage when train

hello, thanks your clean code.
But i found a strange phenomenon that is my cpu usage is very high(almost 100% usage per kernel spied by linux system monitor) so that it run smoothly and stop for a while and continue run...
Have you also experienced this phenomenon when training?What may cause this phenomenon in code?

Error(s) in loading state_dict for SiamFC

Error happened when loading state_dict from the pretrained model (download from google drive) for SiamFC.

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    tracker = TrackerSiamFC(net_path=net_path)
  File "F:\code\MOT\DRL_MOT\CDRL\siamfc-pytorch\siamfc.py", line 84, in __init__
    net_path, map_location=lambda storage, loc: storage))
  File "D:\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 721, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for SiamFC:
        Unexpected key(s) in state_dict: "feature.1.num_batches_tracked", "feature.5.num_batches_tracked", "feature.9.num_batches_tracked", "feature.12.num_batches_tracked".

question about the hann_window

您好，请问在resize之后的响应图上加余弦窗，为什么不是乘法，而是加法呢？？
response = (1 - self.cfg.window_influence) * response +
self.cfg.window_influence * self.hann_window # 添加余弦窗
这里用的加法，为什么不是乘法？？

question about the true label

In the siamfc paper, the true label is composed of 1 and -1, but I think the true label here uses 1 and 0.
I'm wondering whether this can cause any performance drop or not.
Thanks in advance :)

Question about running speed

Hello
        The original author gave a running speed of 86fps (GPU, search factor of 3). I ran your code at 4fps (cpu, VOT15),
       Is the gap so big?

Questions about test accuracy

I used your code to train with VID data set, and then tested the trained model. I found that the test results on OTB50 and VOT2018 were quite different from the results you gave, my test results:
OTB:SR:0.517, pr:0.698
VOT2018:
"speed_fps": 74.3267303229599,
"robustness": 60.76630229928382,
"accuracy": 0.4865348029482427

tset on vot2018

When i test on vot2018

root_dir = os.path.join('G:/practice/data'+'/VOT2018')
e = ExperimentVOT(root_dir,version=2018,experiments='unsupervised')

e.run(tracker)

e.report([tracker.name])

I got

Found results, skipping tiger
--Sequence 57/59: traffic
Found results, skipping traffic
--Sequence 58/59: wiper
Found results, skipping wiper
--Sequence 59/59: zebrafish1
Found results, skipping zebrafish1
Evaluating triplet
Traceback (most recent call last):

File "G:\practice\TripleFC\tools\test.py", line 37, in
e.report([tracker.name])

File "E:\anaconda\1\envs\siamfc\lib\site-packages\got10k\experiments\vot.py", line 361, in report
ious[seq_name][:len(seq_ious), :] = seq_ious

ValueError: could not broadcast input array from shape (0) into shape (0,325)

How to solve this problem？I don't know where I went wrong.

got10K训练到一半时报错

再用got10K训练到11个epoch的时候，出现了opencv中torgb的错误，可以告诉我是数据集的哪里出了问题吗

install opencv

in README.md: intall OpenCV using menpo channel, can i just install without add any -c parameter? it must install by menpo channel?

question about run demo.py

when i run demo.py
Traceback (most recent call last):
File "demo.py", line 7, in
from siamfc import TrackerSiamFC
ModuleNotFoundError: No module named 'siamfc'

Minor bug in crops.py?

Hi,

I might be completely wrong about this but in line 12-13 in crops.py it reads:

xright_pad = max(0, int(round(pos_x + c)) - frame_sz[1])
ybottom_pad = max(0, int(round(pos_y + c)) - frame_sz[0])

From my understanding, the first variable is checking if the crop exceeds the right-hand side limit of the frame (x direction) and second one for bottom (y direction), so shouldn't it instead be:

xright_pad = max(0, int(round(pos_x + c)) - frame_sz[0])
ybottom_pad = max(0, int(round(pos_y + c)) - frame_sz[1])

question about how to prepare dataset

Hi, I'm a newer in tracking. So, how to prepare training datasets? just download and put it in ./data?

VOT2018测评结果

Huang Lianghua 你好！
我将你写的 pytorch 版本 SiamFC 和 bilylee 的 tensorflow 版本SiamFC用 VOT 的 toolkit 进行了测评（VOT2018数据集），结果如下（截取部分）：


表中：

• SiamFC 是VOT官方提供的数据。
• SiamFC_tensorflow 是bilylee代码的测评结果。
• siamfc_pytorch_master_opencv / siamfc_pytorch_master_PIL 是你的代码，采用的是仓库中提供的预训练权重，opencv / PIL 代表的是测试时读取图片的方法。

想请教一下：

1. 为何 opencv 读取方式（转RGB）和 PIL.Image 读取方式对性能有影响？
2. Readme.md 中显示的 Accuracy: 0.502 是不是用got10k测评得到的？为何VOT toolkit达不到这种效果呢？
3. 在更大的数据集上训练之后，性能竟然比tensorflow版本稍差，可能是什么原因呢？

Add a license

Please consider adding a license file to this repository so that others know under what conditions this code may be used.

关于运行demo.py

为什么我运行demo.py的时候会出现ModuleNotFoundError: No module named 'siamfc'，要怎么解决，求问。

GPU version of the SiamFC

Hi,

I found out that all the model was conducted under cpu version.

The path error.

I follow the instruction to set up the environment.
When I run the command python tools/train.py
I got some error:
from siamfc import TrackerSiamFC
ImportError: No module named 'siamfc'

It seems the absolute path is set inappropriate.

Would you please release the demo model (siamfc_alexnet_e50.pth)?

It is great of you to release this code.
However, it is really time-consuming to download the GOT-10k dataset and train from scratch. Would you please release the demo model, i.e., the siamfc_alexnet_e50.pth model, for a quick start?

Much appreciate.

训练问题

请问您在训练GOT-10k全部序列的时候，是使用的单卡训练的吗？

安装menpo的OpenCV：failed

`(base) csd@amax:~$ conda install -c menpo opencv
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

`

验证损失问题

请问有没有遇到过验证损失一开始减小后来越来越大的问题？

Segmentation fault in running train.py

Hi huanglianghua,
Thanks for your brilliant work.
I have encountered a 'Segmentation fault' bug when I ran the train.py.
There are no traceback. no any other information, only segmentation fault.
Could you please help me with this issue? Many thanks

Question on image augmentation during training.

Dear sir, when you are doing image augmentation during training. You define transform_z and transform_x separately so the exemplar image z and search image x do not go through the same augmentation, which can be a big problem. Could you please tell me what am I missing?

can you share any user friendly web download link of dataset, google drive best

Hi Mr Huang
Very thanks for the marvel and light reproduce work in pytorch ,a good job.

And a bit suggestion , could you share some dataset's download link by google drive ? However, the office download link was too slow to irk myself .

Can you fill in miss blank by google_drive_link by manual ?

Dataset	Google Drive	Baidu cloud	office download page or link
GOT-10k	copyright/ private link	copyright/private link	obtain
DukeMTMC-VideoReID	link	optional	office
DukeMTMC-reID	link / ref	optional	office
dukemtmc i don't known	link / ref	optional	github_open_reid
OTB (2013/2015)	miss	optional	office
VOT (2018)	miss	optional	office
DTB70	box	pan	office
TColor128	miss	optional	office
NfS	goto_office	goto_office	office
UAV123	link / ref	optional	office
ILSVRC2017_VID	miss	optional	office
ILSVRC2015_VID	miss	optional	office
Dataset-NUS-PRO	miss	link / ref	office

modify a little in tools/demo.py will work on

need to modify this line: anno = np.loadtxt(seq_dir + 'groundtruth_rect.txt') to anno = np.loadtxt(seq_dir + 'groundtruth_rect.txt', delimiter=','), otherwise it will raise error like this:ValueError: could not convert string to float

the got-10k dataset can not be downloaded

when I dowanload the got-10k dataset in http://got-10k.aitestunion.com/downloads.It shows the error that 500 Internal Server Error.So I can not download this dataset. can you help me solve this problem?thank you！

关于label生成的疑问

您好，对于label生成部分的代码，存在一些疑问，询问一下。
对于siamfc.py中的函数_create_labels(self, size)部分，存在如下代码：

 r_pos = self.cfg.r_pos / self.cfg.total_stride
 r_neg = self.cfg.r_neg / self.cfg.total_stride
 labels = logistic_labels(x, y, r_pos, r_neg)

其中，在该代码的默认配置中：cfg.r_pos = 16,cfg.r_neg = 0,cfg.total_stride = 8。

我的疑问：

• cfg.total_stride代表什么含义？（依据原文中所表述的输入图片尺寸和输出特征图尺寸，找不到结果为8的比例关系）
• cfg.r_pos和cfg.r_neg代表什么含义？这两个变量的数值定义有什么依据么，可以自行定义得更大么？
• 经计算后的r_pos和r_neg代表什么含义？
• 注：我可以理解logistic_labels()函数的意思

NEGATIVE TRAINING EXAMPLES

Hi, I have a question about the training process.
I downloaded the GOT-10K dataset, but in the subset 'train', there are only positive samples ( which contain the target object). So, in the training, we use only positive x-z pairs but should not we use also negative pairs (which do not contain the target object) for training the backbone model ?

Link to pretrained networks broken

It seems that the primary source to the pretrained networks is unavailable (https://bit.ly/cfnet_networks)
If you have the weights, could you please upload them elsewhere?

Thanks!

How do I get it to train using ImageNetVID?

Change "seqs = GOT10k(root_dir, subset='train', return_meta=True)" in the original tool/train code to "seqs = ImageNetVID(root_dir, subset =('train', 'val'))", but "ValueError: num_samples should be a positive integer value, but got num_samples=0".

Which dataset is used by the pretrained model

Is the pretrained model trained on the VID dataset?

about number of prediected seqs

Why I run test.py to evaluate OTB2015 result, but it just generate 74 sequences result, it should be 100 sequences result in OTB2015, right? So where's the problem?

Any help is highly, appreciated
Best regards

Why is this port of SiamFC so slow? I get about 10fps with 1080Ti but author of the original paper claims 50-100 fps?

about the results

Thank you for the job! I have got the results, can you show the results on you machine? We want to compare them. Thank you for your reply.

Question about data transformer

Hi,

Thank you for sharing the great work.
I have a question about the data transformer.

self.transforms_x = Compose([
RandomStretch(),
CenterCrop(instance_sz - 8),
RandomCrop(instance_sz - 2 * 8),
ToTensor()])

Why the size applied on the search image is "instance_sz - 2 * 8" which would be 255- 16 ??
Why not is just 255?
If you know the answer please share with me.
Thank you in advance.

Best Regards,
Lai

hello,Dr.huang,can you share the "pretrained/siamfc_new" file in run_training.py

I want to use your initial weight to train the net.

Hope you share the BaiduYun link.

Thank you very much!

0.467 success rate and 0.603 precision for OTB2013...

Can you offer the pretrained model?

Question about running speed

Dear sir, when I evaluate the tracker using the got10k tool on a single GPU, I got only 20 fps using the AlexNetV1 as backbone while 4 fps when using AlexNetV2, which is far from the reported fps. Could you please tell me why this is happening? Is it because my GPU is not good enough or it has something to do with the implementation of the codes. Thanks!

关于数据预处理的疑问

您好，感谢您提供的GOT数据集！我有一些疑问，想请教一下。
在pairwise.py文件中存在如下代码：

class Pairwise(Dataset):

    def __init__(self, seq_dataset, **kargs):
        super(Pairwise, self).__init__()
        self.cfg = self.parse_args(**kargs)

        self.seq_dataset = seq_dataset
        self.indices = np.random.permutation(len(seq_dataset))
        # augmentation for exemplar and instance images
        self.transform_z = Compose([
            RandomStretch(max_stretch=0.05),
            CenterCrop(self.cfg.instance_sz - 8),
            RandomCrop(self.cfg.instance_sz - 2 * 8),
            CenterCrop(self.cfg.exemplar_sz),
            ToTensor()])
        self.transform_x = Compose([
            RandomStretch(max_stretch=0.05),
            CenterCrop(self.cfg.instance_sz - 8),
            RandomCrop(self.cfg.instance_sz - 2 * 8),
            ToTensor()])

疑问：

• CenterCrop(self.cfg.instance_sz - 8)和RandomCrop(self.cfg.instance_sz - 2 * 8)所代表的含义是什么？
• 对于transform_z，为什么需要 CenterCrop(self.cfg.instance_sz - 8)以及RandomCrop(self.cfg.instance_sz - 2 * 8)?
• 对于transform_x，为什么最终不需要 CenterCrop(self.cfg.instance_sz )？

我的理解：

• 对于transform_z 是不是应该只保留CenterCrop(self.cfg.exemplar_sz)？
• 对于transform_x 是不是应该只保留CenterCrop(self.cfg.instance_sz)？
• 这样的话，得到的z和x，才能最准确的和label相对应?

运行train.py出错

_pickle.PicklingError: Can't pickle

Hi~

I wanted to retrain the model. During training, I face the following issue
_pickle.PicklingError: Can't pickle <class 'pairwise.GenericDict'>: attribute lookup GenericDict on pairwise failed

I rerun the train.py for GOT-10k and OTB and error occur at line # 45.

Regards

About the results

About train loss

train loss always 0.2 and 0.3 ,however,test can get a good result,why loss?

Question about out_scale=0.001

Hi @huanglianghua
I have a question about the hyperparameter out_scale in heads.py
Why this parameter is necessary? I can't find this parameter in the official code (Matlab ver) and in the paper.
Hope to find some inspirations.
Anyway, Thank you for your great work.

about fps of demo

when i try demo.py,my video has a very low fps

关于OTB数据测试速度的疑问

一开始的几个序列测试速度还很快，但是越往后速度越慢。

运行demo.py问题

运行一直卡在anno = np.loadtxt(seq_dir + 'groundtruth_rect.txt') 提示ValueError: could not convert string to float: '205,151,17,50'

Hanning Window

Hi, I am looking into the TrackerSiamFC class. In the init function, you created a hanning window. I am confused about the usage of the hanning window, do you mind explaining a little?