abewley/sort

Simple, online, and realtime tracking of multiple objects in a video sequence.

SORT

A simple online and realtime tracking algorithm for 2D multiple object tracking in video sequences. See an example video here.

By Alex Bewley

Introduction

SORT is a barebones implementation of a visual multiple object tracking framework based on rudimentary data association and state estimation techniques. It is designed for online tracking applications where only past and current frames are available and the method produces object identities on the fly. While this minimalistic tracker doesn't handle occlusion or re-entering objects its purpose is to serve as a baseline and testbed for the development of future trackers.

SORT was initially described in this paper. At the time of the initial publication, SORT was ranked the best open source multiple object tracker on the MOT benchmark.

Note: A significant proportion of SORT's accuracy is attributed to the detections. For your convenience, this repo also contains Faster RCNN detections for the MOT benchmark sequences in the benchmark format. To run the detector yourself please see the original Faster RCNN project or the python reimplementation of py-faster-rcnn by Ross Girshick.

Also see: A new and improved version of SORT with a Deep Association Metric implemented in tensorflow is available at https://github.com/nwojke/deep_sort .

License

SORT is released under the GPL License (refer to the LICENSE file for details) to promote the open use of the tracker and future improvements. If you require a permissive license contact Alex ([email protected]).

Citing SORT

If you find this repo useful in your research, please consider citing:

@inproceedings{Bewley2016_sort,
  author={Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben},
  booktitle={2016 IEEE International Conference on Image Processing (ICIP)},
  title={Simple online and realtime tracking},
  year={2016},
  pages={3464-3468},
  keywords={Benchmark testing;Complexity theory;Detectors;Kalman filters;Target tracking;Visualization;Computer Vision;Data Association;Detection;Multiple Object Tracking},
  doi={10.1109/ICIP.2016.7533003}
}

Dependencies:

To install required dependencies run:

$ pip install -r requirements.txt

Demo:

To run the tracker with the provided detections:

$ cd path/to/sort
$ python sort.py

To display the results you need to:

1. Download the 2D MOT 2015 benchmark dataset
2. Create a symbolic link to the dataset

$ ln -s /path/to/MOT2015_challenge/data/2DMOT2015 mot_benchmark

0. Run the demo with the --display flag

$ python sort.py --display

Main Results

Using the MOT challenge devkit the method produces the following results (as described in the paper).

Sequence	Rcll	Prcn	FAR	GT MT PT ML	FP FN IDs FM	MOTA MOTP MOTAL
TUD-Campus	68.5	94.3	0.21	8 6 2 0	15 113 6 9	62.7 73.7 64.1
ETH-Sunnyday	77.5	81.9	0.90	30 11 16 3	319 418 22 54	59.1 74.4 60.3
ETH-Pedcross2	51.9	90.8	0.39	133 17 60 56	330 3014 77 103	45.4 74.8 46.6
ADL-Rundle-8	44.3	75.8	1.47	28 6 16 6	959 3781 103 211	28.6 71.1 30.1
Venice-2	42.5	64.8	2.75	26 7 9 10	1650 4109 57 106	18.6 73.4 19.3
KITTI-17	67.1	92.3	0.26	9 1 8 0	38 225 9 16	60.2 72.3 61.3
Overall	49.5	77.5	1.24	234 48 111 75	3311 11660 274 499	34.0 73.3 35.1

Using SORT in your own project

Below is the gist of how to instantiate and update SORT. See the 'main' section of sort.py for a complete example.

from sort import *

#create instance of SORT
mot_tracker = Sort() 

# get detections
...

# update SORT
track_bbs_ids = mot_tracker.update(detections)

# track_bbs_ids is a np array where each row contains a valid bounding box and track_id (last column)
...

🔗 See also

• DeepSORT - The successor of SORT with a Deep Association Metric used injecting appearance information to improve the association in difficult scenarios such as occlusions and fast moving objects.
• Local Metrics for Multi-Object Tracking - A framework to help better measure and understand how well your tracker performs at association across time as opposed to the standard metrics (MOTA, HOTA) which as heavilty biased towards raw detection performance.
• Human Scene Transformer - A recent repository for trajectory forecasting, a future direction beyond tracking (pun intended).

sort's Issues

parameters of Kalman filter

Hi,
how do you set the parameters of Kalman filter? is it depends on the data or we can use the parameters as you set?

The tracking bboxs are not exactly the location of the object,some lag exists, I suppose it's the problem of Kalman Filter parameters? Am i right?

A way to evaluate tracker performance

Hello, your algorithm seems to work great! However, I'm seeking to evaluate it using usual metrics (MOTA, MOTP). Are you planning on adding such feature or can refer to some git repository which could be of a help?

Trackers not deleting:

Hi abewley ,

I was trying to run SORT on videos.But I found that there are some instances when trackers are not getting deleted(Continuing their linear trajectory) when no detection is present for few frames.
Code:



Kalman filter update function:

Results seems something like this:

I have used max_age = 2 , min_hits = 5.
It will be a a great help if you could help me out of this issue.

Thanks in advance
Aayush

Tracking with frame skips

Hello, I have some issues regarding tracking with skip in detections. I'm trying to get frameskip to work with this tracker.
How I expect it to work:

1. Initialise tracking by feeding it with x detections with no frame skip (e.g. 5 frames).
2. Now only use detection every q frame (q frame could be 3 frames)
3. For every frame feed tracker detection list, if no detection is run feed an empty list. According to requirement for Sort function.

According to Sort(object) comment: Requires: this method must be called once for each frame even with empty detections.

How it works:
Tracks fine in the initialization, but when I feed the tracker an empty detection it gives an empty tracking result out. Furthermore on the upcoming third frame (after 2 frame skips) it makes a new detection and input this list into the tracker but the tracker output is still empty.

Do you know what I'm doing wrong in my implementation?

My implementation (edited for easier readability):
It's running in a while loop where each loop counts ix up which grabs the next frame of the video.

if skip_count == detection_frame_skip or ix < 5: # detection_frame_skip = 2
            # Make a detection on the image
            r = detect_np(net, meta, img) #
            skip_count = 0
        else:
            r = []
            skip_count += 1
        detections = []
        if r:
            for detection in r:
                # Some data-handling
                probability = detection[1]
                x, y, w, h = detection[2][0], detection[2][1], detection[2][2], detection[2][3]
                x_min, y_min, x_max, y_max = convertBack(float(x), float(y), float(w), float(h))
                # Save detection in sort format
                detections.append([x_min, y_min, x_max, y_max, probability])
        # Convert detection from list to numpy array for sort
        detections = np.array(detections)
        print(detections)
        tracker_results = tracker.update(detections)
        print(tracker_results)

Output of the terminal (edited, removed many detections and only kept one for simplicity)

$ python main.py
Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.75000000e+02   5.99000000e+02   4.74000000e+02   6.67000000e+02
    9.92387056e-01]
[[  1.76200000e+03   2.52000000e+02   1.89700000e+03   3.56000000e+02
    2.30000000e+01]
Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.99000000e+02   4.74000000e+02   6.67000000e+02
    9.92466390e-01]
[[  1.74125536e+03   2.58809752e+02   1.88174823e+03   3.66188550e+02
    2.30000000e+01]

Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.98000000e+02   4.74000000e+02   6.67000000e+02
    9.92073357e-01]
[[  1.72014292e+03   2.55391698e+02   1.86403637e+03   3.64960492e+02
    2.30000000e+01]

Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.97000000e+02   4.74000000e+02   6.66000000e+02
    9.92073357e-01]
[[  1.70176462e+03   2.55298312e+02   1.84845438e+03   3.66359289e+02
    2.30000000e+01]

Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.98000000e+02   4.75000000e+02   6.67000000e+02
    9.92194295e-01]
[[  1.69376420e+03   2.51968522e+02   1.84135284e+03   3.63011087e+02
    2.30000000e+01]

Img load: 0.06 seconds
[]
[]
Img load: 0.06 seconds
[]
[]
Img load: 0.06 seconds
[]
[]
Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.75000000e+02   5.99000000e+02   4.74000000e+02   6.67000000e+02
    9.92482007e-01]
[] 
Img load: 0.07 seconds
[]
[]
Img load: 0.06 seconds
[]
[]
Img load: 0.07 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.98000000e+02   4.75000000e+02   6.68000000e+02
    9.92348790e-01]
[]

installing some dependencies from requirements.txt in a conda environment fails

For reference, I ran pip install -r requirements.txt in a conda environment. The following appears when trying to install numba.

running build_ext
building 'numba._dynfunc' extension
C compiler: gcc -pthread -B /home/amao1/anaconda3/envs/sort/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/numba
compile options: '-I/home/amao1/anaconda3/envs/sort/include/python3.7m -c'
gcc: numba/_dynfuncmod.c
In file included from numba/_dynfuncmod.c:1:0:
numba/_dynfunc.c: In function ‘dup_string’:
numba/_dynfunc.c:238:9: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
tmp = PyString_AsString(strobj);
^
numba/_dynfunc.c: In function ‘generator_dealloc’:
numba/_dynfunc.c:350:10: error: ‘_Py_Finalizing’ undeclared (first use in this function); did you mean ‘_Py_IsFinalizing’?
if (!_Py_Finalizing)
^~~~~~~~~~~~~~
_Py_IsFinalizing
numba/_dynfunc.c:350:10: note: each undeclared identifier is reported only once for each function it appears in
In file included from numba/_dynfuncmod.c:1:0:
numba/_dynfunc.c: In function ‘dup_string’:
numba/_dynfunc.c:238:9: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
tmp = PyString_AsString(strobj);
^
numba/_dynfunc.c: In function ‘generator_dealloc’:
numba/_dynfunc.c:350:10: error: ‘_Py_Finalizing’ undeclared (first use in this function); did you mean ‘_Py_IsFinalizing’?
if (!_Py_Finalizing)
^~~~~~~~~~~~~~
_Py_IsFinalizing
numba/_dynfunc.c:350:10: note: each undeclared identifier is reported only once for each function it appears in
error: Command "gcc -pthread -B /home/amao1/anaconda3/envs/sort/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/amao1/anaconda3/envs/sort/include/python3.7m -c numba/_dynfuncmod.c -o build/temp.linux-x86_64-3.7/numba/_dynfuncmod.o" failed with exit status 1

---

Failed building wheel for numba

new improvement about sort

I combine SSD detection with sort tracking,it performs well,I will give a specific report about the results,Do you have new method or ideas about improving sort algorithm

Smoothing detections using tracking

I want to smooth out my detection using tracking. E.g if some object is detected by my detection algorithm in frame 5 and not detected in next 4 frame, in that case .....i want to track that object for at least 4 frames(if it is not detected in 4 frames continuously, I won't track it.). I hope, I'm clear with that.

Can you guide me with the required changes?

Confidence score hardcoded as 1

The confidence score of predictions after "update" is hardcoded as 1 in Line 282 although it takes the confidence score of original detection in its input in "dets". How can I get the confidence score of bounding box from the tracker output?

Thanks

implicit bug of integer conversion

you use an unsigned int conversion here

        for d in trackers:
          print('%d,%d,%.2f,%.2f,%.2f,%.2f,1,-1,-1,-1'%(frame,d[4],d[0],d[1],d[2]-d[0],d[3]-d[1]),file=out_file)
          if(display):
            d = d.astype(np.uint32)
            ax1.add_patch(patches.Rectangle((d[0],d[1]),d[2]-d[0],d[3]-d[1],fill=False,lw=3,ec=colours[d[4]%32,:]))
            ax1.set_adjustable('box-forced')

        if(display):
          fig.canvas.flush_events()
          plt.draw()
          ax1.cla()

And the bound of box in d might be a negative number. So it might be d = d.astype(np.int32) instead of d = d.astype(np.uint32).

How to use my own data

I want to use sort on my own data,but i do not know how to do that

"Bidirectional" Kalman Filter

Hi,

Thanks for great and simple code, used it and it worked beautifully!

I was thinking to improve the tracking even further yet having real-time speed as it's a requirement in my project. Specifically, the main problem in my pipeline (and in general with CNNs) is there will be some noise in the image and detections will either be FPs or FNs.

I was thinking to add a backwards Kalman Filter which is similar to the Bidirectional Recurrent Neural Networks. This way we would have 2 filters, one in forward direction (as in SORT) and another in backward both of which would predict a current location and two results will be merged using another simple method. From my understanding of Kalman filters it would only be possible to have a backward filter only if I were to recompute the full state at every sequence step given next n states.

What do you think?

how to do tracking with just 1 frame detection [detect and track]

Hi,

I have a vehicle detection algorithm that provides a bounding box for only 1 frame for 30 frames/. i.e. every 30 frames I get a new bounding box.

In this context, for the remaining 29 frames I would like to apply SORT to track. I am able to run the demo script. however, when I try to track for more frames, it does not. Could you provide some info as to how to adapt SORT for detecting and tracking?

Understanding the physics intuition of: if kf.x[6] + kf.x[2] <=0.

First off. Thank you so much for the library and the research paper.

I had a simple doubt.

Why are we checking if the measurement of scale (area covered by box) and the estimate of scale sum up to less than or equal to zero or not? The relevant line of code.
(I modified the self.kf.x[6] *= 0.0 to self.kf.x[6] = 0.0)

(Please correct me if my understanding of the problem is wrong as well)

What is the physics intuition to understand this line of code? Why are we checking this?

Thank you so much.

(not an issue)is there a way to track live data from a camera?

hi @abewley
I run the demo successfully and the result is impressive ! good work!

I'm wandering if sort could be used for live stream data tracking?The data may not be a video file but some frames captured by a camera.and if it could, how to do it?thank you very much!

How to use sort on another data set

Hi,
We are a forth year student at faculty of Engineering and our graduation project is to track people in a certain scene and count them. We want to use SORT to track people but with our data set. what are we need to change in the code. and I want to know what the benefit of using the output folder in code.

(not really an issue) C/C++ implementation?

Are you aware of any implementations in C/C++? I'm looking to use this in a bare metal project.

Initialisation of kalman parameters

Can you guide me about, which parameter should be tracked? And how do tune the measurement and process noise for this type of application. Here I am getting detection from some detection model like SSD and tracking those detections. I am considering 20 FPS and getting detections at each frame(i.e. I am getting measurements at each frame).

I am doing for FCW system where I am getting different objects like car, truck, person, bicycle, motorbikes etc

print class value in python?

Hi do you know how to show the class value in python ?
I want to debug it

Using faster-RCNN for SORT

I would like to know the basic idea, do we detect and save the bounding box parameters later used for tracking or it can track which the detection(s) are in progress?

Also, is there any script for outputting the labels directly?

2DMOT2015 dataset label question

data/PETS09-S2L1/det.txt
the line 1 is 1,-1,649.441,231.502,44.417,86.13,0.995474,-1,-1,-1
what does it mean and what the det.txt can do ? ( including every -1 and 0.995474

About Data

Hello! I have run your demo successfully! However,I wonder that How to get 'data' in your demo? Thank you!

Complexity of Hungarian Algorithm

I've searched your paper and couldn't find the complexity of hungarian algorithm. I know there are implementations of o(n^3) complexity, is yours o(n^4)?

score

(x, y, w, h, score) score what is the meaning of how to calculate the score d [4]

can I feed detection boxes every 5 frame?

Hi,
In you code, you feed detection boxes every frame, and use tracker.update(detections) to update the location and track_id. I have a question, can I feed detection boxes every 5 frame?

Tracker re-initialization

Hi, I am trying the Sort for tracking objects in multiple videos, but I noticed that when I re-initialize the tracker for the next video, mot_tracker=Sort(), the tracker ID do not reset and keeps increasing, I tried removing the tracker after each video but that does not help, do you know why is this happening?

Output visualisation

I used the sort.py --display and ran the program ,the output of the tracking is not visible, by opening a new window. How to see the output of that?

Bug in def convert_bbox_to_z(bbox):

r = w / h should be replaced with r = w / float(h) to avoid integer rounding with python 2.7 downwards (which may result in ratios of 0)

No module named numba

hi

I am trying to install numba for python but after following the instruction from the homepage I got this error that no model named numba can not be found.

sudo -H python3 -m pip install --user numba

Tracking dense scenes

Hi Alex,

First of all thank you for sharing this great library with us!
Gradually I went to applying it to more and more complicated scenarios playing around with different parameters. This time I either reached a limit or my random tweaking of parameters / hard-coded values is too much of a "dance in the dark".
Thing I'm struggling with is being able to track very dense scenes (~100 objects).

When I apply sort tracker to such scenes it doesn't do anything (i.e. does not match detections between frames). If I limit the detection area to a small sub-region of the image it all goes back to normal.
Please advice what parameters (and within what ranges) would it be sensible to tweak, or what other approach do you recommend.

Thanks,
Tom

Detection

where is the detector?

Where do the dets came from?

As far as I can see, this project just saves markup and displays the results of the data , right?
How can I get the tracking tag from the image or video?

convert_x_to_bbox function output

Thanks for making the code public.

1. It seems that 'convert_x_to_bbox' output points at the bottom left and the top right.

   Is this a typo in the function comment?

2. Also would you please advise that why a minus iou_matrix passed to linear assignment module?

Where's the pretrained model

In fact I don't understand how this project works since I can't find the pretrained model. Will you give the model or the training code?

Wrong format in comment

In update method of the tracker on this line, it says in the comment that input format is supposed to be x,y,w,h,score while in reality, it expects x1,y1,x2,y2,score as can be further seen in usage example.

Also might be a good idea to explicitly state return format.

trackers = sort.update(detections) gives the same bb as the detections

Maybe I am missing something,
but calling update with an array of detections always give me the same bb as the detections.
therefore i dont track anything but stay in the same place.
btw - if i want to detect only once objects, than only to update through the tracker - shall i call update with [] empty list?

Runnig SORT on own dataset.

Hello,I am looking at SORT and successfully ran for the mot data-set provided.But, how do I run the SORT for my own data-set?

Format used in your data

Hello Sir,
The data given in your det.txt is of the form:
1,-1,1691.97,381.048,152.23,352.617,0.995616,-1,-1,-1

whereas the mot challenge is of the form:

<3>

Why is the id kept as -1?

function of max_age

Hi,

I just confused when I set max_age=3
the latest pop results as flows: x- and y-coordinate
[266.673,303.671, ...],
[266.673,303.671, ...],
[266.673,303.671, ...],
[266.673,303.671, ...],
why they are same ? from my prospective, Kalman predicts all the trackers,so the results should be different.

Could I know what's your opinion about this question?

Many thanks!

Interpretation of det.txt

Every row in det.txt has data like

1,-1,286.552,154.138,71.337,167.328,0.998331,-1,-1,-1.

So can anyone explain how to interpret this data? I understand the floating point numbers represent bounding box and prediction confidence.

#get detections

What is meant by get detections? What is this detection we refer to? Can I use faster RCNN for this?

Minimum FPS requirement

Can you please suggest minimum fps required for tracking?
(In other words will it work for 1 FPS or 2 FPS because relative change in objects will be large)

Thanks

The use of List 'history'

Hello,
Thank you for sharing and I have an issue about the List 'history' defined in the code.
I see you define a List called 'history' in the line 99 and use it in the line 109, 125 , 126 and realize this List is used to record the previous predict values, but it seems has nothing about the other works such as predict and associate and can be removed. (I removed it, it seems nothing happened)
So why you want record this value? Could you tell me something about it?

Thanks for your any replies.

About the Main Results

Hi,
Could you tell me more details about the Main Results in README.md?

1. Are these results are generated by Faster RCNN and SORT?
2. How about the overall speed on this validation set?

What's meaning of each column of detection txt file? Is there detection score?

Hi, dear developer. I wonder what's meaning in the column of gt.txt OR det.txt.
Especially in det.txt file, is there any detection score in these elements? I guess the 7th column is the fasterRCNN score, but not sure.
Would u mind detail each column's meaning and how to genearte it? Thx.

MOTA and MOTP

how to test the MOTA and MOTP for sort?
I know the https://motchallenge.net/
but I still don't have any idea

How to use

Detection independent tracking

Hi,
So [correct me if i am wrong], this tracking algorithm is heavily dependent on detection in the sense that if the detector fails to detect the object for one frame then the corresponding tracker will be discarded and a new one will be spawned and instantiated the next time the object is detected by the detector.
Is there a way to ensure retention of tracking for a couple of frames despite non-detection by the detector?

Run with MOT test files

Hi, Alex, thanks for your great job
I'm trying to run this work with MOT test files:
I modify sort.py to read the dets file in '$MOT2015/test/PETS09-S2L2/det/det.txt'
and set '$MOT2015/test/PETS09-S2L2/img1/xxxxxx.jpg' as input frame
But it cannot work correctly, No tracking bbox on my display window
Anything I missed ?

python launcher crashing

while running sort.py with the argument display, the python launcher is crashing and no output is available.

Steps to track objects on custom dataset

Is there any tutorial to implement SORT on custom dataset?