abewley/sort

Simple, online, and realtime tracking of multiple objects in a video sequence.

sort's Issues

Interpretation of det.txt

Every row in det.txt has data like

1,-1,286.552,154.138,71.337,167.328,0.998331,-1,-1,-1.

So can anyone explain how to interpret this data? I understand the floating point numbers represent bounding box and prediction confidence.

Output visualisation

I used the sort.py --display and ran the program ,the output of the tracking is not visible, by opening a new window. How to see the output of that?

new improvement about sort

I combine SSD detection with sort tracking,it performs well,I will give a specific report about the results,Do you have new method or ideas about improving sort algorithm

2DMOT2015 dataset label question

data/PETS09-S2L1/det.txt
the line 1 is 1,-1,649.441,231.502,44.417,86.13,0.995474,-1,-1,-1
what does it mean and what the det.txt can do ? ( including every -1 and 0.995474

Tracker re-initialization

Hi, I am trying the Sort for tracking objects in multiple videos, but I noticed that when I re-initialize the tracker for the next video, mot_tracker=Sort(), the tracker ID do not reset and keeps increasing, I tried removing the tracker after each video but that does not help, do you know why is this happening?

print class value in python?

Hi do you know how to show the class value in python ?
I want to debug it

The use of List 'history'

Hello,
Thank you for sharing and I have an issue about the List 'history' defined in the code.
I see you define a List called 'history' in the line 99 and use it in the line 109, 125 , 126 and realize this List is used to record the previous predict values, but it seems has nothing about the other works such as predict and associate and can be removed. (I removed it, it seems nothing happened)
So why you want record this value? Could you tell me something about it?

Thanks for your any replies.

Minimum FPS requirement

Can you please suggest minimum fps required for tracking?
(In other words will it work for 1 FPS or 2 FPS because relative change in objects will be large)

Thanks

Format used in your data

Hello Sir,
The data given in your det.txt is of the form:
1,-1,1691.97,381.048,152.23,352.617,0.995616,-1,-1,-1

whereas the mot challenge is of the form:

<3>

Why is the id kept as -1?

Detection independent tracking

Hi,
So [correct me if i am wrong], this tracking algorithm is heavily dependent on detection in the sense that if the detector fails to detect the object for one frame then the corresponding tracker will be discarded and a new one will be spawned and instantiated the next time the object is detected by the detector.
Is there a way to ensure retention of tracking for a couple of frames despite non-detection by the detector?

#get detections

What is meant by get detections? What is this detection we refer to? Can I use faster RCNN for this?

Confidence score hardcoded as 1

The confidence score of predictions after "update" is hardcoded as 1 in Line 282 although it takes the confidence score of original detection in its input in "dets". How can I get the confidence score of bounding box from the tracker output?

Thanks

Runnig SORT on own dataset.

Hello,I am looking at SORT and successfully ran for the mot data-set provided.But, how do I run the SORT for my own data-set?

Bug in def convert_bbox_to_z(bbox):

r = w / h should be replaced with r = w / float(h) to avoid integer rounding with python 2.7 downwards (which may result in ratios of 0)

Understanding the physics intuition of: if kf.x[6] + kf.x[2] <=0.

First off. Thank you so much for the library and the research paper.

I had a simple doubt.

Why are we checking if the measurement of scale (area covered by box) and the estimate of scale sum up to less than or equal to zero or not? The relevant line of code.
(I modified the self.kf.x[6] *= 0.0 to self.kf.x[6] = 0.0)

(Please correct me if my understanding of the problem is wrong as well)

What is the physics intuition to understand this line of code? Why are we checking this?

Thank you so much.

About the Main Results

Hi,
Could you tell me more details about the Main Results in README.md?

1. Are these results are generated by Faster RCNN and SORT?
2. How about the overall speed on this validation set?

A way to evaluate tracker performance

Hello, your algorithm seems to work great! However, I'm seeking to evaluate it using usual metrics (MOTA, MOTP). Are you planning on adding such feature or can refer to some git repository which could be of a help?

Wrong format in comment

In update method of the tracker on this line, it says in the comment that input format is supposed to be x,y,w,h,score while in reality, it expects x1,y1,x2,y2,score as can be further seen in usage example.

Also might be a good idea to explicitly state return format.

trackers = sort.update(detections) gives the same bb as the detections

Maybe I am missing something,
but calling update with an array of detections always give me the same bb as the detections.
therefore i dont track anything but stay in the same place.
btw - if i want to detect only once objects, than only to update through the tracker - shall i call update with [] empty list?

Smoothing detections using tracking

I want to smooth out my detection using tracking. E.g if some object is detected by my detection algorithm in frame 5 and not detected in next 4 frame, in that case .....i want to track that object for at least 4 frames(if it is not detected in 4 frames continuously, I won't track it.). I hope, I'm clear with that.

Can you guide me with the required changes?

Run with MOT test files

Hi, Alex, thanks for your great job
I'm trying to run this work with MOT test files:
I modify sort.py to read the dets file in '$MOT2015/test/PETS09-S2L2/det/det.txt'
and set '$MOT2015/test/PETS09-S2L2/img1/xxxxxx.jpg' as input frame
But it cannot work correctly, No tracking bbox on my display window
Anything I missed ?

Using faster-RCNN for SORT

I would like to know the basic idea, do we detect and save the bounding box parameters later used for tracking or it can track which the detection(s) are in progress?

Also, is there any script for outputting the labels directly?

Detection

where is the detector?

Trackers not deleting:

Hi abewley ,

I was trying to run SORT on videos.But I found that there are some instances when trackers are not getting deleted(Continuing their linear trajectory) when no detection is present for few frames.
Code:



Kalman filter update function:

Results seems something like this:

I have used max_age = 2 , min_hits = 5.
It will be a a great help if you could help me out of this issue.

Thanks in advance
Aayush

"Bidirectional" Kalman Filter

Hi,

Thanks for great and simple code, used it and it worked beautifully!

I was thinking to improve the tracking even further yet having real-time speed as it's a requirement in my project. Specifically, the main problem in my pipeline (and in general with CNNs) is there will be some noise in the image and detections will either be FPs or FNs.

I was thinking to add a backwards Kalman Filter which is similar to the Bidirectional Recurrent Neural Networks. This way we would have 2 filters, one in forward direction (as in SORT) and another in backward both of which would predict a current location and two results will be merged using another simple method. From my understanding of Kalman filters it would only be possible to have a backward filter only if I were to recompute the full state at every sequence step given next n states.

What do you think?

How to use sort on another data set

Hi,
We are a forth year student at faculty of Engineering and our graduation project is to track people in a certain scene and count them. We want to use SORT to track people but with our data set. what are we need to change in the code. and I want to know what the benefit of using the output folder in code.

(not really an issue) C/C++ implementation?

Are you aware of any implementations in C/C++? I'm looking to use this in a bare metal project.

(not an issue)is there a way to track live data from a camera?

hi @abewley
I run the demo successfully and the result is impressive ! good work!

I'm wandering if sort could be used for live stream data tracking?The data may not be a video file but some frames captured by a camera.and if it could, how to do it?thank you very much!

Steps to track objects on custom dataset

Is there any tutorial to implement SORT on custom dataset?

implicit bug of integer conversion

you use an unsigned int conversion here

        for d in trackers:
          print('%d,%d,%.2f,%.2f,%.2f,%.2f,1,-1,-1,-1'%(frame,d[4],d[0],d[1],d[2]-d[0],d[3]-d[1]),file=out_file)
          if(display):
            d = d.astype(np.uint32)
            ax1.add_patch(patches.Rectangle((d[0],d[1]),d[2]-d[0],d[3]-d[1],fill=False,lw=3,ec=colours[d[4]%32,:]))
            ax1.set_adjustable('box-forced')

        if(display):
          fig.canvas.flush_events()
          plt.draw()
          ax1.cla()

And the bound of box in d might be a negative number. So it might be d = d.astype(np.int32) instead of d = d.astype(np.uint32).

How to use

Where's the pretrained model

In fact I don't understand how this project works since I can't find the pretrained model. Will you give the model or the training code?

Tracking with frame skips

Hello, I have some issues regarding tracking with skip in detections. I'm trying to get frameskip to work with this tracker.
How I expect it to work:

1. Initialise tracking by feeding it with x detections with no frame skip (e.g. 5 frames).
2. Now only use detection every q frame (q frame could be 3 frames)
3. For every frame feed tracker detection list, if no detection is run feed an empty list. According to requirement for Sort function.

According to Sort(object) comment: Requires: this method must be called once for each frame even with empty detections.

How it works:
Tracks fine in the initialization, but when I feed the tracker an empty detection it gives an empty tracking result out. Furthermore on the upcoming third frame (after 2 frame skips) it makes a new detection and input this list into the tracker but the tracker output is still empty.

Do you know what I'm doing wrong in my implementation?

My implementation (edited for easier readability):
It's running in a while loop where each loop counts ix up which grabs the next frame of the video.

if skip_count == detection_frame_skip or ix < 5: # detection_frame_skip = 2
            # Make a detection on the image
            r = detect_np(net, meta, img) #
            skip_count = 0
        else:
            r = []
            skip_count += 1
        detections = []
        if r:
            for detection in r:
                # Some data-handling
                probability = detection[1]
                x, y, w, h = detection[2][0], detection[2][1], detection[2][2], detection[2][3]
                x_min, y_min, x_max, y_max = convertBack(float(x), float(y), float(w), float(h))
                # Save detection in sort format
                detections.append([x_min, y_min, x_max, y_max, probability])
        # Convert detection from list to numpy array for sort
        detections = np.array(detections)
        print(detections)
        tracker_results = tracker.update(detections)
        print(tracker_results)

Output of the terminal (edited, removed many detections and only kept one for simplicity)

$ python main.py
Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.75000000e+02   5.99000000e+02   4.74000000e+02   6.67000000e+02
    9.92387056e-01]
[[  1.76200000e+03   2.52000000e+02   1.89700000e+03   3.56000000e+02
    2.30000000e+01]
Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.99000000e+02   4.74000000e+02   6.67000000e+02
    9.92466390e-01]
[[  1.74125536e+03   2.58809752e+02   1.88174823e+03   3.66188550e+02
    2.30000000e+01]

Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.98000000e+02   4.74000000e+02   6.67000000e+02
    9.92073357e-01]
[[  1.72014292e+03   2.55391698e+02   1.86403637e+03   3.64960492e+02
    2.30000000e+01]

Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.97000000e+02   4.74000000e+02   6.66000000e+02
    9.92073357e-01]
[[  1.70176462e+03   2.55298312e+02   1.84845438e+03   3.66359289e+02
    2.30000000e+01]

Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.98000000e+02   4.75000000e+02   6.67000000e+02
    9.92194295e-01]
[[  1.69376420e+03   2.51968522e+02   1.84135284e+03   3.63011087e+02
    2.30000000e+01]

Img load: 0.06 seconds
[]
[]
Img load: 0.06 seconds
[]
[]
Img load: 0.06 seconds
[]
[]
Img load: 0.06 seconds
Detect: 0.04 seconds
[[  3.75000000e+02   5.99000000e+02   4.74000000e+02   6.67000000e+02
    9.92482007e-01]
[] 
Img load: 0.07 seconds
[]
[]
Img load: 0.06 seconds
[]
[]
Img load: 0.07 seconds
Detect: 0.04 seconds
[[  3.76000000e+02   5.98000000e+02   4.75000000e+02   6.68000000e+02
    9.92348790e-01]
[]

python launcher crashing

while running sort.py with the argument display, the python launcher is crashing and no output is available.

Complexity of Hungarian Algorithm

I've searched your paper and couldn't find the complexity of hungarian algorithm. I know there are implementations of o(n^3) complexity, is yours o(n^4)?

Initialisation of kalman parameters

Can you guide me about, which parameter should be tracked? And how do tune the measurement and process noise for this type of application. Here I am getting detection from some detection model like SSD and tracking those detections. I am considering 20 FPS and getting detections at each frame(i.e. I am getting measurements at each frame).

I am doing for FCW system where I am getting different objects like car, truck, person, bicycle, motorbikes etc

installing some dependencies from requirements.txt in a conda environment fails

For reference, I ran pip install -r requirements.txt in a conda environment. The following appears when trying to install numba.

running build_ext
building 'numba._dynfunc' extension
C compiler: gcc -pthread -B /home/amao1/anaconda3/envs/sort/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/numba
compile options: '-I/home/amao1/anaconda3/envs/sort/include/python3.7m -c'
gcc: numba/_dynfuncmod.c
In file included from numba/_dynfuncmod.c:1:0:
numba/_dynfunc.c: In function ‘dup_string’:
numba/_dynfunc.c:238:9: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
tmp = PyString_AsString(strobj);
^
numba/_dynfunc.c: In function ‘generator_dealloc’:
numba/_dynfunc.c:350:10: error: ‘_Py_Finalizing’ undeclared (first use in this function); did you mean ‘_Py_IsFinalizing’?
if (!_Py_Finalizing)
^~~~~~~~~~~~~~
_Py_IsFinalizing
numba/_dynfunc.c:350:10: note: each undeclared identifier is reported only once for each function it appears in
In file included from numba/_dynfuncmod.c:1:0:
numba/_dynfunc.c: In function ‘dup_string’:
numba/_dynfunc.c:238:9: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
tmp = PyString_AsString(strobj);
^
numba/_dynfunc.c: In function ‘generator_dealloc’:
numba/_dynfunc.c:350:10: error: ‘_Py_Finalizing’ undeclared (first use in this function); did you mean ‘_Py_IsFinalizing’?
if (!_Py_Finalizing)
^~~~~~~~~~~~~~
_Py_IsFinalizing
numba/_dynfunc.c:350:10: note: each undeclared identifier is reported only once for each function it appears in
error: Command "gcc -pthread -B /home/amao1/anaconda3/envs/sort/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/amao1/anaconda3/envs/sort/include/python3.7m -c numba/_dynfuncmod.c -o build/temp.linux-x86_64-3.7/numba/_dynfuncmod.o" failed with exit status 1

---

Failed building wheel for numba

Where do the dets came from?

As far as I can see, this project just saves markup and displays the results of the data , right?
How can I get the tracking tag from the image or video?

score

(x, y, w, h, score) score what is the meaning of how to calculate the score d [4]

convert_x_to_bbox function output

Thanks for making the code public.

1. It seems that 'convert_x_to_bbox' output points at the bottom left and the top right.

   Is this a typo in the function comment?

2. Also would you please advise that why a minus iou_matrix passed to linear assignment module?

parameters of Kalman filter

Hi,
how do you set the parameters of Kalman filter? is it depends on the data or we can use the parameters as you set?

The tracking bboxs are not exactly the location of the object,some lag exists, I suppose it's the problem of Kalman Filter parameters? Am i right?

MOTA and MOTP

how to test the MOTA and MOTP for sort?
I know the https://motchallenge.net/
but I still don't have any idea

function of max_age

Hi,

I just confused when I set max_age=3
the latest pop results as flows: x- and y-coordinate
[266.673,303.671, ...],
[266.673,303.671, ...],
[266.673,303.671, ...],
[266.673,303.671, ...],
why they are same ? from my prospective, Kalman predicts all the trackers,so the results should be different.

Could I know what's your opinion about this question?

Many thanks!

can I feed detection boxes every 5 frame?

Hi,
In you code, you feed detection boxes every frame, and use tracker.update(detections) to update the location and track_id. I have a question, can I feed detection boxes every 5 frame?

About Data

Hello! I have run your demo successfully! However,I wonder that How to get 'data' in your demo? Thank you!

How to use my own data

I want to use sort on my own data,but i do not know how to do that

how to do tracking with just 1 frame detection [detect and track]

Hi,

I have a vehicle detection algorithm that provides a bounding box for only 1 frame for 30 frames/. i.e. every 30 frames I get a new bounding box.

In this context, for the remaining 29 frames I would like to apply SORT to track. I am able to run the demo script. however, when I try to track for more frames, it does not. Could you provide some info as to how to adapt SORT for detecting and tracking?

No module named numba

hi

I am trying to install numba for python but after following the instruction from the homepage I got this error that no model named numba can not be found.

sudo -H python3 -m pip install --user numba

Tracking dense scenes

Hi Alex,

First of all thank you for sharing this great library with us!
Gradually I went to applying it to more and more complicated scenarios playing around with different parameters. This time I either reached a limit or my random tweaking of parameters / hard-coded values is too much of a "dance in the dark".
Thing I'm struggling with is being able to track very dense scenes (~100 objects).

When I apply sort tracker to such scenes it doesn't do anything (i.e. does not match detections between frames). If I limit the detection area to a small sub-region of the image it all goes back to normal.
Please advice what parameters (and within what ranges) would it be sensible to tweak, or what other approach do you recommend.

Thanks,
Tom

What's meaning of each column of detection txt file? Is there detection score?

Hi, dear developer. I wonder what's meaning in the column of gt.txt OR det.txt.
Especially in det.txt file, is there any detection score in these elements? I guess the 7th column is the fasterRCNN score, but not sure.
Would u mind detail each column's meaning and how to genearte it? Thx.