mooch443/trex

TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields.

Now with native Apple Silicone (M1) and ML Compute support. How to install TRex (arm64).

Documentation: https://trex.run/docs

Hey there

Welcome to the git repository of TRex (https://trex.run) -- a software designed to track and identify individuals and other moving entities using computer vision and machine learning. The work-load is split into two (not entirely separate) tools:

• TGrabs: Record or convert existing videos, perform live-tracking and closed-loop experiments
• TRex: Track converted videos (in PV format), use the automatic visual recognition, explore the data with visual helpers, export task-specific data, and adapt tracking parameters to specific use-cases

TRex can track 256 individuals in real-time, or up to 128 with all fancy features like posture estimation enabled, and for up to 100 individuals allows you to (when realtime speed is not required) visually recognize individuals and automatically correct potential tracking errors.

TGrabs, which is used to directly process already saved videos or to record directly from webcams and/or Basler machine-vision cameras with integrated and customizable closed-loop support. Camera support can be extended for other APIs with a bit of C++ knowledge, of course.

Installation

TRex supports all major platforms. You can create a new virtual environment (named tracking here) using Anaconda or miniconda/miniforge by running:

# macOS (Intel/arm64 M1), Windows
conda create -n tracking -c trexing trex 

# Linux
conda create -n tracking -c defaults -c conda-forge -c trexing trex 

macOS with an arm64 / M1 processor

If you own a new Mac with an Apple Silicone CPU, the Intel version (above) works fine in Rosetta. However, I would strongly encourage installing TRex via miniforge, a flavor of miniconda that natively supports arm64 packages. Simply follow the instructions here for installing miniforge: https://github.com/conda-forge/miniforge#download.

Once you're done, you can run this command to create the virtual environment:

# macOS (arm64/M1)
conda create -n tracking -c trexing trex 

Installing tensorflow on the M1 is a bit more complicated, which is why TRex will not allow you to use machine learning unless you install the following extra packages manually. Instructions will be printed out after you created the environment Apple provides their own tensorflow version for macOS including a native METAL (https://developer.apple.com/metal/tensorflow-plugin/) plugin. To install tensorflow inside your environment, just run:

# activate the TRex environment
conda activate tracking

# install tensorflow dependencies and metal plugin
conda install -c apple -y tensorflow-deps==2.7.0
python -m pip install tensorflow-macos==2.7.0 tensorflow-metal==0.3.0

Manual compilation

Pre-built binaries are compiled with fewer optimzations and features than a manually compiled one (due to compatibility and licensing issues) and thus are slightly slower =(. For example, the conda versions do not offer support for Basler cameras. If you need to use TGrabs with machine vision cameras, or need as much speed as possible (or the newest version), please consider compiling the software yourself.

If you want compatibility with the Basler API (or other things with licensing/portability issues), please use one of the manual compilation options (see https://trex.run/docs/install.html).

Usage

Within the conda environment, simply run:

trex

Opening a video directly and adjusting parameters:

trex -i /path/to/video.pv -track_threshold 25 -track_max_individuals 10

If you don't want a graphical user interface and save/quit when tracking finishes:

trex -i /path/to/video.pv -nowindow -auto_quit

To convert a video to our custom pv format (for usage in TRex) from the command-line:

tgrabs -i /full/path/to/video.mp4 -o funny_name

Read more about parameters for TRex here and for TGrabs here.

Contributors, Issues, etc.

This project has been developed, is still being updated, by Tristan Walter. If you want to contribute, please submit a pull request on github and I will be happy to credit you here, for any substantial contributions!

If you have any issues running the software please consult the documentation first (especially the FAQ section) and if this does not solve your problem, please file an issue using the issue tracker here on github. If you experience problems with Tensorflow, such as installing CUDA or cuDNN dependencies, then please direct issues to those development teams.

License

Released under the GPLv3 License (see LICENSE).

Reference

If you use this software in your work, please cite our open-access paper:

@article {walter2020trex,
  article_type = {journal},
  title = {TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields},
  author = {Walter, Tristan and Couzin, Iain D},
  editor = {Lentink, David},
  volume = 10,
  year = 2021,
  month = {feb},
  pub_date = {2021-02-26},
  pages = {e64000},
  citation = {eLife 2021;10:e64000},
  doi = {10.7554/eLife.64000},
  url = {https://doi.org/10.7554/eLife.64000},
  journal = {eLife},
  issn = {2050-084X},
  publisher = {eLife Sciences Publications, Ltd},
}

trex's Issues

Correcting Old Imgstores With Incorrect Size

FWIW Instead of the fix you have

trex/Application/src/commons/common/video/VideoSource.cpp

Lines 374 to 379 in 586ce98

	/**
	* Because of a bug in loop bio software while saving specific videos,
	* they sometimes put slightly wrong metadata in the accompanying npz files.
	* In this case we have to disregard the dimensions stated within and use
	* the actual ones from the frame.
	**/

You can take the same fix we have, which doesn't require opening / reading the first movie

loopbio/imgstore#12
https://github.com/loopbio/imgstore/blob/444fc61bd7cd00e1ec8cd1bc3cab51f7bcc59352/imgstore/stores.py#L1015

which is basically shape &= -2 to truncate to even because the underlying cause was me not respecting+checking h264 size limitations

Positioning of GUI elements

Some GUI elements (such as datasets and the timeline-hover) currently have weird positioning. Need to investigate this.

blob_split

1. The blob_split_global_shrink_limit and blob_split_max_shrink have a default value of 0.2 listed in the documentation but my TRex output indicates them being 0.4.

2. How is decided which threshold is used when splitting blobs? Whichever is smaller?

Make heatmaps saveable in TRex

Currently this can only be done using pvinfo.

Error when building from source

Hi Tristan,
I'm trying to compile trex on my system (Ubuntu 18.04.5, gcc-8, g++-8) since I am using Basler cameras, and the build fails for some reason that is unclear to me. I've attached the terminal output after running trex_build_unix.sh. The main issue seems to be with building glfw.

Thanks,
Elhanan
trex_build_201020.txt

Definitions of positional data variables

Hi, this is a great tool, thanks so much for developing and maintaining it!

I'm wondering if there is somewhere I can find definitions and units of all of the default variables associated with positional data. The tables on this page (https://trex.run/docs/formats.html) provide some, but not all. I can guess what some probably mean but I want to make sure.

For example, I am wondering what "ANGLE" means. Also, some variables have "#pcentroid" after them, how does this differ from "#wcentroid"?

One issue I have with tracking is sometimes the midline and head position are off (or completely swapped in orientation from head to tail) for a few frames and but then go back to the correct position. I was hoping I could filter out these frames later if there is a variable that would indicate that the orientation of the head with respect to the centroid changed too fast, or reversed.

Thanks!

Documentation: -nowindow also works with TGrabs

Just a quick note for the documentation: the -nowindow option is only described as a TRex parameter, but I was happy to realise it also works with TGrabs, so it will be worthwhile including it there as well. Especially as it allows the user to make use of cluster computing, which can be SUPER useful for bigger tracking projects. :-D

Add a hint on arm64 to install tensorflow if not present

Originally posted by @mooch443 in #9 (comment)

Show reason why blobs are not available for matching

Reminder to myself to implement a feature that will show why a certain blob is not available for matching. Possibly only if hovered. Additional data-load will be necessary, but could also be made optional (upon restarting the program).

Segmentation fault in some .pv files

Describe the bug
Trex stops abruptly with a seg fault

If possible, please also include the full terminal output as a file attachment. : I have compiled some of the different errors as screenshots

Expected behavior (if it is not a simple crash)
What works? :For some videos it returns the tracked data results as npz files and there is no seg error and the tracking is for all the frames.
What doesn't work? : But this is not the same for other files which have the same format as the ones that it works on. it consistently does not track after a frame(depending on that video) and just throws out, core dumped message. For some files, it just stops tracking abruptly, and no message is given and the results are just a bunch of numbers. For the same file, it works on a ROI on one half and not on the other half, and the half on which it works is not consistent across different videos to actually see a trend

Computer (please complete the following information):

• Operating system: Ubuntu 18
• -Build hash [search for build in the settings box]
• Terminal used : bash

Additional context
Add any other context about the problem here. For crashes it may be very helpful to include the file (and settings) that made the program crash. Although please be aware that anything posted here will be public.

Screenshots

Cannot get auto_train working

log_trex.txt

I am trying to use the training network on trex, for example adding the option -auto_train but I always get the same error:

[EXCEPTION 16:15:55 gui.cpp:4889] The training process failed. Please check whether you are in the right python environment and check previous error messages.

attached is the full log. Trex was able to find the GPU so I'd say that can't be the problem and looking on forums it seems messages raised by TensorFlow are warnings.

I am using python 3.7.10 and miniconda 4.9.2 on a x86_64 Linux Machine

thanks in advance

giulio

trex crashes/closes immediately

Describe the bug
Forgive my inexperienced report as I only have a primary understanding of programming. Currently working with the provided sample video of four mice. Tgrabs successfully creates a png and pv file. However, when I run trex -i video.pv, a new window opens and then immediately closes/crashes. This is also true when I run trex. The file explorer menu opens, but when I select either a raw or converted video file, the program closes immediately.

To Reproduce
Steps to reproduce the behavior:

1. From Anaconda Prompt on Windows 10, run conda activate tracking
2. Run trex -i trex_video.pv
3. Window titled 'TRex v1.1.3 (trex_video)' opens and closes

Computer (please complete the following information):

• Operating system: Windows 10
• Terminal used: Anaconda Prompt

Additional context
In Anaconda Prompt, when running trex, I get the following error displayed after the 'LOADING COMMANDLINE' text:

Property('wd') = ""
"[ERROR ... CommandLine.cpp:146] Cannot change directory to ' '."

Initially, I had tried using WSL1/2, but unfortunately, it can't initialize OpenGL. However, when I run trex in WSL, I actually don't get the CommandLine.cpp error. Instead, it reads without error:

Property('wd') = "/home/zwiebellab/anaconda3/envs/tracking/bin"

I noticed that this Ubuntu-based folder contains 160 files, including thing like .scikit-learn-post-link.sh, ffmpeg, python3, and xzmore, to name a few. However, the folder in 'C:/Users/stf/Anaconda3/envs/tracking/bin' only contains 9 items, including omptarget.dll and the tgrabs.exe and trex.exe. I also tried simply moving the files form the Ubuntu folder to the C drive folder, but that didn't work, and a fresh re-installation and update of Trex also failed to solve the problem. My next step will be to get a proper Linux distribution up and running in lab, but for now, I can't seem to get past this hiccup on Windows.

Looking forward to getting this up and running. Thanks for publishing this great resource and thanks for the help!

.csv output

Hiya,

I have utilised TRex for looking at ant activity, the .npz file I uploaded to python and extracted as .csv files.

This ended in 31 separate files (attached image)

.

The reason I did this was to use in R (I am not well versed in python). However, when opening I don't see how I can use the information in R to plot, is anyone able to help with this please?

Some plots I was after are plots of the trail for each ant, plots with all together and then graphs with speed etc of species. And then general heat maps showing activity. So not a bug per se just a way to interpret the csv files. Here's a csv file as well.

somepathSPEED.csv

I tried uploading the .npz file but it isn't supported here.

I am sure there is an easy answer to this, but I am a newbie when it comes to utilising this type of information.

THanks in advance and thanks for the awesome application

All the best!
Mark

Example settings file

Hi Tristan,

Thank you for creating this software, putting it out there for others to use, and supporting open science. I've been applying TRex to some mouse videos I have. I was wondering if you could post an example of a .settings file? I would like to create one, but I don't know what the formatting should be (I should mention that I am pretty new to python - my main language is MATLAB, so please forgive me for not being able to reverse-engineer this answer!)

If you will allow me one more unrelated request - could you please post the TGrabs/TRex parameters you used to generate those amazing mouse tracking results in Appendix 12/Figure A1?

Appreciate your help!
Uday

TGrabs FATAL: Segmentation Fault

Hey, could you help me with the following TGrabs error?
I get it with two of 40 videos, always around the same time point. There don't seem to be extremely many blobs in the video and I can play it fine with a video player.

Thanks!

[10:35:26] 12968/44967 (t+9:38min) @ 42.5fps (eta:14:50min load:19ms proc:4ms track:0us save:4us) tgrabs[0x70a353] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f3c13fdf3c0] tgrabs(_ZN3cmn11CPULabeling8run_fastEPNS0_6DLListE+0x5ea)[0xcaec2a] tgrabs(_ZN3cmn11CPULabeling3runERKN2cv3MatEb+0x331)[0xcb0a51] tgrabs[0x6e4b62] tgrabs[0x6e6507] /home/stefan/anaconda3/envs/tracking/bin/../lib/libstdc++.so.6(+0xc819d)[0x7f3c125c319d] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f3c13fd3609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3c122c6293] FATAL: Segmentation Fault. Logged StackTrace tgrabs[0x70a353] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f3c13fdf3c0] /home/stefan/anaconda3/envs/tracking/bin/../lib/libstdc++.so.6(_ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base+0xf)[0x7f3c125b495e] tgrabs(_ZN6Timing16conclude_measureEd+0x124)[0xc94034] tgrabs[0x6e510c] Didn't clean up FrameGrabber properly. Waiting for analysis to be paused...Trying to close file... tgrabs[0x6e6507] /home/stefan/anaconda3/envs/tracking/bin/../lib/libstdc++.so.6(+0xc819d)[0x7f3c125c319d] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f3c13fd3609] [10:35:26] Index table is 103.74KB big. /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3c122c6293] FATAL: Segmentation Fault. Logged StackTrace /home/stefan/Desktop/batchTrex: line 8: 730915 Segmentation fault (core dumped) tgrabs -i ${dir}/allVids/${exp}_${col}_${cam}${pt}.MP4 -s ${dir}/trex/tgrabs${cam}.settings -d $dir/tGrabOut

Common background in batch processing

Hi Tristan, hope you're well!

After massive struggles on the experimental side, I'm now finally beginning to do some actual tracking - finally! So now I'm back to provide - and get - some feedback. This is not a feature request really, more of a clarification question.

My experiments are quite long (24-48h), so I record videos in 6h blocks to be on the safe side. And so, I was wondering if there is a way to compute the background mean using all the videos to be used across all videos? (For my purpose, max has worked very well and seems very fast).
I haven't done much batch processing, so along the same lines I struggled to use TGrabs (haven't actually gotten to the TRex part yet) for batch processing. Could you maybe share an example of how you achieve it in the terminal? E.g. process every file in a destination folder (specified with -d). That could maybe be worth adding to the Batch Processing docs page too I think. :-)

Fail on building tgrabs when building from source

Hi Tristan, thanks for the great work on this project! I recently got a new workstation and tried to build TRex again, but got an error.

When building TRex from source on Ubuntu 18.04 with Basler support, the build fails on Linking CXX executable ../../tgrabs .

The full error is:
/home/benyishay_la/anaconda3/envs/tracking/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/pylon/lib/libpylonutility.so: undefined reference to clock_gettime@GLIBC_2.17'
/home/benyishay_la/anaconda3/envs/tracking/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/pylon/lib/libpylonbase.so: undefined reference to memcpy@GLIBC_2.14'
collect2: error: ld returned 1 exit status
src/grabber/CMakeFiles/tgrabs.dir/build.make:294: recipe for target 'tgrabs' failed
make[2]: *** [tgrabs] Error 1
CMakeFiles/Makefile2:747: recipe for target 'src/grabber/CMakeFiles/tgrabs.dir/all' failed
make[1]: *** [src/grabber/CMakeFiles/tgrabs.dir/all] Error 2
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2

I hope it's okay I didn't use the exact report format.

Best,
Elhanan

Directional data

Hiya,
I just wondered if there was a way to know if my subject was moving left or right. Basically, food is left, nest is right, when under stress (or not) I wanted to see if they are moving more left than right, and then look at speed average.

Which data from the .npz file would give me that information?
And is there a way to calculate distance covered?

Here is a screenshot of the ants, square boxes is blocking of the tunnel where they seem to meet and greet quite a lot.

Thanks in advance!
Mark

Comparison of tracks with video frame in tracking view

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I get imperfect blobs with TGrabs (sometimes multiple are included in one blob, sometimes a subject is missed) so when looking at tracks, I want to make sure that all of the subjects in view are accounted for correctly. It's frustrating to not be able to compare the tracks to the base video; I have no idea how accurate these tracks are to my data.

Describe the solution you'd like
A clear and concise description of what you want to happen.

When I scan through tracks in tracking mode, I would like to see the corresponding video frame instead of just the background image.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

The best option I can think of is to pull up the video next to my Trex window and try to match them up by time, then compare the contours with the video. This can generally show if individuals are untracked, but it would be very useful to be able to see the contours superimposed over the actual base video frame to see how well each animal is tracked.

Additional context
Add any other context or screenshots about the feature request here.
Thank you! This is a really compelling project.

An alternative solution for calibrating cm_per_pixel

Is your feature request related to a problem? Please describe.
The current solution of using meta_real_width is somewhat limiting, eg. when there are height differences because of walls, and often you may know a certain distance within the frame instead. It would be very useful to have an alternative way of getting cm_per_pixel.

Describe the solution you'd like
The TRex GUI already allows you to click and obtain coordinates within the frame, so an easy solution could be a function where you provide 2 coordinates with a known distance between them (or asks you to click on two coordinates, then asks for the distance).

Additional context
This type of feature is common in ImageJ, tracker, and other image processing software which requires the user to specify a known length within the image/video (example from tracker, calibration stick). The function could default to False, and if provided cm_per_pixel could use it instead of meta_real_width. It requires a bit of basic pythagorean calculation, but shouldn't be very hard to incorporate.

Hard limits to time & distance traveled between points of subsequent frames

Problem

1. Tracks jumping around over long distances & times, even though I have set a max speed (which is the max observed speed for my animal).
2. Animals enter & exit from the same hole in my setup, so an animal disappearing into the hole and another reappearing thereafter are connected

Possible solutions?

1. an option for a hard cm limit of how far a track can jump.
2. an option for making tracks always terminate after track_max_reassign_time has passed

Is something like this possible?

tGrabs says video is empty

I cut long videos into 30min pieces with an FFmpeg script and run trex over the shorter videos. On some, trex works just fine, while with others it creates averages but then indicates that the video contains 0 frames with a framerate of -1.
I tried doing the FFmpeg conversion several times with reproducible outcome.
I run the latest trex version.

Any pointers to how to resolve this?
Thank you!

One case:

[23:30:13] 11 / 10 ('/home/stefan/Desktop/2020_clone/allVids/HRM_W2_SW10.MP4')
draw_transform(3x3, CV_32F) = 
[	2941.925537	0.000000	691.170105	]
[	0.000000	2533.762451	685.370728	]
[	0.000000	0.000000	1.000000	]

cam(3x3, CV_32F) = 
[	2945.089600	0.000000	617.255432	]
[	0.000000	2942.825195	682.473633	]
[	0.000000	0.000000	1.000000	]

[23:30:15] Property<float>('cm_per_pixel') = 0.026042
[23:30:15] Copying _original_average (3840x2160) back to _average and preparing...
[23:30:15] cam_scale = 1.000000
[23:30:15] Calculating relative luminance...
[23:30:15] Copying _average 3840x2160
[23:30:15] --- done preparing
[23:30:15] ThreadedAnalysis started (3840x2160 | 3840x2160).
[WARNING 23:30:15 grabber.cpp:1257] Copying average to GPU.
[23:30:15] Copying _original_average (3840x2160) back to _average and preparing...
[23:30:15] cam_scale = 1.000000
[23:30:15] Calculating relative luminance...
[23:30:15] Using OpenGL3.2 (seems supported, 4.6.0 NVIDIA 460.91.03).
[23:30:15] Property<float>('gui_interface_scale') = 1
[EXCEPTION 23:30:15 GLImpl.cpp:94] Cant find application icon 'gfx/TGrabsIcon16.png'.
[EXCEPTION 23:30:15 GLImpl.cpp:94] Cant find application icon 'gfx/TGrabsIcon32.png'.
[EXCEPTION 23:30:15 GLImpl.cpp:94] Cant find application icon 'gfx/TGrabsIcon64.png'.
[23:30:16] Copying _average 3840x2160
[23:30:16] --- done preparing
[23:30:16] Creating queue...
[23:30:16] Done. -1
[23:30:16] pv::File<V7, 8.29MB, '/home/stefan/Desktop/2020_clone/tGrabOut/HRM_W2_SW10', [3840 x 2160], 0 frames, no mask>

crop_offsets: [0,0,0,0]
Time of recording: 'Tue Jan 25 23:30:16 2022'
Length of recording: '00d:00h:00m:00s'
Framerate: -1fps (0ms)

Metadata empty.
[23:30:16] 1/40563 (t+40ms) @ 0.1fps (eta:7:02h load:23ms proc:4ms track:0us save:1us)
[23:30:16] Creating images.
tgrabs[0x883633]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fbab5fce3c0]
tgrabs(_ZN4grab3GUI4drawERN3gui13DrawStructureE+0x516)[0x87c3a6]
tgrabs(_ZN4grab3GUI11update_loopEv+0xab)[0x87e9cb]
tgrabs(main+0x2fbc)[0x7b028c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fbab41b90b3]
tgrabs[0x7ec9fa]
FATAL: Segmentation Fault. Logged StackTrace
Didn't clean up FrameGrabber properly.
Waiting for analysis to be paused...Trying to close file...
[23:30:16] Index table is 56.00bytes big.
[23:30:16] Metadata: '{"meta_species": "", "meta_age_days": -1, "meta_conditions": "", "meta_misc": "", "cam_limit_exposure": 5500, "meta_real_width": 100, "meta_source_path": "/home/stefan/Desktop/2020_clone/allVids/HRM_W2_SW10.MP4", "meta_cmd": " tgrabs -i /home/stefan/Desktop/2020_clone/allVids/HRM_W2_SW10.MP4 -s /home/stefan/Desktop/2020_clone/trex/tgrabsSW.settings -d /home/stefan/Desktop/2020_clone/tGrabOut", "meta_build": "0c4577eed732fe8f2c9d8deee969dc8ee71fac50", "meta_conversion_time": "25-01-2022 23:30:07", "frame_rate": 25, "cam_undistort_vector": [-0.257663,-0.192336,0.002455,0.003988,0.35924], "cam_matrix": [2945.0896,0,617.255432,0,2942.825195,682.473633,0,0,1]}'
[23:30:16] Updating number of frames with 7, index offset 8294543, timestamp 1643178616261898 8294599
Closing camera/video...
tgrabs[0x883633]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fbab5fce3c0]
tgrabs(_ZN3cmn11CPULabeling4Node3Ref13release_checkEv+0x8)[0xeeadc8]
tgrabs[0xeedbcd]
tgrabs(_ZN3cmn11CPULabeling3runERKN2cv3MatEb+0x352)[0xef16c2]
tgrabs[0x857177]
tgrabs[0x858fa5]
/home/stefan/anaconda3/envs/tracking/bin/../lib/libstdc++.so.6(+0xc9039)[0x7fbab45b2039]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7fbab5fc2609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fbab42b4293]
FATAL: Segmentation Fault. Logged StackTrace
Closed.
/home/stefan/Desktop/batchTrex: line 9: 1912626 Segmentation fault      (core dumped) tgrabs -i 

Another, slightly different case:

[23:29:56] pv::File<V7, 8.29MB, '/home/stefan/Desktop/2020_clone/tGrabOut/HRM_W2_SW08', [3840 x 2160], 0 frames, no mask>

crop_offsets: [0,0,0,0]
Time of recording: 'Tue Jan 25 23:29:56 2022'
Length of recording: '00d:00h:00m:00s'
Framerate: -1fps (0ms)

Metadata empty.
[23:29:56] 1/45015 (t+40ms) @ 0.1fps (eta:8:40h load:22ms proc:3ms track:0us save:2us)
[23:29:56] Creating images.
tgrabs[0x883633]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fa9f226c3c0]
tgrabs(_ZN4grab3GUI4drawERN3gui13DrawStructureE+0x516)[0x87c3a6]
tgrabs(_ZN4grab3GUI11update_loopEv+0xab)[0x87e9cb]
tgrabs(main+0x2fbc)[0x7b028c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fa9f04570b3]
tgrabs[0x7ec9fa]
FATAL: Segmentation Fault. Logged StackTrace
terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(4.5.5) /home/tristan/anaconda3/conda-bld/trex_1642026713641/work/Application/build/opencv/src/CustomOpenCV/modules/core/src/ocl.cpp:5655: error: (-215:Assertion failed) u->refcount == 0 && "UMat deallocation error: some derived Mat is still alive" in function 'deallocate'

Overwrites settings selection

Describe the bug
If you have a settings file with the same name as the video, but want to supply a different settings file from the command line the selection is overwritten. Or, to be more precise, the provided settings file is loaded, but immediately after, the identically named settings file is loaded.

To Reproduce
trex -i last_dark -s sleepy-ant.settings

Highlighted output

[14:08:07] ------------------------------------------------------
[14:08:07] LOADING '/Users/roaldarbol/Videos/sleepy-ant.settings'
[14:08:07] ------------------------------------------------------
[14:08:07] Property('individual_prefix') = "ant"
[14:08:07] Property('track_max_individuals') = 5
[14:08:07] Property('track_max_reassign_time') = 0.25
[14:08:07] Property('track_max_speed') = 100
[14:08:07] Property('track_posture_threshold') = 150
[14:08:07] Property('track_speed_decay') = 1
[14:08:07] Property('track_threshold') = 25
[14:08:07] -----------------------------------------------------
[14:08:07] LOADED '/Users/roaldarbol/Videos/sleepy-ant.settings'
[14:08:07] -----------------------------------------------------
[14:08:07] Property('cm_per_pixel') = 0.015625
[14:08:07] Initialized with 8 threads.
[14:08:07] Setting home to '/opt/anaconda3/envs/tracking'
[14:08:07] -----------------------------------------------------
[14:08:07] LOADING '/Users/roaldarbol/Videos/last_dark.settings'
[14:08:07] -----------------------------------------------------
[14:08:07] Property('individual_prefix') = "ant"
[14:08:07] Property('blob_size_ranges') = [[0.1,0.85]]
[14:08:07] Property('track_max_individuals') = 5
[14:08:07] Property('track_max_reassign_time') = 0.25
[14:08:07] Property('track_max_speed') = 100
[14:08:07] Property('track_posture_threshold') = 45
[14:08:07] Property('track_speed_decay') = 1
[14:08:07] Property('track_threshold') = 25
[14:08:07] ----------------------------------------------------
[14:08:07] LOADED '/Users/roaldarbol/Videos/last_dark.settings'
[14:08:07] ----------------------------------------------------

Computer (please complete the following information):

• Operating system: MacOS Catalina
• Build hash: a718edb
• Terminal used: zsh

Additional context
I noticed this behaviour as I had supplied a settings file, edited settings from the GUI, saved config - and then the supplied settings file hadn't changed, however the named settings file had.

Inf in data

Describe the bug
Getting on with the tracking, again, thanks for having made this! This issue is about the data output. After tracking, there's a few Inf observations. First of all, I'm not sure why they are there (I assume it's from frames where the animal wasn't tracked), and why it defaults to Inf rather than NaN. An undesired effect it has is that the observation that follows the Inf's get much higher values, creating some serious outlier observations. For now, I'll just remove the affected observations, but just thought I bring up that it might not be the best default behaviour.
For e.g. speeds or accelerations (anything derivative really), I'd think the best default would be to only supply values when there's actually two consecutive observations, and otherwise to NaN?

Screenshots
An example of this (don't mind the angles):

How to specify additional parameters to TGrabs.

The documentation at https://trex.run/docs/run.html mentions that I can specify additional parameters to TGrabs at the command line. However, how to do this is not specified. Could you update the docs to have a brief example of setting one or two parameters? Thanks!

Option to pause

Is your feature request related to a problem? Please describe.
Working with long videos (and no GPU...), it takes a long time to track videos. If I do it during the day and it's not finished by the time I'm heading home from the office I'm always in a tight spot. It would be great to have the option of pausing the tracking and then un-pausing once home.

Describe the solution you'd like
The option to pause during both TGrabs and TRex tracking.

Describe alternatives you've considered
Currently I'm working around these by leaving my laptop to track overnight. I've considered making use of a cluster al well, but my bash skills are not quite there yet I feel like.

Additional context
I have no clue whether this is easy or super tricky to implement, so if it is a bit of a hornet's nest just spend your time on better things. ;-) Also, I'm finally running good experiments and with TRex I'm generating masses of data - so a massive thanks your way! Cheers!

.csv Export Bugs

These are not critical bugs and I could work around them, but I still wanted to make you aware of them.

• sometimes, an empty .csv file is exported (among others that are fine). It contains the headers but no data.
• sometimes, 2 additional, unspecified empty columns (no header, no data) are output in a .csv file (again, while most other files are fine).
  For both issues I could not find an obvious pattern or trigger. "Sometimes" means in about 5% of the files.

Please let me know which additional information you need, if any.

Collision detection

Hey, thanks for the rad tool Tristan!

I'm trying to detect collisions of tadpoles. Specifically I'm looking to find velocity and displacement caused by collisions.

Looks like we're getting a lot of the data needed to make these calculations (amazing!!), but I'm wondering if you've seen any related use cases implemented already?

OpenCV error during trex analysis

Bug description:

During video analysis of .pv file trex suddenly closes and following message is shown in powershell terminal:

OpenCV: terminate handler is called! The last OpenCV error is:ame 25519/69769 (8.64MB/s @ 1161.51fps eta 47s)) OpenCV(4.3.0) Error: Assertion failed (0 < thickness && thickness <= MAX_THICKNESS) in cv::line, file C:\Users\mooch\Anaconda3\condabld\trex_1603052489921\work\Application\build\opencv\src\CustomOpenCV\modules\imgproc \src\drawing.cpp, line 1782

System info:

• Operating system: Windows 10
• TRex 1.0
• Terminal used: Powershell through Anaconda Navigator

Additional context
It seems the bug is perhaps specific to individual files as I have successfully analysed smaller files. Although the above stopped at this point it seems random as to when the error occurs in the file.

screenshots

Checking for updates automatically

Is your feature request related to a problem? Please describe.
Since users are not usually aware of new anaconda releases, and there is no notification system, it would be good to have an internal update notification.

Describe the solution you'd like
When the user starts TRex for the first time, it should ask whether they want to automatically check for updates. If an update is available, the user is presented with instructions to update. A check can be performed a maximum of once per session, and only when the program is idling.

Additional context
This can be implemented using the github api (https://api.github.com/repos/mooch443/trex/releases) to check for new releases in the TRex repo.

File opening through double-click only half-works

It opens the File dialog, but it does not instantly bring up the file as "selected".

Orientation Swaps

Hello. Thanks for this awesome software!

I am tracking fruit flies and notice that the orientation of the flies (head vs tail) is inconsistent and occasionally rapidly swaps back and forth on a stationary individual.

I've tried altering some settings, including: setting peak_mode to broad and posture_direction_smoothing to true.
None of these changes improved the orientation predictions.

Are there any other settings you would recommend I change?

Thanks again!

Combining Blobs

Hey, I am having an issue where some of my fish are getting separated into two or more blobs, this is putting them below the threshold, which is no issue to just match them manually. But is there a way to assign multiple blobs to one fish, this way I could get a more accurate reading of the center of the fish? I've tried adjusting the threshold for pixel values as I assume it is mostly caused by some dropped pixels in the fish causing it to separate into two blobs but I have had limited success in doing so.

Thank you for all your work, this has been invaluable to our project!

blob_size_range flag affected by junk files within the current directory causing a crash

Describe the bug
I was getting very strange behavior with tgrabs where it failed to process the video if I specified "-blob_size_range" Eventually I realized it only happened when I was running from within a sub directory, not from my home directory, even though in both cases I specified the full path. On further investigation, I noticed that I had a empty file called '2' in my directory (apparently generated by a sloppy batch script I'm debugging.) Obviously this is a very specific problem, but it might reveal some underlying vulnerability.

To Reproduce
-create an empty file within your current directory with "2" as the filename (e.g. "touch 2")
-run tgrabs using the -blob_size_range argument

you get the following error:

terminate called after throwing an instance of 'cmn::illegal_syntax'
what(): Cannot parse array '2'
Aborted (core dumped)

Moving to a different directory, or dropping the -blob_size-range argument removes it. Deleting the empty file, or renaming it (e.g., 2.txt) also fixes it. Interestingly, this also occurs if you have a file called '1' but not '3'. I haven't tried any other numbers.

Computer (please complete the following information):

• Operating system: Ubuntu 18.04.06
• Build hash 0c4577e
• Terminal used: bash

Averaging mode 'mode' does not work in TRex preview

auto-correct does not appear to assign recognized identities

This could certainly be user error, but from what I can tell, visual identification has worked successfully, but auto-correct isn't changing the identities based on the recognition. I updated to the most recent version to confirm it wasn't an older bug.

As you can see in the image, it's very confident that the green fish should be fish0 (which is correct) but it has still assigned it as fish1

Full terminal readout when I run auto-correct, in case that's helpful:
trex-output.txt

I'm using the most recent version of trex (1.1.6). The same thing was happening in 1.1.3 as well.

Thanks!

EDIT:
For background, it's not just individual frames, but an entire tracks (>100 consecutive frames all with the same assigned locID)
This also occurs both in sections (marked yellow or red) with stable tracks for all individuals.

Here's the source video, if you want to try to recreate the issue:
https://drive.google.com/drive/folders/1Uopqn_he3FGuGDsPYrN_IBKJvcu7xlYa?usp=sharing

Parameters used:

-blob_size_range [1,6] -threshold 40
-track_intensity_range [50,150] -track_max_speed 30

This is just a little pilot test to get the pipeline set up, so I haven't made sure the time and distance measures are real-world accurate, but I could do that if that's important.

error in trex/Application/src/commons/common/video/GenericVideo.cpp

calibration error in trex/Application/src/commons/common/video/GenericVideo.cpp line 20:

if(map1.cols == disp.cols && map1.rows == disp.cols && map2.cols == disp.cols && map2.rows == disp.rows)

should be:

if(map1.cols == disp.cols && map1.rows == disp.rows && map2.cols == disp.cols && map2.rows == disp.rows)

Evaluate focal loss with new network architecture

Originally posted by @mooch443 in #9 (comment)

error in easy installation on windows

I got the following error while attempting easy installation on windows 10, through anaconda 1.9:

LinkError: post-link script failed for package trexing::trex-1.0.2-saur
location of failed script: D:*\anaconda\envs\tracking\Scripts.trex-post-link.bat
==> script messages <==

==> script output <==
stdout:
stderr:
return code: 1

()

Coordinate Unit Clarification

Hello -

After tgrabs and trex, I export the tracking results and I am able to read the .npz files. However, I can't seem to figure out the unit associated with the coordinates X and Y. My raw data video is ~1800x1800px^2 and all of my coordinates fall within a box of ~30x30 (units unknown).

I have not changed any parameters associated with the image size from their default value. I've browed the documentation and see a couple parameters (meta width, for example) but can't seem to follow the logic.

Could you clarify the units or the method used to scale the raw pixel positions? That would be helpful for me. It might also be helpful for others if this is more straightforward. Or if this info is already provided, could you point me to it?

I appreciate your help,
Pasha

Extraneous \ when using TRex to start conversion on Windows

Describe the bug
When users try to open TGrabs using the TRex GUI, they might run into the issue that the program cannot be found. This is due to an extra \ in the command-line string.

To Reproduce
Steps to reproduce the behavior:

1. Start TRex from CMD
2. Click on "RAW"
3. Select a video
4. Click 'Open'

Additional context
Anaconda_trex_output_new.txt

Improvements to the FileChooser dialog

Is your feature request related to a problem? Please describe.
The general set of features in the file chooser has to be extended. Currently, especially for TGrabs, there are only limited options available. The rest has to be "known" or figured out by users, as well as where to find the command-line text.

Describe the solution you'd like
There needs to be a more "drag and drop" kind of feel to the dialog. A couple of options should be pre-selected, but most options should be chosen from a normal settings dropdown. These should expand for edits and condense when set. Only TGrabs commands are considered at first, since TRex options can easily be set from within the program.

No OCL devices available.

When I start any tGrabs session I get the following exception message multiple times:
No OCL devices available. Please check your graphics drivers and point the PATH variable to the appropriate folders.
tGrabs still runs through, but it gets killed half the time and I want to exclude this issue as the cause of the killings.

Are you able to provide some additional information? Specifically, where the PATH variable can be found and what 'appropriate' means. Or just by what the issue is caused?

I work on Ubuntu 20.04 with an NVIDIA GPU and supposedly the latest drivers.

Thanks for any pointers you can give me!

update?

Hi,

I am trying to use Trex on a mac (macOs BigSur). While it struggles to find any objects, it seems to look for an update.
However, I just recreated the venv. any idea what is going on?

I very much appreciate any suggestions!

Cheers
Rob

[15:57:23] Property<app_update_check_t>('app_check_for_updates') = automatically
[15:57:32] Adding to queue...
[15:57:32] Added.
[15:57:32] Property('analysis_paused') = false
[15:57:32] Removing frames after and including 0
[15:57:32] Inactive individuals: [3,2,1,0]
[15:57:32] Active individuals: []
[15:57:32] After removing frames: 0
[15:57:32] posture: 0
[15:57:32] physical props: 0
[15:57:32] all blobs: 0
[15:57:32] Range: -1--1
[15:57:32] Clearing from 0.000000 to 1840.000000
[15:57:32] Property('analysis_paused') = false
[15:57:32] Paused.
[ ] 0.00% frame 0/9999 (0.65MB/[15:57:33] Property('analysis_paused') = true
[15:57:46] Saved '/Users/rob/Videos/acclim_raw.settings'.
[15:57:49] [CHECK_UPDATES] It has been a week. Let us check for updates...
[15:57:49] Property('app_last_update_check') = 1633960669915309
[WARNING 15:57:49 GPURecognition.cpp:739] Python not yet initialized. Waiting...
[EXCEPTION 15:57:49 GPURecognition.cpp:1008] ModuleNotFoundError: No module named 'requests'

At:
(2):

[ERROR 15:57:49 CheckUpdates.cpp:302] Failed to retrieve github status to determine what the current version is. Assuming current version is the most up-to-date one.
[EXCEPTION 15:57:49 CheckUpdates.cpp:223] Status suggested the check failed.
[ERROR 15:57:49 CheckUpdates.cpp:227] There was an error checking for the newest version:

ModuleNotFoundError: No module named 'requests'
see terminal for full stack...
<string>(2): <module>

Please check your internet connection and try again. This also happens if you are checking for versions too often, or if GitHub changed their API (in which case you should probably update).
[16:02:03] Writing backup of settings...
[16:02:03] Saved '/Users/rob/Videos/acclim_raw.settings.backup'.

TRex crashes on video "Open"

Hi there - I'm trying to get TRex running on some footage I collected this summer. It seems that once I navigate to a folder, TRex doesn't see the video files in it, and I have to manually enter the filepath (whereas going through folders can be point-and-clicked). Then, when I try to Open the video and get started, TRex crashes. I've found this to be the case for both .mov and .mp4 videos. Here is the splash screen:

MacOS Catalina v10.15.6
2.8 GHz Quad-Core Intel Core i7
Installed using conda create -n tracking -c trexing trex

Use of mask

This is not really a recommendation as much as a question. I've noticed that there is a parameter called video_mask, and when generating the .pv, there are mentions of no mask in the command line output. However, it doesn't appear in the documentation (yet, I'm sure).
So I was wondering how one could make use of a masking feature? In my videos, it could be quite useful to use a mask layer with a few circles (see attached image), so if you could shed some light on it that would be greatly helpful! Maybe also a pointer on how to create a mask.
Thanks!

GUI considerations

Hi Tristan, just came across TRex yesterday and I'm super excited to get started with it - so thanks! I just thought I'd leave a bit of initial feedback which might be useful. I've tried to stay with using the GUI for now, in the hope that it may be feasable also for the non-programming lab members in my lab, so some feedback or feature requests for the GUI implementation here only. :-)

Is your feature request related to a problem? Please describe.
There are two initial things which are complicating the GUI use for now:

1. When using a raw video, if you click 'open' before the average is done calculating, the program crashes.
2. In the same scenario, if you want to tweak parameters, TRex starts computing anew every time a parameter is changed, not waiting until you have changed all the parameters to what you want - in my case slowing the GUI down substantially, eventually causing it to crash.

Describe the solution you'd like
For 1) it would be great to have it stated explicitly somewhere that the average needs to be finished calculating before clicking 'open', and preferably that the button cannot be clicked until that is the case.
For 2) it would be great to have another responsive button to begin the averaging process, allowing the user to change the desired parameters and only then begin computing (e.g. an 'apply parameters' or 'calculate average')

I hope it might be helpful, and if so, I'll provide some feedback when I've worked with TRex a bit more.
Thanks again!

Difficulties finding good parameters

TGrabs seems to work fine for me, judging by the red boxes constantly drawn around the ants in the video (see screenshot below). However, the .csv TRex is exporting always lack a portion of the detected blobs. I have tried finding the optimal parameter set through a systematic testing pipeline, but ended up realizing that some parameter sets 'lose' detections in some tracks/parts of the video and other sets lose detections in other parts (see plots below). I cannot identify any clear patterns of which parameters lead to which accuracy.

Is there a way of exporting all the detection stored in the .pv file to .csv, without caring about identity? This would give me a ground truth and basis for testing other algorithms.

I would greatly appreciate any help you could provide!

---

Snapshot of TGrabs

Plots of some of the best performing parameter sets. GrThr: threshold, match: matching_probability_threshold, reass: track_max_reassign_time, split: blob_split_global_shrink_limit & blob_split_max_shrink, lumin: correct_luminance true

---

Other TGrabs parameters;
approximate_length_minutes = 28 averaging_method = max average_samples = 10 image_invert = true threshold = 78 blob_size_range = [0.0001,0.002] reset_average = false framerate = 25

Other TRex parameters:
auto_minmax_size = true auto_number_individuals = true calculate_posture = false output_min_frames = 4 track_include = [[[3840,14],[2920,18],[2920,2160],[3840,2160]],[[2920,38],[2300,50],[2300,2160],[2920,2160]],[[2300,50],[1850,52],[1850,2160],[2300,2160]],[[1850,58],[1210,68],[1210,2160],[1850,2160]],[[1210,42],[950,62],[718,2160],[1210,2160]],[[950,86],[124,94],[160,2160],[1390,2160]]] track_max_speed = 45 track_speed_decay = 0.7

support for matroska videos

Describe the solution you'd like
We have a lot of videos saved in .mkv containers (vp8, vp9 and h264 codecs). The Matroska (MKV) format is a free, open-standard container format. It would be great if TGrabs could support reading .mkv videos like it supports reading .mp4 videos.

Describe alternatives you've considered
I have considered converting my videos using ffmpeg. For example:

ffmpeg -i FILENAME.mkv -vcodec copy FILENAME.mp4

will do the conversion without re-encoding the video. This would not be necessary if TGrabs would directly support MKV videos.

Additional context
Add any other context or screenshots about the feature request here.

Excluding blobs / petri dish borders

Hi :)

Great software. Is there any way to prevent the sides of the petri dish from being identified as blobs? Or to click on them an "exclude" them somehow?

I've tried:

• increasing threshold (this works but also decreases the detection of the animals on the plate)
• modifying the video beforehand by adding a circular mask to get rid of the petri dish borders (this also works but it'd be very time consuming when using many plates)
• using the categorize feature to train a model by telling the network which blobs are animals and which are just borders, but for some reason it keeps showing me plate borders identified as "isopod" in the "category/skip" tile selector

EDIT: I'm sure this is a user thing. I'm going over the documentation again and again but I'm unable to find a way out so that's why I decided to drop a message.