jarvis-mocap/jarvis-annotationtool

AnnotationTool to create multi-view annotations for the JARVIS 3D Markerless Pose Estimation Toolbox

JARVIS AnnotationTool

This is the official Github Repository for the JARVIS Annotation Tool. To find out more about our 3D markerless motion capture toolbox have a look at our website.

All you need to get started are synchronized multi-camera recordings (check out our AcquisitionTool) and calibration recordings using a simple checkerboard or ChArUco-board. The AnnotationTool has functionallity to extract representative frames from your recordings in a semi-supervised fashion and it can be used to calibrate your cameras.
It then uses live updating reprojection-error statistics to make the process of creating 3D keypoint annotations as intuitive and precise as possible. If you have real world measurements of the animal or object you're annotating (e.g. the length of all finger segments) you can also use those metrics to guide you during the annotation process.

Installing our prebuild packages is easy! Just go to our downloads page and grab the installer for your operating system. We currently support Windows, MacOS and Ubuntu 20.04/18.04. Installers for the current and previous versions can also be found under Releases.

If you want to build the tool yourself here's a step by step guide on how to do it.

Building from Source

Linux

Installing the dependencies

On Debian based systems (e.g. Ubuntu and Mint) run the follwing command:

  sudo apt install cmake git build-essential libxcb-xinerama0 libdouble-conversion-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-gl ffmpeg libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libxcb-xinput0 libpcre2-dev libeigen3-dev libgl-dev zlib1g-dev libfontconfig-dev libjpeg-dev libharfbuzz-dev '^libxcb.*-dev' libx11-xcb-dev libglu1-mesa-dev libxrender-dev libxi-dev libxkbcommon-dev libxkbcommon-x11-dev  

Important: If you're using Ubuntu 18.04 or 20.04 updating CMake is required. See the FAQ section for instructions.

On Arch based systems (e.g. Manjaro) run the following command (Currently there is a problem with building Qt6 on Arch based systems):

  sudo pacman -S base-devel git cmake double-conversion gst-libav gst-plugins-good gst-plugins-base ffmpeg eigen zlib libjpeg fontconfig harfbuzz

Cloning the repository

Next clone our repository with

 git clone --recursive https://github.com/JARVIS-MoCap/JARVIS-AnnotationTool.git

Change to the repositories main directory

 cd JARVIS-AnnotationTool

Building and installing

Build Qt and OpenCV using the provided setup script by runnning

 sh setup.sh

Create and enter a build directory

mkdir build && cd build

Run cmake to configure and build the AnnotationTool

cmake .. && cmake --build . --parallel 8

If you want to create a debian package go to the deployment folder and run (replace XX04 by your Ubuntu Version)

 sh deploy_Ubuntu_XX04.sh

And finally install with (replacing the Xs with the numbers in the package you created)

 sudo apt install ./JARVIS-AnnotationTool_X.X-X_amd64_XX04.deb

If you want to remove it run

 sudo dpkg -r AnnotationTool

MacOS

• Initialize the Xcode tools by running the following command in the terminal:

  xcode-select --install
  

• Install cmake (either using Homebrew or by downloading it from here

Cloning the repository

Next clone our repository with

 git clone --recursive https://github.com/JARVIS-MoCap/JARVIS-AnnotationTool.git

Change to the repositories main directory

 cd JARVIS-AnnotationTool

Building and installing

Build Qt and OpenCV using the provided setup script by runnning

 sh setup.sh

Create and enter a build directory

mkdir build && cd build

Run cmake to configure and build the AnnotationTool

cmake .. && cmake --build . --parallel 8

Windows

• Install a version of Visual Studio (tested on 2015 or newer). The latest versioon can be found here
• Install Git for Windows from here
• Install Strawberry Perl from here

Cloning the repository

Next clone our repository with

 git clone --recursive https://github.com/JARVIS-MoCap/JARVIS-AnnotationTool.git

Change to the repositories main directory

 cd JARVIS-AnnotationTool

Building and installing

Switch to a x64 VS Developer Command Prompt and run the setup batch file:

setup.bat

Create a build directory

mkdir build && cd build

Then run cmake

cmake -DCMAKE_BUILD_TYPE=RELEASE .. -G "Ninja" && cmake --build . --parallel 8 --config Release

To run the AnnotationTool.exe without inistalling it you need to copy all opencv dlls to the build directory!

We currently use the free version Advanced Installer to create our '.msi' installer files. This is not an optimal solution, so if you know how to build a better pipeline to build them please feel free to implement that!

FAQ

Qt does not compile throwing 'CMake 3.21 or higher is required.'

This will occur on Ubuntu 20.04 or earlier. To fix it install the latest cmake release with the following commands.

1. Remove the old cmake install

   sudo apt remove --purge --auto-remove cmake
   

2. Prepare install

   sudo apt update && sudo apt install -y software-properties-common lsb-release && sudo apt clean all
   

3. Get kitware's signing key

   wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null
   

4. Add repo to list of sources

   sudo apt-add-repository "deb https://apt.kitware.com/ubuntu/ $(lsb_release -cs) main"
   

5. Install kitware-archive-keyring package:

   sudo apt update && sudo apt install kitware-archive-keyring && sudo rm /etc/apt/trusted.gpg.d/kitware.gpg
   

6. Add public key

   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 6AF7F09730B3F0A4
   

7. Install cmake

   sudo apt update && sudo apt install cmake
   

Contact

JARVIS was developed at the Neurobiology Lab of the German Primate Center (DPZ). If you have any questions or other inquiries related to JARVIS please contact:

Timo Hüser - @hueser_timo - [email protected]

jarvis-annotationtool's Issues

Suggestions for Sparse projection on multiple cameras

Hi Timo,

Currently our system projects the subject on half of the cameras(4 out of 9) at a time. I have some queries regarding this. We did a first pass of the all the models in the hybrid net and it does not look very good.

1. We have a lot of frames that are unlabeled because the subjects are not in them. Could it be biasing the model? Should we do suppress keypoint instead of leaving keypoints unlabeled to mitigate this issue? Or should we make a different 2D dataset that has less unlabeled frames without the subject?
2. We have a freely moving subject projected on half of the cameras. Do you have an ball park estimate of how many frames we should label or at what frequency we should get segments/frames?
3. We have 5 1280x720 cameras and 3 640x360 cameras. Right now I am downsampling 720p to 480 keeping the ratio and padding black on the 360 cameras. How do you think we should set up the videos? Undersample 720p to 360p or keep it as it is?

Thanks again for your help and this great tool!

-Ken

Speeding up Extracting image features

Hi Timo,
We have been using the Annotation-Tool to create new datasets for training.
The frame extraction takes though particularly long and only uses CPU.
We were wondering first of all if no GPU usage is normal here and if so how could we speed up the Extraction of image features for kmeans which currently takes more than 6-8 hours per video with videos of around 30min lenght.
Thanks a bunch for your help!
Anto

Charuco Calibration Not Working

Hi,

We have shifted from Anipose to Jarvis, but we have calibration in Anipose format for our previous recording. I could not get the CHaruco to get working on Jarvis. I have attached our calibration video and charuco board parameter(under the toml file) here. I was wondering if you could guide me to the problem.

https://drive.google.com/file/d/1ZKJEH7MikZAsPpsFvqcfZMbWz0jbG9RG/view?usp=sharing

Let me know if the link does not work.

Thanks,
Ken

AnnotationTool crashes when using trackpad with mac app

I'm using V1.2, off the shelf, on osx monterey. I haven't tried compiling myself.

When I trey to do annotations with a mac using my trackpad to scross in the annotation window, I get the following error:

/AnnotationTool
2022-10-04T13:43:03: [W] Populating font family aliases took 153 ms. Replace uses of missing font family "Sans Serif" with one that exists to avoid this cost.  (unknown - unknown:0)
[1]    2775 floating point exception  ./AnnotationTool

The same issue does not occur with a mouse/scrolls wheel.

Callibration Unsuccessful in tutorial

I get this error, even though the callibration appears to find the checkerboard. Any idea why?

Calibration unsuccessful! Camera Camera_RBB: Found only 1 valid checkerboards, aborting calibration. Make sure your checkerboard parameters are set correctly.

Here is the debugging image. No debugging images are saved for extrinsic calibration (is that the issue here?)

Error while loading calibration files

Hi, thank you for developing this tool! Looks very promising and the user interface is very smooth.

I am trying a new calibration with some calibration files I acquired. We tried them in .avi format and in .mp4 with different compressions, but we are always having the same error printed in the terminal:

OpenCV(4.5.1) C:\opencv\opencv\modules\videoio\src\cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): D:/luigi/jarvis/jarvis_directory_calib3/Extrinsics/SM. in function 'icvExtractPattern'

It looks like the file extension is missing in the filename, is it possible that that is the reason opencv can't open it? And if yes, what might be the problem? The naming is currently just cameraname.mp4 or cameraname.avi.

AnnotationTool segmentation fault on Create Trainingset (Ubuntu 20.04)

I am running through the tutorial and get a segmentation fault core dumped error when I click Create in the trainingset section. I had the same error on two different machines.

I'm trying to build from source to see if I get the same error currently.

Error when exporting training set

I am trying to export a training dataset from some annotated movies. I am working on a macOS and installed the Annotation Tool available on the website. Running stuff on a mac solved the issues described #1, even though those are still standing on the Windows machine.

I have annotated a bunch of frames and I wanted to generate a training dataset from them. I am working with 3 cameras, I annotated from 1 camera all key points, and from the other 2 I suppressed one in each as it was not visible.

This is the window I have just before the export:

If I export, the tool crashes and I get the following report:

Process:               AnnotationTool [19789]
Path:                  /private/var/folders/*/AnnotationTool.app/Contents/MacOS/AnnotationTool
Identifier:            ???
Version:               ??? (???)
Code Type:             X86-64 (Native)
Parent Process:        ??? [1]
Responsible:           AnnotationTool [19789]
User ID:               501

Date/Time:             2022-05-24 09:05:29.983 +0200
OS Version:            Mac OS X 10.15.7 (19H512)
Report Version:        12
Anonymous UUID:        C279620A-BC34-6883-34DB-45EF599243A2

Sleep/Wake UUID:       59D21917-B77C-4D98-96E5-81734769A3CF

Time Awake Since Boot: 56000 seconds
Time Since Wake:       410 seconds

System Integrity Protection: enabled

Notes:                 Translocated Process

Crashed Thread:        3  QThread

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!'
abort() called
terminating with uncaught exception of type NSException
...

Any way you can help me?

Generating Calibration Board

Hi I am trying to generate calibration board from this site: https://markrmakr.herokuapp.com/charuco_board
However, on the annotation toolbox there is a parameter called side length. I was wondering what that would correspond to if I use a board from the website mentioned.

Also do you know of any better option for generating charuco boards?

Thanks

Inaccurate Calibration

Hi,

I have been trying to run the calibration on a 7x4 checkerboard. However, there seem to be huge discrepancies between cameras. When I reproject some of the key points using the annotation tool, the error goes very high(>30). I was wondering if you could take a look at the calibration and give some suggestions.

https://drive.google.com/file/d/1DgYuZCVzb2UBp2KxiFrPaGtPA5caMG-P/view?usp=sharing

Thanks,
Ken

Calibration Stuck at 99%

Hi,

When I run the calibration, the extrinsic calibration seems to be stuck at 99%. If I close the calibration then it seems to save the YAML file but they do not seem to be right.

Here is an example YAML that was saved:
%YAML:1.0

intrinsicMatrix: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [ 5.0108589414539790e+02, 0., 0., 0., 5.0217136123022357e+02,
0., 3.1740749857840819e+02, 1.8371580711700071e+02, 1. ]
distortionCoefficients: !!opencv-matrix
rows: 1
cols: 5
dt: d
data: [ 1.1327725023237396e-01, -2.4446001902304867e-01, 0., 0., 0. ]
R: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [ 1., 0., 0., 0., 1., 0., 0., 0., 1. ]
T: !!opencv-matrix
rows: 3
cols: 1
dt: d
data: [ 0., 0., 0. ]

Thanks

Memory leak on Annotation

This is on a machine with 32GB ram on ubuntu 20.04 with a dataset with 1000 frames and 5 cameras.

With each frame annotated, my ran slowly fills up. After annotating less than 20 frames, the program crashes with full RAM.

This is the full readout:

$ AnnotationTool 
2022-09-27T16:21:50: [W] Initializing RHI with OpenGL backend (unknown - unknown:0)
Killed

Correct for dropped frames in annotation tool

Hi Timo,

We managed to acquire a dataset using the Jarvis acquisition tool and we experienced for every recording 8-10 frame drops in all the cameras around 30 frames in the acquisition, making them not amenable to processing in the annotation tool directly, as referenced in the issue JARVIS-MoCap/JARVIS-AcquisitionTool#10.

You mentioned this issue would be fixed in a next release of the Annotation tool by adding blank frames in place of the missing ones as detected from metadata (example attached) and we were wondering if there was any progress.

Thanks a lot for your help with this.

metadata.zip

Best,
Anto

Calibration unsuccessful!

When I try to calibrate the extrinsics of the example recording as the website said, it became unsuccessful and gave a hint that "camera pairs [camera_t, camera_ltf]: found only 0 valid checkerboards, aborting calibration. Make sure your checkerboard parameters are set correct. " However, I do not change any parameters and make it untouched and the parameters are correct.

New Videos Not adding

Hi,

I am having a weird issue where I cannot add new segments to existing projects. I have created the segments using same project and the csvs of new labels are identical. I have also updated the yaml with correct segments. Would you know why this is happening? I have attached a sample of csv
I am guessing csv are different but I cannot seem to pin down the differences.
addsegement_timo.zip
I can add one of them as dataset but not other. I need both of them in the same dataset.

Thanks