aifarms/avat

Animal Video Analysis Tool (AVAT)

AVAT

The front-end dashboard that provides the ability to annotate videos of livestock for the use in research.

Goals

The goal of this project is to create a tool that is easy to use, has a small learning curve and can scale easily to provide the ability to annotate videos efficiently and at scale. The intuition of this project is gained from analyzing similar projects and analyzing the weaknesses and points of frustration and improving them. Another goal is to have an annotation workflow that is extremely fast and efficient, as there are terabytes of video that needs to be processed at large.

USAGE

Please refer to USAGE.md for a detailed guide on how to use the tool.

Local Deployment (Frontend)

These instructions will get you a copy of the project up and running on your local machine for viewing.

Prerequisites:

You will need to have npm, firefox/chrome and git installed on your system for this to work.

Installing and Building:

First, you will need to clone the repository to your desired location

git clone https://github.com/AIFARMS/pig-annotation-tool

Secondly, you will need to install the node_modules which can be done by:

npm install

Lastly, run start to get the project running on your local machine. The website will be on http://localhost:4000/ This is for the front-end portion of the dashboard. Currently, there is no backend portion to this website.

npm start

If the goal is to simply access the website, then go to the build folder to index.html to access the website.

Docker setup

Currently docker is setup to run the the node project. This will be further expanded onto in the future to include Clowder integration, backend servers, etc

These commands are written for a UNIX based machine running Docker. Support for windows is uncertain

Easy command list:

• To build and run docker container sudo docker build --tag avat:latest . && sudo docker run -it --rm -v ${PWD}:/avat -v /AVAT/node_modules -e CHOKIDAR_USEPOLLING=true --name avat -p 3001:3000 avat && sudo docker start avat && sudo docker ps -a | grep avat
• To stop and remove container sudo docker stop avat && sudo docker rm avat
• To remove pre-existing container and build new container sudo docker stop avat && sudo docker rm avat && sudo docker build --tag avat:latest . && sudo docker run -it --rm -v ${PWD}:/avat -v /AVAT/node_modules -e CHOKIDAR_USEPOLLING=true --name avat -p 3001:3000 avat && sudo docker start avat && sudo docker ps -a | grep avat

Flask server

There is also support for running this off of a flask server. This will be further expanded on into the future to include any external models and such.

These commands are written for a UNIX based machine running Docker. Support for windows is uncertain

Setup a virtual environment and install the proper dependencies from backend/requirements.txt. Then change current directory to backend and type in flask run. The website should be up on http://127.0.0.1:5000

Built With

FrontEnd

• React - Main Front-End Framework
• React Bootstrap - UI Library
• react-player - Video Player
• fabric.js - Canvas renderer

Authors

• Pradeep Senthil pradeepsen99 - AVAT
• Ana Lucic alucic2 - Data Conversion Scripts
• Zimu Li neuroZ68 - Usage Documentation

avat's Issues

Video playback disabled

The video playback functionality is currently broken and thus disabled.

Currently the issue is that while the video is playing, rendering each of the frames is too taxing on the system resources which causes severe lag and stuttering.

Delete function do not delete polygon

Ensure all variables used in state management are used in rendering

More of a sanity check but double checking to make sure that all of the state variables are correctly being used to render only UI components. If variables are found not being used for state then it needs to be converted to normal variables.

Add more automated selenium testing

Have the ability to input multiple video types beyond traditional mp4

Output JSON is not reusable

Operation:
1>Upload Images and JSON.
2> Do any changes.
3> OUTPUT
4> Reupload images and OUTPUT.
Issue: polygon on first image flash appeared, then gone.
Issue: the polygons do not match randomly at some frames with the pig contour (first uploading does not have this issue).

***Use images and JSON in "debug-2" folder.

Allow input beyond .jpeg and .png

Allow support for more types of image data such as matlab output files, depth sensor files, etc.

Potential way is to create a module that converts the outside formats into jpeg internally and shown to the end user.

Video issues

Suggestion： Make the default frame number to be 15？

Issues：
Use data in the "debug-3" folder.
Polygon and table do not update when clicking the next frame.

Outdated documentation

Documentation for code is outdated and needs to be updated.

Frame number issue

The annotated frame should be 0, 15, 30….. But they were displayed at 1,16, 31….which does not match with the polygon. Here is the sample video and annotation file for debugging.https://uofi.box.com/s/b23hx6s0vlw6rc78gun2qurdahwgelog

Update frame number and table with play function

Frame number and table content is not updated along with the playing of the video.

Image distortion/stretching with Image set upload

With image set upload option, images that are smaller in size and resolution appear to get stretched/distorted. Suggestion: add in the instructions that the images should be of certain size/resolution or adapt the tool to accept smaller size and resolution without distortion.

Add in new table controller and logic

The react-bootstrap-2 table functionality was not sufficient in scalability due to some bugs present and also the main contributor to the lib had stopped making changes to the codebase. Switching to react-table provides flexibility and the library is well maintained.

Export to JSON generates malformed output

After importing in annotations, doing annotations and exporting it, the resulting JSON output is malformed. The JSON sometimes contains null values or is entirely empty.

Export file miss annotation information

I send you the JSON file on slack

Migrate all state management to Redux

Video metadata resolution stuck at 1280x720

In react redux the values stored under horizontal_res and vertical_res are always set to 1280 and 720 respectively. This might be caused during video initialization.

Clean up handleSetCurrentFrame

The function handleSetCurrentFrame under main_upload.js is an important function in making sure that the current frame values are setup properly. Currently there is a lot of redundancy, this needs to be cleaned up to allow for image and video data. Also special consideration needs to be made in regards to multiple video streams and how to manage that internally.

Export json do not export segmentation

Annotations don't save when added to single frame and exported

Issue:

When the user uploads a video, the annotations added to the screen are not saved iff the user does not go to another frame.

Cause:

The current callbacks for a mouse move, up, down and object move events are supposed to call the save_data function. For some reason, the function call does not go through leading to a mismatch of the data and the current state.

Fix ideas:

• There is a hacky workaround that could be used to save the data by diffing the changes and seeing which annotations need to be redrawn, however this is very prone to breaking upon any changes to the annotation logic. It is better to simply address the function callback issues.

Metadata update - Data Type and Image filename

The JSON output file should include what data type each frame/image is (i.e. Video or Image data). If image data is used then the specific file name of the image being annotated should be saved.

Add a new component that solely takes care of the video and <img> elements

This element should dynamically switch between the None element or the react video player depending on what input type was chosen.

Default column configuration setup

Currently in AVAT, the user is responsible for uploading a configuration file for columns. Easy improvement here is to add in a default config file and present the user with the option to override.

Annotation export format streamline

Currently the JSON exported by AVAT is a simple JSON dump into a massive array of length=frame number. This is inefficient on storage and cumbersome to work with. Instead, we can streamline this by using a nested json approach where each array element has a frame identifier indicating the frame it's from and the redux store smartly grabs the data upon upload.

Potential data format:

{
  frame_number: 1
  Annotation_type: “bounding_box”
  Dimensions: ….data dump of coordinates for box/segmentation
}