wml-frc/cj-vision

Layered computer vision platform with C++ UDP networking for data transfer and embedded tools for easy deployment to UNIX-based coprocessors.

CJ-Vision

Layered computer vision platform with C++ UDP networking for data transfer and embedded tools for easy deployment to UNIX-based processors.

Build Status

Features:

• Platform for Computer Vision (CV) applications in C++

• Common OpenCV wrapper functions

• Local debugging with webcam

• Your choice in coprocessor, RPI, Jetson, Tinker Board, etc...

• Uses layering system for programmable design

• Debian UNIX-based deployment (uses Embedded tools for building/deploying to Debian UNIX-based systems)

• Socket networking (sends CV data through custom socket programming https://github.com/wml-frc/UDP_TransferNT)

• Stream MJPEG Video to browser or MJPEG Viewer

Local Setup

Installing

All platforms

1. Clone or download the project.

Windows

1. Download Java (JDK 11 for best support https://www.oracle.com/java/technologies/javase-jdk11-downloads.html)

2. "Desktop development with C++" from VS required for local debugging https://visualstudio.microsoft.com/downloads/

3. Once complete, download or clone the project either into existing code (using the lib as a submodule) or download as is. Then inside the root dir of the lib run gradlew generateFramework build If all downloaded correctly, it should read BUILD SUCCESSFUL

Linux

1. Linux requires the build-essentials (GCC, CMake, etc... along with OpenCV)

2. If you are running a Debian-based machine, then an easier option is to use the included bootstrap for local machines. Use sudo wget -qO - https://github.com/wml-frc/CJ-Vision/blob/master/bootstrap/bootstrapLocal.sh?raw=1 | bash

3. Alternatively you can install it manually

# Update & Install required packages
sudo apt-get update
sudo apt-get install -y avahi-daemon libnss-mdns v4l-utils imagemagick
sudo apt-get install -y build-essential cmake cmake-curses-gui pkg-config
sudo apt-get install -y libjpeg-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libeigen3-dev libxvidcore-dev libx264-dev libgtk2.0-dev
sudo apt-get install -y libv4l-dev v4l-utils
sudo modprobe bcm2835-v4l2
sudo apt-get install -y libatlas-base-dev gfortran

2. OpenCV will also need to be installed on your local machine to run this program. (Bootstrap script installs it for you. Skip this step if used).

• You can install it manually from OpenCV's official page https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html

Mac

1. Download Xcode from the app store https://apps.apple.com/au/app/xcode/id497799835?mt=12

2. You can utilize the bootstrap script using /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/wml-frc/CJ-Vision/cj-vision-2.0/bootstrap/bootstrapLocal_mac.sh)"

3. Or you can install OpenCV manually https://medium.com/@jaskaranvirdi/setting-up-opencv-and-c-development-environment-in-xcode-b6027728003

Coprocessor

Coprocessor Setup

Get started with the coprocessors.

Getting Started

Get started with the library.

_{^{readme written by @CJBuchel, 14/10/21}}

cj-vision's Issues

Store bounding values inside some kind of container

Bounding box wrapper only provides x and y values but no structure for the rectangle/bound or any other values produced through the bounding wrapper.

Azure rebuild

Rebuild azure for 2.0 system

Add modular imshow() so they can have the code output their images when running sim

Possible Implementation of Virtual Layers

I like the idea of layers, and it surprisingly makes sense for a visual-based program. The current setup is chaotic threading, each thread starts one by one and waits for the prior required thread before being freed. But it's mostly a "hope and pray" design.

However, a virtual layering system is more secure in theory. It allows the user to program based on implementing their layer (whatever custom processing they do of the image) into the layer queue and be executed in the correct order. Threading with this logic can be difficult, but orderly asynchronous looping through a queue does exist. And not everything needs to be threaded, too much can decrease performance.

Neural Detector Implementation

Fancy name for Trained Neural Network object detection.

Current detection speeds vary on performance, camera and the process which filters the contours in an image. Training a neural network using images of existing objects (users target object) to get a confidence level to add to the deploy map, would greatly increase the speed. The map would later get deployed and used for detecting objects on the coprocessor.

Should eliminate filtering and contour detection slowing detection down. May also provide better ways to track uniquely shaped targets.

Desktop Simulation has bug displaying image

Create Vision library specific build.gradle, for teams to add the library to their code

wmllib/src/vision/build.gradle files needs to be accessed by the teams separate vision/build.gradle file. Because unlike regular WML, it has binaries specific to external processors

Speed up bounding box thread

Bounding box it too large and holds to many for loops for contour sizing. Reduce or split up into many smaller threads and pass around the values instead. This will also help teams using slower processors. Like pi 2's.

New Documentation for 2.0 version

Fix threads outrunning themselves without using lines in vision.cpp, like mutex locks or waitkeys in the library

Element size unstable over 2 pixels squared

2020 version of the code has issues building with the 2020 tinkerboard plugin. Trying to fix groovy plugin in buildsrc. Will also add to documentation

2020 version has issue building/deploying. (Version Upgrade Issue)

Make Threads continuous

Currently the program spawns a thread and continues with the rest of the process in the main thread. But the second thread it not in a loop, after it has finished it's process it will be joined back up with the main thread...
Effectively i'm creating and destroying threads at a ridiculous pace. Instead it should be creating threads and keeping them alive and passing all the information between threads. E.g image from main thread is passed to the sub thread, processed by sub thread. then the image is passed back.

I'm still working on a way to solve this. Until then i'm using the joining cycle.

inodes full

Issue with constant deployment and replacement of small folders fill up the inodes in linux. Stopping user from deploying causing a failed build
Only encountered twice so far. But It's a good idea to find a solution around this when deploying.

Vision Tracking doesn't start on boot when deployed to coprocessor

COPROC Def not being defined during compile time for coprocessor

Makefile bug due to method of -DCOPROC

The known fix is to move the define before the src files and headers are included in the gcc/g++ maker

Networking Handshake

When the network is initialised the only security it has against errors is a boolean (dataTrue) at the end of the datapacket.

But if in future events where the lib is updated or you change the buffersize. The program cannot detect these changes and will either undershoot the buffer or overshoot if not configured correctly.

Task:
Make a handshake program which sets both server and client to the same buffersize. And checks the version of the program they are running. Before sending data between them.

Need to find a way to thread the output of images.

It's not completely required because the info is sent through network tables at full speed. But for users who want to see the stream at full speed when their testing it on their computers. It would be nice.

Not too sure what the viewing threads issue is tho?? I am quite confused as why every other instance has no problem getting the sink info. But CJ::Output be like, NAH we don't do that here....

Add basic documentation

Program Issue when on the TinkerBoard, (Not running, or not connecting to the camera)

This is a major bug, Or it could be a non issue. It really depends if somethings really wrong or not. But i'm unable to get our previous test cases (e.g 2018 vision tracking) deployed and working. So i'm very confused. Could just be tinkerboard. I know these things have many issues.

Network data not being serialized/deserialized properly

Networking Issue between machines

First issue:
data seems glitchy when it's being sent and received between machines. Often overshoots undershoots buffer. (even when hard set)

Second issue:
If client starts first before server, the program will set to error mode. But once it connects it still displays info in error mode yet still sends data regardless.

Vision doesn't build with current network implementation

This is my bad.
Whoops.

The rebuild of the UDP network has changed the way the library should be linked with the project build.
Should be easy fix

Display() has an issue outputting to camera servers

Not to sure why, i'll do some testing. For the moment teams will have to use output.Putframe() in their main thread. Not to big of a deal. But an issue none the less

Add basic processes.

Makefile Remake

Makefile needs to be remade to increase compile speeds. (Also might not be building on coprocessor from gradle subfolder changes)

nt implementation for networking

Add network tables to the project for the easier coprocessor to rio network for students. (slower, but easier)

Rebuild gradle for better support

the vision.gradle/build.gradle is currently using our old 2019 system when wpi was still prototyping deploy support. 2020 has changed the way that they deploy to pi's/tinkers.
And tinkerboards are outright not even supported anymore. Bit of an issue for curtin. Because chances are we're sponsored next year by them and are required to use them.

So to upgrade i'm probably going to downgrade. In 2018 we used an old but simple system for our vision tracking. Where the bootstrap would download opencv for the pi/tinker, and instead of deploying OS specific binaries, we would just deploy the code and let the pi/tinker do it's thing to run it using it's own OS.

Better support for dev boards. (Not just pi's and tinkers) But also fixes the dependency of gradlerio/wpi's system. So Users don't get chucked in the dark when an update occurs, or wpi change their minds on something. It's own dedicated system. Should also have less build issues, if i build it right.

Downside, the bootstrap process goes from being 10 minutes to an hour. But i think it's fine for a one off thing. Makes it a pain if you make a bunch of backup sd's tho.

CJ-Vision 2.0

New system. Not just the gradle changes. But code changes.

For a prototype it wasn't a bad library and got the job done.
But changes are required to increase performance, usability and efficiency.

• Restructure program using linear style with main include
• create custom datatype that has Mat, coords, and contour vectors inside. And use that to pass to threads. Also allows us to split threads more evenly and get's rid of for loops.
• build shape detection.
• distance via two points.
• distance via blob.
• look into proper object detection using jpg to match objects.