beealarmed/beealarmed

A camera based bee-hive monitoring that identifies and tracks bees. It also identifies bee characteristics with a neural network, such as pollen packages, wasps, varroa mite infestations or bees trying to cool the hive.

BeeAlarmed 🐝🐝🐝🎥

A camera based bee-hive monitoring that identifies and tracks bees. It also identifies bee characteristics with a neural network, such as pollen packages, wasps, varroa mite infestations or bees trying to cool the hive. Therefore, it utilizes tensorflow, opencv, filterpy and others.

Here is an example video! (YouTube.com)

This is the result of a private project to develop a camera based bee monitoring system. It was initially made to run on the JetsonNano, but it is not limited to it.

What it can

It can:

• count bees entering or leaving the hive
• detect and count varroa infected bees
• detect and count bees that are cooling the hive
• detect and count bees with pollen
• find and count wasps

What it does

It monitors all bees and uses a neural network to detect the above mentioned characteristics, counts all findings and sends them to the ThingsNetwork using LoRaWAN. If you don't like to use LoRaWAN you can change that easily in the code.

How does it work

Each frame provided by the camera (or video file) will be processed to identify the bees in the image. The found bee positions will then be used to reconstruct the bee movements and paths using kalman filters. The paths are then used to count the bees entering or leaving the hive. Where the hive entry is on the upper part of the filmed pane and the exit is on the bottom.

Each detected bee will then be cut from the image, rotated and forwarded to a neural network for classification.

A bee carrying a packet of pollen

A bee infested by varroa mites.

A bee that is coooling the hive. To do so, the bee stays stationary and moves its wings to move fresh air into the hive

A typical wasp that can be found stealing honey from the hive

The neural network performs simple classification tasks to identify bees with pollen, varroa mite infected bees, bees cooling the hive or wasp and counts them. The results can also be visualized. The neural network runs in a separate process and the results may be too late to visualize them, as the bee may have already left the filmed area. But that depends on the performance of the used system. On the JetsonNano, you will probably see that phenomena once in a while.

Below you can see that two bees were detected cooling the hive (green dots), one of them is even infected by varroa (red dot).

Look close, you can see the varroa mite.

You want to give it a try?

Just check out this repository and install the dependencies:

git clone https://github.com/BeeAlarmed/BeeAlarmed.git
cd BeeAlarmed
git submodule init
git submodule update
pip3 install -r requirements.txt

Download the pre trained neural network and unzip it to ./SavedModel: SavedModel_20201121.zip

wget "https://raspbee.de/FH/Github/SavedModel_20201121.zip" -O SavedModel_20201121.zip
unzip SavedModel_20201121.zip -d ./SavedModel

Note: I will publish the dataset and neural network once I finished working on them. The dataset is currently based on manually selected and annotated images (>6000).

Download the example video file: Sample Video (314 MB)

More videos can be found at YouTube: YouTube Channel: Raspbee - Videos

wget "https://raspbee.de/FH/Github/cooling_varroa_small.avi"

Now start the monitoring system by calling:

python3 main.py --video=./cooling_varroa_small.avi

This is what you should get:

• Blue dot: Pollen
• Red dot: Varroa mite
• Green dot: Bee is cooling the hive
• Black dot: Detected a wasp

Please note: The 'cooling_varroa_small.avi' video file was compressed and thus lost quality compared to the original camera output. The neural network is trained on the original data and may produce false detections with other data. I'll try to link an uncompressed file as well, but it will be large. Detection and tracking of the bees will not be affected by this.

JetsonNano

On the JetsonNano you have to follow the steps above, but additionally you should install a more lightweight desktop environment. The default one consumes roughly 1.5GB of the available 4GB. I suggest to install the lubuntu-desktop or similar. Here is a guide on how to do it: https://www.zaferarican.com/post/how-to-save-1gb-memory-on-jetson-nano-by-installing-lubuntu-desktop

Configuration

You can modify several parameters in the Config.py file.

You tried & liked it. Now you want to build it yourself?

Below you can find a rough explanation on how to build your own camera system. A more detailed description will follow in the beginning of the next year. I'm currently working on a more robust wooden solution.

How does camera system looks like?

The below shown camera-system is the first prototype made out of cardboard. But it worked quite well!

The schematic is shown below. The system is simple. A camera films the bees on a pane when they are entering or leving the hive.

The inside looks like this:

As you can see the filmed area is green and illumined by two LED-stripes in the front and back.

Remarks

• The neural network was trained with data that was collected with the above mentioned camera system. If you build your own system bee monitoring system, you may experience different results due to different camera angles, resolutions, sharpness, background color and so on. In this case you should create your own dataset and train the neural network with it. Once complete, I'll provide my scripts and the dataset that I used to train the network.

What is still to do?

• Optimize the neural network with live data. Currently everything is based on video material that was captured at the end of the year.
• The neural network is trained on images of the size 75x150 pixels. The results may be better with larger images, but the impact to the performance has to be evaluated.
• Optimize tracking. Some bee tracks are still lost, especially when bees fly around in the camera system.
• Camera input was not yet implemented. Just a minor thing, but I cannot test anything, because the bees are already hibernating.
• Write documentation

beealarmed's Issues

keras unable to load .PB file (tf Saved model)

I'm newbie to this but:

• Firstly, seem like there is problem with codec, which could be fix by adding syntax .encode("utf-8").
• Secondly seem like your code require .h5 file instead of .PB(in /SavedModel). So I can't load your pre-trained model
• Thirdly if I force it to load .PB file by syntax _model = tf.saved_model.load(get_config("NN_MODEL_FOLDER").encode("utf-8")). It'll cause error on line _model.predict_on_batch(tf.convert_to_tensor(imgs)). It said there are no attribute 'predict_on_batch'.

At this point I have no Idea about what else could I do because seem like it up to your SavedModel.zip

Issue with codec:

Issue with .PB file:

Issue with Predict_on_batch:

what video camera did you use?

Hello Fabian!

Very interesting project and I would like to deploy it for my apiary on Jetson Nano. Could you please share info about architecture and camera characteristic and manufacturer?

Thank you!
Best regards,
Vadim

tensor flow version?

I'm trying the saved model, and get an error:

tensorflow - WARNING -        SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are sa
ving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in th
e SavedModel directory.

what tensor flow version was used to create the saved model? what save method was used?

Overtrained?

Hey Fabian & team. I'm trying to adapt and run the app.
I got it working with the provided video, but with custom inputs seems like the detection rate and precision drops significantly.
Seems to me like it may be due to green background or maybe video resolution

upd. looks to be more sensitive to green background

Screen.Recording.2023-07-21.at.10.12.51.mov