appsilon/mbaza

Save 99% of Your Time Classifying Camera-Trap Footage. Completely Free.

Mbaza

A desktop application which allows bioconservation researchers to classify camera trap animal images and analyze the results.

Installation

Visit the latest release page and download Mbaza installer for your system (X.Y.Z will be the version number, e.g. 2.0.0):

• Windows: Mbaza.AI.Setup.X.Y.Z.exe
• Linux: Mbaza.AI-X.Y.Z.AppImage
• macOS:
  • ⚠️ Mbaza requires additional setup to run on macOS. Read the instructions below.
  • x86-64 (Intel): Mbaza.AI-X.Y.Z-mac.zip
  • ARM64 (Apple M1): Mbaza.AI-X.Y.Z-arm64-mac.zip

macOS

By default macOS doesn't allow running app files downloaded from the Internet and it will display a "file is damaged" error. To fix it:

1. Extract the app from the ZIP archive.
2. Open a terminal in the directory with the extracted app.
3. Run the following command: xattr -cr 'Mbaza AI.app'.

Links

• Landing page: information, testimonials, news.
• User guide / Mode d’emploi (v1.2.1)

Development

If you are seeking to get involved in the development of Mbaza, take a look at our contributing guidelines. There you'll find useful technical information about app architecture, ML models, npm commands, release process, and more.

License

AGPLv3

mbaza's Issues

A number of improvements to Explore page

• improve file selection UI -- add explanation, put all into a nice segment
• add tooltip with info what animals are considered rare
• add note "To save the plot to disk, hover on the plot and use the button with a camera icon"

Add Gabon model to the repository

At this point we only have the Serengeti model in the repo. We need to add the Gabon model as well.

Please add a second directory in models and follow the structure from https://github.com/Appsilon/wildlife-explorer/tree/master/models/serengeti

Make a usable sidebar

Acceptance criteria:

• sidebar visible in all the routes
• sidebar look active when in a specified page

Facilitate predictions directory choice

Determine license and whether the app will be open source?

Image Exif data update

A nice-to-have option where the user can tell the application to update the images classified by the model with location data (not always present in the images, because of e.g. faulty GPS units in the camera traps) and perhaps species names – to be discussed with Robbie.

Message from Robbie: "The only other thing I can think of is to check the date and time are correct. Often people make mistakes when they set the camera, and we check this by taking a photo of a blackboard with the time and date of deployment written on it. I've made a simple GUI that loads the first ten images of the 'check' the user then slides to the image with the blackboard and checks it matches the time in the exif. This will be easier to understand if I send you some screenshots, will do now. Also, if this is too tricky then please don't treat it as a priority!"

Here's a proper description of the problem we might want to resolve in this issue: https://docs.google.com/document/d/1Tv0mR6fyTrHEwgaI7yh27JNYFzErtIbc2dXYD-Om9ho/edit#heading=h.qjod6ia0n33w

Translation feature

Starting with French – Robbie, then Spanish – Tadeusz.

Classifier doesn't work under Windows

Steps to reproduce

• Install the app on a Windows machine (build the installer with yarn cross-env DEBUG_PROD=true yarn package-win or download from releases).
• Go to the "Detect animals" page.
• Choose input and output locations (optional - the problem appears even without it).
• Click on "Find animals!" button.

Expected behavior

The app runs the classifier and streams the progress to the user.

Actual behavior

No output is shown. The debug console shows Cannot find module '../build/Debug/pty.node' and ...d03948eb.tmp.node is not a valid Win32 application errors (only if built with debug enabled). See log and screenshot.

Automatically determine an appropriate number of PyTorch workers

The option --pytorch_num_workers must be set to 0 on Windows; otherwise the classifier script fails in a cryptic way (see #90). A better solution would be to automatically determine a proper number of workers based on the system (keep the option so the user can overwrite this setting).

In addition it would be good to research what setting is actually appropriate and document the choice. Does having more workers make the inference faster? Does it come with increased memory usage (it might lead to crashes, see #74) or other drawbacks?

Fill settings, app name, app icon

How to set icon: https://www.electron.build/icons

[?] Gather usage stats and app errors

Gather statistics about app usage.

We need this to be able to show that app is having active users and to measure if it is being used, and gather info to improve the app.

User should be informed about this during installation.
Stats should be submitted regularly when user has an internet connection, and when there is no internet connection they should be collected and submitted to a server when connection is restored.

Determine what to collect specifically.

Ideally, we should also submit information about app crashes and errors.

[1h] Add logos of institutions + update text on the home page

• Add logos of institutions provided by Robbie on the home page. Write the word ECOFAC6 under the EU flag in normal font.
• update the email address on home page from tadeusz to [email protected]

Fix security warning in console

When you run the app, devtools console displays a warning as below. We need to fix it.

Electron Security Warning (Insecure Content-Security-Policy) This renderer process has either no Content Security
Policy set or a policy with "unsafe-eval" enabled. This exposes users of
this app to unnecessary security risks.

For more information and help, consult
https://electronjs.org/docs/tutorial/security.
This warning will not show up
once the app is packaged.

Set icon for Mbaza that works in all places

Acceptance criteria

• new icon is displayed in the app, on the desktop shortcuts, in system bar
• works at least on Windows and Linux, and ideally also on Mac (paste screenshots to streamline code review)

More info

Documentation is here, but I think it's not exact https://www.electron.build/icons. Instead you may need to add all files as in the resources folder in https://github.com/electron-react-boilerplate/electron-react-boilerplate/tree/master/resources (I deleted most of them at some point because from the docs it seemed they were not necessary)

Final design approved by Robbie below.

JPG: https://drive.google.com/open?id=15e-1JaNTV89qiE2qwpp-oXbknief4Qtv
SVG: https://drive.google.com/file/d/1dS5pEkiijpPOzpuBtHzp94Z60UnstJ4U/view?usp=sharing

Show classification progress

When the classifier script is run from the terminal it displays a text-based progress bar. Using node-pty a pseudo-terminal was allocated for the script and that progress bar could be directly displayed in the UI. Due to problems with node-pty (#88), PR #89 uses Node's built-in child_process module instead. However it does not allocate a pseudo-terminal and thus the classifier script only shows the beginning and end of the classification process. A different solution is needed to display progress to the user.

Acceptance criteria

The progress of the classification process is clearly presented to the user.

Rare species counter

Basic functionality

• for the count of rare species, use the list of species from IUCN Red List, as proposed by Robbie, that is:

• Gabon: Cat_Golden, Chimpanzee, Duiker_Yellow_Backed, Elephant_African, Gorilla, Leopard_African, Mandrillus, Pangolin

• Serengeti: cheetah, elephant, giraffe, hippopotamus, hyenastriped, koribustard, leopard, lionfemale, lionmale, rhinoceros, secretarybird, topi, vulture

• provide an explanation in an infobox (available on hover), or "about" section on which species are selected and why. The above were chosen since they are endangered, vulnerable or near threatened (see this Slack thread)

Extended functionality

• on click in the "rare species found" change the selection of the species viewed in the species dropdown to the relevant rare ones.

Log the output of the python script in a text box

[2h] Classification script should take image locations from a national grid file

1. The app checks if a stations grid file is expected in models/biomonitoring_stations.csv. If it's not present, show a warning message and assume missing locations in output.
2. Classification script adds new columns grid_file_lat, grid_file_long (filled based on the grid file and station number) and coordinates_lat, coordinates_long (from EXIF coords if present, else from grid coords if present)
3. Explore page uses the new coordinates_lat, coordinates_long

File is available from @marekrogala

Data explorer – map

The map should show points where the animal was present. The points’ size should be scaled according to the number of individual images taken within the chosen time period. Upon clicking any given point, a window should pop up with related pictures and descriptions. The descriptions should come from the underlying CSV/Excel/data structure – i.e. species name, number of individuals, etc.

There should be an option to easily save the map – .jpg or .png for copying into a word processor.

Classifier dies when processing a larger data set

When running the classifier on a "small sample of data" provided by Robbie (628 images), the process runs quite slowly and dies on the second batch. I was using my work laptop with 16 GB of RAM. @marekrogala suggests the images might have unnecessarily large resolution and scaling them down could possibly help.

[2h] Classify page: check the number of images to be classified and show a warning if there are many

If you're classifying >1000 images, show a warning message that this can take a long time.

App auto updates

App should auto update when connected to internet. This is simple to configure

electron-react-boilerplate/electron-react-boilerplate#1289

https://www.electron.build/auto-update

Packaged models for Windows don't work

Steps to reproduce

• Download packaged models from Windows from Drive (Appsilon internal).
• Unpack the zip archive under Windows and attempt to run it from the command line.

Expected behavior

The classifier runs and outputs a CSV file with classification results.

Actual behavior

The classifier runs but hangs. Multiple warnings are emitted. The process doesn't use much CPU at all. The process can be killed by hitting Ctrl+C. Note the usage info in the end - it seems that the classifier is somehow run again without arguments.

By redirecting the warnings it is possible to see, that the inference actually starts and outputs the progress bar.

Explore page: Display results on the map

Similar to #5 , but display results from the file on the map (which was implemented in #40 but does not display observations yet)

Acceptance criteria

• sites with observations are displayed as points on the map
• The map should show points where the animal was present. The points’ size should be scaled according to the number of individual images taken within the chosen time period.
• when you click at a site, you can see details about the observations in this site and a list of photos from this site that you can browse, The descriptions should be species name, number of individuals, etc
• there should be an option to easily save the chart output – .jpg or .png for copying into a word processor.

Introduce Definiton of Done

The UI doesn't notice if the classifier dies

If the classifier dies while processing images (e.g. as in #74), the UI should notify the user. Right now, from the user's perspective it looks as if the progress stalled.

Downscale images before feeding them to the classifier

The classifier crashes when processing larger data sets (#74). Downscaling the images before feeding them to the classifier should hopefully solve the issue.

[?] Run the app with model on Windows + Windows application installer

Acceptance criteria

• Installer for Windows is a must. We should also have an installer for Linux to make debugging easier.
• Installation steps are: run installer, then unzip models environment into application folder - this adds a models directory, and contains all model files in models/serengeti, models/gabon, the stations grid in models/stations_grid.csv and models runner in models/runner
• document build process for this zip file for both systems and create scripts where possible
• Document installation process for Windows and Linux in README

Image classification flow

• choose images data folder
• upload Excel/CSV with previously classified images (in case someone is adding additional data to an existing project), we have to account for a situation where a user adds data to this existing project but from a new location
• choose where to save the files
• Run mock Python process
• observe that process finished (with success or error)
• observe progress from stdout (propose communnication format)

Set up CI with app packaging

Base application

• decide what technology to use
• create a base app code structure without any functionality
• add some UI framework (bootstrap 4? fomantic ui?)

[4h?] Classify page: Run ML model on sample images dataset

Acceptance criteria

• Make sure that the actual Serengeti model is ran on images from the provided folder in dev version and that a correct output is produced, in the same format as used in #5 .
• test this on test images dataset from Robbie
• progress should be displayed correctly and be informative, and the entire flow needs to be helpful to non-technical user
• test Explore page and resolve any issues in Explore page that show up or create issues for them
• show error if models runner or model files are not present - inform that they need to unpack models in the application directory

Add an offline map of Gabon

[3h] Recover date also from folder structure

Acceptance criteria

1. Classification script extracts date from the folder structure as a new column path_date.
2. It outputs a column date, which uses uses for each photo path_date if present, else exif_datetime if present, else null.

[1h] Package the app and send to Robbie for tests

Icon doesn't show in the application top bar when installed

Steps to reproduce

• Package the app with yarn cross-env DEBUG_PROD=true yarn package.
• Install the the package created in release/ directory.
• Run the app.

Expected results

MBaza icon is displayed in the top left corner (within the application window).

Actual results

A placeholder "image failed to load" icon is displayed instead. The console log says the the icon file couldn't be found.

[2h] Classify page: Add Gabon model + add model selection option

Acceptance criteria

• user can choose between Savannah model and Gabon model for classification
• both models work and are used and produce output in the expected format. @swiezew can assist with this

Make it so that we can fit in the 2GB artifacts limit on Github (we can upload artifacts to GCP, we have credits for this project there!) and reset spending limit back to $0

Store partial results as the classifier runs

Right now the classification results are only stored after the process finishes for the whole data set. If the process is interrupted (e.g. user kills it or it dies due to #74), all results computed so far are lost. This could be especially bad for large data sets (hours to process).

A possible solution would be to add --append option to the classifier (mutually exclusive with --overwrite, enabled by default). The script would first read the output file and only process the images which are not there yet; partial results could be appended after each batch of images is processed.

Explore page: Show all observations and not first 100

We have limited the results to first 100 in development, but before release we should fix this and make sure it all works with a large (~30K) number of photos

Add basic map to exploration page

@wojdziw I think this is a great task for you :)

Goal: Display photos on a map - the more photos the bigger point on the map. Map needs to work offline.

These looks like good resources how to do that:

• https://digitalki.net/2017/12/13/offline-maps-with-mapbox-gl-js-and-electron/
• https://github.com/osm2vectortiles/offline-maps

Potential alternative in case of problems with the above: https://github.com/kothic/kothic-js

We need map tiles for Gabon, or ideally for entire Africa or Earth.

Explore page: add results filtering

Acceptance criteria

• user is able to filter results based on the values in columns (date from-to selection, species multiselect, site/"check" multiselect).
• the filtering can be reset
• the filtering applies to all other places on the explore page

[2h] Explore page: Add a toggle that applies a certainty threshold filter to results.

1. Add a toggle in the filters area saying ">= X% certainty" (X to be defined)
2. if toggle is enabled, consider non blank images with score_1 < X as blank

[8h] Explore page: show photos

Show photos

• add photos view - thumbnails + option to enlarge and show predictions for a single photo
• when you click at a site on the map (or hover, this can be a tooltip), you can see details about the observations in this site and a list of photos from this site that you can browse, The descriptions should be species name, number of individuals, etc

[4h] Make sure all texts are using the translation feature, get French translation from Robbie and add to app

The app needs to be available in French.
We already have the basic translation feature, but many texts are not using it.

• use file backend as in https://phrase.com/blog/posts/building-an-electron-app-with-internationalization-i18n/ instead of a hardcoded json
• make sure all texts use the translation feature
• prepare a file to be translated to French
• get French translation from Robbie and add it to the app

GUI/Excel for choosing images/creating a project

We will most likely start with a simple Excel template to provide basic project information (locations, cameras, latitude and longitude) and build on that in the following weeks.

Explore page: display data from the opened file on the chart and arrange layout

The core feature of the app is the "Explore" page. It should display data from a chosen file. User will open file in this format (this file is artificially generated by @swiezew, but the format is the same):
output_corrected_groups.tar.gz

Acceptance criteria

• show basic statistics about the data like number of animals, dates range, number of sites
• show data from the loaded file on the simple chart (at the minimum show the number of animals at a given location across time)
• layout on the Explore page is arranged so that it's nice, organized and usable. Add placeholders for controls that will be needed in #46 that will filter data based on the values in columns (date from-to selection, species multiselect, site/"check" multiselect).

[2h] Customize menu commands

Now these commands are straight from the boilerplate. Replace some of them with relevant actions or links and generally make sure they make sense

Disable Node integration in production

PR #83 enables Node integration both in development and production (previously it was only enabled in dev). In app/main.dev.ts (stripped for clarity):

  mainWindow = new BrowserWindow({
    webPreferences: { nodeIntegration: true }
  });

This fixes Unable to load preload script: 'renderer.prod.js' error which appeared only in production and made the app display nothing but a blank screen. However, enabling Node integration has security implications. It would be better to understand the underlying issue and fix the problem without this hack.

Acceptance criteria

• Node integration is disabled in production.
• Production app works without issues, both when:
  • run with yarn start (this might be faster to test),
  • run after packaging with yarn package and installing (additional issues might surface).

[2h] Explore page: map should show circles with size relative to how many species were detected in this spot

This is the most relevant from the bioconservation perspective.