tryolabs/luminoth

Deep Learning toolkit for Computer Vision.

---

(Jan 2020) Luminoth is not maintained anymore. We recommend switching to Facebook's Detectron2, which implements more modern algorithms supporting additional use cases.

---

Luminoth is an open source toolkit for computer vision. Currently, we support object detection, but we are aiming for much more. It is built in Python, using TensorFlow and Sonnet.

Read the full documentation here.

DISCLAIMER: Luminoth is still alpha-quality release, which means the internal and external interfaces (such as command line) are very likely to change as the codebase matures.

Installation

Luminoth currently supports Python 2.7 and 3.4–3.6.

Pre-requisites

To use Luminoth, TensorFlow must be installed beforehand. If you want GPU support, you should install the GPU version of TensorFlow with pip install tensorflow-gpu, or else you can use the CPU version using pip install tensorflow.

Installing Luminoth

Just install from PyPI:

pip install luminoth

Optionally, Luminoth can also install TensorFlow for you if you install it with pip install luminoth[tf] or pip install luminoth[tf-gpu], depending on the version of TensorFlow you wish to use.

Google Cloud

If you wish to train using Google Cloud ML Engine, the optional dependencies must be installed:

pip install luminoth[gcloud]

Installing from source

First, clone the repo on your machine and then install with pip:

git clone https://github.com/tryolabs/luminoth.git
cd luminoth
pip install -e .

Check that the installation worked

Simply run lumi --help.

Supported models

Currently, we support the following models:

• Object Detection
  • Faster R-CNN
  • SSD

We are planning on adding support for more models in the near future, such as RetinaNet and Mask R-CNN.

We also provide pre-trained checkpoints for the above models trained on popular datasets such as COCO and Pascal.

Usage

There is one main command line interface which you can use with the lumi command. Whenever you are confused on how you are supposed to do something just type:

lumi --help or lumi <subcommand> --help

and a list of available options with descriptions will show up.

Working with datasets

See Adapting a dataset.

Training

See Training your own model to learn how to train locally or in Google Cloud.

Visualizing results

We strive to get useful and understandable summary and graph visualizations. We consider them to be essential not only for monitoring (duh!), but for getting a broader understanding of what's going under the hood. The same way it is important for code to be understandable and easy to follow, the computation graph should be as well.

By default summary and graph logs are saved to jobs/ under the current directory. You can use TensorBoard by running:

tensorboard --logdir path/to/jobs

Why the name?

The Dark Visor is a Visor upgrade in Metroid Prime 2: Echoes. Designed by the Luminoth during the war, it was used by the Champion of Aether, A-Kul, to penetrate Dark Aether's haze in battle against the Ing.

-- Dark Visor - Wikitroid

License

Copyright © 2018, Tryolabs. Released under the BSD 3-Clause.

luminoth's Issues

Support training with multiple GPUs in same machine

Test Dataset

Failing multiple overlap test

Related to #29

It seems as if we're always setting the last foreground proposal as foreground and ignoring the rest, instead of setting the proposal with the highest IoU as foreground and ignoring the rest.

Your website is exceptionally unreadable because of poor font color choices.

Contrast, please. ^_^

Add batches/sec & secs/batch to summary

Test RPN Anchor Target

• Test base case, one background, one foreground, one background (ignored using minibatch of 2)
• Test empty case, no matching proposals, but it should assign at least one anchor.
• Test positive clobbering behaviour
• Test with anchors outside the border to check that when filtering inside anchors and using unmap the order or anchors is correct.

Test RCNN Proposal

• Test class NMS
• Test limit per class
• Test image clipping
• Test that the correct bbox_pred are used

Class and method docstrings

Using Google style:
http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

Image Viz module revamp

It was build as a debugging tool but it adds real value when being able to view intermediate images in TensorBoard.

We should:

• Revamp the code
• Remove functions without a clear objective
• Avoid duplicated code when post processing raw Tensor values
• Use consistent colors for GT, Anchors, Predictions
• Add text tags to Objects Detection visualizations

Add abstraction to train/eval

Currently it is focused on FasterRCNN (the pretrained bit, the dataset). We should abstract it so that any model with or without a pretrained net, a specific dataset can work.

Upload and use pretrained weights in Google Cloud

(Maybe we can use our org and share the link publicly)

Rewrite anchor reference module

It is very confusing in the way it does things.

Add RPN Anchor Target test

• Test for allowed border
• Test for closest GT label
• Foreground vs background fraction

Support distributed training

Should be independent of Google Cloud but also support it.

Travis CI can release new versions to PyPI

Define what branch/tags, etc. See here.

Possible workflow.

Logo Luminoth

Test RCNN

• Test returning shapes
• Test behaviour with and without Gt boxes (minibatch or all limit)
• Test RCNN loss
  • Perfect case

Add ImageNet and COCO support

Also, try to reuse Pascal VOC methods when doing it.

Google Apis Error

Hi,
I'm trying to install luminoth on Windows 8, and I'm getting an error installing googleapis-common-protos.

Collecting googleapis-common-protos (from gcloud)
Using cached googleapis-common-protos-1.5.3.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\setuptools_init_.py", line 10, in
from setuptools.extern.six.moves import filter, filterfalse, map
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\setuptools\extern_init_.py", line 1, in
from pkg_resources.extern import VendorImporter
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init_.py", line 3017, in
@call_aside
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 3003, in cal
l_aside
f(*args, **kwargs)
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 3045, in ini
tialize_master_working_set
dist.activate(replace=False)
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 2577, in acti
vate
declare_namespace(pkg)
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init_.py", line 2151, in decl
are_namespace
handle_ns(packageName, path_item)
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 2091, in _han
dle_ns
rebuild_mod_path(path, packageName, module)
File "c:\users\mariano-dediego\appdata\local\programs\python\python36\lib\site-packages\pkg_resources_init.py", line 2120, in _reb
uild_mod_path
orig_path.sort(key=position_in_sys_path)
AttributeError: '_NamespacePath' object has no attribute 'sort'

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\MARIAN~1\AppData\Local\Temp\pip-build-_ld46zq1\googleapis-common-pro

Any idea?

My platform:
Windows 8
Python 3.6.3
Pip 9.0.1

Thank you very much.

Add CPU and GPU options to setup.py

Sonnet and Tensorflow have both different packages with and without GPU support. Our setup.py should be able to handle both cases.

Checkout Sonnet setup.py

Allow instantiating FasterRCNN without ground truth boxes

So it can be used in prediction-only mode.

Checkpoints trained in Python2 are not compatible to load in Python3

Full usage example

Detailed instructions for running luminoth on a custom dataset, along with example code. (examples/ dir?)

Generate config "template" for optimizer and learning rate decay

The same way there is a get_initializer there should be a get_optimizer or get_learning_decay which allows for configurable options than they currently exist.

We have to be careful since we might be removing the possibility of those values to be easily available from CLI.

Documentation improvements

• Installation instructions
• Usage instructions
• Testing instructions
• High-level description & better name explanation
• Using your own dataset (explanation of .tfrecords format)

Use TensorFlow's recommended Datasets API

See here where it says:

Note: The preferred way to feed data into a tensorflow program is using the Datasets API.

How to install luminoth using Anaconda?

I am using Anaconda Navigator 1.6.8 at Windows 7 environment.
Python 3.6.3 |Anaconda, Inc.| (default, Oct 6 2017, 10:25:46) [MSC v.1900 64 bit (AMD64)] on win32

How to install luminoth using Anaconda? Or using command prompt?

Thank you very much in advance.

Regards,
Suryadi

Tensorboard on Google Cloud ML training job

How can I use Tensorboard for a training job which ran on the Google Cloud?
I used luminoth (checked out from GIT) with default settings, voc2012 (transformed and uploaded to a bucket) and the sample_config.yml (?)

I know how to use the tensorboard command, but only locally..

thanks!

Test FasterRCNN.network

• Output shapes
• Generate anchors

Add base_config file to package

When submitting package to Google Cloud ML luminoth/models/fasterrcnn/base_config.yml is not being included.

Add "How does it compare to ..." section

Add comparison with slim/object_detection, DIGITS and TensorBox

Add support for model_type in config file

For easier Google Cloud setup using a single file.

Paths can be expanded (like the shell) and use environment variables

Make something like this work:

dataset:
  dir: '$DATASET_DIR/voc/tf'

or

dataset:
  dir: '~/datasets/voc/tf'

or

lumi train -o dataset.dir=~/datasets/voc-2012/tf

Should be consistent and not only do dataset.dir, but every config option that is a path.

Would this be a problem under some configuration?
Do we want to make this an optional feature?

Test RCNN Target

• Test base case, one GT, 3 proposals (one background, one foreground, one background ignored using minibatch size of 2 and fraction of 0.5)
• Test empty case, it should choose the closer one (what if there no overlap with GT?)
• Test multiple overlap with GT, it should choose the one with the highest IoU

Share builds configurations in CircleCI

Basic CI for running tests and styling checks on PRs

Better behavior & error handling when importing datasets

Ie. lumi dataset ...

• Check that the files/directories exist
• Graceful exit
• Warning when overwriting files

Add data augmentation techniques

• Add random patching
• Add patching based on IoU
• Add random resize within limits
• Add photo-metric distortions

Fix FasterRCNN when running without any sort of RCNN config

When setting use_rcnn = False and removing the subconfig rcnn: it crashes.

Instant feedback for --help commands.

Currently, when running a simple lumi --help command it takes more than 7s for the message to appear. This is because of all the loading of models/sonnet/etc.

We should change the way we import/load submodules to have an instant reply.

Get rid of Sparse to Dense warning

I believe the warning is caused when reading from the .tfrecords. I don't think it's really a memory performance issue (as its says), but we should look into it.

Predict with custom config fails.

Fix is simple. Config files need to be loaded and merged with base_config before all the processing!

Test for ROI Pool

• Test crop pooling
• Test max pooling

Evaluation script for running on val/test splits

• Validation/test loss w/summaries on tensorboard
• VOC2007 mAP calculation w/summaries on tensorboard
• Support for other dataset splits (even train itself)
• Change mAP to VOC2012 style
• Cleanup the evaluation command file

Add bbox test

• Test for case of no overlap
• Test for case of same bboxes
• Test for case of incorrect boxes (negative area)

Server error

I created my virtual env with conda, everthing is working great but when I try access the server I receive the error:

Traceback (most recent call last)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/luminoth/tools/server/web.py", line 105, in index
return render_template('index.html')
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/templating.py", line 133, in render_template
return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/jinja2/environment.py", line 869, in get_or_select_template
return self.get_template(template_name_or_list, parent, globals)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/jinja2/environment.py", line 830, in get_template
return self._load_template(name, self.make_globals(globals))
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/jinja2/environment.py", line 804, in _load_template
template = self.loader.load(self, name, globals)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/jinja2/loaders.py", line 113, in load
source, filename, uptodate = self.get_source(environment, name)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/templating.py", line 57, in get_source
return self._get_source_fast(environment, template)
File "/home/marco/miniconda2/envs/DeepVision/lib/python3.5/site-packages/flask/templating.py", line 85, in _get_source_fast
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: index.html

Sometimes getting 0 foreground samples in RCNN

We're sometimes getting 0 foreground samples in RCNN, which makes reg_loss==nan, and therefore train_loss=nan. testEmptyCase and testAbsolutelyEmptyCase in rcnn_target_test.py were designed to catch this problem and failed, so we should consider rewriting them to be more solid.

Pre-launch checklist

• Make pre-launch checklist
• [Doc] Installation instructions
• [Doc] Usage instructions
• [Doc] Testing instructions
• [Doc] High-level description & better name explanation
• Test runner
• Simple CI setup

Test for RPN Proposal

• Test outside anchors are filtered
• Test clipping of proposals
• Test filter of negative area proposals
• Test top N pre NMS
• Test NMS (with simple cases, and high IoU)
• Test top N post NMS

You can use bbox_encode to generate valid bbox_pred.

Add docstrings to files with LICENSE reference

Use Google style:
http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html