aschampion/diluvian

Flood filling networks for segmenting electron microscopy of neural tissue

diluvian's Issues

Training and prediction of the "CREMI" data set. PART 1

Hello. Could I ask you whether the training and prediction of the "CREMI" data set work as expected or there is some problem with that? I used that data set for training using a very small learning rate value to be sure that the training went ok. My training was terminated automatically at the 158th epoch, however, the maximum value were set to be equal 1000. I didn't use either "early_abort_epoch" or "early_abort_loss" values while performing the training. Literally, I followed the example.

However, after the training I ran the prediction mode and the result of that was just a bunch of hdf5 files filled with zeros. All my predictions are just fully black images. Sometimes I can get the results with just 3 or 5 white points within the entire volume and that's all.

Could you please give me some piece of advice what can be wrong?

Thanks in advance.

'pip install diluvian' is not working

Hi! aschmpion.

First, thank you to implement FFN using keras.
I tried to install diluvian using 'pip install diluvian', but It didn't work
I got below error message

"""
Reading https://pypi.python.org/simple/keras-contrib/
Couldn't find index page for 'keras_contrib' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or working download links found for keras_contrib
error: Could not find suitable distribution for Requirement.parse('keras_contrib')
"""

If you know the reason why it is not working, please tell me how to deal with.
thank you.

Save labeled bodies to HDF5 for inspection (single region sparse)

Multi GPU fn's should optionally take list of devices instead of GPU count

Batches should optionally be drawn from a single subvolume generator

Would require Roundrobin being batch size aware.

UNet should downsample dimensions to nearest isotropy

Instead of using unet_downsample_rate, dimensions should be downsampled until they are nearest the lowest common near-isotropic resolution, and then be downsampled every level.

Optional noise/dropout in mask input channel for robustness training

Support dilated conv layers

Reduces the need for depth to increase directly with FOV

setup.py doesn't work

Hello,

I had the same issue as issue #14 so I downloaded the sources and I removed 'Kera==2.1.6' from requirements/prod.txt and installed keras using the suggestion in #14.

Then, when I tried python setup.py install I got this:

Traceback (most recent call last):
File "setup.py", line 59, in
tests_require=parsed_test_requirements
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
cmd.run()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 418, in run
self.easy_install(spec, not self.no_deps)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 660, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 707, in install_item
self.process_distribution(spec, dist, deps)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 752, in process_distribution
[requirement], self.local_index, self.easy_install
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 782, in resolve
replace_conflicting=replace_conflicting
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 1065, in best_match
return self.obtain(req, installer)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 1077, in obtain
return installer(requirement)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
run_setup(setup_script, args)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
raise
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-_T9Yb2/pyn5-0.1.0/setup.py", line 7, in

ImportError: No module named setuptools_rust

So I tried downloading and installing the most recent setuptools-rust version for python 2.7 from https://pypi.org/project/setuptools-rust/0.5.1/#files

and then when I tried python setup.py install , I get this:

running install
running bdist_egg
running egg_info
writing requirements to diluvian.egg-info/requires.txt
writing diluvian.egg-info/PKG-INFO
writing top-level names to diluvian.egg-info/top_level.txt
writing dependency_links to diluvian.egg-info/dependency_links.txt
writing entry points to diluvian.egg-info/entry_points.txt
reading manifest file 'diluvian.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching 'pycache' found under directory ''
warning: no previously-included files matching '.py[co]' found under directory ''
warning: no files found matching '.jpg' under directory 'docs'
warning: no files found matching '.png' under directory 'docs'
warning: no files found matching '.gif' under directory 'docs'
warning: no files found matching '*.txt' under directory 'diluvian'
writing manifest file 'diluvian.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/init.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/octrees.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/preprocessing.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/postprocessing.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/network.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/util.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/diluvian.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/config.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/regions.py -> build/bdist.linux-x86_64/egg/diluvian
creating build/bdist.linux-x86_64/egg/diluvian/conf
copying build/lib/diluvian/conf/default.toml -> build/bdist.linux-x86_64/egg/diluvian/conf
copying build/lib/diluvian/conf/cremi_datasets.toml -> build/bdist.linux-x86_64/egg/diluvian/conf
copying build/lib/diluvian/conf/cremi_test_datasets.toml -> build/bdist.linux-x86_64/egg/diluvian/conf
copying build/lib/diluvian/main.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/volumes.py -> build/bdist.linux-x86_64/egg/diluvian
copying build/lib/diluvian/training.py -> build/bdist.linux-x86_64/egg/diluvian
byte-compiling build/bdist.linux-x86_64/egg/diluvian/init.py to init.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/octrees.py to octrees.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/preprocessing.py to preprocessing.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/postprocessing.py to postprocessing.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/network.py to network.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/util.py to util.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/diluvian.py to diluvian.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/config.py to config.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/regions.py to regions.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/main.py to main.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/volumes.py to volumes.pyc
byte-compiling build/bdist.linux-x86_64/egg/diluvian/training.py to training.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying diluvian.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/diluvian-0.0.6-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing diluvian-0.0.6-py2.7.egg
removing '/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/site-packages/diluvian-0.0.6-py2.7.egg' (and everything under it)
creating /home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/site-packages/diluvian-0.0.6-py2.7.egg
Extracting diluvian-0.0.6-py2.7.egg to /home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/site-packages
diluvian 0.0.6 is already the active version in easy-install.pth
Installing diluvian script to /home/ncmir-lab/miniconda2/envs/diluvian/bin

Installed /home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/site-packages/diluvian-0.0.6-py2.7.egg
Processing dependencies for diluvian==0.0.6
Searching for pyn5==0.1.0
Reading https://pypi.org/simple/pyn5/
Downloading https://files.pythonhosted.org/packages/4f/ba/0026c6f64d107092aa377f7225c943722c2f9a7612f966b3328fb835f640/pyn5-0.1.0.tar.gz#sha256=7c6c52b5b36fb710f8a014aafd78c95cbbb68f7bc2bcf49960b9d1adb62f4688
Best match: pyn5 0.1.0
Processing pyn5-0.1.0.tar.gz
Writing /tmp/easy_install-CrtnJC/pyn5-0.1.0/setup.cfg
Running pyn5-0.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-CrtnJC/pyn5-0.1.0/egg-dist-tmp-m01Akx
Traceback (most recent call last):
File "setup.py", line 59, in
tests_require=parsed_test_requirements
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/init.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
cmd.run()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 418, in run
self.easy_install(spec, not self.no_deps)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 660, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 707, in install_item
self.process_distribution(spec, dist, deps)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 752, in process_distribution
[requirement], self.local_index, self.easy_install
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 782, in resolve
replace_conflicting=replace_conflicting
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 1065, in best_match
return self.obtain(req, installer)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 1077, in obtain
return installer(requirement)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 705, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
run_setup(setup_script, args)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
raise
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/home/ncmir-lab/miniconda2/envs/diluvian/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/home/ncmir-lab/.local/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-CrtnJC/pyn5-0.1.0/setup.py", line 7, in

ImportError: cannot import name Binding

fill_subvolume_with_model overwrites CUDA_VISIBLE_DEVICES

I'm running your code on a machine with 2 gpus: 470 and Titan Black
I was able to run "diluvian train" after setting CUDA_VISIBLE_DEVICES (to ignore the 470).
However, "diluvian fill" would always end up running on the CPU because CUDA_VISIBLE_DEVICES was being overwritten and the 470 isn't suitable.

I think checking if CUDA_VISIBLE_DEVICES is in os.environ before setting should fix this.

Let me know if more information is helpful.

Training and prediction of the "CREMI" data set. PART 2

Hello, one more time. I decided to be more precise and wrote down the exact steps that I did. The problem that I described in the previous issue still remains but I hope that information will be useful in debugging the code

We did the following steps:

1. all diluvian and tensorflow packages were removed from the system. All files
   were also removed from /user/.keras/dataset

2. Following the instruction, diluvian was installed as following:
   pip install diluvian

   ... Installing collected packages: tensorflow, diluvian Successfully installed diluvian-0.0.3 tensorflow-1.1.0

3. At the next step tensorflow-gpu was installed:
   pip install 'tensorflow-gpu==1.2.1'

   ... Installing collected packages: tensorflow-gpu Successfully installed tensorflow-gpu-1.2.1

4. I ran diluvian with the default arguments to check whether everything worked ok

   diluvian train

   the "cremi" database was downloaded from the website and diluvian ran just
   for two epochs

   During the trail run I got the following warning:
   "/user/anaconda2/lib/python2.7/site-packages/keras/callbacks.py:120:
   UserWarning: Method on_batch_end() is slow compared to the batch update (1.893568).
   Check your callbacks."

5. I generated the convig file to change the number of epochs and run the training for a long term

   diluvian check-config > myconfig.toml

   <myconfig.toml>
   ...
   total_epochs = 1000
   ...
   num_gpus = 2
   ...
   

   diluvian train -c ./myconfig.toml

   By default, the u-net architecture was chosen:

   <myconfig.toml>
   ...
   factory = "diluvian.network.make_flood_fill_unet"
   ...
   

   The training was terminated automatically at the 76th epoch
   the values of loss and validation loss were about the same ( 0.63 )

6. At the next step I ran the prediction mode to estimate how good the training
   was.

   diluvian fill -c ./myconfig.roml -m ./model-output.hdf5 ./file-{volume}

   where file-{volume} was just a dummy that contained nothing. As far as
   I understood diluvian takes the file to name the output files (*.hdf5)

7 Results:
At the end of the filling procedure I got three files as the output.
The problem is that all of them contain only zeros. Basically it means
that the output mask was not predicted

IMPORTANT: 
1) even reducing the learning rate during the training doesn't
help to improve the result of both prediction and training

2) ffn architecture produced the same result that u-net did

3)  we tried to run deluvian with different data set but the result was about the same

Please, can you tell us where a bug can be or what we’re doing wrong?

Thanks in advance

Best regards,
Ravil

Dense segmentation should fill in parallel

Subvolume bounds generator should apply mask constraints based on output, not input, FOV shape

Coalesce adjacent partitions

Reimplement a transposed convolution for UNet upscaling

Had this working with old Keras, but switched to an upsample+conv kludge for Keras 2.