Training and prediction of the "CREMI" data set. PART 2

[aschampion]

Training and prediction of the "CREMI" data set. PART 2

[aschampion]

@sflender like most network training paradigms, diluvian terminates training early if the validation loss (or actually in this case, validation_metric which is by default F_0.5 score of masks of single-body validation volumes) has not improved in several epochs. The number of epochs is controlled by the CONFIG.training.patience parameter (see docs). To effectively disable this, just set patience to be larger than total_epochs.

[aschampion]

Hi Ravil,

My guess as to what's happening is this:

the values of loss and validation loss were about the same ( 0.63 )

For a near-default configuration this loss is high and it's unlikely the network is generating high probability, structured output yet. Hence the filling volumes are empty because the network is not yet trained to the point where it's outputting probability high enough to cause FOV moves. You can tell this is happening if during the filling step the lower progress bar is just flashing by quickly and shows 1 / 1 at the end (this is the progress bar for filling each region). Another way to check is that during training if you increase the output verbosity (-l INFO) the training generators will show the average numbers of moves in the training subvolumes. In normal training this should increase to be near or above 20 once the network is learning to move.

These networks take a very long time to train to have performance competitive with other approaches -- days on 8 GPUs. However, on 2 GPUs you should be able to train the network to a point where the output starts to look like neuronal segmentation in less than 8 hours.

Some things to try for fast training to good results:

• switching from SGD to Adam
• orders of magnitude larger training and validation sizes than the defaults

[sflender]

Hi Ravil, Andrew, do you understand why training was terminated automatically after the 76th epoch, even though you did not configure it to stop early? I observed similar behavior.

Best,
-Samuel.

[Jingliu1994]

@aschampion I do the things that you suggest,including changing the optimizer and enlarging training and validation sizes,but still can't get a good results.The training loss can't fall when drop to about 0.3,and the validation loss is about 0.5. The number of gpu is 1,batch size is 12.I change the learning rate,but get no progress.What should i do next,thanks

[aschampion]

@Jingliu1994 You can continue to sweep the parameter space, but there are several things I would suggest first:

• If you're using the CREMI data, the official data still has many ground truth errors and quality problems (e.g., random labels in blank sections). The MALA v2 submission on the CREMI front page has realigned ground truth volumes with much better ground truth labels. This is what the ground truth I was using when working on FFNs was based on, but for various reasons don't automatically distribute it with diluvian.
• Ignore the validation loss (which is often meaningless because of the FFN training process
  -- this is why in Google's paper they validate with the skeleton metrics instead) and pay attention only to the F_beta validation metric. Even if the training loss improvement is miniscule the validation metric may still be improving. An example pulled at random from my logs ("val subvolumes" is the validation metric):
  
• Apply the network at higher resolution, 8nm or 4nm. (Will greatly increase inference time)
• Use a larger input FOV. (Will greatly increase training and inference time)

You should also be aware that Google released their implementation a few weeks ago. If you just want good results and aren't that concerned with having a simple sandbox to experiment with FFN-like architectures, it's probably a better choice than diluvian for you. Multi-segmentation consensus and FFN-based merging are both crucial to the quality of FFN results reported in Google's original paper; diluvian doesn't implement either of these.