slow training progress

[aurora95]

slow training progress

[ahundt]

For reference that means these parameters (and SGD is the optimizer):

    model_name = 'AtrousFCN_Resnet50_16s'
    batch_size = 16
    batchnorm_momentum = 0.95
    epochs = 250
    lr_base = 0.01 * (float(batch_size) / 16)
    lr_power = 0.9
    resume_training = False
    if model_name is 'AtrousFCN_Resnet50_16s':
        weight_decay = 0.0001/2
    else:
        weight_decay = 1e-4
    classes = 21
    target_size = (320, 320)
    dataset = 'VOC2012_BERKELEY'

[aurora95]

To be honest I didn't test the new code...

But my old code running on an old Titan X takes about 600-700s per epoch for AtrousFCN_Resnet50_16s, batch_size=16, target_size=(320, 320) on the 11k dataset. Also the accuracy after the first epoch should be 0.78 or something around. So you must have big problems...

[ahundt]

Oh man so silly... the branch on my laptop was different from my training workstation, sorry about that. The numbers look like you describe now once I checked out the right branch and enabled SGD.

 1398/11127 [==>...........................] - ETA: 6660s - loss: 1.0014 - sparse_accuracy_ignoring_last_label: 0.7451

For some reason epochs are still super slow for me but that's probably particular to my machine.