SegDataGenerator is very slow

[aurora95]

SegDataGenerator is very slow

[hilal-t]

@ahundt thank you.
It is working fine after vectorizing binarylab for loop.
It was taking 40% of batch generation time.
Thanks again.

[ahundt]

@hilal-t Happy to hear it, what did the improved version look like?

I'll close this since it seems your problem is solved.

[aurora95]

@hilal-t If you set everything correctly it shouldn't be very slow since your input size is not considerably larger than the original one. Can you provide more information, for instance, your codes? And how much time exactly does the generator take to process one batch?

[hilal-t]

@aurora95

Thank you for your reply
the following is what I have added

target_size = (224,224)
dataset = 'test_dataset'
train_file_path = 'data/Seg/train.txt'
val_file_path   = 'data/Seg/val.txt'
data_dir        = 'data/Seg/train_imgs/'
label_dir       = 'data/Seg/label/'
data_suffix='.jpg'
label_suffix='.png'
classes = 2
class_weight = None
data_suffix='.jpg'
label_suffix='.png'
input_shape=(224,224,3)
batchnorm_momentum=0.9
ignore_label = None
label_cval = 0
loss_shape = (target_size[0] * target_size[1] , classes,)
model_name = 'unet'
batch_size =64
batchnorm_momentum = 0.95
epochs = 250
lr_base = 0.01 * (float(batch_size) / 16)
lr_power = 0.9

model=get_unet()

model.compile(optimizer='sgd', loss='categorical_crossentropy',metrics=['accuracy'])

train_datagen = SegDataGenerator(zoom_range=0.,
                                 zoom_maintain_shape=True,
                                 crop_mode='none',
                                 crop_size=None,
                                 rotation_range=0.,
                                 shear_range=0,
                                 horizontal_flip=False,
                                 channel_shift_range=0.,
                                 rescale=1/255.,
                                 fill_mode='constant',
                                 label_cval=0)
val_datagen = SegDataGenerator()

train_generator=  train_datagen.flow_from_directory(
        file_path=train_file_path,
        data_dir=data_dir, data_suffix=data_suffix,
        label_dir=label_dir, label_suffix=label_suffix,
        classes=classes,
        target_size=target_size, color_mode='rgb',
        batch_size=batch_size, shuffle=True,
        loss_shape=loss_shape,
        ignore_label=ignore_label,
        # save_to_dir='Images/'
    )
history = model.fit_generator(
    generator=train_generator,
    steps_per_epoch=get_file_len(train_file_path),
    epochs=epochs,
    callbacks=callbacks,
    workers=4,
    class_weight=class_weight
   )

`

I have added the following to the SegDataGenerator

def binarylab(labels, size=224, nb_class=2):
    y = np.zeros((size,size,nb_class))
    for i in range(size):
        for j in range(size):
            if labels[i,j]==0 :
                y[i,j,0]=1
            else:
                y[i,j,1]=1
    return y
    

if self.ignore_label:
     y[np.where(y == self.ignore_label)] = self.classes
        
y = binarylab (y, 224, 2)
if self.loss_shape is not None:
y = np.reshape(y, self.loss_shape)

batch_x[i] = x
batch_y[i] = y

this is example of the output

2691/38621 [=>............................] - ETA: 134460s - loss: 0.6930 - acc: 0.5071

[ahundt]

Try using a profiling tool on both your disk and the python code to see the source of the problem and post the results. Then we can determine how to fix it.

Also are you on an HDD or SSD?

[ahundt]

Also you will definitely need to vectorize that binary lab for loop consult google/stack overflow for details