aitorzip/keras-icnet

Keras implementation of Real-Time Semantic Segmentation on High-Resolution Images

Keras-ICNet

[paper]

Keras implementation of Real-Time Semantic Segmentation on High-Resolution Images. Training in progress!

Requisites

• Python 3.6.3
• Keras 2.1.1 with Tensorflow backend
• A dataset, such as Cityscapes or Mapillary (Mapillary was used in this case).

Train

Issue ./train --help for options to start a training session, default arguments should work out-of-the-box.

You need to place the dataset following the next directory convention:

.
├── mapillary                   
|   ├── training
|   |   ├── images             # Contains the input images
|   |   └── instances          # Contains the target labels
|   ├── validation
|   |   ├── images
|   |   └── instances
|   └── testing
|   |   └── images

These are the results of training for 300 epochs ./train --epochs 300

Training

Validation

Test

Issue ./test --help for options to start a testing session, default arguments should work out-of-the-box.

Output examples

TODO

• Perform class weighting

keras-icnet's Issues

Ok that's my bad, the keras implementation is very fast too !

Ok that's my bad, the keras implementation is very fast too !
I ran it on 20 images on a single batch and as you can see it goes from 1 FPS to 90 FPS after only one prediction !

[GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335]

Generated segmentations in 0.879213809967041 seconds -- 1.1373797689068224 FPS
Generated segmentations in 0.011792182922363281 seconds -- 84.80194096239386 FPS
Generated segmentations in 0.011512994766235352 seconds -- 86.85837354262875 FPS
Generated segmentations in 0.01149892807006836 seconds -- 86.96462782500518 FPS
Generated segmentations in 0.011469125747680664 seconds -- 87.1906038873298 FPS
Generated segmentations in 0.011708974838256836 seconds -- 85.40457331351429 FPS
Generated segmentations in 0.011548042297363281 seconds -- 86.59476422495665 FPS
Generated segmentations in 0.011418342590332031 seconds -- 87.57838469890586 FPS
Generated segmentations in 0.01206517219543457 seconds -- 82.88319336033989 FPS
Generated segmentations in 0.011431455612182617 seconds -- 87.47792354057606 FPS
Generated segmentations in 0.0111236572265625 seconds -- 89.89849108367626 FPS
Generated segmentations in 0.010841846466064453 seconds -- 92.23521132955095 FPS
Generated segmentations in 0.010785341262817383 seconds -- 92.71843844640449 FPS
Generated segmentations in 0.010878801345825195 seconds -- 91.92189177935086 FPS
Generated segmentations in 0.010807514190673828 seconds -- 92.52821530994926 FPS
Generated segmentations in 0.010857343673706055 seconds -- 92.10355958628868 FPS
Generated segmentations in 0.010789155960083008 seconds -- 92.68565619958898 FPS
Generated segmentations in 0.010899066925048828 seconds -- 91.75097344358402 FPS
Generated segmentations in 0.010824441909790039 seconds -- 92.38351578159072 FPS
Generated segmentations in 0.010749101638793945 seconds -- 93.03103027614506 FPS

note that it doesn't count the image loading and parsing

So you might want to use this repo, it's well coded and easy to understand

Originally posted by @MathiasGilson in #2 (comment)

train under multi gpu

Have you tried to enable multi gpu to train?
If I use “-n_gpu 2” to train model, I get this error:
File "Keras-ICNet/model.py", line 195, in
y = Lambda(lambda x: tf.image.resize_bilinear(x, size=(int(x.shape[1])//2, int(x.shape[2])//2)), name='data_sub2')(x)
TypeError: int returned non-int (type NoneType)

keras 2.1.3
tensorflow 1.4.1

Network speed

Hi.

I'm going to start experimenting with your project, but first wanted to ask, if you achieved similar speeds (~30fps on a high-end gpu or equivalent) as stated in the article?

Best,
Lauri

the accuracy of the testing results

Hi, I wonder to know your accuracy of the predicted results. How did you calculate the testing results? IoU or other methods? Because I apply this program to my own dataset, I get the very low accuracy.

Input normalization

why do you normalize the input with this function ?

Keras-ICNet/model.py

Line 16 in bcc13bf

x = Lambda(lambda x: (x - 127.5)/255.0)(inp)

It looks like an input value of 255 (let’s say in the green channel) would be normalized to 0.5 instead of 1 in that same channel.
I don’t understand why, can you help me ?

[ERROR] StopIteration: index 117 is out of bounds for axis 1 with size 66

Hi,
I tried to run the model but I get the following error

Epoch 1/100
Traceback (most recent call last):
File "/home/user/.local/lib/python3.5/site-packages/keras/utils/data_utils.py", line 578, in get
inputs = self.queue.get(block=True).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/user/.local/lib/python3.5/site-packages/keras/utils/data_utils.py", line 401, in get_index
return _SHARED_SEQUENCES[uid][i]
File "/home/user/Desktop/Keras-ICNet/utils.py", line 95, in getitem
self.Y1[n] = to_categorical(cv2.resize(label, (label.shape[1]//4, label.shape[0]//4)), self.n_classes).reshape((label.shape[0]//4, label.shape[1]//4, -1))
File "/home/user/.local/lib/python3.5/site-packages/keras/utils/np_utils.py", line 31, in to_categorical
categorical[np.arange(n), y] = 1
IndexError: index 117 is out of bounds for axis 1 with size 66

The above exception was the direct cause of the following exception:

Traceback`(most recent call last):
File "train", line 63, in
use_multiprocessing=False, shuffle=True, max_queue_size=10, initial_epoch=opt.epoch)
File "/home/user/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/user/.local/lib/python3.5/site-packages/keras/engine/training.py", line 2192, in fit_generator
generator_output = next(output_generator)
File "/home/user/.local/lib/python3.5/site-packages/keras/utils/data_utils.py", line 584, in get
six.raise_from(StopIteration(e), e)
File "", line 3, in raise_from
StopIteration: index 117 is out of bounds for axis 1 with size 66

Any ideas ?

Many thanks

NameError: name 'tf' is not defined when I loda the checkpoint file at training.

After I trained on my own dataset with width 800 height 480 without checkpoint successfully, I fill in the checkpoint parameter and tried the test image and this error showed up. The exact error is like this:

Traceback (most recent call last): File "train", line 55, in net = load_model(opt.checkpoint) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\engine\saving.py", line 419, in load_model model = _deserialize_model(f, custom_objects, compile) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\engine\saving.py", line 225, in deserialize_model model = model_from_config(model_config, custom_objects=custom_objects) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\engine\saving.py", line 458, in model_from_config return deserialize(config, custom_objects=custom_objects) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\layers_init.py", line 55, in deserialize printable_module_name='layer') File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\utils\generic_utils.py", line 145, in deserialize_keras_object list(custom_objects.items()))) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\engine\network.py", line 1032, in from_config process_node(layer, node_data) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\engine\network.py", line 991, in process_node layer(unpack_singleton(input_tensors), **kwargs) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\engine\base_layer.py", line 457, in call output = self.call(inputs, **kwargs) File "C:\DDDDZZZZYYYY\envs\kerass\lib\site-packages\keras\layers\core.py", line 687, in call return self.function(inputs, **arguments) File "C:\Users\In-Car DVR\Desktop\DZY-work\Keras-ICNet-master\model.py", line 195, in y = Lambda(lambda x: tf.image.resize_bilinear(x, size=(int(x.shape[1])//2, int(x.shape[2])//2)), name='data_sub2')(x) NameError: name 'tf' is not defined

The training process showed the same error when I put in the checkpoint parameters. Is the file in the output folder the checkpoint file? Or the checkpoint file is in some other locations?

Thanks in advance!

Can you add a license?

Hey,

Can you add a license like MIT?

Can't build the output

Hi,

I have trained the model and I am trying to visualize the predictions, however I am having trouble. The error occurs on the second last line;
output = apply_color_map(np.argmax(y[0], axis=-1), labels)
I also noticed that there is y[1] and y[3], as well.

Can anyone help me please?

Testing Performance

When I test the ADEKChallenge2016 dataset and VOC2012 dataset, I find the result is bad ,how was your test result?

Error in the code leading to bad performances

Hi,
I found that you resize images with opencv for the validation (see val_generator arguments)

Keras-ICNet/utils.py

Lines 61 to 63 in 987a6aa

	if self.resize_shape:
	image = cv2.resize(image, self.resize_shape)
	label = cv2.resize(label, self.resize_shape)

cv2.resize deforms the images and thus leads to bad learning

Instead I used the random crop function to always crop the image instead of deforming it
it's not and ideal fix but it leads to way better performances

change this line

Keras-ICNet/train

Line 48 in 987a6aa

val_generator = MapillaryGenerator(mode='validation', batch_size=opt.batch_size, crop_shape=None, resize_shape=(opt.image_width, opt.image_height))

to

val_generator = MapillaryGenerator(mode='validation', batch_size=opt.batch_size, crop_shape=(opt.image_width, opt.image_height))

and put this lines

Keras-ICNet/utils.py

Lines 91 to 92 in 987a6aa

	if self.crop_shape:
	image, label = _random_crop(image, label, self.crop_shape)

out of the if self.mode == 'training': condition
Now in each validation iteration the image will not be resized but randomly cropped

train loss

@aitorzip Hi, thanks your great repo, when I train in my dataset, I find such following
, every new epoch loss fall from the beginning ，not from the last epoch , why does it happen ?

This keras implementation has different order of bn and activation

Hi, thank you for your work. One thing I want to point out here is that you have some tiny difference compared with original implementaion, which may lead to more latency.
Your implementation here is conv + activation + bn as shown below:

y = Conv2D(32, 3, strides=2, padding='same', activation='relu', name='conv1_1_3x3_s2')(y)
y = BatchNormalization(name='conv1_1_3x3_s2_bn')(y)

But the original caffe implementaion is conv + bn + activation, which normally won't have much differnce, but this kind of order make it possible to merge bn into conv, thus leads to smaller latency.

Missing conv layers in the pyramid pooling module of first branch?

In the original PSPnet ,there appear to be a projection conv after the feature map is pooled, but it seems to be lefted out in this repo. So can a convolution between pooling and upsample help improve performance, did anyone test that?