titu1994/neural-image-assessment

Implementation of NIMA: Neural Image Assessment in Keras

NIMA: Neural Image Assessment

Implementation of NIMA: Neural Image Assessment in Keras + Tensorflow with weights for MobileNet model trained on AVA dataset.

NIMA assigns a Mean + Standard Deviation score to images, and can be used as a tool to automatically inspect quality of images or as a loss function to further improve the quality of generated images.

Contains weights trained on the AVA dataset for the following models:

• NASNet Mobile (0.067 EMD on valset thanks to @tfriedel !, 0.0848 EMD with just pre-training)
• Inception ResNet v2 (~ 0.07 EMD on valset, thanks to @tfriedel !)
• MobileNet (0.0804 EMD on valset)

Usage

Evaluation

There are evaluate_*.py scripts which can be used to evaluate an image using a specific model. The weights for the specific model must be downloaded from the Releases Tab and placed in the weights directory.

Supports either passing a directory using -dir or a set of full paths of specific images using -img (seperate multiple image paths using spaces between them)

Supports passing an argument -resize "true/false" to resize each image to (224x224) or not before passing for NIMA scoring. Note : NASNet models do not support this argument, all images must be resized prior to scoring !

Arguments:

-dir    : Pass the relative/full path of a directory containing a set of images. Only png, jpg and jpeg images will be scored.
-img    : Pass one or more relative/full paths of images to score them. Can support all image types supported by PIL.
-resize : Pass "true" or "false" as values. Resize an image prior to scoring it. Not supported on NASNet models.

Training

The AVA dataset is required for training these models. I used 250,000 images to train and the last 5000 images to evaluate (this is not the same format as in the paper).

First, ensure that the dataset is clean - no currupted JPG files etc by using the check_dataset.py script in the utils folder. If such currupted images exist, it will drastically slow down training since the Tensorflow Dataset buffers will constantly flush and reload on each occurance of a currupted image.

Then, there are two ways of training these models.

Direct-Training

In direct training, you have to ensure that the model can be loaded, trained, evaluated and then saved all on a single GPU. If this cannot be done (because the model is too large), refer to the Pretraining section.

Use the train_*.py scripts for direct training. Note, if you want to train other models, copy-paste a train script and only edit the base_model creation part, everythin else should likely be the same.

Pre-Training

If the model is too large to train directly, training can still be done in a roundabout way (as long as you are able to do inference with a batch of images with the model).

Note : One obvious drawback of such a method is that it wont have the performance of the full model without further finetuning.

This is a 3 step process:

1. Extract features from the model: Use the extract_*_features.py script to extract the features from the large model. In this step, you can change the batch_size to be small enough to not overload your GPU memory, and save all the features to 2 TFRecord objects.

2. Pre-Train the model: Once the features have been extracted, you can simply train a small feed forward network on those features directly. Since the feed forward network will likely easily fit onto memory, you can use large batch sizes to quickly train the network.

3. Fine-Tune the model: This step is optional, only for those who have sufficient memory to load both the large model and the feed forward classifier at the same time. Use the train_nasnet_mobile.py as reference as to how to load both the large model and the weights of the feed forward network into this large model and then train fully for several epochs at a lower learning rate.

Example

Requirements

• Keras
• Tensorflow (CPU to evaluate, GPU to train)
• Numpy
• Path.py
• PIL

neural-image-assessment's Issues

from path import Path

when i run 'evaluate_mobilenet.py',there occured an error——No module named path.so i want to know what dose this module depend on.

An Error in IO

When I run the @tfriedel's training script, an error raises that
IOError: cannot identify image file u'/home/xxx/dataset/AVA_dataset/images/images/1617.jpg'

Things happen because PIL cannot open the "1617.jpg" image.

So, how to avoid such error? @tfriedel

How to set arguments:

I understand this issue is very basic problem.
In evaluate_mobilenet.py, how should I set my directry (C:\users\ ...), images, and resize option.
Could you show a sample in your case to me??
###############
parser = argparse.ArgumentParser(description='Evaluate NIMA(Inception ResNet v2)')
parser.add_argument('-dir', type=str, default=None,
help='Pass a directory to evaluate the images in it')
parser.add_argument('-img', type=str, default=[None], nargs='+',
help='Pass one or more image paths to evaluate them')
parser.add_argument('-resize', type=str, default='false',
help='Resize images to 224x224 before scoring')
parser.add_argument('-rank', type=str, default='true',
help='Whether to tank the images after they have been scored')
###############

Thank you so much ...

Can not get the same weights file as in the release

Hi titu, thanks for your work!
I run the training code you wrote and the generated weight file looks quite different with file posted in the release. I run a test with some images using the weight file I got and the release weight file, and find almost every image got different scores with these two weight file.
How does this happen?

loss become nan

hello! Just got another question.When i train the nasnet mobile model, the loss becomes nan during the fifth epoch. Any possible reason for that? Should i tune some parameters? I used some other data before and it did not happen. Thanks

error for the final epoch

Thank you so much for your fantastic work!
I have just read you code and try to redo it again.
Then i just change the batch to 10, epochs to 2, and use the first 250 pics as training set and the last 50 as val set.
At the first, It seems to be fine.
But I get the Index error in the final epoch.
Exception ignored in: <generator object train_generator at 0x7f8c7433a360> Traceback (most recent call last): File "/home/haoyuzhang/workspace/NIMA_zhanghaoyu/utils/data_loader.py", line 123, in train_generator yield (X_batch, y_batch) File "/home/haoyuzhang/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1503, in __exit__ exec_type, exec_value, exec_tb) File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__ self.gen.throw(type, value, traceback) File "/home/haoyuzhang/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4342, in get_controller if self.stack[-1] is not default: IndexError: list index out of range
I know there is something wrong with keras.fit_generator but I can't figure it now.
Did some one else have the same situation?

failed at the last step when I train nasnet

When I run "python train_nasnet_mobile.py", It failed as follows:

1250/1250 [==============================] - 1439s 1s/step - loss: 0.0667 - val_loss: 0.0653
Exception IndexError: IndexError('list index out of range',) in <generator object val_generator at 0x24cf7c30> ignored
Exception IndexError: 'list index out of range' in <generator object get_controller at 0xd611500> ignored
Exception IndexError: IndexError('list index out of range',) in <generator object train_generator at 0x24cf7be0> ignored
Exception IndexError: 'list index out of range' in <generator object get_controller at 0xd611730> ignored

1)I didnt find anywhere a list index out of range.
2)I didnt find any file contains the Intermediate results,did you save the model that trained every epoch?
thks very much

Tensorflow version Issue

I am having trouble loading the .so(shared object) files of tensorflow during training .May i get to know the tensorflow version on which the code is executed so that i could create my environment according to that

Wrong loss while computing mean.

Suppose we have 3 samples, each y_truth and y_pred as following:

import numpy as np
import tensorflow as tf
from keras import backend as K

y_truth = np.array([
    [0, 0, 0, 0, 0, 0, 0, 0.9, 0.1, 0],
    [0, 0, 0, 0, 0, 0, 0, 0.9, 0.1, 0],
    [0, 0, 0, 0, 0, 0, 0, 0.9, 0.1, 0]])

y_pred = np.array([
    [0, 0, 0, 0, 0, 0, 0.9, 0, 0.1, 0],
    [0.9, 0, 0, 0, 0, 0, 0, 0, 0.1, 0],
    [0.8, 0, 0, 0, 0, 0, 0, 0, 0.1, 0.1]])

Each sample has its own EMD. Let's calculate them.

sess = tf.Session()

def emd_sample_wise(y_truth, y_pred):
    return K.sqrt(K.mean(K.square(K.abs(K.cumsum(y_truth, axis=-1) - K.cumsum(y_pred, axis=-1))), axis=-1))

sess.run(emd_sample_wise(y_truth, y_pred))
## array([ 0.28460499,  0.75299402,  0.67082039])

Looks right. First sample has the smallest loss.

Now, we need the mean of them, which is (0.28460499 + 0.75299402 + 0.67082039)/3 = 0.56947313

def emd(y_truth, y_pred):
    return K.mean(emd_sample_wise(y_truth, y_pred))

sess.run(emd(y_truth, y_pred))
## 0.56947313551525303

Comparing with the original version:

def earth_mover_loss(y_true, y_pred):
    return K.sqrt(K.mean(K.square(K.abs(K.cumsum(y_true, axis=-1) - K.cumsum(y_pred, axis=-1)))))

sess.run(earth_mover_loss(y_truth, y_pred))
## 0.60497933849016694

That's incorrect.

memory leak when evaluating

swig/python detected a memory leak of type 'int64_t *', no destructor found.

dropout for eval

Hi, do you turn off Dropout when evaluating?
I couldn't find the related code.

train mobilenet test error

it encouter below error when I training test data with 45 images for train_dataset and 25 images for val_dataset.

------------------- validation exception!
Exception ignored in: <generator object DataSetLoader.val_generator at 0x10eb6d888>
Traceback (most recent call last):
File "/Users/jia/IdeaProjects/lizhifm_projects/image_quality_assessment/image_assessment/data/dataset_loader.py", line 196, in val_generator
yield (x_batch, y_batch)
File "/Users/jia/.virtualenvs/tensorflow1.3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1507, in exit
exec_type, exec_value, exec_tb)
File "/Users/jia/anaconda3/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/Users/jia/.virtualenvs/tensorflow1.3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3972, in get_controller
if self.stack[-1] is not default:
IndexError: list index out of range
------------------- train exception!
Exception ignored in: <generator object DataSetLoader.train_generator at 0x10eb6d830>
Traceback (most recent call last):
File "/Users/jia/IdeaProjects/lizhifm_projects/image_quality_assessment/image_assessment/data/dataset_loader.py", line 147, in train_generator
yield (x_batch, y_batch)
File "/Users/jia/.virtualenvs/tensorflow1.3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1507, in exit
exec_type, exec_value, exec_tb)
File "/Users/jia/anaconda3/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/Users/jia/.virtualenvs/tensorflow1.3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3972, in get_controller
if self.stack[-1] is not default:
IndexError: list index out of range
Exception ignored in: <generator object get_controller at 0x10eb6d9e8>
Traceback (most recent call last):
File "/Users/jia/.virtualenvs/tensorflow1.3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3972, in get_controller
if self.stack[-1] is not default:
IndexError: list index out of range
Exception ignored in: <generator object get_controller at 0x112e5e360>
Traceback (most recent call last):
File "/Users/jia/.virtualenvs/tensorflow1.3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3972, in get_controller
if self.stack[-1] is not default:
IndexError: list index out of range

for nasnet model

why does the model not support resizing option for evaluation?

Can i train ava with inception_v3?

Thanks for job!
I read your codes. you train ava dataset wich mobiblenet,but I think this model design for mobile and its accuracy is not as good as inception_v3. Since this is a classification task, it can also train wich inception_v3. How do you think?

Not able to load utils,need extra setting？

Hi ，

I tried with below inputs and get error outputs that functions in utils not found

~/code/ML/neural-image-assessment-0.5$ python3 evaluate_nasnet.py -dir ./inputs/ -resize true
Using TensorFlow backend.
Traceback (most recent call last):
  File "evaluate_nasnet.py", line 8, in <module>
    from utils.nasnet import NASNetMobile, preprocess_input
ImportError: No module named 'utils.nasnet'

do I need add below code to each script or I miss some settings?

import sys    
sys.path.append(path To Folder Containing Scripts) 

edit： using the 0.5 release version

Weight freezing in train*.py

I see that the weights besides the very top layer are frozen in train*.py files:

base_model = InceptionResNetV2(input_shape=(image_size, image_size, 3), include_top=False, pooling='avg')
for layer in base_model.layers:
    layer.trainable = False

Are the weights provided in the releases page also obtain this way?

for trainable params

Total params: 4,280,286
Trainable params: 10,570
Non-trainable params: 4,269,716

this is what i got, i tried different dataset but it's always like this. Is this normal?Or any cause for so many non-trainable params? Also do the training images also need to be resized to 224*224 before training? Thanks.

Can't get the similar scores for the readme images.

Thanks for your jobs!
I download the model in 'Releases Tab' and test the predict score for the images in your README.md,but can not get the similar scores. The inception_resnet and mobilenet have the same problem. why?

version of tf, keras and python?

Hi

could I ask which version of tf, keras and python you used to train and test all the networks please?

Need to normalize

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/_impl/keras/applications/imagenet_utils.py#L82

def _preprocess_symbolic_input(x, data_format, mode):
  """Preprocesses a symbolic image tensor.
  Arguments:
      x: symoblic tensor, 3D or 4D.
      data_format: data format of the image tensor.
      mode: One of "caffe", "tf".
          - caffe: will convert the images from RGB to BGR,
              then will zero-center each color channel with
              respect to the ImageNet dataset,
              without scaling.
          - tf: will scale pixels between -1 and 1,
              sample-wise.
  Returns:
      Preprocessed tensor.
  """
  global _IMAGENET_MEAN

  if mode == 'tf':
    x /= 127.5
    x -= 1.
    return x

  if data_format == 'channels_first':
    # 'RGB'->'BGR'
    if K.ndim(x) == 3:
      x = x[::-1, ...]
    else:
      x = x[:, ::-1, ...]
  else:
    # 'RGB'->'BGR'
    x = x[..., ::-1]

  if _IMAGENET_MEAN is None:
    _IMAGENET_MEAN = K.constant(-np.array([103.939, 116.779, 123.68]))
  # Zero-center by mean pixel
  if K.dtype(x) != K.dtype(_IMAGENET_MEAN):
    x = K.bias_add(x, K.cast(_IMAGENET_MEAN, K.dtype(x)), data_format)
  else:
    x = K.bias_add(x, _IMAGENET_MEAN, data_format)
  return x

https://github.com/titu1994/neural-image-assessment/blob/master/data_loader.py#L59

1. Which order RGB/BGR does MobileNet accept?
2. Should we use IMAGENET_MEAN = [103.939, 116.779, 123.68] or just [127, 127, 127]?

scores correct?

Hi,
thanks for implementing this paper!
I just played a little bit with it, and I noticed almost every image, even really good or really bad ones all had scores of at least 4.8 and rarely more than 5.x.
Could this be a bug? Or maybe is more training needed? I was using your weights.
How long did you train for?

Can you share a well-trained model or settings?

Hi,

I'm using this code on AVA dataset and have encountered some problem when training. The model converges at around 10th epoch but the prediction result is not as expected. The score of all pictures are around 5.x and also the model can not figure out which picture is better. I'm using the directly training method and have tried mobilenet / vgg16 / resnet50 / inceptionv3 but none of them works.

So could you please share the exact settings for the one used in the demo images? Or can you share the well trained models ? Many thanks.

Where I can find weights for Inception ResNet v2?

README.md says that @tfriedel trained weights for Inception ResNet v2, but I can't find it. Is that weights available for downloading anywhere?

training time

Hello, how long does it take for you to train with 250000 images? I am training with a small sample of 1000 images, and each epoch is taking about 1 hour. Is this time normal I haven't done much machine learning before so I don't have a good concept of this. The machine i use have a good nvidia gpu. Do i need any adjustment or tensorflow can automatically find the gpu and use it? Currently i only installed tensorflow 1.4. thanks for the help!

dataset

where did you get the dataset?

about AVA dataset

Hi @titu1994 , would you please share the AVA dataset files? I could not access the original AVA dataset page due to some reason in China mainland.

Releasing the VGG version

Hi:

The work is very interesting and valuable. Could you also release the VGG16 version of the NIMA in the paper. I checked the repository and only found the MobileNet , InceptionNet versions available.

Many thanks

Different version with same h5 file get different Evaluating?

Hi titu,

Sorry to bother you again , I just tried with 0.5 release and the HEAD of git with mobilenet（seems nasnet only process 224224 image? cut big image to 224224 to evaluate?） ,the test images are same but the score turn out to be quite different, a very bad image get high score in the the HEAD of git
do you got any idea on it？

outputs from 0.5 release:

1) output.jpg : Score = 3.76305                 <<<<< optimized images
2) img_20170531_215051.jpg : Score = 3.69984
3) img_20170531_215157.jpg : Score = 3.67825
4) img_20170531_215130.jpg : Score = 3.66773
5) img_20170531_215126.jpg : Score = 3.65906
6) img_20170531_215106.jpg : Score = 3.65685
7) img_20170531_215153.jpg : Score = 3.65632
8) img_20170531_215143.jpg : Score = 3.65036
9) img_20170531_215139.jpg : Score = 3.64920
10) img_20170531_215112.jpg : Score = 3.64874
11) img_20170531_215117.jpg : Score = 3.64370
12) img_20170531_215134.jpg : Score = 3.64089
13) img_20170531_215147.jpg : Score = 3.62634
14) single_shoot_test3.jpg : Score = 3.22735    <<<< very bad images

and output from head of git

1) output.jpg : Score = 4.98324                <<<<< optimized images
2) single_shoot_test3.jpg : Score = 4.82322 <<<< very bad images
3) img_20170531_215139.jpg : Score = 4.82291
4) img_20170531_215134.jpg : Score = 4.81500
5) img_20170531_215051.jpg : Score = 4.78660
6) img_20170531_215153.jpg : Score = 4.77603
7) img_20170531_215130.jpg : Score = 4.76752
8) img_20170531_215117.jpg : Score = 4.75221
9) img_20170531_215147.jpg : Score = 4.73955
10) img_20170531_215126.jpg : Score = 4.72469
11) img_20170531_215106.jpg : Score = 4.71353
12) img_20170531_215157.jpg : Score = 4.71176
13) img_20170531_215143.jpg : Score = 4.69852
14) img_20170531_215112.jpg : Score = 4.68663

0.5 release modifications to read imgs in dir in my env

import numpy as np
import os

from keras.models import Model
from keras.layers import Dense, Dropout
from keras.applications.mobilenet import MobileNet
from keras.applications.mobilenet import preprocess_input
from keras.preprocessing.image import load_img, img_to_array
import tensorflow as tf

import sys
sys.path.append("./utils")
from score_utils import mean_score, std_score

#get jpg file from path
def get_imlist(path):
    return [os.path.join(path,f) for f in os.listdir(path) if f.endswith('.jpg')]

def envaluation(model,img_path):
    img = load_img(img_path)
    x = img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)

    scores = model.predict(x, batch_size=1, verbose=1)[0]

    mean = mean_score(scores)
    std = std_score(scores)

    file_name = os.path.basename(img_path).lower()
    score_list.append((file_name, mean))

    print("Evaluating : ", img_path)
    print("NIMA Score : %0.3f +- (%0.3f)" % (mean, std))

with tf.device('/CPU:0'):
    base_model = MobileNet((None, None, 3), alpha=1, include_top=False, pooling='avg', weights=None)
    x = Dropout(0.75)(base_model.output)
    x = Dense(10, activation='softmax')(x)

    model = Model(base_model.input, x)
    model.load_weights('weights/mobilenet_weights.h5')

    #Get All the original Photo
    imgs = get_imlist("./inputs/")
    score_list = []

    for img_path in imgs[1:]:
        try:
            envaluation(model,img_path)
        except:
             print (img_path + '...skipped')

    print("*" * 40, "Ranking Images", "*" * 40)
    score_list = sorted(score_list, key=lambda x: x[1], reverse=True)

    for i, (name, score) in enumerate(score_list):
        print("%d)" % (i + 1), "%s : Score = %0.5f" % (name, score))

problem at training

Hello, i encountered the following error when using train_nasnet_mobile.py. I actually encountered the same error when using all the other training models. Evaluate models worked very well.

Total params: 4,280,286
Trainable params: 10,570
Non-trainable params: 4,269,716

---

Epoch 1/20
Traceback (most recent call last):
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
preferred_dtype=default_dtype)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float64: 'Tensor("arg0:0", shape=(), dtype=float64)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_nasnet_mobile.py", line 93, in
validation_steps=(5000. // batchsize))
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/keras/engine/training.py", line 1426, in fit_generator
initial_epoch=initial_epoch)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/keras/engine/training_generator.py", line 155, in fit_generator
generator_output = next(output_generator)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/keras/utils/data_utils.py", line 793, in get
six.reraise(value.class, value, value.traceback)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/keras/utils/data_utils.py", line 658, in _data_generator_task
generator_output = next(self._generator)
File "/Users/jiazhengyuan/Documents/workspace/neural-image-assessment/utils/data_loader.py", line 101, in train_generator
train_dataset = train_dataset.map(parse_data, num_parallel_calls=2)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 853, in map
return ParallelMapDataset(self, map_func, num_parallel_calls)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1870, in init
super(ParallelMapDataset, self).init(input_dataset, map_func)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1839, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1802, in tf_map_func
ret = map_func(*nested_args)
File "/Users/jiazhengyuan/Documents/workspace/neural-image-assessment/utils/data_loader.py", line 61, in parse_data
image = tf.read_file(filename)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 528, in read_file
"ReadFile", filename=filename, name=name)
File "/Users/jiazhengyuan/venv3.62/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 533, in _apply_op_helper
(prefix, dtypes.as_dtype(input_arg.type).name))
TypeError: Input 'filename' of 'ReadFile' Op has type float64 that does not match expected type of string.

CoreML model input type is MultiArray instead of Image

I was able to successfully save the MobileNet NIMA model as a coreML model however the input is not what I was expecting. It shows the input as MultiArray instead of an Image

base_model = MobileNet((None, None, 3), alpha=1, include_top=False, pooling='avg', weights=None)
x = Dropout(0.75)(base_model.output)
x = Dense(10, activation='softmax')(x)

model = Model(base_model.input, x)
model.load_weights('weights/mobilenet_weights.h5')
coreml_model = coremltools.converters.keras.convert(model)
coreml_model.save("mobilenet_nima2.mlmodel")

CoreML metaData

Ideally what the input and output should resemble:

I also tried converting the model by passing in an image_input_names parameter

coreml_model = coremltools.converters.keras.convert(model, image_input_names='img', input_names='img')

but I received an unpack error from the coreML builder tools

/coremltools/models/neural_network/builder.py", line 2547, in set_pre_processing_parameters
    channels, height, width = array_shape
ValueError: need more than 1 value to unpack

Code where the error originates
https://github.com/apple/coremltools/blob/master/coremltools/models/neural_network.py

        # Add image inputs
        for input_ in spec.description.input:
            if input_.name in image_input_names:
                if input_.type.WhichOneof('Type') == 'multiArrayType':
                    array_shape = tuple(input_.type.multiArrayType.shape)
                    channels, height, width = array_shape

Any idea why the keras model is converting to a CoreML model with input MultiArray type instead of an image?

There have been MobileNet models that have been successfully converted to a CoreML model with image as an input e.g. https://github.com/hollance/MobileNet-CoreML

question for evaluation

Hello. You mentioned that evaluation requires cpu, train needs gpu.Just wondering would evaluation by gpu make the process faster? Thanks

I've computed histograms of the ground truth and predicted scores

Histograms of the ground truth and predicted scores from the article, p.7

And did the same for two models here:

It shows that MobileNet will show correct scores for very few ground truth images with score <4 and > 7.

Better one is this implementation (mobilenet): https://github.com/idealo/image-quality-assessment

Actually, I'm having troubles myself trying to fit MobileNet2, I'm getting something similar to your mobilenet image.

My histograms are bult on 0.1 subset of entire set.

Could you reach SRCC=0.51 with Mobilenet and SRCC=0.61 with Inception_v2?

I had the same try with you about NIMA, but I cannot reach such high SRCC in the paper. Could you reach that? By the way, I have a different implement with you, but I agree with your loss function. Thank you!

error at the last epoch

Epoch 00020: val_loss did not improve from 0.04966
Exception ignored in: <generator object val_generator at 0x7f588c71da20>
Traceback (most recent call last):
File "/home/tiger/jiazheng/neural-image-assessment/utils/data_loader.py", line 158, in val_generator
yield (X_batch, y_batch)
File "/home/tiger/jiazheng/venv34/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1596, in exit
self._default_graph_context_manager.exit(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.4/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/home/tiger/jiazheng/venv34/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
context.context().context_switches.pop()
File "/home/tiger/jiazheng/venv34/lib/python3.4/site-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list
Exception ignored in: <generator object train_generator at 0x7f58ac2835e8>
Traceback (most recent call last):
File "/home/tiger/jiazheng/neural-image-assessment/utils/data_loader.py", line 123, in train_generator
yield (X_batch, y_batch)
File "/home/tiger/jiazheng/venv34/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1596, in exit
self._default_graph_context_manager.exit(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.4/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/home/tiger/jiazheng/venv34/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
context.context().context_switches.pop()
File "/home/tiger/jiazheng/venv34/lib/python3.4/site-packages/tensorflow/python/eager/context.py", line 136, in pop
self.stack.pop()
IndexError: pop from empty list

after the last epoch, this error happened. I checked the dataset, the ava.txt matches the training images. all the epochs before were good. Thanks!

Hello author, why did I run the first epoch without a problem, and the second epoch program crashed.

Hello author, why did I run the first epoch without a problem, and the second epoch program [crashed.[]

)

i face the error when using it for other dataset not AVA

hello!

i tried to train with vgg16 structure.
but, after epochs are done, error codes come out like this.
and the data's format is same as AVA.

Exception ignored in: <generator object val_generator at 0x7f878d041728>
Traceback (most recent call last):
File "/home/kpark2/neural-image-assessment/utils/data_loader.py", line 160, in val_generator
yield (X_batch, y_batch)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1558, in exit
self._default_graph_context_manager.exit(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.5/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 4973, in get_controller
context.context_stack.pop()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/eager/context.py", line 100, in pop
self.stack.pop()
IndexError: pop from empty list
Exception ignored in: <generator object train_generator at 0x7f878d0416d0>
Traceback (most recent call last):
File "/home/kpark2/neural-image-assessment/utils/data_loader.py", line 125, in train_generator
yield (X_batch, y_batch)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1558, in exit
self._default_graph_context_manager.exit(exec_type, exec_value, exec_tb)
File "/usr/lib/python3.5/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 4973, in get_controller
context.context_stack.pop()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/eager/context.py", line 100, in pop
self.stack.pop()
IndexError: pop from empty list

help me!

why large training/validation loss difference between VGG16 vs inception_v2?

i got two troubles

1. i achieved large training/validation loss (0.12/0.13) with VGG16 on TID2013, this result differs with nima paper released (0.054).
   My experimental settings are pre-trained top layer with dropout rate 0.5, lr=1e-4, epoch=200, batch_size=50, batch=192, then fine-tuning the whole model with dropout rate 0.5, lr=3e-7, epoch=30, batch_size=50, batch=192. The dataset has 25 scenes processed by 24 distorted types with 5 levels. I use leave-one-distorted-level-out for validation, others for training.

2. With similar training manner, we divide dataset into three parts: training/validation/test. we use leave-one-scene-out as test, we got 25 models for each scene. But some scene predictions (10/25) are extremely bad, almost got same score with 3. i checked the image and found some failure cases are actually similar with training scene. Is this result reasonable?

does training need resizing the pics?

I see that evaluation need to resize the pic to 224*224, is that the same for training? Thanks

How to use faeture in feature_generator?

Hi titu1994,
I have just run your 3-step code and when I try to run pretrain_.py
I got this problem.
Traceback (most recent call last): File "pretrain_inception_resnet.py", line 82, in <module> model.fit_generator(features_generator(TRAIN_RECORD_PATH, batchsize=batchsize, shuffle=True), TypeError: features_generator() missing 1 required positional argument: 'faeture_size'
I knew it is because you didn't set feature in pretrain_.py
But I just can't figure out what does the faeture mean?

Can't instantiate abstract class DatasetV1 with abstract methods _as_variant_tensor, _inputs

Using the AVA dataset, we came across the following problems.
It shows that 'TypeError: Can't instantiate abstract class DatasetV1 with abstract methods _as_variant_tensor, _inputs'. We've googled it but cannot find the suitable answer.

Please help us.

Environment:

tensorflow-gpu 1.12.0
keras 2.24
python 3.6.6

Logs:

Traceback (most recent call last):
  File "train_mobilenet.py", line 88, in <module>
    validation_steps=(5000. // batchsize))
  File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 181, in fit_generator
    generator_output = next(output_generator)
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 709, in get
    six.reraise(*sys.exc_info())
  File "/home/hadoop/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 685, in get
    inputs = self.queue.get(block=True).get()
  File "/home/hadoop/anaconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/home/hadoop/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 626, in next_sample
    return six.next(_SHARED_SEQUENCES[uid])
  File "/home/hadoop/Documents/demo-ava-1/neural-image-assessment-master/utils/data_loader.py", line 101, in train_generator
    train_dataset = tf.data.Dataset().from_tensor_slices((train_image_paths, train_scores))
TypeError: Can't instantiate abstract class DatasetV1 with abstract methods _as_variant_tensor, _inputs

why high score from the distorted image

test two images (one for good quality and another distorted image) from the three trained models, i got scores as below:
Evaluating_inception_resnet : D:\test\bad.PNG
NIMA Score : 5.215 +- (1.719)

Evaluating_inception_resnet : D:\test\good.PNG
NIMA Score : 5.694 +- (1.815)

Evaluating_mobilenet : D:\test\bad.PNG
NIMA Score : 5.132 +- (1.723)

Evaluating_mobilenet : D:\test\good.PNG
NIMA Score : 5.653 +- (1.651)

Evaluating_nasnet : D:\test\bad.PNG
NIMA Score : 4.879 +- (1.759)

Evaluating_nasnet : D:\test\good.PNG
NIMA Score : 6.736 +- (1.593)

it makes me confused why the distored image still got high score?

EMD loss function maybe wrong

I found the emd definition here https://github.com/titu1994/neural-image-assessment/blob/master/train_mobilenet.py#L49

def earth_mover_loss(y_true, y_pred):
    return K.sqrt(K.mean(K.square(K.abs(y_true - y_pred))))

You are missing the CDF function. According to the paper: EMD(y_true, y_pred) = sqrt(mean(abs(CDF(y_true) - CDF(y_pred))))

y_true  = [0, 0, 0, 0, 0, 0, 0, 0.9, 0.1, 0]
y_pred1 = [0, 0, 0, 0, 0, 0, 0.9, 0, 0.1, 0]
y_pred2 = [0.9, 0, 0, 0, 0, 0, 0, 0, 0.1, 0]

y_pred1's loss should be less than y_pred2's loss

Error when I run train_mobilenet.py

I meet error when train mobilenet according to your guide. Please help me, thanks~

Traceback (most recent call last):
File "train_mobilenet.py", line 88, in
validation_steps=(5000. // batchsize))
File "/home/zw/work/NIMA/venv/local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/zw/work/NIMA/venv/local/lib/python2.7/site-packages/keras/engine/training.py", line 2192, in fit_generator
generator_output = next(output_generator)
File "/home/zw/work/NIMA/venv/local/lib/python2.7/site-packages/keras/utils/data_utils.py", line 793, in get
six.reraise(value.class, value, value.traceback)
File "/home/zw/work/NIMA/venv/local/lib/python2.7/site-packages/keras/utils/data_utils.py", line 658, in _data_generator_task
generator_output = next(self._generator)
File "/home/zw/work/NIMA/neural-image-assessment/utils/data_loader.py", line 102, in train_generator
train_dataset = tf.data.Dataset().from_tensor_slices((train_image_paths, train_scores))
TypeError: Can't instantiate abstract class Dataset with abstract methods _as_variant_tensor, output_shapes, output_types

The dataset print result is as below:
('Train set size : ', (250508,), (250508, 10))
('Val set size : ', (5000,), (5000, 10))
Train and validation datasets ready !

nasnet weights incorrectly loaded

So, I was just refactoring my fork of the code and I got bad results in the nasnet evaluation script.
It turns out there's a bug when loading the weights.

model.load_weights('weights/nasnet_weights.h5', by_name=True)

Remove by_name=true! Most layers have names generated on the fly, except maybe the fully connected layer. So what this does is probably only loading the fully connected layer's weights!
If you load the complete network, you will get good results!

how to install

please provide the tips on how to install.

when i train model on TID2013 dataset,something went wrong

I want to train a model on tid2013,so i changed some details in the script of data_loader.py.but i got an error:
Traceback (most recent call last):
File "train_mobilenet_tid.py", line 90, in
validation_steps=(500. // batchsize))
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training_generator.py", line 181, in fit_generator
generator_output = next(output_generator)
File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 709, in get
six.reraise(*sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 685, in get
inputs = self.queue.get(block=True).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
raise self._value
TypeError: Can't instantiate abstract class Dataset with abstract methods _as_variant_tensor, output_classes, output_shapes, output_types
I am so confused of this error,could you help me with this?

Regarding results

Hi,

Just asking, Does this code produce the same accuracy of the NIMA paper?

Nice job.

dead in the first step when train

when I tried "python train_nasnet_mobile.py", it failed as follows:
Epoch 1/20
2018-03-22 21:01:16.558680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
Exception in thread Thread-2:
Traceback (most recent call last):
File "software/python/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
File "software/python/lib/python2.7/threading.py", line 761, in run
self.__target(*self.__args, **self.__kwargs)
File "software/python/lib/python2.7/site-packages/keras/utils/data_utils.py", line 630, in data_generator_task
generator_output = next(self._generator)
File "prj/NIMA/neural-image-assessment/utils/data_loader.py", line 100, in train_generator
train_dataset = tf.data.Dataset().from_tensor_slices((train_image_paths, train_scores))

TypeError: Can't instantiate abstract class Dataset with abstract methods _as_variant_tensor, output_shapes, output_types

Traceback (most recent call last):
File "train_nasnet_mobile.py", line 94, in
validation_steps=(5000. // batchsize))
File "software/python/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "software/python/lib/python2.7/site-packages/keras/engine/training.py", line 2083, in fit_generator
generator_output = next(output_generator)
StopIteration

How to solve this problem

I run the train_model, and find some problems:
Traceback (most recent call last):
File "/home/ztt/下载/neural-image-assessment-master/train_model.py", line 196, in
main()
File "/home/ztt/下载/neural-image-assessment-master/train_model.py", line 188, in main
initial_epoch=0
File "/home/ztt/anaconda3/envs/tensorflow/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/ztt/anaconda3/envs/tensorflow/lib/python3.5/site-packages/keras/engine/training.py", line 2145, in fit_generator
generator_output = next(output_generator)
File "/home/ztt/anaconda3/envs/tensorflow/lib/python3.5/site-packages/keras/utils/data_utils.py", line 561, in get
six.raise_from(StopIteration(e), e)
File "", line 3, in raise_from
StopIteration: [Errno 2] No such file or directory: 's:\datasets\AVA_dataset\images/950742.jpg'