feiyanhu/fastsal

FastSal

FastSal is a fast saliency map predictor base on MobielNet V2 backbone. FastSal (A) achieves near real time speed on CPU (>30FPS). There are 2 versions of FastSal, A(ddition) and C(oncatenetion), which can be chosen by specifying -model_type. Two sets of pretrained weights from the SALICON 2017 and COCO 2019 detection tasks are provided.

Publication

Find our pre-print on arXiv: FastSal: a Computationally Efficient Network for Visual Saliency Prediction.

@article{hu2020fastsal,
  title={FastSal: a Computationally Efficient Network for Visual Saliency Prediction},
  author={Feiyan Hu and Kevin McGuinness},
  journal={arXiv preprint arXiv:2008.11151},
  year={2020}
}

Install Dependencies

Use the package manager pip to install python library dependencies.

pip install -r requirements.txt

Usage

Download pretrained teacher models and/or weights

The SALGAN teacher weights can be downloaded from here.

The DeepGaze II computational graph can be downloaded from here

Pretraining

The SALGAN teacher weights are needed for pretraining.

Supports salicon or coco for -dataset_name. Supports A or C for -model_type.

Pretrain on SALICON 2017:

python pretrain.py \
-batch_size 30 \
-dataset_name salicon \
-dataset_path DATASET_PATH \
-teacher_path SALGAN_PATH \
-save_dir checkpoint \
-model_name pretrained_salicon

Pretrain on COCO:

python pretrain.py \
-batch_size 30 \
-dataset_name coco \
-dataset_path DATASET_PATH \
-teacher_path SALGAN_PATH \
-save_dir checkpoint \
-model_name pretrained_coco

Finetuning

The SALGAN teacher weights are needed for fine tuning. Fine tuning can be either from pretrained weights or random initialized weights. If -pretrain_path is not provided then randomly intialized weights are used.

Supports salicon or coco for -dataset_name. If -dataset_name is coco, -pseudo_path needs to be specified. Supports A or C for -model_type

Fine tune FastSal A with a SALGAN teacher and pretrained weights from SALICON 2017:

python fine_tune.py \
-model_type A \
-batch_size 30 \
-dataset_name salicon \
-dataset_path DATASET_PATH \
-teacher_path SALGAN_PATH \
-save_dir checkpoint \
-model_name finetune_salicon \
-pretrain_path weights/salicon_salgan_pretrain.pth

Fine tune FastSal C with pseudo saliency maps generated by DeepGaze II and pretrained weights from COCO. In order to train on the COCO dataset, pseudo labels have to be generated and saved at PSEUDO_PATH. In our example, we use Deepgaze II to generate pseudo saliency maps. We save the probability saliency map with central bias in numpy files. We can provide this data upon request.

python fine_tune.py \
-model_type C \
-batch_size 30 \
-dataset_name coco \
-dataset_path DATASET_PATH \
-save_dir checkpoint \
-model_name finetune_coco \
-pretrain_path weights/coco_salgan_pretrain.pth \
-pseudo_path PSEUDO_PATH

Evaluation

Supports salicon or mit1003 for -dataset_name. If mit1003 is chosen for -dataset_name, -batch_size needs to be set 1. Supports A or C for -model_type

Evaluation of FastSal A on the SALICON 2017 validation set:

python eval.py \
-model_type A \
-batch_size 30 \
-dataset_name salicon \
-dataset_path DATASET_PATH \
-model_path weights/salicon_A.pth

To load FastSal in python

Examples of how to load FastSal are in simple_load.py:

model = fastsal.fastsal(pretrain_mode=False, model_type='A')
state_dict, opt_state = load_weight(MODEL_PATH, remove_decoder=False)
model.load_state_dict(state_dict)

To predict using a image or a folder with images

In this example, the input and output folder are both image_example. Currently the script only takes images files with '.jpg', '.png' and '.jpeg' extensions.

python fastSal_predict.py \
-input_path image_examples/
-output_path image_examples/

The default settings for prediction -model_type is A, finetune_dataset is salicon, -batch_size is 1, prebability_output is False and gpu is True. Supports A or C for -model_type. Supports salicon or coco for -fintune_dataset. If -probability_output is True, the output files are saved in numpy data format.

The other arguments are:

python fastSal_predict.py \
-model_type A \
-finetune_dataset coco \
-input_path image_examples/ \
-output_path image_examples/ \
-batch_size 30 \
-probability_output True \
-gpu False

Future Development

TensorFlow implementation.

Acknowledgement

This work has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/15/SIRG/3283 and SFI/12/RC/2289 P2.

fastsal's Issues

why salicon_A outperform than salicon_C on SALICON test data?

Hello, I test your model on SALICON2017,and why salicon_A outperform than salicon_C on SALICON test data?

Image input for pretrained SalGAN teacher network?

As far as I saw, the SalGAN teacher network gets fed with 'BGR' image data where only the ImageNet mean values were subtracted, no further normalization using the standard deviation. Is my understanding correct?

Is there any more information on what is the appropriate input for the networks? Because from the torchvision documentation of 'vgg16' I would have assumed that the color channel order is 'RGB' and the mean and std values to apply would be the same as for the student network.

Information on downloading Coco and Salcon datasets

For reproduction it would be nice to have some more information on how to download the appropriate datasets. Can more information or download links get provided?

Regarding the mentioned data in the data of the COCO Detection 2019 task, is there anymore information available on how to download this data specifically?
On the COCO page I can only find information on how to download data from 2014 or 2017 fro training.

Thanks.

Hello, can you provide the pretrain model of fastsal ，thank you very much

Creation of pseudo saliency maps (COCO)

Can additional information be shared how the pseudo saliency maps for the COCO dataset got created?

Was just the trained DeepGazeII model used to create them?

Thanks

UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.The local variable vgg_img in the function read_vgg_img() in utils.py is referenced before assignment.

I try to "predict using a image or a folder with images" according to the instruction on the website https://github.com/feiyanhu/FastSal, and get an error as pasted below. The local variable vgg_img in the function read_vgg_img() in utils.py is referenced before assignment. I wonder how this error can be solved?
Note that the python version (3.7) is not provided on the github, but is selected by myself.

(fsal) w@X299-WU8:/demo/w/00-ivc/FastSal$ python fastSal_predict.py -input_path image_examples -output_path image_examples
image folder is image_examples

Traceback (most recent call last):

File "fastSal_predict.py", line 91, in

args.probability_output, args.batch_size, gpu=args.gpu)

File "fastSal_predict.py", line 48, in predict

for x, original_size_list, output_path_list in simple_loader:

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in next

data = self._next_data()

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data

return self._process_data(data)

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data

data.reraise()

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise

raise self.exc_type(msg)

UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.

Original Traceback (most recent call last):

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop

data = fetcher.fetch(index)

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch

data = [self.dataset[idx] for idx in possibly_batched_index]

File "/home/w/anaconda3/envs/fsal/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in

data = [self.dataset[idx] for idx in possibly_batched_index]

File "fastSal_predict.py", line 33, in getitem

vgg_img, original_size = read_vgg_img(img_path, (192, 256))

File "/demo/w/00-ivc/FastSal/dataset/utils.py", line 176, in read_vgg_img

original_size = vgg_img.size

UnboundLocalError: local variable 'vgg_img' referenced before assignment