imatge-upc/egocentric-2016-saliency

Research on the prediction of visual saliency in egocentric vision.

egocentric-2016-saliency's Issues

Compute the saliency maps for a collection of images

Once a saliency map from a single image has been computed, the next step is to compute the saliency maps for a collection of them. Please check this internal ticket and ask @marccarne for details and examples about how to do this.

State of the art on egocentric saliency prediction

You should identify and read some (3-5) scientific papers or works where similar to your research.

I think that in the world of egocentric they use a lot the term "attention" as a similar concept to saliency.

I have found some papers that I would like you to look at and write a short summary (one paragraph for each):

Yamada, Kentaro, Yusuke Sugano, Takahiro Okabe, Yoichi Sato, Akihiro Sugimoto, and Kazuo Hiraki. "Can saliency map models predict human egocentric visual attention?." In Computer Vision–ACCV 2010 Workshops, pp. 420-429. Springer Berlin Heidelberg, 2010.

Matsuo, Kenji, Kentaro Yamada, Satoshi Ueno, and Sei Naito. "An attention-based activity recognition for egocentric video." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 551-556. 2014.

Bettadapura, Vinay, Irfan Essa, and Caroline Pantofaru. "Egocentric field-of-view localization using first-person point-of-view devices." In Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on, pp. 626-633. IEEE, 2015.

Fathi, Alireza, Yin Li, and James M. Rehg. "Learning to recognize daily actions using gaze." In Computer Vision–ECCV 2012, pp. 314-327. Springer Berlin Heidelberg, 2012

In particular, I want you to answer this questions:

• Which dataset did they use ?
• Which metric did they use ?
• Is there any aspect in their methodology we could adopt ?
• Could we compare our results with them somehow ?

Answer to this issue with a paragraph every time you finish reading each paper. Make sure you answer the questions I posed.

Privacy issues for the dataset recordings

Professor O'Connor wrote:

I think that the suggested data set would be very valuable and if we succeed in creating it, it would be great to be able to release it for others to use. Given thus, we should try to keep privacy considerations in mind when creating the data set so that there are no problems if we decide to go this route e.g. we could make sure that if people appear in the video, they are members of the group here who can give formal consent, make sure there are no car license plates, etc. From our ACM Grand Challenge data sets we learnt that its much better to consider these things up front rather than try to fix them later

Dataset link not working

The link to download the dataset is not working. Could you please provide a link to download the dataset.
https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/
Thank You

Evaluation metric to compare saliency map and eye gaze

Choose among the metrics used in MIT saliency benchmark which one can be used in the case when a saliency map (2D grayscale image) is to be compared with the actual location provided by the eye tracker. Check also its implementation in the toolbox provided by MIT300.

Can you compute a saliency map from an egocentric frame ?

Hi @monicachs ,

I was wondering if, after Junting's instructions, you are able to compute a saliency map from an image. If so, you should run it on one of Sergi's frames.

What is the status of this stage ?

Extract frames from the Tobi video

Learn how to conveniently extract frames from the video sequence providede by Tobi glasses. Check Sergi Imedio's thesis as well as 'srun' command to be used in GPI servers.

Evaluation on the Georgia Tech Egocentric Activities - Gaze (+)

I just found a dataset which seems to exactly do what we are expecting to do. It is the Georgia Tech Egocentric Activities - Gaze (+). It should be referred in your report. Not sure if we have time for the submission in Dublin, but for sure in the one for Terrassa.

It would also be desirable to generate and assess the saliency maps over them.

Locations to record a new dataset

Places were we could record:

• University
• Residence
• Supermarket
• Gym
• Touristic places: Phoenix park, Trinity college, Spire, Leffey, ...