Question Architecture (images)

[abduallahmohamed]

Question Architecture (images)

[abduallahmohamed]

Hello, Thanks for asking. Using images or videos is optional in the architecture. This is what you see in the ablation study in the paper. If set to True, it should extract the images or videos iirc. Best, ------------------------------------------ Abduallah Mohamed abduallahmohamed.com

…

On Thu, 23 Feb 2023 at 02:07, MZ82020 ***@***.***> wrote: Hello, Thank you for your amazing work! But I have a question. You use images in your architecture but in the code the configuration is set to False: Load_image, video_back and background_back are set set false in your code. How is this? Can you explain me please? Where is the part of the extract and use of images please? How you use your images please? Thanks! — Reply to this email directly, view it on GitHub <#6>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIVLGYIAVUV3Z54U5CH6BTWY4ZGNANCNFSM6AAAAAAVFNNX6E> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[MZ82020]

I didn't really understand the part related to the use of images. What is the difference between Load_image, video_back and background_back please? Can you give me more details on the part where you use the images please?

[abduallahmohamed]

Sorry for the delayed response.
The model has three modes:

1. Without any images or video data, this is where we set load_img = false
   We set load_img = true so the data loader load all images in these two modes:
2. One that uses the last image observed in a sequence, we set background_back = true in this case, so the model uses the last observed image to fuse with the observation embedding.
3. One that uses all of the images observed in a sequence (aka video), we set video_back = true, so the model uses the whole sequence as embedding

I hope this helps