Using the methods for a custom architecture

[hila-chefer]

Using the methods for a custom architecture

[hila-chefer]

Happy to help :)

it sounds like the model you’re describing is more similar to the DETR case than the CLIP case. DETR too has a CNN backbone and then a Transformer encoder-decoder and outputs logits indicating which object is in the image.
The implementation of our rules for DETR can be found here and please refer to our colab notebook for examples.

I hope this helps, if I missed something from your description please let me know.

[hila-chefer]

Hi @SreeHarshaNelaturu, thanks for your interest!

(1) as most explainability methods, ours is focused on classification models, however it is possible to make adjustments in cases where the output can be interpreted similarly to classification scores (e.g. the CLIP similarity score can be viewed as a classification logit).
Could you please specify the output of your model, and what it means in the context of the inputs?

(2) indeed, the attention weights are extracted after the softmax layer.

best,
Hila.

[SreeHarshaNelaturu]

Hello, thanks for the prompt response!

The input to the model is a set of video frames of shape (1, 1, 240, 88, 88) (Batch Size x Channels x Number of Frames x H x W). This is passed to an encoder model that provides tokens of shape (240, 1, 768).

Encoder first uses ResNet like encoder to generate embeddings which are then sent through transformer encoder layers.

This is then passed through a Transformer decoder which returns an output of shape (1, seq_len, vocab_size) (example: 1, 39, 1000) which are basically logits.

I assume this would fit the template as you described as more or less a classification problem if we perform greedy decoding on the final logits (top 1)?

The objective is to see how a predicted word maps to the set of frame (s) that led to its prediction.

Would appreciate any suggestion on how to proceed.

Thanks!

[May-226]

Hi! I am reading the colab notebook of DETR example, in definition of funtion 'evaluate', there seems to be some redundant codes generating 'conv_features, enc_attn_weights, dec_attn_weights'. These variables seem not to be used after being generated, but I felt they are supposed to be meaningful. Could you suggest how should I interpret those variables and clarify a little what's the original motivation of this piece of code? Many thanks!