echo0125/mat-memory-and-anticipation-transformer

[ICCV 2023] Official implementation of Memory-and-Anticipation Transformer for Online Action Understanding

mat-memory-and-anticipation-transformer's Issues

What is 'knn' in DotProductAttention Class?

https://github.com/Echo0125/Memory-and-Anticipation-Transformer/blob/main/src/rekognition_online_action_detection/models/transformer/multihead_attention.py

DotProductAttention Class is different from the one in LSTR

class DotProductAttention(nn.Module):

def __init__(self, dropout=0.0):
    super(DotProductAttention, self).__init__()

    self.dropout = dropout

def forward(self, q, k, v, attn_mask=None, knn=False):
    B, N1, N2 = q.shape[0], q.shape[-2], k.shape[-2]
    attn_output_weights = torch.bmm(q, k.transpose(1, 2))
    if attn_mask is not None:
        attn_output_weights += attn_mask
    if knn:
        mask=torch.zeros(B,N1,N2,device=q.device,requires_grad=False)
        index=torch.topk(attn_output_weights,k=int(N2 * 3 // 4),dim=-1,largest=True)[1]
        mask.scatter_(-1,index,1.)
        # attn_output_weights = torch.where(mask>0,attn_output_weights,torch.full_like(attn_output_weights,-1e7))
        attn_output_weights = torch.where(mask > 0, attn_output_weights, torch.full_like(attn_output_weights, float('-inf')))

    attn_output_weights = F.softmax(attn_output_weights, dim=-1)
    attn_output_weights = F.dropout(attn_output_weights,
                                    p=self.dropout,
                                    training=self.training)
    attn_output = torch.bmm(attn_output_weights, v)
    return attn_output

What is this?
is there any explanation in the paper?

Cannot reproduce THUMOS Kinetics result

Hi, I trained a model from scratch as written in readme, however, my mAP is only 69 ~ 69.5, which is lower than the reported performance 71.6 (I used kinetics for THUMOS) and LSTR

does this difference come from inference mode?

Did you get the result by running the script?
my torch version is 1.13.1+cu117

Request for the extracted features of TV Series and HDD datasets

I have seen that the author cannot share the extracted features owing to the copyright.
I would really appreciate it if anybody can share the pre-extracted features of TV Series and HDD with me or enlighten me with the detailed ways to extract the features myself.

Hello @mugenggeng, sorry to bother, I have read about the issue you opened in request for the features, and I am wondering if you have finished the extraction. If yes, I sincerelly hope that you can share that with me, Thanks a lot!

The project cannot be run

Sorry to bother you.,I'd like to ask what the config_file is needed to run the program.,The program error is because I didn't specify the path of the config_file.,Can you please help me solve it?Thank you.

Feature extraction

Sorry! I have another question about the feature extraction. Because I want to apply MAT to other datasets, so I try to extract the features by myself. As section 5.2 Implementation details, "extract the frames at 4 FPS for training and validation". I have resample the videos at 24 FPS and extract RGB and flow(视频帧和光流) using mmaction2 But I don't know how to set the video chunk size to 6. How could you do that and are there any tools in mmaction2 can help to realize this step?

about target_perframe file

I'm uncertain about how to create the target_perframe file. Could you please describe how to create it and provide the relevant code? It would greatly assist me.

Questions about essays

“We then employ NL learnable tokens QL ∈ R NL×D as the long-term memory queries and a weight-shared transformer decoder block to query Classifier each segment.”In it, the parameter NL is defined according to what？ “we use NF future queries QF ∈ RNF ×D and a trans�former decoder block to query ME.”In it, the parameter NF is defined according to what？Thank you for your kind answers!

Some questions about papers.

Thank you for your excellent work! I would like to ask a question about the paper.
As section 5.3 Future Queries Renewal outline, "renewing times = 0” denotes no renewal and utilizes FA. I think renewing times = 1 denotes replacing the FA with Q'f. But what is the meaning of renewing times = 2 ? Is it the same as the one shown in the picture below?

can't find the module 'rekognition_online_action_detection'

Hi! Why i can't find the module 'rekognition_online_action_detection'?

The precision with batch inference is not as good as when trained

Sorry to bother you, I recently had a problem with the batch test model: the accuracy obtained with the batch test was not as good as the test accuracy obtained during training. Do you know why?

FUTURE_SECONDS and ANTICIPATION_SECONDS

The code shows FUTURE_SECONDS is 12, ANTICIPATION_SECONDS is 2, and fps is 4.

However, the ablation experiment Figure 5 indicates that the performance published in the article is achieved when the anticipation length is 24 seconds.

Which of FUTURE_SECONDS and ANTICIPATION_SECONDS controls the anticipation length?

Why does that variable represent the time domain of 24 seconds?

Thanks for your reply.

Regarding checkpoint

Congratulations on your fantastic work and all the best for your presentation. I wanted to know:

1. Link to the supplementary.

2. Can you release the checkpoints for EK-100 and THUMOS online action detection?

3. To reproduce EK-100 and THUMOS online action detection, what changes do I need to make to the code? And any other change to config is needed?

4. How is the work-memory sample rate and long-memory sample rate decided in https://github.com/Echo0125/Memory-and-Anticipation-Transformer/blob/b84186438d748c599146a512a089ddc6a93a1761/configs/EK100/MAT/mat_long_64_work_5_kinetics_1x.yaml#L18?

Question about the code

                anticipation_queries = self.pos_encoding(
                    self.final_query.weight[:self.cfg.MODEL.LSTR.ANTICIPATION_LENGTH
                                            :self.cfg.MODEL.LSTR.ANTICIPATION_SAMPLE_RATE, ...].unsqueeze(1).repeat(1,work_memories.shape[1], 1),
                    padding=self.work_memory_num_samples)
                work_memories = torch.cat((work_memories, anticipation_queries), dim=0)

Hello @Echo0125 , I have a another question about your code of MAT. In the code above, an anticipation_queries is concatenated with the work_memories. Could you please tell me what is the function of anticipation_queries, and why it is necessary to combine the work_memories and the anticipation_queries? Thanks so much!

HDD and TVSeries dataset

I'm sorry to bother you. I'm very interested in your research. I would appreciate it if you could provide us with features about HDD and TVSeries.

Issue about creating

Thanks for your great work and sharing.

When I try to add modules in the original MAT framework, I find that only creating new modules without using them, the detection results will change. Have you encountered this problem? How can I fix it?

Looking forward to your reply