minguinho26/prefix_aac_icassp2023

Official Implementation of "Prefix tuning for Automated Audio Captioning(ICASSP 2023)"

prefix_aac_icassp2023's Issues

About pretrained model

Hello，
I would like to know when using Huggingface's gpt2 model， is the pt file which you shared by your
GoogleDrive from your own training， or does it correspond to the gpt2 on the website?

Thank you for your early reply.

the validation set of audiocaps dataset

According to the download link you provided, the audiocaps dataset does not have a validation set. Can you provide audiocaps validation set data？

Can't find Inference.py

Thank you for sharing your codes.

But, I can't find the Inference.py.
Could you upload this file?

Pretrained Models for reproducibility

Hi,

First of all, your project is great and I am working on this project. However, I am not able to reproduce your results reported in the ICASSP paper. It would be great if you could share your pre-trained models for reproducibility.

Thanks!

ERROR in the Ablation study

I'm sorry for bothering you.
I have confronted the problem when I set both temporal_prefix_size and global_prefix_size to 0. I know this operatioin means no mapping network in the model.
The error message is ''the size of tensor a (88) must match the size of tensor b (31) at non-singleton dimension 3''
as following,

I do not konw why it happened .
I tried to figure it out, but I failed.
I would be appreciated for your early response!!!
Thank you very much!

requirements

Can you show me the requirements for running environment?

About Prefix vector in ablation study

Hello, I have another question when I want to replicate ablation study about the mapping network.

    if self.temporal_prefix_length > 0:
        # temporal_prefix_vector = self.temporal_mappingnetwork(temporal_feature).view(-1, self.temporal_prefix_length, self.gpt_embedding_size)
        temporal_prefix_vector = self.temporal_mappingnetwork(temporal_feature).view(-1, self.temporal_prefix_length, self.gpt_embedding_size).to(self.device)
    elif self.global_prefix_length + self.temporal_prefix_length == 0:
        temporal_feature = temporal_feature.permute(0, 2, 1, 3).contiguous()
        temporal_feature = torch.reshape(temporal_feature, (temporal_feature.size()[0], temporal_feature.size()[1], -1))
        temporal_prefix_vector = self.temporal_mappingnetwork(temporal_feature)

    if self.global_prefix_length > 0:
        global_prefix_vector = self.global_mappingnetwork(global_feature).view(-1, self.global_prefix_length, self.gpt_embedding_size)
        global_prefix_vector = global_prefix_vector.to("cuda:1")
    elif self.global_prefix_length + self.temporal_prefix_length == 0:
        global_prefix_vector = self.global_mappingnetwork(global_feature)
        global_prefix_vector = global_prefix_vector.view(global_feature.size()[0], 11, 768)  # [bsize, 11, 768]
        global_prefix_vector = global_prefix_vector.to("cuda:1")

        

    if self.temporal_prefix_length > 0 and self.global_prefix_length == 0:
        prefix_vectors = temporal_prefix_vector
    elif self.temporal_prefix_length == 0 and self.global_prefix_length > 0:
        prefix_vectors = global_prefix_vector
    else:
        prefix_vectors = torch.cat(
            (temporal_prefix_vector, global_prefix_vector), dim=1)

if I set self.global_prefix_length > 0 and self.temporal_prefix_length = 0, does it means the method with a mapping network and global feature,vice versa.
And is I set self.global_prefix_length = 0 and self.temporal_prefix_length = 0, does it means the method without mapping networks, just two features which is fed into gpt2?

Thank you for your early reply!

About Evaluation files

Hello, Nice work!!!
When I replicate your work， I found there is not any evaluation files you mentioned in README.

Evaluation_AudioCaps.py and Evaluation_Clotho.py

Are these two files just using the eval_model() in Train.py?
Or anything else? If so, could you release these tow files?

Thank you for your early reply!!!