Research : Change Attention Transformer Inputs

[dreamquark-ai]

Research : Change Attention Transformer Inputs

[mustaphabenhajm]

@Optimox Hello, could you please clear more the idea ? do you mean the input of attentive transformer will be initial data + previous mask which will replace the priors of the previous step ? Thanks

[Optimox]

Hello @MustaphaBM

I'll try to rephrase what I meant at that time.

The attentive transformer from step 1 is taking a vector of size n_a as input, which has been computed by the initial feature transformer (number 0). Until here I'm totally fine with the idea of masking certain features from this.

The attentive transformer 2 however gets as input the n_a output of feature transformer 1, but this feature transformer 1 has never seen the full data because it was masked by the attentive transformer 1. And here I think there might be something wrong, how can you chose which feature to use if you have only seen part of them?

Obviously this would be a real problem if the mask did not change at instance level, here the mask can adapt to each instance. However I feel that it would be interesting to try to create the mask from the original data and not from the previous attentive transformer.

This would somehow lower the 'sequential' attention of TabNet but I think that keeping the previous mask as a prior for the update of the next mask could mitigate this.

Actually I think this would be quite easy to implement and try, but I'm not sure on which dataset I should to the benchmark to see whether there is a real improvement.

Hope this is clearer, let me know otherwise. Let me know if you perform some experiments I would be interested to know about the results.