[Question] Use of MLP in Transformer Encoder/Decoder

[pi-tau]

[Question] Use of MLP in Transformer Encoder/Decoder

[phlippe]

Hi @pi-tau, there exist many variations of Transformers, and you can indeed replace the MLP with a different layer like convolution. In Computer Vision, combining convolutional methods with Transformers was a hot topic for a while, e.g. https://arxiv.org/abs/2107.06263. But in general, when you go to scale, it is usually best to give the model full flexibility by only using MLPs, and have the attention as arbitrary ways of communicating. For very small datasets, such inductive biases are more important. This is why RNNs can still beat Transformers when you train them from scratch on very small NLP datasets.

[pi-tau]

Update:
I found this paper on text-to-speech and for their FFT module (chap 3.1.) they use an encoder with two convolutional layers as non-linearity instead of the MLP (probably modelling a depthwise separable convolution idk?).

Their argument is that nearby sequence elements are more closely related in the character/phoneme sequence in speech tasks. But with byte pair encoding in language tasks wouldn't it make sense to also use this design?