rishikksh20/fnet-pytorch

Unofficial implementation of Google's FNet: Mixing Tokens with Fourier Transforms

FNet: Mixing Tokens with Fourier Transforms

Pytorch implementation of Fnet : Mixing Tokens with Fourier Transforms.

Citation:

@misc{leethorp2021fnet,
      title={FNet: Mixing Tokens with Fourier Transforms}, 
      author={James Lee-Thorp and Joshua Ainslie and Ilya Eckstein and Santiago Ontanon},
      year={2021},
      eprint={2105.03824},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

fnet-pytorch's Issues

are you ready to Implement embedding part?

thanks for your impl, and I found that you may not code emdedding part, I think the part is important too.

I suggest that you could write a demo from input to prediction if you have free time, excespecially how to code embedding part.

Layer Normalization

Hi,
thanks for a great implementation!

I wanted to clarify one thing that mismatches with the code, proposed in the article itself. In your code, you pre-normalize inputs, so that they are passed through LayerNorm before FFT. In the code, presented in the article, they have:

class FNetEncoderBlock ( nn . Module ) :
30 f o u r i e r _ l a y e r : Fou rie rT ran sfo rmLa ye r
31 f f _ l a y e r : FeedForwardLayer
32
33 @nn. compact
34 def _ _ c a l l _ _ ( s e l f , x , d e t e r m i n i s t i c ) :
35 m i x i n g _ o ut p ut = s e l f . f o u r i e r _ l a y e r ( x )
36 x = nn . LayerNorm (1 e−12 , name=" mixing_laye r_no rm " ) ( x + &
m i x i n g _ o ut p ut )
37 fe ed _fo rw a rd _o utp ut = s e l f . f f _ l a y e r ( x , d e t e r m i n i s t i c )
38 r e t u r n nn . LayerNorm (
39 1e−12 , name=" output_la ye r_no rm " ) ( x + fee d_fo rwa rd _outp ut )

which in my view is done in the opposite order.
Am I mistaken or is it indeed a bug?

FNet in TTS Encoders

Dear @rishikksh20 ,

Thanks for your implementation! Can I ask what do you think about the effectiveness of Furier features as well as FNet when it's used to replace multi-head attention in TTS encoders which is quite a concrete approach so far! I'm thinking about if it can help to reduce the complexity of the TTS models but still maintain good performance 🤔

Thanks!

Max

Masking/padding tokens in sequences ?

How can we mask/pad tokens for sequences of varied length ?
When we use fft along dimension (-2) , sequences , if we just use zero padding the result will be skewed.
torch.fft.fft(torch.fft.fft(hidden_states.float(), dim=-1), dim=-2).real

post-norm or pre-norm

Hi,

From the Figure in the paper, I think the post-norm is used rather than pre-norm.

What is your opinion?