some implementations are different from the original paper

[lucidrains]

some implementations are different from the original paper

[lucidrains]

@ZihaoH nevermind i found the bug and it converges ok, just released in 0.1.0

their design for number 2 may have slightly better gradients

[lucidrains]

@ZihaoH

yea you are correct, i didn't do the implementation to the exact detail. i've built this sort of thing too many times and just wanted to capture the overall idea

number one, this will be negligible difference, if what you mean is that they used a separate fine token embedding at the coarsest level for patching

i just tried out their number two, but didn't see better results than the way i did it, and code ends up being more complicated. you are welcome to give it a run (or if you can spot a bug, let me know)