Some question about the MEGABYTE

[lucidrains]

Some question about the MEGABYTE

[relic-yuexi]

For character-level model, how to set the termination mark, the sequence generated behind are meaningless spaces, but afraid that some sentences have spaces.

like "" will be sepreate "<" "E" "O" "F" ">"

[Law-Lawliet]

@relic-yuexi small question since looks like you were able to make the code work. Can you show me in some way how to make this work ? Sorry I'm new to AI stuff

[relic-yuexi]

Me too. I just want to makea small GPT-like model in chinese. By my experiment, it looks like it has a faster speed than the tradtional Transfomer. And it can use none complex tokenizer, i just use the func ord() to tokenizer the unicode.

Just input some text as the dataset, then modify the train.py as you like.

[relic-yuexi]

When i used the ord(" ") as a pad_id, it will lead to nan loss and hard training and error code.