Alphabet based training

[wenet-e2e]

Alphabet based training

[robin1001]

token means word piece? alphabet means character?

[pKrysenko]

Yes, you understand correct

[robin1001]

character can be used as model unit, just change the pipeline to use char. Generally, char has lower accuracy than word piece, so word piece is used as default model unit.

[pKrysenko]

With low resource data character based model shows better results then word piece model, based on my small research.
As far as i understand, only i need to change is the format.data file for training. if i use characters to train, i have to change, for instance, "▁CHAPTER ▁ONE" to "CHAPTER ONE" and corresponding token ids and obviously tokens shape and i don't need to decode text_bpe in test stage, am i right?

[robin1001]

for "CHAPTER ONE", the format.data should be "C H A P T E R [space] O N E", and the dict should be A-Z plus [space].

[robin1001]

for "CHAPTER ONE", the format.data should be "C H A P T E R [space] O N E", and the dict should be A-Z plus [space].

where [space] could be also replaced with a special char "▁" to simplify data processing.

[pKrysenko]

For dict is:
blank 0
unk 1
A 2
...
Z 28
sos/eos 29
Am i right?

[robin1001]

add one special symbol for space.

[pKrysenko]

Okey, many thanks and that's all what i need to change?

[robin1001]

1. the dict
2. training corpus should be prepared as char, such as "CHAPTER ONE", the format.data should be "C H A P T E R [space] O N E"

[pKrysenko]

Many thanks, sorry for dumb questions !