Filter HTML entities in training scripts data

[argosopentech]

Filter HTML entities in training scripts data

[PJ-Finlay]

Interesting, good catch. I was able to reproduce this. The tokenization looks fine so the sequence to sequence model is just outputting HTML characters. The training data (OpenSubtitles, ParaCrawl, UNPC from OPUS) is generally created by scraping real world data so I'm guessing some escaped HTML codes made their way into the training data. It might make sense to add a filter in the training scripts to filter out any lines that have HTML escape codes in them.

[PJ-Finlay]

I ended up removing this filtering because it substantially slowed down training. I'm still looking it a better solution but hoping that better quality models in general will do this less.

[pierotofy]

You could also use: https://docs.python.org/3/library/html.html#html.unescape

And a regex or beautifulsoup https://stackoverflow.com/questions/9662346/python-code-to-remove-html-tags-from-a-string

Which would make it more generic.

[PJ-Finlay]

That might make sense but for now I'm just removing data that has one of these strings instead of trying to clean it up.

[PJ-Finlay]

This should be done for any models trained going forward.

html_entities = [''', ' ', '<', '>', '"']
html_tags = ['<a>', '<p>', '<h1>', '<i>']

Suggestions for additional things that might be in training data to filter appreciated.

[Qix-]

Still seeing this on models downloaded fresh today, was there perhaps a regression?

https://github.com/Qix-/bellingbot/issues/2 has some sample text; RU->EN.