cisnlp/glotlid

GlotLID: Language Identification with Support for More Than 2000 Labels -- EMNLP 2023

glotlid's Issues

Model Output to a 2 character output

For language translation, most of the models need a 2-character source input language (en, hi) etc. Is there a way to get that kind of output from the model?

add source

Group A: Please add here any possible speculation to have cleaner sources and evaluation data.

Group B: Please add any possible new sources here, especially those concerning languages not included.

Which ISO format?

Hello, in the description you mention that you use "three-letter ISO codes with script)". But which exactly format you use? Could you please provide the specific ISO format you rely on, please?

Modify to output BCP-47 labels?

Firstly, thanks to the authors for releasing the model on GitHub 🙂!

I'm currently benchmarking different language identification models. As highlighted in Section 5.2 of the GlotLID paper, matching the language code metadata between models and datasets can be difficult.

Given that BCP-47 is backwards compatible with ISO-639-3 and is the IETF's best practice for language codes, would it be possible to have an alternative version of GlotLID that uses BCP-47 labels?

Thanks!