Comments (6)
Group B:
* add domain and multilple langs from [Pontoon-Translations](https://huggingface.co/datasets/ayymen/Pontoon-Translations): cleaning is a bit challenging
Are you talking about cleaning the data itself or the metadata (lang codes)? I intend to release new versions of both Pontoon Translations and Weblate Translations (which has more languages BTW, but probably less quality for LID), but I'm not really sure how I'm going to fix lang codes.
about the cleaning, I meant more the tags like <playIcon>
or {$goal}
, for LID it should be removed, or otherwise it learn bad features. It's not too difficult, but it should be done. I will check your HF every once in a while to see if you publish anything new.
from glotlid.
Group A:
-
solved in v3, Nllbseed and Flores (zgh_Tfng, tzm_Tfng, taq_Tfng):
facebookresearch/flores#63 -
solved in v3, Flores (yue_Hant):
facebookresearch/flores#61 -
Nllbseed (ary_Arab):
facebookresearch/flores#64" -
solved in v3, yue bible is tokenized by space. We should concatenate them without space and then train the LID. update: after concatenation we found the label of yue for this data is not true, we removed this part of data.
-
clean further dag_Latn from wikipedia (citations).
from glotlid.
Group B:
-
add snk (GlotWeb, GlotSparse)
-
add ykg (GlotWeb, GlotSparse)
-
add srp-Latn (GlotWeb, GlotSparse): hbs languages (hrv, srp, cnr, bos) are misclassified more than other langs.
-
add domain and multilple langs from Pontoon-Translations: cleaning is a bit challenging
-
add lzz: https://incubator.wikimedia.org/wiki/Special:PrefixIndex/Wp/lzz
-
add evn: https://incubator.wikimedia.org/wiki/Special:PrefixIndex/Wp/evn
-
add syl: https://incubator.wikimedia.org/wiki/Special:PrefixIndex/Wp/syl/
-
add srb:
- https://www.vishwavani.org/wp-content/uploads/2022/09/Soura-oct-2022-Samarpam-PDF.pdf
- https://www.vishwavani.org/wp-content/uploads/2021/02/SOURA-MAR2021.pdf
- https://www.vishwavani.org/wp-content/uploads/2023/07/09-SOURA-AUG2023.pdf
- https://www.vishwavani.org/wp-content/uploads/2023/03/11-SOURA-APR2023.pdf
- https://www.vishwavani.org/wp-content/uploads/2023/05/08-SOURA-MAY-2023-1.pdf
- https://www.vishwavani.org/wp-content/uploads/2021/04/SOURA-MAY-2021.pdf
- https://www.vishwavani.org/wp-content/uploads/2024/02/06-SOURA-MAR2024.pdf
-
add pbs:
-
add kpo:
-
bible in australian indigenous languages: https://aboriginalbibles.org.au/
-
some ebibles in html format: https://download.sabda.org/yesusmesias/__ebible/html/
-
many langs data for cdc: https://wwwn.cdc.gov/pubs/other-languages
-
many picture book bible here: https://www.globalrecordings.net/en/scripts for example bkx: https://www.globalrecordings.net/en/script/812 or mhs: https://www.globalrecordings.net/en/script/mhs/395
from glotlid.
Group B:
* add domain and multilple langs from [Pontoon-Translations](https://huggingface.co/datasets/ayymen/Pontoon-Translations): cleaning is a bit challenging
Are you talking about cleaning the data itself or the metadata (lang codes)?
I intend to release new versions of both Pontoon Translations and Weblate Translations (which has more languages BTW, but probably less quality for LID), but I'm not really sure how I'm going to fix lang codes.
from glotlid.
Can you clarify why facebookresearch/flores#61 is solved? I don't see any update in their data.
from glotlid.
@laubonghaudoi For my project (GlotLID), the issue is resolved because I deleted the yue in my Flores benchmark. This project is GlotLID, which trains a better language identification system. Flores-200 is one of the benchmarks I used.
But to answer your question in general, this issue is not resolved in Flores-200 at its root. They made another project to maintain Flores: https://github.com/openlanguagedata/flores, but that also does not address this issue! Maybe someone needs to bring up this issue in the new project again.
from glotlid.
Related Issues (4)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glotlid.