thomasthiebaud/spacy-fastlang

Language detection using Spacy and Fasttext

spacy_fastlang

Install

Assuming you have a working python environment, you can simply install it using

pip install spacy_fastlang

Usage

The library exports a pipeline component called language_detector that will set two spacy extensions

• doc._.language = ISO code of the detected language or xx as a fallback
• doc._.language_score = confidence

import spacy_fastlang  # noqa: F401 # pylint: disable=unused-import
nlp = spacy.load("...")
nlp.add_pipe("language_detector")
doc = nlp(en_text)

doc._.language == "..."
doc._.language_score >= ...

Options

Check the tests to see more examples and available options

License

Everythin is under MIT except the default model which is distributed under Creative Commons Attribution-Share-Alike License 3.0 by facebook here

spacy-fastlang's Issues

french number not detected as french

french word for number are not detected as 'fr' language
small test program below yield result:

Text: ceci est du francais LANG: fr SCORE: 0.8692260980606079
Text: donnez quarante cinq cinquante euros LANG: xx SCORE: 0.44422411918640137
Text: donnez 45 50 euros LANG: en SCORE: 0.20330046117305756

import spacy
import spacy_fastlang
nlp_lang = spacy.blank('xx')
nlp_lang.add_pipe('language_detector', config={"supported_languages": ["fr","en"]})

texts = ["ceci est du francais","donnez quarante cinq cinquante euros", "donnez 45 50 euros"]

for t in texts:
    doc = nlp_lang(t)
    print( "Text: {} LANG: {} SCORE: {}".format( t, doc._.language, doc._.language_score))

Lower spacy version requirements

Currently the required spaCy version is set to 3.6.0 or higher. However, AFAIK the LanguageDetector does not use any spaCy features specific to that version so the version constraint can safely be relaxed to a lower version (probably >= 3.0.0)

I personally tried it with spaCy 3.4.4 and it worked without any issues

Error when installing spacy_fastlang

Hi,
source: https://spacy.io/universe/project/spacy_fastlang
This is what I entered in the Google Colab notebook but I get the following error

!pip install spacy_fastlang

ERROR: Could not find a version that satisfies the requirement spacy_fastlang (from versions: none) ERROR: No matching distribution found for spacy_fastlang

How can I fi this? Thank you.

Not good at detecting english?

Hi, I tried this against youtube transcripts generated by Whisper, and I found that it doesn't do a good job of predicting english.

Is this expected behavior? Is it better at some languages than others?

Thanks for any help, Kastan