rspeer/language_data

An optional supplement to `langcodes` that stores names and statistics of languages.

language_data: a supplement to langcodes

This package is not meant to be used on its own. Please see langcodes for documentation.

language_data is a supplement to the langcodes module, for working with standardized codes for human languages. It stores the more bulky and hard-to-index data about languages, particularly what they are named in various languages.

For example, this stores the data that tell you that the code "en" means "English" in English, or that "francés" is the Spanish (es) name for French (fr).

The functions and test cases for working with this data are in langcodes, because working with the data correctly requires parsing language codes.

Data

The data included in this package is:

• The names of various languages, in various languages
• The estimated population that speaks each language
• The estimated population that writes each language

These are all extracted from the Unicode CLDR data package, version 40, plus a few additional language names that fill in gaps in CLDR.

Caveats

• The estimates for "writing population" are often overestimates, as described in the CLDR documentation on territory data. In most cases, they are derived from published data about literacy rates in the places where those languages are spoken. This doesn't take into account that many literate people around the world speak a language that isn't typically written, and write in a different language.

• The writing systems of Chinese erase most (but not all) of the distinctions between spoken Chinese languages. You'll see separate estimates of the writing population for Cantonese, Mandarin, Wu, and so on, even though you'll likely consider these all to be zh when written.

• CLDR doesn't have language population data for sign languages. Sign languages end up with a speaking_population() and writing_population() of 0, and I suppose that is literally true, but there's no data from which we could provide a signing_population() method.

Dependencies

language_data has a dependency on the marisa-trie package so that it can load a compact, efficient data structure for looking up language names.

Installation

language_data is usually installed as a dependency of langcodes, and doesn't make much sense without it. You can pip install language_data anyway if you want.

To install the language_data package in editable mode, run poetry install in the package root. (This is the equivalent of pip install -e ., which will hopefully become compatible again soon via PEP 660.)

language_data's Issues

Missing dependency on setuptools

Hey!

    language_name = reduced.language_name(language, max_distance)
  File "/nix/store/slc2gq45p8rm5fl1vc7phwlfa8jyqql6-python3-3.10.8-env/lib/python3.10/site-packages/langcodes/__init__.py", line 942, in language_name
    return self._get_name('language', language, max_distance)
  File "/nix/store/slc2gq45p8rm5fl1vc7phwlfa8jyqql6-python3-3.10.8-env/lib/python3.10/site-packages/langcodes/__init__.py", line 853, in _get_name
    from language_data.names import code_to_names
  File "/nix/store/slc2gq45p8rm5fl1vc7phwlfa8jyqql6-python3-3.10.8-env/lib/python3.10/site-packages/language_data/names.py", line 4, in <module>
    from language_data.util import data_filename
  File "/nix/store/slc2gq45p8rm5fl1vc7phwlfa8jyqql6-python3-3.10.8-env/lib/python3.10/site-packages/language_data/util.py", line 5, in <module>
    from pkg_resources import resource_filename
ModuleNotFoundError: No module named 'pkg_resources'

setuptools should be declared as a dependency to get pkg_resources. I am not familiar enough with Poetry to know where this should be specified.

Tag the source

Could you please tag the source again (for 1.1)? Otherwise all distributions which were shipping this Python module and were using the GitHub releases need to change their description to build the packages.

Thanks

Add Python 3.12 support

This project depends on marisa-trie = "^0.7.7" (<0.8), but this package does not support Python 3.12 until v1.1.

Would it be possible to bump the dependency?

Maintanance

Hello @rspeer

I came to the conclusion this package is not maintained anymore because of the following reasons:

• The last commit is >3 years old
• There were 4 new releases of CLDR that are not reflected
• There is no python 12 compatibility
• Several open issues and PRs are not answered.

I am fully aware you did all the great work on your free time and you don't owe us anything! Its highly appreciated you took the effort and published this great library as open source. Thank you for this! 🙏

However, I and probably other people rely on the package and I'd be therefore interested in helping with the maintenance. With respect to your work in the past, I am therefore trying to reach you to find a solution for the lack of updates. Please let me know if you are interested in either sharing the responsibility for the package or transferring the package, let me know!

Just to be fully transparent, in case there is no reply I am considering to request a name transfer following PEP-0541.