pythainlp/nlpo3

Thai Natural Language Processing library in Rust, with Python and Node bindings.

PyThaiNLP: Thai Natural Language Processing in Python

PyThaiNLP is a Python package for text processing and linguistic analysis, similar to NLTK with a focus on Thai language.

PyThaiNLP เป็นไลบารีภาษาไพทอนสำหรับประมวลผลภาษาธรรมชาติ คล้ายกับ NLTK โดยเน้นภาษาไทย ดูรายละเอียดภาษาไทยได้ที่ README_TH.MD

News

Now, You can contact with or ask any questions of the PyThaiNLP team.

Version	Description	Status
5.0.4	Stable	Change Log
dev	Release Candidate for 5.1	Change Log

Getting Started

• PyThaiNLP 2 requires Python 3.7+. Python 2.7 users can use PyThaiNLP 1.6. See 2.0 change log | Upgrading from 1.7 | Upgrading ThaiNER from 1.7
• PyThaiNLP Get Started notebook | API document | Tutorials
• Official website | PyPI | Facebook page
• Who uses PyThaiNLP?
• Model cards - for technical details, caveats, and ethical considerations of the models developed and used in PyThaiNLP

Capabilities

PyThaiNLP provides standard linguistic analysis for Thai language and standard Thai locale utility functions. Some of these functions are also available via the command-line interface (run thainlp in your shell).

Partial list of features:

• Convenient character and word classes, like Thai consonants (pythainlp.thai_consonants), vowels (pythainlp.thai_vowels), digits (pythainlp.thai_digits), and stop words (pythainlp.corpus.thai_stopwords) -- comparable to constants like string.letters, string.digits, and string.punctuation
• Linguistic unit segmentation at different levels: sentence (sent_tokenize), word (word_tokenize), and subword (subword_tokenize)
• Part-of-speech tagging (pos_tag)
• Spelling suggestion and correction (spell and correct)
• Phonetic algorithm and transliteration (soundex and transliterate)
• Collation (sorted by dictionary order) (collate)
• Number read out (num_to_thaiword and bahttext)
• Datetime formatting (thai_strftime)
• Thai-English keyboard misswitched fix (eng_to_thai, thai_to_eng)

Installation

pip install --upgrade pythainlp

This will install the latest stable release of PyThaiNLP.

Install different releases:

• Stable release: pip install --upgrade pythainlp
• Pre-release (nearly ready): pip install --upgrade --pre pythainlp
• Development (likely to break things): pip install https://github.com/PyThaiNLP/pythainlp/archive/dev.zip

Installation Options

Some functionalities, like Thai WordNet, may require extra packages. To install those requirements, specify a set of [name] immediately after pythainlp:

pip install pythainlp[extra1,extra2,...]

Possible extras:

• full (install everything)
• attacut (to support attacut, a fast and accurate tokenizer)
• benchmarks (for word tokenization benchmarking)
• icu (for ICU, International Components for Unicode, support in transliteration and tokenization)
• ipa (for IPA, International Phonetic Alphabet, support in transliteration)
• ml (to support ULMFiT models for classification)
• thai2fit (for Thai word vector)
• thai2rom (for machine-learnt romanization)
• wordnet (for Thai WordNet API)

For dependency details, look at the extras variable in setup.py.

Data Directory

• Some additional data, like word lists and language models, may be automatically downloaded during runtime.
• PyThaiNLP caches these data under the directory ~/pythainlp-data by default.
• The data directory can be changed by specifying the environment variable PYTHAINLP_DATA_DIR.
• See the data catalog (db.json) at https://github.com/PyThaiNLP/pythainlp-corpus

Command-Line Interface

Some of PyThaiNLP functionalities can be used via command line with the thainlp command.

For example, to display a catalog of datasets:

thainlp data catalog

To show how to use:

thainlp help

Licenses

	License
PyThaiNLP source codes and notebooks	Apache Software License 2.0
Corpora, datasets, and documentations created by PyThaiNLP	Creative Commons Zero 1.0 Universal Public Domain Dedication License (CC0)
Language models created by PyThaiNLP	Creative Commons Attribution 4.0 International Public License (CC-by)
Other corpora and models that may be included in PyThaiNLP	See Corpus License

Contribute to PyThaiNLP

• Please fork and create a pull request :)
• For style guides and other information, including references to algorithms we use, please refer to our contributing page.

Who uses PyThaiNLP?

You can read INTHEWILD.md.

Citations

If you use PyThaiNLP in your project or publication, please cite the library as follows:

Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, & Pattarawat Chormai. (2016, Jun 27). PyThaiNLP: Thai Natural Language Processing in Python. Zenodo. http://doi.org/10.5281/zenodo.3519354

or by BibTeX entry:

@misc{pythainlp,
    title = "{P}y{T}hai{NLP}: {T}hai Natural Language Processing in {P}ython",
    author = "Phatthiyaphaibun, Wannaphong  and
      Chaovavanich, Korakot  and
      Polpanumas, Charin  and
      Suriyawongkul, Arthit  and
      Lowphansirikul, Lalita  and
      Chormai, Pattarawat",
    month = jun,
    year = "2016",
    doi = {10.5281/zenodo.3519354},
    publisher = {Zenodo},
    url = {http://doi.org/10.5281/zenodo.3519354}
}

Our NLP-OSS 2023 paper:

Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, Pattarawat Chormai, Peerat Limkonchotiwat, Thanathip Suntorntip, and Can Udomcharoenchaikit. 2023. PyThaiNLP: Thai Natural Language Processing in Python. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 25–36, Singapore, Singapore. Empirical Methods in Natural Language Processing.

and its BibTeX entry:

@inproceedings{phatthiyaphaibun-etal-2023-pythainlp,
    title = "{P}y{T}hai{NLP}: {T}hai Natural Language Processing in {P}ython",
    author = "Phatthiyaphaibun, Wannaphong  and
      Chaovavanich, Korakot  and
      Polpanumas, Charin  and
      Suriyawongkul, Arthit  and
      Lowphansirikul, Lalita  and
      Chormai, Pattarawat  and
      Limkonchotiwat, Peerat  and
      Suntorntip, Thanathip  and
      Udomcharoenchaikit, Can",
    editor = "Tan, Liling  and
      Milajevs, Dmitrijs  and
      Chauhan, Geeticka  and
      Gwinnup, Jeremy  and
      Rippeth, Elijah",
    booktitle = "Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)",
    month = dec,
    year = "2023",
    address = "Singapore, Singapore",
    publisher = "Empirical Methods in Natural Language Processing",
    url = "https://aclanthology.org/2023.nlposs-1.4",
    pages = "25--36",
    abstract = "We present PyThaiNLP, a free and open-source natural language processing (NLP) library for Thai language implemented in Python. It provides a wide range of software, models, and datasets for Thai language. We first provide a brief historical context of tools for Thai language prior to the development of PyThaiNLP. We then outline the functionalities it provided as well as datasets and pre-trained language models. We later summarize its development milestones and discuss our experience during its development. We conclude by demonstrating how industrial and research communities utilize PyThaiNLP in their work. The library is freely available at https://github.com/pythainlp/pythainlp.",
}

Sponsors

Logo	Description
	Since 2019, our contributors Korakot Chaovavanich and Lalita Lowphansirikul have been supported by VISTEC-depa Thailand Artificial Intelligence Research Institute.
	We get support of free Mac Mini M1 from MacStadium for running CI builds.

---

Made with ❤️ | PyThaiNLP Team 💻 | "We build Thai NLP" 🇹🇭

---

We have only one official repository at https://github.com/PyThaiNLP/pythainlp and another mirror at https://gitlab.com/pythainlp/pythainlp

Beware of malware if you use codes from mirrors other than the official two on GitHub and GitLab.

nlpo3's Issues

Clarify method in Tokenizer trait

From Tokenizer trait declaration:

pub trait Tokenizer {
    fn segment(&self, text: &str, safe: Option<bool>, parallel: Option<bool>) -> Vec<String>;

    fn segment_to_string(
        &self,
        text: &str,
        safe: Option<bool>,
        parallel: Option<bool>,
    ) -> Vec<String>;
}

It provides 2 methods that look do the same thing. And in Newmm, it do the same things for both methods (Ref here). I purpose to clarify which one we should use and clean up trait protocol.

No Cargo.lock in the project

Currently, we haven't Cargo.lock in the project to lock dependency version. Do we need to add it?

Can't install on apple silicon (arm64) platform

Current python is 3.8.13 using conda (as apple silicon can't run python 3.7 (x86_64 only))

Running

$> pip install "nlpo3>=1.2.2"

caused the following error

Collecting nlpo3>=1.2.2
  Using cached nlpo3-1.2.2.tar.gz (9.6 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [87 lines of output]
      configuration error: `project.license` must be valid exactly by one definition (2 matches found):
      
          - keys:
              'file': {type: string}
            required: ['file']
          - keys:
              'text': {type: string}
            required: ['text']
      
      DESCRIPTION:
          `Project license <https://www.python.org/dev/peps/pep-0621/#license>`_.
      
      GIVEN VALUE:
          "Apache-2.0"
      
      OFFENDING RULE: 'oneOf'
      
      DEFINITION:
          {
              "oneOf": [
                  {
                      "properties": {
                          "file": {
                              "type": "string",
                              "$$description": [
                                  "Relative path to the file (UTF-8) which contains the license for the",
                                  "project."
                              ]
                          }
                      },
                      "required": [
                          "file"
                      ]
                  },
                  {
                      "properties": {
                          "text": {
                              "type": "string",
                              "$$description": [
                                  "The license of the project whose meaning is that of the",
                                  "`License field from the core metadata",
                                  "<https://packaging.python.org/specifications/core-metadata/#license>`_."
                              ]
                          }
                      },
                      "required": [
                          "text"
                      ]
                  }
              ]
          }
      Traceback (most recent call last):
        File "/Users/user/miniconda3/envs/python38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/user/miniconda3/envs/python38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/user/miniconda3/envs/python38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 4, in <module>
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 159, in setup
          dist.parse_config_files()
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py", line 62, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py", line 126, in read_configuration
          validate(subset, filepath)
        File "/private/var/folders/9n/7pyff06d2x78r2sktbrb18680000gn/T/pip-build-env-jslccyui/overlay/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py", line 51, in validate
          raise ValueError(f"{error}\n{summary}") from None
      ValueError: invalid pyproject.toml config: `project.license`.
      configuration error: `project.license` must be valid exactly by one definition (2 matches found):
      
          - keys:
              'file': {type: string}
            required: ['file']
          - keys:
              'text': {type: string}
            required: ['text']
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

This package is used in pythainlp>=3.0, so it caused some problem down the line

Update regex rule and dict

After #557 & #562, PyThaiNLP have update regex rule and dict. oxidized-thainlp needs to update regex rule and dict.

Using FST to speed up tokenziation?

See the use from Charabia

• https://github.com/meilisearch/charabia/blob/main/src/segmenter/thai.rs
• https://github.com/meilisearch/charabia/blob/main/src/segmenter/utils.rs

They use the longest matching approach.
And I'm sure their segmenter doesn't check the valid boundaries, like Thai Character Cluster.

There can be a room for improvement.

range end index 4 out of range, for a dictionary file with blank line

A dictionary file with a blank line will result an error:

thread 'main' panicked at 'range end index 4 out of range for slice of length 0', /Users/arthit/.cargo/registry/src/github.com-1ecc6299db9ec823/nlpo3-1.1.2/src/fixed_bytes_str/four_bytes.rs:233:30

Remove dictionary from default install

Current dictionary is quite big and probably not every users are going to use that.

Also, PyThaiNLP may consider to include nlpO3 Python-binding as one of the word tokenzation options. That means two exact copies of a same dictionary will be installed.

Dictionary can be installed if the user explicitly put it in a feature flag, maybe something like:

nlpo3 = {version = "1.3.0", features = ["dict-pythainlp", "dict-libthai"]}

More dictionary options can be offered in this way as well.

If user just install it without the feature flag:

nlpo3 = {version = "1.3.0"}

the user have to supply a dictionary of their own.

How to manage breaking changes from main lib for binding?

Now, each binding uses a published crate of main lib, but what would happen if the main lib signature is changed like in #31?

Should the changes in main lib be merged into some "dev" branch, then publish new version of the main crate from approved "dev" branch and then update main crate dependency of each binding, fix binding and then make a pull request to main?

[TODO] build and publish on npm

build and publish nlpO3-Node.js binding to npm

[Python] load_dict() can't deal with blank line

If a dictionary file has a blank file, will get this error:

thread '<unnamed>' panicked at 'range end index 4 out of range for slice of length 0',
/nlpo3-1.3.1/src/fixed_bytes_str/four_bytes.rs:296:30

Update TCC+

After PyThaiNLP's TCC has major update, TCC in PyThaiNLP has 2 version PyThaiNLP/pythainlp#741. nlpO3 needs to update the TCC rule to improve newmm tokenizer but the new TCC in PyThaiNLP still isn't the final version because some rule maybe missing and wait to found the bug from reporting.

Consider remove optional out of safe and parallel flag in Tokenizer trait

From this line, I think we can remove Option from safe and parallel to be just a boolean. The code should looks simpler and the binding just call unwrap_or or unwrap_or_default to convert option type to a boolean.