pythainlp/attacut

A Fast and Accurate Neural Thai Word Segmenter

AttaCut: Fast and Reasonably Accurate Word Tokenizer for Thai

How does AttaCut look like?

TL;DR: 3-Layer Dilated CNN on syllable and character features. It’s 6x faster than DeepCut (SOTA) while its WL-f1 on BEST is 91%, only 2% lower.

Installation

$ pip install attacut

Remarks: Windows users need to install PyTorch before the command above. Please consult PyTorch.org for more details.

Usage

Command-Line Interface

$ attacut-cli -h
AttaCut: Fast and Reasonably Accurate Word Tokenizer for Thai

Usage:
  attacut-cli <src> [--dest=<dest>] [--model=<model>]
  attacut-cli [-v | --version]
  attacut-cli [-h | --help]

Arguments:
  <src>             Path to input text file to be tokenized

Options:
  -h --help         Show this screen.
  --model=<model>   Model to be used [default: attacut-sc].
  --dest=<dest>     If not specified, it'll be <src>-tokenized-by-<model>.txt
  -v --version      Show version

High-Level API

from attacut import tokenize, Tokenizer

# tokenize `txt` using our best model `attacut-sc`
words = tokenize(txt)

# alternatively, an AttaCut tokenizer might be instantiated directly, allowing
# one to specify whether to use `attacut-sc` or `attacut-c`.
atta = Tokenizer(model="attacut-sc")
words = atta.tokenize(txt)

For better efficiency, we recommend using attacut-cli. Please consult our Google Colab tutorial for more detials.

Benchmark Results

Belows are brief summaries. More details can be found on our benchmarking page.

Tokenization Quality

Speed

Retraining on Custom Dataset

Please refer to our retraining page

Related Resources

• Tokenization Visualization
• Thai Tokenizer Dockers

Acknowledgements

This repository was initially done by Pattarawat Chormai, while interning at Dr. Attapol Thamrongrattanarit's NLP Lab, Chulalongkorn University, Bangkok, Thailand. Many people have involed in this project. Complete list of names can be found on Acknowledgement.

attacut's Issues

attacut-sc doesn't tokenise space properly

Example from https://colab.research.google.com/drive/11nMfWmPGR_82voL37okn4XlxMPVbsu9r#scrollTo=v5sGX_dlQ2_B

It seems that spaces aren't tokenised properly. Please see the issue below:

|Blognone |Tomorrow |2019 |ประกาศ|ชื่อ |speaker |เพิ่มเติม |1 |ท่าน|คือ |คุณธนาธร |จึงรุ่งเรืองกิจ |หัวหน้า|พรรคอนาคต|ใหม่ |จะ|มา|พูด|ใน|หัวข้อ |Hyperloop |and |Path |Skipping |Development 
|Strategy |หรือ|แปล|เป็น|ภาษา|ไทย|คือ |"|Hyperloop |กับ|การ|พัฒนา|แบบ|เสือ|กระโดด|"

custom_dict param is not working properly on attacut tokenizer

Env:

• Google Colab

Version :

• pythainlp (3.0.5)
• attacut (1.0.6)

as Docs pythainlp.tokenize.word_tokenize said Attacut tokenizer provided custom_dict param. However, it seem not working properly.

from pythainlp.corpus.common import thai_words
from pythainlp.util import dict_trie
from pythainlp.tokenize import Tokenizer

custom_words_list = set(thai_words())
custom_words_list |= set(['ต้องการระบาย'])
trie = dict_trie(dict_source=custom_words_list)

_tokenizer = Tokenizer(custom_dict=trie, engine='attacut')

_tokenizer.word_tokenize('ต้องการ')

output

['ต้องการ', 'ระบาย']

expected output

['ต้องการระบาย']

PS1. I also tested with pythainlp.tokenize.word_tokenize. it's worked similar to above.
PS2. newmm and longest engine still work with custom_dict param.

Add custom dictionary

Is there a way to add custom dictionary? like deepcut?

the accuracy between "AttaCut" and "newmm".

Why the accuracy of "AttaCut" model more worse than "newmm" throgh PyThaiNLP in TNHC(Thai National Historical Corpus) dataset?
Is there any problem that needs my attention？

And, do you what's the meaning of "、#、$" in TNHC dataset？

Convert models to torchscript format

Torch provides a way to save models in Torchscript. Using this format, several computations can be further optimised, hence faster models.

More information: https://pytorch.org/docs/stable/jit.html#frequently-asked-questions

We might also to this for datasets (collated_func)

beta release

Tasks

• make it pipable
• use logger instead of print
• add more test
• add linting
• refactor training.py
• [ ] batch processing for attacut-cli (moved to #5)

Documentations (now available on https://pythainlp.github.io/attacut)

• how to train a new model
• module overview in this repo

Testing & Final Benchmark

• install via pip
• run tokenization to check whether the implement is correctly migrated.
• testing speed
• update visualisation results

เพิ่ม option `attacut-sc` และ `attacut-c` ใน engine เมื่อเรียก from pythainlp.tokenize import word_tokenize

Tokenizer(model = "attacut-sc").tokenize("วัดพระแก้วกรุงเทพ") และ word_tokenize("วัดพระแก้วกรุงเทพ", engine="attacut") ถึงได้ผลลัพธ์เป็น ["วัดพระแก้วกรุงเทพ"]

ในขณะที่ Tokenizer(model = "attacut-c").tokenize("วัดพระแก้วกรุงเทพ") ได้ผลลัพธ์เป็น ["วัดพระแก้ว", "กรุงเทพ"]

หากเพิ่มเติมรายละเอียดความแตกต่างระหว่าง attacut-sc และ attacut-c ในหน้าหลักของ https://thainlp.org/pythainlp/docs/2.0/api/tokenize.html จะดีมาก

Batch processing for CLI

It would be faster if we can do multiple lines per inference, aka batching lines.

Add a factory function

This is similar to https://github.com/rkcosmos/deepcut/blob/master/deepcut/deepcut.py#L23

This function will be a proxy function for other modules, such as PyThaiNLP, to consume.

It should receive text and instantiate a AttaCut tokenizer when needed.

UnicodeDecodeError Problem

using tokenize function caused UnicodeDecodeError from load_dict function in utils.py

Provide a way to make part of text not to be tokenized

I've been using attacut to process Thai chat corpus. The problem is many texts contain hyperlink, emoji, email addr, etc. I detect these entities and try to use placeholders to replace them before sending them to attacut hoping that attacut will leave them untoudhed. The placeholders I use are long english word string like "EMOJIPLACEHOLDER". It works for other tokenizers like deepcut, but attacut occasionly cut the placeholder into pieces: "EMO JIPLACEHOLDER".

So is there any way that I can tell attacut not to cut a string in a line of text?
Or any way to work around this problem?

Thanks.

Retraining on floydhub.com is not available

Can you provide corpus? We cannot retraining this model on floydhub.com in readme. Thanks a lot.

Can't install on Windows

I used pip install https://github.com/PyThaiNLP/attacut/archive/master.zip on Windows but it has a installation problems. torch can't install on Windows.

>pip install https://github.com/PyThaiNLP/attacut/archive/master.zip
Collecting https://github.com/PyThaiNLP/attacut/archive/master.zip
  Downloading https://github.com/PyThaiNLP/attacut/archive/master.zip
     \ 2.4MB 1.1MB/s
Requirement already satisfied: docopt==0.6.2 in c:\users\tc\anaconda3\lib\site-packages (from attacut==0.0.3.dev0) (0.6.2)
Collecting fire==0.1.3 (from attacut==0.0.3.dev0)
  Downloading https://files.pythonhosted.org/packages/5a/b7/205702f348aab198baecd1d8344a90748cb68f53bdcd1cc30cbc08e47d3e/fire-0.1.3.tar.gz
Collecting nptyping==0.2.0 (from attacut==0.0.3.dev0)
  Downloading https://files.pythonhosted.org/packages/a5/0f/9b44a1866c7911d03329669d82d2ebb1b8e6dac15803fdb6588549a44193/nptyping-0.2.0-py3-none-any.whl
Collecting numpy==1.17.0 (from attacut==0.0.3.dev0)
  Downloading https://files.pythonhosted.org/packages/26/26/73ba03b2206371cdef62afebb877e9ba90a1f0dc3d9de22680a3970f5a50/numpy-1.17.0-cp37-cp37m-win_amd64.whl (12.8MB)
     |████████████████████████████████| 12.8MB 3.3MB/s
Requirement already satisfied: python-crfsuite==0.9.6 in c:\users\tc\anaconda3\lib\site-packages (from attacut==0.0.3.dev0) (0.9.6)
Collecting pyyaml==5.1.2 (from attacut==0.0.3.dev0)
  Downloading https://files.pythonhosted.org/packages/bc/3f/4f733cd0b1b675f34beb290d465a65e0f06b492c00b111d1b75125062de1/PyYAML-5.1.2-cp37-cp37m-win_amd64.whl (215kB)
     |████████████████████████████████| 225kB 3.2MB/s
Requirement already satisfied: six==1.12.0 in c:\users\tc\anaconda3\lib\site-packages (from attacut==0.0.3.dev0) (1.12.0)
Collecting ssg==0.0.4 (from attacut==0.0.3.dev0)
  Downloading https://files.pythonhosted.org/packages/05/e0/226b4fb9144d80a3efc474e581097d77abc4e8c3ce8e751469cb1c25e671/ssg-0.0.4-py3-none-any.whl (473kB)
     |████████████████████████████████| 481kB 2.2MB/s
Collecting torch==1.2.0 (from attacut==0.0.3.dev0)
  ERROR: Could not find a version that satisfies the requirement torch==1.2.0 (from attacut==0.0.3.dev0) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.2.0 (from attacut==0.0.3.dev0)

Uplift version pin of nptying

I'm wondering that if there is any reason that the nptying dependency is pinned at <=0.3.1 while the latest version of nptying is 1.4.0?

What is the format of the input data？

I want to continue training attacut on my own data set, but I am not sure what the format of the data set should be?

The data set link here is no longer valid, so I cannot view the format of the data set, can you help me?

the meaning of label data

@cnlinxi sorry again for my response. You can find the data at https://codeforthailand.s3-ap-southeast-1.amazonaws.com/attacut-related/data.zip

Please unzip and make sure the root directory is at ./data. The content of the archive contains

Only the first two are relevant for training; sampling-0 means all the dateset, while sampling-10 means only 10 files are used. You can use sampling-10 for quick training.

Before running the training command below, make sure that you have the ./artifacts directory.

python ./scripts/train.py --model-name seq_sy_ch_conv_concat \
 --model-params "embc:8|embs:8|conv:8|l1:6|do:0.1" \
 --data-dir ./data/best-syllable-crf-and-character-seq-feature-sampling-0  \
 --output-dir ./artifacts/model-xx  \
 --epoch 2 \
 --batch-size 1024 \
 --lr 0.001 \
 --lr-schedule "step:5|gamma:0.5"

Originally posted by @heytitle in #20 (comment)

Datasets cannot be accessed

The datasets on https://pythainlp.github.io/attacut/training.html could not be accessed.
In additional, without taking a look at the datasets, the example in https://pythainlp.github.io/attacut/training.html#how-to-retrain-on-custom-dataset is not clear enough to follow (in my opinion).