pythainlp/mudyom

MudYom (มัดย้อม)

MudYom is a module for pre/post-processing text. It combines, aka มัด, words that should be together into one token. This process is done according to a user-defined dictionary.

---

Installation

Because it's still in beta, installation has to be done via

$ pip install git+https://github.com/pythainlp/mudyom.git

Usage (WIP)

$ mudyom-cli --input "..." --dictionary "..." --output "..."

Remark: Vocabs in the dictionary should be sorted from longest to shortest one.

If not, you can use the command line below to sort the dictionary:

$ cat dictionary.txt | awk '{ print length, $0 }' | sort -g -r | cut -d" " -f2  > sorted_dictionary.txt

Example

# input.txt
ฉัน|ขวัญ|หนี|ตี|ฝ่อ|ใจ|สลาย

# dictionary.txt
หลบลี้
คิดถึง
ตีฝ่อ

# output.txt
ฉัน|ขวัญ|หนี|ตีฝ่อ|ใจ|สลาย

Public Dictionaries

Name	Vocaburary Size	Author
Food and Restuarant menues	~400k	Wongnai
Names and Acronyms	~2k	Thachaparn Bunditlurdruk
Name Entity in BEST	..	..

Acknowledgements

• The implementation of this module is majorily drawn from Wongnai's post, written by Ekkalak Thongthanomkul.

mudyom's Issues

What is a license? #1

I not found a license on MudYom. Can you add license?

Problems with your sort example when used with mixed language

I tried your example for sorting from largest to smallest:
cat dictionary.txt | awk '{ print length, $0 }' | sort -g -r | cut -d" " -f2 > sorted_dictionary.txt

with the example text:
A
A รหัส A
AB
ABC

and the result is:
A
ABC
AB
A

with "รหัส A" deleted.

A possible kludge that I came up is:
cat dictionary.txt | awk '{ print length, $0 }' | sort -g -r | sed 's/^ *//' | cut -d" " -f2- > sorted_dictionary.txt