jakartaresearch/maleo

Wrapper library for text cleansing, preprocessing in NLP

maleo's Issues

hashtag extraction

stemmer and lemmatizer

Customize Regex pattern (override the current one)

conver word to number

encode numbers like in information retrieval

jadi daripada di remove angkanya, mending di encode misal 123 > xxx
atau no tlpn 0123 4567 8901 >

remove stopword

Use Indonesian stopword dictionary, must prepare it first

remove punctuation

handling date

Feat/emoji_to_tag

INPUT = 😊
OUTPUT = >EMOJI<

emoticon to word doesn't add whitespace

when using emoji_to_word the converted emoji has no whitespace to separate each emoji.

input

"untuk besok komen disini ya.. 😊😊😊😊🙏🙏🙏 https://t.cowok/nxeojVug3z "

output

utk besok komen disini ya.. "smiling_face_with_smiling_eyessmiling_face_with_smiling_eyessmiling_face_with_smiling_eyessmiling_face_with_smiling_eyesfolded_handsfolded_handsfolded_hands https://t.co/nxeojVug3z"

expected output

"smiling_face_with_smiling_eyes smiling_face_with_smiling_eyes smiling_face_with_smiling_eyes smiling_face_with_smiling_eyes folded_hands folded_hands folded_hands https://t.co/nxeojVug3z"

So that, we can tokenize the emoji word

Price parser

Put static file to cloud

Add Pytest Coverage

https://pytest-cov.readthedocs.io/en/latest/

encode email link to 'EMAIL'

Unit Testing

requirements.txt need add library

add Beautiful Soup library

pip install beautifulsoup4

Update slang_dict.json

remove non-ascii character

spelling correction

Create documentation

Explore tools for create documentation

Option:

• Sphinx
• Doxygen
• Pdoc

Choosen --> Pdoc

Handle Emoji & Emoticon

remove single character

remove http link, pic.twitter.com link

convert slang word to formal word

remove duplicate text

remove consecutive repeating character

Update slang-formal dictionary

Currently, already have one slang-formal dictionary but need to check and update

Add docstring for public function

remove multiple whitespace

choose text cleaning steps

toggle on off for (remove number yes or no)

phrase detection

POS Tagging model
https://github.com/jakartaresearch/maleo/releases/tag/v0.0.6.2

remove HTML tag

Add Travis CI for checking unit test