mohbadar/pashto-stopwords

Pashto Stopwords Corpus and Text Cleansing

Pashto Stopwords Corpus and Text Cleansing

By emerging of the World Wide Web (WWW), huge amount of data is generated in a variety of subjects. In fact, that makes it very challenging for Natural Language Processing (NLP) systems to process all these contents. So, it is important to do some pre-processing in order to distinguish between more and less informative data. During pre-processing, the words, which only have grammatical roles and do not contribute to information content should be identified and eliminated. Such words are called stop words. Stop words exist, almost, in all languages. For other languages, dictionaries of stopwords are available. However, as far as I know there is no such corpus (list of stopwords) for the Pashto Language. Finding that gap, we have developed a collection of the Pashto stopwords to facilitate the Pashto text processing. Also, we feel that this practical research will encourage other researchers to start working on our national languages to adopt our languages with new technologies. And it would be a foundation work for Pashto Natural Language processing task.

We also have proposed a new Hybrid approach to identify and eliminate Stopwords from Pashto Text. And it is going to be published very soon.

pashto-stopwords's Issues

Pashto text preprocessing

This issue includes

• text cleaning
• text normalization
• canonical format conversion
• special characters control