goru001/nlp-for-malyalam

State of the Art Language models and Classifier for Malayalam, which is spoken by the Malayali people in the Indian state of Kerala and the union territories of Lakshadweep and Puducherry

NLP for Malayalam

This repository contains State of the Art Language models and Classifier for Malayalam, which is spoken by the Malayali people in the Indian state of Kerala and the union territories of Lakshadweep and Puducherry.

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

1. Malayalam Wikipedia Articles

2. Malayalam News Dataset

Open Source Datasets

1. iNLTK Headlines Corpus - Malayalam : Uses the Malayalam News Dataset prepared above

Results

Language Model Perplexity (on validation set)

Architecture/Dataset	Malayalam Wikipedia Articles
ULMFiT	26.39
TransformerXL	25.79

Classification Metrics

ULMFiT

Dataset	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Malayalam	95.56	93.29	Link

Visualizations

Word Embeddings

Architecture	Visualization
ULMFiT	Embeddings projection
TransformerXL	Embeddings projection

Results of using Transfer Learning + Data Augmentation from iNLTK

On using complete training set (with Transfer learning)

Dataset	Dataset size (train, valid, test)	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Malayalam	(5036, 630, 630)	95.56	93.29	Link

On using 10% of training set (with Transfer learning)

Dataset	Dataset size (train, valid, test)	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Malayalam	(503, 630, 630)	82.38	73.47	Link

On using 10% of training set (with Transfer learning + Data Augmentation)

Dataset	Dataset size (train, valid, test)	Accuracy	MCC	Notebook to Reproduce results
iNLTK Headlines Corpus - Malayalam	(503, 630, 630)	84.29	76.36	Link

Pretrained Models

Language Models

Download pretrained Language Model from here

Tokenizer

Trained tokenizer using Google's sentencepiece

Download the trained model and vocabulary from here

nlp-for-malyalam's Issues

Malayalam Common Crawl Dataset

Hi, I have worked on extracting common crawl data for Malayalam and has made it publicly available. Please check it out and see if it can be used to improve your results.

It has around 8 GB of Raw Malayalam text from the 2020-05 crawl.

https://github.com/qburst/common-crawl-malayalam

LFS data limit exceeded

@goru001 While cloning the repo, getting the following error message:

Downloading language-model/embeddings.tsv (45 MB)
Error downloading object: language-model/embeddings.tsv (0509f1b): Smudge error: Error downloading language-model/embeddings.tsv (0509f1b101ebcace773f9eccdd9be921002e751a4288804e64d49b410a8a1755): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to /media/athul/ee4ac28b-dca1-4dd5-b1ef-1e0be6e5e867/home/athul/nlp/nlp-for-malyalam/.git/lfs/logs/20200211T154509.772925999.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: language-model/embeddings.tsv: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'``
`