hiredscorelabs/tamnun-ml

An easy to use open-source library for advanced Deep Learning and Natural Language Processing

Tamnun ML

tamnun is a python framework for Machine and Deep learning algorithms and methods especially in the field of Natural Language Processing and Transfer Learning. The aim of tamnun is to provide an easy to use interfaces to build powerful models based on most recent SOTA methods.

For more about tamnun, feel free to read the introduction to TamnunML on Medium.

Getting Started

tamnun depends on several other machine learning and deep learning frameworks like pytorch, keras and others. To install tamnun and all it's dependencies run:

$ git clone https://github.com/hiredscorelabs/tamnun-ml
$ cd tamnun-ml
$ python setup.py install

Or using PyPI:

pip install tamnun

Jump in and try out an example:

$ cd examples
$ python finetune_bert.py

Or take a look at the Jupyer notebooks here.

BERT

BERT stands for Bidirectional Encoder Representations from Transformers which is a language model trained by Google and introduced in their paper. Here we use the excellent PyTorch-Pretrained-BERT library and wrap it to provide an easy to use scikit-learn interface for easy BERT fine-tuning. At the moment, tamnun BERT classifier supports binary and multi-class classification. To fine-tune BERT on a specific task:

from tamnun.bert import BertClassifier, BertVectorizer
from sklearn.pipeline import make_pipeline

clf = make_pipeline(BertVectorizer(), BertClassifier(num_of_classes=2)).fit(train_X, train_y)
predicted = clf.predict(test_X)

Please see this notebook for full code example.

Fitting (almost) any PyTorch Module using just one line

You can use the TorchEstimator object to fit any pytorch module with just one line:

from torch import nn
from tamnun.core import TorchEstimator

module = nn.Linear(128, 2)
clf = TorchEstimator(module, task_type='classification').fit(train_X, train_y)

See this file for a full example of fitting nn.Linear module on the MNIST (classification of handwritten digits) dataset.

Distiller Transfer Learning

This module distills a very big (like BERT) model into a much smaller model. Inspired by this paper.

from tamnun.bert import BertClassifier, BertVectorizer
from tamnun.transfer import Distiller

bert_clf =  make_pipeline(BertVectorizer(do_truncate=True, max_len=3), BertClassifier(num_of_classes=2))
distilled_clf = make_pipeline(CountVectorizer(ngram_range=(1,3)), LinearRegression())

distiller = Distiller(teacher_model=bert_clf, teacher_predict_func=bert_clf.decision_function, student_model=distilled_clf).fit(train_texts, train_y, unlabeled_X=unlabeled_texts)

predicted_logits = distiller.transform(test_texts)

For full BERT distillation example see this notebook.

Support

Getting Help

You can ask questions and join the development discussion on Github Issues

License

Apache License 2.0 (Same as Tensorflow)

tamnun-ml's Issues

Tamnun-ml for multilabel classifier

Hi,
I looked over the notebook for one label classifier, can you explain how to use this method for multi label classifier? or save the probabilities for each label?
Thank you!

control hyperparameters and number of epochs

Hi,
Thank you for this tool! How I can control on the number of the epochs or change the other hyperpararmeters?
Thank you!

Is it true that it may not work as an estimator for StackingClassifier from Sklearn?

I included it as a base estimator for 'sklearn.ensemble.StackingClassifier' but it gives an error saying "The estimator Pipeline should be a classifier"

How do I get it to run on more than one GPU?

Saving and Reloading

What would be the best way to save and reload the model?

pytorch version for tamnun

Hi,
Can I use tamnun lib. by using other versions of pytorch, and not only torch 1.1.0?
Best.

training on gpu, predicting on cpu

Hi,
I tried predicting with the trained (pretrained+fine tuned) model using a server on a docker container, but had a problem.
looks like the model can only predict in a GPU environment (and indeed I managed to upload a local server on my machine).
I tried both loading the model with pickle and loading the model using torch.load with map_device = 'cpu', but still it had an error whenever i tried to predict.

Training Bert on Hebrew dataset

Is it possible to train BERT on Hebrew dataset? (For example Wiki dump)

Support for other pretrained BERT models

Thanks for the great module, it seems to work right out of the box!

Is there a way to specify a different pretrained model other than those listed in here?

It would be great to use bioBERT or sciBERT, the latter of which has a better vocabulary. A quick look suggest that you're using pytorch_transformers.BertTokenizer which seems to have methods for custom calls.