kormalev/pytorch-lifestream

A library built upon PyTorch for building embeddings on discrete event sequences using self-supervision

PyTorch-LifeStream or PTLS a library built upon PyTorch for building embeddings on discrete event sequences using self-supervision. It can process terabyte-size volumes of raw events like game history events, clickstream data, purchase history or card transactions.

It supports various methods of self-supervised training, adapted for event sequences:

• Contrastive Learning for Event Sequences (CoLES)
• Contrastive Predictive Coding (CPC)
• Replaced Token Detection (RTD) from ELECTRA
• Next Sequence Prediction (NSP) from BERT
• Sequences Order Prediction (SOP) from ALBERT
• Masked Language Model (MLM) from ROBERTA

It supports several types of encoders, including Transformer and RNN. It also supports many types of self-supervised losses.

The following variants of the contrastive losses are supported:

• Contrastive loss (paper)
• Triplet loss (paper)
• Binomial deviance loss (paper)
• Histogramm loss (paper)
• Margin loss (paper)
• VICReg loss (paper)

Install from PyPi

pip install pytorch-lifestream

Install from source

# Ubuntu 20.04

sudo apt install python3.8 python3-venv
pip3 install pipenv

pipenv sync  --dev # install packages exactly as specified in Pipfile.lock
pipenv shell
pytest

Demo notebooks

• Supervised model training notebook
• Self-supervided training and embeddings for downstream task notebook
• Self-supervided embeddings in CatBoost notebook
• Self-supervided training and fine-tuning notebook
• Self-supervised TrxEncoder only training with Masked Language Model task and fine-tuning notebook
• Pandas data preprocessing options notebook
• PySpark and Parquet for data preprocessing notebook
• Fast inference on large dataset notebook
• Supervised multilabel classification notebook
• Text features demo:
  • Using pretrained encoder to text featuresnotebook

Docs

Documentation

Library description index

Experiments on public datasets

pytorch-lifestream usage experiments on several public event datasets are available in the separate repo

PyTorch-LifeStream in ML competitions

• Data Fusion Contest 2022 report (in Russian)
• Data Fusion Contest 2022 report, Sber AI Lab team (in Russian)
• VK.com Graph ML Hackaton report (in Russian)
• VK.com Graph ML Hackaton report, AlfaBank team (in Russian)
• American Express - Default Prediction Kaggle contest report (in Russian)

How to contribute

1. Make your chages via Fork and Pull request.
2. Write unit test for new code in ptls_tests.
3. Check unit test via pytest: Example.