siddsax/xml-cnn

Pytorch implementation of the paper Deep learning for extreme multi-label text classification

This repo is from my past life. I am now co-founder and CTO @ Merlin. Check out our blog and extension

XML-CNN

Pytorch implementation of the paper Deep Learning for Extreme Multi-label Text Classification with dynamic pooling

Dependencies

• NLTK (stopwords)
• Pytorch >= 0.3.1
• Gensim
• Matplotlib

Directory Structure:

+-- code
|   +-- cnn_test.py  
|   +-- cnn_train.py
|   +-- header.py
|   +-- main.py
|   +-- models
|   |   +-- classifier.py
|   |   +-- cnn_encoder.py
|   |   +-- embedding_layer.py
|   |   +-- header.py
|   |   +-- xmlCNN.py
|   +-- precision_k.py
|   +-- score_matrix.mat
|   +-- test_manik.m
+-- datasets
|   +-- Folder where datasets need to be kept
+-- embedding_weights
+-- saved_models
|   +-- Directory created by default where models are saved
+-- utils
|   +-- data_dive.py
|   +-- data_helpers.py
|   +-- fiddle_clusters.py
|   +-- futils.py
|   +-- loss.py
|   +-- process_eurlex.py
|   +-- w2v.py
+-- README.md

Glove embeddings are needed by default as pre-training for the model. They can be download from here and placed in embedding_weights directory. The Default embedding dimension is 300 with 6 Billion (6B) tokens. Otherwise you can set --model_variation = 0 for starting from scratch.

Sample dataset RCV can be downloaded from here. Trained model on RCV1 can be downloaded from below.

Trained model

Precision Scores

1	2	3	4	5
0.96	0.8848	0.7809	0.6436	0.5457

Note: The scores are slighltly higher as the tested dataset is part of the full test dataset.

Procedure to train and test the model is as follows. The code by default doesn't plot graphs. Though that can be enabled wth a visdom server running [Visdom] and turing on flag --d

python main.py # train a model
python main.py --mn=rcv # train a model and save in directory rcv [inside saved_models]

This will create multiple files inside the folder saved_models/rcv in the above case. Checkpoints are saved after every save_step epochs, this can be changed with --ss option in command line. Also a checkpoint is made according to best test precision@1 score and best training batch precision@1.

python main.py --lm=$DESTINATION OF SAVED MODEL # This resumes trainign from the given checkpoint

In order to test the model run

python main.py --lm=$DESTINATION OF SAVED MODEL --tr=0 

This will first print the training error and then the test error while also saving a score_matrix.mat in the folder the model is loaded that can be used to run test scripts from here

xml-cnn's Issues

How to solve the problem that topK's K is different for every input text?

The output topK's K is fixed now.

Do you think training a classifier to predict the value of K for every input is a good solution?

Thank you very much.

Regarding file not found "x_train.npz"

Saving Model to: Gen_data_CNN_Z_dim-100_mb_size-20_hidden_dims-512_preproc-0_loss-BCELoss_sequence_length-500_embedding_dim-300_params.vocab_size=30000
Traceback (most recent call last):
File "main.py", line 57, in
x_tr, x_te, y_tr, y_te, params.vocabulary, params.vocabulary_inv, params = save_load_data(params, save=params.load_data)
File "../utils\futils.py", line 153, in save_load_data
x_tr = sparse.load_npz(params.data_path + '/x_train.npz')
File "C:\Users\dc\anaconda\envs\riya\lib\site-packages\scipy\sparse_matrix_io.py", line 131, in load_npz
with np.load(file, **PICKLE_KWARGS) as loaded:
File "C:\Users\dc\anaconda\envs\riya\lib\site-packages\numpy\lib\npyio.py", line 415, in load
fid = open(os_fspath(file), "rb")
FileNotFoundError: [Errno 2] No such file or directory: '../datasets/rcv/x_train.npz'

Pooling Method

Hi siddsax,

I wanna ask about the pooling layer because you're using a sliding max pooling. XML-CNN use dynamic max pooling with the definition "For a p document with m words, we evenly divide its m-dimensional feature map into p chunks, each chunk is pooled to a single feature by taking the largest value within that chunk".

What would be the format of the input dataset?

Hi there,

I am interested in trying XML-CNN on my own dataset. I have collection of documents, and their labels. Could you please help me understand how I can feed it to your tool? Or, if you provide me samples, that would also be helpful. I tried to go through the RCV file you mentioned in the README file, but it's not really clear. Thanks.

about data preprocessing

Hello, I want to run XML-CNN on several benchmark, but i don't know how to deal with data, can you provide the script you use for data preprocessing in WIKI31K , AMAZON and others, and the downloading source of dataset?
Thanks~

The link of RCV1 dataset is invalid

Hi, when I got into the link of the RCV dataset, I found "404 not found", could you provide another link of the RCV dataset? If possible could you provide other datasets in your paper. It's a little hard for me to understand the code without the dataset. Thank you very much!