CCCapsNet

A PyTorch implementation of Compositional coding Capsule Network based on the paper Compositional coding capsule network with k-means routing for text classification.

Requirements

Anaconda
PyTorch

conda install pytorch torchvision -c pytorch

PyTorchNet

pip install git+https://github.com/pytorch/tnt.git@master

PyTorch-NLP

pip install pytorch-nlp

capsule-layer

pip install git+https://github.com/leftthomas/CapsuleLayer.git@master

Datasets

The original AGNews, AmazonReview, DBPedia, YahooAnswers, SogouNews and YelpReview datasets are coming from here.

The original Newsgroups, Reuters, Cade and WebKB datasets can be found here.

The original IMDB dataset is downloaded by PyTorch-NLP automatically.

We have uploaded all the original datasets into BaiduYun(access code:kddr) and GoogleDrive. The preprocessed datasets have been uploaded to BaiduYun(access code:2kyd) and GoogleDrive.

You needn't download the datasets by yourself, the code will download them automatically. If you encounter network issues, you can download all the datasets from the aforementioned cloud storage webs, and extract them into data directory.

Usage

Generate Preprocessed Data

python utils.py --data_type yelp --fine_grained
optional arguments:
--data_type              dataset type [default value is 'imdb'](choices:['imdb', 'newsgroups', 'reuters', 'webkb', 
                         'cade', 'dbpedia', 'agnews', 'yahoo', 'sogou', 'yelp', 'amazon'])
--fine_grained           use fine grained class or not, it only works for reuters, yelp and amazon [default value is False]

This step is not required, and it takes a long time to execute. So I have generated the preprocessed data before, and uploaded them to the aforementioned cloud storage webs. You could skip this step, and just do the next step, the code will download the data automatically.

Train Text Classification

visdom -logging_level WARNING & python main.py --data_type newsgroups --num_epochs 70
optional arguments:
--data_type              dataset type [default value is 'imdb'](choices:['imdb', 'newsgroups', 'reuters', 'webkb', 
                         'cade', 'dbpedia', 'agnews', 'yahoo', 'sogou', 'yelp', 'amazon'])
--fine_grained           use fine grained class or not, it only works for reuters, yelp and amazon [default value is False]
--text_length            the number of words about the text to load [default value is 5000]
--routing_type           routing type, it only works for capsule classifier [default value is 'k_means'](choices:['k_means', 'dynamic'])
--loss_type              loss type [default value is 'cross'](choices:['margin', 'focal', 'cross', 'mf', 'mc', 'fc', 'mfc'])
--embedding_type         embedding type [default value is 'cwc'](choices:['cwc', 'cc', 'normal'])
--classifier_type        classifier type [default value is 'capsule'](choices:['capsule', 'linear'])
--embedding_size         embedding size [default value is 64]
--num_codebook           codebook number, it only works for cwc and cc embedding [default value is 8]
--num_codeword           codeword number, it only works for cwc and cc embedding [default value is None]
--hidden_size            hidden size [default value is 128]
--in_length              in capsule length, it only works for capsule classifier [default value is 8]
--out_length             out capsule length, it only works for capsule classifier [default value is 16]
--num_iterations         routing iterations number, it only works for capsule classifier [default value is 3]
--num_repeat             gumbel softmax repeat number, it only works for cc embedding [default value is 10]
--drop_out               drop_out rate of GRU layer [default value is 0.5]
--batch_size             train batch size [default value is 30]
--num_epochs             train epochs number [default value is 30]
--num_steps              test steps number [default value is 100]

Visdom now can be accessed by going to 127.0.0.1:8097/env/$data_type in your browser, $data_type means the dataset type which you are training.

Benchmarks

Adam optimizer (lr=0.0001, weight_decay=5e-4) is used without learning rate scheduling. The models are trained with 30 epochs and batch size of 30 on one NVIDIA Tesla V100 (32G) GPU.

The texts are preprocessed as only number and English words, max length is 5000.

Here is the dataset details:

Dataset	agnews	dbpedia	yahoo	sogou	yelp	yelp fine grained	amazon	amazon fine grained
Num. of Train Texts	120,000	560,000	1,400,000	450,000	560,000	650,000	3,600,000	3,000,000
Num. of Test Texts	7,600	70,000	60,000	60,000	38,000	50,000	400,000	650,000
Num. of Vocabulary	62,535	548,338	771,820	106,385	200,790	216,985	931,271	835,818
Num. of Classes	4	14	10	5	2	5	2	5

Here is the model parameter details:

Dataset	agnews	dbpedia	yahoo	sogou	yelp	yelp fine grained	amazon	amazon fine grained
Normal-Linear	4,448,192	35,540,864	49,843,200	7,254,720	13,296,256	14,333,120	60,047,040	53,938,432
CC-Linear	2,449,120	26,770,528	37,497,152	4,704,040	8,479,856	9,128,040	45,149,776	40,568,416
CWC-Linear	2,449,120	26,770,528	37,497,152	4,704,040	8,479,856	9,128,040	45,149,776	40,568,416
Normal-Capsule	4,455,872	35,567,744	49,862,400	7,264,320	13,300,096	14,342,720	60,050,880	53,948,032
CC-Capsule	9,537	3,570	4,110	9,537	3,570	4,110	9,537	3,570
CWC-Capsule	3,783	1,530	4,110	3,783	1,530	4,110	3,783	1,530

Here is the accuracy details:

Dataset	agnews	dbpedia	yahoo	sogou	yelp	yelp fine grained	amazon	amazon fine grained
Normal-Linear	92.03%	98.54%	72.28%	95.00%	95.85%	63.95%	93.44%	56.91%
CC-Linear	72.22%	93.77%	52.31%	94.10%	88.31%	52.65%	82.25%	45.08%
CWC-Linear	92.21%	98.90%	72.65%	96.08%	95.72%	64.29%	93.65%	57.21%
Normal-Capsule	101	51	600	101	51	600	101	51
CC-Capsule	9,537	3,570	4,110	9,537	3,570	4,110	9,537	3,570
CWC-Capsule	3,783	1,530	4,110	3,783	1,530	4,110	3,783	1,530

Results

The train/test loss、accuracy and confusion matrix are showed with visdom. The pretrained models and more results can be found in BaiduYun(access code:xer4) and GoogleDrive.

agnews