freedomintelligence/textclassificationbenchmark

A Benchmark of Text Classification in PyTorch

textclassificationbenchmark's Issues

loadData in utils is not compatible with the transformer model

It seems loadData in utils is not the same as the one in dataHelper

When to support more dataset?

I am so sad it's not use straight.

we I want use trec dataset to test ,it cant run ,and dataSet file cant find i have to said,it NOT friendly to me.I just want quickly iterator,IT waste my time.

Does it need a new Data preprocess to raplace torchText?

Visualization with confusion matrix for text classfication

docs in English as well please?

Hello,

Can I raise a PR with docs in English?

This would help your work reach a larger audience.

More Models supporting?

write a new readme in English, please

Difference performance between the following results

Windows/Pytorch 0.3/Python3
linux/pytorch 0.2/Python2
Results on windows:
0 ieration 0 epoch with loss : 1.11731
0 ieration 100 epoch with loss : 0.95308
0 ieration 200 epoch with loss : 0.50708
0 ieration 300 epoch with loss : 0.75614
0 ieration with percision 0.8398
1 ieration 0 epoch with loss : 0.16429
1 ieration 100 epoch with loss : 0.10894
1 ieration 200 epoch with loss : 0.05845
1 ieration 300 epoch with loss : 0.34559
1 ieration with percision 0.8438
2 ieration 0 epoch with loss : 0.09230
2 ieration 100 epoch with loss : 0.01140
2 ieration 200 epoch with loss : 0.23463
2 ieration 300 epoch with loss : 0.00004
2 ieration with percision 0.8346
3 ieration 0 epoch with loss : 0.05743
3 ieration 100 epoch with loss : 0.40399
3 ieration 200 epoch with loss : 0.01134
3 ieration 300 epoch with loss : 0.30481
3 ieration with percision 0.8086
4 ieration 0 epoch with loss : 0.03121
4 ieration 100 epoch with loss : 0.00000
4 ieration 200 epoch with loss : 0.36847
4 ieration 300 epoch with loss : 0.07362
4 ieration with percision 0.7786
5 ieration 0 epoch with loss : 0.10139
5 ieration 100 epoch with loss : 0.02151
5 ieration 200 epoch with loss : 0.00000
5 ieration 300 epoch with loss : 0.00000
5 ieration with percision 0.8099

Linux results:
0 ieration 0 epoch with loss : 1.09381
0 ieration 100 epoch with loss : 0.69792
0 ieration 200 epoch with loss : 0.69200
0 ieration 300 epoch with loss : 0.67666
0 ieration with percision 0.6022
1 ieration 0 epoch with loss : 0.66189
1 ieration 100 epoch with loss : 0.61229
1 ieration 200 epoch with loss : 0.50442
1 ieration 300 epoch with loss : 0.48552
1 ieration with percision 0.7489
2 ieration 0 epoch with loss : 0.24260
2 ieration 100 epoch with loss : 0.14823
2 ieration 200 epoch with loss : 0.22468
2 ieration 300 epoch with loss : 0.27329
2 ieration with percision 0.6626
3 ieration 0 epoch with loss : 0.06171
3 ieration 100 epoch with loss : 0.05115
3 ieration 200 epoch with loss : 0.04176
3 ieration 300 epoch with loss : 0.03525
3 ieration with percision 0.6915
4 ieration 0 epoch with loss : 0.01683
4 ieration 100 epoch with loss : 0.01829
4 ieration 200 epoch with loss : 0.00905
4 ieration 300 epoch with loss : 0.02068
4 ieration with percision 0.6919
5 ieration 0 epoch with loss : 0.00466
5 ieration 100 epoch with loss : 0.00262
5 ieration 200 epoch with loss : 0.00331
5 ieration 300 epoch with loss : 0.00265
5 ieration with percision 0.6734
6 ieration 0 epoch with loss : 0.00100
6 ieration 100 epoch with loss : 0.00166
6 ieration 200 epoch with loss : 0.02555
6 ieration 300 epoch with loss : 0.00685
6 ieration with percision 0.6747
7 ieration 0 epoch with loss : 0.00118
7 ieration 100 epoch with loss : 0.00065
7 ieration 200 epoch with loss : 0.00031
7 ieration 300 epoch with loss : 0.00016
7 ieration with percision 0.6703
8 ieration 0 epoch with loss : 0.00030
8 ieration 100 epoch with loss : 0.00029
8 ieration 200 epoch with loss : 0.00029
8 ieration 300 epoch with loss : 0.00006
8 ieration with percision 0.6694
9 ieration 0 epoch with loss : 0.00016
9 ieration 100 epoch with loss : 0.00093
9 ieration 200 epoch with loss : 0.00020
9 ieration 300 epoch with loss :

Saving and Loading saved models gives different results

Hi,
There is a serious problem in the current codebase. If you save a model then reload it in a DIFFERENT time (not the same execution of main.py) the accuracy is 50% on IMDB. As a sanity check if you save and reload in the execution of main.py then there is no problem. If I had to guess this is a dataloading problem where there is a mistmatch between the saved model and the newly loaded model in a second execution.

why donot use optimizer.zero_grad in each training step?

https://github.com/wabyking/TextClassificationBenchmark/blob/b5ac116c74493c28f2af2541f21385df7c73ef93/main.py#L71

TODO : add ensemble strategy

Benchmark results in README

Hi, would it be possible for the authors to add the accuracy results that they're getting to the README? Right now I'm seeing numbers which are mid 80's for certain models when I know certain people have reported 89/90 with Deep Learning methods on IMDB.

Add BERT benchmarks

Thanks for this repo. It could be very interesting to add the latest Google's BERT model that claims to be state-of-the-art in recent NLP tasks among them text classification. They have a classifier implementation to adapt, here some details.

There is a small error in the ./docs/data_config.md and ./docs/data_config_en.md respectively.

for the ./docs/data_config.md file : "TREC 1 2 问题分类数据集下载到.data/imdb" may be "TREC 1 2 问题分类数据集下载到.data/trec"

for the ./docs/data_config_en.md file : "Download TREC Question Classification 2 dataset to .data/imdb" may be "Download TREC Question Classification 2 dataset to .data/trec"

In general, "imdb" may be replaced by "trec".

TODO: use different pretrained Model like GPT and XLNET

Automatically download embeddings

Hi,

thanks creating this text classification benchmark!

I wanted to run the basic example python3 main.py --model cnn and I could see that the GloVe embeddings were not downloaded automatically.

The dataHelper.loadData(opt) never calls the Glove constructor, so the embeddings won't be downloaded. But when I change from_torchtext = False to from_torchtext = True the utils.loadData(opt) method calls the Glove constructor.

I guess calling the Glove constructor would be enough to call it before the glove_file declaration (from here)?

How to run?

Where is main.py? Thanks

TODO: With Ragu to test the Robustness of the Models

Ragu : https://github.com/wabyking/Ragu

It would be much better that you can provide the best results or average results over multiple runs for each dataset and each model

it’s not a good idea to commit the commented out codes.

In order to keep the neatness of the commit codes, I think it would be better to remove the commented out codes before commit, or just integrating the commented out codes to publish version.

how to get start the program

may I ask you for the step of starting your program,please?
because I am green hand.

Before we start to introduce our project， we should add enough citation for our code