domerin0/neural-sentiment

A sentiment analyzer using deep rnns, built with TensorFlow

neural-sentiment

A sentiment analyzer using deep rnns, built with TensorFlow.

Installation & Dependency Instructions

1. Python 2 or 3
2. Install [TensorFlow 1.2]https://www.tensorflow.org/install/)
3. $ pip install nltk

Next you will need to download the nltk tokenizer dataset:

1. run a python session in terminal $ python
2. >>> import nltk
3. >>> nltk.download()
4. type 'd', then hit enter
5. type 'punkt' then hit enter

You can exit the python terminal once the download is done, your dependencies will be all setup.

Usage Instructions

To run with preset hyper-parameters just run:

$ python train.py

To try other hyper-parameters, you can change them in the config file. If you want to provide your own:

$ python train.py --config_file="path_to_config"

Descripton of hyper parameters:

Name	Type	Description
hidden_size	int	number of hidden units in hidden layers
num_layers	int	number of hidden layers
batch_size	int	size of batchs in training
max_epoch	int	max number of epochs to train for
learning_rate	float	beggining learning rate
steps_per_checkpoint	int	number of steps before running test set
lr_decay_factor	float	factor by which to decay learning rate
batch_size	int	size of batchs in training
max_seq_length	int	maximum length of input token sequence
checkpoint_dir	string	directory to store/restore checkpoints
dropout	float	probability of hidden inputs being removed
grad_clip	int	max gradient norm

When you first run train.py, it will download the data set, unzip, and process it. This will take several minutes. The training will begin automatically after this is done.

After your model is trained you can run:

$ python sample.py --text="Your text to sample here"

It will return 0 or 1. This number corresponds to a positive (1), or a negative (0) score.

Model

The model is: embedding layer -> LSTMCells -> logistic regression output layer. I'll provide a picture in the future.

The hidden state averaged across all time steps is what is passed to the logistic regression layer.

Tensorboard Usage

I've begun to implement tensorboard variables and histograms.

You can access tensorboard with this application by using:

$ tensorboard --logdir=/tmp/tb_logs/

Then copy and paste the localhost url your terminal window gives you, into your browser of choice.

There isn't much to see right now except an overview of the graph. I will include screenshots of this down the road as I add more tensorboard functionality to this project.

Results

Pre-trained model will be committed soon. Currently with defaults, it gets about 70%. I would like to get it to about 80-85% before committing the model.

Attribution

Thanks to reddit user /u/LeavesBreathe for the help with getting the last hidden state

The dataset used was the Large Movie Review Dataset.

I also referred to this tutorial as a starting point

neural-sentiment's Issues

tf.pack() issue while building the model..

Hi Dominik,

Thanks very much for this great tool. I tried to use this tool kit and i am ending up with an error. I have all the dependencies.
The code downloads the database, builds the vocab and 4 data files in processed directory and than when i am trying to build a model it give the following error. Please can you suggest what might be the problem..

Traceback (most recent call last):
File "train.py", line 149, in
main()
File "train.py", line 69, in main
model = createModel(sess, vocab_size)
File "train.py", line 123, in createModel
FLAGS.learning_rate, FLAGS.lr_decay_factor)
File "/home/neural-sentiment/models/sentiment.py", line 59, in init
concat_states = tf.pack(encoder_state)
File "/home/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 228, in pack
return gen_array_ops._pack(values, name=name)
File "/home/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 523, in _pack
return _op_def_lib.apply_op("Pack", values=values, name=name)
File "/home/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py", line 357, in apply_op
(input_name, op_type_name, values))
TypeError: Expected list for 'values' argument to 'Pack' Op, not Tensor("RNN/concat_199:0", shape=(?, 200), dtype=float32).

regards
Kannadaraj

logistics regression layer not properly implemented

I will need to fix the logistic regression layer to properly output the label.

padding all sequences to same length is inefficient

Will add bucketing to resolve.

There is no restoreHyperParameters function in the sample.py

There is no restoreHyperParameters function in the sample.py
def loadModel(session, vocab_size):
hParams = restoreHyperParameters()
model = models.sentiment.SentimentModel(int(hParams[0]), int(hParams[1]),
float(hParams[2]), int(hParams[3]), int(hParams[4]), int(hParams[5]),
float(hParams[6]),float(hParams[7]) ,True)
ckpt = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir)
if ckpt and gfile.Exists(ckpt.model_checkpoint_path):
print ("Reading model parameters from {0}".format(ckpt.model_checkpoint_path))
model.saver.restore(session, ckpt.model_checkpoint_path)
else:
print ("Double check you got the checkpoint_dir right...")
print ("Model not found...")
model = None
return model

memory is blowing up

This is only occurring on the dev branch, but it seems to freeze and take up all the available memory. I probably have some stupid mistake somewhere, and will fix it shortly.

Problem with tensorflow version 0.12 head

Hi, I have installed tensorflow from source.
In the sentiment.py I had to change tf.contrib.rnn.LSTMCell, tf.contrib.rnn.DropoutWrapper, tf.contrib.rnn.MultiRNNCell at line 75. But cannot find any replacement for rnn.rnn. I have tried tf.nn.rnn but it gave an error "'module' object has no attribute 'rnn' ". I have also tried tf.contrib.rnn but got an error "'module' object is not callable "

Tensorflow version

Which tensorflow version should we use with this code?

Model not found...

Hi @inikdom
I've test the model with this command(python sample.py --text="she is ugly.")
but I got this error
/Projects/neural-sentiment$ python sample.py --text="very good"
2018-05-18 23:51:37.258393: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-18 23:51:37.258419: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-18 23:51:37.258434: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-05-18 23:51:37.258441: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-18 23:51:37.258456: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Double check you got the checkpoint_dir right...
Model not found...

How can I fix it!

Missing util/hyperparams?

Hi Domnik,
I am wondering if the hyper parameters should be added from our side or is it missing from the repository (under util/hyperparams.py). Thank you for the code.

Steven

Training is not working?

I've just cloned your repo, downloaded the nltk tokenizer dataset and just tried to train the model using the default config. And that's what I've got after training it for 3 hours on CPU

The accuracy is just 50% all the time. Am I doing something wrong or maybe something has changed?
Output of the python --version is
Python 2.7.13 :: Anaconda custom (64-bit)
The TensorFlow version is 1.2.1
Ubuntu 16.04

Code does not run with the latest Tensorflow version.

Mostly it breaks at wherever there is RNN cell. I am trying to fix it, but too many changes. I would suggest you to keep the repo updated if possible.

Dropout used incorrectly!

during the test the dropout_keep_prob parameter is not set to 1 as it should, the dropout_keep_prob parameter shouldn't be constant but a placeholder or at least a tf.Variable

checkpoint names

I made some changes to your code to run it on Tensorflow 1.0.1. Training works fine, and I see checkpoints getting saved in the data/checkpoints/ folder as follows:

sentiment0.501447366256.ckpt-50.data-00000-of-00001
sentiment0.501447366256.ckpt-50.index
sentiment0.501447366256.ckpt-50.meta
sentiment0.494999998494.ckpt-100.data-00000-of-00001
sentiment0.494999998494.ckpt-100.index
sentiment0.494999998494.ckpt-100.meta

However, in the checkpoints file, the checkpoint names are saved as follows:

model_checkpoint_path: "sentiment0.494999998494.ckpt-100"
all_model_checkpoint_paths: "sentiment0.501447366256.ckpt-50"
all_model_checkpoint_paths: "sentiment0.494999998494.ckpt-100"

Now when I run python sample.py , it says model not found, please check that your checkpoint_dir is right. Why is this happening? I tried adding ".meta" or ".index" to the end of model_checkpoint_path. Then the model is found but there are other errors and the model doesn't get loaded. Any idea why this is happening?

ValueError: setting an array element with a sequence

I was trying to use this model. When I tried on my local system, it worked correctly. Same thing I tried to run on aws server, it gave me

ValueError: setting an array element with a sequence

Error code:

WARNING:tensorflow:<tensorflow.python.ops.rnn_cell.LSTMCell object at 0x7f56e6c2cb10>: The input_size parameter is deprecated.
Traceback (most recent call last):
  File "train.py", line 194, in <module>
    main()
  File "train.py", line 63, in main
    model = create_model(sess, hyper_params, vocab_size)
  File "train.py", line 124, in create_model
    hyper_params["batch_size"])
  File "/home/datametica/karim/deeplearning/neural-sentiment/models/sentiment.py", line 73, in __init__
    self.initial_state = tf.zeros([self.batch_size, self.cell.state_size])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1184, in zeros
    shape = ops.convert_to_tensor(shape, dtype=dtypes.int32, name="shape")
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 657, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 180, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 163, in constant
    tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 354, in make_tensor_proto
    nparray = np.array(values, dtype=np_dt)
ValueError: setting an array element with a sequence.

I thought it is due to numpy version. so I tried changing it, but no help.

What I found is, this code works well with tensorflow 0.8.0 .

If you install latest tensorflow and try this code, it will give this error.

So I uninstalled latest version and installed 0.8.0, now again it works fine.