xueyouluo/my_seq2seq

My seq2seq based on tensorflow

my_seq2seq's Issues

inference stage still not quite right

The training stage converges pretty fast for some generated toy data, but the inference stage does not output correct result.

A question has been posted on stackoverflow:
https://stackoverflow.com/questions/44353824/tensorflow-seq2seq-inference-stage-wired-result

Thanks!

Pointer-generator model

I would like to ask how could I run the Pointer-Generator model?

beam_size=1?

Does anyone meet the problem that the parameter 'beam_size' can only be set to 1, or it will result in errors？

bidirectional encoder后面的decoder

如果用biRNN， decoder的hidden_size不应该是encoder的两倍吗？我看你的code里面好像没有这样的设定，关键是code还能跑。请指正？

另外，我发现当num_layers>2 时会出错。事实上，只有当encoder的layer等于1，decoder的layer等于2的时候，code是work的。从下面的code可以看出：
fw_cell = get_rnn_cell('gru',self.dim_size,num_layers= 1, train_phase=self.train_phase, keep_prob=self.keep_prob_config)
bw_cell = get_rnn_cell('gru',self.dim_size,num_layers= 1, train_phase=self.train_phase, keep_prob=self.keep_prob_config)
dec_cell = get_rnn_cell('gru', self.dim_size, num_layers=self.num_layers, train_phase=self.train_phase,
keep_prob=self.keep_prob_config)

Thank you very much

This is not an actual issue, I just want to thank you very much, such a great repository, and I did fix some bugs for deprecated issue, but that just around less than 1% what you already done. Also I added Luong Attention for your pointer generator model.

I tested on real dataset, X = body of news, Y = title of news. Accuracy is based on sequential cross-entropy, never tested using rouge. All this after 10 epochs, 80% to train 20% to test.

On bahdanau attention, https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/abstractive-summarization/7.xueyouluo-pointer-generator-bahdanau.ipynb

epoch: 9, avg loss: 6.818628, avg accuracy: 0.913240
epoch: 9, avg loss test: 16.073720, avg accuracy test: 0.870706

On luong attention, https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/abstractive-summarization/9.xueyouluo-pointer-generator-luong.ipynb

epoch: 9, avg loss: 4.113721, avg accuracy: 0.988244
epoch: 9, avg loss test: 14.028113, avg accuracy test: 0.915375

Thank you so much!

运行pointer模型，报以下错误

File "D:\App\novel\FunText\src\models\pointernet\PointerNetGenerator.py", line 108, in _build_decoder_cell
initial_state = cell.zero_state(self._batch_size, tf.float32).clone(cell_state=cell_state)
File "D:\App\novel\FunText\src\models\pointernet\PointerNetHelper.py", line 202, in zero_state
for _ in self._attention_mechanisms))
TypeError: new() missing 1 required positional argument: 'attention_state'

attention mechanism on the top layer

From this repository, https://github.com/JayParks/tf-seq2seq/blob/master/seq2seq_model.py , I noticed a note saying:
# Note: We implement Attention mechanism only on the top decoder layer

Correspondingly, the code looks like:

    self.decoder_cell_list[-1] = attention_wrapper.AttentionWrapper(
        cell=self.decoder_cell_list[-1],
        attention_mechanism=self.attention_mechanism,
        attention_layer_size=self.hidden_units,
        cell_input_fn=attn_decoder_input_fn,
        initial_cell_state=encoder_last_state[-1],
        alignment_history=False,
        name='Attention_Wrapper')

Why do you add attention to the zeroth decoder layer ? I think the top layer version sounds more reasonable. Thanks!