Improve performance of target prefix in batch mode

[opennmt]

Improve performance of target prefix in batch mode

[guillaumekln]

Hi,

You can still pass a batch of inputs, but internally they will be processed individually. Did you encounter performance issues while using a target prefix?

[guillaumekln]

@vince62s Were you able to try passing a batch of inputs and observe the behavior?

[vince62s]

not yet sorry, still finetuning a model to test this.

[vince62s]

@guillaumekln In fact I am not sure if we are talking about the same thing.
I don't think that using a target prefix itself is the issue, we just observe a slow translation because of the lack of batching.
End result yes it is slower than normal inference.

[guillaumekln]

Yes, I understand. I just thought that my comment in OpenNMT/OpenNMT-py#1800 led you to think that we are not accepting a batch of inputs when using a target prefix.

So this issue is about improving batch performance of target prefix.

[guillaumekln]

How are you using the target prefix? Is it a fixed prefix for all inputs, like a context?

I'm just not sure to find use cases where a target prefix is available for many inputs at once.

[vince62s]

as soon as you deal with many special tokens (politeness, extractiveness for summarization, length control, context, ...) , it becomes convoluted, not always fixed.

[guillaumekln]

Why not adding these tokens on the source side? The issue with the target side is that the first decoding step on <s> is not constrained by these tokens.

[vince62s]

this is not necessarily what we are doing exactly, but this gives you an example.
https://www.aclweb.org/anthology/W17-5702.pdf