Nested Structure in ListOps

[google-research]

Nested Structure in ListOps

[vanzytay]

The brackets are required. This issue has been fixed internally and we are going to push an update to this codebase soon (In Jan). Since this public version acts as a mirror, we have not sync-ed for awhile now.

[adamsolomou]

Thanks for clarifying!

So the correct way to go would be to tokenize according to this vocabulary

( ']', '1', '4', '7', '9', '8', '2', '5', '3', '0', '6', '[MIN', '[SM', '[MAX', '[MED'])?

[cifkao]

I suppose the fix would be to pass something like reserved_tokens=['[', ']'] to tfds.deprecated.text.Tokenizer to prevent it from ignoring these?

[adamsolomou]

If you address the issue this way it gives rise to the following vocabulary

('MED', '5', '8', '4', '0', ']', '1', '3', '6', '7', '2', 'MIN', '9', '[', 'MAX', 'SM')

which differs from this

( ']', '1', '4', '7', '9', '8', '2', '5', '3', '0', '6', '[MIN', '[SM', '[MAX', '[MED')

in the sense that [ is encoded as a token itself rather than being part of '[MIN', '[SM', '[MAX', '[MED'.

@vanzytay could you please clarify which is the right way to go?