sereni/joint-parsing

joint-parsing's Issues

Ambiguous tokens order

Things break violently if the input corpus has something like this:

1-3 Blah
1 ...
2 ...
3 ...
1 ...
2 ...
4

Note the 2 before the 4, if the range does not end on the max index, the tokens won't be read correctly. If you do 1-2 first and then 1-2-3, works fine.

Output format for multi-word tokens

Output format does not align with the gold corpora. In gold, the first word gets the surface form, and the subsequent words are marked with a *. In our case, word form is duplicated everywhere. This makes file comparisons less convenient.

ours:
10 басымды басымды _ ...
11 басымды е _

gold:
10 басымды басымды
11 * е _