Urdu sentence alignment

[zbhatti-dev]

Urdu sentence alignment

[zbhatti-dev]

If you make an improvement to the Moses sentence splitter, please share your code so others can benefit. You can send it to me or create a pull request
…
On Sat, Feb 27, 2021, 3:20 PM zbhatti-dev @.***> wrote: OK yes is working now; the above code change I mentioned doesnt make a difference...so thanks; now working on an urdu postprocessing model to filter out the mis-matches — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#199 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFI4FGSQQM3MXUAJCCDPXLTBF42VANCNFSM4YALHJ2A .

Hi yes if I do change anything and it is useful will let you know...issue I am seeing with urdu and similar languages is that sentences do not quite match...so now looking at constructing an urdu bicleaner...to filter out those translations not matching

[lpla]

Hi. Seems like the Urdu dictionary is well generated. It could be an issue with Urdu in the language detector, the sentence splitter or the word tokenizer.

Could you please upload the intermediate files generated by Bitextor in the "dataDir" and "transientDir" folders? That way it will be easier to find the problem.

[zbhatti-dev]

Hi, thanks for you reply...
please find attached directory contents:

https://drive.google.com/file/d/1Um6iDFMjF3YkhRlVZbP2SBGJEHAqNlAW/view?usp=sharing

https://drive.google.com/file/d/1Pl50qsX2VpzoEFKUoij7rkroKcfjyfaJ/view?usp=sharing

thanks
Zaf

[lpla]

Looks like it is the sentence splitter.

By default, Bitextor uses a Python port of split-sentences.perl (from mosesdecoder) coded by mediacloud because it is the fastest we tested for a broader selection of languages. But this port doesn't have support for Urdu.

However, Bitextor supports user provided sentence splitters commands, and looks like the original split-sentences.perl got support for Urdu lately. Then, if you want to use that sentence splitter, you should add this to your Bitextor configuration file (following the wiki page for non-European languages):

sentenceSplitters: { 'ur' : "~/bitextor/preprocess/moses/ems/support/split-sentences.perl -q -b -l ur" }

[zbhatti-dev]

OK, thanks for this, brilliant help - it seems to have worked; although I have changed one of the code lines in the split-sentences.perl:

(?: [.?!\x{06D4}] | ..+ ) to (?: [.?!\x{6D4}] | ..+ )

but I am not sure if that was needed...will keep you posted

[zbhatti-dev]

OK yes is working now; the above code change I mentioned doesnt make a difference...so thanks; now working on an urdu postprocessing model to filter out the mis-matches

[hieuhoang]

If you make an improvement to the Moses sentence splitter, please share your code so others can benefit. You can send it to me or create a pull request

…

On Sat, Feb 27, 2021, 3:20 PM zbhatti-dev ***@***.***> wrote: OK yes is working now; the above code change I mentioned doesnt make a difference...so thanks; now working on an urdu postprocessing model to filter out the mis-matches — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#199 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFI4FGSQQM3MXUAJCCDPXLTBF42VANCNFSM4YALHJ2A> .