Hunalign and Bicleaner errors

[bjarnigithub]

Hunalign and Bicleaner errors

[cgr71ii]

Hi!

Could you share the specific command that you ran in order to try to reproduce the problem (and the provided configuration file if used any)?

[bjarnigithub]

Here is the command I used:

bitextor-full --configfile yamls/config.yaml --cores 7

And here is the yaml file I used:

# DATA FOLDERS
permanentDir: data/permanent/bitextor-output
dataDir: data/permanent/data
transientDir: data/transient
tempDir: data/transient

# WORKFLOW
parallelWorkers: {split: 2, tokenise: 2, docalign: 2, segalign: 2, bifixer: 2, bicleaner: 2, sents: 2} 

# DATA SOURCES
warcs: ["data/warcs/skatturinn.warc.gz"]

# PREPROCESSING
preprocessor: warc2preprocess
langID: cld3
ftfy: True
cleanHTML: True
html5lib: True
boilerplateCleaning: True 
paragraphIdentification: True 
parser: bs4

# SHARDING
shards: 8
batches: 3072

# LANGS
lang1: en
lang2: is

# DOCUMENT ALIGNMENT
documentAligner: DIC
dic: data/bilingLex/en-is_is-en_glossary_21-09/en-is.dic

# SEGMENT ALIGNMENT
sentenceAligner: hunalign 
sentenceAlignerThreshold: 0.1

# PARALLEL DATA FILTERING
bicleaner: True 
bicleanerModel: data/bicleaner-model/bitextor_bicleaner_model/en-is/en-is.yaml 
bicleanerFlavour: classic
bicleanerThreshold: 0.5

# POST-PROCESSING
bifixer: True
elrc: True
tmx: True
deduped: True
biroamer: False

[cgr71ii]

The problem is fixed in master branch. The problem was related to the config value parallelWorkers: {..., bicleaner: 2, ...}. If you want to fix the problem in your conda installation instead of downloading and installing manually Bitextor, modifying $CONDA_PREFIX/bitextor/bitextor/Snakefile and applying the changes applied in commit f033675, the problem should be gone (we will not release a new conda build for now).

Closing issue. Reopen if needed.