Conversion scripts not working

[hu-ner]

Conversion scripts not working

[yetinam]

Hey @nakumgaurav,
thanks for reporting the issue. I tried reproducing the issue, but the command runs successfully on my machine. Can you give more detail on your version of python, pexpect, opennlp and java?
@leonweber : Could you check, if you can reproduce the issue on mac? It seems to be an encoding related bug. I'm genuinely suprised to see the Bell character \x07 here.

[leonweber]

@nakumgaurav I have just pushed a commit that should fix this issue: https://github.com/hu-ner/huner/tree/docker_for_ner_scripts . Conversion is handled by the docker container and not the system python on that branch. Could you please verify that this fixes your issue? Then, I'll merge it into master.

[nakumgaurav]

Hi, the bc2gm conversion works but I am facing an error in Biosemantics:

Converting Biosemantics 0%| | 2/4693 [00:00<36:18, 2.15it/s] Traceback (most recent call last): File "scripts/biosemantics_to_conll.py", line 35, in <module> tmp_sentences = sentence_splitter.split(text) File "/usr/ner_scripts/scripts/opennlp_wrapper.py", line 65, in split result, err = p.communicate(text.encode('utf-8')) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 83: ordinal not in range(128) 0%| | 2/4693 [00:00<36:04, 2.17it/s] Traceback (most recent call last): File "scripts/biosemantics_to_conll.py", line 35, in <module> tmp_sentences = sentence_splitter.split(text) File "/usr/ner_scripts/scripts/opennlp_wrapper.py", line 65, in split result, err = p.communicate(text.encode('utf-8')) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 83: ordinal not in range(128)

Also to make Bioinfer work, I had to update the Dockerfile and add lxml since it resulted in the error "No module named lxml". See the new pull request

[nakumgaurav]

Hi @leonweber, it seems lxml and tqdm are still missing from the Docker container. Could you please fix that?

Error on CDR data:
Traceback (most recent call last): File "scripts/cdr_to_conll.py", line 3, in <module> import extractors File "/usr/ner_scripts/scripts/extractors.py", line 2, in <module> from lxml import etree ImportError: No module named lxml
Additionally, I am not sure if we are running python3 in the image, since running the following command:
docker run -t -i huner/huner:dev /bin/bash -c "python --version"
yields
Python 2.7.17

EDIT: Yup, I think that's the issue. The use of python in convert_corpora.sh should be replaced by python3. I will run this on all datasets and let you know

[leonweber]

Hey @nakumgaurav. Could you please verify that you are on the latest commit of https://github.com/hu-ner/huner/tree/docker_for_ner_scripts? convert_corpora.sh should only use python3 now.

[nakumgaurav]

Hi @leonweber , yes I can now see the latest commit has python3. I am able to convert almost all datasets (still running a few). Also, for LocText could you maybe specify the directory structure in the downloads file. I think it requires all json files in the data/LocText directory, since I am facing this error on running covert_corpora.sh:

Converting Loctext 0%| | 0/2 [00:00<?, ?it/s] Traceback (most recent call last): File "scripts/loctext_gene_to_conll.py", line 27, in <module> with open(os.path.join(args.input_dir, file), 'r') as f_in: IsADirectoryError: [Errno 21] Is a directory: 'data/LocText/ann.json' 0%| | 0/2 [00:00<?, ?it/s] Traceback (most recent call last): File "scripts/loctext_gene_to_conll.py", line 27, in <module> with open(os.path.join(args.input_dir, file), 'r') as f_in: IsADirectoryError: [Errno 21] Is a directory: 'data/LocText/ann.json'

Thanks a lot for your constant help!

[leonweber]

Hi @nakumgaurav . I will check this on Monday, thanks for reporting!

[leonweber]

@nakumgaurav LocText can now be downloaded automatically again. Does that resolve the issue for you? And did you convert the other corpora successfully?

[nakumgaurav]

Hi @leonweber, thanks a lot for fixing LocText. I am still having an issue with Variome disease conversion:

Disease 30%|████████████████████████████████████████████████▊ | 116/392 [00:54<02:20, 1.97it/s]Traceback (most recent call last): File "scripts/variome_to_conll.py", line 20, in <module> sentences, entities, document_ids = extractors.extract_from_biocreative(f_in, types, split_sentences=True) File "/usr/ner_scripts/scripts/extractors.py", line 69, in extract_from_biocreative assert annotated_entity == true_entity AssertionError 30%|████████████████████████████████████████████████▊ | 116/392 [00:54<02:10, 2.11it/s]

I have been able to convert all other datasets

[yetinam]

Hey @nakumgaurav ,
thanks for reporting. We didn't get to reproduce and debug the issue yet, but will look at it in the next days.

[yetinam]

Hey @nakumgaurav ,
the issue should be fixed in 850da3b. It resulted from incorrect annotations in Variome, annotating a lowercase "cancer" as the entity, while having an uppercase "Cancer" in the text.
Could you please verify this solves the issue? I'll merge the branches afterwards.

[nakumgaurav]

Wonderful! I am now able to convert Variome disease. Many thanks for your constant help!