Error when writting with ` ZBuilder::<Bgzf, _>::new().num_threads(8)...`

[sstadick]

Error when writting with ` ZBuilder::::new().num_threads(8)...`

from gzp.

[sstadick]

Good catch! The fix to make sure large writes were handled ended up skipping the EOF marker. As you noted it's not technically needed but the HTS suit of tools will complain if it isn't present.

v0.9.5 has the fixes. Thanks for following up!

[sstadick]

@mrvollger Sorry the delay in getting to this! I'll take a look later tonight or tomorrow. I don't see anything immediately wrong with your example code.

[mrvollger]

Thanks so much for taking a look! I am having a hard time confirming this but I think it might have to do with the length of the sequences. I have run basically this code on short reads and never had a problem, but then I tried some contigs from assemblies and started to get this error.

Thanks again!

[sstadick]

Yep, I have a fix incoming to gzp. It looks like when the write calls have more bytes than the BGZF max buffer size the final call to flush will try to send all the remaining bytes at once, which will then fail in BGZF since the number of bytes exceeds the max buffer size allowed by the BGZF spec.

Thank you for making this issue! I would not have run into this normally I don't think and missed it in all my test cases as well.

[sstadick]

Try the v0.9.3 release

[sstadick]

And thank you again for the issue!

[mrvollger]

Just tried it and it got through the file and wrote all the records! Thanks!

However, I do get this error/warning from samtools when I try to index the resulting file.

$ cat .test/large.test.fa.gz | cargo run --release --bin rb -- -vvv fasta-split 1.fa.gz
$ samtools faidx 1.fa.gz
[W::bgzf_read_block] EOF marker is absent. The input is probably truncated

The index does appear to be created after this, and it is correct in this case, but it seems a little worrying.