exit code: -6

[fergsc]

exit code: -6

[wjq1981]

Thank you for providing your files @wjq1981! We are currently investigating the issue. In the mean time, could you please post your logs/hapog_chunks_8.e and logs/hapog_chunks_8.o?

I have almost sent you an email.

[bistace]

Hello,
could you please paste the content of the logs/hapog_chunks_69.e and logs/hapog_chunks_69.o files?

[fergsc]

I had to rerun hapo-g (I'm running on a HPC and the logs were not saved when the job crashed). Same error just reporting a different chunk this time

ERROR: Hapo-G didn't finish successfully, exit code: -6
Faulty command: /g/data/xe2/scott/gadi_modules/hapog/build/hapog -b chunks_bam/chunks_9.bam -f chunks/chunks_9.fasta -o hapog_chunks/chunks_9.fasta -c hapog_chunks/chunks_9.changes

I have included all logs. There was no chunk_69.
logs.zip

Thanks for the help.

[wjq1981]

I had to rerun hapo-g (I'm running on a HPC and the logs were not saved when the job crashed). Same error just reporting a different chunk this time

ERROR: Hapo-G didn't finish successfully, exit code: -6
Faulty command: /g/data/xe2/scott/gadi_modules/hapog/build/hapog -b chunks_bam/chunks_9.bam -f chunks/chunks_9.fasta -o hapog_chunks/chunks_9.fasta -c hapog_chunks/chunks_9.changes

I have included all logs. There was no chunk_69. logs.zip

Thanks for the help.

Hi, did you get your problem solved?

[bistace]

Hello,
we've pushed a few changes on Conda and on Github that specifically solve some problems encountered when polishing with long reads. Could you please check it out?

[wjq1981]

Hello, we've pushed a few changes on Conda and on Github that specifically solve some problems encountered when polishing with long reads. Could you please check it out?

Okay, I'll tell you the results when I'm done testing.

[wjq1981]

Hello, we've pushed a few changes on Conda and on Github that specifically solve some problems encountered when polishing with long reads. Could you please check it out?

Hi, The program still has the same problem.

Checking dependencies...
Found BWA.
Found Samtools.

Generating bwa index...
Done in 912 seconds

Launching mapping on genome...
Done in 12247 seconds

Indexing the BAM file...
Done in 664 seconds

Fragmenting the genome into 40 chunks of 28,860,154 bases (depending of scaffold sizes)
Done in 7 seconds

Extracting bam for each chunk
Done in 320 seconds

Launching Hapo-G on each chunk
ERROR: Hapo-G didn't finish successfully, exit code: -9
Faulty command: /share/database/HAPO-G-1.3/build/hapog -b chunks_bam/chunks_8.bam -f chunks/chunks_8.fasta -o hapog_chunks/chunks_8.fasta -c hapog_chunks/chunks_8.changes

[bistace]

Would it be possible for you to share your chunks_bam/chunks_8.bam and chunks/chunks_8.fasta? If so, could you please shoot an email to [email protected] so I can send you a secure link to upload your data?

[bistace]

Thank you for providing your files @wjq1981! We are currently investigating the issue. In the mean time, could you please post your logs/hapog_chunks_8.e and logs/hapog_chunks_8.o?

[bistace]

This is surprising, I can't reproduce the issue on our system with your input files and there is no error in the log files. Could you please try to launch the failing command separately? I mean this one:

/share/database/HAPO-G-1.3/build/hapog -b chunks_bam/chunks_8.bam -f chunks/chunks_8.fasta -o hapog_chunks/chunks_8.fasta -c hapog_chunks/chunks_8.changes

Would it be possible that something exterior to Hapo-G is killing the command before it can finish like a job scheduler if you are using a computing cluster?

[wjq1981]

This is surprising, I can't reproduce the issue on our system with your input files and there is no error in the log files. Could you please try to launch the failing command separately? I mean this one:

/share/database/HAPO-G-1.3/build/hapog -b chunks_bam/chunks_8.bam -f chunks/chunks_8.fasta -o hapog_chunks/chunks_8.fasta -c hapog_chunks/chunks_8.changes

Would it be possible that something exterior to Hapo-G is killing the command before it can finish like a job scheduler if you are using a computing cluster?

It is possible to run it alone, I have tested it. But running together does not work. I don't know what the problem is.

---

/share/HAPO-G-1.3/build/hapog -b chunks_bam/chunks_${SLURM_ARRAY_TASK_ID}.bam
-f chunks/chunks_${SLURM_ARRAY_TASK_ID}.fasta
-o hapog_chunks/chunks_${SLURM_ARRAY_TASK_ID}.fasta
-c hapog_chunks/chunks_${SLURM_ARRAY_TASK_ID}.changes

cat hapog_chunks/chunks_.fasta > hapog.fasta && cat hapog_chunks/chunks_.changes > hapog.changes

This is no problem.

But,

/share/HAPO-G-1.3/hapog.py --genome /share/genome/07.polish/hap2-1/hapog.fasta
--pe1 /share/02.dnaclean/raw_R1.fq.gz
--pe2 /share/02.dnaclean/raw_R2.fq.gz
--output hap2-2
--threads 46

The above mentioned problems will occur.

[wjq1981]

This is surprising, I can't reproduce the issue on our system with your input files and there is no error in the log files. Could you please try to launch the failing command separately? I mean this one:

/share/database/HAPO-G-1.3/build/hapog -b chunks_bam/chunks_8.bam -f chunks/chunks_8.fasta -o hapog_chunks/chunks_8.fasta -c hapog_chunks/chunks_8.changes

Would it be possible that something exterior to Hapo-G is killing the command before it can finish like a job scheduler if you are using a computing cluster?

Like you said, there could be some reason on the cluster that caused it to be terminated early. Thank you very much.

[bistace]

I suspect that you might be running out of memory due to the 46 Hapo-G running simultaneously (you mentioned that you used --threads 46). However, this is quite surprising since Hapo-G doesn't use that much RAM, do you know how much memory is available on the machine you are using?

One test you could do if you have some time to spare would be to reduce the number of threads to see if that helps.

[wjq1981]

I suspect that you might be running out of memory due to the 46 Hapo-G running simultaneously (you mentioned that you used --threads 46). However, this is quite surprising since Hapo-G doesn't use that much RAM, do you know how much memory is available on the machine you are using?

One test you could do if you have some time to spare would be to reduce the number of threads to see if that helps.

One node has 252Gb of memory.
Okay, I'll try reducing the number of threads later.