processes have been killed by the cgroup out-of-memory handler.

[nextomics]

processes have been killed by the cgroup out-of-memory handler.

[moold]

Hi, our tests show a typical minimap2-nd task usually consumes about 40g of memory, but you can use -I parameter to reduce the max memory requirement. In addition, you can also adjust -t in minimap2_options_raw and minimap2_options_cns to adapt the maximum sub-jobs running on each nodes simultaneously.

[biowackysci]

hello again,
i tried with the modified options for minimap2_options_raw = -x ava-ont -t 32 -I 100 and minimap2_options_cns = -x ava-ont -t 32 -k17 -w17 but now the pipeline stops at 01.raw_align/02.raw_align.sh.work/raw_align000/ with a error description of "slurmstepd: error: *** JOB 1734386 ON comp054 CANCELLED AT 2020-02-08T21:42:13 DUE TO TIME LIMIT ***
"
Is it something I can modify in the script or just restart the job?

Also, is there an option where I can filter for reads above certain read lengths ?

Thanks in advance so much for your help

S

[moold]

It seems that your system limits the running time of a job, so you can reduce blockize and increase seed_cutfiles to reduce the size of each subfile and speed up the map task. But the total runing time maybe will longer.

[biowackysci]

Thanks will try this now and update soon with the outcome

Regards,
S

[biowackysci]

Hello again ,
I modified the script with reduced blocksize of 3g and increased the seed_cutfiles to 10 but it still stalls at [ERROR] 2020-02-21 06:48:13,170 /group/pasture/Saila/NextDenovo/01.raw_align/02.raw_align.sh.work/raw_align159/nextDenovo.sh.e
with an error description of slurmstepd: error: *** JOB 1809950 ON comp035 CANCELLED AT 2020-02-21T07:00:20 DUE TO TIME LIMIT ***
Not sure if i should reduce the blocksize still to smaller size ?
Can you please advice
Thanks
S

[moold]

Yes or contact your system administrator for help, your system limits the running time of a job.

[lifan18]

Dear Dr. Hu,

Thank you for your replies above. I have the same question, "out-of-memory". I tried to adjust the -t in minimap2_options_raw and minimap2_options_cns as -t 8, but it still failed to pass this error.

see run.cfg parameters:
"
[correct_option]
read_cutoff = 1k
genome_size = 3.23g # estimated genome size
sort_options = -m 20g -t 8
minimap2_options_raw = -t 8
pa_correction = 5
correction_options = -p 30

[assemble_option]
minimap2_options_cns = -t 8
nextgraph_options = -a 1
"

see error message:
minimap2-nd --step 1 --dual=yes -t 8 -x ava-pb /01.raw_align/input.seed.005.2bit /01.r aw_align/input.part.004.2bit -o input.seed.005.2bit.163.ovl slurmstepd-c03b06n04: error: Detected 1 oom-kill event(s) in step 593045.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

The slurm cluster I used has enough memory (q_fat, 72 cpus/1.5T memory/2 nodes), but the default normal queue has limited memory. The subtasks submitted by paralleltask always use default normal queue even though I submit the main job to q_fat Therefore, I tried to use submit specify the queue for paralleltask, for example submit = sbatch -q q_fat, but it seems it was a wrong way to specify queue.

I also tried the way in [https://nextdenovo.readthedocs.io/en/latest/FAQ.html#how-to-optimize-parallel-computing-parameters].

Could you tell me how to specify the queue in a correct way or give some suggestion for this out-of-memory issue?

Thank you very much!

Regards,

LF

[moold]

Two solutions:

1. use submit = sbatch -q q_fat --cpus-per-task={cpu} --mem-per-cpu={mem} -o {out} -e {err} {script}
2. set job_type = local and submit the main job to q_fat

[lifan18]

Thank you! I will update you once it is done.

[lifan18]

Thank you! This issue solved through job_type= slurm and submit = sbatch -p q_fat --cpus-per-task=1 --mem-per-cpu=64g -o {out} -e {err} {script} ;D