kallisto subworkflow runs out of memory (reiteration of #38)

[nf-core]

kallisto subworkflow runs out of memory (reiteration of #38)

[Khajidu]

I set the memory as the following:

`process {
withLabel:process_high {
memory = 500.GB
}
withName:'KALLISTOBUSTOOLS_REF' {
memory = 250.GB
}
withName:'KALLISTOBUSTOOLS_COUNT' {
memory = 250.GB
}

}`

[Khajidu]

Solved by giving more memory to the process in the config file.

[apeltzer]

You could try supplying a separate config that overwrites what kallisto is using for all steps, e.g. https://nf-co.re/usage/configuration#custom-configuration-files and then supplying something for memory:

withName: 'KALLISTOBUSTOOLS_REF' {
        memory : '32.GB'
    }

[Khajidu]

It didn't work, same error.

Here are the logs:

`[2022-06-22 09:29:53,353] INFO [ref_lamanno] Preparing sc_ncbi_genome.fasta, sc_gene_models_ncbi_no_genes_no_contigs_notrnas.gtf

[2022-06-22 09:30:27,852] INFO [ref_lamanno] Splitting genome sc_ncbi_genome.fasta into cDNA at /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmp_aaaucz5

[2022-06-22 09:31:08,887] INFO [ref_lamanno] Creating cDNA transcripts-to-capture at /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmpxx9d3eue

[2022-06-22 09:31:09,220] INFO [ref_lamanno] Splitting genome into introns at /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmp_4zx6l_e

[2022-06-22 09:37:03,403] INFO [ref_lamanno] Creating intron transcripts-to-capture at /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmpw8sjit6k

[2022-06-22 09:37:11,916] INFO [ref_lamanno] Concatenating 1 cDNA FASTAs to cdna.fa

[2022-06-22 09:37:12,346] INFO [ref_lamanno] Concatenating 1 cDNA transcripts-to-captures to cdna_t2c.txt

[2022-06-22 09:37:12,369] INFO [ref_lamanno] Concatenating 1 intron FASTAs to intron.fa

[2022-06-22 09:37:28,411] INFO [ref_lamanno] Concatenating 1 intron transcripts-to-captures to intron_t2c.txt

[2022-06-22 09:37:28,593] INFO [ref_lamanno] Concatenating cDNA and intron FASTAs to /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmp4jp7quuj

[2022-06-22 09:37:46,283] INFO [ref_lamanno] Creating transcript-to-gene mapping at t2g.txt

[2022-06-22 09:38:01,695] INFO [ref_lamanno] Indexing /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmp4jp7quuj to kb_ref_out.idx

[2022-06-22 09:47:41,663] ERROR [ref_lamanno]

[build] loading fasta file /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmp4jp7quuj

[build] k-mer length: 31

[build] warning: clipped off poly-A tail (longer than 10)
from 242 target sequences

[build] warning: replaced 24801554 non-ACGUT characters in the input sequence
with pseudorandom nucleotides

[build] counting k-mers ...

[2022-06-22 09:47:41,679] ERROR [main] An exception occurred

Traceback (most recent call last):

File "/usr/local/lib/python3.9/site-packages/kb_python/main.py", line 856, in main
COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)

File "/usr/local/lib/python3.9/site-packages/kb_python/main.py", line 131, in parse_ref
ref_lamanno(

File "/usr/local/lib/python3.9/site-packages/ngs_tools/logging.py", line 62, in inner
return func(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/kb_python/ref.py", line 661, in ref_lamanno
index_result = kallisto_index(combined_path, index_path, k=k or 31)

File "/usr/local/lib/python3.9/site-packages/kb_python/ref.py", line 212, in kallisto_index
run_executable(command)

File "/usr/local/lib/python3.9/site-packages/kb_python/dry/init.py", line 24, in inner
return func(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/kb_python/utils.py", line 195, in run_executable
raise sp.CalledProcessError(p.returncode, ' '.join(command))

subprocess.CalledProcessError: Command '/usr/local/bin/kallisto index -i kb_ref_out.idx -k 31 /shared/ifbstor1/projects/bsbii/sc_single_cell_brain/work/ca/8de08f1ecfab6f41d8cc1f94e45c52/tmp/tmp4jp7quuj' died wit
h <Signals.SIGKILL: 9>.

slurmstepd: error: Detected 1 oom-kill event(s) in StepId=23442481.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.`

[Khajidu]

Link to profile configuration if needed: https://github.com/nf-core/configs/blob/master/conf/ifb_core.config

[apeltzer]

Ok, then lets add another profile for you that automatically does that for your cluster

[Khajidu]

Good!

[apeltzer]

Can you share what/how you modified the memory in the process config file? Then we can simply copy that

[grst]

Any chance this got fixed also for less memory now that we added the -m <MEMORY> flag in the kallisto module?

[ogibson]

This issue seems to be resolved. @apeltzer, is there anything else that can be done here?

[apeltzer]

No, if it works we can just close here 👍🏻