Insufficient memory？

[chhylp123]

Insufficient memory？

[chhylp123]

How much RAM does my computer have?

[DongnaMa]

I'm running in a linux environment with 1000G RAM.

[chhylp123]

Based on our experiments, 200Gb fasta needs about 200G RAM, so 1000G RAM should be enough. Could you have a look at the log file of linux? It should be at the file like: /var/log/messages. That may tell why it was killed.

[DongnaMa]

I'm sorry, I don't have permission to view the files under this path, why don't I have log file output? I also assembled it with another linux server, didn't killed, but it's been running for a week without any file output, is that normal?

[chhylp123]

Could you please check the status of the running thread on another server? I worry hifiasm didn't successfully run on that server. If there is no log, that means hifiasm stop at k-mer counting step, which is abnormal...

[DongnaMa]

It's weird, I rerun it again and still no file output.

[chhylp123]

Sorry, what do you mean by no file output? For 200G fasta, the k-mer counting may need several hours. So it shouldn't have any output to stderr in the first several hours.

[lh3]

@DongnaMa What is the exact command line you are using? How many threads are you specifying?

[DongnaMa]

At first, my run command: hifiasm -z 50 -o THS.asm -t 20 ../1_data/THS_pacbio.fasta and ran for a week without any files output. I'm running commands on another server：hifiasm -z 50 -o THS.asm -t 60 THS_pacbio.fasta , I tried different threads parameters including 10，30, 100, and run for about a few hours and get killed.

[HenrivdGeest]

My fasta sequences are 61Gbyte and hifiasm used 179Gbyte ram as maximum:
User time (seconds): 1420965.92
System time (seconds): 11862.12
Percent of CPU this job got: 13456%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:57:27
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 179872840
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 615657759
Voluntary context switches: 133891585
Involuntary context switches: 150357957
Swaps: 0
File system inputs: 72
File system outputs: 68024272
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Which means fastaSize3. Nonetheless 2003 is less that 1TByte. I would check for dmesg to see if the OOM killer killed you program.

Just on a side note, it would be handy in this case that hifiasm starts to write to the log what is is doing already from the start. If you have no log file at all, you start to suspect that something else went completely wrong, I had a similar experience once.

[lh3]

PacBio assembled the 26Gb redwood genome at ~33X using 710GB memory at the peak, so 1TB should be ok. @DongnaMa are you using a job scheduler such as Slurm/LSF/SGE? Also, are you using hifi data?

it would be handy in this case that hifiasm starts to write to the log what is is doing already from the start.

That makes sense. Thanks for the suggestion.

[DongnaMa]

My server is SGE type, but I'm running the program in the largest node not using a job scheduler. I only used PacBio data and didn't use any other data.

[DongnaMa]

I'm guessing that my 1TB memory may have set a limit，so it got killed, Is there any way to limit the highest memory used by the program？

[lh3]

By "PacBio data", do you mean CLR (noisy reads) or HiFi/CCS (accurate reads)? Note that hifiasm only works with HiFi data, not CLR.

[DongnaMa]

It's a CCS type.

[DongnaMa]

Sorry, I made sure, It's a CLR type, it doesn't fit, thanks guys.

[chhylp123]

np, have a good night.