nextomics/nextdenovo

Fast and accurate de novo assembler for long reads

nextdenovo's Issues

processes have been killed by the cgroup out-of-memory handler.

When I run the pipeline for a 206.442627 input fastq file, the progress was struck at step /01.raw_align/02.raw_align.sh.work/. The error reported was "slurmstepd: error: Detected 1 oom-kill event(s) in step 1481924.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
".
My run.cfg file is
[General]
job_type = slurm # here we use SGE to manage jobs
job_prefix = nextDenovo
task = all # 'all', 'correct', 'assemble'
rewrite = yes # yes/no
deltmp = yes
rerun = 3
parallel_jobs = 22
input_type = raw
input_fofn = /group/pasture/Saila/NextDenovo/smartdenovo.input.fofn # input file
workdir = /group/pasture/Saila/NextDenovo

[correct_option]
read_cuoff = 1k
seed_cutoff = 20000 # the recommended minimum seed length
blocksize = 5g
pa_correction = 5
seed_cutfiles = 5
sort_options = -m 50g -t 30 -k 50
minimap2_options_raw = -x ava-ont -t 8
correction_options = -p 30
cluster_options = --cpus-per-task={cpu} --mem-per-cpu={vf}

[assemble_option]
random_round = 100
minimap2_options_cns = -x ava-ont -t 8 -k17 -w17
nextgraph_options = -a 1

I am not sure if I need to increase the memory and if its the case can you please suggest how much I need to ?
My cluster is slurm.

Thanks
S

ovl_sort.c:688: main: Assertion `opts.max_mem > 100000000' failed

I am trying to use nextdenovo to assembly the data of a fungi. but the following problem occurs. what is it about and how could I solve it ?
ovl_sort: ovl_sort.c:688: main: Assertion `opts.max_mem > 100000000' failed.
Command terminated by signal 6

Questions about "seed_cutoff" option

Hi, Dr. Hu, thanks for your excellent work at NextOmics. I have a few questions about the "seed_cutoff" option and I would appreciate it very much if you could help me:

1. If I set "read_cutoff" to 1000 and "seed_cutoff" to 1001, will all reads that are longer than 1001 bp be corrected?
2. If I have enough CPUs, should I correct all the reads longer than 1kb I have on my hand?
3. I did a small test on the "seed_cutoff" option. The 1st time I set seed_cutoff = 13k, and the best one of the 100 results is: assembly_size = 550Mb, contigN50 = 1.1Mb. The 2nd time I set seed_cutoff = 1001 to correct all the reads and the best result is: assembly_size = 665Mb, contigN50 = 940kb. The genome size estimation(by kmerfreq) is about 1Gb and we only have ~20Gb nanopore data, by the way. Do you think the difference between the assembly sizes(550Mb vs. 665Mb) is because of the different seed_cutoff values? If so, how should I decide how much data to use during the reads correction stage?
   Thanks in advance!
   YU Jin.

self.drmaa.exit() stuck

jobs have been finished normally but the program was stuck at the stage of self.drmaa.exit() in script task_control.py.

how can I solve this?

Segmentation fault

Hi,
I run nextDenovo in a local machine, and get an error in final assembly step,
............
[INFO] 2019-10-21 16:05:15,963 cns_align done
[INFO] 2019-10-21 16:05:15,968 analysis tasks done
[INFO] 2019-10-21 16:05:15,974 total jobs: 11
[INFO] 2019-10-21 16:05:15,976 Throw jobID:[43608] jobCmd:[/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00/nextDenovo.sh] in the local_cycle.
..................
..................
..................
[ERROR] 2019-10-21 16:07:50,953 ctg_graph failed: please check the following logs:
[ERROR] 2019-10-21 16:07:50,954./01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00/nextDenovo.sh.e

the content of nextDenovo.sh.e file:

hostname

• hostname
  cd ./01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00
• cd ./01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00
  time ~/NextDenovo/bin/nextgraph -a 1 -f ./01_rundir/03.ctg_graph/01.ctg_graph.input.seqs ./01_rundir/03.ctg_graph/01.ctg_graph.input.ovls -o nextgraph.assembly.contig.fasta;
• ~/NextDenovo/bin/nextgraph -a 1 -f ./01_rundir/03.ctg_graph/01.ctg_graph.input.seqs ./01_rundir/03.ctg_graph/01.ctg_graph.input.ovls -o nextgraph.assembly.contig.fasta
  [INFO] 2019-10-21 16:05:15 Initialize graph and reading...
  [INFO] 2019-10-21 16:06:07 Initial Node(s): 245385, Edge(s): 2928398
  [INFO] 2019-10-21 16:06:08 Depth stat, Ave: 48.591 Max: 97182.656 Repeat: 82.710 Nor:Ext: 435534:58696
  [INFO] 2019-10-21 16:06:08 Outdegree stat, Ave: 11.974 Max: 23947.166 Repeat: 18.166 Nor:Ext: 485220:5550
  [INFO] 2019-10-21 16:06:09 Chimeric node ratio: 0.150% (candidate: 0.449%)
  [INFO] 2019-10-21 16:06:16 Assembly done and outputting...
  ./01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00/nextDenovo.sh: line 5: 43612 Segmentation fault (core dumped) ~/NextDenovo/bin/nextgraph -a 1 -f ./01_rundir/03
  .ctg_graph/01.ctg_graph.input.seqs ./01_rundir/03.ctg_graph/01.ctg_graph.input.ovls -o nextgraph.assembly.contig.fasta

the size of nextgraph.assembly.contig.fasta is about 32Mb.

This is my run.cfg file:
[General]
job_type = local # here we use SGE to manage jobs
job_prefix = nextDenovo
task = all # 'all', 'correct', 'assemble'
rewrite = yes # yes/no
deltmp = no
rerun = 3
parallel_jobs = 30
input_type = raw
input_fofn = ./input.fofn # input file
workdir = ./01_rundir
#cluster_options = -l vf={vf} -q all.q -pe smp {cpu} -S {bash} -w n

[correct_option]
read_cuoff = 1k
seed_cutoff = 19703 # the recommended minimum seed length
blocksize = 10g
pa_correction = 50
#seed_cutfiles = 5
sort_options = -m 30g -t 5 -k 40
minimap2_options_raw = -x ava-pb -t 8
correction_options = -p 5

[assemble_option]
random_round = 100 #same error if commented out
minimap2_options_cns = -x ava-pb -t 8 -k17 -w17
nextgraph_options = -a 1

What should I do? Thank you!

how to restart the whole pipeline when some tasks is failed?

i run NextDenovo in lsf system, and some ovl_sort task are failed, how to restart the pipeline from the failed ovl_sort other than from beging?

how to select a prefect results

I assembly about 100X ONT (raw data) using NextDenovo. And my genome is about 1G.

when I used default parameter the result is following.

• /ifs1/User/shehongbing/soft/NextDenovo/bin/nextgraph -a 1 -f /ifs1/User/shehongbing/zhuang/next/merge/merge_01/03.ctg_graph/01.ctg_graph.input.seqs /ifs1/User/shehongbing/zhuang/next/merge/merge_01/03.ctg_graph/01.ctg_graph.input.ovls -o nextgraph.assembly.contig.fasta
  [INFO] 2019-12-30 02:54:05 Initialize graph and reading...
  [INFO] 2019-12-30 02:55:11 Initial Node(s): 175253, Edge(s): 2333852
  [INFO] 2019-12-30 02:55:13 Depth stat, Ave: 74.823 Max: 149645.938 Repeat: 155.362 Nor:Ext: 254094:97640
  [INFO] 2019-12-30 02:55:13 Outdegree stat, Ave: 13.449 Max: 26898.375 Repeat: 20.755 Nor:Ext: 340684:9823
  [INFO] 2019-12-30 02:55:26 Chimeric node ratio: 0.930% (candidate: 1.353%)
  [INFO] 2019-12-30 02:55:40 Assembly done and outputting...
  [INFO] 2019-12-30 03:01:39 Assembly stat:
  Type Length (bp) Count (#)
  N10 116204120 1
  N20 114308081 2
  N30 107094691 3
  N40 31652831 5
  N50 7783900 12
  N60 2368060 41
  N70 1638194 97
  N80 1054077 184
  N90 683091 316

Min. 49014 -
Max. 116204120 -
Ave. 1775606 -
Total 1099099761 619

also, I noticed that a max N50 using the random parameter is following.

• /ifs1/User/shehongbing/soft/NextDenovo/bin/nextgraph -a 0 -u 1 -f /ifs1/User/shehongbing/zhuang/next/merge/merge_01/03.ctg_graph/01.ctg_graph.input.seqs /ifs1/User/shehongbing/zhuang/next/merge/merge_01/03.ctg_graph/01.ctg_graph.input.ovls
  [INFO] 2019-12-30 02:55:38 Initialize graph and reading...
  [INFO] 2019-12-30 02:56:13 Initial Node(s): 98576, Edge(s): 747728
  [INFO] 2019-12-30 02:56:13 Depth stat, Ave: 73.900 Max: 147799.938 Repeat: 152.905 Nor:Ext: 144616:54866
  [INFO] 2019-12-30 02:56:13 Outdegree stat, Ave: 7.809 Max: 15618.035 Repeat: 12.303 Nor:Ext: 187703:9450
  [INFO] 2019-12-30 02:56:16 Chimeric node ratio: 1.520% (candidate: 2.085%)
  [INFO] 2019-12-30 02:56:18 Assembly done and outputting...
  [INFO] 2019-12-30 02:56:18 Assembly stat:
  Type Length (bp) Count (#)
  N10 102599639 1
  N20 79963541 3
  N30 45136607 4
  N40 33626804 7
  N50 24232552 11
  N60 10614944 17
  N70 4181683 33
  N80 2435309 65
  N90 1002506 132

Min. 109350 -
Max. 102599639 -
Ave. 3141996 -
Total 999154593 318

we could noticed that these two parameter seems the same except -u, but the N50 value is total different. So I wanna know which I could choose.

Thx

Problem: ValueError: zero length field name in format

Hi,

Once I run nextdenovo, I met the problem "ValueError: zero length field name in format".
Traceback (most recent call last):
File "../nextDenovo", line 394, in
main(args)
File "../nextDenovo", line 278, in main
task.set_subtasks(job_prefix = job_prefix)
File "../NextDenovo/lib/taskControl.py", line 128, in set_subtasks
subtask_dir = self.prefix + '{:0>{}}'.format(i, len(str(task_count)))
ValueError: zero length field name in format

Could you help me with the problem?
Thank you!

Assemble the genome with corrected fasta using NextDenovo2.1

HI,
I am trying to assemble the genome with corrected fasta using NextDenovo2.1.
I have set 'task = assemble', 'input_type = corrected' and 'input.fofn' containing corrected fasta files but something is wrong.
The input fasta files were generated by NextDenovo1 and have been compressed and moved to another place. How could I just running the assemble step with the corrected data?
Thank you!

How to get genome assembly larger than 3.5G?

Hi Hu:

Here is part of the log from graph step:
Min. 22857 - Max. 8011581 - Ave. 402903 - Total 4973432366 12344 [WARNING] 2019-11-19 10:28:10 Unfinished assembly, this is a limited version, currently only supports assembly for genome size < 3500000000 bp, please ask for help.
May I ask is there ways to get genome assembly larger than 3.5G?

Thanks!

Problem with NextDenovo

Hi,

I got some problem when running NextDenovo test example

My command line is ./nextDenovo test_data/run.cfg -l log.txt

run.cfg files is shown below:

[General]
job_type = local
job_prefix = nextDenovo
task = all
rewrite = yes # yes/no
input_type = raw
input_fofn = ./input.fofn
workdir = ../01_rundir
sge_options = -l vf={vf} -q dev.q -pe smp {cpu}

[correct_option]
read_cuoff = 1k
seed_cutoff = 25k
blocksize = 100m
pa_raw_align = 5
pa_correction = 5
seed_cutfiles = 3
sort_options = -m 5g -t 2 -k 50
minimap2_options = -x ava-ont -t 8
correction_options = -p 15

And I got error log like this:

[INFO] 2019-03-07 22:19:14,163 start...
[INFO] 2019-03-07 22:19:14,163 logfile: pid2085.log.txt
[WARNING] 2019-03-07 22:19:14,167 Re-write workdir
[INFO] 2019-03-07 22:19:14,167 options: 
[INFO] 2019-03-07 22:19:14,167 {'sort_threads': 2, 'sge_options': '-l vf={vf} -q dev.q -pe smp {cpu}', 'rewrite': 1, 'blocksize': '100m', 'job_prefix': 'nextDenovo', 'job_type': 'local', 'cns_threads': 15, 'read_cuoff': '1k', 'seed_cutoff': '25000', 'input_fofn': '/public1/home/windz/software/NextDenovo/test_data/./input.fofn', 'input_type': 'raw', 'sort_options': '-m 5g -t 2 -k 50', 'sort_mem': '5g', 'sge_queue': ['dev.q'], 'ctg_graphdir': '/public1/home/windz/software/NextDenovo/test_data/../01_rundir/03.ctg_graph', 'pa_correction': '5', 'pa_raw_align': '5', 'minimap2_options': '-x ava-ont -t 8', 'workdir': '/public1/home/windz/software/NextDenovo/test_data/../01_rundir', 'minimap2_threads': 8, 'cns_aligndir': '/public1/home/windz/software/NextDenovo/test_data/../01_rundir/02.cns_align', 'seed_cutfiles': '5', 'raw_aligndir': '/public1/home/windz/software/NextDenovo/test_data/../01_rundir/01.raw_align', 'task': 'all', 'rerun': 3, 'correction_options': '-p 15 -max_lq_length 10000'}
[INFO] 2019-03-07 22:19:14,169 mkdir: /public1/home/windz/software/NextDenovo/test_data/../01_rundir
[INFO] 2019-03-07 22:19:14,172 mkdir: /public1/home/windz/software/NextDenovo/test_data/../01_rundir/01.raw_align
[INFO] 2019-03-07 22:19:14,174 mkdir: /public1/home/windz/software/NextDenovo/test_data/../01_rundir/02.cns_align
[INFO] 2019-03-07 22:19:14,177 mkdir: /public1/home/windz/software/NextDenovo/test_data/../01_rundir/03.ctg_graph
[INFO] 2019-03-07 22:19:14,186 analysis tasks done
[INFO] 2019-03-07 22:19:14,196 total jobs: 1
[INFO] 2019-03-07 22:19:14,197 Throw jobID:[2107] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:15,927 db_split done
[INFO] 2019-03-07 22:19:15,932 analysis tasks done
[INFO] 2019-03-07 22:19:16,052 total jobs: 20
[INFO] 2019-03-07 22:19:16,053 Throw jobID:[2146] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align00/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:16,554 Throw jobID:[2170] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align01/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:17,056 Throw jobID:[2194] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align02/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:17,557 Throw jobID:[2213] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align03/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:18,058 Throw jobID:[2234] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align04/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:18,560 Throw jobID:[2292] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align05/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:19,061 Throw jobID:[2315] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align06/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:19,563 Throw jobID:[2338] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align07/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:20,065 Throw jobID:[2342] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align08/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:20,566 Throw jobID:[2363] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align09/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:21,067 Throw jobID:[2404] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align10/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:21,569 Throw jobID:[2424] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align11/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:22,070 Throw jobID:[2447] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align12/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:22,572 Throw jobID:[2472] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align13/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:23,073 Throw jobID:[2506] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align14/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:23,575 Throw jobID:[2529] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align15/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:24,076 Throw jobID:[2549] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align16/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:24,578 Throw jobID:[2570] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align17/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:25,079 Throw jobID:[2591] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align18/nextDenovo.sh] in the local_cycle.
[INFO] 2019-03-07 22:19:25,581 Throw jobID:[2620] jobCmd:[/public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align19/nextDenovo.sh] in the local_cycle.
[ERROR] 2019-03-07 22:19:26,103 raw_align failed: please check the following jobs:
[ERROR] 2019-03-07 22:19:26,103 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align00/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,103 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align01/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,103 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align02/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,103 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align03/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,103 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align04/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align05/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align06/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align07/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align08/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align09/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align10/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align11/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align12/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align13/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,104 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align14/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,105 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align15/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,105 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align16/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,105 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align17/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,105 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align18/nextDenovo.sh
[ERROR] 2019-03-07 22:19:26,105 /public1/home/windz/software/NextDenovo/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align19/nextDenovo.sh

Can you tell me how to figure out? Thanks!

Cheers,
Windz

Ask help for software usage

Dear Hu,
Thanks for your update of nextDoneov v2.0.
However, I'm not sure if I need to change the parameter in demo file run.cfg when I run test data. Because there are some errors when I run test data based on demo run.cfg file.
After check final results step by step, there were two files listed in 01.ctg_graph.input.seqs. And I got the empty result file nextgraph.assembly.contig.fasta run bellow comment:
"nextgraph -a 1 -f 01.ctg_graph.input.seqs 01.ctg_graph.input.ovls -o nextgraph.assembly.contig.fasta"
I'm not sure which step emerged error. And why I couldn't run test data successfuly. Please the process file and log file as bellow.
Please help me debug this problem. Many thanks for your help!
Best wishes.

how can I reduce the number of process and storage at raw align step？

Hi ,

I want to use the nextdenovo to assembly PB reads. And I run it by SGE. I notice that the submissions of step2 raw reads align require lots of resources and processes. That led to those tasks that can't be run. So the question is how can I reduce the number of processes and storage at raw align step？ Hope your help. Thanks a lot.

Raw data or corrected data ??

Hi,
Dr. Hu, raw data and corrected data (corrected by canu), which is better for NextDenovo?

get_cns error

HI,
I got errors when running get_cns step, the errors are listed below.
The 'segmentation fault' also happened at sort_align stage, but disappeared when I rerun the wrong tasks. However, things didn't get well when I rerun the jobs at get_cns stage.
Could you help me with the problems?
Thank you!

ERROR 1:
/home/SystemSoftware/tsce/torque6/share/nodes1/mom_priv/jobs/1270566.mu01.SC: line 5: 642 Segmentation fault (core dumped) python /1.Software/NextDenovo/lib/nextCorrector.py -f /nextdenovo/01.correct/.//02.cns_align//01.get_cns.input.idxs -i /nextdenovo/01.correct/01.raw_align/03.sort_align.sh.work/sort_align006/input.seed.088.sorted.ovl -p 20 -max_lq_length 1000 -fast -o cns.fasta

ERROR 2:
Traceback (most recent call last):
File "/1.Software/NextDenovo/lib/nextCorrector.py", line 258, in
main(args)
File "/1.Software/NextDenovo/lib/nextCorrector.py", line 198, in main
worker, read_seq_data(args, corrected_seeds), chunksize=1):
File "/1.Software/python_lib/nextdenovo/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
SystemError: NULL result without error in PyObject_Call

ERROR 3:
*** Error in `python': double free or corruption (out): 0x00002b70bbe79010 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7cfe1)[0x2b6e4bdb0fe1]
/1.Software/NextDenovo/lib/ovlSeq.so(bit2seq+0x1f7)[0x2b6e54d0e867]
/1.Software/python_lib/nextdenovo/lib/python2.7/lib-dynload/_ctypes.so(ffi_call_unix64+0x4c)[0x2b6e548fb35c]
/1.Software/python_lib/nextdenovo/lib/python2.7/lib-dynload/_ctypes.so(ffi_call+0x1f5)[0x2b6e548faab5]
/1.Software/python_lib/nextdenovo/lib/python2.7/lib-dynload/_ctypes.so(_ctypes_callproc+0x3e6)[0x2b6e548f2166]
/1.Software/python_lib/nextdenovo/lib/python2.7/lib-dynload/_ctypes.so(+0x9cf3)[0x2b6e548e9cf3]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x2b6e4b071d23]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6a24)[0x2b6e4b122f54]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x6dcec)[0x2b6e4b095cec]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x686cd)[0x2b6e4b0906cd]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2e1f)[0x2b6e4b11f34f]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x6dcec)[0x2b6e4b095cec]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x686cd)[0x2b6e4b0906cd]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2e1f)[0x2b6e4b11f34f]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x2b6e4b125a2e]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x794a8)[0x2b6e4b0a14a8]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x2b6e4b071d23]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6267)[0x2b6e4b122797]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8665)[0x2b6e4b124b95]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8665)[0x2b6e4b124b95]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x2b6e4b125a2e]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x793a1)[0x2b6e4b0a13a1]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x2b6e4b071d23]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x5c4bf)[0x2b6e4b0844bf]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53)[0x2b6e4b071d23]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x2b6e4b11b633]
/1.Software/python_lib/nextdenovo/bin/../lib/libpython2.7.so.1.0(+0x135d42)[0x2b6e4b15dd42]
/lib64/libpthread.so.0(+0x7dc5)[0x2b6e4b416dc5]
/lib64/libc.so.6(clone+0x6d)[0x2b6e4be2a1cd]
======= Memory map: ========
......
......
/home/SystemSoftware/tsce/torque6/share/nodes2/mom_priv/jobs/1270586.mu01.SC: line 5: 25161 Aborted (core dumped) python /1.Software/NextDenovo/lib/nextCorrector.py -f /nextdenovo/01.correct/.//02.cns_align//01.get_cns.input.idxs -i /nextdenovo/01.correct/01.raw_align/03.sort_align.sh.work/sort_align026/input.seed.021.sorted.ovl -p 20 -max_lq_length 1000 -fast -o cns.fasta

A problem I met when running NextDenovo

Hi!
I ran the test example(the command is nohup nextDenovo test/run.cfg & )
Here is run.cfg file:

[General]
job_type = local
job_prefix = nextDenovo
task = all # 'all', 'correct', 'assemble'
rewrite = yes # yes/no
deltmp = yes
rerun = 3
parallel_jobs = 2
input_type = raw
input_fofn = ./input.fofn
workdir = ./01_rundir

[correct_option]
read_cutoff = 1k
seed_cutoff = 32k
blocksize = 1g
pa_correction = 2
seed_cutfiles = 2
sort_options = -m 1g -t 2 -k 50
minimap2_options_raw = -x ava-ont -t 8
correction_options = -p 15

[assemble_option]
random_round = 10
minimap2_options_cns = -x ava-ont -t 8 -k17 -w17
nextgraph_options = -a 1

But I got error like this:

[INFO] 2019-10-21 17:04:16,871 start...
[INFO] 2019-10-21 17:04:16,871 logfile: pid111095.log.info
[WARNING] 2019-10-21 17:04:16,872 Re-write workdir
[INFO] 2019-10-21 17:04:16,872 options: 
[INFO] 2019-10-21 17:04:16,873 {'sort_threads': 2, 'nodelist': '', 'rewrite': 1, 'blocksize': '1g', 'job_prefix': 'nextDenovo', 'job_type': 'local', 'minimap2_options_raw': '-x ava-ont -t 8', 'cns_threads': 15, 'sort_mem': '1g', 'seed_cutoff': '32000', 'input_fofn': '/h
[INFO] 2019-10-21 17:04:16,873 skip mkdir: /home/lailab/wangzijian/software/NextDenovo/test_data/./01_rundir
[INFO] 2019-10-21 17:04:16,873 skip mkdir: /home/lailab/wangzijian/software/NextDenovo/test_data/./01_rundir/01.raw_align
[INFO] 2019-10-21 17:04:16,873 skip mkdir: /home/lailab/wangzijian/software/NextDenovo/test_data/./01_rundir/02.cns_align
[INFO] 2019-10-21 17:04:16,873 skip mkdir: /home/lailab/wangzijian/software/NextDenovo/test_data/./01_rundir/03.ctg_graph
[INFO] 2019-10-21 17:04:16,874 analysis tasks done
[INFO] 2019-10-21 17:04:16,874 total jobs: 1
[INFO] 2019-10-21 17:04:16,876 Throw jobID:[111096] jobCmd:[/home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh] in the local_cycle.
[ERROR] 2019-10-21 17:04:17,400 db_split failed: please check the following logs:
[ERROR] 2019-10-21 17:04:17,400 /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh.e

Then I checked /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh.e to find out what is the error , which is shown below:

hostname
+ hostname
cd /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0
+ cd /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0
time /home/lailab/wangzijian/software/NextDenovo/bin/seq_dump -f 1k -s 32000 -b 1g -n 2 -d /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align /home/lailab/wangzijian/software/NextDenovo/test_data/input.fofn
+ time /home/lailab/wangzijian/software/NextDenovo/bin/seq_dump -f 1k -s 32000 -b 1g -n 2 -d /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align /home/lailab/wangzijian/software/NextDenovo/test_data/input.fofn
/home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh: 5: /home/lailab/wangzijian/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh: time: not found 

Could you tell me how to deal with it? Thanks a lot!

What is the memory requirement for nodes when run a huge input fastq?

Hi, Hu

When I run the pipeline for a 2.8Tb input fasta file, the progress was struck at step 02.cns_align/01.get_cns.sh.work, it reported report an error out-of-memory. how can I carry on the job, or need larger memory nodes?
My cluster is slurm and the node memory is 192GB and 36 cpus.

Blew is the configure file details
[General]
job_type = slurm
job_prefix = Pp
task = all # 'all', 'correct', 'assemble'
rewrite = yes # yes/no
deltmp = no
rerun = 3
parallel_jobs = 50
input_type = raw
input_fofn = ./input.fofn
workdir = ./01_rundir
#usetempdir = /tmp/test
#nodelist = avanode.list.fofn

cluster_options = -p q_cn -J nextDenovo -o nextDenovo.out -N 1 -n 1 -c 19
[correct_option]
read_cutoff = 1k
seed_cutoff = 15k
blocksize = 1g
pa_correction = 50
seed_cutfiles = 50
sort_options = -m 20g -t 20 -k 50
minimap2_options_raw = -x ava-ont -t 30
correction_options = -p 15

[assemble_option]
random_round = 10
minimap2_options_cns = -x ava-ont -t 30 -k17 -w17
nextgraph_options = -a 1

Add GFA in output

Many assembler output GFA, such as Canu, miniasm.

This file can be visualized in Bandage for check the assembly.

Could you add the this format output in future development?

assemble result

when I used raw Nanopore data, the assemble.fa N50 is about 39 Mb, but the alignment is not good in comparison to published genome. however, when I used collected data (corrected by canu), the assemble.fa N50 is about 2 Mb, and with the good alignment in comparison to the polished genome. So I do not know why. it suggested that should I used the corrected data rather than raw data? and the two methods with the huge different in N50

The error of the seed_cutoff

Hi,
when I used the seq_stat to analysis my data, but the results equal to 0 bp

*Suggested length cutoff of reads (genome size: 980000000, expected seed depth: 45) to be corrected: 0 bp

problem:The data from canu's correctedReads had no results with the nextdenovo software.

Hi, I have a problem about the nextdenovo assemble task. when I used the data from canu's correctedReads to assemble, I couldn,t get any results .
run.cfg:
[General]
job_type = sge
job_prefix = ***
task =assemble # 'all', 'correct', 'assemble'
rewrite = yes # yes/no
deltmp = yes
rerun = 3
parallel_jobs = 200
input_type = corrected
......
[assemble_option]
random_round = 100
minimap2_options_cns = -x ava-pb -t 8 -k17 -w17
nextgraph_options = -a 1

Please give me some advises to solve this problem.
Thanks a lot !!!

there is not enough disk space

HI,
I have 80G raw data and my disk space is 2T. Actually, it not enough disk space for me, so what should I do?

Cann't get the cns.fasta

HI,
there are 20 get_cns** directory but only in get_cns01 directory that I can not get the cns.fasta. and the nextDenovo.sh.e shown is following:

/ifs1/User/shehongbing/zhuang/next/n02/01_rundir_yy/02.cns_align/01.get_cns.sh.work/get_cns01/nextDenovo.sh: 行 5: 448178 段错误 python /ifs1/User/shehongbing/soft/NextDenov o/lib/nextcorrect.py -f /ifs1/User/shehongbing/zhuang/next/n02/01_rundir_yy/02.cns_align//01.get_cns.input.idxs -i /ifs1/User/shehongbing/zhuang/next/n02/01_rundir_yy/01.raw_align/03.sort _align.sh.work/sort_align01/input.seed.002.sorted.ovl -p 7 -max_lq_length 10000 -o cns.fasta

but I compare the nextDenove.sh with others in get_cns03 directory, actually, its no mistakes at all.

about the bug in nextdenovo2.0

Dear Hu,
Thanks for your excellent work in denovo genome assembly of nextdenovo-2.0 and it truly helped us a lot in our project; but i found a bug while running another genome assembly in nextdenovo-2.0

I have finished step 1 of raw align ; the main program was at substep 1, get cns, of step 2 , cns align, and several sub tasks were submitted into running; most of them(26/30) had been normally finished, but some (4/30) were trapped into sleeping state; while I checked the results, I found that the size of cns.fasta was basically the same as other finished ones, and the total number of sequence was 9584, but the corresponding index was imcomplete with only 9490 lines representing the sequence information and the last line had only a number "2" which was the begining of the sequence id;
unfortunately, while I killed the main program and manually submitted the trapped subtask, it still could not normally finish and the results were exactly the same as the one I described before.

Could you please help me debug this problem? Thanks a lot, and looking forward to your reply!

Bestwishes!

jobs in step 02.cns_align/02.cns_align.sh.work runs extremely slow

Hi:

I'm using nextdenovo to assemble a highly repetitive plant genome. It turns out that jobs in step 02.cns_align/02.cns_align.sh.work runs extremely slow. It takes about 1 hours for finishing one single job and I got 9000 jobs in total. Here is the setting I'm using minimap2-nd -I 6G --step 2 --dual=yes -x ava-pb -t 32 -k17 -w17, and the memory allocated for each job is 120 Gb. The target and query fasta file is about 500 Mb.
Here is the log file:
`[M::mm_idx_gen::11.8761.35] collected minimizers
[M::mm_idx_gen::13.3421.78] sorted minimizers
[M::main::13.3421.78] loaded/built the index for 23552 target sequence(s)
[M::mm_mapopt_update::13.4731.78] mid_occ = 2280
[M::mm_idx_stat] kmer size: 17; skip: 17; is_hpc: 1; #seq: 23552
[M::mm_idx_stat::13.5611.77] distinct minimizers: 5972457 (46.92% are singletons); average occurrences: 6.770; average spacing: 12.505
[M::worker_pipeline::3513.41728.06] mapped 23340 sequences
[M::worker_pipeline::3822.019*25.92] mapped 210 sequences
[M::main] Version: 2.17-r941
[M::main] CMD: /nextomics/NextDenovo2.1/NextDenovo/bin/minimap2-nd -I 6G --step 2 --dual=yes -x ava-pb -t 32 -k17 -w17 -o cns.filt.dovt.ovl /nextdenovotest/02.cns_align/01.get_cns.sh.work/get_cns002/cns.fasta /nextdenovotest/02.cns_align/01.get_cns.sh.work/get_cns027/cns.fasta
[M::main] Real time: 3822.115 sec; CPU: 99073.192 sec; Peak RSS: 23.554 GB

real 63m42.143s
user 1648m22.115s
sys 2m51.097s`

Is there ways to speedy up this step?

Thanks in advance!

python3 ?

Interesting software, but python2 ? End of life in less than 6 months ?

Is a port to python3 in progress ?

cheers!

nextgraph: asg.c:1270: update_graph: Assertion `mode == 10' failed.

Hi! I get the following error in "01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00/nextDenovo.sh.e":
"nextgraph: asg.c:1270: update_graph: Assertion `mode == 10' failed.
01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph01/nextDenovo.sh: line 5: 208229 Aborted (core dumped) /apps/NextDenovo/bin/nextgraph -a 0 -n 45 -f 01_rundir/03.ctg_graph/01.ctg_graph.input.seqs 01_rundir/03.ctg_graph/01.ctg_graph.input.ovls". What should I do to solve it? Thanks!

analysis tasks done but the following pipeline died (KeyError: '009')

Hi, Dr,Hu
Recently, I use nextDenovo to assembly a plant genome. It is extremely fast doing reads alignment. But when this step is done, the pipeline died and throwed the follwing error:
[INFO] 2019-12-17 10:15:14,462 analysis tasks done
[INFO] 2019-12-17 10:15:14,463 skip step: db_split
[INFO] 2019-12-17 10:15:14,485 analysis tasks done
[INFO] 2019-12-17 10:15:14,487 skip step: raw_align
[INFO] 2019-12-17 10:15:14,554 analysis tasks done
Traceback (most recent call last):
File "/home/software/bin/nextDenovo", line 518, in
main(args)
File "/home/software/bin/nextDenovo", line 410, in main
pre_sort_align_input(sort_align_input, task.run.unfinished_tasks)
File "/home/software/bin/nextDenovo", line 153, in pre_sort_align_input
for input_file in sort_align_input[seed_idx]:
KeyError: '009'

I also rerun the pipeline, the same error came again. Any suggestion?

Thanks.

Parallel job parameter of NextDenovo when run in local mode

Hi,

Could you tell me what is the relationship among the following parallel parameters

• paralle_job
• pa_correction
• minimap2_options_raw
• sort_options
• correction_options
• minimap2_options_cns

In the default parameters(parallel_jobs = 10, pa_correction = 15, minimap2_options_raw = -x ava-ont -t 10 ), would the nextDenovo use 10 * 15 *10 = 1500 CPU or 15 * 10 = 150 in correction stage.

only few reads were left

Hi, I try to correct the nanopore reads by nextdenovo, I have about 20G rawdata and lest only 2G corrected reads. Is it normal？ I have used -fast for speeding up.

Thanks for your help~

how to run on LSF system?

Hi! I want to use NextDenovo on my LSF system, but when I set "lsf" on " job_type" of "run.cfg", the system get the following error: drmaa.errors.InternalException: code 1: Please set the environment variable SGE_ROOT.
So my questions are:

1. Does NextDenovo work on lsf system？how to set run.cfg (especially "cluster_options" )?
2. how to set "SGE_ROOT" or somethings like "LSF_ROOT" of drmaa-python in LSF system? Does drmaa-python work on lsf system? And I found a another tool named lsf-drmaa? Does lsf-drmaa work on NextDenovo? what is the the relationship between drmaa-python and lsf-drmaa？

minimap2 option "-I"

Hej,

In nextDenovo, lines 165-171 contain hard-coded chunk size -I 3G:

			if part_file != seed_file:
				cmd += SCRIPT_PATH + '/bin/minimap2-hack -I 3G --dual=yes ' + cfg['minimap2_options'] + ' ' + seed_file  + ' ' + part_file + \
					' > ' + os.path.basename(seed_file) + '.' +  str(k) + '.ovl;'
				cmd += 'ln -sf ' + os.path.basename(seed_file) + '.' +  str(k) + '.ovl ' + os.path.basename(part_file) + '.' +  str(k) + '.ovl;'
			else:
				cmd += SCRIPT_PATH + '/bin/minimap2-hack -I 3G ' + cfg['minimap2_options'] + ' ' + seed_file  + ' ' + part_file + \
					' > ' + os.path.basename(seed_file) + '.' +  str(k) + '.ovl;'

If one adds additional minimap2 options, the command results in two "-I" flags. Not sure how minimap2 interprets such situation.

Is there any reason for 3G chunks, or can it be just removed?

Cheers,
Iggy

NextDenovo assemble has no result file

My problem is that my process runs out, but there are no final assembled result,There ais only 01_rundir/02.cns_align/01.get_cns.sh.work/get_cns0/cns.fasta.

my run.cfg is:
[General]
job_type = local
job_prefix = nextDenovo
task = all
rewrite = yes # yes/no
input_type = raw
input_fofn = ./input.fofn
workdir = ./nb_assmble

[correct_option]
read_cuoff = 1k
seed_cutoff = 32k
blocksize = 1g
pa_raw_align = 2
pa_correction = 2
seed_cutfiles = 2
sort_options = -m 1g -t 2 -k 50
minimap2_options = -x ava-pb -t 8
correction_options = -p 15

[assemble_option]
random_round = 20
minimap2_options_cns = -x ava-pb -t 8 -k17 -w17
nextgraph_options = -a 1

The final content of the report file:
[INFO] 2019-11-07 10:18:43,231 remove temporary result: /share/microlab/student/ruanyingfei/biosoft/NextDenovo/test_data/nb_assmble/01.raw_align/02.raw_align.sh.work/raw_align18/input.seed.002.2bit.18.ovl
[INFO] 2019-11-07 10:18:43,352 sort_align done
[INFO] 2019-11-07 10:18:43,387 analysis tasks done
[INFO] 2019-11-07 10:18:43,544 total jobs: 2
[INFO] 2019-11-07 10:18:43,545 Throw jobID:[4798] jobCmd:[/share/microlab/student/ruanyingfei/biosoft/NextDenovo/test_data/nb_assmble/02.cns_align/01.get_cns.sh.work/get_cns0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-11-07 10:18:44,079 Throw jobID:[4821] jobCmd:[/share/microlab/student/ruanyingfei/biosoft/NextDenovo/test_data/nb_assmble/02.cns_align/01.get_cns.sh.work/get_cns1/nextDenovo.sh] in the local_cycle.

Looking forward to your valuable advice, thank you very much!!!

some questions of output files

Hello, I m an ungreatute of lifescience institute, Lanzhou University. my name is Wenjie

I have tested your tools using some public data yesterday. It is amazing that the running speed of NextDenovo so fast, but after program processing, I have some questions confused

first is I do not know which is assembled most properest seqence, there have nine output directories of 02.cns_algin,please tell me how to selcet best and what are the differents among its.

secord is the folder named 03.ctg_graph is empty, whether I made some mistakes when I run program or input is improper. And what is means of 'graph' in thrid step, How is it different from 'assembly'?

my test data is ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR217/003/ERR2173373/ERR2173373.fastq.gz
and my deploy file was uploaded in appendix.

all best.
runcfg.txt

task assemble error

Hi,
After correct task, I run 'assemble task', but I got this error.

Could you give me a help?

Traceback (most recent call last):
File "/home/wangbo/softwares/NextDenovo/nextDenovo", line 518, in
main(args)
File "/home/wangbo/softwares/NextDenovo/nextDenovo", line 448, in main
cns_output = get_cns_output(task.subtasks)
UnboundLocalError: local variable 'task' referenced before assignment

Running error assembling Pacbio data

Hello!
I am trying to use NextDenovo these days and found it's okay using Ont data. But when I was trying to assemble Pacbio data some errors appeared and I cannot figure out.
There were Errors in the ./02.cns_align/02.cns_align.sh.work/cns_align*/nextDenovo.sh.e which showed
[ERROR] failed to find seq file type or empty file .
I wrote

 [correct_option]
read_cutoff = 1k
seed_cutoff = 1057
blocksize = 3g
pa_correction = 20
seed_cutfiles = 20
sort_options = -m 20g -t 10 -k 20
minimap2_options_raw = -x ava-pb -H -t 8
correction_options = -p 15

[assemble_option]
random_round = 20
minimap2_options_cns = -x ava-pb -H -t 8 -k17 -w17
nextgraph_options = -a 1

in my run.cfg when the errors come out.
I need help, thanks!

How do I run NextDenovo on my SLURM / SGE / PBS / LSF / Torque system?

Sorry, I do not know how to run only on the local machine, specify useGrid=false in run. ctf?

multithreading in local

Hi,
Are there have some parameters to assign the CPU number in a local computer?
thank you!

sort_overlap error

/home/soft/NextDenovo/bin/sort_overlap -m 100M -t 3 -k 50 -i /home/soft/NextDenovo/test_data/./01_rundir/01.raw_align/.input.seed.001.idx -o input.seed.001.sorted.ovl input.fofn
sort_overlap: sort_overlap.c:1023: main: Assertion `opts.max_mem > 2.2 * 100000000' failed.
Aborted (core dumped)

This error happens not matter how I change -m and -t ......

About data size after processing nextdenovo

Hi,

I've run nextdenovo on my ONT data and got cns reads. However, the data size of final corrected reads is too small to support de novo human genome assembly. I wonder if I run the tools correctly.

My data are about a list of 147Gb fastq files and a 90Gb fastq.gz file. After processing nextdenovo, the cns reads includ 101054 reads (>=5000bp), which is 4019661934 bp length, 15651442 bins.

My run.cfg file for nextdenovo is as followed:
[General]
job_type = local
job_prefix = nextDenovo
task = all
rewrite = yes # yes/no
input_type = raw
input_fofn = ./input.fofn

[correct_option]
read_cuoff = 1k
seed_cutoff = 32k
blocksize = 10g
pa_raw_align = 10
pa_correction = 15
seed_cutfiles = 10
sort_options = -m 5g -t 8 -k 50
minimap2_options = -x ava-ont -t 8
correction_options = -p 10

Looking for your help. Thanks!

NextDenovo is unfinished

Hi,
My NextDenovo has run five days in cns_align step, there are 49 jobs and 15 have finished, I can see the jobs using "top" and the status are "S", but the log files has not been updated too long time.
can you give me some help?
Thank you.

read_cuoff = 1k 应为 read_cutoff = 1k 问一下现在的参数设置

[General]
job_type = local
job_prefix = nextDenovo
task = all
rewrite = yes # yes/no
input_type = raw
input_fofn = ./input.fofn
workdir = ./01_rundir

[correct_option]
read_cuoff = 1k
seed_cutoff = 32k
blocksize = 1g
pa_raw_align = 2
pa_correction = 2
seed_cutfiles = 2
sort_options = -m 50g -t 20 -k 50
minimap2_options = -x ava-ont -t 20
correction_options = -p 40

Assembly size is smaller than estimated genome size

你好，在软件释放后立即初步体验了一下。使用的基因组预测大小为320M，杂合度为0.7%。最终N50达到了11M，基本达到了染色体水平，对这个水平非常惊喜。很感谢能开发出这样的软件。
但有一个问题，目前组装的基因组大小为273M，使用数据为nanopore raw data，参数均为默认。与其他软件相比（wtdbg2:293M，minimap+minisam：321M，SMARTdenovo：297M），还有一定差距，请问调节什么参数可以适当增大组装的基因组大小呢？
期待开发者的解答。后续也会继续使用反馈。再次感谢开发者做出的贡献~

The question of error cerrection

Hi,Hu,
Whether this version(nextDenovo v2.1-beta.0) of the error correction will result in high repeat area assembly bug?
Thx.

main: Assertion `opts.max_mem > 100000000' failed

it suggest that my memory is too small?

The SGE Option

Hi,hujiang:
The SGE parameter is not so fit to my SGE system .On my SGE, We must echo the Project name ,vf,and cpu number,the job will be submitted.I found that ,your mode is like falcon,so could you add some parameters like falcon?
job_queue = all.q
sge_option_da = -P 1234 -l vf=18G -q %(job_queue)s -l num_proc=4
sge_option_la = -P 1234 l vf=30G -q %(job_queue)s -l num_proc=4
Thank you !

test run is finished, but assembly file is none

I install the software and run the test.The log file is below.

[INFO] 2019-10-19 06:47:40,949 start...
[INFO] 2019-10-19 06:47:40,950 logfile: pid28989.log.info
[WARNING] 2019-10-19 06:47:40,951 Re-write workdir
[INFO] 2019-10-19 06:47:40,952 options:
[INFO] 2019-10-19 06:47:40,952 {'sort_threads': 2, 'nodelist': '', 'rewrite': 1, 'blocksize': '1g', 'job_prefix': 'nextDenovo', 'job_type': 'local', 'minimap2_options_raw': '-x ava-ont -t 8', 'cns_threads': 15, 'sort_mem': '1g', 'seed_cutoff': '32000', 'input_fofn': '/data/software/NextDenovo/test_data/./input.fofn', 'read_cutoff': '1k', 'input_type': 'raw', 'sort_options': '-m 1g -t 2 -k 50', 'parallel_jobs': '2', 'cluster_options': '', 'ctg_graphdir': '/data/software/NextDenovo/test_data/./01_rundir/03.ctg_graph', 'pa_correction': '2', 'workdir': '/data/software/NextDenovo/test_data/./01_rundir', 'random_round': '10', 'minimap2_threads': (8, 8), 'minimap2_options_cns': '-x ava-ont -t 8 -k17 -w17', 'cns_aligndir': '/data/software/NextDenovo/test_data/./01_rundir/02.cns_align', 'seed_cutfiles': '2', 'raw_aligndir': '/data/software/NextDenovo/test_data/./01_rundir/01.raw_align', 'task': 'all', 'deltmp': 1, 'rerun': 3, 'correction_options': '-p 15 -max_lq_length 10000', 'nextgraph_options': '-a 1'}
[INFO] 2019-10-19 06:47:40,952 mkdir: /data/software/NextDenovo/test_data/./01_rundir
[INFO] 2019-10-19 06:47:40,953 mkdir: /data/software/NextDenovo/test_data/./01_rundir/01.raw_align
[INFO] 2019-10-19 06:47:40,953 mkdir: /data/software/NextDenovo/test_data/./01_rundir/02.cns_align
[INFO] 2019-10-19 06:47:40,953 mkdir: /data/software/NextDenovo/test_data/./01_rundir/03.ctg_graph
[INFO] 2019-10-19 06:47:40,954 analysis tasks done
[INFO] 2019-10-19 06:47:40,955 total jobs: 1
[INFO] 2019-10-19 06:47:40,956 Throw jobID:[29002] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:41,635 db_split done
[INFO] 2019-10-19 06:47:41,636 analysis tasks done
[INFO] 2019-10-19 06:47:41,637 total jobs: 5
[INFO] 2019-10-19 06:47:41,638 Throw jobID:[29011] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:42,140 Throw jobID:[29017] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align1/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:44,229 Throw jobID:[29074] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align2/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:45,404 Throw jobID:[29099] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align3/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:47,565 Throw jobID:[29141] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align4/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:51,040 raw_align done
[INFO] 2019-10-19 06:47:51,042 analysis tasks done
[INFO] 2019-10-19 06:47:51,043 total jobs: 2
[INFO] 2019-10-19 06:47:51,045 Throw jobID:[29206] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/03.sort_align.sh.work/sort_align0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:51,547 Throw jobID:[29212] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/01.raw_align/03.sort_align.sh.work/sort_align1/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:54,208 remove temporary result: /data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align2/input.seed.002.2bit.2.ovl
[INFO] 2019-10-19 06:47:54,208 remove temporary result: /data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align3/input.seed.002.2bit.3.ovl
[INFO] 2019-10-19 06:47:54,208 remove temporary result: /data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align4/input.seed.002.2bit.4.ovl
[INFO] 2019-10-19 06:47:54,209 remove temporary result: /data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align0/input.seed.001.2bit.0.ovl
[INFO] 2019-10-19 06:47:54,209 remove temporary result: /data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align1/input.seed.001.2bit.1.ovl
[INFO] 2019-10-19 06:47:54,209 remove temporary result: /data/software/NextDenovo/test_data/01_rundir/01.raw_align/02.raw_align.sh.work/raw_align2/input.seed.001.2bit.2.ovl
[INFO] 2019-10-19 06:47:54,213 sort_align done
[INFO] 2019-10-19 06:47:54,214 analysis tasks done
[INFO] 2019-10-19 06:47:54,216 total jobs: 2
[INFO] 2019-10-19 06:47:54,217 Throw jobID:[29254] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/02.cns_align/01.get_cns.sh.work/get_cns0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:47:54,719 Throw jobID:[29278] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/02.cns_align/01.get_cns.sh.work/get_cns1/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:01,304 get_cns done
[INFO] 2019-10-19 06:48:01,304 cns finished, and final corrected reads file:
[INFO] 2019-10-19 06:48:01,304  /data/software/NextDenovo/test_data/./01_rundir/02.cns_align/01.get_cns.sh.work/get_cns*/cns.fasta
[INFO] 2019-10-19 06:48:01,305 analysis tasks done
[INFO] 2019-10-19 06:48:01,306 total jobs: 3
[INFO] 2019-10-19 06:48:01,307 Throw jobID:[29330] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/02.cns_align/02.cns_align.sh.work/cns_align0/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:01,809 Throw jobID:[29337] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/02.cns_align/02.cns_align.sh.work/cns_align1/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:04,389 Throw jobID:[29388] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/02.cns_align/02.cns_align.sh.work/cns_align2/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:07,400 cns_align done
[INFO] 2019-10-19 06:48:07,401 analysis tasks done
[INFO] 2019-10-19 06:48:07,403 total jobs: 11
[INFO] 2019-10-19 06:48:07,404 Throw jobID:[29442] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:07,906 Throw jobID:[29448] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph01/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:08,610 Throw jobID:[29456] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph02/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:09,112 Throw jobID:[29462] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph03/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:09,762 Throw jobID:[29481] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph04/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:10,282 Throw jobID:[29500] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph05/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:11,198 Throw jobID:[29507] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph06/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:11,907 Throw jobID:[29514] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph07/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:12,656 Throw jobID:[29522] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph08/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:13,158 Throw jobID:[29528] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph09/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:13,826 Throw jobID:[29548] jobCmd:[/data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph10/nextDenovo.sh] in the local_cycle.
[INFO] 2019-10-19 06:48:15,029 ctg_graph done
[INFO] 2019-10-19 06:48:15,030 nextDenovo finished
[INFO] 2019-10-19 06:48:15,030 final assembly file:
[INFO] 2019-10-19 06:48:15,030  /data/software/NextDenovo/test_data/./01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph*/nextgraph.assembly.contig.fasta
[INFO] 2019-10-19 06:48:15,031 final stat files:
[INFO] 2019-10-19 06:48:15,031  /data/software/NextDenovo/test_data/./01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph*/nextDenovo.sh.e

But the assembly file is only in /data/software/NextDenovo/test_data/01_rundir/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph00/,and file is empty. Is this right? My linux system is ubuntu18.04!

NextDenovo/bin/seq_dump: cannot execute binary file

Hello, I meet a problem when I run nextdenovo. Please give me suggestions about how to fix this. Details are shown as follows. I download this in my labtop, MACpro, 64bit.
Thanks!

(python2) on-campus-10-138-71-78:### NextDenovo yongzhou$ nextDenovo test_data/run.cfg
[INFO] 2019-11-13 22:25:58,016 start...
[INFO] 2019-11-13 22:25:58,016 logfile: pid933.log.info
[WARNING] 2019-11-13 22:25:58,016 It seems that you are using the default value of "seed_cutoff", it is recommended to use "bin/seq_stat" to calculate this value, because this value will be greatly affected by reads length and sequencing depth, and an inappropriate value can significantly reduce assembly quality.
[WARNING] 2019-11-13 22:25:58,017 Re-write workdir
[INFO] 2019-11-13 22:25:58,017 options:
[INFO] 2019-11-13 22:25:58,017 {'sort_threads': 2, 'nodelist': '', 'rewrite': 1, 'blocksize': '1g', 'job_prefix': 'nextDenovo', 'job_type': 'local', 'minimap2_options_raw': '-x ava-ont -t 8', 'cns_threads': 15, 'sort_mem': '1g', 'seed_cutoff': '29999', 'input_fofn': '/Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./input.fofn', 'read_cutoff': '1k', 'input_type': 'raw', 'sort_options': '-m 1g -t 2 -k 50', 'parallel_jobs': '2', 'cluster_options': '', 'ctg_graphdir': '/Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir/03.ctg_graph', 'pa_correction': '2', 'workdir': '/Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir', 'random_round': '10', 'minimap2_threads': (8, 8), 'minimap2_options_cns': '-x ava-ont -t 8 -k17 -w17', 'cns_aligndir': '/Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir/02.cns_align', 'seed_cutfiles': '2', 'raw_aligndir': '/Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir/01.raw_align', 'task': 'all', 'deltmp': 1, 'rerun': 3, 'correction_options': '-p 15 -max_lq_length 10000', 'nextgraph_options': '-a 1'}
[INFO] 2019-11-13 22:25:58,017 skip mkdir: /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir
[INFO] 2019-11-13 22:25:58,017 skip mkdir: /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir/01.raw_align
[INFO] 2019-11-13 22:25:58,017 skip mkdir: /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir/02.cns_align
[INFO] 2019-11-13 22:25:58,018 skip mkdir: /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/./01_rundir/03.ctg_graph
[INFO] 2019-11-13 22:25:58,019 analysis tasks done
[INFO] 2019-11-13 22:25:58,019 total jobs: 1
[INFO] 2019-11-13 22:25:58,020 Throw jobID:[940] jobCmd:[/Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh] in the local_cycle.
[ERROR] 2019-11-13 22:25:58,540 db_split failed: please check the following logs:
[ERROR] 2019-11-13 22:25:58,540 /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh.e
(python2) on-campus-10-138-71-78:NextDenovo yongzhou$ less /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh.e
(python2) on-campus-10-138-71-78:NextDenovo yongzhou$
hostname

• hostname
  cd /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0
• cd /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0
  time /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/bin/seq_dump -f 1k -s 29999 -b 1g -n 2 -d /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/input.fofn
• /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/bin/seq_dump -f 1k -s 29999 -b 1g -n 2 -d /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/input.fofn
  /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/test_data/01_rundir/01.raw_align/01.db_split.sh.work/db_split0/nextDenovo.sh: line 5: ### /Users/yongzhou/Softwares/13_NextDenovo/NextDenovo/bin/seq_dump: cannot execute binary file

How to run nextdenovo on lsf system

job_type = lsf
i don't know how to set cluster_options

task = ？

What is the relationship between 'correct' and 'assemble' ?