Comments (11)
Could you provide more details about the error?
I also recommend to run the pipeline with the new update and contact if the problem persist in this new issue. I've adapted the output plotting scripts to a big number of samples.
from nanoclust.
您能否提供有关该错误的更多详细信息?
我还建议使用新的更新来运行管道,如果此问题仍然存在,请联系。我已经将输出绘图脚本改编为大量示例。
thank,you.
yes,i am using the new pipline to run ,and later i will tell you the result whatever.
from nanoclust.
#######
cutor > local (112)
[49/e62e0e] process > QC (15) [100%] 34 of 34 ✔
[ac/29994e] process > fastqc (34) [100%] 34 of 34 ✔
[- ] process > multiqc [ 0%] 0 of 1
[90/a2d271] process > kmer_freqs (34) [100%] 34 of 34 ✔
[3e/2b620e] process > read_clustering (9) [ 18%] 6 of 33, failed: 3, retries: 3
[- ] process > split_by_cluster [ 0%] 0 of 3
[- ] process > read_correction -
[- ] process > draft_selection -
[- ] process > racon_pass -
[- ] process > medaka_pass -
[- ] process > consensus_classification -
[- ] process > join_results -
[- ] process > get_abundances -
[- ] process > plot_abundances -
[bb/067704] process > output_documentation [100%] 1 of 1 ✔
[0;35m[nf-core/nanoclust] Pipeline completed with errors
[dc/639563] NOTE: Process read_clustering (5)
terminated with an error exit status (137) -- Execution is retried (1)
WARN: Killing pending tasks (3)
Execution aborted due to an unexpected error
#######
i just run 34 samples ,the problems is coming,
i wander that one samlpes one by one to run and all samples run togehter ,whether the results are consistent?
from nanoclust.
_ __ ________ __ _____________
/ | / /___ _____ ____ / ____/ / / / / / ___/_ __/
/ |/ / __ `/ __ \/ __ \ / / / / / / / /\__ \ / /
/ /| / // / / / / // / / // // // // // /
// |/_,// //_/ _/__/_//___///
NanoCLUST v1.0dev
Run Name : ridiculous_kare
Reads : 16s/*.fastq
Max Resources : 128 GB memory, 16 cpus, 10d time per job
Container : docker - [:]
Output dir : result_nanopore6
Launch dir : /Volumes/rest/work/16s/0522/13
Working dir : /Volumes/rest/work/16s/0522/13/work
Script dir : /Volumes/rest/work/16s/0522/13/NanoCLUST
User : root
Config Profile : docker
executor > local (5)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
executor > local (6)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
executor > local (6)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
executor > local (6)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
executor > local (6)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
[31/6f7e04] process > fastqc (7) [100%] 7 of 7, cached: 7 ✔
executor > local (7)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
[31/6f7e04] process > fastqc (7) [100%] 7 of 7, cached: 7 ✔
executor > local (7)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
executor > local (9)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
[31/6f7e04] process > fastqc (7) [100%] 7 of 7, cached: 7 ✔
executor > local (11)
executor > local (12)
executor > local (1117)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
[31/6f7e04] process > fastqc (7) [100%] 7 of 7, cached: 7 ✔
[f6/04a5fb] process > multiqc [100%] 1 of 1 ✔
[d9/5e7d70] process > kmer_freqs (7) [100%] 7 of 7, cached: 7 ✔
executor > local (1118)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
[31/6f7e04] process > fastqc (7) [100%] 7 of 7, cached: 7 ✔
[f6/04a5fb] process > multiqc [100%] 1 of 1 ✔
[d9/5e7d70] process > kmer_freqs (7) [100%] 7 of 7, cached: 7 ✔
executor > local (1118)
[ed/b232b2] process > QC (4) [100%] 7 of 7, cached: 7 ✔
[31/6f7e04] process > fastqc (7) [100%] 7 of 7, cached: 7 ✔
[f6/04a5fb] process > multiqc [100%] 1 of 1 ✔
[d9/5e7d70] process > kmer_freqs (7) [100%] 7 of 7, cached: 7 ✔
[3f/aac4b5] process > read_clustering (6) [100%] 13 of 13, failed: 6, retries: 6 ✔
[d5/67e0c3] process > split_by_cluster (7) [100%] 7 of 7 ✔
[54/70be5a] process > read_correction (745) [100%] 746 of 746 ✔
[4e/9b2532] process > draft_selection (345) [ 46%] 346 of 746
[1c/0ad410] process > racon_pass (5) [ 1%] 5 of 345, failed: 1
[- ] process > medaka_pass [ 0%] 0 of 2
[- ] process > consensus_classification -
[- ] process > join_results -
[- ] process > get_abundances -
[- ] process > plot_abundances -
[bb/067704] process > output_documentation [100%] 1 of 1, cached: 1 ✔
Execution cancelled -- Finishing pending tasks before exit
Error executing process > 'racon_pass (4)'
Caused by:
Process racon_pass (4)
terminated with an error exit status (1)
Command executed:
minimap2 -ax map-ont --no-long-join -r100 -a draft_read.fasta corrected_reads.correctedReads.fasta -o aligned.sam
racon --quality-threshold=9 -w 250 corrected_reads.correctedReads.fasta aligned.sam draft_read.fasta > racon_consensus.fasta
Command exit status:
1
Command output:
(empty)
Command error:
[M::mm_idx_gen::0.0390.11] collected minimizers
[M::mm_idx_gen::0.0400.14] sorted minimizers
[M::main::0.0410.14] loaded/built the index for 1 target sequence(s)
[M::mm_mapopt_update::0.0410.14] mid_occ = 2
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
[M::mm_idx_stat::0.0410.14] distinct minimizers: 305 (100.00% are singletons); average occurrences: 1.000; average spacing: 5.472
[M::worker_pipeline::0.0860.09] mapped 7 sequences
[M::main] Version: 2.17-r941
[M::main] CMD: minimap2 -ax map-ont --no-long-join -r100 -a -o aligned.sam draft_read.fasta corrected_reads.correctedReads.fasta
[M::main] Real time: 0.088 sec; CPU: 0.009 sec; Peak RSS: 0.003 GB
[racon::Polisher::initialize] loaded target sequences 0.000883 s
[racon::Polisher::initialize] loaded sequences 0.000965 s
[racon::Polisher::initialize] error: empty overlap set!
Work dir:
/Volumes/rest/work/16s/0522/13/work/42/179274480642c682e93cc5a2f6bbc7
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh
[1]+ Stopped sudo nextflow run NanoCLUST/main.nf --reads '16s/.fastq' -profile docker --db db/16S_ribosomal_RNA --tax db/taxdb/ --polishing_reads 8 --min_cluster_size 20 --outdir result_nanopore6 -resume nanaopore136
(base) dhys-MacBook-Air:13 dhy$ sudo nextflow run NanoCLUST/main.nf --reads '16s/.fastq' -profile docker --db db/16S_ribosomal_RNA --tax db/taxdb/ --polishing_reads 5 --min_cluster_size 20 --outdir result_nanopore6 -resume nanaopore136
Password:
N E X T F L O W ~ version 20.04.1
Launching NanoCLUST/main.nf
[cheesy_morse] - revision: 5166b629f0
Unable to acquire lock on session with ID 26c6a5d3-da6b-462a-b45a-1082dd786649
Common reasons of this error are:
- You are trying to resume the execution of an already running pipeline
- A previous execution was abruptly interrupted leaving the session open
You can check what process is holding the lock file by using the following command:
- lsof /Volumes/rest/work/16s/0522/13/.nextflow/cache/26c6a5d3-da6b-462a-b45a-1082dd786649/db/LOCK
#########
this is the error ,i just run 7 samples
from nanoclust.
Hi, thank you for the logs.
I've found some issues when running the pipeline setting the parameters min_cluster_size and polishing_reads with too low values. This may be a cause of problem in canu/racon/medaka processes.
The values assigned in the test profile (50 and 20) are too low and could be not suitable for real samples (no mock community) and bigger files. I recommend to set polishing_reads with 500-1000 and also provide a higher min_cluster_size value (100-300) and see if you don't get an error.
Thank you for your time and testing the tool.
from nanoclust.
ok,thans you,i will try,
from nanoclust.
i have changed the parament
nextflow run NanoCLUST/main.nf --reads 'no/*.fastq' -profile docker --db db/16S_ribosomal_RNA --tax db/taxdb/ --polishing_reads 500 --min_cluster_size 200 --outdir result_nanopore22222 -name nanaopore1all2
Password:
N E X T F L O W ~ version 20.04.1
Launching NanoCLUST/main.nf
[nanaopore1all2] - revision: 5166b629f0
_ __ ________ __ _____________
/ | / /___ _____ ____ / ____/ / / / / / ___/_ __/
/ |/ / __ `/ __ \/ __ \ / / / / / / / /\__ \ / /
/ /| / // / / / / // / / // // // // // /
// |/_,// //_/ _/__/_//___///
NanoCLUST v1.0dev
Run Name : nanaopore1all2
Reads : no/*.fastq
Max Resources : 128 GB memory, 16 cpus, 10d time per job
Container : docker - [:]
Output dir : result_nanopore22222
Launch dir : /Volumes/rest/work/16s/0522/13
Working dir : /Volumes/rest/work/16s/0522/13/work
Script dir : /Volumes/rest/work/16s/0522/13/NanoCLUST
User : root
Config Profile : docker
executor > local (109)
[97/baa6ed] process > QC (26) [100%] 26 of 26 ✔
[37/6bdebb] process > fastqc (26) [100%] 26 of 26 ✔
executor > local (110)
[97/baa6ed] process > QC (26) [100%] 26 of 26 ✔
[37/6bdebb] process > fastqc (26) [100%] 26 of 26 ✔
[76/65c519] process > multiqc [100%] 1 of 1 ✔
[f8/47682c] process > kmer_freqs (25) [100%] 26 of 26 ✔
executor > local (110)
[97/baa6ed] process > QC (26) [100%] 26 of 26 ✔
[37/6bdebb] process > fastqc (26) [100%] 26 of 26 ✔
[76/65c519] process > multiqc [100%] 1 of 1 ✔
[f8/47682c] process > kmer_freqs (25) [100%] 26 of 26 ✔
[51/3b98cb] process > read_clustering (3) [ 68%] 27 of 40, failed: 17, retries: 16
[0e/3f4833] process > split_by_cluster (1) [ 0%] 0 of 9
[- ] process > read_correction -
[- ] process > draft_selection -
[- ] process > racon_pass -
[- ] process > medaka_pass -
[- ] process > consensus_classification -
[- ] process > join_results -
[- ] process > get_abundances -
[- ] process > plot_abundances -
[0e/0abcd4] process > output_documentation [100%] 1 of 1 ✔
[0;35m[nf-core/nanoclust] Pipeline completed with errors
[8a/9f3abc] NOTE: Process read_clustering (26)
terminated with an error exit status (137) -- Execution is retried (1)
WARN: Killing pending tasks (3)
Error executing process > 'read_clustering (1)'
Caused by:
Process read_clustering (1)
terminated with an error exit status (137)
Command executed [/Volumes/rest/work/16s/0522/13/NanoCLUST/templates/umap_hdbscan.py]:
#!/usr/bin/env python
import numpy as np
import umap
import matplotlib.pyplot as plt
from sklearn import decomposition
import random
import pandas as pd
import hdbscan
df = pd.read_csv("input.1", delimiter="min_dis")
#UMAP
motifs = [x for x in df.columns.values if x not in ["read", "length"]]
X = df.loc[:,motifs]
X_embedded = umap.UMAP(n_neighbors=15, min_dist=0.1, verbose=2).fit_transform(X)
df_umap = pd.DataFrame(X_embedded, columns=["D1", "D2"])
umap_out = pd.concat([df["read"], df["length"], df_umap], axis=1)
#HDBSCAN
X = umap_out.loc[:,["D1", "D2"]]
umap_out["bin_id"] = hdbscan.HDBSCAN(min_cluster_size=int(200), cluster_selection_epsilon=int(0.5)).fit_predict(X)
#PLOT
plt.figure(figsize=(20,20))
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=umap_out["bin_id"], cmap='Spectral', s=1)
plt.xlabel("UMAP1", fontsize=18)
plt.ylabel("UMAP2", fontsize=18)
plt.gca().set_aspect('equal', 'datalim')
plt.title("Projecting " + str(len(umap_out['bin_id'])) + " reads. " + str(len(umap_out['bin_id'].unique())) + " clusters generated by HDBSCAN", fontsize=18)
for cluster in np.sort(umap_out['bin_id'].unique()):
read = umap_out.loc[umap_out['bin_id'] == cluster].iloc[0]
plt.annotate(str(cluster), (read['D1'], read['D2']), weight='bold', size=14)
plt.savefig('hdbscan.output.png')
umap_out.to_csv("hdbscan.output.tsv", sep=" ", index=False)
Command exit status:
137
Command output:
(empty)
Command error:
.command.run: line 159: 22 Killed /usr/bin/env python .command.sh
Work dir:
/Volumes/rest/work/16s/0522/13/work/c9/fe155a066d9a48a085ce3e826e5aa4
Tip: when you have fixed the problem you can continue the execution adding the option -resume
to the run command line
(######
from nanoclust.
This keeps happening when min_cluster_size is 50 and polishing_reads is still 500?
An excessive number of clusters due to low min_cluster_size is 50 could kill the conda env. If you are still getting this error I would try even higher values than 50 for min_cluster_size.
from nanoclust.
Hi,
I've been inspecting your log and the error 137 you are getting in the process is because it ran out of memory. I've updated the nextflow.config file to use 8GB initially and try with more RAM if a process fail due to this error. We recommend at least 16GB of RAM in your machine. I hope this time you dont get memory errors.
EDIT: now fixing an issue with that commit. I will update when fixed
from nanoclust.
ok,
thkan you
i. will try,
and. my ram is 64GB, when i run the pipline ,i monitor the cpu and ram at the same time,the ram maxused is 54 GB.
from nanoclust.
The issue with the commit is fixed and the memory adjustments for these processes have changed.
EDIT: Before the commit, the memory per process was limited up to 7GB. Even if your machine has enough memory, the process would fail.
I’m working also on limit the processes that can be run at certain pipeline stages to improve multiple sample files per run. We have tested it with 12 samples.
from nanoclust.
Related Issues (20)
- Error executing process > 'get_abundances (1)' HOT 2
- Can you do OTU clustering with ASVs that are different lengths?
- Process read_correction is applied to a subset of reads from the fasta file
- Blast DB error during consensus_classification
- Error executing process > 'consensus_classification (1)' HOT 4
- Merge fastq files prior to Nanoclust
- Process `consensus_classification (5)` terminated with an error exit status (255) HOT 1
- some samples terminate at read_correction due to gzip: corrected_reads.correctedReads.fasta.gz: No such file or directory HOT 2
- How we get reads for each consensus fasta file in each cluster, and coverage of our data
- Error executing process > 'get_abundances (1)' HOT 4
- ERROR: Nextflow DSL1 is no longer supported HOT 1
- medaka consensus error HOT 1
- Error read_clustering (ValueError: could not convert string to float: 'TTTTG')
- touch Permission denied with test
- Remote resource not found when running via Nextflow with nf-core/nanoclust HOT 2
- No taxonomy in the summary file
- Create a database (DB) from a custom database.
- Unable to attach MultiQC report to summary email
- NanoCLUST script not supported by latest Nextflow HOT 1
- Error when using Nextflow 22.10.7: Unable to parse config file...compile failed for sources FixedSetSources...
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nanoclust.