uphl-biongs/cecret

Reference-based consensus creation

cecret's Issues

All N's in MPX consensus

Our local MPX sequence had all N's in it's consensus sequence, whereas a comparative SRR sequence did not. So a complete SNP matrix was not produced, though bcftools_variants ran successfully for both.

I therefore uncommented 3 lines in cecret.config to make sure ivar_consensus was working, but had the same results.

Do I need to use ivar_variants instead of bcftools_variants?

Set up tests for multifasta support

So that the next releases will have in-house tests that we can try.

Issue setting maxcpus with v.2.0.2021115

I just updated to the newest version but I'm having issues with the cpu settings. I've been running the previous versions for a few months now (thanks for your hard work on this!) without issue on a machine with 6 cores + Docker, but now I'm getting errors such as:
Error executing process > 'bwa '

Caused by:
Process requirement exceed available CPUs -- req: 8; avail: 6

I've tried to override the params.maxcpus & params.medcpus and even hard coding -t or --threads to 6 throughout the Cecret.nf file for each process, but so far the error persists and the message starting the run displays:
The maximum number of CPUS used in this workflow is 8

Any ideas on what I should try next?

Adjust defaults to artic V4 primers

This is in the works, but probably won't get worked on until someone requests this feature.

fastp version is not captured correctly

When using fastp, the version is still in the log file, but it doesn't make it to the summary file.

I think this is due to fastp printing the version to stderr.

creation of a parameter schema

All nf-core workflows include a json parameter schema file in the top level directory that outlines all of the parameters of the workflow. Here are some examples:
viralrecon
mycosnp
bactopia

nf-core includes a tool to help with creating the file if you are using nf-core schema builder.

StaPH-B Toolkit v2 will use this file to tell users which parameters can be supplied with each workflow. If there isn't a schema file the toolkit will just inform the user that help is not available for this workflow. Including this file for Cecret will be really helpful for users once I release v2 of the toolkit.

iqtree2 failure: nextalign.aligned.fasta.renamed has wrong outgroup header for iqtree2

iqtree2 fails because of a wrong header for the outgroup.

outgroup="-o NC_063383"
cat nextalign.aligned.fasta | sed 's/NC_063383.*/NC_063383/g' > nextalign.aligned.fasta.renamed

nextalign.aligned.fasta header is:

MPXV_USA_2022_MA001 in NC_063383 coordinates

So nextalign.aligned.fasta.renamed header is:

MPXV_USA_2022_MA001 in NC_063383

Instead of this:

NC_63383

Solution is to fix sed command like this:
sed 's/^.NC_063383./NC_063383/g'

segmentation fault at ivar trim step

Hello @erinyoung

The pipeline fails at ivar trim step.
Here's the log:

WARNING: The BED file provided did not have the expected score column, but iVar will continue trimming

iVar uses the standard 6 column BED format as defined here - https://genome.ucsc.edu/FAQ/FAQformat.html#format1.
It requires the following columns delimited by a tab: chrom, chromStart, chromEnd, name, score, strand

WARNING: The BED file provided did not have the expected score column, but iVar will continue trimming

Found 218 primers in BED file
Amplicons detected:
Segmentation fault

Any ideas what might be the issue?

Add option to bypass trimming

Actually, this entire workflow needs a facelift and needs to be simplified.

Cecret will be more useful if there's an option to bypass primer trimming.

Error executing process > 'fastqc (12)'

I don't know what is the problem. I reinstalled fastqc from the link, but I can't fix. Please.

Furthermore, I left the complete output:

Error executing process > 'fastqc (12)'

Caused by:
Process fastqc (12) terminated with an error exit status (1)

Command executed:

mkdir -p fastqc logs/fastqc
log_file=logs/fastqc/12.427d29a5-7e5f-482e-82cd-7be02fedbeea.log
err_file=logs/fastqc/12.427d29a5-7e5f-482e-82cd-7be02fedbeea.err

time stamp + capturing tool versions
date | tee -a $log_file $err_file > /dev/null
fastqc --version >> $log_file

fastqc --outdir fastqc --threads 1 12_S12_L001_R1_001.fastq.gz 12_S12_L001_R2_001.fastq.gz 2>> $err_file >> $log_file

zipped_fastq=($(ls fastqc/*fastqc.zip) "")

raw_1=$(unzip -p ${zipped_fastq[0]} */fastqc_data.txt | grep "Total Sequences" | awk '{ print $3 }' )
raw_2=NA
if [ -f "${zipped_fastq[1]}" ] ; then raw_2=$(unzip -p fastqc/*fastqc.zip */fastqc_data.txt | grep "Total Sequences" | awk '{ print $3 }' ) ; fi

if [ -z "$raw_1" ] ; then raw_1="0" ; fi
if [ -z "$raw_2" ] ; then raw_2="0" ; fi

Command exit status:
1

Command output:
(empty)

Command error:
WARNING: failed to set O_CLOEXEC flags on image
WARNING: failed to set O_CLOEXEC flags on image
ERROR : Failed to set securebits: Invalid argument
ERROR : Failed to set securebits: Invalid argument

pipeline crash after updating to latest pangolin version v4

error: ValueError: Usecols do not match columns, columns expected but not found: ['pango_version', 'pangoLEARN_version', 'status']

column headings have changed/been modified

Add collection dates for Freyja

Freyja can create a cool figure if collection dates are added for each sample.

Right now there's no way to way to incorporate this into Cecret.

fastp and samtools versions look ... weird

When fastp is used instead of seqyclean, the fastp version doesn't look right.

Also a similar issues when using samtools ampliconclip.

Parsing of Freyja results into cecret results output

Thank you for adding Freyja in Cecret. It has been very helpful.
However, freyja results are not getting parsed correctly in the cecret result output. An additional row is being created in the table to output the freyja result. I have attached an example result file for your reference.

example_cecret_results.csv

Fix DAG image of Cecret

Self explanatory

adding MPX specific tools

I'm keeping a list of things to add:

https://github.com/insapathogenomics/mutation_profile/blob/main/README.md

allow nf-core like samplesheet

Some people actually like using a sample sheet for their samples.

Segmentation fault for iVar variants process

When running iVar occasionally throws a segmentation fault possibly related to samtools usage of memory.

Command exit status:
  139
Command output:
  (empty)
Command error:
  .command.sh: line 11:    37 Broken pipe             samtools mpileup -A -d 8000 -B -Q 0 --reference MN908947.3.fasta 091176-44249.primertrim.sorted.bam 2>> $err_file
          38 Segmentation fault      (core dumped) | ivar variants -p ivar_variants/091176-44249.variants -q 20 -t 0.6 -m 10 -r MN908947.3.fasta -g MN908947.3.gff 2>> $err_file >> $log_file

Giving the process addition memory after a retry seems to fix the issue:

memory {2.GB * task.attempt}
errorStrategy {'retry'}
maxRetries 2

I'm opening a pull request to add this to the process in case you want to include the changes to the workflow.

change summary file to be more flexible with pangolin updates

error with single reads

Full post at StaPH-B/staphb_toolkit#72

[heads up] Nextclade v2.0

Today, Nextclade v2.0 was released, with a lot of improvements (especially in performance). However, the new version contains a couple of breaking changes that may break your workflow…

• The input FASTA argument is now a positional argument, and it's now possible to specify more than one input FASTA file (no more need to cat them together!).
• The --output-dir flag has now been superseded by the --output-all flag. (Also, the --output-{json,csv,tsv,tree} flags have become redundant if you only use the default names for those output files. Here is the file that specifies the CLI flags and the default output file names.)

Best of luck with the new version!

nextclade error

There is an error when I run the cecret workflow using staphb-toolkit today.I run this command for many datasets before and it works fine.
I saw this issue for the nextcade update. However, when I check the errors, it seems like it still using the --output-dir instead of --output-all. I have upgrade the staphb-toolkit before I run the workflow but it doesn't work. I think it probably because staphb-toolkit doesn't update. I have checked latest release date for staphb-toolkit is April 25.

Here is my command:
staphb-wf cecret fastq_dir --output out --config cecret.config

nextflow error:

Error executing process > 'nextclade (Clade Determination)'

Caused by:
  Process `nextclade (Clade Determination)` terminated with an error exit status (2)

Command executed:

  mkdir -p nextclade dataset logs/nextclade
  log_file=logs/nextclade/nextclade.5cb932f9-fc35-4591-84a7-5fafd637a83e.log
  err_file=logs/nextclade/nextclade.5cb932f9-fc35-4591-84a7-5fafd637a83e.err

  date | tee -a $log_file $err_file > /dev/null
  nextclade --version >> $log_file
  nextclade_version=$(nextclade --version)

  nextclade dataset get --name sars-cov-2 --output-dir dataset

  for fasta in 1.consensus.fa 2.consensus.fa 3.consensus.fa 4.consensus.fa 5.consensus.fa
  do
    cat $fasta >> ultimate_fasta.fasta
  done

  nextclade        --input-fasta=ultimate_fasta.fasta       --input-dataset dataset       --output-json=nextclade/nextclade.json       --output-csv=nextclade/nextclade.csv       --output-tsv=nextclade/nextclade.tsv       --output-tree=nextclade/nextclade.auspice.json       --output-dir=nextclade       --output-basename=nextclade       2>> $err_file >> $log_file
  cp ultimate_fasta.fasta nextclade/combined.fasta

Command exit status:
  2

Command output:
  (empty)

Work dir:
  /storage/hpc/group/cov-sur/datasets/data/fastq/work/44/e4b578e9b613d8ea3fc2973cae69e8

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Update Cecret on StaphB toolkit

Feature request : Insert Size

Add Insert size to summary table

bedtools multicov overlap threshold is too low

Cecret runs bedtools to calculate the read depth of each amplicon using this command:
bedtools multicov -bams !{bam} -bed amplicon.bed
This counts the number of alignments in the bam file that overlap each amplicon region defined in the bed file. Bedtools' default definition of "overlap" is 1bp, so when bedtools counts up the number of alignments overlapping amplicon 5, it reports the number of alignments for amplicons 4, 5, and 6, since amplicons 4 and 6 overlap 5. If you require an overlap of 50% (-f .5), then the overlapping alignments from amplicons 4 and 6 won't be included in the count.
This will bring the number of "failed amplicons" reported by bedtools more in line with the number reported by samtools.

Here's an example of bedtools multicov with the default 1bp overlap:
MN908947.3 30 410 nCoV-2019_1 7450
MN908947.3 320 726 nCoV-2019_2 13592
MN908947.3 642 1028 nCoV-2019_3 13859
MN908947.3 943 1337 nCoV-2019_4 10224
MN908947.3 1242 1651 nCoV-2019_5 7396
MN908947.3 1573 1964 nCoV-2019_6 8599
MN908947.3 1875 2269 nCoV-2019_7 6648
MN908947.3 2181 2592 nCoV-2019_8 4817
MN908947.3 2505 2904 nCoV-2019_9 6788
MN908947.3 2826 3210 nCoV-2019_10 10485

And here's the same thing but requiring an overlap of 50% (-f .5)
MN908947.3 30 410 nCoV-2019_1 383
MN908947.3 320 726 nCoV-2019_2 6930
MN908947.3 642 1028 nCoV-2019_3 5911
MN908947.3 943 1337 nCoV-2019_4 906
MN908947.3 1242 1651 nCoV-2019_5 3347
MN908947.3 1573 1964 nCoV-2019_6 3047
MN908947.3 1875 2269 nCoV-2019_7 2151
MN908947.3 2181 2592 nCoV-2019_8 1358
MN908947.3 2505 2904 nCoV-2019_9 1283
MN908947.3 2826 3210 nCoV-2019_10 4117

When an overlap of 50% is required bedtools correctly reports the 383 alignments for amplicon 1 and 6,930 for amplicon 2. Running with the default 1bp overlap the number of amplicon 1 alignments is reported as 7,450, which is (approximately) the number of alignments to amplicons 1 and 2 (383 + 6,930 = 7313).

what if I want the input as assembled fastas?

Hi,
Impressive binf tool! May I ask what shall I do if I just want the input to be the assembled fastas to get the analysis such as phylogeny, pangolin, vadr, etc.?
Regards,
Shaokang

iqtree2 failure for monkeypox - wrong outgroup header in nextalign.aligned.fasta

Running from UPHL-BioNGS/Cecret with the following execution error:

[24/8ac928] NOTE: Process msa:iqtree2 (Creating phylogenetic tree with iqtree) terminated with an error exit status (2) -- Execution is retried (1)
Error executing process > 'msa:iqtree2 (Creating phylogenetic tree with iqtree)'

Caused by:
Process msa:iqtree2 (Creating phylogenetic tree with iqtree) terminated with an error exit status (2)

Command executed:

mkdir -p iqtree2 logs/msa:iqtree2
log_file=logs/msa:iqtree2/msa:iqtree2.7be2fac2-15c3-4970-aa30-aadd189177a2.log
err_file=logs/msa:iqtree2/msa:iqtree2.7be2fac2-15c3-4970-aa30-aadd189177a2.err

date | tee -a $log_file $err_file > /dev/null
iqtree2 --version >> $log_file

if [ -n "NC_063383.1" ] && [ "NC_063383.1" != "null" ] && [ "nextalign" != "nextclade" ]
then
outgroup="-o NC_063383.1"
cat nextalign.aligned.fasta | sed 's/NC_063383.1.*/NC_063383.1/g' > nextalign.aligned.fasta.renamed
else
outgroup=""
mv nextalign.aligned.fasta nextalign.aligned.fasta.renamed
fi

creating a tree

iqtree2 -ninit 2 -n 2 -me 0.05 -m GTR -o NC_063383.1 -nt AUTO -ntmax 8 -s nextalign.aligned.fasta.renamed -pre iqtree2/iqtree2 $outgroup >> $log_file 2>> $err_file

Command exit status:
2

Command output:
(empty)

Work dir:
/data/Sequence_analysis/Cecret/Analyses/monkeypox/iSeqs_Runs_220727/work/14/3f4bbbaca74412d37dbeb3cfe80aa3

==
The sequence header in nextalign.aligned.fasta was:

ref_in_coord Reference sequence in coord.fasta coordinates

It should have been:

NC_063383.1

So the sed command above didn't work:
sed 's/NC_063383.1.*/NC_063383.1/g'

WARN: Unable to fetch attribute for file...

Are these warning messages a problem for the outgroup files or can I safely ignore them?

WARN: Unable to fetch attribute for file: /app/becksts/.nextflow/assets/UPHL-BioNGS/Cecret/configs/NC_063383.1.gff - Hash is inferred from Git repository commit Id
WARN: Unable to fetch attribute for file: /app/becksts/.nextflow/assets/UPHL-BioNGS/Cecret/configs/NC_063383.1.gff - Hash is inferred from Git repository commit Id

Count bases with > 10X coverage

Update iqtree to iqtree

mpx:vadr (QC metrics) error

The last error I am seeing with Cecret is that vadr is terminating:

[e4/abdf4a] NOTE: Process mpx:vadr (QC metrics) terminated with an error exit status (255) -- Error is ignored

setup github action

Update nextclade container

It's already been posted on the staph-b toolkit (StaPH-B/staphb_toolkit#69), but I'll post this here as well.

The default nextclade container and the commands need to be updated.

Error executing process > 'pangolin (SARS-CoV-2 lineage Determination)'

This was using the git cloned version of Cecret. Error from NextFlow:

Error executing process > 'pangolin (SARS-CoV-2 lineage Determination)'

Caused by:
Process pangolin (SARS-CoV-2 lineage Determination) terminated with an error exit status (2)

Command executed:

mkdir -p pangolin logs/pangolin
log_file=logs/pangolin/pangolin.11a368d3-9f2d-4abc-b735-f07ad370a00d.log
err_file=logs/pangolin/pangolin.11a368d3-9f2d-4abc-b735-f07ad370a00d.err

date | tee -a $log_file $err_file > /dev/null
pangolin --all-versions >> $log_file

for fasta in 2111180087.consensus.fa
do
cat $fasta >> ultimate_fasta.fasta
done

pangolin --threads 4 --outdir pangolin ultimate_fasta.fasta 2>> $err_file >> $log_file
cp ultimate_fasta.fasta pangolin/combined.fasta

Command exit status:
2

Command output:
(empty)

Command error:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
usage: pangolin [options]
pangolin: error: unrecognized arguments: --all-versions

Work dir:
/home/mdubfx/cecret_test/Cecret/work/be/1e5b21170bcb2897537dc02b9f6f8b

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

The StaPH-B version of Cecret is also crashing at the pangolin stage. Thanks!

Add freyja

Freyja is a tool for co-occurring SARS-CoV-2 infections

Feature Request : samtools fixmate when using ivar trim and samtools amplicon stats

Long story, but sometimes when ivar trims primers and then the stats are being gathered by samtools ampliconstats, the depth becomes negative. There's a workaround involving samtools fixmate that will help the issue.

Apple M1 & Docker Issue

I've run the 'nextflow run Cecret.nf -c configs/docker.config' command to test 3 samples and I have encountered an error.

N E X T F L O W ~ version 21.10.1
Launching Cecret.nf [hungry_faggin] - revision: 9cede0c1f7
Currently using the Cecret workflow for use with amplicon-based Illumina hybrid library prep on MiSeq

Author: Erin Young
email: [email protected]
Version: v.2.2.20211220

Fastq file found : 7093-MS-1_80
Fastq file found : 7093-MS-1_7
Fastq file found : 7093-MS-1_81
The maximum number of CPUS used in this workflow is 8
The files and directory for results is /Users/vestalg/Cecret-master 2/cecret
A table summarizing results will be created: /Users/vestalg/Cecret-master 2/cecret/summary.txt and /Users/vestalg/Cecret-master 2/cecret_run_results.txt

Reference Genome : /Users/vestalg/Cecret-master 2/configs/MN908947.3.fasta
GFF file for Reference Genome : /Users/vestalg/Cecret-master 2/configs/MN908947.3.gff
Primer BedFile : /Users/vestalg/Cecret-master 2/configs/artic_V3_nCoV-2019.bed
Amplicon BedFile : /Users/vestalg/Cecret-master 2/configs/nCoV-2019.insert.bed

executor > local (6)
[be/112020] process > fastqc (7093-MS-1_81) [ 0%] 0 of 3
[09/afc9fb] process > seqyclean (7093-MS-1_81) [ 0%] 0 of 3
executor > local (6)
[be/112020] process > fastqc (7093-MS-1_81) [ 0%] 0 of 3
[09/afc9fb] process > seqyclean (7093-MS-1_81) [ 25%] 1 of 4, failed: 1, r..
[- ] process > bwa -
[- ] process > sort -
executor > local (10)
[04/185650] process > fastqc (7093-MS-1_80) [ 40%] 2 of 5, failed: 2, r..
[e8/d25ab3] process > seqyclean (7093-MS-1_80) [ 50%] 3 of 6, failed: 3, r..
[- ] process > bwa -
[- ] process > sort -
[- ] process > filter -
[- ] process > ivar_trim -
[- ] process > ivar_variants -
[- ] process > ivar_consensus -
[- ] process > fasta_prep -
[- ] process > bcftools_variants -
[- ] process > bamsnap -
[- ] process > samtools_stats -
executor > local (12)
[35/a63224] process > fastqc (7093-MS-1_81) [ 50%] 3 of 6, failed: 3, r..
[ed/d3f88a] process > seqyclean (7093-MS-1_80) [ 50%] 3 of 6, failed: 3, r..
[- ] process > bwa -
[- ] process > sort -
[- ] process > filter -
[- ] process > ivar_trim -
[- ] process > ivar_variants -
[- ] process > ivar_consensus -
[- ] process > fasta_prep -
[- ] process > bcftools_variants -
[- ] process > bamsnap -
[- ] process > samtools_stats -
[- ] process > samtools_coverage -
[- ] process > samtools_flagstat -
executor > local (12)
[35/a63224] process > fastqc (7093-MS-1_81) [ 50%] 3 of 6, failed: 3, r..
[ed/d3f88a] process > seqyclean (7093-MS-1_80) [ 50%] 3 of 6, failed: 3, r..
[- ] process > bwa -
[- ] process > sort -
[- ] process > filter -
[- ] process > ivar_trim -
[- ] process > ivar_variants -
[- ] process > ivar_consensus -
[- ] process > fasta_prep -
[- ] process > bcftools_variants -
[- ] process > bamsnap -
[- ] process > samtools_stats -
[- ] process > samtools_coverage -
[- ] process > samtools_flagstat -
[- ] process > samtools_depth -
[- ] process > kraken2 -
[- ] process > bedtools_multicov -
[- ] process > samtools_ampliconstats -
[- ] process > samtools_plot_ampliconstats -
[- ] process > pangolin -
[- ] process > nextclade -
[- ] process > vadr -
[- ] process > summary -
[- ] process > combine_results -
[09/afc9fb] NOTE: Process seqyclean (7093-MS-1_81) terminated with an error exit status (127) -- Execution is retried (1)
[b3/7b5c47] NOTE: Process fastqc (7093-MS-1_7) terminated with an error exit status (127) -- Execution is retried (1)
[ae/30a22f] NOTE: Process fastqc (7093-MS-1_80) terminated with an error exit status (127) -- Execution is retried (1)
[af/4e7788] NOTE: Process seqyclean (7093-MS-1_7) terminated with an error exit status (127) -- Execution is retried (1)
[e8/d25ab3] NOTE: Process seqyclean (7093-MS-1_80) terminated with an error exit status (127) -- Execution is retried (1)
[be/112020] NOTE: Process fastqc (7093-MS-1_81) terminated with an error exit status (127) -- Execution is retried (1)
Error executing process > 'fastqc (7093-MS-1_80)'

Caused by:
Process fastqc (7093-MS-1_80) terminated with an error exit status (127)

Command executed:

mkdir -p fastqc logs/fastqc
log_file=logs/fastqc/7093-MS-1_80.f574859a-d7c7-4223-a402-3fd94fdd9e50.log
err_file=logs/fastqc/7093-MS-1_80.f574859a-d7c7-4223-a402-3fd94fdd9e50.err

time stamp + capturing tool versions

date | tee -a $log_file $err_file > /dev/null
fastqc --version >> $log_file

fastqc --outdir fastqc --threads 1 7093-MS-1_80_S1_L005_R1_001.fastq.gz 7093-MS-1_80_S1_L005_R2_001.fastq.gz 2>> $err_file >> $log_file

zipped_fastq=($(ls fastqc/*fastqc.zip) "")

raw_1=$(unzip -p ${zipped_fastq[0]} */fastqc_data.txt | grep "Total Sequences" | awk '{ print $3 }' )
raw_2=NA
if [ -f "${zipped_fastq[1]}" ] ; then raw_2=$(unzip -p fastqc/*fastqc.zip */fastqc_data.txt | grep "Total Sequences" | awk '{ print $3 }' ) ; fi

if [ -z "$raw_1" ] ; then raw_1="0" ; fi
if [ -z "$raw_2" ] ; then raw_2="0" ; fi

Command exit status:
127

Command output:
(empty)
executor > local (12)
[04/185650] process > fastqc (7093-MS-1_80) [ 66%] 4 of 6, failed: 4, r..
[ed/d3f88a] process > seqyclean (7093-MS-1_80) [ 50%] 3 of 6, failed: 3, r..
[- ] process > bwa -
[- ] process > sort -
[- ] process > filter -
[- ] process > ivar_trim -
[- ] process > ivar_variants -
[- ] process > ivar_consensus -
[- ] process > fasta_prep -
[- ] process > bcftools_variants -
[- ] process > bamsnap -
[- ] process > samtools_stats -
[- ] process > samtools_coverage -
[- ] process > samtools_flagstat -
[- ] process > samtools_depth -
[- ] process > kraken2 -
[- ] process > bedtools_multicov -
[- ] process > samtools_ampliconstats -
[- ] process > samtools_plot_ampliconstats -
[- ] process > pangolin -
[- ] process > nextclade -
[- ] process > vadr -
[- ] process > summary -
[- ] process > combine_results -
[09/afc9fb] NOTE: Process seqyclean (7093-MS-1_81) terminated with an error exit status (127) -- Execution is retried (1)
[b3/7b5c47] NOTE: Process fastqc (7093-MS-1_7) terminated with an error exit status (127) -- Execution is retried (1)
[ae/30a22f] NOTE: Process fastqc (7093-MS-1_80) terminated with an error exit status (127) -- Execution is retried (1)
[af/4e7788] NOTE: Process seqyclean (7093-MS-1_7) terminated with an error exit status (127) -- Execution is retried (1)
[e8/d25ab3] NOTE: Process seqyclean (7093-MS-1_80) terminated with an error exit status (127) -- Execution is retried (1)
[be/112020] NOTE: Process fastqc (7093-MS-1_81) terminated with an error exit status (127) -- Execution is retried (1)
Error executing process > 'fastqc (7093-MS-1_80)'

Caused by:
Process fastqc (7093-MS-1_80) terminated with an error exit status (127)

Command executed:

mkdir -p fastqc logs/fastqc
log_file=logs/fastqc/7093-MS-1_80.f574859a-d7c7-4223-a402-3fd94fdd9e50.log
err_file=logs/fastqc/7093-MS-1_80.f574859a-d7c7-4223-a402-3fd94fdd9e50.err

time stamp + capturing tool versions

date | tee -a $log_file $err_file > /dev/null
fastqc --version >> $log_file

fastqc --outdir fastqc --threads 1 7093-MS-1_80_S1_L005_R1_001.fastq.gz 7093-MS-1_80_S1_L005_R2_001.fastq.gz 2>> $err_file >> $log_file

zipped_fastq=($(ls fastqc/*fastqc.zip) "")

raw_1=$(unzip -p ${zipped_fastq[0]} */fastqc_data.txt | grep "Total Sequences" | awk '{ print $3 }' )
raw_2=NA
if [ -f "${zipped_fastq[1]}" ] ; then raw_2=$(unzip -p fastqc/*fastqc.zip */fastqc_data.txt | grep "Total Sequences" | awk '{ print $3 }' ) ; fi

if [ -z "$raw_1" ] ; then raw_1="0" ; fi
if [ -z "$raw_2" ] ; then raw_2="0" ; fi

Command exit status:
127

Command output:
(empty)

Command error:
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
/bin/bash: line 0: export: `2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/FastQC/': not a valid identifier
.command.sh: line 2: mkdir: command not found

Work dir:
/Users/vestalg/Cecret-master 2/work/04/185650992b0ea86d57547adb49c938

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

Is this related to the Apple M1 chip I am using? I've attempted to use Option 1 and Option 2 to run from Docker and Singularity and I encounter an issue with the StaphB FastQC Docker container:

FATAL: Unable to pull docker://staphb/fastqc:latest: conveyor failed to get: no descriptor found for reference "5d67fad373325a597d0eb75d0986ad7dd78f8b2b67e78435d9176c434a8cec04"

Is this an issue with the Apple M1 chips?

Process `summary` input file name collision

When both params.pangolin and params.vadr are set to false, nextflow complains:

Error executing process > 'summary (P120-S09-00846-4-S_S24)'

Caused by:
  Process `summary` input file name collision -- There are multiple input files for each of the following file names: Cecret.nf

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

Possible offending lines:

Cecret/Cecret.nf

Lines 1134 to 1136 in c683a19

	} else {
	pangolin_file = Channel.fromPath(workflow.projectDir + "/Cecret.nf", type:'file')
	}

Cecret/Cecret.nf

Lines 1233 to 1235 in c683a19

	} else {
	vadr_file = Channel.fromPath(workflow.projectDir + "/Cecret.nf", type:'file')
	}

This was run with version 1.3.3 of the StaPH-B toolkit, which may not include an up-to-date workflow file.

Fix stageinmode

Nextclade needs an update

The erin-dev branch actually has a fix for this.

The error goes something like this

  --2021-11-16 22:05:08--  https://raw.githubusercontent.com/nextstrain/nextclade/master/data/sars-cov-2/tree.json
  Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...
  Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
  HTTP request sent, awaiting response... 404 Not Found
  2021-11-16 22:05:08 ERROR 404: Not Found.

NextStrain told everyone months ago that this would happen, so there is effort to get this corrected. We've had internal issues that have prevented us from releasing the next version of Cecret.

Update readme for kraken2 database

The initial kraken2 download in the readme is broken, BUT! there's a new WORKING one :

I already have it in a readme here and it would be simple to adjust the wording.

No way to tell "trimmer" in summary file

The "aligner" and "cleaner" are readily apparent in the final summary file, but the "trimmer" is not.

That should be added to the summary file.

Add tools specific for MonkeyPox

Please add tools specific for working with MonkeyPox

update freyja

Pangolin lineage missing from summary file

Descriptive title

skip ampliconstats if not using primers

Update Nextclade

Copied over from SLACK

Quick note for those upgrading Nextclade CLI to v1.10.*:
If you don’t use --input-dataset, you need to add a new arg --input-virus-properties
In v.1.10.0, you won’t get a helpful error message, this is fixed in patch release 1.10.1
Details in the thread replies here so as not to spam.

Test run ...

Hello @erinyoung -

I installed singularity, nextflow on my mac.
I git cloned this repo and have a test fastq file (single end).
I was hoping I can run the command,

nextflow Cecret/Cecret.nf

But, it fails with fastqc command not found.
I was assuming the script might pull necessary containers and thus no need to install dependencies on my laptop before running this script.
I see containers.config in configs directory.
If I want to run this script using singularity, how should I invoke the command or configure this software?
Thank you in advance.

Best,
Mahesh

Feature Request : Amplicon bedfile

For people that use custom primers, an amplicon bedfile would be useful for bedtools. I create one for SARS-CoV-2 artic V3 primers, but it would be better if it was user supplied with some documentation on how to perform it locally.

Does not parse pangolin file correctly

All the pangolin results are null in the summary file. Pangolin still seems to be running, however.