Hi! I have an issue downloading the databases for HPC: Looks like my

<a href="https://github.com/shandley/hecatomb/files/10521260/2023-01-27T102534.391649.

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Progress! I think this is the same issue as <a class="issue-link js-issue-link" data-e

lucyintheskyzzz commented on September 26, 2024

Update- I read the wiki and created a snakemake profile. I tried to run the install for HPC and get an error, see below.

Thanks!

(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress2 slurm]$ hecatomb install --profile slurm

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗
██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗
███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝
██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗
██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝
╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

[2023:01:26 16:07:35] Config file hecatomb.config.yaml already exists. Using existing config file.
[2023:01:26 16:07:35] Writing runtime config file to hecatomb.config.yaml
[2023:01:26 16:07:35] ------------------
[2023:01:26 16:07:35] | Runtime config |
[2023:01:26 16:07:35] ------------------

BigJobCpu: 24
BigJobMem: 64000
BigJobTimeMin: 1440
COMPRESSION: 1
CONTIG_MINLENGTH: 1000
CUTTAIL_WINDOW: 25
DEDUP_ACCURACY: 4
Databases: null
ENTROPY: 0.5
ENTROPYWINDOW: 25
Host: human
MediumJobCpu: 16
MediumJobMem: 32000
MoreRamCpu: 2
MoreRamMem: 16000
Output: hecatomb.out
Preprocess: paired
QSCORE: 15
READ_MINLENGTH: 90
Reads: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data
Search: sensitive
SmallJobCpu: 8
SmallJobMem: 16000
canuSettings: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False genomeSize=10M
-nanopore
filtAAprimary: --min-length 30 -e 1e-3
filtAAsecondary: --min-length 30 -e 1e-5
filtNTprimary: --min-length 90 -e 1e-10
filtNTsecondary: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode
3
perfAA: --start-sens 1 --sens-steps 3 -s 7 --lca-mode 2 --shuffle 0
perfAAfast: -s 4.0 --lca-mode 2 --shuffle 0
perfNT: --start-sens 2 -s 7 --sens-steps 3
perfNTfast: -s 4.0
taxIdIgnore: 0 1 2 10239 131567 12429 2759

[2023:01:26 16:07:35] ---------------------
[2023:01:26 16:07:35] | Snakemake command |
[2023:01:26 16:07:35] ---------------------

snakemake -s /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/workflow/DownloadDB.smk --configfile hecatomb.config.yaml --rerun-incomplete --printshellcmds --nolock --show-failed-logs --profile slurm
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Job stats:
job count min threads max threads

all 1 1 1
download_db_file 7 1 1
total 8 1 1

Select jobs to execute...

[Thu Jan 26 16:07:39 2023]
rule download_db_file:
output: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/nt/virus_primary_nt/sequenceDB
jobid: 45
reason: Forced execution
wildcards: file=nt/virus_primary_nt/sequenceDB
resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

Traceback (most recent call last):
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in call
return func(*args, **kwargs)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/cli/main.py", line 76, in _main
init_loggers(context)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/cli/main.py", line 58, in init_loggers
if context and context.json:
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in get
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 974, in load
match.value(self._element_type),
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 267, in value
return make_immutable(self._raw_value)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/collection.py", line 24, in make_immutable
elif isiterable(value):
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/compat.py", line 37, in isiterable
return not isinstance(obj, string_types) and isinstance(obj, collections.Iterable)
AttributeError: module 'collections' has no attribute 'Iterable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/lustre/project/taw/share/conda-envs/hecatomb/bin/conda", line 13, in
sys.exit(main())
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/cli/main.py", line 150, in main
return conda_exception_handler(_main, *args, **kwargs)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1371, in conda_exception_handler
return_value = exception_handler(func, *args, **kwargs)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1082, in call
return self.handle_exception(exc_val, exc_tb)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1126, in handle_exception
return self.handle_unexpected_exception(exc_val, exc_tb)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1137, in handle_unexpected_exception
self.print_unexpected_error_report(error_report)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1193, in print_unexpected_error_report
if context.json:
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in get
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 974, in load
match.value(self._element_type),
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 267, in value
return make_immutable(self._raw_value)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/collection.py", line 24, in make_immutable
elif isiterable(value):
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/compat.py", line 37, in isiterable
return not isinstance(obj, string_types) and isinstance(obj, collections.Iterable)
AttributeError: module 'collections' has no attribute 'Iterable'
Traceback (most recent call last):
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/init.py", line 757, in snakemake
success = workflow.execute(
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/workflow.py", line 1089, in execute
raise e
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/workflow.py", line 1085, in execute
success = self.scheduler.schedule()
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/scheduler.py", line 592, in schedule
self.run(runjobs)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/scheduler.py", line 641, in run
executor.run_jobs(
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 155, in run_jobs
self.run(
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 1156, in run
self.write_jobscript(job, jobscript)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 884, in write_jobscript
exec_job = self.format_job_exec(job)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 442, in format_job_exec
self.general_args,
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/common/init.py", line 218, in get
value = self.method(instance)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 334, in general_args
w2a("conda_base_path", skip=not self.assume_shared_fs),
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 294, in workflow_property_to_arg
value = getattr(self.workflow, property)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/workflow.py", line 300, in conda_base_path
return Conda().prefix_path
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 667, in init
shell.check_output(self._get_cmd(f"conda info --json"), text=True)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/shell.py", line 63, in check_output
return sp.check_output(cmd, shell=True, executable=executable, **kwargs)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 1.
[2023:01:26 16:07:41] Error: Snakemake failed

from hecatomb.

beardymcjohnface commented on September 26, 2024

This is a weird issue and I'm guessing there's something wrong with the environment. Did you load anaconda as a module on your HPC?
collections.iterable was deprecated in 3.10 and it looks like the workflow is calling packages from both 3.8 and 3.10. I would suggest trying a fresh install of miniconda on your home directory with python 3.10 if that's possible.

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

Looks like it worked when I load the module right before running! Thanks

(base) [kvigil@cypress1 ~]$ module load anaconda3/2020.07
(base) [kvigil@cypress1 ~]$ conda activate hecatomb
(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress1 ~]$ hecatomb install --profile slurm

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗
██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗
███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝
██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗
██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝
╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

[2023:01:27 07:37:56] Config file hecatomb.config.yaml already exists. Using existing config file.
[2023:01:27 07:37:56] Writing runtime config file to hecatomb.config.yaml
[2023:01:27 07:37:56] ------------------
[2023:01:27 07:37:56] | Runtime config |
[2023:01:27 07:37:56] ------------------

BigJobCpu: 24
BigJobMem: 64000
BigJobTimeMin: 1440
COMPRESSION: 1
CONTIG_MINLENGTH: 1000
CUTTAIL_WINDOW: 25
DEDUP_ACCURACY: 4
Databases: null
ENTROPY: 0.5
ENTROPYWINDOW: 25
Host: human
MediumJobCpu: 16
MediumJobMem: 32000
MoreRamCpu: 2
MoreRamMem: 16000
Output: hecatomb.out
Preprocess: paired
QSCORE: 15
READ_MINLENGTH: 90
Reads: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data
Search: sensitive
SmallJobCpu: 8
SmallJobMem: 16000
canuSettings: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False genomeSize=10M
-nanopore
filtAAprimary: --min-length 30 -e 1e-3
filtAAsecondary: --min-length 30 -e 1e-5
filtNTprimary: --min-length 90 -e 1e-10
filtNTsecondary: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode
3
perfAA: --start-sens 1 --sens-steps 3 -s 7 --lca-mode 2 --shuffle 0
perfAAfast: -s 4.0 --lca-mode 2 --shuffle 0
perfNT: --start-sens 2 -s 7 --sens-steps 3
perfNTfast: -s 4.0
taxIdIgnore: 0 1 2 10239 131567 12429 2759

[2023:01:27 07:37:56] ---------------------
[2023:01:27 07:37:56] | Snakemake command |
[2023:01:27 07:37:56] ---------------------

snakemake -s /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/workflow/DownloadDB.smk --configfile hecatomb.config.yaml --rerun-incomplete --printshellcmds --nolock --show-failed-logs --profile slurm
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Nothing to be done (all requested files are present and up to date).
Complete log: .snakemake/log/2023-01-27T073806.383518.snakemake.log
[2023:01:27 07:38:09] Snakemake finished successfully

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

So looks like when I run the HPC test I get a snakemake failed error:

(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress1 ~]$ hecatomb test --profile slurm

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗
██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗
███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝
██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗
██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝
╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

[2023:01:27 07:40:59] Config file hecatomb.config.yaml already exists. Using existing config file.
[2023:01:27 07:40:59] Writing runtime config file to hecatomb.config.yaml
[2023:01:27 07:40:59] ------------------
[2023:01:27 07:40:59] | Runtime config |
[2023:01:27 07:40:59] ------------------

BigJobCpu: 24
BigJobMem: 64000
BigJobTimeMin: 1440
COMPRESSION: 1
CONTIG_MINLENGTH: 1000
CUTTAIL_WINDOW: 25
DEDUP_ACCURACY: 4
Databases: null
ENTROPY: 0.5
ENTROPYWINDOW: 25
Host: human
MediumJobCpu: 16
MediumJobMem: 32000
MoreRamCpu: 2
MoreRamMem: 16000
Output: hecatomb.out
Preprocess: paired
QSCORE: 15
READ_MINLENGTH: 90
Reads: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data
Search: sensitive
SmallJobCpu: 8
SmallJobMem: 16000
canuSettings: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False genomeSize=10M
-nanopore
filtAAprimary: --min-length 30 -e 1e-3
filtAAsecondary: --min-length 30 -e 1e-5
filtNTprimary: --min-length 90 -e 1e-10
filtNTsecondary: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode
3
perfAA: --start-sens 1 --sens-steps 3 -s 7 --lca-mode 2 --shuffle 0
perfAAfast: -s 4.0 --lca-mode 2 --shuffle 0
perfNT: --start-sens 2 -s 7 --sens-steps 3
perfNTfast: -s 4.0
taxIdIgnore: 0 1 2 10239 131567 12429 2759

[2023:01:27 07:40:59] ---------------------
[2023:01:27 07:40:59] | Snakemake command |
[2023:01:27 07:40:59] ---------------------

snakemake -s /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/workflow/Hecatomb.smk --configfile hecatomb.config.yaml --use-conda --conda-frontend mamba --conda-prefix /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/conda --rerun-incomplete --printshellcmds --nolock --show-failed-logs --profile slurm
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/immutable.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Traceback (most recent call last):
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in call
return func(*args, **kwargs)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/cli/main.py", line 76, in _main
init_loggers(context)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/cli/main.py", line 58, in init_loggers
if context and context.json:
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in get
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 974, in load
match.value(self._element_type),
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 267, in value
return make_immutable(self._raw_value)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/collection.py", line 24, in make_immutable
elif isiterable(value):
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/compat.py", line 37, in isiterable
return not isinstance(obj, string_types) and isinstance(obj, collections.Iterable)
AttributeError: module 'collections' has no attribute 'Iterable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/lustre/project/taw/share/conda-envs/hecatomb/bin/conda", line 13, in
sys.exit(main())
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/cli/main.py", line 150, in main
return conda_exception_handler(_main, *args, **kwargs)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1371, in conda_exception_handler
return_value = exception_handler(func, *args, **kwargs)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1082, in call
return self.handle_exception(exc_val, exc_tb)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1126, in handle_exception
return self.handle_unexpected_exception(exc_val, exc_tb)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1137, in handle_unexpected_exception
self.print_unexpected_error_report(error_report)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/exceptions.py", line 1193, in print_unexpected_error_report
if context.json:
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in get
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 1207, in
matches = [self.type.load(self.name, match) for match in raw_matches]
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 974, in load
match.value(self._element_type),
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/common/configuration.py", line 267, in value
return make_immutable(self._raw_value)
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/collection.py", line 24, in make_immutable
elif isiterable(value):
File "/share/apps/anaconda/3/2020.07/lib/python3.8/site-packages/conda/_vendor/auxlib/compat.py", line 37, in isiterable
return not isinstance(obj, string_types) and isinstance(obj, collections.Iterable)
AttributeError: module 'collections' has no attribute 'Iterable'
Traceback (most recent call last):
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/init.py", line 757, in snakemake
success = workflow.execute(
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/workflow.py", line 928, in execute
dag.create_conda_envs(
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/dag.py", line 338, in create_conda_envs
env.create(dryrun)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 405, in create
pin_file = self.pin_file
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/common/init.py", line 218, in get
value = self.method(instance)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 109, in pin_file
f".{self.conda.platform}.pin.txt"
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/common/init.py", line 218, in get
value = self.method(instance)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 102, in conda
return Conda(
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 667, in init
shell.check_output(self._get_cmd(f"conda info --json"), text=True)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/site-packages/snakemake/shell.py", line 63, in check_output
return sp.check_output(cmd, shell=True, executable=executable, **kwargs)
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/lustre/project/taw/share/conda-envs/hecatomb/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 1.
[2023:01:27 07:41:06] Error: Snakemake failed

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

2023-01-27T102534.391649.snakemake.log
2023-01-27T102517.189801.snakemake.log
2023-01-27T102344.153011.snakemake.log
2023-01-27T102208.438015.snakemake.log
2023-01-27T102146.386328.snakemake.log
2023-01-27T102120.713968.snakemake.log
2023-01-27T074100.928790.snakemake.log
2023-01-27T073806.383518.snakemake.log

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

Hi did anybody checkout these logs? Thanks katie

from hecatomb.

beardymcjohnface commented on September 26, 2024

Hi,
Sorry I meant to follow up on this. I'm not sure which of those logs is the latest, but based on what you copy pasted it looks like the same issue as before and I think the anaconda module might be too old. I would try installing miniconda with python 3.10. You should be able to install a local copy to your home directory even on an HPC.

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

@beardymcjohnface Hi thank you! I will definitely try this out. Thanks for the response.

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

Hi I am still getting a snakemake error. Also, it doesn't look like this run was saved in a log like it stated below. Thanks!

(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress2 conda-envs]$ unset PYTHONPATH
(/lustre/project/taw/share/conda-envs/hecatomb) [kvigil@cypress2 conda-envs]$ hecatomb test --profile slurm

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗
██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗
███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝
██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗
██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝
╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

[2023:03:06 13:20:24] Config file hecatomb.config.yaml already exists. Using existing config file.
[2023:03:06 13:20:24] Writing runtime config file to hecatomb.config.yaml
[2023:03:06 13:20:24] ------------------
[2023:03:06 13:20:24] | Runtime config |
[2023:03:06 13:20:24] ------------------

BigJobCpu: 24
BigJobMem: 64000
BigJobTimeMin: 1440
COMPRESSION: 1
CONTIG_MINLENGTH: 1000
CUTTAIL_WINDOW: 25
DEDUP_ACCURACY: 4
Databases: null
ENTROPY: 0.5
ENTROPYWINDOW: 25
Host: human
MediumJobCpu: 16
MediumJobMem: 32000
MoreRamCpu: 2
MoreRamMem: 16000
Output: hecatomb.out
Preprocess: paired
QSCORE: 15
READ_MINLENGTH: 90
Reads: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data
Search: sensitive
SmallJobCpu: 8
SmallJobMem: 16000
canuSettings: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False genomeSize=10M
-nanopore
filtAAprimary: --min-length 30 -e 1e-3
filtAAsecondary: --min-length 30 -e 1e-5
filtNTprimary: --min-length 90 -e 1e-10
filtNTsecondary: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode
3
perfAA: --start-sens 1 --sens-steps 3 -s 7 --lca-mode 2 --shuffle 0
perfAAfast: -s 4.0 --lca-mode 2 --shuffle 0
perfNT: --start-sens 2 -s 7 --sens-steps 3
perfNTfast: -s 4.0
taxIdIgnore: 0 1 2 10239 131567 12429 2759

[2023:03:06 13:20:24] ---------------------
[2023:03:06 13:20:24] | Snakemake command |
[2023:03:06 13:20:24] ---------------------

snakemake -s /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/workflow/Hecatomb.smk --configfile hecatomb.config.yaml --use-conda --conda-frontend mamba --conda-prefix /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/conda --rerun-incomplete --printshellcmds --nolock --show-failed-logs --profile slurm
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/immutable.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Using shell: /bin/bash
Provided cluster nodes: 100
Job stats:
job count min threads max threads

PRIMARY_AA_parsing 1 2 2
PRIMARY_AA_taxonomy_assignment 1 24 24
PRIMARY_NT_parsing 1 2 2
PRIMARY_NT_reformat 1 2 2
PRIMARY_NT_taxonomic_assignment 1 24 24
SECONDARY_AA_generate_output_table 1 2 2
SECONDARY_AA_parsing 1 2 2
SECONDARY_AA_refactor_finalize 1 2 2
SECONDARY_AA_taxonomy_assignment 1 24 24
SECONDARY_AA_tophit_lineage 1 2 2
SECONDARY_NT_convert 1 24 24
SECONDARY_NT_generate_output_table 1 2 2
SECONDARY_NT_summary 1 2 2
SECONDARY_NT_taxonomic_assignment 1 24 24
all 1 1 1
archive_for_assembly 10 1 1
assembly_kmer_normalization 10 8 8
bam_index 1 2 2
calculate_gc 2 2 2
calculate_tet_freq 2 2 2
cluster_similar_sequences 10 24 24
combine_AA_NT 1 1 1
concatenate_contigs 1 1 1
concatentate_contig_count_tables 1 1 1
contig_krona_plot 1 1 1
contig_krona_text_format 1 1 1
contig_read_taxonomy 1 2 2
coverage_calculations 10 8 8
create_contig_count_table 10 1 1
create_host_index 1 8 8
create_individual_seqtables 10 24 24
dumpSamplesTsv 1 1 1
fastp_preprocessing 4 16 16
host_removal_mapping 10 8 8
individual_sample_assembly 10 16 16
krona_plot 1 1 1
krona_text_format 1 1 1
link_assembly 1 1 1
mapSampleAssemblyPairedReads 10 8 8
mapSampleAssemblyUnpairedReads 10 8 8
map_seq_table 1 16 16
merge_seq_table 1 1 1
mmseqs_contig_annotation 1 24 24
mmseqs_contig_annotation_summary 1 24 24
nonhost_read_combine 10 1 1
nonhost_read_repair 10 8 8
poolR1Unmapped 1 1 1
poolR2Unmapped 1 1 1
poolUnpairedUnmapped 1 1 1
population_assembly 1 16 16
pullPairedUnmappedReads 10 8 8
pullPairedUnmappedReadsMateMapped 10 8 8
rescue_read_kmer_normalization 1 24 24
secondary_nt_calc_lca 1 24 24
secondary_nt_lca_table 1 1 1
seq_properties_table 2 2 2
tax_level_counts 1 2 2
unmapped_read_rescue_assembly 1 16 16
zip_fastq 60 1 1
total 250 1 24

Select jobs to execute...

[Mon Mar 6 13:20:34 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log
jobid: 43
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-258-124-06_CGTACG.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq
wildcards: sample=A13-258-124-06_CGTACG
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log

Submitted job 43 with external jobid 'Submitted batch job 2287003'.

[Mon Mar 6 13:20:34 2023]
rule dumpSamplesTsv:
output: hecatomb.out/results/hecatomb.samples.tsv
jobid: 252
reason: Missing output files: hecatomb.out/results/hecatomb.samples.tsv
resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

Submitted job 252 with external jobid 'Submitted batch job 2287004'.

[Mon Mar 6 13:20:34 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log
jobid: 49
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-256-117-06_ACTGAT.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq
wildcards: sample=A13-256-117-06_ACTGAT
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log

Submitted job 49 with external jobid 'Submitted batch job 2287005'.

[Mon Mar 6 13:20:34 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log
jobid: 31
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-253-140-06_GTCCGC.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq
wildcards: sample=A13-253-140-06_GTCCGC
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log

Submitted job 31 with external jobid 'Submitted batch job 2287006'.

[Mon Mar 6 13:20:34 2023]
rule create_host_index:
input: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz
output: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx
log: hecatomb.out/stderr/create_host_index.log
jobid: 7
benchmark: hecatomb.out/benchmarks/create_host_index.txt
reason: Missing output files: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx
threads: 8
resources: mem_mb=16000, mem_mib=15259, disk_mb=1939, disk_mib=1850, tmpdir=, time=1440

    minimap2 -t 8 -d /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx <(cat /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz) 2> hecatomb.out/stderr/create_host_index.log
    rm hecatomb.out/stderr/create_host_index.log

Submitted job 7 with external jobid 'Submitted batch job 2287007'.

[Mon Mar 6 13:20:34 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log
jobid: 55
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-256-115-06_GTTTCG.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq
wildcards: sample=A13-256-115-06_GTTTCG
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log

Submitted job 55 with external jobid 'Submitted batch job 2287008'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:44 2023]
Error in rule fastp_preprocessing:
jobid: 43
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287003

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 43, external: Submitted batch job 2287003, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.43.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 43.
Select jobs to execute...

[Mon Mar 6 13:20:44 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log
jobid: 43
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-258-124-06_CGTACG.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq
wildcards: sample=A13-258-124-06_CGTACG
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log

Submitted job 43 with external jobid 'Submitted batch job 2287009'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:44 2023]
Error in rule dumpSamplesTsv:
jobid: 252
output: hecatomb.out/results/hecatomb.samples.tsv
cluster_jobid: Submitted batch job 2287004

Error executing rule dumpSamplesTsv on cluster (jobid: 252, external: Submitted batch job 2287004, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.dumpSamplesTsv.252.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 252.
Select jobs to execute...

[Mon Mar 6 13:20:44 2023]
rule dumpSamplesTsv:
output: hecatomb.out/results/hecatomb.samples.tsv
jobid: 252
reason: Missing output files: hecatomb.out/results/hecatomb.samples.tsv
resources: mem_mb=2000, mem_mib=1908, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

Submitted job 252 with external jobid 'Submitted batch job 2287010'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:45 2023]
Error in rule fastp_preprocessing:
jobid: 49
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287005

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 49, external: Submitted batch job 2287005, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.49.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 49.
Select jobs to execute...

[Mon Mar 6 13:20:45 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log
jobid: 49
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-256-117-06_ACTGAT.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq
wildcards: sample=A13-256-117-06_ACTGAT
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log

Submitted job 49 with external jobid 'Submitted batch job 2287011'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:45 2023]
Error in rule fastp_preprocessing:
jobid: 31
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287006

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 31, external: Submitted batch job 2287006, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.31.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 31.
Select jobs to execute...

[Mon Mar 6 13:20:45 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log
jobid: 31
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-253-140-06_GTCCGC.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq
wildcards: sample=A13-253-140-06_GTCCGC
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log

Submitted job 31 with external jobid 'Submitted batch job 2287012'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:45 2023]
Error in rule create_host_index:
jobid: 7
input: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz
output: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx
log: hecatomb.out/stderr/create_host_index.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4ea269956b660de957aed37e1acd2903_
shell:

    minimap2 -t 8 -d /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx <(cat /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz) 2> hecatomb.out/stderr/create_host_index.log
    rm hecatomb.out/stderr/create_host_index.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287007

Logfile hecatomb.out/stderr/create_host_index.log not found.

Error executing rule create_host_index on cluster (jobid: 7, external: Submitted batch job 2287007, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.create_host_index.7.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 7.
Select jobs to execute...

[Mon Mar 6 13:20:45 2023]
rule create_host_index:
input: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz
output: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx
log: hecatomb.out/stderr/create_host_index.log
jobid: 7
benchmark: hecatomb.out/benchmarks/create_host_index.txt
reason: Missing output files: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx
threads: 8
resources: mem_mb=16000, mem_mib=15259, disk_mb=1939, disk_mib=1850, tmpdir=, time=1440

    minimap2 -t 8 -d /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx <(cat /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz) 2> hecatomb.out/stderr/create_host_index.log
    rm hecatomb.out/stderr/create_host_index.log

Submitted job 7 with external jobid 'Submitted batch job 2287013'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:45 2023]
Error in rule fastp_preprocessing:
jobid: 55
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287008

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 55, external: Submitted batch job 2287008, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.55.sh). For error details see the cluster log and the log files of the involved rule(s).
Trying to restart job 55.
Select jobs to execute...

[Mon Mar 6 13:20:45 2023]
rule fastp_preprocessing:
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log
jobid: 55
benchmark: hecatomb.out/benchmarks/fastp_preprocessing.A13-256-115-06_GTTTCG.txt
reason: Missing output files: hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq
wildcards: sample=A13-256-115-06_GTTTCG
threads: 16
resources: mem_mb=32000, mem_mib=30518, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log

Submitted job 55 with external jobid 'Submitted batch job 2287014'.
Invalid jobs list: Submitted
[Mon Mar 6 13:20:55 2023]
Error in rule fastp_preprocessing:
jobid: 43
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-258-124-06_CGTACG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-258-124-06_CGTACG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-258-124-06_CGTACG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287009

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-258-124-06_CGTACG.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 43, external: Submitted batch job 2287009, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.43.sh). For error details see the cluster log and the log files of the involved rule(s).
Invalid jobs list: Submitted
[Mon Mar 6 13:20:55 2023]
Error in rule dumpSamplesTsv:
jobid: 252
output: hecatomb.out/results/hecatomb.samples.tsv
cluster_jobid: Submitted batch job 2287010

Error executing rule dumpSamplesTsv on cluster (jobid: 252, external: Submitted batch job 2287010, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.dumpSamplesTsv.252.sh). For error details see the cluster log and the log files of the involved rule(s).
Invalid jobs list: Submitted
[Mon Mar 6 13:20:56 2023]
Error in rule fastp_preprocessing:
jobid: 49
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-117-06_ACTGAT_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-117-06_ACTGAT_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-117-06_ACTGAT.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287011

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-256-117-06_ACTGAT.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 49, external: Submitted batch job 2287011, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.49.sh). For error details see the cluster log and the log files of the involved rule(s).
Invalid jobs list: Submitted
[Mon Mar 6 13:20:56 2023]
Error in rule fastp_preprocessing:
jobid: 31
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json, hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-253-140-06_GTCCGC_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-253-140-06_GTCCGC_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-253-140-06_GTCCGC.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287012

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-253-140-06_GTCCGC.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 31, external: Submitted batch job 2287012, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.31.sh). For error details see the cluster log and the log files of the involved rule(s).
Invalid jobs list: Submitted
[Mon Mar 6 13:20:56 2023]
Error in rule create_host_index:
jobid: 7
input: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz
output: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx
log: hecatomb.out/stderr/create_host_index.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4ea269956b660de957aed37e1acd2903_
shell:

    minimap2 -t 8 -d /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz.idx <(cat /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/host/human/masked_ref.fa.gz) 2> hecatomb.out/stderr/create_host_index.log
    rm hecatomb.out/stderr/create_host_index.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287013

Logfile hecatomb.out/stderr/create_host_index.log not found.

Error executing rule create_host_index on cluster (jobid: 7, external: Submitted batch job 2287013, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.create_host_index.7.sh). For error details see the cluster log and the log files of the involved rule(s).
Invalid jobs list: Submitted
[Mon Mar 6 13:20:56 2023]
Error in rule fastp_preprocessing:
jobid: 55
input: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz, /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../../databases/contaminants/vector_contaminants.fa
output: hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json, hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html
log: hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log (check log file(s) for error details)
conda-env: /lustre/project/taw/share/conda-envs/hecatomb/snakemake/conda/4fabc0d9baac0a21fd5dfed339929234_
shell:

    fastp -i /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R1.fastq.gz -I /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data/A13-256-115-06_GTTTCG_R2.fastq.gz -o hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R1.s1.out.fastq -O hecatomb.out/processing/temp/p01/A13-256-115-06_GTTTCG_R2.s1.out.fastq             -z 1             -j hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.json -h hecatomb.out/processing/stats/p01/A13-256-115-06_GTTTCG.s1.stats.html             --qualified_quality_phred 15             --length_required 90             --detect_adapter_for_pe             --cut_tail --cut_tail_window_size 25 --cut_tail_mean_quality 15             --dedup --dup_calc_accuracy 4             --trim_poly_x             --thread 16 2> hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log
    rm hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 2287014

Logfile hecatomb.out/stderr/fastp_preprocessing.A13-256-115-06_GTTTCG.log not found.

Error executing rule fastp_preprocessing on cluster (jobid: 55, external: Submitted batch job 2287014, jobscript: /lustre/project/taw/share/conda-envs/.snakemake/tmp.skvkwg2w/snakejob.fastp_preprocessing.55.sh). For error details see the cluster log and the log files of the involved rule(s).
Exiting because a job execution failed. Look above for error message

FATAL: Hecatomb encountered an error.
       Dumping all error logs to "hecatomb.errorLogs.txt"Complete log: .snakemake/log/2023-03-06T132026.448612.snakemake.log

[2023:03:06 13:21:06] Error: Snakemake failed

from hecatomb.

beardymcjohnface commented on September 26, 2024

Progress! I think this is the same issue as #87, can you check your profile config.yaml file to make sure you've got --parsable for sbatch?

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

2023-03-06T145158.281223.snakemake.log
2023-03-06T145440.340562.snakemake.log
2023-03-06T145809.129136.snakemake.log
2023-03-06T145820.167039.snakemake.log
2023-03-06T130924.336960.snakemake.log
2023-03-06T132026.448612.snakemake.log

Here are all my logs if this helps

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

Here is what that file says:

#####################################################

| | _ _ | | ___ _ _ | |__

| '_ \ / _ / / `| / _ \| '_` _ | ' \

| | | | __/ (| (| | || () | | | | | | |) |

|| ||_|,|__/|| || ||_.__/

#####################################################

For more information see: https://github.com/shandley/hecatomb and https://hecatomb.readthedocs.io/en/latest/

##################

Run Parameters

##################

Inputs and outputs (specify Reads here or on the command line, Output blank = use default)

Reads:
Output:

Database installation location, leave blank = use hecatomb install location

Databases:

STICK TO YOUR SYSTEM'S CPU:RAM RATIO FOR THESE

BigJobMem: 64000 # Memory for MMSeqs in megabytes (e.g 64GB = 64000, recommend >= 64000)
BigJobCpu: 24 # Threads for MMSeqs (recommend >= 16)
BigJobTimeMin: 1440 # Max runtime in minutes for MMSeqs (this is only enforced by the Snakemake profile)
MediumJobMem: 32000 # Memory for Megahit/Flye in megabytes (recommend >= 32000)
MediumJobCpu: 16 # CPUs for Megahit/Flye in megabytes (recommend >= 16)
SmallJobMem: 16000 # Memory for BBTools etc. in megabytes (recommend >= 16000)
SmallJobCpu: 8 # CPUs for BBTools etc. (recommend >= 8)

Some jobs need more RAM; go over your CPU:RAM ratio if needed

MoreRamMem: 16000 # Memory for slightly RAM-hungry jobs in megabytes (recommend >= 16000)
MoreRamCpu: 2 # CPUs for slightly RAM-hungry jobs (recommend >= 2)

############################

Optional Rule Parameters

############################

Preprocessing

QSCORE: 15 # Read quality trimming score (rule fastp_preprocessing in 01_preprocessing.smk)
READ_MINLENGTH: 90 # Minimum read length during QC steps (rule fastp_preprocessing in 01_preprocessing.smk)
CONTIG_MINLENGTH: 1000 # Read minimum length (rule contig_reformating_and_stats in 01_preprocessing.smk)
CUTTAIL_WINDOW: 25 # Sliding window size for low qual read filter rule fastp_preprocessing in 01_preprocessing.smk)
DEDUP_ACCURACY: 4 # Specify the level (1 ~ 6). The higher level means more memory usage and more running time, but lower risk of incorrect deduplication marking (rule fastp_preprocessing in 01_preprocessing.smk)
COMPRESSION: 1 # Compression level for gzip output (1 ~ 9). 1 is fastest, 9 is smallest. Default is 1, based on assumption of large scratch space (rule fastp_preprocessing in 01_preprocessing.smk)
ENTROPY: 0.5 # Read minimum entropy (rule remove_low_quality in 01_preprocessing.smk)
ENTROPYWINDOW: 25 # entropy window for low qual read filter

CLUSTER READS TO SEQTABLE (MMSEQS EASY-LINCLUST)

-c = req coverage of target seq

--min-seq-id = req identity [0-1] of alignment

linclustParams:
--kmer-per-seq-scale 0.3
-c 0.8
--cov-mode 1
--min-seq-id 0.97
--alignment-mode 3

If the following LCA-calculated TaxIDs are encountered, defer to the top hit TaxID

0 = unclassified

1 = root

2 = bacteria root

10239 = virus root

131567 = cell org root

12429 = unclass virus

2759 = eukaryota root

12333 = unclass phage

taxIdIgnore: 0 1 2 10239 131567 12429 2759

###########################################

Canu settings (for longread assemblies)

###########################################

recommended correctedErrorRate is 0.16 to 0.12 (depending on coverage) for nanopore and

0.105 to 0.040 (depending on coverage) for pacbio (non-HiFi) - below is low-coverage nanopore

https://canu.readthedocs.io/en/latest/faq.html#what-parameters-can-i-tweak

canuSettings:
correctedErrorRate=0.16
maxInputCoverage=10000
minInputCoverage=0
corOutCoverage=10000
corMhapSensitivity=high
corMinCoverage=0
useGrid=False
stopOnLowCoverage=False
genomeSize=10M
-nanopore

-pacbio

-pacbio-hifi

###################

MMSeqs settings

###################

ALIGNMENT FILTERING CUTOFFS

--min-length for AA should be equal or less than 1/3 of READ_MINLENGTH

--min-length for NT should be equal or less than READ_MINLENGTH

filtAAprimary:
--min-length 30
-e 1e-3
filtAAsecondary:
--min-length 30
-e 1e-5
filtNTprimary:
--min-length 90
-e 1e-10
filtNTsecondary:
--min-length 90
-e 1e-20

PERFORMANCE SETTINGS - SEE MMSEQS DOCUMENTATION FOR DETAILS

sensitive AA search

perfAA:
--start-sens 1
--sens-steps 3
-s 7
--lca-mode 2
--shuffle 0

fast AA search

perfAAfast:
-s 4.0
--lca-mode 2
--shuffle 0

sensitive NT search

perfNT:
--start-sens 2
-s 7
--sens-steps 3

fast NT search

perfNTfast:
-s 4.0

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

OOPs I found the correct yam file: I added --parsable at the end of sbatch and it is running now!

cluster:
mkdir -p logs/{rule}/ &&
sbatch
--cpus-per-task={threads}
--mem={resources.mem_mb}
--time={resources.time}
--job-name=smk-{rule}
--output=logs/{rule}/{jobid}.out
--error=logs/{rule}/{jobid}.err
--parsable
default-resources:

mem_mb=2000
time=1440
jobs: 100
latency-wait: 60
local-cores: 8
restart-times: 1
max-jobs-per-second: 20
keep-going: True
rerun-incomplete: True
printshellcmds: True
scheduler: greedy
use-conda: True
conda-frontend: mamba
cluster-status: ~/.config/snakemake/slurm/slurm-status.py
max-status-checks-per-second: 10

from hecatomb.

lucyintheskyzzz commented on September 26, 2024

Hi I am still getting an error with the slurm test:

(base) [kvigil@cypress01-123 ~]$ hecatomb test --profile slurm

██╗ ██╗███████╗ ██████╗ █████╗ ████████╗ ██████╗ ███╗ ███╗██████╗
██║ ██║██╔════╝██╔════╝██╔══██╗╚══██╔══╝██╔═══██╗████╗ ████║██╔══██╗
███████║█████╗ ██║ ███████║ ██║ ██║ ██║██╔████╔██║██████╔╝
██╔══██║██╔══╝ ██║ ██╔══██║ ██║ ██║ ██║██║╚██╔╝██║██╔══██╗
██║ ██║███████╗╚██████╗██║ ██║ ██║ ╚██████╔╝██║ ╚═╝ ██║██████╔╝
╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═════╝

Hecatomb version v1.1.0

[2023:03:09 11:54:15] Config file hecatomb.config.yaml already exists. Using existing config file.
[2023:03:09 11:54:15] Writing runtime config file to hecatomb.config.yaml
[2023:03:09 11:54:15] ------------------
[2023:03:09 11:54:15] | Runtime config |
[2023:03:09 11:54:15] ------------------

BigJobCpu: 24
BigJobMem: 64000
BigJobTimeMin: 1440
COMPRESSION: 1
CONTIG_MINLENGTH: 1000
CUTTAIL_WINDOW: 25
DEDUP_ACCURACY: 4
Databases: null
ENTROPY: 0.5
ENTROPYWINDOW: 25
Host: human
MediumJobCpu: 16
MediumJobMem: 32000
MoreRamCpu: 2
MoreRamMem: 16000
Output: hecatomb.out
Preprocess: paired
QSCORE: 15
READ_MINLENGTH: 90
Reads: /lustre/project/taw/share/conda-envs/hecatomb/bin/../test_data
Search: sensitive
SmallJobCpu: 8
SmallJobMem: 16000
canuSettings: correctedErrorRate=0.16 maxInputCoverage=10000 minInputCoverage=0 corOutCoverage=10000
corMhapSensitivity=high corMinCoverage=0 useGrid=False stopOnLowCoverage=False genomeSize=10M
-nanopore
filtAAprimary: --min-length 30 -e 1e-3
filtAAsecondary: --min-length 30 -e 1e-5
filtNTprimary: --min-length 90 -e 1e-10
filtNTsecondary: --min-length 90 -e 1e-20
linclustParams: --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode
3
perfAA: --start-sens 1 --sens-steps 3 -s 7 --lca-mode 2 --shuffle 0
perfAAfast: -s 4.0 --lca-mode 2 --shuffle 0
perfNT: --start-sens 2 -s 7 --sens-steps 3
perfNTfast: -s 4.0
taxIdIgnore: 0 1 2 10239 131567 12429 2759

[2023:03:09 11:54:15] ---------------------
[2023:03:09 11:54:15] | Snakemake command |
[2023:03:09 11:54:15] ---------------------

snakemake -s /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/workflow/Hecatomb.smk --configfile hecatomb.config.yaml --use-conda --conda-frontend mamba --conda-prefix /lustre/project/taw/share/conda-envs/hecatomb/bin/../snakemake/conda --rerun-incomplete --printshellcmds --nolock --show-failed-logs --profile slurm
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/dbFiles.yaml is extended by additional config specified via the command line.
Config file /lustre/project/taw/share/conda-envs/hecatomb/snakemake/workflow/../config/immutable.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Using shell: /bin/bash
Provided cluster nodes: 100
Job stats:
job count min threads max threads

PRIMARY_AA_parsing 1 2 2
PRIMARY_AA_taxonomy_assignment 1 24 24
PRIMARY_NT_parsing 1 2 2
PRIMARY_NT_reformat 1 2 2
PRIMARY_NT_taxonomic_assignment 1 24 24
SECONDARY_AA_generate_output_table 1 2 2
SECONDARY_AA_parsing 1 2 2
SECONDARY_AA_refactor_finalize 1 2 2
SECONDARY_AA_taxonomy_assignment 1 24 24
SECONDARY_AA_tophit_lineage 1 2 2
SECONDARY_NT_convert 1 24 24
SECONDARY_NT_generate_output_table 1 2 2
SECONDARY_NT_summary 1 2 2
SECONDARY_NT_taxonomic_assignment 1 24 24
all 1 1 1
bam_index 1 2 2
calculate_gc 2 2 2
calculate_tet_freq 2 2 2
cluster_similar_sequences 10 24 24
combine_AA_NT 1 1 1
concatenate_contigs 1 1 1
concatentate_contig_count_tables 1 1 1
contig_krona_plot 1 1 1
contig_krona_text_format 1 1 1
contig_read_taxonomy 1 2 2
coverage_calculations 10 8 8
create_contig_count_table 10 1 1
create_individual_seqtables 10 24 24
krona_plot 1 1 1
krona_text_format 1 1 1
link_assembly 1 1 1
map_seq_table 1 16 16
merge_seq_table 1 1 1
mmseqs_contig_annotation 1 24 24
mmseqs_contig_annotation_summary 1 24 24
population_assembly 1 16 16
rescue_read_kmer_normalization 1 24 24
secondary_nt_calc_lca 1 24 24
secondary_nt_lca_table 1 1 1
seq_properties_table 2 2 2
tax_level_counts 1 2 2
unmapped_read_rescue_assembly 1 16 16
total 81 1 24

Select jobs to execute...

[Thu Mar 9 11:54:19 2023]
rule cluster_similar_sequences:
input: hecatomb.out/processing/temp/p04/A13-135-177-06_AGTTCC_R1.all.fastq
output: hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_R1_rep_seq.fasta, hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_R1_cluster.tsv, hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_R1_all_seqs.fasta
log: hecatomb.out/stderr/cluster_similar_sequences.A13-135-177-06_AGTTCC.log
jobid: 10
benchmark: hecatomb.out/benchmarks/cluster_similar_sequences.A13-135-177-06_AGTTCC.txt
reason: Missing output files: hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_R1_cluster.tsv, hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_R1_rep_seq.fasta; Updated input files: hecatomb.out/processing/temp/p04/A13-135-177-06_AGTTCC_R1.all.fastq
wildcards: sample=A13-135-177-06_AGTTCC
threads: 24
resources: mem_mb=64000, mem_mib=61036, disk_mb=1000, disk_mib=954, tmpdir=, time=1440

    mmseqs easy-linclust hecatomb.out/processing/temp/p04/A13-135-177-06_AGTTCC_R1.all.fastq hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_R1 hecatomb.out/processing/temp/p05/A13-135-177-06_AGTTCC_TMP             --kmer-per-seq-scale 0.3 -c 0.8 --cov-mode 1 --min-seq-id 0.97 --alignment-mode 3             --threads 24 &> hecatomb.out/stderr/cluster_similar_sequences.A13-135-177-06_AGTTCC.log
    rm hecatomb.out/stderr/cluster_similar_sequences.A13-135-177-06_AGTTCC.log