When running the encode chip-seq-pipeline on HPC with SLURM, it consistently hangs on the line that reads "task=chip.read_genome_tsv:-1, retry=0, status=WaitingForReturnCode". I have tried running the pipeline with and without an active server and cannot overcome this error.
2021-05-13 12:39:33,587|caper.caper_base|INFO| Creating a timestamped temporary directory. /home/hpc/batchus1/caper_tmp/chip/20210513_123933_586540
2021-05-13 12:39:33,587|caper.caper_runner|INFO| Localizing files on work_dir. /home/hpc/batchus1/caper_tmp/chip/20210513_123933_586540
2021-05-13 12:39:35,039|autouri.autouri|INFO| cp: skipped due to name_size_match, size=872949833, mt=1549739698.0, src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz, dest=/home/hpc/batchus1/caper_tmp/caf534ed3cf684406e731d19be272b4a/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
2021-05-13 12:39:35,796|autouri.autouri|INFO| cp: skipped due to md5_match, md5=05297d96dd1f7cfb45a7b637d6dd7036, src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz, dest=/home/hpc/batchus1/caper_tmp/f43b63a83784d3ec8055f1a22168ed89/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz
2021-05-13 12:39:36,631|autouri.autouri|INFO| cp: skipped due to md5_match, md5=393688b4f06c9ce26165d47433dd8c37, src=https://www.encodeproject.org/files/ENCFF356LFX/@@download/ENCFF356LFX.bed.gz, dest=/home/hpc/batchus1/caper_tmp/f183dcba5d34f959d8b55ed438ee2e22/ENCFF356LFX.bed.gz
2021-05-13 12:39:38,246|autouri.autouri|INFO| cp: skipped due to md5_match, md5=c95303fb77cc3e11d50e3c3a4b93b3fb, src=https://www.encodeproject.org/files/GRCh38_EBV.chrom.sizes/@@download/GRCh38_EBV.chrom.sizes.tsv, dest=/home/hpc/batchus1/caper_tmp/c52f52c7bfa357f55a39b1de7e4d0b0c/GRCh38_EBV.chrom.sizes.tsv
2021-05-13 12:39:39,532|autouri.autouri|INFO| cp: skipped due to name_size_match, size=3749246230, mt=1571469011.0, src=https://www.encodeproject.org/files/ENCFF110MCL/@@download/ENCFF110MCL.tar.gz, dest=/home/hpc/batchus1/caper_tmp/3ff4ac4c3f59d096b1a3842a182072ae/ENCFF110MCL.tar.gz
2021-05-13 12:39:40,386|autouri.autouri|INFO| cp: skipped due to md5_match, md5=80b263f6ea6ff65d547eef07102535db, src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz, dest=/home/hpc/batchus1/caper_tmp/df5193e07055d13c48be59bacd0f56b8/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz
2021-05-13 12:39:41,713|autouri.autouri|INFO| cp: skipped due to name_size_match, size=4318261891, mt=1549723866.0, src=https://www.encodeproject.org/files/ENCFF643CGH/@@download/ENCFF643CGH.tar.gz, dest=/home/hpc/batchus1/caper_tmp/8c692fba4640609720272154ab0faa30/ENCFF643CGH.tar.gz
2021-05-13 12:39:42,490|autouri.autouri|INFO| cp: skipped due to md5_match, md5=7e088c24a017a43b1db5e8f50060eec1, src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index.tar.gz, dest=/home/hpc/batchus1/caper_tmp/d3dff25534e93d893902540d81e4f475/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index.tar.gz
2021-05-13 12:39:43,365|autouri.autouri|INFO| cp: skipped due to md5_match, md5=aca8cf959206aa3ad257fc46dc783266, src=https://www.encodeproject.org/files/ENCFF493CCB/@@download/ENCFF493CCB.bed.gz, dest=/home/hpc/batchus1/caper_tmp/0fa7d04b32e66fa02fb2c1ae39e41447/ENCFF493CCB.bed.gz
2021-05-13 12:39:44,640|autouri.autouri|INFO| cp: skipped due to name_size_match, size=14377496, mt=1592463730.0, src=https://www.encodeproject.org/files/ENCFF304XEX/@@download/ENCFF304XEX.bed.gz, dest=/home/hpc/batchus1/caper_tmp/805e179275a9c0fb7a37def40c4312d1/ENCFF304XEX.bed.gz
2021-05-13 12:39:45,485|autouri.autouri|INFO| cp: skipped due to md5_match, md5=91047588129069ff91ec1b0664179f8e, src=https://www.encodeproject.org/files/ENCFF140XLU/@@download/ENCFF140XLU.bed.gz, dest=/home/hpc/batchus1/caper_tmp/0cbd2c602ddad252bc39729fc8a29286/ENCFF140XLU.bed.gz
2021-05-13 12:39:46,764|autouri.autouri|INFO| cp: skipped due to name_size_match, size=18381891, mt=1592463727.0, src=https://www.encodeproject.org/files/ENCFF212UAV/@@download/ENCFF212UAV.bed.gz, dest=/home/hpc/batchus1/caper_tmp/1d3aa436b05f16a509edb94789c061d3/ENCFF212UAV.bed.gz
2021-05-13 12:39:46,923|autouri.autouri|INFO| cp: skipped due to md5_match, md5=df624401f76fbd4d651e736068c43a1a, src=https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz, dest=/home/hpc/batchus1/caper_tmp/3b39284516e676ea52238f0636c0bbbf/hg38_dnase_avg_fseq_signal_formatted.txt.gz
2021-05-13 12:39:46,996|autouri.autouri|INFO| cp: skipped due to md5_match, md5=ced0c653d28628654288f7a8ab052590, src=https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz, dest=/home/hpc/batchus1/caper_tmp/c73f434c3fa4f3f54bc2ecad09c065c2/hg38_celltype_compare_subsample.bed.gz
2021-05-13 12:39:47,086|autouri.autouri|INFO| cp: skipped due to md5_match, md5=3f7fd85ab9a4c6274f28c3e82a79c10d, src=https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt, dest=/home/hpc/batchus1/caper_tmp/a9745b33b4ffdd83d7d2c5a7d3c8036a/hg38_dnase_avg_fseq_signal_metadata.txt
2021-05-13 12:39:47,850|caper.cromwell|INFO| Validating WDL/inputs/imports with Womtool...
2021-05-13 12:39:51,199|caper.cromwell|INFO| Womtool validation passed.
2021-05-13 12:39:51,200|caper.caper_runner|INFO| launching run: wdl=/home/hpc/batchus1/chip-seq-pipeline2/chip.wdl, inputs=/home/hpc/batchus1/caper_tmp/home/hpc/batchus1/param_short.local.json, backend_conf=/home/hpc/batchus1/caper_tmp/chip/20210513_123933_586540/backend.conf
2021-05-13 12:40:00,685|caper.cromwell_workflow_monitor|INFO| Workflow: id=3b6d19ac-dd11-4e6f-9246-af6ffb9af467, status=Submitted
2021-05-13 12:40:00,729|caper.cromwell_workflow_monitor|INFO| Workflow: id=3b6d19ac-dd11-4e6f-9246-af6ffb9af467, status=Running
2021-05-13 12:40:10,133|caper.cromwell_workflow_monitor|INFO| Task: id=3b6d19ac-dd11-4e6f-9246-af6ffb9af467, task=chip.read_genome_tsv:-1, retry=0, status=Started, job_id=44018
2021-05-13 12:40:10,139|caper.cromwell_workflow_monitor|INFO| Task: id=3b6d19ac-dd11-4e6f-9246-af6ffb9af467, task=chip.read_genome_tsv:-1, retry=0, status=WaitingForReturnCode
2021-05-13 12:39:52,796 INFO - Running with database db.url = jdbc:hsqldb:mem:4bb55e12-96c4-4bba-a1f8-78ea07e02915;shutdown=false;hsqldb.tx=mvcc
2021-05-13 12:39:59,327 INFO - Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
2021-05-13 12:39:59,338 INFO - [RenameWorkflowOptionsInMetadata] 100%
2021-05-13 12:39:59,417 INFO - Running with database db.url = jdbc:hsqldb:mem:cac1db66-f6bc-4aaa-a0e5-adb74c67f90b;shutdown=false;hsqldb.tx=mvcc
2021-05-13 12:39:59,754 INFO - Slf4jLogger started
2021-05-13 12:39:59,952 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO - Workflow heartbeat configuration:
{
"cromwellId" : "cromid-ee56600",
"heartbeatInterval" : "2 minutes",
"ttl" : "10 minutes",
"failureShutdownDuration" : "5 minutes",
"writeBatchSize" : 10000,
"writeThreshold" : 10000
}
2021-05-13 12:40:00,005 cromwell-system-akka.dispatchers.service-dispatcher-13 INFO - Metadata summary refreshing every 1 second.
2021-05-13 12:40:00,096 cromwell-system-akka.dispatchers.service-dispatcher-12 INFO - WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
2021-05-13 12:40:00,100 cromwell-system-akka.actor.default-dispatcher-4 INFO - KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
2021-05-13 12:40:00,123 cromwell-system-akka.dispatchers.engine-dispatcher-42 INFO - CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
2021-05-13 12:40:00,124 WARN - 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf)
2021-05-13 12:40:00,610 cromwell-system-akka.dispatchers.engine-dispatcher-42 INFO - JobExecutionTokenDispenser - Distribution rate: 1 per 2 seconds.
2021-05-13 12:40:00,637 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO - SingleWorkflowRunnerActor: Version 59
2021-05-13 12:40:00,643 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO - SingleWorkflowRunnerActor: Submitting workflow
2021-05-13 12:40:00,685 cromwell-system-akka.dispatchers.api-dispatcher-47 INFO - Unspecified type (Unspecified version) workflow 3b6d19ac-dd11-4e6f-9246-af6ffb9af467 submitted
2021-05-13 12:40:00,711 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - SingleWorkflowRunnerActor: Workflow submitted UUID(3b6d19ac-dd11-4e6f-9246-af6ffb9af467)
2021-05-13 12:40:00,714 cromwell-system-akka.dispatchers.engine-dispatcher-42 INFO - 1 new workflows fetched by cromid-ee56600: 3b6d19ac-dd11-4e6f-9246-af6ffb9af467
2021-05-13 12:40:00,722 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - WorkflowManagerActor: Starting workflow UUID(3b6d19ac-dd11-4e6f-9246-af6ffb9af467)
2021-05-13 12:40:00,728 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - WorkflowManagerActor: Successfully started WorkflowActor-3b6d19ac-dd11-4e6f-9246-af6ffb9af467
2021-05-13 12:40:00,729 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - Retrieved 1 workflows from the WorkflowStoreActor
2021-05-13 12:40:00,743 cromwell-system-akka.dispatchers.engine-dispatcher-42 INFO - WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
2021-05-13 12:40:00,804 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - MaterializeWorkflowDescriptorActor [UUID(3b6d19ac)]: Parsing workflow as WDL 1.0
2021-05-13 12:40:03,540 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - MaterializeWorkflowDescriptorActor [UUID(3b6d19ac)]: Call-to-Backend assignments: chip.align_R1 -> slurm, chip.bam2ta_ctl -> slurm, chip.idr_pr -> slurm, chip.filter_no_dedup -> slurm, chip.pool_ta_ctl -> slurm, chip.error_subsample_pooled_control_with_mixed_endedness -> slurm, chip.overlap_ppr -> slurm, chip.filter -> slurm, chip.qc_report -> slurm, chip.error_wrong_aligner -> slurm, chip.call_peak_pooled -> slurm, chip.filter_R1 -> slurm, chip.pool_ta_pr2 -> slurm, chip.spr -> slurm, chip.filter_ctl -> slurm, chip.call_peak_pr1 -> slurm, chip.error_custom_aligner -> slurm, chip.count_signal_track_pooled -> slurm, chip.error_ctl_fastq_input_required_for_control_mode -> slurm, chip.call_peak_ppr1 -> slurm, chip.idr_ppr -> slurm, chip.call_peak -> slurm, chip.error_control_required -> slurm, chip.align -> slurm, chip.jsd -> slurm, chip.error_input_data -> slurm, chip.reproducibility_overlap -> slurm, chip.idr -> slurm, chip.call_peak_ppr2 -> slurm, chip.bam2ta -> slurm, chip.pool_ta -> slurm, chip.xcor -> slurm, chip.macs2_signal_track -> slurm, chip.overlap -> slurm, chip.subsample_ctl -> slurm, chip.macs2_signal_track_pooled -> slurm, chip.call_peak_pr2 -> slurm, chip.gc_bias -> slurm, chip.read_genome_tsv -> slurm, chip.pool_blacklist -> slurm, chip.error_use_bowtie2_local_mode_for_non_bowtie2 -> slurm, chip.error_use_bwa_mem_for_non_bwa -> slurm, chip.fraglen_mean -> slurm, chip.bam2ta_no_dedup_R1 -> slurm, chip.bam2ta_no_dedup -> slurm, chip.pool_ta_pr1 -> slurm, chip.subsample_ctl_pooled -> slurm, chip.count_signal_track -> slurm, chip.align_ctl -> slurm, chip.overlap_pr -> slurm, chip.error_ctl_input_defined_in_control_mode -> slurm, chip.choose_ctl -> slurm, chip.reproducibility_idr -> slurm
2021-05-13 12:40:03,813 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,814 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,815 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,815 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,816 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,816 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,816 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,817 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,817 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,817 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,817 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,818 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,818 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,818 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,818 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,819 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,819 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,819 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,820 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,820 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,820 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,820 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,821 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,821 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,821 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,821 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,822 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,822 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,822 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,822 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,823 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,823 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,824 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,824 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,824 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,825 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,825 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,825 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,826 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,826 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,826 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,826 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,826 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,827 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,827 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,827 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,827 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,827 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,828 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,830 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,831 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,831 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:03,831 cromwell-system-akka.dispatchers.backend-dispatcher-101 WARN - slurm [UUID(3b6d19ac)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-05-13 12:40:05,631 cromwell-system-akka.dispatchers.engine-dispatcher-152 INFO - Not triggering log of token queue status. Effective log interval = None
2021-05-13 12:40:08,202 cromwell-system-akka.dispatchers.engine-dispatcher-152 INFO - WorkflowExecutionActor-3b6d19ac-dd11-4e6f-9246-af6ffb9af467 [UUID(3b6d19ac)]: Starting chip.read_genome_tsv
2021-05-13 12:40:08,645 cromwell-system-akka.dispatchers.engine-dispatcher-152 INFO - Assigned new job execution tokens to the following groups: 3b6d19ac: 1
2021-05-13 12:40:08,808 cromwell-system-akka.dispatchers.engine-dispatcher-112 INFO - 3b6d19ac-dd11-4e6f-9246-af6ffb9af467-EngineJobExecutionActor-chip.read_genome_tsv:NA:1 [UUID(3b6d19ac)]: Could not copy a suitable cache hit for 3b6d19ac:chip.read_genome_tsv:-1:1. No copy attempts were made.
2021-05-13 12:40:08,836 cromwell-system-akka.dispatchers.backend-dispatcher-156 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(3b6d19ac)chip.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks
2021-05-13 12:40:08,914 cromwell-system-akka.dispatchers.backend-dispatcher-156 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(3b6d19ac)chip.read_genome_tsv:NA:1]: `echo "$(basename /home/hpc/batchus1/RT4_output/chip/3b6d19ac-dd11-4e6f-9246-af6ffb9af467/call-read_genome_tsv/inputs/955696674/hg38.local.tsv)" > genome_name
# create empty files for all entries
touch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2
touch mito_chr_name
touch regex_bfilt_peak_chr_name
python <<CODE
import os
with open('/home/hpc/batchus1/RT4_output/chip/3b6d19ac-dd11-4e6f-9246-af6ffb9af467/call-read_genome_tsv/inputs/955696674/hg38.local.tsv','r') as fp:
for line in fp:
arr = line.strip('\n').split('\t')
if arr:
key, val = arr
with open(key,'w') as fp2:
fp2.write(val)
CODE`
2021-05-13 12:40:09,169 cromwell-system-akka.dispatchers.backend-dispatcher-156 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(3b6d19ac)chip.read_genome_tsv:NA:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;
ITER=0
until [ $ITER -ge 3 ]; do
sbatch \
--export=ALL \
-J cromwell_3b6d19ac_read_genome_tsv \
-D /home/hpc/batchus1/RT4_output/chip/3b6d19ac-dd11-4e6f-9246-af6ffb9af467/call-read_genome_tsv \
-o /home/hpc/batchus1/RT4_output/chip/3b6d19ac-dd11-4e6f-9246-af6ffb9af467/call-read_genome_tsv/execution/stdout \
-e /home/hpc/batchus1/RT4_output/chip/3b6d19ac-dd11-4e6f-9246-af6ffb9af467/call-read_genome_tsv/execution/stderr \
-t 60 \
-n 1 \
--ntasks-per-node=1 \
--cpus-per-task=1 \
--mem=2048 \
-p long \
--account science \
\
\
--wrap "/bin/bash /home/hpc/batchus1/RT4_output/chip/3b6d19ac-dd11-4e6f-9246-af6ffb9af467/call-read_genome_tsv/execution/script" \
&& break
ITER=$[$ITER+1]
sleep 30
done
2021-05-13 12:40:10,132 cromwell-system-akka.dispatchers.backend-dispatcher-156 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(3b6d19ac)chip.read_genome_tsv:NA:1]: job id: 44018
2021-05-13 12:40:10,139 cromwell-system-akka.dispatchers.backend-dispatcher-157 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(3b6d19ac)chip.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCode