Feature Request - add support to singularity

[icgc-argo]

Feature Request - add support to singularity

[edsu7]

Investigation summary:

• will have to change a lot of files namely nextflow.config and potentially main.nf per wfpm module
• Two scenarios A) and B)
• have to add profiles to nextflow.config, e.g.
• B) is more work but technically keeps in line with how code has been organized

profiles {
	docker {
    	docker.enabled = true
        singularity.enabled = false
    	runOptions = '-u \$(id -u):\$(id -g)'
	}

	singularity {
    	docker.enabled = false
        singularity.enabled = true
        singularity.cacheDir = "~/.singularity/cache"
	}
}

A) Can add process.container to either nextflow.config:

profiles {
   singularity {
   process.container = 'docker://quay.io/biocontainers/multiqc:1.3--py35_2'
   }
}

B) Or define in main.nf

### Append and inform singularity to pull from docker
if (workflow.profile!='docker'){
  params.container='docker://'
} else {
  params.container = ""
}

### Restructure container string
process payloadJsonToTsvs {
  container "${params.container}${container[params.container_registry ?: default_container_registry]}:${params.container_version ?: version}" 
  publishDir "${params.publish_dir}/${task.process.replaceAll(':', '_')}", mode: "copy", enabled: params.publish_dir ? true : false
  errorStrategy 'terminate'
  cpus params.cpus
  memory "${params.mem} GB"

Modules to change:

github.com/icgc-argo-workflows/data-processing-utility-tools/[email protected]
github.com/icgc-argo-workflows/data-processing-utility-tools/[email protected]
github.com/icgc-argo-workflows/data-processing-utility-tools/[email protected]
github.com/icgc-argo-workflows/dna-seq-processing-tools/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/argo-data-submission/[email protected]
github.com/icgc-argo/nextflow-data-processing-utility-tools/[email protected]

[edsu7]

Linda started a PR, scong-score-wf step not working. Will need to investigate.

[edsu7]

Error found. It looks like score-client is trying to write /score-client/logs within the container without appropriate permissions. See below to allow for log writing.

  docker {
    docker.enabled = true
    singularity.enabled = false
    docker.runOptions = '-u \$(id -u):\$(id -g)'
	}
    singularity {
    docker.enabled = false
    singularity.enabled = true
    singularity.autoMounts = true
    singularity.engineOptions = '-s'
    singularity.runOptions = '--writable'
	}

Side note - I had validate-seqtools fail once me because it detected a newer version (pulled the latest image which is running the latest seq-tools version...)