azure/azure-hpc

Microsoft Azure HPC & Big Compute

azure-hpc's Issues

If /dev/shm is too small, use local disk

Support custom database upload

Increase default /dev/shm size to ensure DBs fit

Clearly show 'database downloading' state

Currently the query state shows running even thow it might be downloading databases for 10-15 minutes. It would be good to highlight this state.

Support for viewing and editing the various prep scripts

It would be gret to be able to view and edit the various scripts used by the systems. This would also allow post-processing logic to be inserted.

Snakemake Input File Missing

Hi There,

I've followed all of the install instructions, and now I seem to be running into permission issues. My yaml files all seem to be working correctly now, and it's successfully create a pool.

But some of the rules at the beginning of the Snakemake file seem to be causing a permission issues when creating directories.

Provided is the error I'm getting:

I've attached my snakefile and yaml files for the batch-shipyard configuration. Some help would be really appreciated on this! I've followed the installation guide very carefully so I think it's something specific with Snakemake. What's odd is that it works no problem when installing snakemake and the dependencies in Conda.

Snakefile for the RNA-Seq analysis pipeline using test data from zebrafish

You should not need to edit this file unless you are changing the programs in the pipeline

configfile: "config_zebrafish.yaml"
SAMPLES = config['samples']

R1_suffix=config['input_file_R1_suffix']
R2_suffix=config['input_file_R2_suffix']
genome_fasta_file = config['genome_fasta_file']
genome_index_base = config['genome_index_base']
merged_transcripts_file=config['merged_transcripts_file']

rule trim_and_qc_all:
input:
html=expand("{sample}_R1.trimmed_paired_fastqc.html", sample=SAMPLES)

rule trim_reads:
input:
R1_reads="data/{sample}" + R1_suffix,
R2_reads="data/{sample}" + R2_suffix
output:
"1_trimmed_reads/{sample}_R1.trimmed_paired.fastq",
"1_trimmed_reads/{sample}_R1.trimmed_unpaired.fastq",
"1_trimmed_reads/{sample}_R2.trimmed_paired.fastq",
"1_trimmed_reads/{sample}_R2.trimmed_unpaired.fastq"
threads: config['threads']
params:
run_params=config['trimmomatic_params']
shell:
"echo -e "#!/usr/bin/env bash\ncd $FILESHARE;\n trimmomatic PE -threads {threads} {input.R1_reads} {input.R2_reads} {output}" > $FILESHARE/jobrun.sh ;\n $SHIPYARD/shipyard jobs add --configdir $FILESHARE/azurebatch --tail stderr.txt\n"

Query results should be available even after a Batch job is deleted

Currently the portal uses Batch jobs to determine what's been run, and then find job outputs in storage. If the job is deleted, the portal doesn't know the job exists.

I'll update the logic here so that outputs are always available, unless explicitly deleted.

Support for email notifications on job completion