k8hertweck/fieldlabcomparison

microbiome analysis of ant colonies

FieldLabComparison

Scripts for microbiome analysis of bacterial 16S sequences from ants.

Data:

• SRA accessions:
  • PRJNA170251 (Mycocepurus): metadata in SRA correct, data/SraRunTableSRP018247.txt
  • PRJNA170250 (Trachymyrmex/Cyphomyrmex): metadata in SRA is not correct, revised metadata in data/SraRunTableSRP018246corrected.txt
• Silva database:
  • Use Full length sequences and taxonomy references for version 128, which includes taxonomy file (these are downloaded as a part of the workflow below). Another option is to use the silva.bacteria.fasta which is a silva reference file from the MiSeq SOP. The silva.bacteria.fasta file can be trimmed to the V1-V3 region using the align.seqs() and the E.coliV1V3.fas file.

Dependencies:

• SRA Toolkit v2.8.1or higher for data download
• seq_crumbs only if using dataCheck.sh
• MothuR for raw data cleaning and OTU assessment; currently set to Mac version, v.1.40.5 (last updated 06/19/18)
• QIIME 1 for raw data cleaning and OTU assessment

Setting up data

• dataDownload.sh download data from NCBI SRA and Silva database (takes several minutes depending on internet connectivity)
  • data/dataCheck.sh check SRA data against archived sequence files, do not need to run again

Mothur Workflow

• MothuR analysis based on 454 and MiSeq tutorials to create OTU table
• mothur_SilvaRef.bat create custom Silva alignment for reference (takes ~10 minutes)
• mothur_prep.sh split, trim, and aggregate sequence files (takes several minutes)
• mothur_otu.bat processes combined sequences and outputs OTU table

QIIME 1 Workflow

• QIIME 1 analysis based on the 454 tutorial to create OTU table
• qiime_workflow.sh uses 3 scripts to take fastq files and obtain an OTU table biom file
  • convert_fastaqual_fastq.py converts fastq to fasta & qual files in qiime
  • add_qiime_labels.py combines all fasta files into one fasta file in qiime
    • metadata_mapping_file.txt contains data that assist in combining all fasta files into one fasta file in qiime
  • pick_de_novo_otus.py a series of 7 scripts that outputs OTU table biom file
    • qiime_parameters.txt changes default settings of python script

Hypothesis testing and data visualization

• R package phyloseq

fieldlabcomparison's Issues

Add updated silva database

Modify mothur_SilvaRef.bat to reflect new reference

Document in README: URL to download version 128 file

Include taxonomy file

Taxonomy file can be downloaded with reference sequences (described in #6 ).

Referencing the MiSeq tutorial, we need to add the taxonomy file into the workflow.

line 45 in mothur_otu.bat

Next make.group command doesn't recognize fasta files

optimizing mother otu calling

• check sequences removed in pre-processing
• cluster step (just after distance matrix) memory error (need to optimize)

Add other dependencies

sratoolkit 2.8.1 or higher

Add qiime workflow

Extract particular taxa from raw sequences and build phylogeny?

Add description of data to README

include SRA accession numbers, and caveat that metadata may be incorrect

add metadata file to GitHub

Automate silva download and setup?

• use wget or curl to download from URL
• use tar to expand
• mv files to silva/

managing Illumina data

• use SeqPrep (or something similar) to merge paired end sequences

• include in pipeline

Use phyloseq for data visualization

Tutorial here, including instructions for importing results from mothur.

update readme with complete instructions

• version numbers of software (and installation)

• links to tutorials

• instructions for running

Need to add rest of sequence data (and exclude unnecessary controls)

Following the separation of fastq files into fasta + qual in the batch file, the resulting data files need to be moved to analysis.

Add samples from SRP018247

• update SRP018247.lst

• uncomment lines 16-19 in mothur_prep.sh

• update line 39 in mothur_prep.sh

• update line 45 in mothur_prep.sh

• update line 48 in mothur_prep.sh

• run mothur_prep.sh to test

• checking parameters in mothur_otu.bat

ignore other files

List of files to be ignored by git are found in .gitignore; silva files (and other intermediate data files) should be added to this list as needed

Update and organize project directory

Scripts in highest level, other directories include data/, figures/, silva/, ...?

Plan figures and analyses for publication