pyrpipe (Pronounced as "pyre-pipe") is a python package to easily develop bioinformatic or any other computational pipelines in pure python. pyrpipe provides an easy-to-use framework for importing any UNIX command in python. pyrpipe comes with specialized classes and functions to easily code RNA-Seq processing workflows. Pipelines in pyrpipe can be created and extended by integrating third-party tools, executable scripts, or python libraries in an object oriented manner.
Read the paper here
Read the docs here
NOTE: Due to change in API designs, pyrpipe version 0.0.5 and above is not compatible with lower versions. All the tutorials and documentation have been updated to reflect v0.0.5.
Allows fast and easy development of bioinformatics pipelines in python by providing
- a high level api to popular RNA-Seq processing tools -- downloading, trimming, alignment, quantificantion and assembly
- optimizes program parameters based on the data
- a general framework to execute any linux command from python
- comprehensive logging features to log all the commands, output and their return status
- report generating features for easy sharing, reproducing, benchmarking and debugging
- Import any UNIX executable command/tool in python
- Dry-run feature to check dependencies and commands before execution
- Flexible and robust handling of options and arguments (both Linux and Java style options)
- Auto load command options from .yaml files
- Easily override threads and memory options using global values
- Extensive logging for all the commands
- Automatically verify Integrity of output targets
- Resume feature to restart pipelines/jobs from where interrupted
- Create reports, MultiQC reports for bioinformatic pipelines
- Easily integrated into workflow managers like Snakemake and NextFlow (to schedule jobs, scale jobs, identify paralellel steps in pipelines)
- Schedule jobs
- Scale jobs on HPC/cloud
- Identify parallel steps in pipelines
- python 3.6 or higher
- OS: Linux, Mac
| Tool | Purpose |
|---|---|
| SRA Tools (v. 2.10.9 ) | SRA access |
| Trimgalore (v. 0.6.0) | Trimming |
| BBDuk (v. 38.76) | Trimming |
| Hisat2 (v. 2.2.1) | Alignment |
| STAR (v. 2.7.7a) | Alignment |
| Bowtie2 (v. 2.3.5.1) | Alignment |
| Kallisto (v. 0.46.2) | Quantification |
| Salmon (v. 0.14.1) | Quantification |
| Stringtie (v. 2.1.4) | Transcript Assembly |
| Cufflinks (v. 2.2.1) | Transcript Assembly |
| Samtools (v. 1.9) | Tools |
Get started with the basic tutorial. Read the documentation here. Several examples are provided here
Following python code downloads data from SRA, uses Trim Galore to trim the fastq files and STAR to align reads. More detailed examples are provided here
from pyrpipe.sra import SRA
from pyrpipe.qc import Trimgalore
from pyrpipe.mapping import Star
trimgalore = Trimgalore(threads=8)
star = Star(index='data/index',threads=4)
for srr in ['SRR976159','SRR978411','SRR971778']:
SRA(srr).trim(trimgalore).align(star)
This simple example imports and runs the Unix grep command. See this for more examples.
>>> from pyrpipe.runnable import Runnable
>>> grep=Runnable(command='grep')
>>> grep.run('query1','file1.txt',verbose=True)
>>> grep.run('query2','file2.txt',verbose=True)
Please follow these instructions:
NOTE: You need to install the third-party tools to work with pyrpipe. We recomend installing these through bioconda where possible. An example of setting up the environment using conda is provided below. It is best to share your conda environment files with pyrpipe scripts to ensure reproducibility.
- Download and install Conda
conda create -n pyrpipe python=3.8conda activate pyrpipeconda install -c bioconda pyrpipe star=2.7.7a sra-tools=2.10.9 stringtie=2.1.4 trim-galore=0.6.6
The above command will install pyrpipe and the required tools inside a conda environment. Alternatively, use the conda environment.yaml file provided in this repository and build the conda environment by running
conda env create -f pyrpipe_environment.yaml
conda install -c bioconda pyrpipe
pip install pyrpipe --upgrade
If above command fails due to dependency issues, try:
- Download the requirements.txt
pip install -r requirements.txtpip install pyrpipe
To run tests:
- Download the test set (direct link)
pip install pytest- To build test_environment. Please READ THIS
- From pyrpipe root directory, run
pytest tests/test_*
git clone https://github.com/urmi-21/pyrpipe.git
pip install -r pyrpipe/requirements.txt
pip install -e path_to/pyrpipe
#Running tests; From pyrpipe root perform
#To build test_environment (This will download tools):
cd tests ; . ./build_test_env.sh
#in same terminal
py.test tests/test_*
If you face problems with downloading data from SRA, try configuring the SRA-Tools.
Use vdb-config -i to configure SRA Toolkit. Make sure that:
- Under the TOOLS tab, prefetch downloads to is set to public user-repository
- Under the CACHE tab, location of public user-repository is not empty
Use the following pyrpipe_diagnostic command to test if SRA-Tools are setup properly
pyrpipe_diagnostic test
Please see CONTRIBUTING.md
This work is funded in part by the National Science Foundation award IOS 1546858, "Orphan Genes: An Untapped Genetic Reservoir of Novel Traits".
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent