Giter Site home page Giter Site logo

jetbrains-research / snakecharm Goto Github PK

View Code? Open in Web Editor NEW
61.0 7.0 7.0 2.86 MB

Plugin for PyCharm / IntelliJ IDEA Platform IDEs which adds support for Snakemake language.

License: MIT License

Kotlin 30.77% Python 46.83% Roff 0.01% HTML 0.31% Gherkin 21.45% Lex 0.09% Java 0.56%
snakemake pycharm intellij-plugins python intellij bioinformatics-pipeline bioinformatics-tool

snakecharm's Introduction

JetBrains Research License license Tests Linux tests Tests Windows tests

SnakeCharm SnakeCharm

About

SnakeCharm plugin for PyCharm / IntelliJ Platform IDEs adds IDE support for Snakemake workflows language, that is widely used in Bioinformatics. The plugin is developed by JetBrains Research Team, for more details see project home page.

Please report features suggestions or found bugs to project issue tracker.

Features highlights:

NB: To activate all features please enable Snakemake support in Settings | Languages & Frameworks | Snakemake. Snakemake framework should be configured to use python interpreter with snakemake module installed.

Features available in Snakefile and *.smk, *.rule, *.rules files:

  • Python code syntax highlighting

  • Snakemake specific syntax highlighting

    • Highlight rule section names
    • Syntax highlighting for Snakemake string format language, e.g. in strings like "results/sample_{genome}.bam".
    • Special highlighting for rule wildcards and their usages
  • Code completion and goto to declaration for:

    • Snakemake specific variables (e.g. config, rules, etc);
    • Snakemake api methods like expand, temp, etc.
    • Rules names after rules., checkpoints after checkpoints.
    • Rules sections names after rules.my_rule_name.
    • Rule sections keywords arguments, e.g. after rules.my_rule_name.input.
    • Wildcards and sections names in shell, input, message, params and other sections
    • Paths in sections: configfile, workdir, conda, include.
    • Completion for wrappers names in wrapper: sections
    • Wrappers with detailed meta.yaml provides completion for rule section keyword arguments in output, input, params sections
    • Python methods and variables
  • Structure view for Snakemake files

  • Rules blocks folding

  • Open rule declaration by name using Navigate|Symbol...

  • Quick documentation for wrappers, that includes links to wrapper source code and paga in wrappers repository

  • Multiple code inspections for possible errors in snakemake files

Useful links:

Installation

Via JetBrains Plugins Market Place (recommended):

NB: If you've already configured custom file type for Snakefile or *.smk files please delete it (or change file masks to smth else). Otherwise, your custom file type will replace SnakeCharm support in snakemake files.

In PyCharm (or other IntelliJ Platfrom based IDE with Python Community Edition plugin installed) open Preferences|Plugins|Marketplace|SnakeCharm and press Install button.

From ZIP plugin bundle (alternative way):

  • Open IDEA/PyCharm Preferences
  • Choose Plugins section
  • Press gear icon and choose Install Plugin from Disk...
  • Use *.zip bundle download from Plugin Manager or built from sources

Setup Snakemake support

To enable all code insight features (e.g. code completion for directory, touch etc methods) please configure Python SDK with installed snakemake package.

https://github.com/JetBrains-Research/snakecharm/wiki#setup-snakemake-support

IntelliJ Plugins Repository

Plugin is available in IntelliJ Plugins Repository, see SnakeCharm Plugin.

EAP Updates

Early builds of coming releases are available in EAP channel. To receive this updates add EAP repository:

  • Open PyCharm / IntelliJ IDEA IDE settings
  • In Plugins section press ⚙️ (gear icon), select Manage Plugin Repositories...
  • Add repo https://plugins.jetbrains.com/plugins/eap/list using + button, press OK (Complete guide at Custom Release Channels)
  • Switch to Marketplace tab, search for snakecharm
  • EAP builds will be visible as plugin updates if compatible with your IDE build

Support

Contact me at [email protected] or post issues in Issue Tracker


Development

Pull requests are welcome. It is my side project, so I appreciate your help with implementation of desired features. See DEVELPER.md

snakecharm's People

Contributors

artemd24 avatar dakochik avatar iromeo avatar krgolovin avatar nikita-nazarov avatar olegs avatar sharkovadarya avatar simontsirikov avatar xewar313 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

snakecharm's Issues

*.rules files support

E.g. see https://bitbucket.org/snakemake/snakemake-workflows/src/231af1f339c7ba31f88e5a9e4a50257d4d2c8c45/bio/ngs/rules/quality_control/fastqc.rules?at=master&fileviewer=file-view-default

# vim: syntax=python tabstop=4 expandtab
# coding: utf-8


"""
Rules for analysing fasta files with FastQC.

For usage, include this in your workflow.
"""


__author__ = "Johannes Köster (http://johanneskoester.bitbucket.org)"
__license__ = "MIT"


rule fastqc:
    input:
        "{prefix}.fastq.gz"
    output:
        "{prefix}_fastqc.zip"
    shell:
        "fastqc {input}"

Resolve/Completion for named args of input, output, params,...

Resolve/Completion for named args of input, output, params,...
E.g. here: input.bdg

rule bdg_to_clip:
    input:
        bdg="bws/{sample}.bdg",
        cs=rules.download_chrom_sizes.output
    output: temp("bws/{sample}.clip")
    shell: 'bedtools slop -i {input.bdg} -g {input.cs} -b 0 | bedClip stdin {input.cs} {output}'

Includes support

See https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#includes

include_prefix="https://bitbucket.org/johanneskoester/snakemake-workflows/raw/master"

include:
    include_prefix + "/bio/ngs/rules/mapping/bwa_mem.rules"
include:
    include_prefix + "/bio/ngs/rules/mapping/samfiles.rules"
include:
    include_prefix + "/bio/ngs/rules/mapping/gatk.rules"
include:
    include_prefix + "/bio/ngs/rules/variant_calling/gatk_haplotype_caller.rules"
include:
    include_prefix + "/bio/ngs/rules/variant_calling/gatk_variant_recalibrator.rules"


rule all:
    input:
        "variant_calling/genome/all.snp_recalibrated.indel_recalibrated.vcf"

Also see examples here: https://github.com/snakemake-workflows/docs

Completion false positives for snakemake.io, etc.

Current completion implementation adds all top-level methods from snakemake.io to autocompletion, although snakemake imports only some of methods, e.g.

See

# ~/anaconda/envs/snakemake/lib/python3.6/site-packages/snakemake/workflow.py
from snakemake.io import protected, temp, temporary, ancient, directory, expand, dynamic, glob_wildcards, flag, not_iterable, touch, unpack, local, pipe, repeat, report

E.g. contains_wildcard_constraints method shouldn't be visible in autocompletion. We need smarter impl. Related to #1

image

Inspection: wildcards in shell script can be used only with `wildcards.` prefix.

Snakemake error message is quite misleading,

This won't work with strange error sample is unknown.

rule trim_single_fastq:
    input: os.path.join(config['fastq_dir'], '{sample}.fastq')
    output: 'cleaned/{sample}_se.fastq'
    threads: 4
    conda: 'envs/bio.env.yaml'
    shell: 'trim_galore --cores {threads} --nextera {input} -o cleaned/ &> {log}; ' \
           'mv cleaned/{sample}_val.fq {output}'

Changing cleaned/{sample}_val.fq to cleaned/{wildcards.sample}_val.fq fixes the problem.

Resolve/completion for `config` attrs

E.g. config[genome] where genome is defined in configfile or in cmdline using --config argument. Warn only about first usage and consider defined downstream.

rule download_fa:
    output: directory("fa")
    shell: 'rsync -avzP --exclude="*.txt" rsync://hgdownload.cse.ucsc.edu/goldenPath/{config[genome]}/chromosomes/ {output} && '
           'gunzip -f {output}/*.fa.gz'

IDEA CE 2019.1 : Plugin 'SnakeCharm' failed to initialize and will be disabled.

SnakeCharm v0.2-2018.3
Python Community Edition Plugin: v2018.3.183.5912.2

IntelliJ IDEA 2019.1 (Community Edition)
Build #IC-191.6183.87, built on March 27, 2019
JRE: 1.8.0_202-release-1483-b39 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.14

Plugin 'SnakeCharm' failed to initialize and will be disabled.  Please restart IntelliJ IDEA.

java.lang.NoClassDefFoundError: com/jetbrains/python/PythonFileType
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
    at com.intellij.util.lang.UrlClassLoader._defineClass(UrlClassLoader.java:306)
    at com.intellij.util.lang.UrlClassLoader.defineClass(UrlClassLoader.java:302)
    at com.intellij.util.lang.UrlClassLoader._findClass(UrlClassLoader.java:271)
    at com.intellij.ide.plugins.cl.PluginClassLoader.loadClassInsideSelf(PluginClassLoader.java:182)
    at com.intellij.ide.plugins.cl.PluginClassLoader.tryLoadingClass(PluginClassLoader.java:136)
    at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:49)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.jetbrains.snakecharm.SnakemakeFileTypeFactory.createFileTypes(SnakemakeFileTypeFactory.kt:15)
    at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.initStandardFileTypes(FileTypeManagerImpl.java:296)
    at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.<init>(FileTypeManagerImpl.java:256)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.picocontainer.defaults.InstantiatingComponentAdapter.newInstance(InstantiatingComponentAdapter.java:193)
    at com.intellij.util.pico.CachingConstructorInjectionComponentAdapter.doGetComponentInstance(CachingConstructorInjectionComponentAdapter.java:92)
    at com.intellij.util.pico.CachingConstructorInjectionComponentAdapter.instantiateGuarded(CachingConstructorInjectionComponentAdapter.java:66)
    at com.intellij.util.pico.CachingConstructorInjectionComponentAdapter.getComponentInstance(CachingConstructorInjectionComponentAdapter.java:48)
    at com.intellij.openapi.components.impl.ComponentManagerImpl$ComponentConfigComponentAdapter.getComponentInstance(ComponentManagerImpl.java:458)
    at com.intellij.openapi.components.impl.ComponentManagerImpl.createComponents(ComponentManagerImpl.java:104)
    at com.intellij.openapi.application.impl.ApplicationImpl.lambda$createComponents$9(ApplicationImpl.java:449)
    at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:164)
    at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:586)
    at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:532)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:86)
    at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:151)
    at com.intellij.openapi.application.impl.ApplicationImpl.createComponents(ApplicationImpl.java:473)
    at com.intellij.openapi.components.impl.ComponentManagerImpl.init(ComponentManagerImpl.java:88)
    at com.intellij.openapi.application.impl.ApplicationImpl.load(ApplicationImpl.java:407)
    at com.intellij.openapi.application.impl.ApplicationImpl.load(ApplicationImpl.java:393)
    at com.intellij.idea.IdeaApplication.run(IdeaApplication.java:208)
    at com.intellij.idea.IdeaApplication.lambda$initApplication$0(IdeaApplication.java:74)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:764)
    at java.awt.EventQueue.access$500(EventQueue.java:98)
    at java.awt.EventQueue$3.run(EventQueue.java:715)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:734)
    at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:347)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.ClassNotFoundException: com.jetbrains.python.PythonFileType PluginClassLoader[SnakeCharm, 0.2-2018.3] com.intellij.ide.plugins.cl.PluginClassLoader@7647d89
    at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:51)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 50 more

Includes support

See https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#includes

include_prefix="https://bitbucket.org/johanneskoester/snakemake-workflows/raw/master"

include:
    include_prefix + "/bio/ngs/rules/mapping/bwa_mem.rules"
include:
    include_prefix + "/bio/ngs/rules/mapping/samfiles.rules"
include:
    include_prefix + "/bio/ngs/rules/mapping/gatk.rules"
include:
    include_prefix + "/bio/ngs/rules/variant_calling/gatk_haplotype_caller.rules"
include:
    include_prefix + "/bio/ngs/rules/variant_calling/gatk_variant_recalibrator.rules"


rule all:
    input:
        "variant_calling/genome/all.snp_recalibrated.indel_recalibrated.vcf"

https://bitbucket.org/snakemake/snakemake-workflows/src/231af1f339c7ba31f88e5a9e4a50257d4d2c8c45/bio/ngs/workflows/variant_calling_gatk/Snakefile?at=master&fileviewer=file-view-default

Cannot execute `intellij:runIde`

The Gradle task fails with error Could not get unknown property 'intellijPublishToken' for task ':publishPlugin' of type org.jetbrains.intellij.tasks.PublishTask. if a contributor doesn't have intellijPublishToken property defined.

Workaround: comment publishPlugin task in build.gradle

*.rules files support

E.g. see https://bitbucket.org/snakemake/snakemake-workflows/src/231af1f339c7ba31f88e5a9e4a50257d4d2c8c45/bio/ngs/rules/quality_control/fastqc.rules?at=master&fileviewer=file-view-default

fastqc.rules

# vim: syntax=python tabstop=4 expandtab
# coding: utf-8


"""
Rules for analysing fasta files with FastQC.

For usage, include this in your workflow.
"""


__author__ = "Johannes Köster (http://johanneskoester.bitbucket.org)"
__license__ = "MIT"


rule fastqc:
    input:
        "{prefix}.fastq.gz"
    output:
        "{prefix}_fastqc.zip"
    shell:
        "fastqc {input}"

Snakemake wrappers resolve/completion

Resolve / completion from the configured wrappers repo

rule samtools_sort:
    input:
        "mapped/{sample}.bam"
    output:
        "mapped/{sample}.sorted.bam"
    params:
        "-m 4G"
    threads: 8
    wrapper:
        "0.0.8/bio/samtools_sort"

Here, Snakemake will automatically downloads the corresponding wrapper from https://bitbucket.org/snakemake/snakemake-wrappers/src/0.0.8/bio/samtools_sort/wrapper.py.

. Alternatively, e.g., for development, the wrapper directive can also point to full URLs, including URLs to local files with absolute paths file:// or relative paths file:.

Update: Bitbucket wrappers deprecated (https://bitbucket.org/snakemake/snakemake-wrappers/src/62b507647dde?at=master) use GitHub repo (https://github.com/snakemake/snakemake-wrappers)

'shell' section: resolve & completion for `params` args

'shell' section: resolve & completion for params args

rule call_peaks_span_all:
    input:
        bam="cleaned_all/pooled.bam",
        span=rules.download_span.output,
        chrom_sizes=rules.download_chrom_sizes.output
    output: "cleaned_all_peaks/span/pooled_{bin, [0-9]+}.span"
    params:
        outdir=lambda wildcards, output: os.path.dirname(str(output)),
        xmx=lambda wildcards: str(800 // int(wildcards.bin))
    threads: 8
    shell: 'java -Xmx{params.xmx}G  -jar {input.span} analyze -t {input.bam} --workdir {params.outdir} --fragment 0 ' \
           '--bin {wildcards.bin} --cs {input.chrom_sizes} --threads {threads} --model {output} --keep-dup true --debug; '\
           '--foo {params.}'
  • Auto-complete keyword args names after params. in shell section. E.g.
    2019-05-11_18-20-29

  • Resolve keyword args names, e.g. xmx in {params.xmx} into params: section

Multiline strings declaration support

E.g. see shell arg in:

rule download_phantompeakqualtools:
    output: directory("bin/phantompeakqualtools")
    params:
        targz="phantompeakqualtools.tar.gz"
    shell: 'cd bin && '
           'curl --location '
           'https://storage.googleapis.com/google-code-archive-downloads/v2/'
           'code.google.com/phantompeakqualtools/ccQualityControl.v.1.1.tar.gz '
           '--output {params.targz} && '
           'tar xvf {params.targz}'

Resolve/completion for rules names

After rules. in python code

rule clip_to_bw:
    input:
        clip="bws/{sample}_sorted.clip",
        cs=rules.download_chrom_sizes.output
    output: "bws/{sample}.bw"
    shell: 'bedGraphToBigWig {input.clip} {input.cs} {output}'

Subtask of #6

Support highlighting for arguments of format strings

Snakemake automatically substitutes input, output, wildcards, etc. objects into strings used in input/output/message/shell, etc. sections. So it is a bit similar to f-strings appeared in python 3.6. At the moment no highlighting is substitution syntax is provided:

image

Let's support highlighting similar to that which is provided by PyCharm for python f-strings, e.g.:

image

P.S: we cannot use f-strings instead of ordinary strings in snakemake due to undefined variable errors, some vars appear lately on runtime

Resolve/Completion for subworflows names

subworkflow otherworkflow:
    workdir:
        "../path/to/otherworkflow"
    snakefile:
        "../path/to/otherworkflow/Snakefile"
    configfile:
        "path/to/custom_configfile.yaml"

rule a:
    input:
        otherworkflow("test.txt")
    output: ...
    shell:  ...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.