Snakemake wrappers resolve/completion

[jetbrains-research]

Snakemake wrappers resolve/completion

[iromeo]

E.g. see https://bitbucket.org/snakemake/snakemake-wrappers/src/bd3178f4b82b1856370bb48c8bdbb1932ace6a19/bio/samtools/sort/wrapper.py?at=0.31.0&fileviewer=file-view-default

__author__ = "Johannes Köster"
__copyright__ = "Copyright 2016, Johannes Köster"
__email__ = "[email protected]"
__license__ = "MIT"


import os
from snakemake.shell import shell


prefix = os.path.splitext(snakemake.output[0])[0]

shell(
    "samtools sort {snakemake.params} -@ {snakemake.threads} -o {snakemake.output[0]} "
    "-T {prefix} {snakemake.input[0]}")

We could provide code inspections for missing parameters

[iromeo]

E.g we could add reference on a string, see http://www.jetbrains.org/intellij/sdk/docs/tutorials/custom_language_support/reference_contributor.html#define-a-reference-contributor to provide completion/resolve for wrappers from https://github.com/snakemake/snakemake-wrappers or any custom source.

Read about wrappers at https://snakemake-wrappers.readthedocs.io/en/stable/

Desired behavior:

• Utility class which could collect wrappers info from user provided directory
• Utility class that works with local github repo - collects wrappers for all repo relizes to some effective data structure
• Gradle task that checkouts wrappers repo locally + collects wrappers info using the step below
• Plugin bundle will include serialized wrappers info from prev step.
• Maybe some optional wrappers crawler from github master, update it by some "button" in settings or "action".
• Plugin uses cached wrappers info for
  • resolve/completion for wrappers paths
  • documentation popup with useful info (e.g. hyper links on wrapper docs, e.g. https://snakemake-wrappers.readthedocs.io/en/0.62.0/wrappers/bismark/bismark.html + maybe some parameters info extracted from meta.yaml content. When using custom wrappers dir - use links to dependencies.yaml, meta.yaml and wrapper.py files instead of https://snakemake-wrappers.readthedocs.io)
  • Ideally to have useful inspections on input/output/params sections according to meta.yaml or wrappers.py content.

UI:

• Settings page
• Choice for wrappers source: 1) default wrappers repo 2) custom directory (e.g some github branch or just directory with proper structure)
• Refresh button which updates wrappers info from the above source

[iromeo]

In case of local files - paths starts with file:// prefix, e.g.:

rule foo:
  wrapper:
        f"file://{PIPELINE_PATH}/wrappers/bio/bismark/bismark"

also could be relative paths

[sharkovadarya]

Be wary of rate limits for request to repositories: https://confluence.atlassian.com/bitbucket/rate-limits-668173227.html

[iromeo]

API examples:

• Documentation popup: SmkShadowSettingsDocumentation

[iromeo]

Example:

rule virus_reference_genome_chromsizes:
    output: "indexes/{virus_genome}/{virus_genome}.chrom.sizes"
    log:    "indexes/{virus_genome}/{virus_genome}.chrom.sizes.log"
    input:  "indexes/{virus_genome}/{virus_genome}.2bit"
    message: "Indexes: Chrom sizes for {input}"
    shadow: "full"
    wrapper:
        "0.43.1/bio/ucsc/twoBitInfo"

[iromeo]

@SimonTsirikov
Got an exception when playing with features/#18-resolve-completion-for-wrappers

java.nio.file.NoSuchFileException: /github.com/snakemake/snakemake-wrappers/archive/master.zip
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1225)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:727)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:844)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:247)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
	at com.intellij.platform.templates.github.ZipUtil.unzip(ZipUtil.java:94)
	at com.jetbrains.snakecharm.codeInsight.completion.wrapper.SmkWrapperCrawler$runActivity$1$1.run(SmkWrapperCrawler.kt:30)
	at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:935)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsync$5(CoreProgressManager.java:442)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$3(ProgressRunner.java:235)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:170)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:629)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:581)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:157)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:235)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base/java.lang.Thread.run(Thread.java:834)

[iromeo]

Missing at the moment:

• Documentation

  • Show doc for wrapper keyword argument in completion/F1. Use info from meta.yaml
  • Documentation for section name (e.g input/output/...) that shows expected args for this section, not only documentation for wrapper.
  • Documentation for wrapper: show supported sections & params info, not just text from meta.yaml

• Wrappers params parsing:

  • Parse not only wrappers.py but also meta.yaml and collect keyword args and their descriptions
  • In case of using file://, collect such wrappers, parse&resolve params and add to module wrappers info 'storage'
  • Not all wrapper params are obligatory. Maybe understand it from calls like snakemake.get("key", default_value) in wrappers.py or use some conventions in meta.yaml
  • Inspection False positive: Wrapper sections detection false positive: 'shell' is missing #311
  • Inspection False positive: #312

• Wrappers Info bundles:

  • Bundle wrappers should contain information about all wrappers releases, e.g. not all tools available in latest release or parameters could be changed. Otherwise, our completion/inspection will show false positives for such cases
  • Load bundled wrappers storage object once for all projects/modules, not per module. Here we need several layers, e.g. bundled wrappers could be application level object, facet settings dependent wrappers (e.g. custom folder) - module level
  • Think of wrappers info auto-update, e.g. from TeamCity
  • If custom folder is used as input, need to add file watcher and update case if timestamps changed

[iromeo]

Initial support implemented, rest will be in wrappers milestone

[iromeo]

Seems not all mentioned in Missing at the moment: is extracted as issues