Support highlighting for arguments of format strings

[jetbrains-research]

Support highlighting for arguments of format strings

[iromeo]

Features like #43 could be easily implemented if we will detect substitution at parser level (like f-string support in PyCharm)

[iromeo]

Format strings use Python Mini Language Format, see https://docs.python.org/3/library/string.html

Let's try to implement this using language injection approach.

• here format language is much more simple than syntax of f-strings
• mini language format isn't a valid python, e.g. in {foo.some_dict[key]} quotes around key are omitted
• we'd like to support this not for all strings in *.smk file, but only for some cases, e.g. args of sections, "shell" call, etc.
• we don't need to many features there, e.g. highlighting, completion, resolve, inline/extract param are enough
• completion/resolve isn't very complicated there, not required to make python completion work there

See PyDocstringLanguageInjector, PyTypingAnnotationInjector, PyDocstringLanguageDialect, PyDocReference

[iromeo]

Some examples:

Wildcards regexp constraints:

rule index_bams:
    input: '{anywhere}/{sample}.bam'
    output: '{anywhere}/{sample, [^/]*}.bam.bai'
    wrapper: '0.31.1/bio/samtools/index'

Escaped { and }:

rule bam_qc_pbc_nrf:
    input: rules.bam_to_pileup.output
    output: 'qc/pbc_nrf/{sample}.pbc_nrf.tsv'
    params:
          tmp_dir='tmp'
    shell: '''
mkdir -p {params.tmp_dir} &&
(T=$'\\t'
>&2 echo "TotalReadPairs${{T}}DistinctReadPairs${{T}}OneReadPair${{T}}TwoReadPairs${{T}}\
NRF=Distinct/Total${{T}}PBC1=OnePair/Distinct${{T}}PBC2=OnePair/TwoPair"

cat {input} | \
    sort -k1,1 -k3,3n -k2,2n -k6,6 -T {params.tmp_dir} | \
    awk -v OFS='\\t' '{{print $1,$2,$3,$6}}' | uniq -c | \
    awk 'BEGIN{{mt=0;m0=0;m1=0;m2=0}}
    ($1==1){{m1=m1+1}} ($1==2){{m2=m2+1}} {{m0=m0+1}} {{mt=mt+$1}}
    END{{
        if (mt!=0){{m0_t=m0/mt}} else {{m0_t=-1.0}};
        if (m0!=0){{m1_0=m1/m0}} else {{m1_0=-1.0}};
        if (m2!=0){{m1_2=m1/m2}} else {{m1_2=-1.0}};
        printf "%d\\t%d\\t%d\\t%d\\t%f\\t%f\\t%f\\n",mt,m0,m1,m2,m0_t,m1_0,m1_2;
    }}') > {output}
    '''

[nikita-nazarov]

Identifiers of string language in some rule sections that are not executable sections turn into wildcards in snakemake (See snakemake/io.py:565). These sections are:

• conda
• resources
• group
• benchmark
• log
• output
• params
• input

See snakemake/rules.py:596-758

e.g. sample will be considered as wildcard in

rule NAME:
   input: "{sample}.txt"

Notice, that you can't use the same wildcard with different regular expression constraints. The code will fail at runtime with Constraint regex must be defined only in the first occurence of the wildcard in a string. error

Furthermore, there are sections, that generate wildcards, and sections, where wildcards are expanded. The list of first ones is:

• output
• log
• benchmark

(see usages of register_wildcards in snakemake/rules.py:265)

Wildcards are created once by one of these sections in priority in which they are listed above.
E. g.:

rule NAME:  
  log: "{sample}.txt"
  output: "{sample}.txt"

The wildcard sample will be created by output section
If wildcards sets are not equal in output, log, benchmark, the code will fail at runtime with error:
Not all output, log and benchmark files of rule contain the same wildcards...
(see snakemake/rule.py:270)
Another error will occur for example in this situation:

rule NAME:
   input: "{sample}.txt"
   output: "{output_file}"

In this example code will fail with error:
Wildcards in input files cannot be determined from output files: 'sample'
This happens because wildcard sample was not created.

[iromeo]

Related PyCharm bug - https://youtrack.jetbrains.com/issue/PY-37374

[iromeo]

Done