Giter Site home page Giter Site logo

docopt.sh's Introduction

docopt.sh

image

A docopt argument parser for bash 3.2, 4+, and 5+ with no external dependencies.

Quick start

#!/usr/bin/env bash

DOC="Naval Fate.
Usage:
  naval_fate.sh <name> move <x> <y> [--speed=<kn>]
  naval_fate.sh shoot <x> <y>

Options:
  --speed=<kn>  Speed in knots [default: 10]."

naval_fate() {
  eval "$(docopt "$@")"
  if $move; then
    printf "The %s is now moving to %d,%d at %d knots.\n" "$_name_" "$_x_" "$_y_" "$__speed"
  fi
  if $shoot; then
    printf "You shoot at %d,%d. It's a hit!\n" "$_x_" "$_y_"
  fi
  return 0
}
naval_fate "$@"

Run docopt.sh to insert a parser that matches the helptext into the script:

$ sudo pip3 install docopt-sh
$ docopt.sh naval_fate.sh
naval_fate.sh has been updated
$ ./naval_fate.sh Olympia move 1 5 --speed 8
The Olympia is now moving to 1,5 at 8 knots.

Note that the script is completely independent from the python package.
docopt.sh is merely a development tool to insert and update the parser, your scripts will be entirely self contained.
The parser is inlined right beneath the DOC="…​" and takes up ~80 lines of code (depending on the size of your helptext). If you ship multiple scripts in the same project you can reduce that to ~20 lines by moving the generic parts of the parser to a different file and reusing them.

Installation

Install docopt.sh using pip:

sudo pip3 install docopt-sh

How it works

Given a script docopt.sh finds the docopt (DOC="…​") help text, parses it, generates a matching parser in bash, and then inserts it back into the original script. The patched script will have no dependencies and can be shipped as a single file.

To reduce the amount of code added to the it, the script will only contain a parser made for that specific help text. For that reason there is no need for the generator itself to be written in bash, instead that part is written in Python 3.11. Though, this also means that you have to regenerate your parser every time you change the help text (see On-the-fly parser generation for automating that part while developing).

Local vs. global variables

Running docopt "$@" outputs multiple variable declarations (and a function) whose values match the command-line arguments that were used.

As an example, invoking naval_fate.sh from the quick start section with ./naval_fate.sh shoot 1 5 outputs the following.

docopt_exit() { [[ -n $1 ]] && printf "%s\n" "$1" >&2
printf "%s\n" "${DOC:12:87}" >&2; exit 1; }
declare -- __speed="10"
declare -- _name_=""
declare -- _x_="1"
declare -- _y_="5"
declare -- move="false"
declare -- shoot="true"

Evaluating (eval) this in bash will set those variables. If done in a function the variables will be local and only available inside that function (like in naval_fate.sh), otherwise they will be available globally.

Refreshing the parser

docopt.sh embeds a shasum of the help text into the parser to ensure that the two always match. In order to update the parser, simply run docopt.sh again. The existing parser will be replaced with a new one. If the parser was generated with any particular options, these options will be re-applied unless instructed otherwise with --no-auto-params.

$ docopt.sh --line-length 120 naval_fate.sh
naval_fate.sh has been updated.
$ docopt.sh naval_fate.sh
Adding `--line-length=120` from parser generation parameters that were detected
in the script. Use --no-auto-params to disable this behavior.
The parser in naval_fate.sh is already up-to-date.

Once you have generated the parser, you can move the codeblock to any other place in your script. The script patcher will automatically find the codeblock and replace it with an updated version.

In order to avoid "works on my machine" issues, the parser automatically skips the help text check on machines without shasum or sha256sum (a "command not found" error will still be printed though). The check can also manually be disabled with $DOCOPT_DOC_CHECK (see parser options for more on that).

Parser output

Names of arguments, commands, and options are mapped by replacing everything that is not an alphanumeric character with an underscore. This means --speed becomes $_speed, -f becomes $_f, and <name> becomes _name, while NAME stays as $NAME and set stays as $set.

Switches (options without arguments) and commands become true or false. If a switch or command can be specified more than once, the resulting variable value will be an integer that has been incremented the number of times the parameter was specified.

Options with values and regular arguments become strings. If an option with a value or an argument can be specified more than once, the value will be an array of strings.

To clarify, given this (somewhat complex, but concise) doc and invocation:

Usage:
  program -v... -s --val=VAL multicmd... command ARG ARGS...

$ program -vvv -s --val XY multicmd multicmd command A 1 2 3

The variables and their values will be:

_v=3 # -vvv
_s=true # -s
__val=XY # --val XY
multicmd=2 # multicmd multicmd
command=true # command
ARG=A # A
ARGS=(1 2 3) # 1 2 3

You can use $DOCOPT_PREFIX to prefix the above variable names with a custom string (e.g. specifying DOCOPT_PREFIX=prog would change ARG to progARG). See parser options for additional parser options.

Commandline options

The commandline options of docopt.sh only change how the parser is generated, while global variables specified before eval "$(docopt "$@")" itself change the behavior of the parser.

The commandline options are:

Option

Description

--line-length -n N

Max line length when minifying. Disable with 0 (default: 80)

--library -l SRC

Generates the dynamic part of the parser and includes the static parts with source SRC.

--no-auto-params -P

Disable auto-detection of parser generation parameters.

--parser -p

Output the parser instead of inserting it in the script.

--help -h

Show the help screen.

--version

Show docopt.sh version.

Parser options

Parser options change the behavior of the parser in various ways. These options are specified as global variables and must be specified before invoking eval "$(docopt "$@")". You do not need to regenerate the parse when changing any of these options.

Option

Default

Description

$DOCOPT_PROGRAM_VERSION

false

The string to print when --version is specified (false disables the option)

$DOCOPT_ADD_HELP

true

Set to false to disable the --help option

$DOCOPT_OPTIONS_FIRST

false

Set to true to treat everything after the first non-option as commands/arguments

$DOCOPT_PREFIX

""

Prefixes all variable names with the specified value

$DOCOPT_DOC_CHECK

true

Set to false to disable checking whether the parser matches the doc

$DOCOPT_LIB_CHECK

true

Set to false to disable checking whether the library version and the docopt parser version match

Exiting with a usage message

Oftentimes additional verification of parameters is necessary (e.g. when an option value is an enum). In those cases you can use docopt_exit "message" in order to output a message for the user, the function automatically appends a short usage message (i.e. the Usage: part of the doc) and then exits with code 1.

Note that this function is only defined after you have run eval "$(docopt "$@")", it is part of the docopt output.

Library mode

Instead of inlining the entirety of the parser in your script, you can move the static parts to an external file and only insert the dynamic part into your script. This is particularly useful when you have multiple bash scripts in the same project that use docopt.sh. To generate the library run docopt.sh generate-library > DEST. The output is written to stdout, so make sure to add that redirect.

Once a library has been generated you can insert the dynamic part of your parser into your script with docopt.sh --library DEST SCRIPT. The generator will then automatically add a source DEST to the parser. Make sure to quote your library path if it contains spaces like so docopt.sh --library '"/path with spaces/docopt-lib.sh"'. You do not need to specify --library on subsequent refreshes of the parser, docopt.sh will automatically glean the previously used parameters from your script and re-apply them.

--library can be any valid bash expression, meaning you can use things like "$(dirname "$0")/docopt-lib-$v.sh" (the $v is the version of docopt the parser AST was generated with).

On every invocation docopt checks that the library version and the version of the dynamic part in the script match. The parser exits with an error if that is not the case.

On-the-fly parser generation

ATTENTION: The method outlined below relies on docopt.sh being installed and is only intended for development use, do not release any scripts that use this method.

When developing a new script you might add, modify, and remove parameters quite often. Having to refresh the parser with every change can quickly become cumbersome and interrupt your workflow. To avoid this you can use the --parser flag to generate and then immediately eval the output in your script before invoking eval "$(docopt "$@")".

The script from the introduction would look like this (only eval "$(docopt.sh --parser "$0")" has been added):

#!/usr/bin/env bash

DOC="Naval Fate.
Usage:
  naval_fate.sh <name> move <x> <y> [--speed=<kn>]
  naval_fate.sh shoot <x> <y>

Options:
  --speed=<kn>  Speed in knots [default: 10]."

naval_fate() {
  eval "$(docopt.sh --parser "$0")"
  eval "$(docopt "$@")"
  if $move; then
    printf "The %s is now moving to %d,%d at %d knots.\n" "$_name_" "$_x_" "$_y_" "$__speed"
  fi
  if $shoot; then
    printf "You shoot at %d,%d. It's a hit!\n" "$_x_" "$_y_"
  fi
  return 0
}
naval_fate "$@"

Since docopt.sh is not patching the script, you also avoid any line number jumps in your IDE. However, remember to replace this with the proper parser before you ship the script.

Understanding the parser

You can turn of minifaction with -n 0. This outputs the parser in its full form. The parser and the generated AST code is heavily documented and includes references to the analyzed DOC, showing what each part does.

e.g. docopt.sh -n 0 naval_fate.sh

#!/usr/bin/env bash

DOC="Naval Fate.
  ...
  --speed=<kn>  Speed in knots [default: 10]."
# docopt parser below, refresh this parser with `docopt.sh naval_fate.library.sh`
# shellcheck disable=2016,2086,2317
docopt() {
  ...
  # This is the AST representing the parsed doc. The last node is the root.
  # Options are first, as mentioned above. The comments above each node is
  # shows what part of the DOC it is parsing (with line numbers).

  # 03   naval_fate.sh <name> move <x> <y> [--speed=<kn>]
  #                                         ~~~~~~~
  node_0(){
    value __speed 0
  }

  # 03   naval_fate.sh <name> move <x> <y> [--speed=<kn>]
  #                    ~~~~~~
  node_1(){
    value _name_ a
  }
  ...
  # Unset exported variables from parent shell
  # that may clash with names derived from the doc
  for varname in "${varnames[@]}"; do
    unset "$p$varname"
  done
  # Assign internal varnames to output varnames and set defaults

  eval $p'__speed=${var___speed:-10};'\
  ...

}
# docopt parser above, complete command for generating this parser is `docopt.sh --line-length=0 naval_fate.library.sh`

naval_fate() {
  eval "$(docopt "$@")"
  ...
}
naval_fate "$@"

Developers

Testing

docopt.sh uses pytest for testing. You can run the testsuite by executing pytest in the root of the project.

All use cases from the original docopt are used to validate correctness. Per default pytest uses the bash version that is installed on the system to run the tests. However, you can specify multiple alternate versions using --bash-version <versions>, where <versions> is a comma-separated list of bash versions (e.g. 3.2,4.0,4.1). These versions need to be downloaded and compiled first, which you can do with get_bash.py. The script downloads, extracts, configures, and compiles the specified bash versions in the tests/bash-versions folder. Use --bash-version all to test with all the bash versions that are installed.

docopt.sh's People

Contributors

andsens avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

docopt.sh's Issues

Better handling of function scoped commands

docopt.sh should allow indented DOC= strings and detect other script parts that are indented as well, so that commands that are just functions (e.g. included in .bashrc) don't require weird indentation levels.

"shasum command not found"

Thanks a lot for this script! Really useful.

I use it on Centos7 / Summit and shasum is not there. You should use instead sha256.

Thanks again,
Thomas

PS: This is the fix I implemented. I can submit a PR if you want

    if [[ -x "$(command -v shasum)" ]]; then
        doc_hash=$(printf "%s" "$DOC" | shasum -a 256)
    elif [[ -x "$(command -v shas256sum)" ]]; then
        doc_hash=$(printf "%s" "$DOC" | shasum)
    fi

Mechanism to stop processing command line arguments

Hi there!

I'm not very sure if the following should be ever possible, but we faced with this little problem and thought may worth to open an issue :)

We'd like to tell somehow, that the generated parser stops processing options. I don't know if using a delimiter like -- could help.

The use case is an script that's using docopt.sh to parse just a piece of the options given, and internally is calling an underlying command with some other options passed to the script's CLI:

resticw -p profileX stats latest --cache-dir=/tmp/cache

The goal would be to separate somehow the options that are for the outer script (the ones that need to be parsed), to the rest of the line. So in the above example, only the -p profileX option would be parseable by docopt.sh, while the rest should not.

When running the above command, we get the usage help error, as in the example --cache-dir is not something defined as an option for the script (full DOC below).

The following works just fine:

resticw -p profileX stats latest

Thanks!


https://github.com/erikw/restic-systemd-automatic-backup/blob/03d9399686d8d37188a0c100f3080f5a612df0d2/bin/resticw#L3

DOC="A little wrapper over restic just to handle profiles and environment loading.

Usage:
  resticw [options] <restic_arguments_line>...

  The <restic_arguments_line> is just the regular unwrapped restic command arguments, e.g. stats latest

Options:
  -p --profile=<name>        Specify the profile to load or use default [default: default].

Examples:
  resticw --profile profileA snapshots
  resticw stats latest  # this will use the profile: default
"

Script set -u causes error

docopt.sh version: 0.9.17

Hi @andsens, I've observed that when setting the following in scripts that are using docopt.sh, they fail:

set -u

or usually

set -euo pipefail

I tried to reproduce it with the demo in the README, so just setting the above flag after the shebang, then run and the following error shows when the script is run:

./naval.sh                           
./naval.sh: line 85: best_match_idx: unbound variable
./naval.sh: line 125: move: unbound variable./naval.sh Olympia move 1 5 --speed 8
./naval.sh: line 82: match_count: unbound variable
./naval.sh: line 125: move: unbound variable

I'm usually adding the set -eu ... as good practice for scripts. Is that incompatible in some sense with docopt or is it an issue?

Cheers!

Make AST output deterministic

Options may switch places in the AST at times, resulting differing output for the same $DOC. This is likely because dict is used, OrderDict should be used instead.

describe your docopt parser behavior and class role

It seems your parser doesn't use python docopt for parsing bash $DOC

I also realized than docopt installed from pip install docopt (0.6.2) is different from the code available on github.

Could you please explain the way you parse bash $DOC usage from bash script, and the goal of each class?

I started to analyze you code, parsing:

  • main code will call docopt_sh
    • docopt_sh() create an instance of Script object which use a Doc object to extract docopt part from the bash source script
      • Doc() extracts bash $DOC string which is matched from this regexp
      • Doc holds extracted part of the docopt string
    • Script() object then passes the doc instance to Guards object which I don't know the role yet.

Suggestion: support for debugging

I think it would nice to have a built-in mechanism to dump all recognized options and their parsed values. This could be a function included with the parser (say if DOCOPT_INCLUDE_DEBUG=true). I think it will be helpful to someone inexperienced in bash scripting, as well as a good way to show what variables are being defined etc. I would suggest including the option name, the variable name, and its value.

Shellcheck errors

Running docopt over this help string:

DOC="Upload Tool.

Usage:
  ./upload.sh [-h]

Options:
  -h  Print this help string
"

I'm getting a littany of shellcheck errors. SC1009 for a parse error on "parse_long". SC1056 for a "expected }". An undefined error on failure to parse this line left=("${left[@]:0:$i}" "${left[@]:((i+1))}").

Despite these errors, the script runs fine, so either the minification confuses shellcheck. Unsure how to resolve if this is a shellcheck problem unable to properly parse the minified parser.

Minification incorrectly breaks up some newlines.

The newline escape \ is not interpreted as expected by bash here (end of 2nd line):

done;if declare -p var_FIELD >/dev/null 2>&1; then eval $p'FIELD=("${var_FIELD'\
'[@]}")';else eval $p'FIELD=()';fi;eval $p'__purpose=${var___purpose:-'"'"'\
retrieve "$ITEMNAME"'"'"'};'$p'__mechanism=${var___mechanism:-both};'$p'__cach'\

The single line quoting should take effect here so the string is closed before we continue to the new line.

From:

DOC="Output Bitwarden item fields as bash variables
Usage:
  bitwarden-fields [options] ITEMNAME [FIELD...]

Options:
  -p --purpose PURPOSE  Specify why the master password is required.
                        The text will be appended to
                        'Enter your Bitwarden Master Password to ...'
                        [default: retrieve \"\$ITEMNAME\"]
  -m --mechanism=MECH   Use \"bw\" or \"cache\" to retrieve the item [default: both]
  --cache-for=SECONDS   Cache item with socket-credential-cache [default: 0]
  -j --json             Output as JSON instead of bash variables
  --prefix=PREFIX       Prefix variable names with supplied string
  -e                    Output exit codes as eval code (e.g. \"(exit 2)\")
  --debug               Turn on bash -x
Note:
  To retrieve attachments, prefix their name with \`attachment:\`
  For attachment IDs use \`attachmentid:\`
  To retrieve all fields, omit the FIELD argument entirely
"
# docopt parser below, refresh this parser with `docopt.sh bitwarden-fields`
# shellcheck disable=2016,2086,2317,1090,1091,2034,2154
docopt() { source "$pkgroot/.upkg/docopt-lib.sh/docopt-lib.sh" '2.0.0' || {
ret=$?;printf -- "exit %d\n" "$ret";exit "$ret";};set -e
trimmed_doc=${DOC:0:933};usage=${DOC:47:55};digest=9e0b5;options=('-p --purpos'\
'e 1' '-m --mechanism 1' ' --cache-for 1' '-j --json 0' ' --prefix 1' '-e  0' \
' --debug 0');node_0(){ value __purpose 0;};node_1(){ value __mechanism 1;}
node_2(){ value __cache_for 2;};node_3(){ switch __json 3;};node_4(){ value \
__prefix 4;};node_5(){ switch _e 5;};node_6(){ switch __debug 6;};node_7(){
value ITEMNAME a;};node_8(){ value FIELD a true;};node_9(){ optional 0 1 2 3 4 \
5 6;};node_10(){ optional 11;};node_11(){ repeatable 8;};node_12(){ sequence 9 \
7 10;};cat <<<' docopt_exit() { [[ -n $1 ]] && printf "%s\n" "$1" >&2;printf \
"%s\n" "${DOC:47:55}" >&2;exit 1;}';local varnames=(__purpose __mechanism \
__cache_for __json __prefix _e __debug ITEMNAME FIELD) varname;for varname in \
"${varnames[@]}"; do unset "var_$varname";done;parse 12 "$@";local \
p=${DOCOPT_PREFIX:-''};for varname in "${varnames[@]}"; do unset "$p$varname"
done;if declare -p var_FIELD >/dev/null 2>&1; then eval $p'FIELD=("${var_FIELD'\
'[@]}")';else eval $p'FIELD=()';fi;eval $p'__purpose=${var___purpose:-'"'"'\
retrieve "$ITEMNAME"'"'"'};'$p'__mechanism=${var___mechanism:-both};'$p'__cach'\
'e_for=${var___cache_for:-0};'$p'__json=${var___json:-false};'$p'__prefix=${va'\
'r___prefix:-};'$p'_e=${var__e:-false};'$p'__debug=${var___debug:-false};'$p'I'\
'TEMNAME=${var_ITEMNAME:-};';local docopt_i=1;[[ $BASH_VERSION =~ ^4.3 ]] && \
docopt_i=2;for ((;docopt_i>0;docopt_i--)); do for varname in "${varnames[@]}"; \
do declare -p "$p$varname";done;done;}

Support escaped quotes in DOC

The regular expression used to find the DOC string does not account for escaped quotes of any kind. Figure out if regex is still the way to go, or if a more reliable alternative should be implemented instead.

Handle escaped characters in doc check.

Characters like ` need to be escaped with a \ in double quoted docs. This will cause docopt.sh to generate the wrong hash on parser generation. The docstring should be parsed instead and then hashed.

[docopts alternative] provide name mangled compatible for docopts

Following docopt/docopts#35 : this is a twin issue with the project docopts - twin docopt/docopts#36

docopts use can use global vars with or without prefix the names in the option are converted to mangled names (code here) (which is compatible to old python's output, old code here)

Current docopt.sh code

docopts README describing the manged argument conversion.

In order to improve compatibility we need to output compatible variables name from docopt.sh

Usage could be:

DOC="
Usage: pipo new [--switch <value>]
"
DOCOPT_COMPAT_DOCOPTS=mangled
docopt "$@"

compatible Outputs:

new=true
switch=true
value='some_value'

Question: Is there any pattern or reccomended way to use docopt.sh as Git submodule?

I've just found DOCOPT and I love it!
This is just a question, maybe even a nonsense :)

I was wondering if is there any pattern to use docopt.sh as a project dependency or on contrary is this kind of use discouraged. I couldn't manage to make it work in this way.

I mean, include docopt.sh repository as a Git submodule into another project to allow regenerating the parser on-the-fly and reacting to changes on the $DOC without requiring neither to install docopt separately via pip3, nor run it explicitly from command-line ($ docopt.sh SCRIPT).

The aim would be to be able to use someting like: eval "$(./relative/path/to/docopt.sh --parser "$0")" in a script, as the docopt.sh would be present as Git submodule.

If I'm not wrong, this would keep the script clean from parser-generated code. Sure for this to work, the script should live in its Git repository to maintain access to the parser (so no portable script). The aim would only be to allow other developers to quick modify the DOC options & run transparently without extra steps, always shipping the script as a Git repository.

shellcheck disable SC2154

When declaring array vars SC2154 must be disabled, since they are not declared through an eval meaning shellcheck can discover our trickery.

Support short option for --version?

Thank you for this library, it is really useful.

It was a bit surprising that -h and --help both work, but not -v or -V in addition to --version. Would be nice for the short version to be supported as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.