colis-anr/morbig

A static parser for POSIX Shell

morbig's Issues

Does not fail on directories

When running morbig foo where foo is a directory, morbig does not fail but produces an output file foo.json containing only [].

Moreover, when running morbig foo/, morbig does not fail but produces an output file foo/.json which can be pretty annoying.

This is probably what causes #10 to happen.

Alias with trailing space in replacement word

The 2016 edition of the posix standard says:

If the value of the alias replacing the word ends in a <blank>, the shell shall check the next command word for alias substitution; this process shall continue until a word is found that is not a valid alias or an alias value does not end in a <blank>.

I have absolutely no idea what does means (what is the "next command" ?), but there probably is a reason why this was written. We should try to find out what precisely this is supposed to mean.

wrong alias substitution in case of word ending with space

there is a corner case of that weird rule of alias handling when the substitution word ends on a space:

alias x="echo "
x
x x
x x x

in the third line, morbig does not expand the third ocurrence of x. This does not happen when the
replacement word is single-quoted. Also, it happens only with at least three occurrences of the aliased
word in a row.

bad message in case of syntax error

When morbig encounters a syntactically incorrect file, it prints poorly informative
error messages such as Fatal error: exception Libmorbig.Engine.ParseError or (a bit better) Fatal error: exception Failure("Unclosed subshell (got '(').").

It should at least print the location of the error (line, character) and avoid talking about an « exception »

weird lexing error

Hello,

I get a "Failure: lexing: empty token." on the following script :
try.txt

If you remove one of the two comment lines in the middle of the file, or the line between them, then it passes.

-Ralf.

POSIX 2018

There are slight modifications in the grammar since the new POSIX 2018 version.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_10_02

In particular, the starting point of the grammar has now a name! (and it's "program")

program          : linebreak complete_commands linebreak
                 | linebreak
                 ;
complete_commands: complete_commands newline_list complete_command
                 |                                complete_command
                 ;
complete_command : list separator_op
                 | list
                 ;
...

':' wrongly stripped in word_csts

For instance the following:

${parameter:-word}
${parameter-word}

is parsed just like:

${parameter-word}
${parameter-word}

see https://github.com/colis-anr/morbig/blob/master/tests/good/2.6-word-expansions/2.6.2-parameter-expansion/basic.sh for instance.

makefile: use ocamlfind

use ocamlfind to install everything, not only the META file. Make sure that the target directory can be controlled when invicong make (with an option or environment variable)

build dependencies too strict

README.mde says that one needs ocaml <= 4.04.2 to compile. However, it seems to compile and run without problems with ocaml 4.05.0. I sugest to drop the upper limit on the version of ocaml, unless there is compelling reason to keep it.

Cleanup and document word_cst?

I'm using the word_csts in an other project (Morsmall / CoLiS-language) and I'm getting quite confused while using them:

morbig/src/CST.ml

Lines 322 to 335 in 2faccf3

	and word_component =
	| WordSubshell of subshell_kind * program located
	| WordName of string
	| WordAssignmentWord of assignment_word
	| WordDoubleQuoted of word
	| WordSingleQuoted of word
	| WordLiteral of string
	| WordVariable of variable
	| WordGlobAll
	| WordGlobAny
	| WordGlobRange of character_range
	| WordOther
	(* Empty CST. Useful to represent the absence of relevant CSTs. *)
	| WordEmpty

In particular:

1. What are the constructs that can appear in the word of a WordDoubleQuoted?

2. Same question for a WordSingleQuoted. In particular, why not have simple WordSingleQuoted of string? I guess it has to do with 3.

3. I'm still not convinced by these WordName and WordAssignmentWord constructors that are sub-cases of WordLiteral. Wouldn't that be more practical to have only WordLiteral and helpers like is_name : string -> bool and to_assignment_word : string -> name * word?

4. What kind of constructs can one find in a WordAssignmentWord that occurs inside a WordDoubleQuoted (because I guess it can happen)? And inside a WordSingleQuoted?

5. I assume that WordEmpty is the same thing as WordLiteral ''?

6. Do we still need WordOther?

Also, I think we should document all this. I can take care of that, we just need to decide where.

incorrect contents of single-quoted words

when parsing the shell script containing the single line

'a'

that is letter a in single quotes, the simple json output contains

"CmdName_Word", [ "Word", "'"a"", (rest omitted)

with only one single quote at the beginning, and spurious double quotes.

Tagged release?

Hi there,
In e9c9070 you populated the VERSION file with "0.1". Does that mean this commit is the release 0.1? If so, would you mind adding a git tag?

$ git tag 0.1 e9c90708fc64790f2abeb6021d289daa6fe2acdc
$ git push --tags origin master

GitHub automatically generates tarballs for tags, so it’s easier to download a stable version instead of relying on the relative stableness of HEAD.

Thanks!

interaction of alias expansion and parsing

there seems to be a problem with the way alias expansion is integrated in the engine. If alias expansion has an impact on the grammatical structure of the script then this is not recognized properly by morbig, at least not in the following example:

#!/bin/sh

alias si='if'
alias alors='then'
alias sinon='else'
alias cesttout='fi'
alias blabla='echo'

si [ -n a ]
alors
	blabla oui
sinon
	blabla non
cesttout

Morbig recognizes this as a sequence of simple commands insetad of an if-then-else.

alias and unalias can take multiple atrguments

According to POSIX (see "alias" and "unalias" in the Chapter "4: Utilities" of the volume "Shell & Utilities"), both alias and unalias can take multiple arguments, like this :

alias show=echo list=ls
unalias show list

but morbig does not accept this

--as dot is not working

morbig --help promises a dot output fotrmat. However,

% morbig --as dot postinst
Failure: Not me!
.

unquoting : \ inside double quotes not treated correctly

Unquoting is applied to the delimiter of a here document. However, morbig does not treat correctly the case of a backslash inside double quotes. Standard 2.2.3 says about the text inside double quotes:

The shall retain its special meaning as an escape character (see Escape Character (Backslash)) only when followed by one of the following characters when considered special: $ ` " \

That is, the following script should be correct as the \O inside the double quotes shall not be replaced by O but remain intact:

if true
then cat <<"E\OF"
hello
E\OF
fi

dash does this correctly, but morbig thinks that the here document is not completed.

I have added test cases to tests/good/ 2.7-redirection/2.7.4-here-document:

delimiter-backslash-in-double-quotes-todrop.sh
delimiter-backslash-in-double-quotes-tokeep.sh

Design the API

Discussion open! @Niols, shall you start?

continuation lines in here documents

when the delimiter word of a here document is not quoted, the symbol behaves as inside
double quotes (standard 2.7.4). That is, indicates a continuation line. For instance
(test case continuation-line-in-here-documents):

if true
then
cat << EOF
first line
second line
third line with continuation\
EOF
forth and last line ((
EOF
else echo "baeh"
fi

the here document is ended in line 9, not line 7.

Unable to match the ParseError exception when using Libmorbig

When using Libmorbig and parsing a incorrect file, I get:

Fatal error: exception Libmorbig.Errors.ParseError(_)

but when I try to try ... match Libmorbig.Errors.ParseError _ -> ..., I get Error: Unbound module Libmorbig.Errors.

mlpack : make types visible

revisit the mlpack file and make sure that types are correctly exported

dependencies too strong ?

The dependencies in the README file seem to be a bit strong. I can compile morbig on
debian stable, which has older versions than mentioned in the README for the following:

libyojson-ocaml 1.3.2-1+b1
libyojson-ocaml-dev 1.3.2-1+b1
libppx-visitors-ocaml-dev 20170404-1

-Ralf.

Positions are wrong

Morbig miscalculates positions in the input file. For instance, on the input file contining only one line:

true

Morbig reports the word "true" starting at column 4 and ending and columns 5.

possible bug: unclosed subshell

With morbig 20180209 I get an Unclosed Subshell error on the following input:

#!/bin/sh

f() {
    tempdir=`mktemp --directory --suffix=.aptlc`
}

Seems that morbig does not like the "=" sign in the second option". -Ralf.

makefile: doc

add a doc target to the makefile to generate ocamldoc

alias expansion : aliasing alias

Morbig gives an exception Engine.ParseError on the following file:

alias rename=alias
rename endif=fi
if true; then echo "hello"; endif

I cannot see anything in the posix specification that excludes this kind of alias. dash (debian 0.5.8-2.5) executes this correctly, and prints "hello". Interestingly, bash (debian 4.4-5) does not, and gives an error message "rename: command not found".

continuous integration

Troubles when parsing brace-group + here-document with file descriptor

When parsing something like:

{ cat;} 9<<EOF
a
EOF

Morbig complains:

line 1, character 11: Syntax error.

It does not happen when there is no brace group or when there is no file descriptor. It seems to me that it should be accepted by the grammar. dash, bash and zsh do accept the script.

I added a test (68a1cd5) for this particular case.

bug: expansion in here documents

from an anonymous reviewer:

Consider this shell script:

cat <<'EOT'
abc ` def
ghi \
jkl
EOT

The delimiting word is quoted, so that the here-document lines should not be
expanded (§2.7.4). Morbig gives me a here-document containing "abc ` def\nghi
jkl\n". The backslash should have been included verbatim, not treated as
escaping the line break.

If I then put the same document inside a command substitution:

x=$(cat <<'EOT'
abc ` def
ghi \
jkl
EOT
)

then I get an error "Lexing error: unbalanced parentheses". Removing or escaping
the backtick removes the error, but being inside a command substitution should
not change the parsing of here-documents (Per §2.6.3, "Any valid shell script
can be used for command"). The given word is then "$(cat <<'EOT' abc \` def
ghi jkl EOT )", which is missing all of the newlines - I'm not sure whether
that's presentational or another difference.

package for OPAM

requires #2 and #4

the word_cst parser does not recognize globs and character ranges

*, ? and character ranges are always parsed as literals

git migration

finnish the migration of colis.git/software/shparser

Parsing from channel and interactive parsing

This would require a better encapsulation of the lexing buffer.

debug mode should print the pathnames of input files

with the --debug option, and when using --from-stdin, morbig should also tell the pathname of each
script it opens. That would help in case of an error on a large corpus of scripts.

what is the minimal version of ocaml required for building morbig?

README.md says >= 4.02.1, opam says >= 4.02.3. Which one is right?

"Inconsistent assumptions over interface Options" when compiling other projects

Compiling an other project (in this case, shstats) that also has an Options module leads to an error:

Error: The files /path/to/libmorbig/libmorbig.cmi
       and options.cmi make inconsistent assumptions over interface Options

This has not always been the case. I bisected this bug and found out that the last known good commit was 245208c and that the first bad commit is 0ca98a7. Sadly, there are two commits in between that do not compile, and 0ca98a7 is a very large commit, making it hard to find what went wrong.

I did not understand quite well what is causing the issue in shstats, and I didn't manage to produce a minimal example (although I'm still working on it). But it probably should not happen in any way in my opinion.

I will try to find more information about that and I keep you posted.

review exported identifiers

Just a TODO, not a bug: we should review what precisely is exported in the library. For instance, we probably do not want to export all of CSTHelpers.

Travis on OSX

We should integrate OSX as a target of our travis-based continuous integration.

Fatal error: exception Sys_error("plan.sjson: Is a directory") with docker image

Hi,
using the docker image and defining the shell function, I encounter a Fatal error: exception Sys_error("script.sjson: Is a directory") error. The file mentioned is created, but empty.

Is there a workaround?

I haven't tried compiling morbig myself yet, so I don't know if the error also happens in that case.

backslash in here document shoudl behave as inside double quotes when delimiter quoted

When any part of the delimiter of a here document is quoted, the backslash in the here document shall behave as inside doubel quotes, that is it has a special meaning only when followed by $ ` " \ .

For example:

cat << "EOF1"
\a$\
EOF1
cat << EOF2
\a$\
EOF2

In the first case, the contents of the here documt shall be

\a$\

while in the second case it shall be

\a$\

Looking at the module hereDocument, it turns out that the flags which are pushed on the stack expanded are in fact never used (or poped).

C inteface

% cd examples/c/
% make
ocamlfind: Package `libmorbig' not found
ocamlfind: Package `libmorbig' not found
cc -o dump -I /usr/lib/ocaml -I  \
        dump.c -lm -ldl /libmorbigc.a
cc: error: /libmorbigc.a: No such file or directory
Makefile:7: recipe for target 'all' failed
make: *** [all] Error 1

after changing in examples/c/Makefile all libmorbig to morbig:

% make
 cc -o dump -I /usr/lib/ocaml -I /usr/local/lib/ocaml/4.02.3/morbig \
        dump.c -lm -ldl /usr/local/lib/ocaml/4.02.3/morbig/morbigc.a
 cc: error: /usr/local/lib/ocaml/4.02.3/morbig/morbigc.a: No such file or directory
 Makefile:7: recipe for target 'all' failed

bad message in case of lexical error

Issue #15 is fixed but that was only for syntax errors. The same should be done for lexical errors, for instance with the test cases from tests/bad/2.3_token-recognition:

% morbig unterminated-backquote.sh
Fatal error: exception Prelexer.LexingError("Unterminated nesting!")

-Ralf.

Shouldn't parse_file return a complete command located list?

Is there a reason why Morbig's parse_file function returns a list of CST.complete_command and not CST.complete_command located? Wouldn't we want that information?

${#*} → Failure: lexing: empty token.

When running morbig --as simple on the file at the end of this issue, I get

Failure: lexing: empty token.

as an output from Morbig. The Shell script seems to be accepted by other shells. I'll try to narrow this script down to the place where the "bug" occurs.

---

if [ "${0#/usr/share/initramfs-tools/hooks/}" != "$0" ] ||
        [ "${0#/etc/initramfs-tools/hooks/}" != "$0" ]; then
    # called from an initramfs-tools hook script
    TABFILE="$DESTDIR/cryptroot/crypttab"
elif [ "${0#/scripts/}" != "$0" ]; then
    # called at initramfs stage from a boot script
    TABFILE="/cryptroot/crypttab"
else
    TABFILE="${TABFILE-/etc/crypttab}"
fi
export DM_DEFAULT_NAME_MANGLING_MODE=hex # for dmsetup(8)

# Logging helpers. Send the argument list to plymouth(1), or fold it
# and print it to the standard error.
cryptsetup_message() {
    local IFS=' '
    if [ "${0#/scripts/}" != "$0" ] && [ -x /bin/plymouth ] && plymouth --ping; then
        plymouth message --text="cryptsetup: $*"
    elif [ ${#*} -lt 70 ]; then
        echo "cryptsetup: $*" >&2
    else
        # use busybox's fold(1) and sed(1) at initramfs stage
        echo "cryptsetup: $*" | fold -s | sed '1! s/^/    /' >&2
    fi
    return 0
}

# crypttab_parse_options([--export], [--quiet])
#   Parse $_CRYPTTAB_OPTIONS, a comma-separated option string from the
#   crypttab(5) 4th column, and sets corresponding variables
#   CRYPTTAB_OPTION_<option>=<value> (which are added to the environment
#   if --export is set).
#   For error and warning messages, CRYPTTAB_NAME, (resp. CRYPTTAB_KEY)
#   should be set to the (unmangled) mapped device name (resp. key
#   file).
#   Return 1 on parsing error, 0 otherwise (incl. if unknown options
#   were encountered).
crypttab_parse_options() {
    local quiet="n" export="n"
    while [ $# -gt 0 ]; do
        case "$1" in
            --quiet) quiet="y";;
            --export) export="y";;
            *) cryptsetup_message "WARNING: crypttab_parse_options(): unknown option $1"
        esac
        shift
    done

    local IFS=',' x OPTION VALUE
    unset -v CRYPTTAB_OPTION_cipher \
             CRYPTTAB_OPTION_size \
             CRYPTTAB_OPTION_hash \
             CRYPTTAB_OPTION_offset \
             CRYPTTAB_OPTION_skip \
             CRYPTTAB_OPTION_verify \
             CRYPTTAB_OPTION_readonly \
             CRYPTTAB_OPTION_discard \
             CRYPTTAB_OPTION_plain \
             CRYPTTAB_OPTION_luks \
             CRYPTTAB_OPTION_tcrypt \
             CRYPTTAB_OPTION_veracrypt \
             CRYPTTAB_OPTION_swap \
             CRYPTTAB_OPTION_tmp \
             CRYPTTAB_OPTION_check \
             CRYPTTAB_OPTION_checkargs \
             CRYPTTAB_OPTION_tries \
             CRYPTTAB_OPTION_initramfs \
             CRYPTTAB_OPTION_noearly \
             CRYPTTAB_OPTION_noauto \
             CRYPTTAB_OPTION_loud \
             CRYPTTAB_OPTION_quiet \
             CRYPTTAB_OPTION_keyscript \
             CRYPTTAB_OPTION_keyslot \
             CRYPTTAB_OPTION_header \
             CRYPTTAB_OPTION_tcrypthidden
    # use $_CRYPTTAB_OPTIONS not $CRYPTTAB_OPTIONS as options values may
    # contain '\054' which is decoded to ',' in the latter
    for x in $_CRYPTTAB_OPTIONS; do
        OPTION="${x%%=*}"
        VALUE="${x#*=}"
        if [ "$x" = "$OPTION" ]; then
            unset -v VALUE
        else
            VALUE="$(printf '%b' "$VALUE")"
        fi
        if ! crypttab_validate_option; then
            if [ "$quiet" = "n" ]; then
                cryptsetup_message "ERROR: $CRYPTTAB_NAME: invalid value for '${x%%=*}' option, skipping"
            fi
            return 1
        elif [ -z "${OPTION+x}" ]; then
            continue
        fi
        if [ "$export" = "y" ]; then
            export "CRYPTTAB_OPTION_$OPTION"="${VALUE-yes}"
        else
            eval "CRYPTTAB_OPTION_$OPTION"='${VALUE-yes}'
        fi
    done
    IFS=" "

    if [ "$quiet" = "n" ] &&
            [ -z "${CRYPTTAB_OPTION_luks+x}" ] && [ -n "${CRYPTTAB_OPTION_header+x}" ]; then
        cryptsetup_message "WARNING: Option 'luks' missing in crypttab for target $CRYPTTAB_NAME." \
                           "Headers are only supported for LUKS devices."
    fi
    if [ -z "${CRYPTTAB_OPTION_luks+x}" ] && [ -z "${CRYPTTAB_OPTION_tcrypt+x}" ]; then
        # the compiled-in default for these are subject to change
        options='cipher size'
        if [ -n "${CRYPTTAB_OPTION_keyscript+x}" ] || [ "$CRYPTTAB_KEY" = "none" ]; then
            options="$options hash" # --hash is being ignored in plain mode with keyfile specified
        fi
        for o in $options; do
            if [ "$quiet" = "n" ] && eval [ -z "\${CRYPTTAB_OPTION_$o+x}" ]; then
                cryptsetup_message "WARNING: Option '$o' missing in crypttab for plain dm-crypt" \
                    "mapping $CRYPTTAB_NAME. Please read /usr/share/doc/cryptsetup/README.initramfs and" \
                    "add the correct '$o' option to your crypttab(5)."
            fi
        done
    fi
}

# crypttab_validate_option()
#   Validate $OPTION=$VALUE (or flag $OPTION if VALUE is unset).  return
#   1 on error, unsets OPTION for unknown or useless options.
crypttab_validate_option() {
    # option aliases
    case "$OPTION" in
        read-only) OPTION="readonly";;
        key-slot) OPTION="keyslot";;
        tcrypt-hidden) OPTION="tcrypthidden";;
        tcrypt-veracrypt) OPTION="veracrypt";;
    esac

    # sanitize the option name so CRYPTTAB_OPTION_$OPTION is a valid variable name
    local o="$OPTION"
    case "$o" in
        keyfile-offset) OPTION="keyfile_offset";;
        keyfile-size) OPTION="keyfile_size";;
    esac

    case "$o" in
        # value must be a non-empty string
        cipher|hash|header)
            [ -n "${VALUE:+x}" ] || return 1
        ;;
        # numeric options >0
        size|keyfile-size)
            if ! printf '%s' "${VALUE-}" | grep -Exq "0*[1-9][0-9]*"; then
                return 1
            fi
        ;;
        # numeric options >=0
        offset|skip|tries|keyslot|keyfile-offset)
            if ! printf '%s' "${VALUE-}" | grep -Exq "[0-9]+"; then
                return 1
            fi
        ;;
        tmp)
            if [ -z "${VALUE+x}" ]; then
                VALUE="ext4" # 'tmp flag'
            elif [ -z "$VALUE" ]; then
                return 1
            fi
        ;;
        check)
            if [ -z "${VALUE+x}" ]; then
                if [ -n "${CRYPTDISKS_CHECK-}" ]; then
                    VALUE="$CRYPTDISKS_CHECK"
                else
                    unset -v OPTION
                    return 0
                fi
            fi
            if [ -x "/lib/cryptsetup/checks/$VALUE" ] && [ -f "/lib/cryptsetup/checks/$VALUE" ]; then
                VALUE="/lib/cryptsetup/checks/$VALUE"
            elif [ ! -x "$VALUE" ] || [ ! -f "$VALUE" ]; then
                return 1
            fi
        ;;
        checkargs)
            [ -n "${VALUE+x}" ] || return 1 # must have a value (possibly empty)
        ;;
        keyscript)
            [ -n "${VALUE:+x}" ] || return 1 # must have a value
            if [ "${VALUE#/}" = "$VALUE" ]; then
                VALUE="/lib/cryptsetup/scripts/$VALUE"
            fi
            if [ ! -x "$VALUE" ] || [ ! -f "$VALUE" ]; then
                return 1
            fi
        ;;
        # and now the flags
        verify) ;;
        loud) ;;
        quiet) ;;
        initramfs) ;;
        noearly) ;;
        noauto) ;;
        readonly) ;;
        discard) ;;
        plain) ;;
        luks) ;;
        swap) ;;
        tcrypt) ;;
        veracrypt) ;;
        tcrypthidden) ;;
        *)
            if [ "${quiet:-n}" = "n" ]; then
                cryptsetup_message "WARNING: $CRYPTTAB_NAME: ignoring unknown option '$o'";
            fi
            unset -v OPTION
        ;;
    esac
}

# crypttab_resolve_source()
#   Resolve the CRYPTTAB_SOURCE variable, containing value of the second
#   field of a crypttab(5)-like file.
#   On error (non-existing source), CRYPTTAB_SOURCE is not changed and 1
#   is returned.
crypttab_resolve_source() {
    # return immediately if source is a regular file
    [ ! -f "$CRYPTTAB_SOURCE" ] || return 0
    # otherwise resolve the block device specification
    local dev="$CRYPTTAB_SOURCE"
    dev="$(resolve_device_spec "$dev")" && CRYPTTAB_SOURCE="$dev" || return 1
}

# run_keyscript($keyscriptarg, $tried_count)
#   exec()'ute `$CRYPTTAB_OPTION_keyscript $keyscriptarg`.
#   If $CRYPTTAB_OPTION_keyscript is unset or null and $keyscriptarg is
#   "none" (meaning the passphrase is to be read interactively from the
#   console), use `/lib/cryptsetup/askpass` as keyscript with a suitable
#   prompt message.
#   Since the shell process is replaced with the $CRYPTTAB_OPTION_keyscript
#   program, run_keyscript() must be used on the left-hand side of a
#   pipe, or similar.
run_keyscript() {
    local keyscriptarg="$1" CRYPTTAB_TRIED="$2" keyscript;
    export CRYPTTAB_NAME CRYPTTAB_SOURCE CRYPTTAB_OPTIONS
    export CRYPTTAB_TRIED

    if [ -n "${CRYPTTAB_OPTION_keyscript+x}" ] && \
            [ "$CRYPTTAB_OPTION_keyscript" != "/lib/cryptsetup/askpass" ]; then
        # 'keyscript' option is present: export its argument as
        # $CRYPTTAB_KEY
        export CRYPTTAB_KEY="$keyscriptarg"
        keyscript="$CRYPTTAB_OPTION_keyscript"
    elif [ "$keyscriptarg" = none ]; then
        # don't export the prompt message as CRYPTTAB_KEY
        keyscript="/lib/cryptsetup/askpass"
        keyscriptarg="Please unlock disk $CRYPTTAB_NAME: "
    fi

    exec "$keyscript" "$keyscriptarg"
}

# get_crypt_type()
#    Return the mapping's type, depending on its
#    $CRYPTTAB_OPTION_<option> values
get_crypt_type() {
    local type="plain" # assume plain dm-crypt device by default
    if [ "${CRYPTTAB_OPTION_tcrypt-}" = "yes" ]; then
        type="tcrypt"
    elif [ "${CRYPTTAB_OPTION_plain-}" = "yes" ]; then
        type="plain"
    elif [ "${CRYPTTAB_OPTION_luks-}" = "yes" ] ||
            /sbin/cryptsetup isLuks -- "${CRYPTTAB_OPTION_header-$CRYPTTAB_SOURCE}"; then
        type="luks"
    fi
    echo "$type"
}

# unlock_mapping([$keyfile])
#   Run cryptsetup(8) with suitable options and arguments to unlock
#   $CRYPTTAB_SOURCE and setup dm-crypt managed device-mapper mapping
#   $CRYPTTAB_NAME.
unlock_mapping() {
    local keyfile="${1:--}"

    if [ -n "${CRYPTTAB_OPTION_header+x}" ] && [ ! -f "$CRYPTTAB_OPTION_header" ]; then
        cryptsetup_message "ERROR: $CRYPTTAB_NAME: LUKS header '$CRYPTTAB_OPTION_header' missing"
        return 1
    fi

    local type="$(get_crypt_type)"
    if [ "$type" = "luks" ] || [ "$type" = "tcrypt" ]; then
        # ignored for LUKS and TCRYPT devices
        unset -v CRYPTTAB_OPTION_cipher \
                 CRYPTTAB_OPTION_size \
                 CRYPTTAB_OPTION_hash \
                 CRYPTTAB_OPTION_offset \
                 CRYPTTAB_OPTION_skip
    fi
    if [ "$type" = "plain" ] || [ "$type" = "tcrypt" ]; then
        unset -v CRYPTTAB_OPTION_keyfile_size
    fi
    if [ "$type" = "tcrypt" ]; then
        # ignored for TCRYPT devices
        unset -v CRYPTTAB_OPTION_keyfile_offset
    else
        # ignored for non-TCRYPT devices
        unset -v CRYPTTAB_OPTION_veracrypt CRYPTTAB_OPTION_tcrypthidden
    fi

    if [ "$type" != "luks" ]; then
        # ignored for non-LUKS devices
        unset -v CRYPTTAB_OPTION_keyslot
    fi

    /sbin/cryptsetup -T1 \
        ${CRYPTTAB_OPTION_header:+--header="$CRYPTTAB_OPTION_header"} \
        ${CRYPTTAB_OPTION_cipher:+--cipher="$CRYPTTAB_OPTION_cipher"} \
        ${CRYPTTAB_OPTION_size:+--key-size="$CRYPTTAB_OPTION_size"} \
        ${CRYPTTAB_OPTION_hash:+--hash="$CRYPTTAB_OPTION_hash"} \
        ${CRYPTTAB_OPTION_offset:+--offset="$CRYPTTAB_OPTION_offset"} \
        ${CRYPTTAB_OPTION_skip:+--skip="$CRYPTTAB_OPTION_skip"} \
        ${CRYPTTAB_OPTION_verify:+--verify-passphrase} \
        ${CRYPTTAB_OPTION_readonly:+--readonly} \
        ${CRYPTTAB_OPTION_discard:+--allow-discards} \
        ${CRYPTTAB_OPTION_veracrypt:+--veracrypt} \
        ${CRYPTTAB_OPTION_keyslot:+--key-slot="$CRYPTTAB_OPTION_keyslot"} \
        ${CRYPTTAB_OPTION_tcrypthidden:+--tcrypt-hidden} \
        ${CRYPTTAB_OPTION_keyfile_size:+--keyfile-size="$CRYPTTAB_OPTION_keyfile_size"} \
        ${CRYPTTAB_OPTION_keyfile_offset:+--keyfile-offset="$CRYPTTAB_OPTION_keyfile_offset"} \
        --type="$type" --key-file="$keyfile" \
        open -- "$CRYPTTAB_SOURCE" "$CRYPTTAB_NAME"
}

# crypttab_key_check()
#   Sanity checks for keyfile $CRYPTTAB_KEY.  CRYPTTAB_NAME and
#   CRYPTTAB_OPTION_<option> must be set appropriately.
crypttab_key_check() {
    if [ ! -f "$CRYPTTAB_KEY" ] && [ ! -b "$CRYPTTAB_KEY" ] && [ ! -c "$CRYPTTAB_KEY" ] ; then
        cryptsetup_message "WARNING: $CRYPTTAB_NAME: keyfile '$CRYPTTAB_KEY' not found"
        return 0
    fi

    if [ "$CRYPTTAB_KEY" = "/dev/random" ] || [ "$CRYPTTAB_KEY" = "/dev/urandom" ]; then
        if [ -n "${CRYPTTAB_OPTION_luks+x}" ] || [ -n "${CRYPTTAB_OPTION_tcrypt+x}" ]; then
            cryptsetup_message "WARNING: $CRYPTTAB_NAME: has random data as key"
            return 1
        else
            return 0
        fi
    fi

    local mode="$(stat -L -c"%04a" -- "$CRYPTTAB_KEY")"
    if [ $(stat -L -c"%u" -- "$CRYPTTAB_KEY") -ne 0 ] || [ "${mode%00}" = "$mode" ]; then
        cryptsetup_message "WARNING: $CRYPTTAB_NAME: key file $CRYPTTAB_KEY has" \
            "insecure ownership, see /usr/share/doc/cryptsetup/README.Debian."
    fi
}

# resolve_device_spec($spec)
#   Resolve LABEL=<label>, UUID=<uuid>, PARTUUID=<partuuid> and
#   PARTLABEL=<partlabel> to a block special device.  If $spec is
#   already a (link to a block special device) then it is echoed as is.
#   Return 1 if $spec doesn't correspond to a block special device.
resolve_device_spec() {
    local spec="$1"
    case "$spec" in
        UUID=*|LABEL=*|PARTUUID=*|PARTLABEL=*)
            # don't use /dev/disk/by-label/... to avoid gessing udev mangling
            spec="$(blkid -l -t "$spec" -o device)" || spec=
        ;;
    esac
    [ -b "$spec" ] && printf '%s\n' "$spec" || return 1
}

# dm_blkdevname($name)
#   Print the mapped device name, or return 1 if the the device doesn't exist.
dm_blkdevname() {
    local name="$1" dev
    # /dev/mapper/$name isn't reliable due to udev mangling
    if dev="$(dmsetup info -c --noheadings -o blkdevname -- "$name" 2>/dev/null)" &&
            [ -n "$dev" ] && [ -b "/dev/$dev" ]; then
        echo "/dev/$dev"
        return 0
    else
        return 1
    fi
}

# crypttab_find_entry([--quiet], $target)
#   Search in the crypttab(5) for the given $target, and sets the
#   variables CRYPTTAB_NAME, CRYPTTAB_SOURCE, CRYPTTAB_KEY and
#   CRYPTTAB_OPTIONS accordingly.  (In addition _CRYPTTAB_NAME,
#   _CRYPTTAB_SOURCE, _CRYPTTAB_KEY and _CRYPTTAB_OPTIONS are set to the
#   unmangled values before decoding the escape sequence.)  If there are
#   duplicates then only the first match is considered.
#   Return 0 if a match is found, and 1 otherwise.
crypttab_find_entry() {
    local target="$1" quiet="n" IFS
    if [ "$target" = "--quiet" ] && [ $# -eq 2 ]; then
        quiet="y"
        target="$2"
    fi

    if [ -f "$TABFILE" ]; then
        while IFS=" 	" read -r _CRYPTTAB_NAME _CRYPTTAB_SOURCE _CRYPTTAB_KEY _CRYPTTAB_OPTIONS; do
            if [ "${_CRYPTTAB_NAME#\#}" != "$_CRYPTTAB_NAME" ] || [ -z "$_CRYPTTAB_NAME" ]; then
                # ignore comments and empty lines
                continue
            fi

            # unmangle names
            CRYPTTAB_NAME="$(printf '%b' "$_CRYPTTAB_NAME")"
            if [ -z "$_CRYPTTAB_SOURCE" ] || [ -z "$_CRYPTTAB_KEY" ] || [ -z "$_CRYPTTAB_OPTIONS" ]; then
                cryptsetup_message "WARNING: '$CRYPTTAB_NAME' is missing some arguments, see crypttab(5)"
                continue
            elif [ "$CRYPTTAB_NAME" = "$target" ]; then
                CRYPTTAB_SOURCE="$( printf '%b' "$_CRYPTTAB_SOURCE" )"
                CRYPTTAB_KEY="$(    printf '%b' "$_CRYPTTAB_KEY"    )"
                CRYPTTAB_OPTIONS="$(printf '%b' "$_CRYPTTAB_OPTIONS")"
                return 0
            fi
        done <"$TABFILE"
    fi

    if [ "$quiet" = "n" ]; then
        cryptsetup_message "WARNING: target '$target' not found in $TABFILE"
    fi
    return 1
}

# crypttab_foreach_entry($callback)
#   Iterate through the crypttab(5) and run the given $callback for each
#   entry found. Variables CRYPTTAB_NAME, CRYPTTAB_SOURCE, CRYPTTAB_KEY
#   and CRYPTTAB_OPTIONS are set accordingly and available to the
#   $callback.  (In addition _CRYPTTAB_NAME, _CRYPTTAB_SOURCE,
#   _CRYPTTAB_KEY and _CRYPTTAB_OPTIONS are set to the unmangled values
#   before decoding the escape sequence.)
#   Return 0 if a match is found, and 1 otherwise.
crypttab_foreach_entry() {
    local callback="$1" IFS
    local _CRYPTTAB_NAME _CRYPTTAB_SOURCE _CRYPTTAB_KEY _CRYPTTAB_OPTIONS \
           CRYPTTAB_NAME  CRYPTTAB_SOURCE  CRYPTTAB_KEY  CRYPTTAB_OPTIONS

    [ -f "$TABFILE" ] || return
    while IFS=" 	" read -r _CRYPTTAB_NAME _CRYPTTAB_SOURCE _CRYPTTAB_KEY _CRYPTTAB_OPTIONS <&9; do
        if [ "${_CRYPTTAB_NAME#\#}" != "$_CRYPTTAB_NAME" ] || [ -z "$_CRYPTTAB_NAME" ]; then
            # ignore comments and empty lines
            continue
        fi

        # unmangle names
        CRYPTTAB_NAME="$(   printf '%b' "$_CRYPTTAB_NAME"   )"
        CRYPTTAB_SOURCE="$( printf '%b' "$_CRYPTTAB_SOURCE" )"
        CRYPTTAB_KEY="$(    printf '%b' "$_CRYPTTAB_KEY"    )"
        CRYPTTAB_OPTIONS="$(printf '%b' "$_CRYPTTAB_OPTIONS")"

        if [ -z "$CRYPTTAB_SOURCE" ] || [ -z "$CRYPTTAB_KEY" ] || [ -z "$CRYPTTAB_OPTIONS" ]; then
            cryptsetup_message "WARNING: '$CRYPTTAB_NAME' is missing some arguments, see crypttab(5)"
            continue
        fi

        "$callback" 9<&-
    done 9<"$TABFILE"
}

# vim: set filetype=sh :

makefile: uninstall

add a target to the makefile to uninstall

Assertion failure in prelexer

on the corpus sid.amd64.main+contrib+non-free.2018-05-23:

$ examples/scripts/debian-maintainer-scripts corpus
Cleaning up ... done.
Start parsing 31302 files.
Fatal error: exception File "prelexerState.ml", line 287, characters 31-37: Assertion failed
Command exited with non-zero status 2

I am trying to find out on which file this happens, but this is surprisingly not so easy.

alias expansion should apply only to the command word of a simple command

The Posix standard says:

... resulting word that is identified to be the command name word of a simple command shall be examined to determine whether it is an unquoted, valid alias name ...

In my understanding (which may be wrong) this means that only a WORD which is obtained
through this sub-derivation

simple_command -> cmd_name -> WORD

or maybe as the first WORD in

simple_command -> cmd_name cmd_suffix -> WORD cmd_suffix

can be candidate for alias substitution. How these can be found before applying any grammar rules, as the standard requires, escapes me but that is a different matter.

For instance, the following shell script

alias show=echo
show show

should print "show", and "not "echo" . Dash does this right, but when you look at the json produced by morbig you see that it has in fact replaced both occurrences of "show".

The same error occurs on toplevel-unalias.sh in the tests/good directory: If you look at the json produced by morbig you see that the unalias has not been executed. The reason for this is
that the first alias has also replaced the argument of the succeeding unalias, making it ineffective.

What's the point of having the input filename in the "binary" output of Morbig?

I don't really see the use case. In particular because it depends on where morbig has been called.

Is it really useful to have that in the marshalized value?

alias expansion does not always show in word components

When parsing the following file

alias x='true '
x

I obtain (only the simgle command corresponding to the second line shown)

"SimpleCommand_CmdName",
              [
                "CmdName_Word",
                [ "Word", "'true '", [ [ "WordLiteral", "x" ] ] ]
              ]

The x in the list of word components should have been replaced.

The end marker of here-documents ends up in the word in the CST

When parsing here-documents, the current version of Morbig includes in the word describing their contents the end marker.

Here is an example in for_loop_here_document.sh where the contents of the here-document should be foo\n and the separator should be DATA. Morbig indeed says that the marker is DATA. But it also says that the contents is foo\nDATA, which is wrong:

morbig/tests/good/2.7-redirection/2.7.4-here-document/for_loop_here_document.sh.expected

Line 100 in 816ea4c

"foo\nDATA",

C API

write a C API