Bash Shell Mock

Table of Contents

Bash Shell Mock

Overview

Shellmock is a bash shell script mocking utility/framework. It was written to be a companion of the Bash Automated Testing System.

Typically, mocking frameworks return certain outputs when particular inputs are provided. When enabling mocks for scripts, Shellmock defines the "input" as the command line arguments to the script and it defines the "output" as the exit status and any standard output. In addition it will allow you to provide an alternate stubbed behavior. In most testing scenarios just knowing what was called and the command line args is sufficient, but some advanced test cases may require new behavior.

Given the bash command "cp file1 file2", a stub might be defined to return status 0. The stub could also echo "cp file1 file2" to standard out. Using Bats status, line or output variables you can verify that "cp file1 file2" was written which would confirm that the script was called. The example below is a snippet from a bats script. It is assumed that mycmd is the script being tested and it calls "cp file1 file2" during its execution.

run mycmd
[ "${status}" = "0" ]
[ "${line[0]}" = "cp file1 file2" ]

One approach for stubbing is to create bash scripts with the same names as the real scripts or programs and then override the PATH to the stubs and there by short circuiting the real path. Depending on the number of scripts to stub this can be taxing and the stubbing logic can become complex. This is where Shellmock helps. Shellmock lets you define the mocks inline within the bats tests. It creates and manages the stub scripts behind the scenes. The added benefit is that you can view the tests and the test data together making tests easier to manage.

Enabling Mocks in your Bats tests

The first step is to source shellmock in the setup() function and call the shellmock_cleanup function from the bats teardown function. These two steps will make shellmock functions available to your testcases and perform appropriate cleanup in between tests.

Adding the skipIfNot function in setup() will help with troubleshooting. This will allow users to define the TEST_FUNCTION environment variable to run a single test or a subset of tests and turn on the associated debug logs.

The sample.sh script provided in the shellmock repo provides an example of how to setup shellmock in a bats script.

setup()
{
    # Source the shellmock functions into the shell.
    . shellmock

    skipIfNot "$BATS_TEST_DESCRIPTION"

    shellmock_clean
}

teardown()
{
    if [ -z "$TEST_FUNCTION" ]; then
        shellmock_clean
    fi
}

Environment Variables

Name	Purpose
TEST_FUNCTION	Shellmock variable that is used to run single tests and control debugging output in shellmock. When set to the name of a test case then all tmp and debug output is preserved after the test completes. If you give each test a unique name then only that one test will be executed. You have to leverage the skipIfNot function in setup or in the test case to take advantage of the feature.
SHELLMOCK_V1_COMPATIBILITY	v1.2 introduced logic that handles matching arguments differently when they contain spaces. As a result old tests could fail to match. This flag allows the old tests to continue to pass.

Adding Mocks in one of your tests

In the sample.sh script provided in the shellmock repo, the status code from grep is used to control the logic flow of the script. We can use Shellmock’s shellmock_expect command to simulate various success and failures scenarios depending on the arguments passed into the grep command.

sample.sh

echo "sample line"  > sample.out

grep "sample line" sample.out > /dev/null
if [ $? -ne 0 ]; then
    echo "sample not found"
    exit 1
fi

echo "sample found"

In the sections that follow we will create some test cases against sample.sh.

Success Scenario using exact matching

This testcase is simply using bats and calling the real grep command. No mocking was involved. This was included to show that testcases can be a mixture of mocks and real commands.

@test "sample.sh-success" {

    run ./sample.sh

    [ "$status" = "0" ]

    # Validate using lines array.
    [ "${lines[0]}" = "sample found" ]

    # Optionally since this is a single line you can use $output
    [ "$output" = "sample found" ]

}

Failure Scenario using exact matching

In this failure scenario we are creating a stub that will return a status of 1 if the grep is called in one of the two ways below:

grep "sample line" sample.out.

or

grep 'sample line' sample.out

NOTE: These will look the same in the stub's input args.

The testcase is using the default match type which is an exact match.

@test "sample.sh-failure" {


    shellmock_expect grep --status 1 --match '"sample line" sample.out'

    shellmock_debug "starting the test"

    run ./sample.sh

    # Only significant when debugging is occurring it captures bats variables to output files
    # to make it easier to see what you are missing.
    shellmock_dump

    [ "$status" = "1" ]
    [ "$output" = "sample not found" ]

    # called to create the capture array to allow expect verifications.
    shellmock_verify
    [ "${capture[0]}" = 'grep-stub "sample line" sample.out' ]

}

After the status and output of the script has been validated as needed, then the final piece is to verify that all of the expected mocks were called. The function shellmock_verify reads the shellmock.out file which contains a record of all mock invocations. The lines of the file are written to an array variable called capture.

Note

Arguments that contain quotes in them were a challenge. The scripting cannot tell the difference between single or double quotes. Therefore when single quotes are specified in the matching then shellmock converts them to double quotes. The capture output will contain double quotes even if the original script was called with single quotes.

The original version 1 did not make any distinction and this new feature was added in version 2. In v1 no quotes would appear in the verification output. It would appear like three arguments were passed instead of two.

Success Scenario leveraging a partial mock

In this test scenario we are only matching one of the arguments: "sample line". Any filename could be passed and still match the mock.

@test "sample.sh-success-partial-mock" {

    shellmock_expect grep --status 0 --type partial --match '"sample line"'

    run ./sample.sh

    shellmock_dump

    [ "$status" = "0" ]

    # Validate using lines array.
    [ "${lines[0]}" = "sample found" ]

    # Optionally since this is a single line you can use $output
    [ "$output" = "sample found" ]

    shellmock_verify
    [ "${#capture[@]}" = "1" ]
    [ "${capture[0]}" = 'grep-stub "sample line" sample.out' ]

}

Success Scenario demonstrating single vs double quotes

This testcase is the same as the one above except that single quotes where used around the argument.

@test "sample.sh-success-partial-mock-with-single-quotes" {

    shellmock_expect grep --status 0 --type partial --match "'sample line'"

    run ./sample.sh

    shellmock_dump

    [ "$status" = "0" ]

    # Validate using lines array.
    [ "${lines[0]}" = "sample found" ]

    # Optionally since this is a single line you can use $output
    [ "$output" = "sample found" ]

    shellmock_verify
    [ "${#capture[@]}" = "1" ]

    # Note that it is "sample line" in the capture output.
    [ "${capture[0]}" = 'grep-stub "sample line" sample.out' ]

}

Scenario with RegEx matching

This scenario was easier to show just using grep directly from the bats file. I created two mocks for grep, one with file names that start with 's' and one with file names starting with 'b'. The two mocks return 0 and 1 respectively.

@test "sample.sh-mock-with-regex" {

    shellmock_expect grep --status 0 --type regex --match '"sample line" s.*'
    shellmock_expect grep --status 1 --type regex --match '"sample line" b.*'

    # The first two patterns leverage the first mock.
    run grep "sample line" sample.out
    [ "$status" = "0" ]

    run grep "sample line" sample1.out
    [ "$status" = "0" ]

    # These two patterns leverage the second mock.
    run grep "sample line" bfile.out
    [ "$status" = "1" ]

    run grep "sample line" bats.out
    [ "$status" = "1" ]

    shellmock_dump

    shellmock_verify
    [ "${#capture[@]}" = "4" ]
    [ "${capture[0]}" = 'grep-stub "sample line" sample.out' ]
    [ "${capture[1]}" = 'grep-stub "sample line" sample1.out' ]
    [ "${capture[2]}" = 'grep-stub "sample line" bfile.out' ]
    [ "${capture[3]}" = 'grep-stub "sample line" bats.out' ]

}

To see a demonstration of the sample tests running, you will first need to install shellmock as described later and then follow the steps below.

cd sample-bats
bats sample.bats

You should expect to see output as follows:

 ✓ sample.sh-success
 ✓ sample.sh-failure
 ✓ sample.sh-success-partial-mock
 ✓ sample.sh-success-partial-mock-with-single-quotes
 ✓ sample.sh-mock-with-regex

5 tests, 0 failures

The test bats files are another good source for examples as it contains examples of all of the shellmock features.

Shellmock Functions

This section contains a list of the function provided by shellmock also with example usages.

skipIfNot

skipIfNot is a very useful function that would be a great addition to bats itself. There is currently a PR against bats for this ability. For now I have included this function in shellmock. This function will allow you to target particular tests while excluding others. To use it you must define an environment variable called TEST_FUNCTION.

TEST_FUNCTION may contain one or more test names delimited by a pipe. In the example below only tests "sample.sh failure" and "sample.sh success" would be executed. All others would be skipped.

$export TEST_FUNCTION="sample.sh-failure|sample.sh-success"

The next step is to instrument the tests with skipIfNot. skipIfNot requires one parameter which is the test name. The recommended approach is to add skipIfNot to the setup function and leverage the BATS_TEST_DESCRIPTION variable. Alternatively, you can instrument each function with skipIfNot and pass in any alias for the test name you like.

setup()
{
    # Source the shellmock functions into the shell.
    . ../bin/shellmock

    skipIfNot "$BATS_TEST_DESCRIPTION"

    shellmock_clean
}

@test "sample.sh-failure" {

.
.
.

}

shellmock_clean

shellmock_clean cleans up various temp files used by shellmock:

the tmpstubs directory - that is used to store stub data and scripts
shellmock.out - lists every stub call made
shellmock.err - lists errors encountered the stubs (ie not match found)

This command should be placed in the setup and teardown functions. To aid in troubleshooting, I typically recommend only calling it if TEST_FUNCTION is not set. This keeps stubs scripts and data from being deleted and allows you to investigate issues easier.

A useful practice is to place the cleanup in an if statement and ignore cleanup if the TEST_FUNCTION variable is set or some other debug variable. This allows you to have debugging access to the shellmock temp files for troubleshooting tests.

shellmock_debug

shellmock_debug provides a means to capture output statement that might help troubleshoot testing issues.

It can be used in the shellmock script or in your bats scripts if useful.

The output is captured in shellmock-debut.out and will only be available if TEST_FUNCTION is set.

shellmock_dump

shellmock_dump can prove quite useful to troubleshoot testing issues. It will dump the contains of the bats $lines variable which basically equates to any standard out that has been generated by the script under test.

The output is captured in shellmock-debug.out and will only be available if TEST_FUNCTION is set.

shellmock_verify

shellmock_verify converts all shellmock.out lines into a variable array called capture. This allows testers to verify which stubs were called and in what order.

@test "sample.sh-failure" {
.
.
.
    shellmock_verify
    [ "${capture[0]}" = "some-stub arg1 arg2" ]
    [ "${capture[1]}" = "some-stub2 arg1 arg2" ]
}

shellmock_expect

shellmock_expect allows you specify the command to be mocked and how the function should be mocked. The behavior can be in terms of status code, output to echo or a custom behavior that you provide.

usage: shellmock_expect [cmd] [--type partial | exact | regex ] [--status #] --match [arg1 arg2 arg3...] [--exec cmdstring ] [--source cmdstring] [--output texttoecho]

Item	Description	Required?
cmd	unix command to mock	Yes.
-t,--type	Type of argument list matching: partial, exact regex	No. Defaults to exact
-T,--stdin-match-type	Type of stdin matching: partial, exact, or regex	exact
-m,--match,--match-args	Arguments passed to cmd that indicate a match to mock.	No.
-M,--match-stdin	stdin data that is expected to be considered a match.	No.
-e,--exec	Command string to execute for custom behavior.	No.
-S,--source	Command string to source.	No.
-o,--output	Text string to echo if there is a match.	No.
-s,--status	status code to return	No. Defaults to 0 Matching can be defined based on the argument list or the stdin data stream. When both --match and --match-stdin are provided in an expectation then it becomes an AND of the two conditions. shellmock_expect supports returning a single or multiple responses for a given match criteria. The responses will be returned in the order defined. Once all response are seen the last response will be returned indefinitely. ==== examples These examples assume that the "grep string1 file1" is the unix command being mocked to be used in other scripts under test. For simplicity of understanding, I am calling the grep command directly from bats to show what the behavior would look like. ===== Basic mock with success status This example mocks grep to return a 0 status when the input is "string1 file1". In order to verify that the function was called you would need to use shellmock_verify and do a comparison. `bash shellmock_expect grep --match "string1 file2"` `run grep string1 file2 [ "$status" = "0" ] shellmock_verify [ "${capture[@]} = 1 ] [ "${capture[0]} = "grep-stub string1 file2" ]` ===== Basic mock with failed status This scenario show a status of 1 being returned for the same inputs. `bash shellmock_expect grep --status 1 --match "string1 file2"` `run grep string1 file2 [ "$status" = "1" ] shellmock_verify [ "${capture[@]} = 1 ] [ "${capture[0]} = "grep-stub string1 file2" ]` ===== Mock with partial mock If the grep command is run it will return a status 0 if arg1 is "string1" regardless of the rest of the args. Use shellmock_verify verify each invocation if desired. `bash shellmock_expect grep --status 0 --type partial --match string1` `run grep string1 file2 [ "$status" = "0" ] run grep string1 file3 [ "$status" = "0" ] shellmock_verify [ "${capture[@]} = 2 ] [ "${capture[0]} = "grep-stub string1 file2" ] [ "${capture[1]} = "grep-stub string1 file3" ]` ===== Mock with whitespace in the parameters If the grep command is run by the script under test it will return a status 0 if arg1 is "string1" regardless of the rest of the args. In order to verify that the function was called you would need to use shellmock_verify and do a comparison. If the --match argument were "'string1 string2' file", where the double quotes and single quotes are swapped, then shellmock treats the string as if it were '"string1 string2" file'. `bash shellmock_expect grep --status 0 --type partial --match '"string1 string2"'` `run grep "string1 string2" file2 [ "$status" = "0" ] run grep "string1 string2" file3 [ "$status" = "0" ] shellmock_verify [ "${capture[@]} = 2 ] [ "${capture[0]} = 'grep-stub "string1 string2" file2' ] [ "${capture[1]} = 'grep-stub "string1 string2" file3' ]` ===== Mock with regex This example shows the use of regex match type. The regular expression is evaluated by the AWK command. Refer to AWK documentation for details. Any AWK special characters will need to be escaped in the match criteria. `bash shellmock_expect grep --status 0 --type regex --match "s.* f."` `run grep string1 file2 [ "$status" = "0" ] run grep string1 file3 [ "$status" = "0" ] shellmock_verify [ "${capture[0]} = "grep-stub string1 file2" ] [ "${capture[1]} = "grep-stub string1 file3" ]` ===== Mock with custom script If the grep* command is run by a script under test it will return a status 0 if arg1 is "string1" and arg2 is "file1". It will also write "mycustom string1 file1" to stdout. The use of the {} in the --exec script will cause any arguments passed to the mocked script to be expanded in place of the braces as seen below. For this example you can verify the status, the output/line, and the capture variables. `bash shellmock_expect grep --status 0 --type partial --match "string1" --exec "echo mycustom {}"` `run grep string1 file1 shellmock_dump [ "$status" = "0" ] [ "${lines[0]}" = "mycustom string1 file1" ] run grep string1 file2 shellmock_dump [ "$status" = "0" ] [ "${lines[0]}" = "mycustom string1 file2" ] shellmock_verify [ "${#capture[@]}" = "2" ] [ "${capture[0]}" = 'grep-stub string1 file1' ] [ "${capture[1]}" = 'grep-stub string1 file2' ]` This example shows the use of echo as the script, however, it could also be any user defined script that you want in place of the mocked command. The {} braces are a way to forward arguments from the mock script into your script. == Installing Bash Shell Mock from source Check out a copy of the shellmock repository. Then, either add the shellmock `bin` directory to your `$PATH`, or run the provided `install.sh` command with the location to the prefix in which you want to install Shellmock. For example, to install Bats into `/usr/local`, $ git clone [repository_url] $ cd bash_shell_mock $ ./install.sh /usr/local Note that you may need to run `install.sh` with `sudo` if you do not have permission to write to the installation prefix. == Debugging Tests If the shellmock_clean function is short circuited then the temp files will remain. shellmock.out contains all of the mock commands that have been run and is used by the shellmock_verify command. If you following the sample and set TEST_FUNCTION then the tmpstubs directory will remain and not be cleaned up. Inside that directory you will find err out and debug files. For each file there will be two .tmp data files: - shellmock.out - shows which mocks were executed and their parameters - shellmock.err - shows the results of the matches - shellmock-debug.out - shows the results of what would have been sent to standard out array $lines which bats also allows you to match on. - .playback.capture.tmp - shows defines each of the expectations. There will be on of these files for every mocked script. - .playback.state.tmp - keeps track of multiple responses for the same mock == Limitations The Shellmock mocking approach does have impact on how write your scripts. The key to using any mocking in unix scripts is that the scripts must be reached via the PATH variable and you can not use full or relative pathing to the script. Shellmock uses the PATH variable to short circuit calling the "real" script or program. == Contributors We welcome Your interest in Capital One’s Open Source Projects (the “Project”). Any Contributor to the Project must accept and sign an Agreement indicating agreement to the license terms below. Except for the license granted in this Agreement to Capital One and to recipients of software distributed by Capital One, You reserve all right, title, and interest in and to Your Contributions; this Agreement does not impact Your rights to use Your own Contributions for any other purpose. Sign the Individual Agreement Sign the Corporate Agreement == Code of Conduct This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

swipswaps / bash_shell_mock Goto Github PK

bash_shell_mock's Introduction