commander-cli/commander

Test your command line interfaces on windows, linux and osx and nodes viá ssh and docker

commander's Issues

Refactoring Matchers

it is a mess :(

Add retry to test cases

distribute tests via ssh

It would be nice if I could define a test which will be executed on a different host via ssh.

Better filters

--filter="should execute command"
--filter="command*" # With wild card pattern

--filter-node="ssh-host"
--filter-node="ssh*"

Maybe with regex support?

Better diffs

It would be nice to print better diffs between the expected and actual result

Add timeout for commands / tests

It would be nice to define a timeout for each test / executed command

Add sleep time to retries

I want to add a sleep or interval if a test fails with retries.

I.e.

tests:
    echo hello:
        config:
            retries: 4
            interval: 10s

Add unit parsing for timeouts

It would be nice if parsing timeouts could add untis like this 20ms.

See time.ParseDuration

Add asciinema video

Add macOS support

Running `commander add --file tests/commander.yaml my-command` removes test validation from commander.yaml

Prerequisites

• Can you reproduce the problem?
• Are you running the latest version?
• Did you perform a cursory search?

Description

When using the automatic adding of tests, manually added tests validation is removed.

Steps to Reproduce

1. run commander add --file tests/commander.yaml ./setup-update-tag.sh
2. check what has changed in the file (e.g. diff --git a/tests/commander.yaml b/tests/commander.yaml)

Expected behavior: commander add should just add new tests, not modify existing ones (resorting the file may be fine)

Actual behavior: It restructures the file and drops test cases. Basically removing most manually made edits.

diff --git a/tests/commander.yaml b/tests/commander.yaml
index 3156fb7..15b5d6c 100755
--- a/tests/commander.yaml
+++ b/tests/commander.yaml
@@ -1,33 +1,18 @@
 tests:
-  setup with existing .env:
-    command: ./setup.sh
+  ./setup-update-tag.sh:
     exit-code: 1
-    stdout: |-
-      Creating individual env files for containers (if they do not exist already)
-      .env already exists with initial configuration
-      If you want to change the configuration, please edit .env directly
+    stdout: Please install reg in order to run this script.
   ./version.sh:
     exit-code: 0
   ./version.sh core:
     exit-code: 0
-    stdout:
-      not-contains:
-        - core-
-        - Debian
-        - amd64
-        - tar.gz
   ./version.sh webapp:
     exit-code: 0
-    stdout:
-      not-contains:
-        - webapp-
-        - Debian
-        - amd64
-        - tar.gz
   ./version.sh zpush:
     exit-code: 0
-    stdout:
-      not-contains:
-        - Debian
-        - amd64
-        - tar.gz
+  setup with existing .env:
+    exit-code: 1
+    stdout: |-
+      Creating individual env files for containers (if they do not exist already)
+      .env already exists with initial configuration
+      If you want to change the configuration, please edit .env directly

Specifications

• Version: 1.12.0
• Platform: Ubuntu 16.04
• Subsystem:

You can get the version information from executing commander --version.

Expose healthcheck

It would be nice to have a command which exposes commander as a healthcheck

Add os for tests

Add a field which defines on which os a test can be executed, i.e.:

tests:
    echo hello:
        os:
        - darwin
        - linux

or maybe this is even better:

config:
    os: linux # Define default os
tests:
     print hello:
         cmd:
             osx: echo hello osx
             windows: echo hello windows
             linux: echo hello linux

Add tap output

I would like to add a tap like output

Print logs if test failed

Add option to disable colored output of the command under test

At the moment it is needed to remove colored output from the shell with something like this:

echo command | sed -r "s/\x1B[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g"

execute tests against a directory

it would be nice if tests could be executed against a complete directory of files

Help Needed: Understanding PATH and the commander runner shell

Prerequisites

• Can you reproduce the problem?
• Are you running the latest version?
• Did you perform a cursory search?

Related:
#80
#94

Description

Our CLI app makes calls to the shell for other apps, namely docker. During the install step for our cli, it uses docker to pull and configure some containers. If I run install from the shell, it works. If I run install from commander test it fails saying that docker is not found. I had expected config: env: PATH_FROM_SHELL: ${PATH} to fix this, but there is something more going on that I don't understand.

How does each runner use a shell?
Is is an untestable scenario in commander to test a cli application that uses an OS process to call another cli application?

Steps to Reproduce

1. Install binary from this build
2. Make sure docker is installed and running.
3. Run this test:

tests:
  titan install:
    exit-code: 0
    stdout:
      contains:
        - Titan cli successfully installed, happy data versioning :)
config:
  env:
    PATH_FROM_SHELL: ${PATH}
  timeout: 600s

Expected behavior:
install should finish and exit process with 0

Actual behavior:
Test fails with Docker not found

Specifications

• Version: Commander version v1.2.1
• Platform: MacOS

Release deployment

• #6
• install.sh script which can be fetched via curl
• travis deployment for github releases on tag

Update Documentation

• Fix errors in documentation
  • README.md:54
• Add assertion documentation

Create docker containers automatically

• alpine
• ubuntu
• centos

Inerheit env variables from parent shell

Add colored output on windows

Define dependencies

It would be nice to define dependencies on tests. This is necessary especially if I want to test kubernetes resouces.

It should also be possible to activate an interactive mode for debugging purposes.

Example:

it should deploy:
  command: ...

it should be available:
  command: ...
  depends: it should deploy

Add not-contains assertion to stdout and stderr

more documentation

i was a little bit lazy :(

Evaluate how to test interactive commands

It would be nice to test interactive commands, but not necessary at the moment

Need help getting windows server 2019 working

Prerequisites

• Can you reproduce the problem?
• Are you running the latest version?
• Did you perform a cursory search?

Description

Following the instructions in the README to setup a testing environment with github actions and running commander does not have any output. I've tried a few variations of invoking commander, but all with the same results. The file will download, I can see it with ls, but running /commander --version has no output. I have tried adding it to the path (can confirm it's in the path), but running commander --version has the same results.

Steps to Reproduce

Add a github action with the following config:
https://github.com/mcred/titan/blob/feature/e2e/.github/workflows/test.yml

Inspect output:
https://github.com/mcred/titan/commit/ee26a202ea5218314315bbb387705e97161abc17/checks?check_suite_id=268546125

Expected behavior:
Commander version v1.2.1

Actual behavior: [What actually happened]
No output, failure or exceptions.

Specifications

• Version: v1.2.1 commander-windows-amd64
• Platform: Windows Server 2019

Add option to execute tests sequentially

It would be nice to execute tests sequentially.
Maybe it would be nice to define an order.

Start line counting at 1 instead 0

More output assertions

• json
• yaml
• against a file

i.e.

stdout:
  json:
    metadata.name: wordpress

TestRuntime_WithEnvVariables is very instable on windows

Add documentation

• Windows installation
• Different matchers for Stdout and Stderr
• Timeout / Env / Dir variables
• Add more examples

Commander -v missing version number

Prerequisites

• Can you reproduce the problem?
• Are you running the latest version?
• Did you perform a cursory search?

Description

The version option output does not contain version numbers.

Steps to Reproduce

1. Install using go command according to https://github.com/SimonBaeumer/commander#installation :

    go get github.com/SimonBaeumer/commander/cmd/commander
   

2. On terminal, run:

    commander -v
   

Expected behavior:

Output:

Commander version 0.3.0

Actual behavior:

Output:

Commander version

Specifications

• Version: Commander 0.3.0 (As found in commit 3fa13b0, 2019-10)
• Platform: Debian GNU/Linux 9.11 (stretch)
• Subsystem: Perhaps go version? go1.7.4 linux/amd6

You can get the version information from executing commander --version.

No. Whether commander -v or commander --version, version number is still missing.

Improve command arguments and env variables

• Execute a single test
• Validate a specifc file
• Deactivate colors in output
• Concurrent executions
• Logging

Add setup and tear down

It should be possible to define setup and tear_down commands

Add pull request template

Option to not discard environment of current shell?

Prerequisites

• Can you reproduce the problem in safe mode?
  • I could not find any indication how to enable safe mode in this repository (only mentioned in the issue template)
• Are you running the latest version?

$ commander -v
Commander version v1.1.0

• Are you reporting to the correct repository?
• Did you perform a cursory search?

Description

When running programs that are configurable through environment variables (for example inside of a container) running them through commander fails, since previously set variables are not present.

Steps to Reproduce

1. run

$ commander add --stdout --no-file env
tests:
  env:
    exit-code: 0
    stdout: PWD=/home/user/current-dir

2. run env directly in the shell
3. notice that none of the env vars (except for PWD) are listed by commander

Expected behavior: commander has access to the environment of the shell it is executed in.

Actual behavior: no environment is preserved.

Specifications

• Version: v1.1.0
• Platform: Ubuntu 16.04 as host, Debian 9 in container
• Subsystem:

You can get the version information from executing commander --version.

Create a shell for the current test

It would be cool to execute a test in the shell commander would use

Fetch test suites from remotes and stdin

It would be nice to fetch test suites from remote locations like gist.github, http, https etc
Further addin stdin support

Use maps for command environment

• Use maps for command environment.
• Eventually skip the config key in the tests

Use cobra as cli framework

It would be nice to use cobra as the cli framework in v2.0.0

Improve err output for stdout and stderr

At the moment it will only be displayed which root attribute failed the test, i.e.:

Starting test file test.yml...

✗ echo hello

Results

✗ echo hello, on property 'Stdout'
--- Got
+++ Expected
@@ -1 +1 @@
-hello
+fail


Duration: 0.003s
Count: 1, Failed: 1

It is not possible to see in the results which attribute of Stdout failed.

Add logging for tests

It would be nice to get a log of the output for executed tests

Add YAML, HTML and file assertions

• HTML assertion
  • Array support
• YAML support
  • Array support
• File assertion

Add commands reorders test suite

If a new test is added by the add command the suite will be reordered alphabetically.

Use cmd package

https://github.com/SimonBaeumer/cmd

implement add command

It should be possible to create tests by giving the command and automatically create the expectations

$ ./commander add echo hello world
echo hello world:
    stdout:
        exactly: hello-world
    exit-code: 0

Refactoring Validation

It is a mess :(

Add windows support