goss add command to set default assertions for empty stdout/stderr

[gberche-orange]

goss add command to set default assertions for empty stdout/stderr

[aelsabbahy]

Thank you for filing this. There seem to be two parts:

1. A bug
2. A feature request

bug

The bug has to do with "/^$/", you're right that is odd behavior. I know why it's happening, and it's not intuitive for the end user. The workaround suggested in the linked issue will solve it, but it isn't evident:

command:
  # the sleep command produces no stdout or stderr
  sleep_test:
    exec: |
      >&2 echo "hi"
    exit-status: 0
    stderr: [ "!/./" ]
    timeout: 10000

See my comment here for more info: #145 (comment)

Essentially, the check is if any line is empty.. but there are no lines, so it fails to find an empty line. You can test this behavior with echo "" vs echo -n "", the former will pass [ "/^$/" ] the latter won't.

My memory is a bit hazy, but I feel like I explored making a special case for empty output, but it caused a breaking change somewhere.. but I could be wrong. I'll re-evaluate this with fresh eyes and see if it makes sense to special case empty string.

feature request

For the feature request, the upcoming goss v4 #814 provides a way to handle this easier:

command:
  sleep_test:
    exec: |
      >&2 echo hi
    exit-status: 0
    stderr: ""

$ goss v
.F

Failures/Skipped:

Command: sleep_test: stderr:
Expected
    "hi\n"
to equal
    ""

Total Duration: 0.001s
Count: 2, Failed: 1, Skipped: 0

It will also handle newlines correctly, for example:

$ cat goss.yaml 
command:
  sleep_test:
    exec: |
      >&2 echo ""
    exit-status: 0
    stderr: ""
$ goss v
.F

Failures/Skipped:

Command: sleep_test: stderr:
Expected
    "\n"
to equal
    ""

Total Duration: 0.002s
Count: 2, Failed: 1, Skipped: 0

The only way the above will pass is with echo -n ""

I'll have to reflect a bit about defaults when doing a goss add. It probably won't be in the initial v4 release.. but might be added in a future release.

feedback

Would love your thoughts on the above..

[gberche-orange]

Thanks a lot @aelsabbahy for your prompt and detailed answer !

Bug
The workaround suggested in the linked issue will solve it

Thanks for the workaround which properly detects empty strings, with the following known limitations

• the current stdout/stderr isn't displayed

command:
  "echo my_stdout > /dev/stdout; echo my_stderr > /dev/stderr":
    exit-status: 0
    stdout: [ "!/./" ]
    stderr: [ "!/./" ]
    timeout: 10000

$ goss validate
.FF

Failures/Skipped:

Command: echo stdout > /dev/stdout; echo stderr > /dev/stderr: stdout: patterns not found: [!/./]
Command: echo stdout > /dev/stdout; echo stderr > /dev/stderr: stderr: patterns not found: [!/./]

Total Duration: 0.002s
Count: 3, Failed: 2, Skipped: 0

feature request
For the feature request, the upcoming goss v4 #814 provides a way to handle this easier

This looks great to me, fully satisfying my needs !

I'll have to reflect a bit about defaults when doing a goss add. It probably won't be in the initial v4 release.. but might be added in a future release.

With the v4 behavior for "" matcher, the current default behavior (i.e. adding "" matchers in goss add command when observing empty strings) seems a good sensible default to me.

I'm looking forward to using goss v4 !

[aelsabbahy]

Forgot to mention, I released a release candidate v0.4.0-rc1 a few days (weeks?) ago.

Would love your feedback on it to make sure it correctly covers this need.

[aelsabbahy]

v0.4.2 is released with empty strings when there's no output.

Feel free to re-open an issue if this is still a problem. Thank you for reporting!