Display additional information or test origin for failed tests

[pahaeanx]

Display additional information or test origin for failed tests

[sshipway]

This is similar to (but better than!) an idea I had in the past, which was to add a 'description' attribute to tests that could be printed on test failure to give a bit more information to the user on which test failed. Being able to define it at test or file level, defaulting to the file name, would be handy when your tests are split over multiple gossfiles, possibly managed by multiple other sources.

[aelsabbahy]

correct me if I'm wrong but do title and meta cover the "hint"/"description" use case?

example file defining title and meta:

$ cat goss.yaml 
command:
  some_test:
    title: This is some arbitrary title
    meta:
      any-key: any-value
      version: 17
      severity: very severe
      contact: call bob
      hint: node-exporter listening port
    exec: |
      echo "hi"
    stdout:
    - bye

example output:

$ goss v
FF

Failures/Skipped:

Title: This is some arbitrary title
Meta:
    any-key: any-value
    contact: call bob
    hint: node-exporter listening port
    severity: very severe
    version: 17
some_test: exit-status: Error: Missing Required Attribute
Command: some_test: stdout: patterns not found: [bye]

Total Duration: 0.004s
Count: 2, Failed: 2, Skipped: 0

Just realized they're documented in the readme, but not the manual https://github.com/goss-org/goss#manually-editing-goss-files

Basically, title can be set to any string and meta can be set to any key/value (both are stored strings).

[pahaeanx]

Oh wow, I somehow completely missed that.

After playing around with these a bit, the only criticism I have is that it seems that neither can be set at the file level?

# goss.yaml
title: testing file level titles
command:
 fail_w/o_title:
   exec: /bin/false
   exit-status: 0
 fail_w_title:
   title: This test has a title set
   exec: /bin/false
   exit-status: 0

Output:

$ goss v
FF

Failures/Skipped:

Command: fail_w/o_title: exit-status:
Expected
    <int>: 1
to equal
    <int>: 0

Title: This test has a title set
Command: fail_w_title: exit-status:
Expected
    <int>: 1
to equal
    <int>: 0

Total Duration: 0.002s
Count: 2, Failed: 2, Skipped: 0

I guess it'd be nice to be able to set title and meta attributes on a file level that act as default values for tests that don't have these set.

The current limitation can be worked around by setting the same title for each test (I'd simply set the file name) but that seems like a lot of unnecessary work and code.

Apart from that, that's exactly what I was looking for. Thanks!

[aelsabbahy]

@sshipway just want to confirm, does the current functionality of title/meta meet your suggestion above?

@pahaeanx I like your suggestion, but it might be a little (long) while before I get to it to be honest.

Mostly because, I'm thinking as a future enhancement Goss should have the ability to group tests into a test suite.. that suite can then have title/meta.. but it can also have other things like setup/teardown.

Perhaps the file will be the group.. or maybe there would be some other top level suite concept introduced that allows multiple suites to be set for a given file.

Once I get through the final Goss v4 release, I'll open up some issues mapping out the future roadmap. Hopefully, I can stimulate some OSS contributions to the projects for some of them.