jorgebucaran / fishtape Goto Github PK

View Code? Open in Web Editor NEW

342.0 342.0 20.0 139 KB

100% pure-Fish test runner

License: MIT License

Shell 100.00%

fish fish-plugin tap test

fishtape's People

Contributors

Stargazers

Watchers

Forkers

rodoviario ricucremop abaez connectthefuture tarun1209 1bionix1 edouard-lopez ykankaya danhper matchai faho ryotako kairyu fabioantunes longervisionusa artnoc1 diffalot erdal-pb t-nil

fishtape's Issues

Only try to execute all fish files found as tests

Hi @jorgebucaran, I love how you've supported me as I try to make the most of fishtape and I now have one more request if you don't mind.

I'd like to store my tests as such:

tests/
  _resources/ # contains files used by my tests, none of them are .fish files
  test_suite_1
    test1.fish
    test2.fish
  test_suite_2
    test3.fish
    test4.fish

Right now if I execute fishtape tests/*/*, I get fishtape: Invalid file or file not found: _resources/....

Would it be a good idea to just try to execute all the .fish files found rather than failing if a non .fish file is found?

Verbose

Is there a way to have a verbose mode where one can see what's happening? Something similar to bash -x cmd.

I'm asking because:

when writing tests, I'm not sure if it's stuck or just taking some time ;
when test fail I lack insight of what is wrong

For instance:

test "install with fisher"
    'Done+WRONG_VALUE+' = (
        docker run \
            --name pure \
            --rm \
            --tty \
        edouardlopez/pure-fish 'fisher rafaelrinaldi/pure' \
        | grep --only-matching 'Done'
    )
end

Will throw:


$ fishtape tests/ci.test.fish | tap-min
type: Could not find “”

install with fisher
        operator: =
        expected: 'Done2'
        actual:   undefined

3 tests complete (9ms)

Disclaimer: maybe I'm not having the right approach

feature request: before_each and after_each hooks

For pure test suite, I'm isolating each test from the other as much as possible to avoid side-effect. To do so, I regularly create before_each and after_each functions in test files. For instance:

removing global/universal variables using _purge_configs
disabling coloring
recreating fresh git repo for test

Feature Request

Would it be possible to add a before/after hooks mechanism to before/after running the test?

Before running a @test, execute the before_each function defined by the user.
After a @test function was executed, execute the function after_each function defined by the user

Fishtape is broken with Fish 2.6

Getting this when trying to use fishtape with a legitimate path:

$ fishtape test/*test.fish                                                                                                                                            master
- (line 95): Missing end to balance this function definition
function __fishtape@total;	if test $__fishtape_count -eq 0;        fishtape_cleanup;        exit 1;    end;;    printf "\n1..%s\n" $__fishtape_count;    printf "# tests %s\n" $__fishtape_count;    printf "# pass  %s\n" (math $__fishtape_count - $__fishtape_fails);;    if test $__fishtape_fails -gt 0;        printf "# fail  %s\n" $__fishtape_fails;        fishtape_cleanup;        exit 1;    end;;    fishtape_cleanup;    printf "\n# ok\n"
^
from sourcing file -
	called on standard input

source: Error while reading file '-'

This happens on both Ubuntu 16.04 and macOS 10.12 with Fish 2.6.

The functions function changed with Fish 2.6 (fish-shell/fish-shell@2e38cf2) to add a comment to the top of the body like :

$ functions fish_clipboard_copy                                                                                                                                       master
# Defined in /usr/local/Cellar/fish/2.6.0/share/fish/functions/fish_clipboard_copy.fish @ line 1
function fish_clipboard_copy
	if type -q pbcopy
        commandline | pbcopy
    else if type -q xsel
        commandline | xsel --clipboard
    end
end

This comment is breaking https://github.com/fisherman/fishtape/blob/master/fishtape.fish#L62 because sed is only removing the first and last line of the functions output, leaving the function definition.

Post sed the output now looks like:

$ functions fish_clipboard_copy | sed '1d;$d;s/\\\/\\\\\\\/g'                                                                                                         master
function fish_clipboard_copy
	if type -q pbcopy
        commandline | pbcopy
    else if type -q xsel
        commandline | xsel --clipboard
    end

... causing the error I first mentioned.

I've temporarily fixed this locally by removing the top two lines (sed '1,2d;$d;s/\\\/\\\\\\\/g'), but this isn't backwards compatible with anything < 2.6 and probably isn't the best solution.

Make fishtape file compatible with fish_indent

related: #32, fish-shell/fish-shell#5399

From discussion with @jorgebucaran and @faho it appears using fishtape file aren't ~~valid~~compatible fish file. Hence the problem met while trying to format them with fish_indent (cf. #32).

Rationale

Adopting a ~~valid~~compatible fish syntax will let us leverage all the tooling related to fish and help developers adopt testing approaches. So we get better integration and better software quality.

Fishtape example

test "current directory is home"
    $HOME = $DIRNAME
end

test "math still works"
    42 -eq (math 41 + 1)
end

test "test is a builtin"
    "test" = (builtin -n)
end

test "no odds are evens"
    1 3 5 7 != (
        for i in (seq $n)
            if test (math $i%2) = 0
                echo $i
            end
        end
        )
end

Roadblocks

Can we list the roadblocks that prevent fishtape file from being ~~valid~~compatible fish file ?

test already has a specific meaning in fish, and it does not start blocks. test is a keyword in fish 3.0

Is it possible to do multiple assertions?

⚠️ This doesn't work:

test "format seconds to human"
    set --local seconds 1000  # express as milliseconds
    '1s' = ( _pure_format_time (math "1*$seconds") $THRESHOLD)
    '59s' = ( _pure_format_time (math "59*$seconds") $THRESHOLD)
end

If so, how can I do it?

Migrating from V1 to V2

Just upgraded to V2? You'll need to make some adjustments. In this issue, I'll explain what these changes are and how to migrate your tests. 🙌

Background

Fishtape V1 test syntax wasn't valid Fish syntax. Running fishtape test.fish transformed test blocks to valid to test builtin declarations under the hood.

That means you couldn't use fish_indent to format your files. Blocks didn't let you create multiple assertions either, defeating the purpose of a "block" in the first place. Even if you never used fish_indent, why write pseudo-Fish code when you can write real Fish?

What's new?

V2 introduces a new @test function. It wraps the test builtin, so you can specify an optional description for your test. Another small difference is no support for -o or -a operators.

Here are some examples:

@test "current directory is home" $PWD = $HOME

@test "hello has length 5" (string length "hello") = 5

@test "math still works" 42 -eq (math 41 + 1)

@test "test is a builtin" (contains -- test (builtin -n)) $status -eq 0

Also new is the @mesg function. It creates a TAP comment like # my message that will be written together with the TAP output.

Other changes include the removal of all special variables, leaving only $current_filename. If we need more variables, we can always add them later when the need arises.

Doesn't print report properly on Alpine Linux

Similar to jorgebucaran/fisher#565, Fishtape doesn't work properly with BusyBox awk.

Actual testing seems to work just fine, however, reporting is not correct.

On macOS:

$ fishtape test/*.test.fish
TAP version 13
ok 1 print path for a standalone script
ok 2 print path for a Fisher package
ok 3 print path for a manually installed function

1..3
# pass 3
# ok

But on Alpine Linux

$ fishtape test/*.test.fish
TAP version 13
0 print path for a standalone script
1 print path for a Fisher package
2 print path for a manually installed function

1..
# pass 0
# ok

Any idea what might be a problem judging by the output?

isatty is always 1 in tests

When you run fishtape this is essentially what happens:

preprocess $files | fish -c source | produce_tap

Piping to source means isatty will exit with status 1 every time. Here's an example:

function foo
    if isatty
        echo atty
    else
        echo notatty
    end
end

@test -- (foo) = atty
@test -- (echo | foo) = notatty

Sharing functions not working when running single test file

I'm looking for a way to run code before all tests in a test file. Currently I'm following suggestion from #10 and created a _setup.fish:

function setup
    docker pull edouardlopez/pure-fish
end

setup

Files

My tests/ directory contains:

❯ ls -1 tests/
ci.test.fish
__pure_set_default.test.fish
_setup.fish

Running tests using a glob

If I run my tests with a glob the _setup.fish is run but not at the beginning. It seems it runs alphabetically.

$ fishtape tests/*
type: Could not find “”
TAP version 13
ok 1 set my_var default value
ok 2 skip setting value if default already exists
Using default tag: latest
latest: Pulling from edouardlopez/pure-fish
Digest: sha256:255942bf7c89a2423ccc38a3bb6f3ea96456fef5aee96f799e35ba18f7e2ac38
Status: Image is up to date for edouardlopez/pure-fish:latest
ok 3 install manually (default behaviour of `docker-pure-fish`)
ok 4 install with fisher
ok 5 install with OMF (Oh-My-Fish!)
ok 6 install with Fundle

Running specific test case

If I run a test case explicitly, then the _setup.fish is not run at all (no mention of docker pulling an image):

❯ fishtape tests/ci.test.fish
type: Could not find “”
TAP version 13
ok 1 install manually (default behaviour of `docker-pure-fish`)
ok 2 install with fisher
ok 3 install with OMF (Oh-My-Fish!)
ok 4 install with Fundle

1..4
# tests 4
# pass  4

# ok

Expectations

_setup.fish runs even when calling a specific test case.
_setup.fish should always run first.

fish_indent reformatting broke tests

related: swsnr/vscode-fish-ide#6

I'm using fish_indent through vscode-fish-ide which reformat test file as follows:

 test "set my_var default value"
-  'default_value' = (
+'default_value' = (
     __pure_set_default my_var 'default_value'
     echo $my_var
-  )
+    )
 end

After formatting my tests fails:

❯ fishtape tests/__pure_set_default.test.fish
TAP version 13
not ok 1 set my_var default valuedefault_value
  ---
    operator: fail
    expected: '= or !='
    received: '= default_value '
  ...
not ok 2 skip setting value if default already existsdefault_value
  ---
    operator: fail
    expected: '= or !='
    received: '= default_value '
  ...

Exclude test file

On pure I have unit tests and some integration testing for the installation method. Often I would like to exclude them from the test run.

Currently, I do:

fishtape tests/*.test.fish
# make build-pure-on test-pure-on FISH_VERSION=3.1.2 CMD="fishtape tests/*.test.fish"

But I would like to be able to do:

fishtape tests/*.test.fish --exclude=pure_tools_installer.test.fish

Is that a possible feature for the next rewrite?

Detect errors

Sometimes a test may pass but errors are printed to the terminal. This would be a "failure" from a human perspective. Thus, it would be nice if Fishtape could detect if errors are printed, and fail accordingly.

Feature Request: oh-my-fish support

It would be nice if you can also make it an oh-my-fish plugin.

Conditional test not running

In pure we have some test cases conditioned by the fish version running. However, while migrating to 2.1.0 they stop working.

For instance, I got no output for the following

if fish_version_below '3.0.0'
    @test "_pure_prompt_ssh: displays 'user@hostname' when on SSH connection" (
       …
    ) = 0
end

However, if I add an echo whatever in the if block then I got some output

ok 2 _pure_prompt_ssh: displays 'user@hostname' when on SSH connection

if fish_version_below '3.0.0'
    echo whatever
    @test "_pure_prompt_ssh: displays 'user@hostname' when on SSH connection" (
       …
    ) = 0
end

Migrating from V2 to V3

Just upgraded to V3? You'll need to make some changes. In this issue, I'll explain what they are and how to migrate your tests.

Use (status dirname) instead of $current_dirname (needs Fish >=3.2).
Use (status filename) instead of $current_filename.
The test builtin ! operator is no longer supported.
- Not to be confused with Fish ! operator.
@mesg is now @echo. Just s/@mesg/@echo/.
setup and teardown are no longer necessary (see #53).
- The best way to do work before and after a test is simply right there in your test file.

Newlines in test arguments are no longer collapsed. Use echo or string collect:

- @test "first six evens" (seq 2 2 12) = "2 4 6 8 10 12"
+ @test "first six evens" (echo (seq 2 2 12)) = "2 4 6 8 10 12"

-@test "one two three" (seq 3) = "1
+@test "one two three" (seq 3 | string collect) = "1
2
3"

Measure code coverage

Fishtape is a great testing framework, but something that is missing is a description of how much code your tests actually cover.

Using fish --debug-output and fish_trace it's easy to dump the code that was run into a file and then count the lines.

Then, using a tool like cloc one can get the number of fish code lines in the user's project to calculate the percent code coverage.

Steps?

I'm not sure how fishtape's subshells work, but presumably they could add fish --debug-output and fish_trace
A lot of code is repeated when running a test file, so duplicate code sections must be removed
Fishtape has it's own code that would need to be silenced/removed from the output
Somehow get only the number of code lines from cloc. Probably have to go through yaml or something, I don't think there's a way of just printing one number.

This is somewhat outside the scope of the project, so if there's no interest in implementing this that's totally fine 😄 I know it would be useful to myself and likely others though.

When a test fails fishtape return status 0 instead of 1

I have a file with this test:

@test "math is real" (math 41 + 1) -eq 43

Which produces this output:

TAP version 13
not ok 1 math is real
  ---
    operator: -eq
    expected: 43
    actual:   42
  ...

1..1
# pass 0
# fail 1

But the status code is always 0

Some info
Fish version: 3.0.2
OS: macOS Mojave
terminal: iTerm2

Missing $DIRNAME in fishtape 2.0.0

related: 2.0.0, #38

Thanks for the new release, I haven't made the switch yet, but look forward to it.

In the release note for 2.0.0 you mention removing $DIRNAME variable. That annoying, cause I'm using it a on every test files for pure to import function under test, like:

source $DIRNAME/../functions/_pure_prompt_first_line.fish

Sure it's possible to use $filename however such a name is prone to conflict and makes code a less readable:

source (dirname $filename)/../functions/_pure_prompt_first_line.fish

BATS as a series of special variables prefixed by $BATS_TEST_ (e.g. $BATS_TEST_DIRNAME).

I reckon namespacing is a must have, and would love to see dirname come back ❤️

Some namespace ideas:

$fishtape_test_dirname
$ft_test_dirname
$_test_dirname

No way to specify fish arguments

For example, there is no way to let fish run tests in interactive mode (fish --interactive).

Multiple assertions per test

Sorry to open an issue but #33 is locked, I can't comment there.

I've been playing with PowerShell and it's test framework Pester:

The convention is to assert a single expectation for each It block.

Adding support for multiple assertions might not be worth the time and complexity it will add. Sure it can be annoying when you repeat context between tests but does it happens that often?

How to test ssh-add?

I'm having trouble testing a ssh-add command for the herrbischoff/fish-ssh-agent project. So far I've tested almost everything but I feel there is some fish-related catch I'm not aware of.

Source

What I'm testing

…
ssh-add $private >/tmp/ssh-add.log ^ /tmp/ssh-add.err
…

Error: what I got

$ fishtape test/*.fish | tap-min
Could not open a connection to your authentication agent.
Could not open a connection to your authentication agent.

    10 tests complete (9ms)
    null Add given keys
    ---
    operator: =
    expected: '0'
    received: '1'
    ...

Question

Am I in some special context where ssh-agent is unavailable ? Or is it something else?

fishtape parsing depends on whitespace in a manner incompatible with `fish_indent`

~ ⋊> cat test_fishtape.fish
test "tautology"
    1 = 1
end

~ ⋊> fishtape test_fishtape.fish
TAP version 13
ok 1 tautology

1..1
# tests 1
# pass  1

# ok

~ ⋊> fish_indent -w test_fishtape.fish

~ ⋊> cat test_fishtape.fish
test "tautology"
1 = 1
end

~ ⋊> fishtape test_fishtape.fish
TAP version 13
not ok 1 tautology1
  ---
    operator: fail
    expected: '= or !='
    received: '= 1 '
  ...

1..1
# tests 1
# pass  0
# fail  1

~ master * ⧕ fish -v
fish, version 2.6.0

Thanks for fishtape!

Option to stylize output

Hi, I understand one of the point of TAP is that you can combine arbitrary producers with consumers/reporters. However, there doesn't seem to be an up-to-date and low-dependency TAP reporter at https://github.com/sindresorhus/awesome-tap#reporters. So, I'm stuck with the default unstyled output of fishtape currently.

Would it be possible to add a flag to fishtape such as --color to simple use red for errors, green for successes? And the flag can be off by default so it won't break integrations with reporters.

Missing end to balance this function definition

When running [email protected] against pure test suite I got the following error:

❯ fishtape tests/debug.test.fish
- (line 3): Missing end to balance this function definition
function __fishtape@total;      if test $__fishtape_count -eq 0;        fishtape_cleanup;        exit 1;    end;;    printf "\n1..%s\n" $__fishtape_count;    printf "# tests %s\n" $__fishtape_count;    printf "# pass  %s\n" (math $__fishtape_count - $__fishtape_fails);;    if test $__fishtape_fails -gt 0;        printf "# fail  %s\n" $__fishtape_fails;        fishtape_cleanup;        exit 1;    end;;    fishtape_cleanup;    printf "\n# ok\n"
^                                                                                                                                                                                                                                      from sourcing file -
called on standard input
source: Error while reading file “-”

This happen whatever test file I try to run.

Files not included in tab completions

Fishtape should show files from the current directory when tab completions are triggered.

$ fishtape<tab>
--help --version tests/ somefile

Support conditional test

Pure support several version of fish (2.5 to 3.0.0). However some of our tests are only meaningful on some version of the platform.

To manage this we wrap those tests in a if…end block, e.g. _pure_prompt_ssh_host.test.fish:

if fish_version_at_least '3.0.0'
test "_pure_prompt_ssh_host: use native \$hostname"
    (
        set pure_color_ssh_host (set_color grey)

        _pure_prompt_ssh_host > /dev/null
    ) $status -eq 0
end
end

This work as expected but look messy, a decorator would be a lot more elegant, e.g.:

# fish_version_at_least 3.0.0
test "_pure_prompt_ssh_host: use native \$hostname"
    …
end

How to debug?

I'm trying to debug the following function to find out why my assertion fail ()

function pure_symlinks_assets
    printf "\tLink pure's configuration and functions to fish config directory"
    for pure_function in $PURE_INSTALL_DIR/functions/*.fish
        echo $pure_function
        ln -sf $pure_function $FISH_CONFIG_DIR/functions/
    end
end

Tested with:

if test $USER = 'nemo'
    @test "installer: link configuration and functions to fish config directory" (
        pure_symlinks_assets #>/dev/null

        set --local active_prompt $FISH_CONFIG_DIR/functions/fish_prompt.fish
        set --local pure_config $FISH_CONFIG_DIR/conf.d/pure.fish
        test \
            -r "$active_prompt" -a -L "$active_prompt" \
            -a -r "$pure_config" -a -L "$pure_config"   # config is a readable symlink
         # ls -lah $pure_config
         ls -lah $FISH_CONFIG_DIR/functions
    ) $status -eq $succeed
end

Output

Currently, I haven't access to the output of the echo from pure_symlinks_assets but only the value of the command failing the test (here ls):

not ok 13 installer: link configuration and functions to fish config directory
  ---
    operator: -eq
    expected: Link pure's configuration and functions to fish config directorytotal 36K drwxr-s--- 1 nemo nemo 4.0K Nov 26 08:43 . drwxr-s--- 1 nemo nemo 4.0K Nov 26 08:43 .. -rw-r--r-- 1 nemo nemo    0 Nov 26 08:43 fish_prompt.fish.ignore -rw-r--r-- 1 nemo nemo  16K Oct 31 11:45 fisher.fish -rw-r--r-- 1 nemo nemo 7.7K Oct 31 11:45 fishtape.fish 0
    actual:
  ...

Adding breakpoint in the test or code does nothing. Could you provide some advice on how to approach this ?

test vs tests for directory name

Hi Jorge, sorry if this is a bit too nit-picky, but I was wondering on your thoughts on whether plugins that use fishtape should put their tests in a dir called test or tests? I was thinking tests since all the other traditional fish files are plural (functions, completions, fish_plugins, fish_variables). However, in this plugin you called it test. Now I'm not sure what the consensus is. What do you think?

Unsupported -a in fishtape 2.0.0

Great work with the release of fishtape 2.0.0!

Unfortunately, the new test system doesn't allow for multiple assertions in one test, so the test setup would have to be duplicated for every test case.

E.g:

@test "It should not mock blacklisted elements" (
	mock eval \*
) = "The function \"eval\" is reserved and therefore cannot be mocked."

@test "It should not mock blacklisted elements" (
	mock eval \*
) $status -eq 1

instead of:

@test "It should not mock blacklisted elements" (
	mock eval \*
) = "The function \"eval\" is reserved and therefore cannot be mocked." -a $status -eq 1

In this example, the test setup is short, but a fair bit of duplication would need to be added tests where setup is more complex, without a way to have multiple assertions.

Fishtape 2

Fishtape 2 will break the test syntax as we want to make it compatible with fish_indent #32, #34.

Summary

Make test syntax compatible with fish_indent
Run test files in parallel (not assertions, but files)
Remove --pipe in favor of any userland solution like "fish -c "fishtape *.fish | tap-nyan"
Remove the built-in test array extensions (don't know what I am talking about? don't worry then)

Here's the new test syntax I have in mind. See that test blocks are now gone. The test builtin has been "enhanced" to take an optional test description. If you already know how to use the shell's test builtin you already know how to create valid assertions in Fishtape 2.

@test "current directory is home" $PWD = $HOME

@test "math works" 42 -eq (math 41 + 1)

@test "test is a builtin" (contains -- test (builtin -n)) $status -eq 0

for odd in 1 3 5 7 9
    @test "$odd is not an even number" (
        contains -- $odd (seq 10 | awk '!($0%2)')
    ) $status -eq 0
end

@test "git_is_dirty shows changes not staged for commit" (
    pushd /path/to/some/dirty/git/repo
    git_is_dirty
    echo $status
    popd
) = 1

Notice that there is no reason you couldn't write the last test from above as follows (if you prefer):

pushd /path/to/some/dirty/git/repo
git_is_dirty
@test "git_is_dirty shows changes not staged for commit" $status = 1
popd

Issue History

Use @test instead of test #31 (comment)

actual and expected

Hey Jorge, how does Fishtape's @test function figure out which argument is the expected one and which argument is the actual result? Or is it based on the order of the arguments? If it's based on order, then which one goes first? I tried reading the code but I'm not 100% sure because it's not documented anywhere.

Fishtape 3

It's that time of the year again. The plan is to use Fish 3.0, make Fishtape faster, smaller, revise the API, and whatnot.

The biggest breaking change will be going back to running tests serially. This makes your tests behave more predictably and might even make most tests out there faster since we'd no longer block (per file), need that extra subshell, or have to handle background jobs.

Serial processing also means we don't have to preprocess test files anymore, so setup and teardown are no longer necessary. The best way to do work before and after a test is simply right there in your test file.

I also think it's a good time to remove all the magic left in Fishtape, e.g.: automatically collapsing newlines in test arguments.

Consider this (surprisingly) passing test:

@test (seq 3) = "1 2 3"

This is magic. The right way to write this is:

@test (echo (seq 3)) = "1 2 3"

# or

@test (seq 3 | string join " ") = "1 2 3"

Magic makes it more difficult to reason about your code because it violates the principle of WYSIWYG. If you want to turn (seq 3) into "1 2 3" you really need use to echo or pipe it into string join " ", which is a bit more code than before, but at least it's correct, so when things break it will be more likely your code and not Fishtape.