Proposal: Test framework

[phel-lang]

Proposal: Test framework

[jenshaase]

This is now completed! All test of the core library can now be executed by ./vendor/bin/phel test

[jenshaase]

I updated the Report section. What do you think?

[jenshaase]

Since (deftest) is the same as (defn) which is the same as (def) why can't we add (deftest) macro in the core and thus avoid adding :refer to the (ns)?

I see the :refer not only useful for this case. It can also be quite useful for library developers to extend Phel with some nice APIs. For example an if anyone whats to define a routing library, he can define a defroute macro like this:

(defroute "/my/route/:id" [request]
  "hello world")

Could we add a command for vendor/bin/phel test namespace/file/test-name?

Sure!

[paullaffitte]

I really like the idea of using ., F or E and to print the detailed output at the end. I think that most test libraries do like this and it produce a good and readable output.

I think it would be pretty useful to print the expected value on test failure. It would help developers to understand what is going on without having to manually print the test's parameters. Quasi-quotation could help here.

Here is an example of what I'm thinking about:

(defmacro assertt
  [test label]
  `(assert ,test (str ,label ", Expected: " ,(print-str test)))
)

(assertt (= 1 2) "Test case")

And the output of this would be e Test case, Expected: (= 1 2).

Then I have a question about run-tests, we would have to pass as parameters all tests that we want to execute right? Maybe auto-discovery of the tests cases (just like phpunit does) would be nice. Then you don't have to worry about forgetting to add one of your tests to the list. And then (again, just like phpunit), we could add a flag or something similar to run only tests that match a pattern, are in some namespace, or even eventually in a test suite (this would require to add something like deftestsuite btw).

[jenshaase]

@paullaffitte Thanks for your input. I will add more details in the report section, on how failure and error message should look like (with examples).

Regarding run-tests. This function is the base for running all tests. It's a good idea to add auto-discovery. That can be build on top of run-tests. I thinks this can be done with the vendor/bin/phel test command. If no namespace is provided all test should run.

[mabasic]

Great work! Really loving how the language is turning out. Sorry I can't contribute more to the code. I have my hands full atm... but testing and experimenting is what I love to do in the morning, so I will give it a go and report back.

[mabasic]

Looks very useful and simple, but I have some questions/suggestions.

Since (deftest) is the same as (defn) which is the same as (def) why can't we add (deftest) macro in the core and thus avoid adding :refer to the (ns)?

As an alternative, why not just use (defn) and tell people to add metadata @{:test true} for writing tests?

Could we add a command for vendor/bin/phel test namespace/file/test-name?

[mabasic]

I see. Makes sense now. Thank you for explaining the use case a little bit more.

[jenshaase]

Current Status:

• I implemented the framework as described above
• I updated the old tests to the new test framework
• I added the test framework documentation on the website

What is left:

• Autodiscovery of all tests
• ./vendor/bin/phel test command