parroty/excheck

Property-based testing library for Elixir (QuickCheck style).

excheck's Issues

Can't compile triq

After adding

{:excheck, "~> 0.3", only: :test},
{:triq, github: "krestenkrab/triq", only: :test}

to my mix.exs, I get the following.

⇒  mix test      
==> triq (compile)
Compiling src/triq_dom.erl failed:
src/triq_dom.erl:290: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:291: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:370: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:409: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:411: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:448: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:463: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:465: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:469: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:470: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:483: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:484: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:495: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:517: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:519: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:554: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:573: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:588: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:608: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:622: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:634: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:645: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:655: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:660: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:684: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:686: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:691: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:734: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:736: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:785: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:794: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:806: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:811: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:812: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:835: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:916: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:963: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:986: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1002: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1063: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1083: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1135: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1148: random:seed/0: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1149: random:seed/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1153: random:seed/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1155: random:seed/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1304: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1324: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1384: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
src/triq_dom.erl:1386: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
ERROR: compile failed while processing /Users/jimmyhmiller/Documents/Code/trabian/cu1-membership/api/deps/triq: rebar_abort
** (Mix) Could not compile dependency :triq, "/Users/jimmyhmiller/.mix/rebar compile skip_deps=true deps_dir="/Users/jimmyhmiller/Documents/Code/trabian/cu1-membership/api/_build/test/lib"" command failed. You can recompile this dependency with "mix deps.compile triq", update it with "mix deps.update triq" or clean it with "mix deps.clean triq"

If you need anymore information, let me know.

"Such that" is failing

So I have the following code:

for_all {s, t, j} in such_that({s_, t_, j_} in {int, int, int} when(s_ < j_ and j_ < t_)) do
    #...
end

Because I need j to be a value between s and t. Seems correct. However, here's the output I'm getting:

I don't think this is the expected behaviour. (apologies in advance if this is my mistake and/or this is not the place to submit this kind of report)

Consider adding syntax as https://github.com/yrashk/properex

@yrashk has a very nice syntax, not quite unlike yours. I don't know if his stuff still compiles/works with current syntax, but I like it quite a bit more:

    property simple_forall do
        forall x in integer, do: is_integer(x)
    end

His test/example code is here: https://github.com/yrashk/properex/blob/master/test/proper_test.exs

Could this library use normal `for` comprehension syntax for generators instead of something custom?

The current generator syntax--e.g. for_all x in int--requires that I learn a new syntax, but it sure looks similar to Elixir's existing for comprehension syntax. Is there any reason that can't be used instead?

For example, looking over the examples in the readme:

property :square do
  for_all x in int, do: x * x >= 0
end

Could be:

property :squre do
  for x <- int, do: x * x >= 0
end

For a more involved example involving a such_that like:

for_all {x, y} in such_that({xx, yy} in {int, int} when xx < yy) do
  x < y
end

Could perhaps be:

for x <- int, y <- int, x < y do
  x < y
end

I just tried ExCheck for the first time today so I'm sure there are aspects of it's API I'm not understanding, but it would greatly reduce the learning curve if my existing knowledge of Elixir's for comprehensions could be used instead of learning a new API.

Can you include the license file in the repo?

On https://hex.pm/packages/excheck, it says your code is licensed under the MIT license, but in your repo, I don't see a license file or a description of the licensing terms. Could you please add a license file?

Thanks! ❤️

Failed test output difficult to read when ExUnit running with `async: true`

Hello!

It would be nice if we could run tests asynchronously while keeping the debug information from failed tests readable.

Could we use a custom ExUnit formatter for this?

Cheers,
Louis

Question: idiomatic approach for managing custom generators?

Hi,

I'm creating a number of properties that have the form:

property :something do
for_all {a, b} in {[number,number], [number,number]} do
#...
end
end

Is there an idiomatic way of referencing the generator expression [number,number] for easy reuse across test modules? Maybe the test_helper.exs file? Then I can use:

property :something do
for_all {a, b} in {number_list_2, number_list_2} do
#...
end
end

Thanks

Counter examples missing from failing tests

Hi!

When running tests that fail, ExCheck fails to print the failing counter example (which would be the expected behaviour, I assume).

For example, changing the sample to this:

  def prop_push_list do
    for_all({x, y}, {int(), list(int())}) do
      result = push(x, y)
      Enum.at(result, 0) == x and Enum.count(result) == Enum.count(y)
    end
  end

which is expected to fail, the output is the following:

ExCheck.SampleTest
  * test verify sample property (24.8ms)...........................................................................................
..........
  1) test verify sample property (ExCheck.SampleTest)
     test/sample_test.exs:5
     Expected truthy, got false
     code: ExCheck.check(ExCheck.Sample)
     stacktrace:
       test/sample_test.exs:6: (test)

  * test verify sample property with iteration parameter (20.5ms)

  2) test verify sample property with iteration parameter (ExCheck.SampleTest)
     test/sample_test.exs:9
     Expected truthy, got false
     code: ExCheck.check(ExCheck.Sample, 10)
     stacktrace:
       test/sample_test.exs:10: (test)

which doesn't provide me with much help for actually tracking down why a test is failing. Is this expected behaviour?

I'm running Elixir 1.4.0.

Formatter fails with no configured tests

Just a minor issue. When running mix test with no tests configured. The formatter has the error message:

[error] GenServer #PID<0.546.0> terminating
** (KeyError) key :test not found in: %{}
    (excheck) lib/excheck/formatter.ex:66: ExCheck.Formatter.update_tests_counter/1
    (excheck) lib/excheck/formatter.ex:28: ExCheck.Formatter.handle_cast/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:683: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:suite_finished, 36754, nil}}
State: %{colors: [enabled: true], failure_counter: 0, invalid_counter: 0, seed: 627868, skipped_counter: 0, test_counter: %{}, trace: false, width: 80}

This occured when I was smoke testing the minimum configuration before adding tests. Not fatal but it did give me a tiny spook if I misconfigured anything with this library.

Using ExCheck with Espec

So I see that ExCheck is pretty coupled to ExUnit. Is there any way to easily adapt it to work easily with Espec? I'm guessing I would have to make my own implementation of ExUnit.__using__? And my own Espec.Statement module? Anything else I'm missing?

number generator?

Thanks for developing ExCheck, looks very useful!

I might be missing something but could we have a number/1 generator function that generates either a random float or integer?

Currently I'm using oneof([real, int]) and wondering whether this belongs in ExCheck or in a helper function?

Stu

ExCheck usage in an umbrella app causes end-of-run spec counts to be reported incorrectly.

We have an umbrella app and we use ExCheck in the test suite of one of the apps. When we run mix test in our project root (which recursively runs mix test in each app), the end-of-run test counts reported for each app are inflated with an extra 100 tests. For example, we get output like this:

# ...
==> s3
Excluding tags: [:manual, :smoke]


Delorean.S3Test

Delorean.S3AcceptanceTest
  * test both clients calculate `presigned_get_url` in the same way (and the URL works) (195.1ms)
  * test `Delorean.S3.Test.FakeS3` can upload objects to S3 and later list and fetch them (13.2ms)
  * test `Delorean.S3` can upload objects to S3 and later list and fetch them (330.6ms)


Finished in 0.7 seconds
103 tests, 0 failures

Randomized with seed 54640
# ...

3 tests but it is reported as 103. I originally reported this as an elixir issue but eventually realized ExCheck is causing this.

Triq dependency moved to gitlab

https://github.com/triqng/triq appears as archived. So the new repo to add as a dependency would be: https://gitlab.com/triq/triq

Samples not being skipped when using "implies"

I created a property test for a simple sum of max 2 function.

I am using elixir and mix version 1.4.1 installed with homebrew running on macOS Sierra.

When I run my tests I see samples that should not have been skipped were skipped and samples that should have been skipped but seem to have not been.

For example, per my implies expression on line 26 of sum-max2_test.exs, 5 should not have been skipped and 0 should have been but wasn't.

➜   mix test
..1
x0
x1
x0
x5
.11
.4
.6
.0
x1
x...

Finished in 0.08 seconds
8 tests, 0 failures

This happens to me consistently.

Am I misinterpreting the test output, what the functionality of implies is, or could this possibly be an issue with the implementation of implies?

Docs on Hex

Hey!

Could you publish your docs on hexdocs? It would make it a lot easier to glance through the library's functionality.

Thanks!

Compiler warning

I keep getting this warning how to get this resolved?

All I have is a simple module with this:

  use ExUnit.Case, async: false
  use ExCheck

warning: this clause cannot match because a previous clause at line 1 always matches

Env:

Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.6.4) - press Ctrl+C to exit (type h() ENTER for help)

Unclear how to ensure x is not in ys

I have a property that takes a string and a list of strings, and part of this property is that the string is not in the list of strings. I can't seem to find a predicate for this, is there some way of accomplishing this in a single property?

Basically what I'm after is something like:

for_all {x, ys} in such_that({xx, yys} in {binary(), list(binary())} when not_in(xx, yys)) do
  ...
end

Compile Error: module :triq_dom is not loaded and could not be found

Hello! All I've done is add this library as a dep to my mix.exs and added use ExCheck, and I'm getting compile errors.

Is there more than needs to be done? If so, can we add this to the README?

louis ~/projects/osc [master *+] $ MIX_ENV=test mix deps.compile excheck
==> excheck
Compiled lib/excheck/generator.ex
Compiled lib/excheck/predicate.ex
Compiled lib/excheck/error.ex
Compiled lib/excheck.ex
Compiled lib/excheck/statement.ex

== Compilation error on file lib/excheck/sample.ex ==
** (CompileError) lib/excheck/sample.ex:5: module :triq_dom is not loaded and could not be found
    expanding macro: ExCheck.Generator.__using__/1
    lib/excheck/sample.ex:5: ExCheck.Sample (module)
    (elixir) expanding macro: Kernel.use/1
    lib/excheck/sample.ex:5: ExCheck.Sample (module)
    expanding macro: ExCheck.__using__/1
    lib/excheck/sample.ex:5: ExCheck.Sample (module)
    (elixir) expanding macro: Kernel.use/1
    lib/excheck/sample.ex:5: ExCheck.Sample (module)

could not compile dependency excheck, mix compile failed. You can recompile this dependency with `mix deps.compile excheck` or update it with `mix deps.update excheck`

Question: best approach for property raising an exception?

Hi,

I'd like to create a property that should raise an exception if successful, e.g. abs("foo"). Is the when_fail predicate the idiomatic way to handle this situation?

Thanks,

Stu

Generator for floating point value between two values

I think the int/2 generator will generate an integer value between two integers? Could we have a matching real/2 generator that generates a float value between two floats?

Thanks!

`property` with ex_unit context binding

Hey, it'd be useful if an excheck property could accept the ExUnit.Case context.

For example:

property :foo, %{bar: baz} do
  ...
end

I hate to admit it, but I tried and failed to pull this off by following ex_unit's example. S'pose I should read that metaprogramming elixir book.