ferd/rebar3_proper

Run PropEr test suites with rebar3

rebar3_proper's Issues

Fix cover-compile bug since 3.2.0

See erlang/rebar3#1272

undefined parse transform 'vararg' on OTP 17.5

Hello, I have issues using this plugin on OTP 17.5, see error below. On OTP 18+ it works fine. What's weird is it does look like proper works on OTP 17 here: https://github.com/manopapad/proper/blob/master/.travis.yml#L13. Sorry if this is not the proper place to report it.

here's minimal repo to reproduce it, just setting up the plugin on a blank new rebar3 lib project: https://github.com/wojtekmach/proper_bug

---

~/proper_bug[master]% rebar3 report compile
===> Compiling proper
make: `include/compile_flags.hrl' is up to date.
===> Compiling _build/default/plugins/proper/src/proper_gen.erl failed
_build/default/plugins/proper/src/proper_gen.erl:none: undefined parse transform 'vararg'

===> Plugin rebar3_proper not available. It will not be used.
Rebar3 report
 version 3.5.0
 generated at 2018-03-04T20:33:48+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: compile
Entered as:
  compile
-----------------
Operating System: x86_64-apple-darwin16.7.0
ERTS: Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]
Root Directory: /Users/wojtek/.asdf/installs/erlang/17.5
Library directory: /Users/wojtek/.asdf/installs/erlang/17.5/lib
-----------------
Loaded Applications:
bbmustache: 1.3.0
certifi: 2.0.0
cf: 0.2.2
common_test: 1.10
compiler: 5.0.4
crypto: 3.5
cth_readable: 1.3.2
dialyzer: 2.7.4
edoc: 0.7.16
erlware_commons: 1.0.4
eunit: 2.2.9
eunit_formatters: 0.5.0
getopt: 1.0.1
inets: 5.10.6
kernel: 3.2
providers: 1.7.0
public_key: 0.23
relx: 3.24.3
sasl: 2.4.1
snmp: 5.1.1
ssl_verify_fun: 1.1.3
stdlib: 2.4
syntax_tools: 1.6.18
tools: 2.7.2

-----------------
Escript path: /Users/wojtek/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report shell state tar tree unlock update upgrade upgrade upgrade version xref

compile error on vonnegut

Tried debugging some, but quickly got out of my depth. Here's the stack trace and rebar3 report:

===> Compiling vonnegut for PropEr
===> Compiling files in /Users/evan/src/vonnegut2/test to /Users/evan/src/vonnegut2/_build/test/lib/vonnegut/test
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: function_clause
===> Stack trace to the error location:
[{filename,join,[[]],[{file,"filename.erl"},{line,399}]},
 {rebar_dir,'-src_dirs/3-lc$^0/1-0-',1,
            [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_dir.erl"},
             {line,264}]},
 {rebar_dir,src_dirs,3,
            [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_dir.erl"},
             {line,261}]},
 {rebar_erlc_compiler,compile,4,
                      [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_erlc_compiler.erl"},
                       {line,151}]},
 {rebar3_proper_prv,'-compile_dirs/3-lc$^0/1-0-',3,
                    [{file,"/Users/evan/src/vonnegut2/_build/default/plugins/rebar3_proper/src/rebar3_proper_prv.erl"},
                     {line,245}]},
 {rebar3_proper_prv,compile_dirs,3,
                    [{file,"/Users/evan/src/vonnegut2/_build/default/plugins/rebar3_proper/src/rebar3_proper_prv.erl"},
                     {line,241}]},
 {rebar3_proper_prv,find_properties,4,
                    [{file,"/Users/evan/src/vonnegut2/_build/default/plugins/rebar3_proper/src/rebar3_proper_prv.erl"},
                     {line,210}]},
 {rebar3_proper_prv,find_properties,2,
                    [{file,"/Users/evan/src/vonnegut2/_build/default/plugins/rebar3_proper/src/rebar3_proper_prv.erl"},
                     {line,186}]}]

$ rebar3 report proper
===> Compiling rebar3_proper
Rebar3 report
 version 3.4.4
 generated at 2017-11-10T18:06:20+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: proper
Entered as:
  proper
-----------------
Operating System: x86_64-apple-darwin15.6.0
ERTS: Erlang/OTP 20 [erts-9.1.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:0] [hipe] [kernel-poll:false] [dtrace]
Root Directory: /usr/local/Cellar/erlang/20.1.2/lib/erlang
Library directory: /usr/local/Cellar/erlang/20.1.2/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.3.0
certifi: 2.0.0
cf: 0.2.2
common_test: 1.15.2
compiler: 7.1.3
crypto: 4.1
cth_readable: 1.3.0
dialyzer: 3.2.2
edoc: 0.9.1
erlware_commons: 1.0.0
eunit: 2.3.4
eunit_formatters: 0.3.1
getopt: 0.8.2
inets: 6.4.2
kernel: 5.4
providers: 1.6.0
public_key: 1.5
relx: 3.23.1
sasl: 3.1
snmp: 5.2.7
ssl_verify_fun: 1.1.2
stdlib: 3.4.2
syntax_tools: 2.1.3
tools: 2.11

-----------------
Escript path: undefined
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs proper release relup report run send shell state tar tree unlock update upgrade upgrade upgrade version xref ```

modules configuration doesn't work as expected

From IRC:

04:12 <Vorpal> In rebar3 I'm having issues with the PropEr plugin... {proper_opts, [{module, ["mod_a"]}]}. works, but not {proper_opts,
               [{module, ["mod_a", "mod_b"]}]}.
04:12 <Vorpal> ===> Module "mod_amod_b" does not exist or exports no properties
04:13 <Vorpal> I tried various different ways to specify the module. It doesn't like atoms either, it outright crashes then.
04:13 <Vorpal> (And yes, just "mod_b" works as well)
04:16 <Vorpal> Oh, found it... {module, ["mod_a,mod_b"]}. Works without the [] as well. So the documentation "{module, [Modules]}" is
               incorrect
04:16 <Vorpal> Should probably report a bug to that plugin

It appears the file configuration works with the same logic as the CLI, but this is not necessary at all in this case.

listing multiple modules appears to fail to find some of them

debugging regression failures

when working with a large regression suite, if several failures happen early or late in the run it would be nice to be able to use --retry to rerun a single one of the failures to prevent having to wait for the entire regression suite to either pass/fail when debugging.

Deprecated calling warning

Hi, @ferd, when I execute rebar3 proper in my project, it warns me that

===> Calling deprecated rebar_erlc_compiler compiler module. This module has been replaced by rebar_compiler and rebar_compiler_erl, but will remain available.

What about replacing the calling rebar_erlc_compiler with another two modules in rebar3_proper code?

Support `counterexample` mode

When the long_result mode is enabled, it should be possible to store the failed results in a file (say in _build/$PROFILE/rebar3_proper.counterexamples).

When running either (I have yet to decide) rebar3 proper --counterexample or rebar3 proper --retry, the run that previously failed can be automatically retried.

Plugin create .rebar3 folder in test dir

After each run rebar3 proper -v

$ cd ./apps/myapp/test/.rebar3 ; ls -la
-rw-rw-r-- 1 hiend hiend 444 may 30 16:04 erlcinfo

is this the right behavior?

{error, bad_name} when invoking code:priv_dir()

rebar 3.3.2 on Erlang/OTP 18 Erts 7.2.1

When I invoke rebar3 as test proper, I am experiencing the following error:

$ rebar3 as test proper
===> Verifying dependencies...
===> Compiling hermes_config
===> Compiling hermes_stat
===> Compiling hermes_common
===> Compiling hermes_backend_redis
===> Compiling hermes_web
===> Compiling hermes
===> Testing prop_hermes_backend_redis_channel:prop_filter_scores_size_drop()
.x.x.x....x....x.x..xx.x.....x..xx.....x.x..x....x.x.x...x.x.xx..xxx.x....x..x..x..xx.xx.x..x...x.xx.xxx.x..x..x..xx....xx..xx.xxx.xxxx.x.xxx..x..xxxxxx.xxxx..xxx.xx..x.xxxxx.x.x.xx..x....x.x.xx.x..xxxx.xx....xxx...x..x..x.x.x.xx...xxx.xxxx.xxx.xxx.x..x..x..xxx...xx...xxx..x...x.x.x..xxx.xx.xx.xx.x..x....x....x..x.x.xx.xx.xxxx....x.x...xxx...xx......x...xx...x.x.xx..x.xxx..xxx.x.xxxx.xx.xxx.xxxx..x.x.x..x.xxxx.xx..x..xx.x.....xxxx.x.xxx.xxx..x...xxxx...xxxx.xx.....x..xx..x..x.x.xxxx..x..xx.x.x..x..x..xx.....x.xx.....x.xx.x.x..xx.xxx..xx.xx..xx.xx..xxxx...x..xxx....x.xxxxx.x....x...x.xx..xx...xxxx.x.x.x..x.xxx....xx..x.xx..xx...x.x.xx.xx.x....xxx....xx.xx.xx.x.....x..x..xx.xx.x...xx.x.xxxx.x..xx...x.......x.x.x.x....xxx...xxx.x.x..x..x...xxxx..xx.xxx..x..xxxxxx...x...xx.xx..x....x..x.x.xx...x.xx...x....xx...x..xx..x.x..x.x.xxxxxx..x.x..x....xx..xx.x....x.xxx.x.xx...x.xxxxxxx...xx.xxxxxxxxx.....xx...x..x..xxxxxxx.x.....xx...xx...xx.xxx...x.xxxxx.x...xx.......x.xx.x.xx...x..x..xx..x....xxxxxxxxx...xxxx..xx.xx.x...x...x..xxx..x.x.x..xx.xx..x....x..xx.xxx.xxx..xx...xx.xxxx.xx.x.xx..xxx.x..x.xx.xx.xxx...xxxx.x.x...xx....xxxx..x.xx.x.x.xx.x...xxx.x..xxxx.x.x.xxx.xxx.xx...x...x..xxx.x.xxxxxx......x.xx.xxxxx.x..x.xx.....xxx.x.x......xx....x.x.x.....xx...xxxxx.x.xxxx.x.xxx..x.xx.xxxx..xxxxx.x....x....xxx.x....xxxxx.xxxx.x.xxx...xx...x..x.xxx...x.x...xxxx.x.x...xxxxx.xxx.x.....xx..xx.xx......x.x..xx.x..xx.x.x.x..x.x...x.xx.x.x..x.xx.x.x....xx....xxxxx..x........xxxx..xxx.x.xx.xx..x.x...xx.x....xxxx..xx.x...xxx..xxx.x..x....x..xx..x.x.x...x.x..x..xxxxxxxx.xx.xxx.x.x..x.xx.....xx.xx.x...xxx....x...x..x.x..x..xx...x..xxx.x...x.xx...x.xx.x....x.xxxxxx.xxx..x.x..x.x.xx.xx..xxxxxx.x..xx.....x...x...xx.xxx.xx.......x.xx.x..xxxxxx..x..xx.xxxx.x.xxxxx..x.xx.xxxx.x..xxxxxx.x..xxx..x..x..x.xx..x..x.x.xxx.x.x.x.x...x.x.x....xx.xxxx.....xxxx...x....xxx.....xx.x...xx..xx...x.xx.xxxxx..xxx..x.....xxxxxx...........x.x..xxxx..x...xxx...x.x..xxx.x.x.xxxxxx.x.xx....
OK: Passed 1000 test(s).
===> Testing prop_hermes_backend_redis_channel:prop_rank_for_topic_strictly_monotonic()
!
Failed: After 1 test(s).
An exception was raised: error:{case_clause,{error,{hermes_backend_redis,{bad_return,{{hermes_backend_redis_app,start,[normal,[]]},{'EXIT',{bad_name,[{hermes_backend_redis_app,priv_dir,0,[{file,[47,85,115,101,114,115,47,106,111,104,110,47,103,105,116,47,104,101,114,109,101,115,47,95,98,117,105,108,100,47,116,101,115,116,47,108,105,98,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,47,115,114,99,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,95,97,112,112,46,101,114,108]},{line,16}]},{hermes_backend_redis_app,read_lua,1,[{file,[47,85,115,101,114,115,47,106,111,104,110,47,103,105,116,47,104,101,114,109,101,115,47,95,98,117,105,108,100,47,116,101,115,116,47,108,105,98,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,47,115,114,99,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,95,97,112,112,46,101,114,108]},{line,23}]},{hermes_backend_redis_app,'-upload_lua_scripts/0-lc$^0/1-0-',1,[{file,[47,85,115,101,114,115,47,106,111,104,110,47,103,105,116,47,104,101,114,109,101,115,47,95,98,117,105,108,100,47,116,101,115,116,47,108,105,98,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,47,115,114,99,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,95,97,112,112,46,101,114,108]},{line,35}]},{hermes_backend_redis_app,upload_lua_scripts,0,[{file,[47,85,115,101,114,115,47,106,111,104,110,47,103,105,116,47,104,101,114,109,101,115,47,95,98,117,105,108,100,47,116,101,115,116,47,108,105,98,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,47,115,114,99,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,95,97,112,112,46,101,114,108]},{line,35}]},{hermes_backend_redis_app,start,2,[{file,[47,85,115,101,114,115,47,106,111,104,110,47,103,105,116,47,104,101,114,109,101,115,47,95,98,117,105,108,100,47,116,101,115,116,47,108,105,98,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,47,115,114,99,47,104,101,114,109,101,115,95,98,97,99,107,101,110,100,95,114,101,100,105,115,95,97,112,112,46,101,114,108]},{line,45}]},{application_master,start_it_old,4,[{file,[97,112,112,108,105,99,97,116,105,111,110,95,109,97,115,116,101,114,46,101,114,108]},{line,273}]}]}}}}}}}.
Stacktrace: [{prop_hermes_backend_redis_channel,setup,0,
                 [{file,
                      "/Users/john/git/hermes/apps/hermes_backend_redis/test/prop_hermes_backend_redis_channel.erl"},
                  {line,199}]},
             {prop_hermes_backend_redis_channel,
                 '-prop_rank_for_topic_strictly_monotonic/0-fun-2-',1,
                 [{file,
                      "/Users/john/git/hermes/apps/hermes_backend_redis/test/prop_hermes_backend_redis_channel.erl"},
                  {line,50}]}].
[]

Shrinking (0 time(s))
[]

The priv dir for my application hermes_backend_redis is failing to be found despite it being present:

 ls _build/test/lib/hermes_backend_redis/priv
post_message.lua	read_messages.lua

Furthermore, if I enter the shell as test I can retrieve priv_dir as expected:

1> code:priv_dir(hermes_backend_redis).
"/Users/john/git/hermes/_build/test/lib/hermes_backend_redis/priv"
2> code:get_path().
["/Users/john/git/hermes/_build/test/lib/hermes/ebin",
 "/Users/john/git/hermes/_build/test/lib/hermes_web/ebin",
 "/Users/john/git/hermes/_build/test/lib/hermes_backend_redis/test",
 "/Users/john/git/hermes/_build/test/lib/hermes_backend_redis/ebin",
 "/Users/john/git/hermes/_build/test/lib/hermes_common/ebin",
 "/Users/john/git/hermes/_build/test/lib/hermes_stat/ebin",
 "/Users/john/git/hermes/_build/test/lib/hermes_config/ebin",
 "/Users/john/git/hermes/_build/test/lib/gun/ebin",
 "/Users/john/git/hermes/_build/test/lib/hackney/ebin",
 "/Users/john/git/hermes/_build/test/lib/proper/ebin",
 "/Users/john/git/hermes/_build/default/lib/elli/ebin",
 "/Users/john/git/hermes/_build/default/lib/elli_websocket/ebin",
 "/Users/john/git/hermes/_build/default/lib/eredis/ebin",
 "/Users/john/git/hermes/_build/default/lib/jiffy/ebin",
 "/Users/john/git/hermes/_build/default/lib/lager/ebin",
 "/Users/john/git/hermes/_build/default/lib/poolboy/ebin",
 "/Users/john/git/hermes/_build/default/lib/recon/ebin",
 "/Users/john/git/hermes/_build/default/lib/statsderl/ebin",
 "/Users/john/git/hermes/_build/default/lib/uuid/ebin",
 "/Users/john/git/hermes/_build/default/lib/vmstats/ebin",
 "/Users/john/git/hermes/_build/test/lib/certifi/ebin",
 "/Users/john/git/hermes/_build/test/lib/cowlib/ebin",
 "/Users/john/git/hermes/_build/test/lib/idna/ebin",
 "/Users/john/git/hermes/_build/test/lib/metrics/ebin",
 "/Users/john/git/hermes/_build/test/lib/mimerl/ebin",
 "/Users/john/git/hermes/_build/test/lib/ranch/ebin",
 "/Users/john/git/hermes/_build/test/lib/ssl_verify_fun/ebin",
 "/Users/john/git/hermes/_build/default/lib/goldrush/ebin",
 [...]|...]
```

Hard dependency on proper 1.2.0 makes it hard to use another version

rebar3_proper does not even use proper for itself and I can't access it without specifying it as a top level dependency.

Also proper 1.2.0 does create deprecation warnings on a current OTP. Therefore I want to use proper from master where the warnings are fixed.

Still proper 1.2.0 is pulled and compiled, even for prod, because we can't have plugins in profiles.

Last but not least running proper does fail randomly when not using 1.2.0.

Application's include directory is missing from include path when building property tests

When I run rebar3 proper, the compilation of the test/prop_*.erl files fails as they reference include files in my application's include/ directory. Since eunit successfully compiles files with the same include references, this seems like a bug.

Sorry I haven't had time to submit a patch with this.

Hopefully minimal reproduction:

% rebar3 new app rebar3_proper_include_issue && cd rebar3_proper_include_issue
% cat > rebar.config <<REBAR_CONFIG
{deps, [{proper, "1.1.1-beta"}]}.
{plugins, [rebar3_proper]}.
REBAR_CONFIG
% mkdir test && cat > test/prop_test.erl <<PROP_TEST
-module(prop_test).
-include_lib("proper/include/proper.hrl").
-compile([export_all]).
-include("foo.hrl").
PROP_TEST
% mkdir include && touch include/foo.hrl

Note that rebar3 eunit compiles this empty test, while rebar3 proper complains:

===> Verifying dependencies...
===> Compiling rebar3_proper_include_issue
===> Compiling test/prop_test.erl failed
test/prop_test.erl:4: can't find include file "foo.hrl"

Umbrella project support is janky

PropEr fails to find test modules in test/ if this directory is at the top of an umbrella project.

• need to make a sample app to play with
• add support
• figure out how to make counterexample storage and retries work there
• ensure this works fine with test/ subdirectories with apps themselves there

Provider hooks

Support provider hooks so plugins like https://github.com/project-fifo/rebar_erl_vsn can also be used in combination with PropEr testing.

How to silent usless warnings when using PropEr?

Hi @ferd

Is there a way to silent these warnings? I see no reason to add TypeSpec to properties IMHO.

$ rebar3 proper
test/prop_btree.erl:8: Warning: missing specification for function prop_btree_insert/0
test/prop_btree.erl:15: Warning: missing specification for function prop_btree_delete/0
[...]

My rebar.config:

{require_otp_vsn, "2+"}.

{deps, []}.

{project_plugins, [rebar3_proper]}.

{profiles,
   [{test, [
          {deps, [ {proper, {git, "https://github.com/proper-testing/proper.git",
                             {branch, "master"}}}
                 , {covertool, {git, "https://github.com/covertool/covertool.git",
                                {branch, "master"}}}
                 ]}
         ]}
 ]}.

Thanks