well-typed/quickcheck-lockstep

Support for lockstep-style testing using quickcheck-dynamic

Library for lockstep-style testing with quickcheck-dynamic

See https://well-typed.com/blog/2022/09/lockstep-with-quickcheck-dynamic/ for a tutorial.

quickcheck-lockstep's Issues

Enable CI

printed counter-examples don't produce valid code

Older versions of quickcheck-lockstep that were compatible with quickcheck-dynamic 2.x would produce counter examples like so:

 [Var 1 := ANew,
  Var 3 := AInsert (GVar (Var 1) OpId) (Right 0) 0,
  Var 4 := AInsert (GVar (Var 1) OpId) (Right 1) 0,
  Var 8 := AInsert (GVar (Var 1) OpId) (Right 2) 0,
  Var 9 := AInsert (GVar (Var 1) OpId) (Right 4) 0,
  Var 10 := AInsert (GVar (Var 1) OpId) (Left (GVar (Var 9) OpId)) 0,
  Var 13 := AInsert (GVar (Var 1) OpId) (Left (GVar (Var 8) OpId)) 0,
  ...

These obviously include the variable names.

From quickcheck-dynamic-3.0.0 changelog:

Improved printing of counterexamples in DL - they are now printed as code that can be copied more-or-less verbatim to create a runnable counterexample in code.
So the printing of the counter examples changes. However quickcheck-lockstep is not compatible with this new method. For example I get:

do action $ ANew
   action $ AInsert (GVar var1 OpId) (Right 2) 0
   action $ AInsert (GVar var1 OpId) (Right 1) 0
   action $ AInsert (GVar var1 OpId) (Right 3) 0
   action $ AInsert (GVar var1 OpId) (Right 0) 0
   action $ AInsert (GVar var1 OpId) (Left (GVar var20 OpId)) 0
   action $ AInsert (GVar var1 OpId) (Left (GVar var3 OpId)) 0
   ...

obviously the problem here is that variables are used, but never bound, so it's not valid code (and one cannot recover it with a bit of text processing because we never know the variable numbers and they're not sequential in shrunk counter examples).

I understand this is probably because quickcheck-lockstep has its own notion of variables on top of quickcheck-dynamic.

Obviously what would be nice is if the printing of counter examples worked somehow. Otherwise it's impractical to run counter examples, and thus impractical to do debugging. I'll go back to the old version for now :-)

Make `quickcheck-lockstep` compatible with GHC 9.8

Document `.Op` module

Variables should be given an `Ord` instance, so we can "shrink" them