plaidfinch/strictcheck

Keep your laziness in check!

StrictCheck: Keep Your Laziness In Check

StrictCheck is a property-based random testing framework for observing, specifying, and testing the strictness behaviors of Haskell functions. Strictness behavior is traditionally considered a non-functional property; StrictCheck allows it to be tested as if it were one, by reifying demands on data structures so they can be manipulated and examined within Haskell.

For details, see the library on Hackage: https://hackage.haskell.org/package/StrictCheck.

strictcheck's Issues

Show instance for Demand

What do you think of adding an instance of Show for Demand a (or (%) if you prefer) so that we can simply write this:

> observe (`seq` ()) (&&) False False
(False,False :* _ :* Nil)

I find it quite convenient for doing small experiments in ghci. I have this instance as a quick hack but you may know a better implementation:

instance Shaped a => Show (Demand a) where
  show = prettyDemand

An alternative proposal is to have specialized functions to print the output of observe and variants, or a variant of observe that prints in addition to/instead of returning the demand.

observe produces wrong results and is not referentially transparent with ghc-8.6.1 and -O1 or -O2

I was just reading your paper and when reading about entangle and observe I thought that there was a good chance, that memorization would cause observe to use the same entangled value twice, thus giving false results.

Indeed I found the following program, where that seems to be the case:

{-# LANGUAGE GADTs #-}

import Test.StrictCheck
import Data.Foldable (traverse_)
import Data.Bifunctor (bimap)

main :: IO ()
main =
  (print . bimap prettyDemand (\(x :* Nil) -> prettyDemand x)) `traverse_` [
    observe (\() -> ()) (\() -> ()) (),
    observe (\() -> ()) (\_  -> ()) ()
  ]

I would expect the following output:

("()","()")
("()","_")

This matches the actual output for ghc versions 8.2.2 and 8.4.3 with any optimisation level as well as ghc-8.6.1 with -O0. However when compiling with ghc-8.6.1 and -O1 or -O2 the actual output is this:

("()","()")
("()","()")

Even worse, when we change the order of the two observe-calls in the list we get this:

("()","_")
("()","_")

I think the best way to fix this would be to change entangle to this safe (exportable) version:

entangle :: a -> IO (a, IORef (Thunk a))
entangle =   do
    ref <- newIORef Thunk
    return ( unsafePerformIO $ do
               writeIORef ref (Eval a)
               return a
           , ref )

This version of entangle should be referentially transparent and allows the user to observe the evaluation status of the entangled value at different points in time. Furthermore, and most importantly for this issue, leaving out the outer unsafePerformIO makes sure that two calls to entangle will create two different entangled values with two different IORefs, thus making bugs like this impossible. However this of cause would probably require a lot of code to be rewritten to accommodate for the new type and maybe there is a quick and dirty solution that somehow prevents memorization or even not exporting observe and making sure it is only used in safe ways.

Error diagnostic crashes on Windows

Via /u/Tarmen, we have a bug report posted on Reddit. The relevant piece seems to be:

...on windows the error reporting breaks with

*** Failed! Falsifiable (after 15 tests and 6 shrinks):
┌──────────────┐   ┌─────────────────────┐
│ Inputs       ├───┤ Demand on result    │
├──────────────┤   ├─────────────────────┤
│ StrictCheckTest.EXE: <stdout>: commitBuffer: invalid argument (invalid character)

I suspect this is due to issues with Unicode text handling. The crash occurs right before StrictCheck would ordinarily print the Unicode bullet character • (U+2022).

A search of bug reports seems to indicate this is a locale issue, in particular this StackOverflow answer says that this issue might be resolved by runningsetLocaleEncoding utf8 before printing text. It references the documentation for GHC.IO.Encoding, as noted in part of the Hakyll FAQ.