seelengrab/propcheck.jl

A package for simple property based testing in julia.

PropCheck.jl

A simple, thin package for property based testing.

Maintenance only

PropCheck.jl is in maintenance mode which means that no new features will be added. Bugs that compromise the intended behavior that pop up will continue to be fixed (if sufficiently feasible). For future development of property based testing, as well as better performance, consider using Supposition.jl instead.

This package is now intended to serve as an example for how a Haskell project deeply relying on lazy evaluation and type classes could be ported to Julia (thought it may not necessarily be a good example for that ;) ).

Installation

This package is registered with General, so to install do

pkg> add PropCheck

PropCheck.jl currently supports Julia versions 1.6 and up. CI runs on nightly and is expected to pass, but no guarantee about stability on unreleased versions of Julia is given.

Please check out the documentation to learn how you can use PropCheck.jl to fuzz your code.

propcheck.jl's Issues

Automatic test running based on methods

The idea is that, given a function and a signature, it ought to be possible to automatically fuzz the method in question, by creating itype generators for each argument and interleaveing them.

There is also an adjacent, possibly required, objective here - a default generation of instances of a type, purely from that type. This would be done by checking whether there are any methods attached to the given type, taking one random one & recursively generating the input arguments to that method. This will be horrible in the general case (constructors are for rejecting invalid arguments after all, so those shouldn't be put into constructors in the first place), but could be a nice idea for some code.

Allow generation of abstract types

Right now, itype doesn't specify what to do when a non-concrete type is requested - it'll likely error. It would be nice to be able to give itype an abstract type, and have PropCheck.jl automatically generate valid instances for that type through reflection, providing a (very generic) integrated shrinker. This would also allow practical generation of Any value.

`check` doesn't return `Bool` when counterexamples found

Loving the package so far!

In the documented examples, expressions of the form

@test check(...)

are provided.

Since check only returns true or the value of the counter example, I don't think this is appropriate usage. I've had to write my own wrapper for my use cases. The one I found that works for me is

function wrap_check(prop::Function, gen::Integrated)
    chk = check(prop, gen)
    chk isa Bool && return chk
    @test prop(chk)
    return false
end

and then called it as

@test wrap_check(prop, gen)

I'm not too familiar with PropCheck.jl internals. I think the type instability of check's return values are somewhat justified for what it's trying to achieve. But I thought I might bring this up because people wanting to use your package would go down your package documentation's idiomatic usage and then travel down the road of discovery I went on.

On the other hand, we choose to be fine with PropCheck.jl reporting counterexample finds as Expression evaluated to non-Boolean test errors instead of test failures, the former of which I'm not a fan of.

Replace camelcase

The Julia style guide says

functions are lowercase (maximum, convert) and, when readable, with multiple words squashed together (isequal, haskey). When necessary, use underscores as word separators. Underscores are also used to indicate a combination of concepts (remotecall_fetch as a more efficient implementation of fetch(remotecall(...))) or as modifiers.

Would you mind applying that here? It's easier to learn and use an API if it follows the same conventions as the rest of the ecosystem and doesn't stick out in my code.

That would apply to shrinkTowards and any others.

PropCheck Logo

I saw a conversation on Discourse about a logo for a PropCheck package badge. I'd been meaning to try my hand at some Luxor.jl, and thought this might be a good way to learn. No sweat if it isn't want you're looking for, just wanted to share in case it is helpful to you:

The code to generate

using Luxor

avg(x,y) = (x+y)/2

function parabolic_ish(start::Point, vertex::Point, stop::Point)
    curve(
        Point(start.x, avg(start.y, vertex.y)),
        Point(avg(start.x, vertex.x), vertex.y),
        vertex
    )
    curve(
        Point(avg(vertex.x, stop.x), vertex.y),
        Point(stop.x, avg(stop.y, vertex.y)),
        stop
    )
end

@draw begin
background("black")
inner_r = 50
space = 40
total_r = inner_r + space
blade_base_w = 40
half_w = blade_base_w / 2
blade_x_start = sqrt((total_r)^2 - (half_w)^2)

blade_len = 380
max_drag = 100

inner_curve_w = ((blade_len - blade_x_start) / 2.3) + blade_x_start
inner_curve_midpoint = avg(inner_curve_w, blade_x_start) * 1.1

blur_curve_mid_x = avg(inner_curve_w, blade_len) 

colors = ("purple", "red", "green")
rotate(π/6)

for i in 1:3
    sethue(colors[i])
    move(Point(blade_x_start, -half_w))
    # outer curve of the blade
    curve(
        Point(blade_len * 1.2, -(half_w + 200)), 
        Point(blade_len + 90, max_drag - 100), 
        Point(blade_len, max_drag))
    # "blur" curve outer
    parabolic_ish(
        Point(blade_len, max_drag),
        Point(avg(blade_len, blur_curve_mid_x), 20),
        Point(blur_curve_mid_x, max_drag*.7)
    )
    # "blur" curve inner
    parabolic_ish(
        Point(blur_curve_mid_x, max_drag*.7),
        Point(avg(blur_curve_mid_x, inner_curve_w), 20),
        Point(inner_curve_w, max_drag*.4)
    )
    # inner most curve on bottom of blade
    parabolic_ish(
        Point(inner_curve_w, max_drag*.4),
        Point(avg(inner_curve_w, blade_x_start), 20),
        Point(blade_x_start, half_w)
    )
    arc(Point(0,0), total_r, sin(half_w/total_r), sin(half_w/total_r))
    fillpath()

    rotate(2π/3)
    closepath()
end
rotate((π/3))
strokepath() 
sethue("white")
move(Point(0,0))
sector(inner_r, total_r, 0, 2*π, :fill)
# circle(Point(0,0), inner_r, :fill)
end 1000 1000

Improve handling of errors during generation

At the moment, PropCheck.jl assumes the generators don't throw errors. This is suboptimal, due to requiring more stringent checks from users than may be necessary. It would be cool if PropCheck.jl could either try/catch errors during such generation, or provide a way to ensure throwing values are never triggered during generation.

Automatic regression testing

Once a failure is found, it'd be good to record the failure and replay it later on, to catch regressions of known-faulty inputs.

Some considerations about this:

• Don't record too many - we don't want to take up a tremendous amount of diskspace.
  • Perhaps some kind of LRF (Least recently failed) cache could be of use
• The cache needs to be evictable manually - there could be false positives that need to be removed.
• Perhaps some known-good inputs should be saved as well?

Better handling of errors during testing & shrinking

PropCheck.jl currently treats all errors ocurring during shrinking the same, as just another failure of the predicate, as if it had returned false. It does record that an error happened, but it treats all these errors the same - in reality, it would be interesting to see whether shrinking could be made error-aware, so that once an error is hit, only executions also throwing that same error (and not just any failure of the predicate!) are considered "valid" shrinks.

Move `trim` argument of `filter` to keyword?

filter(f, i, trim::Bool) has one of these boolean flag arguments. I don't like these because they're hard to read and dirty up the functional design.

If you feel similarly, some ideas:

• filter(f, i ; trim::Bool) (easier to read, at least)
• filter(f, i) and trimfilter(f, i) (I tend to like this from a functional multiple dispatch design perspective)
• filter(f, i, TRIM) (an enum)

Integrated shrinker producing special cases

Some types have known special cases, e.g. Inf for Float64 or "" for String. It'd be interesting to have an integrated shrinker that first generates these special cases, and only afterwards falls back to generating random instances of the desired type.

Better printing of shrinker types

These objects are printed with way too much detail; it needs to be trimmed down a bit. The method to implement here is the three-arg Base.show(::IO, ::MIME"text/plain", obj), with obj being all subtypes of AbstractIntegrated.

The question is - how much detail is sufficient? I'd at least like to prevent the humongous type parameters from being printed, simply due to the vast majority of them not being all that interesting. The ones that are interesting are return types.

Function aware value generation

An integrated shrinker like itype currently produces all possible values of the given type, irrespective of the function the value ultimately ends up in. It would be interesting to see whether we can guide generation of values based on the branches on the generated value, in the function that is ultimately tested.

For example, in functions like this:

function foo(x::Int)
     x < 5 && throw(ArgumentError("Not at least 5!"))
     # other code
end

the input space is clearly delineated into two regions, those Int smaller than 5 and thsoe at least 5. In the general case, these sorts of branch points are especially interesting, in particular when it comes to testing whether the branch point is correct. It would be great if PropCheck.jl provided not only integrated shrinkers, but also integrated generators - or function aware generators, which are biased to generate values close to these branch points.

This is a very out-there idea though, so this issue is just here for tracking the thought, not as an indicator that this is actively being worked on.

DataGenerators.jl stuff

Just for your interest, I ran across these.

https://www.youtube.com/watch?v=uk6-ZHRu-5k
https://github.com/simonpoulding/DataGenerators.jl
https://github.com/simonpoulding/DataGeneratorTranslators.jl