ikding/escape-from-automanual-testing

A three-hour tutorial on property-based testing with https://hypothesis.works

Escape from automanual testing with Hypothesis!

A three-hour workshop on property-based testing with Hypothesis, first delivered at PyCon 2019.

It is designed for intermediate Python users, with extension activities up to expert level. Some minimal experience of unit testing and pytest are assumed - if you have ever used @pytest.mark.parametrize, you are overqualified; but if you have never written and run unit tests you may have trouble.

How this workshop works

It's a crash-course in the important concepts, alternating between short talks and hands-on excercises. You'll get to know the architecture of the library, but not every detail of the API - you can look up that documentation at any time.

Setting up

Clone this repository, and pip install pytest hypothesis. That's it!

Hypothesis is also available on conda-forge, if you prefer to use conda. For the SciPy edition of this workshop, there are optional excercises that require Numpy and Pandas.

To test that everything is installed correctly, run pytest pbt-101.py. You should see nine passing tests and no errors.

Any recent version of pytest, and any hypothesis>=4.0. Use whatever package manager and environment you prefer - if in doubt, just pip install as above. Hypothesis is compatible with every currently supported version of Python, i.e. 2.7, 3.5, 3.6, and 3.7 - and a similarly wide range of versions for it's optional dependencies.

Property-based testing 101

Each block starts with a short talk (slides separated by a blank black slide), followed by a hands-on excercise where you can apply what you've just learned. In this first block we'll see a taxonomy of testing techniques, define property-based testing, and get an initial overview of Hypothesis.

After the talk:

1. Follow the "Setting up" instructions above
2. Run pytest pbt-101.py. You should see several passing tests.
3. Open the file in your preferred editor and read the detailed instructions within! (in short: fix the test, check the test fails, fix the test, check that it passes)

Hypothesis strategies and property-based "tactics"

This block aims to get you comfortable and productive with Hypothesis, which means covering two things: how to generate all kinds of data, and how to use it in your tests.

strategies are objects which tell @given what to pass to your test function. Hypothesis ships with dozens for standard library types and optional dependencies such as pytz, Django, Numpy, and Pandas - to say nothing of third-party extensions! We'll see what's available to explicitly describe your data or infer it from a schema (e.g. strings from a regular expression), and how you can combine, compose and adjust strategies to produce something quite different.

"Tactics" are design patterns for property based tests. They range from common properties to test, through to embedding assertions in your code (not just tests) for free integration tests, and more.

After the talk, pytest strategies-and-tactics.py, and continue as above.

Testing the Untestable or Scientific Hypothesis

Nothing is ever really untestable - but sometimes you need better tools to make testing worth the trouble. We'll explore two approaches:

• Stateful testing, where you define a finite automaton and Hypothesis uses it to generate whole test programs (by choosing actions as well as values).
• Metamorphic testing, where you don't know exactly what the code should do - but do know something about how changes in the input relate to changes in the output.

The excercises in test-the-untestable.py are deliberately challenging. Choose whichever one is the most interesting to you, and don't worry if it takes you the whole block in class - you can always come back to the others later.

Alternatively, if you use the Numpy / Pandas stack, scientific-hypothesis.py is full of excercises that demonstrate Hypothesis' support for generating arrays, dataframes, and all the related things you might need to test data-centric scripts

• or libraries!

The bigger picture

We'll discuss Hypothesis' performance characteristics and configuration options, get a sense of the community around it and the project roadmap, and look at how Hypothesis fits into the wider testing and correctness landscape in Python.

For this final block, you have a few options:

• Continue working on any unfinished exercises from previous blocks
• Start testing your own code with Hypothesis
• Join a group discussion - what to you plan to test? What 'killer feature' is missing?

Useful links

As well as the links above, you may be interested in:

• Hypothesis' official documentation
• Hypothesis' technical blog
• More practice excercises - if you finish early, these are a good challenge.
• Choosing properties for property-based testing is written for F#, but the ideas are useful in any language.
• PBT with oracle functions and Finding property tests. Hillel Wayne has written several other posts on testing with Hypothesis.
• AFL + QuickCheck = ? - a hardware engineer's perspective on software testing. (Hypothesis did use coverage information for a while, but took it out as the overhead made it net-negative test power)
• How SQLite is Tested