Adventures in Property-Based Testing

Midlands Graduate School 2019

John Hughes

Preparations before the school begins

The exercises in this course will be involve practical Haskell programming using the latest version of QuickCheck, and a theory exploration tool, QuickSpec. It’s important to have all the necessary software installed on your laptop before the School begins. You will need:

· A Haskell installation. I recommend installing the Haskell Platform, if you do not already have one: https://www.haskell.org/platform/

· A recent version of QuickCheck, 2.12 or newer. The most recent version of the Haskell Platform (8.6.3) includes this; older versions include an older version of QuickCheck which lacks some of the features I will be discussing. If you have an older version of QuickCheck installed, you can upgrade it using the command cabal install quickcheck.

· A version of QuickSpec. You can install this using the command cabal install quickspec; however, at the time of writing the installation sometimes fails. If this happens to you, I suggest installing QuickSpec in an Ubuntu virtual machine instead. This is a slow process involving a large download, if you need to create a new Ubuntu VM—don’t wait until you are at the School to do it.

· Basic familiarity with Haskell: you will be adding code to existing modules, so if you are unfamiliar with Haskell syntax then there will be examples for you to work from, but of course, some Haskell knowledge will be helpful. Good sources are Graham Hutton’s book Programming in Haskell, or Learn you a Haskell for Great Good, a free online tutorial.

The lecture slides are available as PDFs below; you may wish to make notes on the slides during the lectures. The exercises are available as zip files containing a problem description as a PDF, and Haskell files needed to solve the problem; some of the Haskell files contain solutions or other “spoilers”, so always read the problem description first.

Sunday

Lecture: Testing the Hard Stuff and Staying Sane

An introduction to property-based testing, along with one or two war stories about interesting bugs.

Monday

Exercise class: Property Driven Development

An introduction to using Haskell QuickCheck, and an experience in using property-based testing in test driven development.

Lecture: Testing with Properties

Anatomy of a QuickCheck property. Where do properties useful for testing come from? Some guidelines and common pitfalls. The QuickSpec property generator.

Tuesday

Exercise class: Testing with Properties

Writing properties for testing binary search trees, based on yesterday’s lecture. Using properties to diagnose bugs. Using QuickSpec to explore properties.

Lecture: Stateful Systems

Testing a stateful API requires specialized properties, and a state-modelling library to express them. Using such a library to test an API modelled on the Erlang process registry.

Wednesday

Exercise class: Testing a Process Registry

Complete the specification begun in yesterday’s lecture, and extend it to test the behavior of the registry when processes may sometimes crash.

Lecture: Effective Properties

Plausible-looking properties can actually be ineffective at finding bugs, if no thought is given to test case distributions. How can we measure distributions? What distributions should we aim for? How can we state—and test—coverage requirements on distributions, to prevent later regressions?

Thursday

Exercise class: Labelling and Tuning Tests

Label tests in an implementation of interval sets (from Monday), and tune the testing of delete. Measure coverage of the registry tests (from Wednesday), and tune them to test the rare cases more frequently.