Property-Based Testing with PropEr, Erlang, and Eliixir

Fred Hebert is a systems architect at a video surveillance, access control management, and license plate recognition company. He previously worked as a tech lead on a cloud platform's networking services. He has also worked in real-time bidding systems, as a professional Erlang trainer and course maintainer, and is one of the maintainers of the principal Erlang build tool (rebar3).

Introduction
The Basics
Foundations of Property-Based Testing
Promises of Property-Based Testing
Properties First
Property-Based Testing in Your Project
Running a Property
Wrapping Up
Writing Properties
Structure of Properties
Default Generators
Putting It All Together
Wrapping Up
Thinking in Properties
Modeling
Generalizing Example Tests
Invariants
Symmetric Properties
Putting It All Together
Wrapping Up
Custom Generators
The Limitations of Default Generators
Gathering Statistics
Basic Custom Generators
Fancy Custom Generators
Wrapping Up
Stateless Properties in Practice
Responsible Testing
The Specification
Thinking About Program Structure
CSV Parsing
Filtering Records
Employee Module
Templating
Plumbing It All Together
Wrapping Up
Properties-Driven Development
The Specification
Writing the First Test
Testing Specials
Implementing Specials
Negative Testing
Wrapping Up
Shrinking
Re-centering with ?SHRINK
Dividing with ?LETSHRINK
Wrapping Up
Targeted Properties
Understanding Targeted Properties
Targeted Properties in Practice
Thinking Outside the Box
Wrapping Up
Stateful Properties
Stateful Properties
Laying Out Stateful Properties
How Stateful Properties Run
Writing Properties
Testing a Basic Concurrent Cache
Testing Parallel Executions
Wrapping Up
Case Study: Bookstore
Introducing the Application
Writing Generators
Broad Stateful Testing
Precise Stateful Modeling
Refining the Tests
Debugging Stateful Properties
Parallel Tests
Wrapping Up
State Machine Properties
Laying Out State Machine Properties
How State Machine Properties Run
Writing Properties
Testing a Circuit Breaker
Modeling the Circuit Breaker
Adjusting the Model
Wrapping Up

Top Five Property-Based Testing Tips 1. Sit back and think. The hardest part of writing property-based tests is coming up with properties, and that’s usually because we’re not quite sure of what the program should do in the first place. It’s fine to leave your IDE or text editor, and just think things over again a few times, draw a bit on paper, and take a step back. If you draw a small diagram of what a solution looks like, or write up an explanation to someone who knows nothing about your program, you may just find properties that way. 2. Start broad, and build up. You don’t need to be specific at first. Start with general data types and properties. It can be as simple as “this type of input always results in this type of output”. Then add some complexity: maybe you need more precise input for some scenario, and that will guide you to new fancier data generators and properties. This approach essentially helps get over blank page syndrome with your tests. Attach the exercise to something discrete and you might soon get going. 3. Assemble multiple well-defined properties rather than writing a more complex one. This is especially true if you take one of the previous iterative approaches. It might be tempting to add more and more checks and rules to a single property test, but by doing so you will make debugging harder, and exploration trickier. By using various well-targeted properties, you increase the chances of keeping everything much more maintainable. Add one or two general properties to unite them all together and you’ll have a solid test suite in no time. 4. Double-check the quality of properties, either through metrics, or by adding random bugs in your code and making sure your properties find them. Properties can often feel very trustworthy; metrics let you know if the generator distribution makes sense, and adding random bugs is just a good general way to let you know whether there are areas of your code that are not validated very well. If you can see a good distribution and your properties appear to catch most bugs you try to willingly add to code, chances are better they’ll find all the unexpected ones as well. 5. Think like an operator debugging an incident. How would you detect a bug in something? How would you search for it in any output? What data would you need to confirm there’s a problem? This is an iterative process. The patterns you’d look for in data to confirm a bug let you figure out your properties. For example, you might find that a backpressure mechanism does not work well if a request takes too long without being cancelled: this can be turned into a property where you check for the quality of response delays. The data you need to figure this out, on the other hand, may point to changes you must make to your system to make it observable. In the same example, you may need to expose metrics or logs regarding response times that the operator would use; then, make your tests use that same data. The testability of the system becomes connected to its operability!