Testing with Hypothesis

Hypothesis is a tool for test case generation in Python.

A normal test method might look like this:

    def test_real_problem( self ):
        p = Problem()
        p.addVariable( 'a', range( 5 ) )
        p.addVariable( 'b', range( 5 ) )
        p.addVariable( 'c', range( 5 ) )
        firstPair = [ (1, 2),
                      (2, 3),
                      (3, 4) ]
        secondPair = [ (2, 3),
                       (3, 0),
                       (4, 0) ]
        p.addConstraint( TupleConstraint( firstPair ), ['a', 'b'] )
        p.addConstraint( TupleConstraint( secondPair ), ['b', 'c'] )
        soln = p.getSolutions()
        self.assertIn( {'a' : 1, 'b': 2, 'c': 3 }, soln )
        self.assertIn( {'a' : 2, 'b': 3, 'c': 0 }, soln )
        self.assertIn( {'a' : 3, 'b': 4, 'c': 0 }, soln )
        self.assertEqual( len( soln ), 3 )

It sets up exactly one test case, runs it, and checks the results. A Hypothesis test generates many test cases, but requires you to write code that verifies the correct behavior in all of them. It can be useful even without an explicit invariant to test, as some inputs may cause an unexpected exception to be thrown.

Here's a Hypothesis test of the same class:

    @given( st.lists( st.text(), min_size=2, max_size=2, unique=True ),
            st.lists( st.integers( 1, 100 ),  min_size=2, max_size=2 ),
            st.data() )
    def test_satisfiability_2( self, variables, ranges, data ):        
        p = Problem()
        p.addVariable( variables[0], range( ranges[0] ) )
        p.addVariable( variables[1], range( ranges[1] ) )

        tuples = data.draw( st.sets( st.tuples( st.integers( 0, ranges[0] - 1 ),
                                                st.integers( 0, ranges[1] - 1 ) ) ) )

        p.addConstraint( TupleConstraint( tuples ), [variables[0], variables[1]] )
        solns = p.getSolutions()
        note( "Solutions: {}".format( solns ) )
        solnTuples = set( ( s[variables[0]], s[variables[1]]) for s in solns )
        self.assertEqual( solnTuples, tuples )

Hypothesis fills in all the "given" arguments based on the "strategies" supplied. The strategies specify what format the inputs must have.

• I used a lists strategy to generate variables because they must be distinct. Otherwise I could have done st.tuples( st.text(), st.text() ) to generate exactly two variable names.
• The data() strategy lets some of the test input depend on other inputs, or even on some of the test output. In this case, I wanted all the input data to be within the ranges selected. Filtering to cases where this occurred naturally was too inefficient.

This was my first experiment using Hypothesis, and it did find bug where I had made a typo: https://github.com/mgritter/soffit/commit/5f15cc3403552355fa74b336e11bce9791d84d86#diff-bd859e02ab048cec00085c33844bae52

I tried inserting some bugs and Hypothesis was able to find them, although I scratched my head a little bit when some modifications turned out not to trigger a failure. They made forward checking more permissive, which doesn't affect the correctness of the final result--- so not really a bug as far as the test above is concerned. It would be significantly harder to define this as an invariant the test could check, I think.

When Hypothesis finds a bug, it tries to minimize the test case that reproduces the same error. For example, if I don't initialize one of the indexes correctly, I get the following failure:

Falsifying example: test_satisfiability_2(self=<test.test_constraint_hyp.TestTupleConstraint
testMethod=test_satisfiability_2>, variables=['', '0'], ranges=[1, 2], data=data(...))
Draw 1: {(0, 1), (0, 0)}
Solutions: []

You can reproduce this example by temporarily adding 
@reproduce_failure('3.82.1', b'AAABAAEAAAABAAEBAA==') as a decorator on your test case

This shows what Hypothesis picked as the inputs, both at the start of the test case and any draws during the test case. It also included the note I added so that I can see what the solution set actually was.