Determining affected tests

Automatically determining affected tests sounds too good to be true. Python developers rightfully have a suspecting attitude towards any tool which tries to be too clever about their source code. Code completion and symbol searching doesn’t need to be 100% reliable but messing with the test suite execution? This page explains what testmon tries and what it does not try to achieve.

There is no heuristics involved. testmon works with these pretty solid assumptions:

  • library can reliably determine executed and unexecuted lines of code of a tested program
  • even in dynamic language, unexecuted line of code doesn’t influence the outcome of execution of the surrounding code
  • method body which is not reached/executed by a specific test cannot influence outcome of that test
  • the lines which are executed can have so many side effects that we don’t try to determine their real dependencies, we re-execute dependent tests even on smallest change

E.g. having

    1    def add(a, b):
    2        return a + b
    4    def subtract(a, b):
    5        return a - b
    7    def test_add(a, b):
    8        assert add(1, 2) == 3

If you run coverage run -m pytest you’ll get:

    1>   def add(a, b):
    2>       return a + b
    4>   def subtract(a, b):
    5!       return a - b
    7>   def test_add():
    8>       assert add(1, 2) == 3

Now you can change the unexecuted line ! return a - b to nuclear_bomb.explode() and it still won’t affect running

Implementation details

How does testmon approach processing the source code and determining the dependencies? It splits the code into blocks. Blocks can have holes which are denoted by a placeholder. ( “transformed_into_block” token ). Each Block also has a start, end (line numbers, 1-based, closed interval)

The above code is transformed into 4 blocks:

Block1: 1-8 (start-end)

    def add(a, b):
    def subtract(a, b):
    def test_add(a, b):

Block2: 2-2

    return a + b

Block3: 5-5

    return a - b

Block4: 8-8

    assert add(1, 2) == 3

After running the test with coverage analysis and parsing the source code, testmon determines which blocks does depend on. In our example it’s Block 1,2 and 4. (and not Block 3). testmon doesn’t store the whole code of the block but just a checksum of it. Block 3 can be changed to anything. As long as the Block 1,2 and 4 stay the same, the execution path for and it’s outcome will stay the same.

The limits and reliability of this method are pretty much the same as limits of (things that cause trouble)