Automatically determining affected tests sounds too good to be true. Python developers rightfully have a suspecting attitude towards any tool which tries to be too clever about their source code. Code completion and symbol searching doesn’t need to be 100% reliable but messing with the test suite execution? This page explains what testmon tries and what it does not try to achieve.
There is no heuristics involved. testmon works with these pretty solid assumptions:
E.g. having test_s.py:
1 def add(a, b): 2 return a + b 3 4 def subtract(a, b): 5 return a - b 6 7 def test_add(a, b): 8 assert add(1, 2) == 3
If you run
coverage run -m pytest test_s.py::test_add you’ll get:
1> def add(a, b): 2> return a + b 3 4> def subtract(a, b): 5! return a - b 6 7> def test_add(): 8> assert add(1, 2) == 3
Now you can change the unexecuted line
! return a - b
nuclear_bomb.explode() and it still won’t affect running test_s.py::test_add.
How does testmon approach processing the source code and determining the dependencies? It splits the code into blocks. Blocks can have holes which are denoted by a placeholder. ( “transformed_into_block” token ). Each Block also has a start, end (line numbers, 1-based, closed interval)
The above code is transformed into 4 blocks:
Block1: 1-8 (start-end)
def add(a, b): transformed_into_block def subtract(a, b): transformed_into_block def test_add(a, b): transformed_into_block
return a + b
return a - b
assert add(1, 2) == 3
After running the test with coverage analysis and parsing the source code, testmon determines which blocks does test_s.py::test_add depend on. In our example it’s Block 1,2 and 4. (and not Block 3). testmon doesn’t store the whole code of the block but just a checksum of it. Block 3 can be changed to anything. As long as the Block 1,2 and 4 stay the same, the execution path for test_s.py::test_add and it’s outcome will stay the same.
The limits and reliability of this method are pretty much the same as limits of coverage.py (things that cause trouble)