pytest-testmon is a pytest plugin which selects and executes only tests you need to run. It does this by collecting dependencies between tests and all executed code (internally using Coverage.py) and comparing the dependencies against changes. testmon updates its database on each test execution, so it works independently of version control.
You are reading docs for testmon version 0.9.x. For 1.0.0. go here.

Quickstart

# build the dependency database and save it to .testmondata
pytest --testmon

# change some of your code (with test coverage)

# only run tests affected by the changes
pytest --testmon
        

Before testmon can select the right subset of your tests you have to run all your tests with the --testmon option.

If you want to read more about how testmon collects and compares changes we wrote about it here.

If you want to run your tests on every file save we recommend using testmon with pytest-watch.

Testmon also remembers and re-reports the failures even if nothing in their execution path is changed.

It's also possible to force re-execution of all failures without changing any source code with the switch --testmon --tlf . (tlf = test-last-failed analogous to pytest native --lf or --last-failed) This is handy when you want to have fresh output (e.g. with -s) or if you accidentally executed tests with failure in the environment (e.g. database down).

Installation

pip install pytest-testmon

All command line options

  --testmon         Select only tests affected by recent changes.
  --testmon-tlf     Re-execute last failures regardless of source change status
  --testmon-off     Turn off (even if activated from config by default)

  --testmon-singleprocess
                    Don't track subprocesses

  --testmon-readonly
                    Don't track, just deselect based on existing .testmondata

  --testmon-project-directory=PROJECT_DIRECTORY
                    Only files under this directory will be tracked, can be repeated.

Configuration

Add testmon to pytest.ini

[pytest]
addopts = --testmon # you can make --testmon a default if you want
# If you want to separate different environments running the same sources.
run_variant_expression = os.environ.get('DJANGO_SETTINGS_MODULE') + ':python' + str(sys.version_info[:2])

More complex run_variant_expression can be written: the os, sys and hashlib modules are available, and there is a helper function md5(s) that will return hashlib.md5(s.encode()).hexdigest().

Troubleshooting - usual problems

Testmon selects too many tests for execution: Depending you your change it most likely is by design. If you changed a method parameter name, you effectively changed the whole hierarchy parameter -> method -> class -> module, so any test using anything from that module will be re-executed.

Tests are failing when running under testmon: It's quite unlikely testmon influenced the execution of the test itself. However set of deselected and executed tests with testmon is highly variable, which means testmon is likely to expose undesired test dependencies. Please fix your test suite. We wrote down a couple of tips and tricks on how to tackle this challenge here.

You can also try if your test is influenced under pytest-cov (coverage) without testmon. For reporting a bug a repository with description of unexpected behavior is the best, but please don't hesitate to report even if your project is closed source. We'll try to fix it!

Test dependencies

There are many things influencing the outcome of a test. Testmon keeps track of some of them, but not all:

  1. code inside the tested project itself, which presumably changes frequently
  2. environment variables (e.g. DJANGO_SETTINGS_MODULE), python version (you are able to separate testmon data by configuring run_variant_expression)
  3. code in third-party packages, which presumably change infrequently
  4. static files (txt, xml, other project assets)
  5. external services (reachable through network)

testmon so far deals with incrementally running tests when faced with the 1. and 2. category of changes.

Later versions can implement some detection of other categories.