In this article, we’ll take a look and set-up pytest framework in PyCharm using our small example project. We’ll also introduce and try two very useful pytest-plugins to run tests more flexibly and effectively to save our time and CI/CD resources.
For the sake of this article, we’ve prepared a small project called MasterMind. It’s a simple console-based game where the computer generates a random number that you have to guess.
You can download the project from here.
Brief description of MasterMind’s project structure.
/mastermind <-------------- top-level dir /src <------------------- source codes / modules /mastermind <---------- main python module __init__.py __main__.py <-------- entry point (startup of game) exceptions.py <------ contains game exceptions game.py <------------ game logic heading.txt <-------- heading at the beginning of the game utils.py <----------- various helpers /tests <----------------- pytests test_mastermind.py <--- test suite setup.py <--------------- package setup script
Please notice that we’re using src folder to separate top-level Python modules from the main directory. This is used to protect from accidental imports and other unwanted effects. If you’re interested in more details, check out this blog.
I’m bringing this up because we’ll have to do some additional steps in PyCharm configuration.
So here we are, firing up PyCharm and opening the mastermind folder as a new project.
If you’re following this tutorial, you can extract
mastermind.zip and open the
directory in PyCharm. This will become the root folder of our project in PyCharm.
Next we’ll need to configure PyCharm, to use Python3 as our project’s interpreter. You can find this setting under Preferences -> Python interpreter. If there are no interpreters at all, we’ll have to add one. This will also create a virtual environment.
Since we are using a clean virtual environment, we also need to install the pytest framework
in order to use it for running our tests. Just stay on the Python Interpreter preference page
and install packages from there. Right under selected Python interpreter is a list of installed
packages (so far just
pip). To install new packages click add "+" sign and search
pytest package. Press install package to install it. This will be added only to our project’s
If you prefer console, you can install any Python packages using pip command. Just open the console by pressing terminal at the left - bottom.
pytest with command
pip install pytest. This has the same effect as installing
with PyCharm interface. Using terminal you should see something like this:
(venv)$ pip install pytest Collecting pytest Downloading https://files.pythonhosted.org/packages/5d/c3/54f607bc9817fd284073ac68e99123f86616f431f9d29a855474b7cf00eb/pytest-4.4.1-py2.py3-none-any.whl (223kB) |████████████████████████████████| 225kB 1.7MB/s ...[MORE PIP OUTPUT]... Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from pytest) (41.0.1) Installing collected packages: pluggy, py, attrs, atomicwrites, more-itertools, six, pytest Successfully installed atomicwrites-1.3.0 attrs-19.1.0 more-itertools-7.0.0 pluggy-0.9.0 py-1.8.0 pytest-4.4.1 six-1.12.0
Now we can configure pytest as our default test runner in PyCharm. You can find this setting under
Preferences -> Tools -> Python Integrated Tools. Set Default test runner to
So far, we’ve been playing with PyCharm and configuration. Now we should try out the app,
if it’s working and if we can proceed further. MasterMind can be executed as a module.
In the terminal, we would need to execute this command to start the game. Please note,
that the current working directory must be in
(venv)$ cd src (venv)$ python -m mastermind
When you execute just the module like that, Python will know to go to the
which contains the
main function and execute it.
So how to do this in PyCharm? We’ll need to create Run Configuration.
In this configuration, we’ll basically do the same thing, as in terminal. Just execute a module
mastermind. Also, in the terminal, we’ve
cd (change directory) to
src, which contains
our mastermind module. We have to configure this in PyCharm as well. Otherwise, it would not
know where to find our module. You can do so, by right clicking on
src directory and marking
it as sources root.
Now, it’s time to run our tests for MasterMind. To create a test configuration, we’ll use IDE
to set it up for us. Just right-click on
tests directory in Project tool window
and select run pytest in tests.
This will do 2 things. It’ll run the tests and create run configuration. Test results are displayed in Test Runner Tab.
Also, note the test configuration in the top right corner. We can save this configuration as our default test run.
There are a couple of features that are useful for everyday workflow.
When we’re writing new tests for our app, PyCharm makes it really easy to execute a single test. E.g. the one that you’ve just finished writing. You can do so by pressing the green play button, next to the test definition.
Of course, that’s not the end of it. PyCharm lets you execute a whole class (containing multiple tests) or a single file (test suite). You can select these options from the context menu of a mouse right click.
From the official documentation:
In many cases, you can initiate a testing session from a context menu. For this purpose, the Run and Debug commands are provided in certain context menus. For example, these commands are available for a test class, directory, or a package in the Project Tool Window. They are also available for a test class or method you are currently working on in the editor.
It’s better to focus on programming and don’t worry about the tests (until it’s too late) so let’s make PyCharm do the hard work for us. In the Test Runner Tab, we can toggle the automatic rerun of the tests on a code change. Pretty neat, right?
So for example, if we introduce an error, we’ll know it immediately. You’ve probably heard that fixing defects later is much more expensive than early. This will give us rapid feedback!
There are a couple of disadvantages IMHO. It’s great to have rapid feedback, however, PyCharm will trigger the tests on each file change. Even if you add a new line. Moreover, PyCharm auto-saves your changes, which leads to even more repeatedly executed tests.
You could set the automatic rerun of a subgroup of the tests (different run configuration). For example, if you are working on particular functionality, you can set a smaller test suite (covering only what’s needed) and set auto tests just for that configuration. However, you have to do this manually, and not to forget to change it back, when you’re finished. But as Raymond Hettinger always says: “There must be a better way”. And a better way there is. Let’s check out two very useful pytest plugins that will make your testing life easier and far more efficient.
pytest-testmon is a pytest plug-in which automatically selects and re-executes only tests affected by recent changes.
Again you can install
pytest-testmon either from PyCharm’s Python Interpreter preferences window
or from terminal using:
(venv)$ pip install pytest-testmon Collecting pytest-testmon Using cached pytest_testmon-1.4.5-py3-none-any.whl (28 kB) ...
After successful installation create
pytest.ini in the project’s root directory to instruct
pytest to always use
testmon plugin and we are good to go:
[pytest] addopts = --testmon
After code change, only tests affected by this change will be selected and executed. In this case running 1 test from 27, saving 26* unaffected from being run (*results may vary)
Another way to level-up your testing skill, is to use pytest-randomly. Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.
(venv)$ pip install pytest-randomly Collecting pytest-randomly Downloading pytest_randomly-3.12.0-py3-none-any.whl (8.5 kB) ...
After successful installation let’s update our
pytest.ini to also use randomly with our testing
[pytest] addopts = --testmon --randomly-seed=last
All affected tests will be shuffled each time they run, which can give us new insights into unwanted (test) dependencies.
In this article, we’ve managed the integration of pytest framework in PyCharm. After setting up an automatic rerun of tests, with the help of two pytest plugins, pytest-testmon and pytest-randomly, we can focus on programming and get rapid feedback from our test cases.
Another nice feature is to execute tests that are being currently developed. We can write the tests and see if everything works as expected. Just with the press of a button, we can get the feedback.