How to make your GitHub Actions builds quicker?

For many projects creating the Python environment and installing dependencies takes a lot of time. For local development environment this happens occasionally and is usually not a big problem. However, in your CI, this happens for every build and might be a significant part of the build time. GH Actions does have a caching mechanism which is used in various ways.

actions/setup-python already does some caching to avoid downloading with the cache input:

- uses: actions/setup-python@v4
  with:
    python-version: 3.9
    cache: pip

From its documentation it’s easy to get the impression that the packages are cached in their installed form and that they magically appear in your Python environment when there is a cache hit. However, this is not the case. This action only caches the downloading of the packages from PYPI. pip itself caches the downloads by default on the system level and this action allows you to use the functionality in the GHA environment where workers start from scratch each time by default. Arguably, you’re just replacing the download from PYPI with the download from GitHub Actions cache. There is quite a discussion about this in the actions/setup-python repository

The other slow steps of pip installing a lot of dependencies are: resolving dependencies, unzipping and moving them to the right locations within the Python environment and compiling extensions if needed. actions/setup-python@v4 does not cache any of these steps.

The good news is that it’s possible to use and cache the whole virtual environment in your GitHub Action workflows.

It is possible to craft this manually using actions/cache, however it’s quite error-prone, and you have to be careful to get everything right. Luckily, the syphar/restore-virtualenv custom action does all the heavy lifting for you.

restore-virtualenv is a simple 1-liner that

  • gives you either a new virtualenv, or restores an old one based on the requirements-file.
  • works on Ubuntu, MacOS and Windows
  • sets $VIRTUAL_ENV and $PATH environment for the virtualenv.
  • restore keys take the OS into account, and the major and minor Python version. for patch-updates the virtualenv can be reused.
  • will use any typical requirements file to build the cache key (poetry, pipenv, pip-requirements-txt)

In its simplest form you have to put this after your setup-python and before your pip install:

- uses: syphar/restore-virtualenv@v1

You can also modify your pip install command to avoid running it altogether if the virtualenv is restored:

- run: pip install -r requirements.txt
  if: steps.restore-virtualenv.outputs.cache-hit != 'true'

We’ve been using this technique for the development of testmon.net and we never had any corrupted virtual environment or any other problem.

To learn about more options how to use this action, head to its documentation
syphar/restore-virtualenv