The modern CI services (GitHub Actions, GitLab Pipelines, etc.) model is very unsatisfactory. They are:
I see clear overlap between automated QA, build systems, CI systems, and the broader OS/platform build-and-execute layer. The logical slicing and composition of these concerns should look very different from today’s model, which revolves around two major players locking everybody into their proprietary systems, leaving universality, simplicity, and developer experience as afterthoughts. This makes me hesitant to invest much into these platforms.
There is another reason I haven’t invested time in testing, documenting, and maintaining an official GitHub/GitLab guide for testmon. We’re developing testmon.net, a hosted service that makes it easy to run testmon in CI. It understands different CI environments and runners and maintains a centralized database of dependencies and test results. That’s far more effective than sharing or caching .testmondata files across CI runners. We expect to charge a modest fee, capturing a small fraction of the CI capacity savings and funneling it back into testmon development. Registration is open and free; if you’d like to try it or have any questions, email me at tibor@testmon.org.
pytest-testmon doesn’t have any CI-specific behavior—it runs the same locally and in CI. Ideally, if you suspect a bug in CI, reproduce it locally and report it. That said, we won’t close GHA/GL issues automatically, and we’ll fix CI-specific bugs when we can.
We’re aware of issue #255 which may be causing problems for some users running testmon in CI. This bug relates to how testmon compares system packages between runs. If you’ve been experiencing issues with testmon in CI, particularly if it’s running all tests when it shouldn’t, please test again with release v2.1.4 or later.