CI/CD Integration

The TestFLARE project is motivated by the fact that flaky tests can be particularly problematic for CI/CD pipelines. The flakefighers extension is therefore developed to integrate seamlessly with your existing pipelines and testing workflows. If you already have a workflow that uses pytest, all you need to do is include pytest-flakefighers as a dependency. The extension will then automatically be run as part of pytest.

Remote Databases

When you run pytest, the flakefighers extension automatically saves the details of each run, including the classification result from each active flakefighter for each test. This historical data can then be used on subsequent runs to inform flaky classification. By default, this data is stored locally in an SQlite database called flakefighters.db. However, this is configurable using the --database-url parameter, so you are free to save your data wherever you like.

If you are using the extension as part of your CI/CD pipeline and want to preserve the data between runs, you will need to use external service such as Supabase to host your database. Fortunately, this is very straightforward as these services make make setup and connection very easy. All you need to do is set up the database using your service of choice and pass the database URL into the --database-url parameter. You don’t even need to create the tables yourself, as this will be done by the extension automatically on the first run! Of course, for security reasons, we suggest making your URL a secret rather than including the raw string in your github workflow.

Tip

If you don’t need to preserve the data between CI/CD runs, but do want to inspect it, you can include it as an artifact.

Hybrid Databases

There may be situations where you sometimes want to store data locally and sometimes want to store it remotely. For example, you may want your CI/CD runs to go into the remote database hosted externally, but your local runs to use a local database so that you don’t clog up the remote database during debugging. Depending on your workflow, you may even want to keep a local clone of your database and occasionally synchronise this with the remote database. Fortunately, database providers such as Supabase make this a relatively straightforward process.

Warning

As with any such setting, synchronisation may result in conflicts or inconsistent behaviour if entries have been made into the remote database after you have cloned your local copy and before you have resynchronised. In particular, the results of flakefighters that examine historical runs can be confusing if historical runs that were performed before a particular pytest run were not added until afterwards.