pytest_flakefighters.flakefighters.traceback_matching

This module implements three FlakeFighters based on failure de-duplication from Alshammari et. al. [https://arxiv.org/pdf/2401.15788].

Module Contents

Classes

TracebackMatching	Simple text-based matching classifier from Section II.A of
CosineSimilarity	TF-IDF cosine similarity matching classifier from Section II.C of

class pytest_flakefighters.flakefighters.traceback_matching.TracebackMatching(run_live: bool, previous_runs: list[pytest_flakefighters.database_management.Run], root: str = '.')

Bases: pytest_flakefighters.flakefighters.abstract_flakefighter.FlakeFighter

Simple text-based matching classifier from Section II.A of Alshammari et al. (2024). We implement text-based matching on the failure logs for each test. Each failure log is represented by its failure exception and stacktrace.

Variables:

run_live – Run detection “live” after each test. Otherwise run as a postprocessing step after the test suite.

classmethod from_config(config: dict)

Factory method to create a new instance from a pytest configuration.

params()

Convert the key parameters into a dictionary so that the object can be replicated. :return A dictionary of the parameters used to create the object.

_flaky_execution(execution, previous_executions) → bool

Classify an execution as flaky if any of its failing executions has a traceback that matches a test previously classed as flaky. :return: Boolean True if the test is classed as flaky and False otherwise.

previous_flaky_executions(runs: list[pytest_flakefighters.database_management.Run]) → list

Extract the relevant information from previous flaky executions and collapse into a single list. :param runs: The runs to consider. Defaults to self.previous_runs. :return: List containing the relative path, line number, column number, and code statement of all previous test executions.

flaky_test_live(execution: pytest_flakefighters.database_management.TestExecution, previous_runs: list[pytest_flakefighters.database_management.Run] = None)

Classify executions as flaky if they have the same failure logs as a flaky execution. :param execution: Test execution to consider. :param previous_runs: The previous runs to which the execution will be compared.

flaky_tests_post(run: pytest_flakefighters.database_management.Run) → list[bool | None]

Classify failing executions as flaky if any if their executions are flaky. :param run: Run object representing the pytest run, with tests accessible through run.tests.

class pytest_flakefighters.flakefighters.traceback_matching.CosineSimilarity(run_live: bool, previous_runs: list[pytest_flakefighters.database_management.Run], root: str = '.', threshold: float = 1)

Bases: TracebackMatching

TF-IDF cosine similarity matching classifier from Section II.C of Alshammari et al. (2024). Test executions are classified as flaky if the stack trace is sufficiently similar to a previous flaky execution.

Variables:

• run_live – Run detection “live” after each test. Otherwise run as a postprocessing step after the test suite.

• previous_runs – List of previous FlakeFighters runs.

• root – The root directory of the code repository.

• threshold – The minimum distance to consider as “similar”, expressed as a proportion 0 <= threshold < 1 where 0 represents no difference and 1 represents complete difference.

classmethod from_config(config: dict)

Factory method to create a new instance from a pytest configuration.

params()

Convert the key parameters into a dictionary so that the object can be replicated. :return A dictionary of the parameters used to create the object.

_tf_idf_matrix(executions)

_flaky_execution(execution, previous_executions) → bool

Classify an execution as flaky if the test execution is sufficiently cosine-similar to any of the previous executions. :return: Boolean True if the test is classed as flaky and False otherwise.