Escaping Flaky Test Hell | 5 Root Cause Categories, Diagnosis, and Prevention

A practical guide to escaping flaky test hell — covering root cause classification, diagnosis, prioritized fixes, and prevention. From Wait design and environment differences to data conflicts and external dependencies, learn how to address each type of flaky test systematically based on real QA engineering experience.

The real danger of flaky tests is not that they fail — it’s that people stop trusting the test results entirely.

“Flaky again.” When that phrase becomes routine, test automation has stopped working. CI turns red, someone hits re-run, it passes, and everyone moves on — never knowing whether the failure was a real bug or just noise. Real bugs get missed. Eventually, nobody trusts the test results at all.

This article is about escaping that state — not by “somehow fixing” flaky tests, but by classifying the root cause and applying the right fix systematically.

What Is a Flaky Test and Why Is It Dangerous?

A flaky test is a test that produces inconsistent results — passing sometimes and failing other times — without any change to the code or environment. At least, that’s how it appears on the surface.

The danger isn’t the failure itself. It’s what happens to the team over time.

The 5 Root Cause Categories of Flaky Tests

Treating flakiness as “just unstable” without categorizing it means you’ll never fully fix it. Identifying which category your flakiness belongs to is the essential first step.

Diagnose First: Which Category Is Your Flakiness?

Use this table to identify the likely category from the symptom — then jump directly to the matching STEP.

STEP 1: Measure and Visualize Flakiness First

“Count before you fix.” Establishing flakiness as a data problem — not a feeling — is what lets you prioritize where to start.

3 Metrics to Track

# Run the same test multiple times to measure flakiness rate
# ⚠️ This requires a plugin — NOT a pytest built-in
# pip install pytest-repeat
#
# Note: adds execution time — recommended for investigation only, not regular CI

pytest tests/test_login.py --count=5

# Example output: 3 passed, 2 failed → flakiness rate = 40% (address immediately)

# To visualize over time, save results in JUnit XML:
# pytest --junitxml=results.xml
# Then import into Datadog / Grafana / Allure Report / ReportPortal for trend tracking

STEP 2: Fix ① Wait-Related Flakiness

The most common and relatively most fixable type of flakiness (though SPA, virtual DOM, and microfrontend environments make it harder). “Element not found” and “cannot click” both trace back to not correctly waiting for an element to be visible and interactive.

Selenium

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# Note: the examples below assume `driver` is already initialized
# In real tests, create it via a pytest fixture or setUp()

# ❌ Flaky: time.sleep is a fixed wait — element may not be ready
import time
time.sleep(3)
driver.find_element(By.ID, "submit").click()

# ❌ Flaky: mixing implicitly_wait and WebDriverWait — interference causes unpredictable behavior
driver.implicitly_wait(10)           # global setting
wait = WebDriverWait(driver, 5)      # local setting — they interfere

# ✅ Recommended: WebDriverWait for "clickable" state
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, "submit")))
element.click()

# ✅ For buttons that transition disabled → enabled asynchronously
# Note: requires the `wait` object defined above
wait.until(lambda d: not d.find_element(By.ID, "submit").get_attribute("disabled"))

Playwright

from playwright.sync_api import expect

# Playwright runs an auto-waiting check (visible / stable / enabled / etc.)
# before click() — but this checks the CURRENT state, not async state changes.
# For things that change asynchronously, combine with expect().

# ❌ Over-relying on auto-waiting: disabled → enabled transition may time out
page.locator("#submit").click()

# ✅ Use expect() to confirm enabled state before clicking
expect(page.locator("#submit")).to_be_enabled()
page.locator("#submit").click()

# ✅ Content that appears after an API response
expect(page.locator(".search-results")).to_be_visible()
# waits until the API response returns and the list renders

STEP 3: Fix ② Environment-Dependent Flakiness

“Passes locally, fails in CI” — viewport size, CPU speed, timezone, headless mode differences. The root cause is the test assuming a specific execution environment.

Main Causes and Fixes

# ✅ Selenium headless config that minimizes environment differences
from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")      # pin viewport
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")                 # stability for Linux CI
options.add_argument("--disable-dev-shm-usage")      # prevent memory issues
driver = webdriver.Chrome(options=options)

# ✅ CI timeout — set higher than local (in pytest.ini or playwright.config.ts)
# timeout = 30000  # ms — tune based on your CI machine's performance

STEP 4: Fix ③ Data Conflict Flakiness

“Only fails in parallel” or “changes when test order changes” — this means tests are sharing data or state. High fix difficulty, but leaving it unresolved means permanent recurrence.

import pytest
import uuid

# ❌ Flaky: tests share static test data
# test_a.py
def test_create_user():
    create_user("test@example.com")   # fixed test data

# test_b.py (conflicts with test_a when run in parallel)
def test_delete_user():
    delete_user("test@example.com")   # same data — race condition

# ✅ Fix: generate unique data per test
@pytest.fixture
def unique_user():
    """Creates a unique user per test and deletes it after."""
    email = f"test_{uuid.uuid4().hex[:8]}@example.com"
    user = create_user(email)
    yield user
    try:
        delete_user(user.id)
    except Exception as e:
        # Cleanup failures should not affect the test result
        # In production, use logger.warning(f"cleanup failed: {e}") instead
        # Note: broad Exception catch is acceptable here for cleanup only —
        # avoid this pattern in regular test logic
        pass

def test_update_user(unique_user):
    # Only touches the unique user — no conflict possible
    update_user(unique_user.id, name="Updated")

STEP 5: Fix ④ External Dependency Flakiness

Slow external APIs, unstable networks, third-party maintenance windows — flakiness caused by factors outside your codebase.

import pytest
import requests
from unittest.mock import patch, MagicMock

# ❌ Flaky: hitting a real external API
def test_user_profile():
    response = requests.get("https://api.example.com/users/1")
    # Slow or unavailable API → flaky
    assert response.status_code == 200

# ✅ Fix option 1: mock the external API
@patch("requests.get")
def test_user_profile_mocked(mock_get):
    mock_get.return_value = MagicMock(
        status_code=200,
        json=lambda: {"id": 1, "name": "Test User"}
    )
    response = requests.get("https://api.example.com/users/1")
    assert response.status_code == 200
    # No external dependency → stable

# ✅ Fix option 2: add retry when external dependency is unavoidable
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=4))
def call_external_api():
    return requests.get("https://api.example.com/users/1", timeout=(3, 10))

STEP 6: Fix ⑤ Design-Caused Flakiness

“Fails when test order changes” or “only fails after a specific other test runs” — design problems require a design fix. The highest difficulty category, requiring structural rethinking.

Common design-caused flakiness patterns

• Implicit prerequisite dependency: Test B assumes test A ran first and left a specific state
• State-modifying session-scope fixture: scope="session" is not bad by itself — it’s safe for read-only data, immutable config, or browser binary caching. The problem is using it for fixtures that modify state, which unintentionally shares state between tests
• Oversized E2E tests: Packing “login → search → purchase → payment” into one test means a mid-flow failure cascades to everything after

import pytest

# ❌ Design-caused flakiness: state-modifying fixture with scope="session"
# Note: scope="session" itself is not the problem —
# the problem is using it for state-changing operations (like login) without cleanup
@pytest.fixture(scope="session")
def logged_in_driver(driver):
    driver.get("/login")
    driver.find_element(By.ID, "email").send_keys("admin@example.com")
    driver.find_element(By.ID, "submit").click()
    yield driver
    # No logout → session state bleeds into the next test

# ✅ Fix: complete setup and teardown within each test
@pytest.fixture
def logged_in_driver(driver):
    """Login and logout independently for every test."""
    driver.get("/login")
    driver.find_element(By.ID, "email").send_keys("admin@example.com")
    driver.find_element(By.ID, "submit").click()
    yield driver
    driver.get("/logout")  # always clean up

STEP 7: Quarantine or Delete Tests That Won’t Stay Fixed

When a flaky test “won’t fix” or the fix cost far exceeds the value, quarantining or deleting the test is the right call. However: deletion reduces coverage and weakens regression detection for that scenario. Only delete when you have an alternative — manual testing or a smaller, more targeted unit test — covering the same scenario.

import pytest

# ✅ Option 1: @pytest.mark.skip — temporary quarantine with a paper trail
@pytest.mark.skip(reason="Flaky: under investigation (2026-05-01 by Yoshi) #issue-123")
def test_payment_flow():
    pass

# ✅ Option 2: @pytest.mark.xfail — mark as "expected to fail"
@pytest.mark.xfail(reason="External payment API unstable - #issue-456", strict=False)
def test_external_payment():
    pass

# ✅ Option 3: custom flaky mark — isolate to a separate CI stage
# In pytest.ini:
# [pytest]
# markers =
#     flaky: unstable tests (excluded from main CI)

@pytest.mark.flaky
def test_unstable_feature():
    pass

# Main CI: exclude flaky tests
# pytest -m "not flaky"
#
# Separate scheduled job: run and monitor flaky tests
# pytest -m "flaky"

Preventing Recurrence: Design Rules That Keep Flakiness Out

Fixing existing flakiness is only half the job. Design rules that prevent it from coming back are what make the escape permanent.

FAQ

A system that “keeps fixing” flaky tests is more expensive long-term than one designed to prevent them in the first place. Measure first. Classify the cause. Fix in priority order. That’s the sequence that makes the escape from flaky test hell permanent.