A flaky test is a test that passes and fails inconsistently without meaningful application changes. Flaky tests reduce confidence in CI/CD pipelines because teams cannot easily distinguish real regressions from testing noise.
What Is a Flaky Test?
Flaky tests produce inconsistent results:
- Pass on one execution
- Fail on another execution
- No relevant code changes between runs
Most flaky tests are caused by synchronization issues, unstable test environments, shared test data, or unreliable selectors.
Common Symptoms of Flaky Tests
Timing-Related Failures
Common examples include:
- Timeout exceeded
- Element not found
- Application state not fully loaded
- Asynchronous operations completing unpredictably
Environment Instability
- Infrastructure latency
- API instability
- Shared database contamination
- Parallel execution conflicts
Order-Dependent Failures
Some tests pass individually but fail when executed as part of larger suites due to shared state or dependencies.
Data Dependency Issues
Tests relying on mutable or reused test data often produce inconsistent execution results.
How to Diagnose Flaky Tests
Step 1 — Measure Failure Frequency
Track:
- Pass/fail ratio
- Failure recurrence
- Affected environments
- Affected browsers or devices
Step 2 — Categorize Failure Types
Create categories such as:
- Synchronization
- Infrastructure
- Test data
- Selector instability
- Network dependency
Step 3 — Compare Execution Artifacts
Analyze available execution artifacts:
- Screenshots
- Execution logs
- Video recordings
- Timing metrics
- API responses
Step 4 — Re-Run Strategically
Compare results using:
- Isolated execution
- Repeated execution
- Parallel versus sequential runs
Example Flaky Test Classification
| Failure Pattern | Likely Cause |
|---|---|
| Random timeout | Synchronization issue |
| Browser-only failure | Rendering inconsistency |
| Suite-order failure | Shared state contamination |
| Night-only failure | Infrastructure load |
Best Practices for Reducing Flaky Tests
- Use explicit waits instead of fixed sleeps
- Isolate test data
- Avoid shared state
- Use stable selectors
- Reduce external dependencies
- Retry selectively rather than globally
Frequently Asked Questions
Are flaky tests real failures?
Sometimes. Flaky tests may expose legitimate race conditions or unstable application behavior.
Should flaky tests be retried automatically?
Retries can reduce noise temporarily, but root cause analysis is still necessary to improve test reliability.
What percentage of flaky tests is acceptable?
High-performing engineering teams continuously reduce flaky tests rather than accept a fixed threshold.