A CI test result taxonomy is a standardized way to classify automated test outcomes in CI/CD pipelines. It helps engineering teams improve visibility, debugging speed, and release confidence by ensuring consistent interpretation of test results.
What Is a CI Test Result Taxonomy?
A test result taxonomy defines a shared language for categorizing CI test outcomes. Instead of treating all failures equally, teams classify results based on behavior, reproducibility, and impact.
Why CI Test Result Taxonomy Matters
- Improves debugging efficiency
- Reduces noise in CI pipelines
- Enables better release decisions
- Clarifies ownership of failures
- Helps track reliability trends over time
Recommended CI Test Result Categories
Passed
Test executed successfully and verified expected behavior.
Failed
Deterministic failure indicating a reproducible issue in application or test logic.
Flaky
Non-deterministic result where the same test alternates between pass and fail without code changes.
Blocked
Test could not execute due to external dependency or environment issue.
Skipped
Test intentionally excluded from execution due to configuration or feature state.
Quarantined
Known unstable test temporarily isolated from release decisions.
Example CI Test Taxonomy Table
| Category | Description | Impact on Release |
|---|---|---|
| Passed | Test completed successfully | No impact |
| Failed | Reproducible defect detected | High |
| Flaky | Inconsistent execution result | Medium |
| Blocked | Execution prevented by environment | Medium |
| Skipped | Test intentionally not run | Low / Visibility risk |
| Quarantined | Isolated unstable test | Controlled |
How to Implement a Test Result Taxonomy
- Define standard categories across all teams
- Enforce consistent CI/CD labeling
- Automate classification where possible
- Track trends over time
- Assign ownership for each category
Best Practices
- Never mix flaky and failed results
- Track blocked tests separately from failures
- Review quarantined tests regularly
- Use taxonomy data for release decisions
Frequently Asked Questions
Is a flaky test a failure?
No. Flaky tests indicate instability rather than confirmed defects.
Should quarantined tests run in CI?
Yes, but they should be excluded from release gating metrics.
Why not treat all failures the same?
Different failure types require different actions and ownership.