The Flaky Test Crisis: Why Trunk, Datadog, and Meta Are Racing to Automate Flaky Test Detection

Every QA engineer knows the feeling: a pull request fails a check, you re-run the pipeline with no code changes, and suddenly it's green. That's a flaky test — a test that produces inconsistent results without any change to the code under test — and it is quietly one of the most expensive problems in modern software delivery. As test suites grow larger and CI pipelines run thousands of times a day, flakiness has moved from an annoyance to a first-class engineering problem, and a new wave of tooling is emerging to tackle it head-on.

Why Flaky Tests Are Back in the Spotlight

Flaky tests are not new, but the cost of ignoring them has grown with the scale of continuous integration. Every re-run burns compute minutes and developer patience, and every "just re-run it" click trains engineers to distrust the test suite — the exact opposite of what automated testing is supposed to deliver. As teams push more code through CI/CD pipelines faster than ever, even a small flake rate compounds into hours of lost velocity and, worse, a culture where failing builds are ignored rather than investigated.

Trunk.io: Quarantine as a First-Class CI Feature

Trunk.io has built flaky test detection directly into its CI product, automatically flagging tests with inconsistent pass/fail history across commits and quarantining them so they no longer block merges while still being tracked for a fix. Instead of engineers manually hunting through CI logs to spot patterns, the platform surfaces flakiness trends per test, per author, and per repository, turning an ad-hoc detective exercise into a dashboard-driven workflow.

Datadog's Test Optimization Brings Observability to Test Suites

Datadog has extended its observability roots into the testing world through its Test Optimization (formerly CI Visibility) product line, which includes flaky test management capabilities that correlate test failures with infrastructure signals, git blame data, and historical run patterns. The pitch is familiar to anyone who has used Datadog for production monitoring: apply the same telemetry-driven approach that diagnoses a flaky microservice to diagnose a flaky test.

Meta's Probabilistic Approach at Massive Scale

At the scale of a company like Meta, manual flaky test triage simply isn't feasible across millions of test executions per day. Meta's engineering teams have published research and internal tooling describing probabilistic models that score tests for flakiness likelihood based on historical re-run outcomes, automatically deprioritizing or quarantining the worst offenders before they can block thousands of engineers' commits. These internal systems have influenced how the broader industry, including open-source projects and smaller CI vendors, thinks about flaky test remediation.

What This Means for QA Teams

• Stop relying on manual re-runs. If your team's answer to a red build is "just re-run the pipeline," you already have a flaky test problem worth measuring.
• Track flakiness as a metric. Treat flake rate the same way you'd treat code coverage or defect escape rate — something visible on a dashboard, not something discovered by accident.
• Quarantine, don't ignore. Automated quarantine keeps unreliable tests from blocking releases while ensuring they remain on someone's backlog rather than disappearing entirely.
• Evaluate CI-native tooling. Whether it's Trunk, Datadog, or a homegrown script, the trend is clear: flaky test detection is moving from a manual chore to a built-in platform capability.

Flaky tests won't disappear, but the tooling to manage them is maturing fast. For QA and platform engineering teams evaluating their 2026 roadmap, investing in flaky test observability may deliver a faster return than adding yet another layer of test automation on top of an already-unreliable suite.