A failing test means one of three things: a genuine product defect, a problem with the test itself, or an environment issue unrelated to either. Treating every red test as a bug to fix (or every red test as noise to ignore) is how teams end up either shipping broken software or burning hours chasing ghosts. The right move is always diagnosis first, repair second, and that sequencing is the single most overlooked discipline in test automation today.
- Flaky tests are common enough that Microsoft attributes roughly 25% of CI failures to them, with some studies putting the figure as high as 50-56% of failures before remediation.
- A failure must be classified into one of three buckets before anyone touches code: application defect, test issue, or environment error.
- Most flakiness traces to a small set of root causes: timing and synchronization problems, shared state between tests, environment drift, and unreliable external dependencies.
- Fixing the test before diagnosing it risks masking a real defect, the opposite of what self-healing tools should do.
- Tools like Amikoo build failure classification into every pull request, flagging whether a red test is a product bug, a test issue, or an environment problem before any repair happens, so developers aren't the ones triaging every red build.
A flaky test is one that produces different results (pass, then fail, then pass again) without any change to the underlying code [testdevlab.com]. That definition matters because it's a negative one: flakiness is defined by the absence of a real cause tied to the code change under test, not by any specific symptom. This is precisely why flaky failures are dangerous. They look identical to real bugs on the surface; both show up as a red X in your CI dashboard.
The scale of the problem is bigger than most teams assume. Microsoft has reported that flaky tests account for roughly 25% of CI failures, and other industry studies put the number as high as 50% to over 56% of test failures before any remediation work is done. If a quarter to half of your "failures" aren't real, then a diagnosis step isn't optional overhead, it's the thing standing between your team and either shipping a real bug or wasting a sprint on a phantom one.
Building on the scale problem above, the harder question is what happens when teams skip diagnosis entirely. If a test is rewritten, retried, or quarantined without first confirming whether the underlying application behavior is actually correct, two bad outcomes become equally likely: a real product defect gets paved over by a "fixed" test, or a perfectly good test gets permanently disabled because nobody traced the real root cause.
This is the core design principle behind self-healing tests done correctly. Self-healing should mean a system that adapts a test to legitimate application changes (a button moved, a selector changed, a new field appeared), not a system that silently makes failures go away. A repair agent that patches a test without first confirming the failure wasn't caused by a real regression is solving the wrong problem. The Amikoo agent is built around this exact sequencing: it confirms whether a failure represents an actual product defect before any auto-fix runs, filing confirmed defects as GitHub issues with Slack notifications so a human sees the real bug rather than a quietly "healed" test hiding it.
A related but distinct question is where flakiness actually comes from, because the root cause determines the fix. Testing frameworks and industry standards converge on four recurring categories:
- Timing and synchronization issues: the test checks for an element or state before the application has actually rendered or updated it.
- Shared state pollution: one test leaves behind data, cookies, or session state that affects a later test run.
- Environment inconsistencies: differences between local, staging, and CI environments (network latency, resource limits, parallel execution) cause behavior to vary.
- Unreliable external dependencies: third-party APIs, test databases, or services that aren't fully mocked or isolated.
Think of it like diagnosing a car that won't start. A mechanic doesn't just replace the battery because the car failed to start once; they check whether it's the battery, the starter motor, or a loose connection, because replacing the wrong part wastes money and doesn't fix the problem. Each of these four flakiness categories needs a different repair, so the diagnosis step is what tells you which "part" to actually replace.
Given those four root-cause categories, the practical question becomes: what does a diagnosis workflow actually look like? Several best practices show up consistently across testing frameworks and documentation:
- Rerun before you react. Running the test multiple times is one of the fastest ways to distinguish a flaky test from a real regression; most CI systems support this natively [sdtimes.com].
- Use automated retries, but track them. Retries separate temporary instability from genuine failures, but only if you're logging retry metrics and watching for patterns rather than letting retries mask a recurring issue.
- Replace brittle waits with web-first assertions. Hardcoded sleeps and brittle CSS selectors are a leading cause of timing-related flakiness; auto-waiting and role-based locators make test execution deterministic instead of a guessing game.
- Isolate shared state. Each test should set up and tear down its own data rather than relying on state left behind by a previous run.
- Quarantine, don't delete. A test suspected of being flaky should be moved out of the main pipeline for investigation, not deleted or permanently disabled, so the signal isn't lost [blog.testunity.com].
The cost of skipping this process is measurable, not abstract. Industry research estimates developers spend between 1.28% and 8% of their working time repairing flaky tests, with each individual investigation taking 5 to 15 minutes. At the scale of a 50-engineer team, that adds up to roughly $120,000 in lost productivity annually, purely from chasing failures that may not even represent real bugs.
Stepping back from the individual diagnostic techniques, a separate concern is where this classification work should live operationally. In a mature devops test automation setup, failure classification shouldn't be a manual triage ritual a QA lead does every Monday morning; it should happen automatically, on every pull request, before a human is even pulled in.
This is where the architecture of your end-to-end testing tools matters more than most teams realize. A platform that only reports pass/fail gives you a wall of red with no diagnosis attached. A platform built to classify failures, by application defect, test issue, or environment error, gives engineering teams a starting point instead of a mystery. Amikoo's reporting layer is built around exactly this distinction, surfacing release health in terms of what's actually broken rather than just what's red, so the developer who opened the PR, or the QA lead on teams that have one, isn't manually re-deriving the root cause every time CI goes sideways.
A related but distinct question worth addressing directly: does switching frameworks actually reduce flakiness, or just change where it shows up? The honest answer is that framework choice affects how much flakiness is baked in by default, even if it can't eliminate the root causes above.
| Comparison | Key Difference |
|---|---|
| Playwright vs Selenium | Playwright includes built-in auto-waiting and web-first assertions, reducing the timing-related flakiness that Selenium scripts often need manual waits to handle. |
| Playwright vs Cypress | Playwright supports multiple browser engines and true parallel test execution across tabs and contexts, which Cypress historically handles with more constraints. |
This is part of why migrations are common: a Selenium to Playwright migration or a Cypress to Playwright migration often isn't about chasing a trend, it's about inheriting fewer timing-related flaky failures by default. Amikoo supports both migration paths directly, converting existing suites into standard Playwright TypeScript code that's committed to the customer's own repository, so there's no proprietary runtime and no vendor lock-in if the team's tooling changes down the line.
Given everything above, the practical takeaway for anyone evaluating ai test automation or self-healing tests is a simple filter question: does the tool diagnose before it repairs, or does it just repair? A tool that auto-patches a broken selector without checking whether the underlying page actually broke is optimizing for a green dashboard, not for product quality.
In internal benchmarking, the Amikoo agent achieved roughly double the repair success rate of Playwright's native test healer (70% vs 34%) while completing repair cycles about five times faster (5 minutes vs 25 minutes) and using around 28% fewer tokens across fewer iterations per test (4 vs 7). Those numbers matter less as a competitive scoreboard and more as evidence that diagnosis-first repair doesn't have to be slower than reckless auto-patching, it can be faster precisely because it isn't repeatedly guessing at the wrong fix.
What's the fastest way to tell if a failure is flaky or real?
Rerun the test in isolation two or three times. If the result changes without any code change, it's likely flaky; if it fails consistently, treat it as a real defect until proven otherwise.
Should flaky tests be deleted from the suite?
No. Quarantine them into a separate pipeline for investigation rather than deleting them, so you don't lose coverage of a real user flow.
Can self-healing tests hide real bugs?
Only if the healing happens before diagnosis. A properly built self-healing system confirms whether a failure is a product defect first, then only repairs the test if the application behavior is actually correct.
Is Playwright inherently less flaky than Selenium or Cypress?
It reduces timing-related flakiness through auto-waiting and web-first assertions, but it doesn't eliminate flakiness caused by shared state, environment drift, or unreliable dependencies.
How much developer time do flaky tests actually cost?
Estimates range from 1.28% to 8% of working time, with each investigation taking 5 to 15 minutes, which can add up to roughly $120,000 a year in lost productivity for a 50-engineer team.
How quickly can a team get this kind of classification running?
Developer setup and handoff with Amikoo typically takes around 30 minutes, focusing on repository connection, not the time it takes a full suite to run. After that, Amikoo classifies and handles failures on every pull request without the developer owning the process.
Amikoo is an AI agent by MuukLabs that owns end-to-end verification for web applications. Developers connect their app and repository in about 30 minutes. From then on, on every pull request, Amikoo understands the change, determines which user flows need verification, creates or updates standard Playwright TypeScript tests in the team's own repository, runs them, diagnoses each failure as a product defect, test issue, or environment issue, and reports real defects. It's built for fast-shipping teams, often using coding agents like Claude Code or Cursor, where E2E testing has fallen on developers and the current workflow can't keep up. Amikoo complements coding agents rather than replacing them: AI writes and reviews the code, Amikoo verifies the product actually works. It connects to GitHub and Azure DevOps, with Bitbucket and GitLab support in progress, with no proprietary runtime or lock-in. MuukLabs also offers MuukTest, an expert-backed services brand under the same parent company, for teams that want hands-on QA support alongside the platform.
If flaky failures and unclear defect triage are slowing down your release cycle, get in touch with Amikoo to see how automated failure classification fits into your pipeline.
- The best ways to deal with flaky tests - SD Times (sdtimes.com)
- What Are Flaky Tests and How to Fix Them? (testdevlab.com)
- Fix Flaky Tests: First Steps to Restore Confidence (2026) (blog.testunity.com)