General-purpose LLMs and coding agents like Claude Code handle your first 20–50 end-to-end test cases well. As your application grows more complex and your suite grows into the hundreds, three problems emerge: poor context management (which can burn 20x more tokens than necessary), test drift (tests that quietly "fix" themselves into false positives), and multi-agent communication failures. Each has a concrete fix, covered below.
That was the core of this Amikoo webinar, where MuukLabs CTO and co-founder Renan Ugalde, hosted by CEO and co-founder Ivan Barajas, dug into what teams actually experience when they use LLMs for end-to-end test automation. Writing test scripts is no longer the hard part; the challenge has shifted to managing context, drift, and agent coordination at scale.
Unit test generation with LLMs is essentially solved; end-to-end is not. E2E tests generated by plain LLMs often miss expected assertions or stay flaky as suites grow.
More context is not better context. The same six-step test consumed ~80K tokens with a "send everything" approach vs. ~4K with scoped context (a nearly 20x difference per failure).
Test drift creates false confidence. An LLM asked to "fix" a failing test may weaken the assertion until it passes, hiding real defects. False confidence is worse than no test at all.
Safeguard agents are the fix for drift. Every self-heal should be validated against the original intent of the test.
At scale, agents fail to communicate. Multi-agent testing systems degrade when agents operate in silos and can't combine information from different sources.
Renan opened with a pattern the industry has seen before. Selenium made web testing possible, but keeping it stable required real engineering skill. Playwright arrived faster and more reliably, and the problem moved up a layer, to environment reliability and test data once you were running a thousand tests.
AI testing is repeating that pattern. Teams tell us they're writing ten times more code with AI, and unit test generation is plug-and-play: tell the LLM which function to cover, done. But end-to-end is different. LLM-generated E2E tests miss the assertions teams expected, or stay just as flaky as hand-written ones. The bottleneck was never really "writing the script." It was — and still is — building a scalable, reliable framework you don't spend your life maintaining.
Send only what's relevant to the failure.
When a test fails, the developer instinct is to hand the LLM everything: the full HTML tree, every screenshot, the complete Playwright trace, console logs, network logs, execution history. The assumption is that more context means better answers. It doesn't. Dump everything, and the model struggles to find what matters, and you burn an enormous number of tokens doing it.
Renan showed a side-by-side comparison of the same six-step test case (load customers, find a user, open edit, fill in new data, save, refresh) run with poor context management versus a well-scoped approach. The first action alone consumed roughly 80K tokens with the "send everything" approach vs. 4K tokens with scoped context. That's a 18.4x difference on a single step, compounding across every failure in your suite.
The fixes:
Scope the dump to the failure. Testing a form? Send the elements around the form, not the entire page tree.
Reuse sessions. Use Playwright's storageState (or cookies) instead of logging in and out and rebuilding accessibility trees on every run.
Send diffs, not snapshots. Keep a baseline and share what changed, not the full history of logs and screenshots.
Test drift is when an LLM "fixes" a failing test by changing what the test checks, instead of investigating why it failed. The test goes green, but it no longer catches the defect it was written for.
"A flaky test that goes red is annoying. A test that drifts to green is dangerous. False confidence is worse than no test at all." - Renan Ugalde, CTO of MuukLabs
Drift comes in two forms:
1. Context drift happens when the LLM optimizes for the wrong goal: making the test pass instead of finding defects. Renan's example: an assertion checking an order total of $129.99 fails, possibly just a data issue, not an application bug. The LLM "fixes" it by weakening the assertion to "the field is visible." Now the total could be $0, and your suite stays green. You've introduced a blind spot, and this is exactly why developers lose trust in E2E suites.
2. Objective drift happens over long multi-step tasks: the agent gradually forgets the original intent of the test. Ask it to validate your critical flows across a sequence of steps, and midway through it's fixated on execution history and no longer answering the original question.
The fix is safeguards: dedicated agents whose job is to hold the line on test intent. Every self-heal should be checked against the original purpose of the test: do the new assertions actually assert the same thing? If they don't, the "fix" gets rejected before it reaches your suite.
Because agents start operating in silos, drawing conclusions without the data other agents collected.
Modern AI testing setups involve multiple agents. At small scale, this works fine. At larger scale, agents fail to combine information from different sources and can even disagree outright. Your system's conclusions degrade exactly when the suite gets big enough to matter.
And these are the easy challenges. Beyond them sit harder ones: scaling from dozens to hundreds or thousands of tests, and complex test repair - where plain LLMs reliably start breaking partway through longer test cases because they can't keep up with the context.
If you're early in AI-powered testing, start simple: documenting test cases, generating scripts, pulling data from your repo. Any LLM handles that well. But when you're ready to scale — connecting tests to Jira stories, tracing failures back to lines of code, repairing hundreds of tests without drift — that's where purpose-built QA agents earn their keep.
Amikoo is a purpose-built AI QA agent system that analyzes your codebase, user flows, and test reports to find coverage gaps and generate tests in Playwright TypeScript (readable code your team owns), with guardrails against drift and the context problems covered above.
Watch the full webinar above, and try Amikoo free at amikoo.ai.
General-purpose LLMs and coding agents handle the first 20–50 end-to-end test cases well, but as application complexity and suite size grow, three problems emerge: poor context management (burning up to 20x more tokens than necessary), test drift (tests weakened into false positives), and multi-agent communication failures. Each requires dedicated safeguards to fix.
Test drift is when an LLM "fixes" a failing test by changing what the test checks (for example, weakening an assertion on an order total to a simple visibility check) instead of investigating the failure. The test passes, but it no longer catches the defect it was written for, creating false confidence.
At 20–50 test cases, a good sign is tests that pass and actually find defects, not just tests that keep getting fixed. Past a hundred test cases, evaluate how you're managing context, how your agents communicate, and how much time you spend creating and maintaining scripts.
Don't expect one-to-one behavior - Selenium and Playwright use different protocols, so timing differs. Migrating the code is the easy part. Run an agent that executes both suites in parallel and validates they assert the same behavior, not the same steps. Planning this? Amikoo can help with a Selenium-to-Playwright migration.
Test drift happens when the agent fixing a test is also the one deciding whether the fix is valid, with no independent check; "make it pass" quietly replaces "find defects." Amikoo prevents this by using dedicated agents as guardrails: every self-heal is validated against the original intent of the test case, and changes that weaken what the test asserts are rejected. Full context makes repairs smarter; the guardrail agents keep them honest.