Why LLM Test Automation Breaks at Scale (and How to Fix It) - Webinar Recap

Webinar:  Why LLM Test Automation Breaks at Scale — Webinar Recap
Speakers: Ivan (CEO & co-founder, MuukLabs) and Renan Ugalde (CTO & co-founder, MuukLabs)

Ivan: Hello, hello. Thank you everyone for joining this webinar from MuukLabs and Amikoo. Today we're happy to have our CTO and co-founder, Renan, talk about what's real in LLM testing — what works, what doesn't — and how we think about using AI for end-to-end testing and automation. This will be an entertaining conversation, and in some parts technical. If you have any questions, we'll take them all at the end. With that, I'll leave you with our CTO and co-founder, Renan Ugalde. Thank you, Renan.

Renan: Thank you, Ivan. As you mentioned, we're going to talk about some technical things related to testing and test automation. Please feel free to ask any questions in the chat or through the LinkedIn link.

We're really excited to see how AI and the different tools out there are improving and making the end-to-end experience much more efficient. Today we're going to talk about the different challenges our customers are facing — because it's very different when you say, "let's start using AI, let's create our first set of test cases." Moving from twenty or fifty test cases to a hundred or even a thousand is a completely different thing. We'll cover the challenges you might face and some solutions you need to take into account when scaling these systems.

As an intro: I'm the CTO here. We built Amikoo — a purpose-built QA agent system with different sub-agents that handle QA automation tasks. If you have questions related to this webinar, or to end-to-end test automation with Playwright in general, please reach out. I'm happy to help.

Renan: So — what's the stigma with AI in the world, specifically for testing? When Claude and OpenAI started creating the first test cases, there was this big feeling that AI was going to solve all test automation problems. No more issues in production, development delivers new features, no bottleneck at all.

In reality, what we're seeing definitely provides value. Development teams are moving faster — when we talk to our customers, they say they're writing at least ten times more code than before. They're developing fast and pushing more code through the pipelines. For unit testing, AI is well adopted: test cases are created and maintained, everything works fine.

But when you go up the stack — integration testing, and especially end-to-end testing — things are not the same. What we're hearing is that unit testing is essentially plug and play: you tell your LLM to build the test case you need for a given function. End-to-end is just different. Yes, you'll get the tests — writing the scripts isn't a problem anymore. But when teams execute those test cases, they notice missing assertions they were expecting, or tests that are just as flaky as they've always been. The problems we faced when Selenium and Playwright were introduced, we're seeing again now with AI.

One of the misconceptions we see is: "No worries, I just need a new agent, a skill for my agent." There's real technical work involved in getting this fixed — it's not that easy to implement for end-to-end.

Some context on Selenium and Playwright, because this is the same problem we saw in the past. Selenium made it easier to test web applications, but you really needed technical skills to make it work — otherwise you'd see flaky tests and exceptions where you definitely needed Java skills, and execution speed was an issue. Then Playwright was released: super fast, lightweight, stable. But the problem moved to environment reliability — things started to break when running a thousand test cases, and maintaining test data became the challenge.

So as you can see, it has always been about architecture — making it scalable and reliable. Writing the script was never the hardest part, and now it's super easy with any LLM. The problem has always been concentrated in how to build a scalable, reliable framework where you don't have to spend all your time maintaining scripts.

Now we're in the AI era, where you want to test at the speed of development. You don't want a bottleneck. You want everything you release tested at the different layers — unit tests for functions, but also end-to-end to make sure all the complicated scenarios and all the different modules working together are validated properly.

If we look at the different areas of AI testing, most customers in the early days target four areas: documenting test cases, creating scripts, and pulling data from the repository. This is very easy — any LLM can do it. If you haven't done it, that should be your first step in moving to AI-powered testing. But honestly, those aren't hard to accomplish. The real value comes from the later ones: real cases connected to your software development cycle. Now you want to pull data from Jira, for instance, and automatically create those cases. And you want all the data generated when executing a script to point back to the user story, or even to the lines of code that made the change. This requires a different set of skills — it's not just writing a test function, it's how you handle context and how you get different agents communicating and working together.

A good example is test repair. Historically, maintaining your test suite has always been a problem. When you have hundreds of test cases failing because a locator changed, if you don't provide the right context — and you don't understand how LLMs work — you'll run into the challenges we're going to discuss.

My advice: start with the simple, straightforward things. But when you want to move to a scalable, reliable mode, you definitely need to take these areas into account. What we're hearing from customers is: when there's a pull request or a commit, they want to create the scripts, run the pipeline, and have any defect reported directly into CI/CD and the GitHub repository — and obviously, detect issues before they're released.

Renan: So what are the challenges? With AI, the challenge isn't making the AI work — we know it works, and with each release from OpenAI, Codex, and Claude, the models are far better than they were a year ago. The challenge now is how AI works with the amount of information we need to provide. For end-to-end testing you need to provide context: lines of code, HTML, files, screenshots, videos. All that context needs to be shared in a way that the LLMs can effectively work with. If you reach the point where you're sharing all the data, you're going to start seeing problems. These are the most common ones we've heard — not the only ones, but the ones we see most.

Challenge 1: Context management. This is the classic example. A test case fails, and as a developer you think: "I'm going to upload all the data I have. Everything. The more context I provide, the better, right?" That's not entirely true for LLMs. If you share the whole HTML dump, screenshots, execution history — everything — you'll hit limitations. If you're running ten or twenty test cases and they fail here and there, maybe don't worry about it. But we're talking to customers who want to build a reliable, scalable framework.

Here's an example of an agent skill with a big problem: every time a test fails, it captures the full outer HTML — the whole tree, every attribute, text, style, everything. Then for the next problem it shares stale information. It keeps all the snapshots, console outputs, everything. The thinking is that the LLM will know what to do with all the data: on failure, provide the full Playwright trace, console log, full network log — all of it. This is bad. You share a huge amount of information the model can't handle well, and you spend a lot of tokens doing it.

The best practices: make sure you're only sharing the dump related to the failure. If you're testing a form, maybe you just need the elements around the form causing the issue — not the whole page. If you're logging in and out repeatedly, reuse sessions: storageState is one of the strategies for authentication in Playwright, cookies are another. The idea is to avoid logging in and out and rebuilding full accessibility trees every time. Use diffs — differentiate what changed. Keep a baseline of how it was working and compare against the new state. And when a test fails, don't send everything historically — just what's relevant to the failure.

Here's a comparison. This is a test case with six steps: load customers, reach a specific user, open the edit view, fill in the new information, click save, refresh. Comparing poor context management against an improved design, you can see the difference in tokens: the first approach consumed roughly eighty thousand tokens on the first action; the improved one, four thousand. Almost twenty times better. That's something you need to take into account.

Challenge 2: Drift. This one's quite interesting. There's a concept of drift that applies to both context and objective, and in both cases it's highly related to flaky tests. There's a quote I really like: a flaky test that goes red is annoying, but a test that drifts to make itself go green — that's dangerous. If a test fails and you ask the LLM to fix it, and the LLM doesn't do it properly, the test will pass — and that's dangerous. False confidence is worse than no test at all. This is one of the most common reasons developers lose trust in end-to-end test cases: the tests drift and they don't know how to handle it.

With context drift, you're effectively pushing the LLM to make the test pass, not to find defects. Here's the example: a test asserting a field with an order total, with specific text — $129.99. Obviously that value might change — maybe the order changed. That doesn't necessarily mean there's an application problem; maybe it's just a data problem. But since the LLM is now focused on making the test pass, it says, "Okay, I'll fix it" — and instead of investigating the data, it changes the assertion to just check the field is visible. Now the value could be zero, and the test passes. You've introduced a defect your test suite can't find.

With objective drift, you're asking the LLM to do many tasks, and it simply forgets the intent of the original test. You might ask, "What are my critical flows?" expecting it to go from step one to step three, following the steps, reviewing execution history — but somewhere down the line it drifts, fixates on the history, and never answers the original question considering all the steps. So when we talk about making test cases robust and self-healing, this introduces the risk that agents simply forget what the test was for.

The solution: build safeguards. You need agents that handle these situations. What is the intent of the test? What are its conditions? Make sure the guardrails are there — that you're asserting the right things, and if a test self-heals, check that the new assertions actually follow the same strategy. And like with context, everything has to be bounded to what's important for your test case.

Challenge 3: Multi-agent communication failures. All of these solutions involve multiple agents talking to each other, and sometimes the agents simply disagree. They operate in silos instead of sharing information, and they fail to combine information from different sources. When the system starts scaling, you'll see this: one agent draws all the conclusions without the data from the other sources.

We can share this presentation after the webinar. These are the three most common challenges we've seen — context management, drift (context and objective), and agent communication — along with the signals of what's happening and the potential solutions. And those are the easy ones. Then you need to consider the not-so-easy challenges: how you scale from a few test cases to hundreds or thousands, and complex test repairs. What we're hearing on test repair is that with plain LLMs, problems start midway through a test case — if you have fifty test cases, it's highly likely they'll break partway through the application, and the LLMs can't keep up with the context.

We'll share this presentation, and the idea is for all of you to try these challenges and work around them. We encourage everyone to start by creating and documenting test cases as part of your workflow — and then move to something more robust and scalable: handle the context, manage it well, and make sure your agents can communicate with each other. I'll pass it back to Ivan for questions. And feel free to reach out.

Ivan: Thank you, Renan — super insightful. We received a couple of questions. The first one: "I'm trying Claude Code right now for test automation. What are the signs that I'm on the right track — or the wrong one?"

Renan: It depends on how many test cases you're running — that's the first thing — and what your plan is moving forward. If you have your web application, you're testing it with twenty to fifty test cases, tests are passing and they're finding defects — there's a balance between test cases that actually find defects and ones that are just being fixed — I think that's a good sign. If you go beyond that, to a hundred or more, that's when you should ask: how am I managing my context? How am I handling communication between agents? How much time am I spending creating those scripts? But having test cases that pass, and that find defects when there is one — that's a positive sign.

Ivan: Thank you, Renan. Second question from the chat: "I have a Selenium test suite and I want to migrate it to Playwright. I'm thinking about using Playwright agents, mostly with some Claude Code. What are your recommendations to get it right?"

Renan: A Selenium-to-Playwright migration — that's a very good one. One challenge when moving from a specific platform — we've seen it with Selenium and with WebdriverIO as well — to Playwright: it's a different framework and a different protocol, so the timing will just be different. One uses a driver, the other uses HTTP. So first, don't expect that what you have in one platform will behave identically in the other, purely because of the protocols.

What we're seeing that works is having an agent that validates both at the same time. Migrating the code itself is easy — that shouldn't be a problem; you'll have the skills and the safeguards for how you migrate the code. But then you need an agent that actually runs the Selenium suite and the Playwright suite and validates they serve the same purpose. It's not about the steps — it's about the behavior: that they're validating the right things.

Ivan: Thank you for the advice. And the final question we got: "I checked the Amikoo webpage, and you mentioned that sometimes drift happens if you have access to all the context. If Amikoo has access to all the context, how do you make sure Amikoo is an independent auditor and doesn't drift?"

Renan: That's a very good question. We've built our own agents to handle exactly that — the guards we mentioned. We've experienced these problems ourselves; these are the challenges we've faced. Guarding is one of the solutions: constantly checking that the agents don't drift, keeping a line to what the purpose of the test case is. That's one of the challenges we faced, and that's how we came up with this type of solution.

Ivan: Okay, thank you, Renan. Those were all the questions. Thank you, everyone, for attending — we'll be sharing more information. We shared general information today, but we're also the founders of Amikoo, so we've seen all of these issues firsthand, as Renan mentioned, and we've solved them with purpose-built agents that make it easier to automate end-to-end testing without these problems. If you're curious, check out amikoo.ai. Thank you, everyone, and we hope you have a great rest of your day. Thank you, Renan.

Renan: Thank you, Ivan.