QA automation with AI agents: what works on a real web app

By Ivan Rublev · Published · Updated · ~11 min read
Ivan Rublev builds qa-signoff, QA sign-off for client web apps, and ran the Kimai measurement. See the Kimai case study

QA automation with AI agents works on a real web app: the agents can write and run its tests, and a person still unblocks the cases the agents can't finish and signs off the release. In the run described here, AI agents landed 55 of 66 test cases as passing Playwright tests and left 11 (about 17%) for a person.

You get one real run of AI agents on Kimai, an open-source time tracker: what they did well and where a person stepped in. For the background on keeping a suite useful release after release, read our guide to automated regression testing for web apps.

What is QA automation?

QA automation is software that runs checks on another piece of software and compares the results with what you expect. A test script opens a page, takes the steps a user would take and reports pass or fail, without a person clicking through.

When an AI agent writes the tests, the checks stay the same and the author changes: the agent writes the scripts and a person reviews them.

Manual testing still fits one-off exploration of a new feature. Automation beats it when the same checks run every release, because a script repeats them without a person clicking through each time.

The International Software Testing Qualifications Board (ISTQB) Certified Tester AI Testing (CT-AI) syllabus focuses on testing AI-based systems. This post is about the other direction, AI software testing: using AI to test an ordinary web app.

How can AI be used in test automation?

A QA automation AI agent can do the writing work of a test suite: read the app, plan what to check, write the test code and fix it when a run fails. A person keeps the reviewing work: reading the test cases and deciding on the release.

  1. Crawl the app per role. Without an account for each role, the crawl misses every page that role sees.
  2. Write a spec and test cases from the user's perspective. The agent writes a plausible expectation that the app never promised.
  3. Review the test cases before any code. Skip it and you get neat tests for wrong expectations.
  4. Write one Playwright test per test case, as shown in how AI agents write Playwright tests. A test that passes by checking nothing is the risk.
  5. Run the suite and track regressions to build a regression suite that runs before every release. A status list nobody reads hides the regressions.

This is AI in test automation as the Kimai run used it. Here is what each stage looked like.

How does an agent crawl the app per role?

The first agent logs in as each role and maps the app. For Kimai that meant two logged-in roles (super admin and regular user) and a visitor without a login. Each role sees different pages, so each gets its own map.

The crawl and the written QA spec took about 10 minutes. A role without a seeded account can't log in, so create one account per role before the crawl starts.

How does an agent write a spec and test cases from the user's perspective?

From the spec, an agent writes test cases in plain language, one per checked behaviour, for example "a regular user can edit a time entry". Each test case lists what must be true afterwards and what must not happen. That produced 212 test cases first, then 350, in about 8 minutes.

The agent can write an expectation that the app never promised. The next stage has to catch it.

Who reviews the test cases before any code?

A second pass checks the test cases against the spec and against the app code, and a person reads the result. The review took about 1 hour of agent time and the fixes about 13 minutes.

This stage is the cheapest place to catch a wrong expectation, because no test code exists yet.

How does an agent write one Playwright test per test case?

Now the agents write code. Every test case becomes one test file. It runs for its role on desktop and on a phone through Playwright projects, and 43 of the 55 files run on both. The result for Kimai: 55 test files so far, about 57 lines per test case.

The acceptance and absence criteria of each test case guard against a test that checks nothing.

How does the suite run and track regressions?

A run of the whole suite takes about 6.1 minutes. After each run, a status list holds one row per test case and role-device pair. It flags tests whose test case has changed and tests that passed before and fail now.

In the latest full run, 96 of the 98 tracked rows passed and 2 passed partly. A row is one test case in one project, and 55 distinct test cases sit behind the 98 rows, out of 350 in the backlog. The count of regressions stayed at zero. The 2 that passed partly skip one check, because the test environment runs over HTTP and the HTTPS-only check cannot apply.

What does QA automation look like on a real app: 350 test cases for Kimai?

In the measured run, up to six agents processed 66 of the 350 test cases in about 1h 48m, and 55 landed as passing tests. We wrote the backlog of 350 test cases for Kimai, an open-source time tracker, with qa-signoff, our set of agent skills for Claude Code and Codex. Up to six Claude Code agents, running the Sonnet 5.5 model, worked through it in parallel against Kimai 2.67.0, Playwright 1.49.0 and Chromium 131.0.6778.33.

The numbers below come from the report in our public repository, kimai-qa-playwright-report. The full cost comparison is on the page for the measured run on Kimai, with the full numbers.

WhatNumberKind
Test cases in the backlog350, for 3 roles on Desktop Chrome and iPhone 13Count
Test cases processed in the run66 in about 1h 48m, 55 landedMeasured
Human QA engineer, whole backlogabout 191 hoursModel estimate
This pipeline, whole backlogabout 43 hours, including human unblockingEstimate from the measured run

Read the limits before you copy any of these numbers.

The run covers one app. Only 55 of the 350 test cases have a landed test, and the rest is extrapolated from those 66. The human column is a model built from Bons et al., a 2023 study of Selenium IDE (6 hours for 11 scripts), since nobody wrote these tests by hand. And the 66 test cases leaned toward top-priority behaviour more than the remaining 284 do, so the estimate holds only if the mix stays similar.

What did the AI agents do well?

In the Kimai run, the agents were fast at the writing stage. Up to six agents processed 66 test cases in about 1h 48m of wall-clock time, which is about 1.6 minutes per test case, on one app.

The agents were also consistent. Every test case became one file in the same format, so a reviewer opens any file and knows where to look. We cover those Playwright test files, one per test case, in a separate post.

The surprise was what they refused to do. In our experience, an agent handed an app and asked for tests says nothing about what it left out. Ours worked from test cases that already had acceptance and absence criteria, so every test had something to prove.

Take TC-WEB-001, "User logs in, works on timesheet and dashboard, and logs out safely". The test case lists this absence criterion:

After logout, the browser Back button does not show protected page content.

The agent's test passed every criterion except that absence check. Kimai sends Cache-Control: max-age=0, must-revalidate, private on the dashboard, without no-store, so Back shows the cached dashboard after logout. The agent left that criterion unticked and blocked the case instead of weakening the check to get a green run. TC-WEB-002, the same journey for the super admin, blocked the same way.

The rest of the suite is green: in the latest full run, 134 test blocks passed (a test file holds one or more blocks), 2 were skipped because the environment runs over HTTP, and 0 failed or were flaky, in 6.1 minutes.

Where is a human still needed in QA automation?

In the Kimai run, 11 of 66 test cases blocked (about 17%), and a person is needed for those and for the review of the test cases. The release decision stays with a person too.

The blocked test cases had three causes. Two sit in the app: defects in Kimai, and criteria the app doesn't meet. The third sits outside it: limits of the test environment.

TC-WEB-001 and TC-WEB-002 show the first kind of cause, a defect in Kimai: the dashboard has no no-store, so Back after logout shows it again. The test case notes record the evidence and two ways out: Kimai sends no-store on authenticated pages, or the criterion changes. A person decides which.

How long that decision or the fix takes was not measured. Two of the three causes sit in the app, so a blocked test case can point at something worth fixing there, as this one did.

The human work in the Kimai run sits almost entirely in the 17% that blocked. Preparation took about 20 minutes of human prompting and review. Unblocking the roughly 58 blocked test cases of a 350-case backlog is estimated at about 32 hours, about 0.55 hours per case. The agents couldn't finish those cases, so they may be harder than average and take longer than modelled.

The review of the test cases is the second job. An agent can write a test for a wrong expectation as neatly as for a right one.

The third job is the release decision. The suite says what passed, and a person decides whether that is enough to ship.

How does QA work without a QA team?

QA without QA team staff works if the suite has an owner who isn't a QA engineer. If you have no QA team, pick that person first. On a small SaaS team the owner is a developer or a tech lead. In a software house it's the lead of the project the suite belongs to.

The hours matter more than the title. That person reads test cases and answers the question the agents cannot: is this what the app should do?

How many hours? This is an estimate: about 32 hours at the start to unblock the first backlog, then about 2 to 4 hours per monthly release for upkeep and for reading the results. Both figures are our estimates, not measurements.

No one measured upkeep on the Kimai suite, which has run for days, not months. If nobody has those hours, look at outsourcing QA to an agency.

Who should not follow this approach? A team where nobody can give review time. The review is the gate, and without it the agents write tests for whatever they assumed.

The approach also fails on an app the agents cannot reach per role, for example when there are no seeded accounts and no test database. A team with fewer than about ten tests gains little from a pipeline.

Automate it or outsource it?

Automate when the same checks run every release; outsource when nobody has the hours. Automating with agents bought us speed, 66 test cases in about 1h 48m, and leaves the review with your team. Put the outsourcing quote next to the 2 to 4 review hours per release. We lay out what outsourcing QA costs compared with automating it in a separate post.

See the measured run on Kimai: what 350 test cases cost by hand, and what the agents and one person did instead.

FAQ

Can AI agents be used in QA testing?

Yes. In our Kimai run, AI agents crawled the app, wrote test cases, wrote Playwright tests and ran the suite. They finished 55 of 66 test cases without a person unblocking them, on one app. The other 11 were blocked and needed a person to decide what to do next, so the release decision stays with a person.

Will QA testers be replaced by AI?

No, not in the Kimai run. The agents took over the writing of tests, but a person still reviewed the test cases and would have to unblock the 17% the agents could not finish. The role shifts from writing scripts to judging whether the checks are right, and the release decision stays with a person. One run on one app cannot prove more than that, and we expect the split to differ on other apps.

Is QA worth it in 2026?

Yes, when a test finds what you would not look for. The cached dashboard after logout in Kimai surfaced because a test case demanded that Back shows no protected page. Automation lowers the price of that check, an estimated 43 hours against 191 hours by hand for the whole backlog, as the Kimai case study shows. It doesn't remove the need for someone to own the suite, and for small teams that owner is often a developer.

Is there any AI for software testing?

Yes. AI coding agents such as Claude Code can write and fix tests. In our Kimai run, agents of that kind wrote Playwright tests for 55 of 66 test cases; the other 11 stayed blocked. Which tool fits depends on your app and the review time you can spare, so start from our overview of AI testing tools.

Updated 13 November 2026: Reworded one unmeasured claim in the human-work section.
The author sells qa-signoff.

See what agents did on a real app: 350 test cases on Kimai.

See the measured run on Kimai