Playwright AI: generate an end-to-end suite from planned test cases

By Ivan Rublev · Published · Updated · ~12 min read
Ivan Rublev builds qa-signoff, QA sign-off for client web apps, and ran the Kimai measurement. See the Kimai case study

Playwright AI is two agent-facing tools, an MCP server and a command line for coding agents, plus three test agents, and none of them decides what your app should be tested for. That decision stays with you. This post shows a suite of about 55 Playwright specs, written from about 350 planned test cases on a real app, and the method behind it.

The suite is the start of automated regression testing for web apps. Here's what each Playwright tool does, where it stops, and what we put in front of it.

On this page

Is Playwright an AI tool?

No. Playwright is a browser testing framework. Since 2025 it ships two AI-facing tools, Playwright MCP (Model Context Protocol) and a command-line interface (CLI) for coding agents, and it adds three test agents. The tests themselves are plain TypeScript that runs without any AI. Codegen, the recorder, uses no AI at all.

What do Playwright AI features do today?

As of October 2026, Playwright has four tools for writing tests with an agent or a recorder. Three need an AI agent behind them. The fourth is a recorder.

What is Playwright MCP, the browser as a tool for an agent?

Playwright MCP is a Model Context Protocol server that gives an AI agent browser control through structured accessibility snapshots. You start it with npx @playwright/mcp@latest. As of October 2026 the releases are v0.0.83 for MCP and v0.1.22 for the CLI. The agent reads element roles and names instead of screenshots, so it needs no vision model.

What is the Playwright CLI for coding agents?

The Playwright command-line interface installs with npm i -g @playwright/cli@latest. Playwright describes it as token-efficient browser automation for coding agents such as Claude Code and GitHub Copilot. It is driven by installable skills.

What are the Playwright test agents: planner, generator, healer?

The Playwright test agents arrived with Playwright 1.56.0 on 6 October 2025. You add them to a project with npx playwright init-agents --loop=claude. Other loops are vscode, codex and opencode.

What does Playwright codegen do: recording actions into tests?

Codegen records what you do in a browser and writes the locators and the test code. Playwright says it looks at the page and picks the best locator: role, text or test id. Run npx playwright codegen --device="iPhone 13" to record on a phone profile. Add --save-storage=auth.json to keep a login for later.

Does Playwright have AI capabilities?

Yes. Playwright MCP and the CLI give an agent a browser, and the planner, generator and healer use that agent to write and repair tests. Codegen is a recorder and uses no AI. The output of all four is ordinary Playwright code that you can read and keep in git.

Where does a Playwright AI agent stop?

A Playwright AI agent stops where the decisions start. Playwright AI testing from a bare prompt such as "write tests for this app" hides every one of them. The planner picks the scenarios.

It doesn't pick priorities, roles, devices, negative checks or absence checks, and it doesn't review its own plan. The Playwright docs say that generated tests "may include initial errors", and that the healer can fix them.

A bare prompt to a general assistant has the same gap. Stefan Judis wrote on 14 July 2025 in Generating end-to-end tests with AI and Playwright MCP on the Checkly blog that asking an LLM for a test without context "doesn't work at all", and that the model needs the page snapshots Playwright MCP supplies. He finds writing tests in plain English with vague instructions "way too risky". Live state doesn't tell the agent which tests the app needs.

We haven't benchmarked Playwright's agents. Artem Bondar of Bondar Academy did, on his own demo app with Sonnet 4.5, and published the review on 18 May 2026. His planner needed 14 minutes 18 seconds for a plan of about 1,500 lines. The generator needed 15 minutes 40 seconds for two tests, and 0 of 2 passed before healing.

The healer needed about 10 minutes, and both tests then passed. That's one app and one model, so read it as one data point.

Bondar also names a limit of the MCP view: the agent sees only the accessibility tree, so it can't see attributes such as data-test-id. That's his claim.

A free-form plan drifts to happy paths. In the Kimai run, TC-WEB-170 is a case such a plan could skip: choosing list columns on the iPhone profile must persist and reset, and must not leak into the desktop profile. Nobody asks for that in a one-line prompt. We cover the AI side of this in QA automation with AI agents.

How do you plan test cases before you generate tests?

To get a Playwright suite from an agent, plan the test cases first, review them, and then write one test per test case. The plan is a set of files a person can read. The agent writes code only after that review. Each of the five steps below names what goes wrong when you skip it.

  1. Crawl the live app per role and write the spec: site map, roles and environments. A role missing from the spec has no test case.
  2. Write one test case file per behaviour, as in how to write test cases: priority, projects, Arrange, Act, Cleanup, and acceptance criteria with absence checks. Poor test isolation makes many tests fail when one behaviour changes.
  3. Review the backlog against the spec and the app code before any test exists. Without the review, the cases the agent missed stay missing.
  4. Generate one Playwright test per test case. Several tests per test case are hard to maintain.
  5. Run the suite on the role and device projects and read the blocked cases. A blocked case doesn't test the behaviour.

These steps are general practice. The Kimai run shows what they look like.

TC-WEB-025 covers batch update and batch delete of timesheet entries. Its file holds priority P1, the projects user-desktop-chrome and user-iphone-13, and five acceptance criteria, two of them absence checks: things that must not happen. In the Kimai run the review pass took about one hour and the fixes about 13 minutes.

How is a Playwright automated testing suite laid out from test cases?

The suite follows the test cases one to one. A reviewer who knows one file knows them all.

Why one test file per test case?

Each test case becomes exactly one file, Tests/<role>/TC-<ID>.spec.ts, and every test title starts with the test case ID. The first line of the file carries a criteria hash. When the test case changes, the hash no longer matches, and the test is flagged for rewrite. This is the base of a regression suite that runs on every release.

How do you test several roles and devices in Playwright?

Use Playwright projects. A setup project logs in once per role and saves the session.

The other projects are named <role>-<environment>, for example user-desktop-chrome and user-iphone-13. The test never sets a viewport or a device, because the project supplies it. Three roles on two devices make six projects.

How is test data created through the app's API and cleaned up after?

Arrange runs in beforeAll and creates the test data through the app's API. Cleanup runs in afterAll and deletes it. In TC-WEB-025 the spec calls the Kimai API to create three timesheet entries before the test. Afterwards it removes what is left.

What are the Playwright e2e test results on a real app (Kimai)?

Of the about 66 test cases processed in the measured run, about 55 landed as specs. About 11, about 17%, were blocked for a person. The backlog holds about 350 test cases.

A fleet of Claude Code instances on Sonnet 5.5 took about 1 hour 48 minutes. The numbers come from the measured run on Kimai, an open-source time-tracking app (Kimai 2.67.0, Playwright 1.49.0, Chromium 131.0.6778.33), and from the public report.

The about 55 landed files hold about 3,145 lines and about 70 test() blocks, about 57 lines per test case. They run as six projects.

In the latest full run, about 98 tracked rows passed. A row is one test case in one project, so about 55 distinct test cases sit behind the 98, out of about 350. Of those, 43 ran on two projects and 12 on one. Two rows passed partially, because an HTTPS-only check is skipped when the test environment runs over HTTP.

The blocked share is the interesting part. Blocked means an agent stopped before forcing a green result. The report names the causes: a Kimai defect or a criterion the app doesn't meet, besides a limit of the test environment.

TC-WEB-001 and TC-WEB-002 show the first kind. After logout, the browser's Back button shows the dashboard again, because the dashboard response carries no no-store cache header. A self-healing agent could have weakened the back-button check or dropped the criterion until something went green. Here the failing check is the finding.

A person who takes over a blocked case needs about 33 minutes, an estimate from the report.

What surprised us was the clock.

About 73 sessions add up to about 8.9 hours of agent time and finished in about 108.5 minutes. That is about 1.6 minutes of wall-clock time per test case, against about 8.1 minutes of agent time. Parallel test writing and the release gate belong to the Pro edition of the skills.

The fleet, the 1 hour 48 minutes and the 1.6 minutes per test case come from that Pro run. The free pack plans, reviews and writes tests one after another.

The Kimai suite was written by plan-first skills on Playwright 1.49.0. It wasn't produced or run with Playwright's test agents, since those arrived in 1.56.0.

Limits: one app. The estimate for a person doing the same work, about 191 hours, is a model built on a rate from Bons et al. (2023), Information and Software Technology 158, a study of Selenium IDE. Nobody measured it on Kimai, and Playwright's own agents weren't part of this run.

How do Playwright test agents compare with plan-first agent skills?

This table compares two methods by what each is for, and it ranks neither. Playwright's agents get tests from a prompt. The qa-signoff skills, which we sell, decide what to test first and keep that list reviewable. Both can run in Claude Code.

Playwright test agentsqa-signoff skills
InputA request and a seed testA crawl of the running app per role
What decides what to testThe planner, in a Markdown planA spec, then one test case per behaviour
Roles and devicesYour seed test and configPlaywright projects named <role>-<environment>
Output layoutspecs/ and tests/Tests/<role>/TC-<ID>.spec.ts, one file per test case
Review stepYou read the planA review pass checks the backlog against the app
Repair of failing testsThe healer fixes the test in placeA test that can't pass is blocked for a person
Blocked itemsThe healer can skip a test it judges brokenBlocked test cases are tracked with the reason
Cost to trynpx playwright init-agents and agent usageThe free GitHub pack and agent usage

The healer fits tests that broke because a locator moved. When the app is wrong, blocking fits better, because a repaired test hides the defect.

Skip plan-first skills if you have about ten tests or an app with no stable test data. Skip them too if nobody has time to review test cases. Playwright's test agents or codegen fit those cases better.

What else do people ask about Playwright and AI?

Can Claude Code be used with Playwright?

Yes. Playwright's test agents install for Claude Code with npx playwright init-agents --loop=claude, and they then run inside your Claude Code session. The Playwright CLI names Claude Code as a target, and Playwright MCP works with any MCP client. The plan-first skills run in Claude Code too, and we wrote the Kimai suite that way.

Which AI tool is best for Playwright automation?

None is best for every job, and we don't rank them. Use Playwright MCP to explore an app, codegen to record a flow, the healer to repair a test whose locator moved, and plan-first skills for a whole suite across roles and devices, because they start from a reviewed list of what to test and keep one spec file per test case.

Are Playwright tests E2E tests?

Yes, Playwright tests can be E2E tests. Playwright drives a real browser through your app, and that's the definition of an end-to-end (E2E) test. It also runs component tests and API tests. The Kimai suite is end to end: a browser logs in as each role and checks what the page shows.

Can I use Playwright for UI testing?

Yes. Playwright UI testing, or Playwright AI testing when an agent writes the tests, means a browser clicks and reads the page the way a user does, on every browser engine Playwright ships. You can emulate devices such as the iPhone 13 profile from the same config. Our six Playwright projects run each role on Desktop Chrome and on an iPhone 13 profile.

How to automate Playwright tests?

Plan the test cases, then let the tests follow them. Five steps:

  1. Crawl the running app and write the spec. A skipped role is never tested.
  2. Write one test case per behaviour and review the backlog. A vague test case gives a vague test.
  3. Write one spec file per test case. A file with two cases hides which one failed.
  4. Run npx playwright test. A failing test shows where the app and its test case disagree.
  5. Rerun the suite on every release, or it goes stale. The Kimai suite takes about 6.1 minutes.

Can Playwright's AI agents fix a failing test on their own?

Partly: the healer replays a failing test, inspects the current UI, proposes a fix and reruns it, but it can't tell whether the app or the test is wrong. A person still decides what a failure means, and in Kimai that was the case for TC-WEB-001. Whether AI replaces QA testers is a larger question, covered in QA automation with AI agents.

Try the plan-first method on your own app: the free qa-signoff skills write the test cases and the Playwright tests. Get them on GitHub. Need parallel test writing and a release gate? See qa-signoff Pro pricing.

Updated 13 November 2026: First publication; Playwright features as of October 2026.
The author sells qa-signoff.

See what agents did on a real app: 350 test cases on Kimai.

See the measured run on Kimai