Test automation for web apps: what to automate first and what to keep manual
Test automation means software runs the checks of your app instead of a person. Start with the checks you repeat before every release, beginning with login and the paths that take money or data. Keep new designs and exploratory checks manual.
This post is for developers and QA leads at small SaaS teams, consultancies and software houses. You get an order to automate in, five steps to start and a first run on a real app, Kimai. The checks that repeat every release are the ones behind automated regression testing for web apps.
What is test automation?
Test automation is the use of software to run tests and compare the result with what you expected. For a web app, a program opens a browser, clicks through the app as a user would and reports pass or fail. The ISTQB glossary defines it as "The conversion of test activities to automated operation."
Tests sit in layers. A team has many small unit tests that check one function, and fewer UI tests that drive the whole app in a browser. A UI test sees what a unit test can't: the page a real user gets.
Which tests do you automate first, and which stay manual?
Automate the checks you run by hand before every release, in the order of what hurts most when it breaks. A broken login stops every user. Rank your checks by that damage, and work down the list.
- Login and roles. If a user can't log in, or sees a page the role should not see, nothing else matters.
- The paths that take money or data. Checkout, invoices, imports, and anything that writes records.
- Paths that broke before. A bug that happened once can come back.
- The checks you repeat before every release. Software runs them the same way every time.
- Breadth last. Cover the rest of the pages once the first four hold.
Some checks stay manual. Exploratory checks need a person who looks for things nobody wrote down. A design that changes every week breaks tests faster than they pass.
How do you start test automation on a web app in five steps?
Start with the people who use the app and leave the tool for later. Describe what each role does, write test cases from that view, and turn each case into a test. Then run the suite before every release and decide on each red result. Each step has a trap, and the list names it.
- Describe what each role does in the app. Goes wrong: a role nobody described is never tested.
- Write test cases from the user's view, with acceptance criteria and absence checks. Goes wrong: test cases that copy the screen break at every redesign.
- Turn each test case into a test, per role and device. Goes wrong: a one-device suite hides what the other device shows.
- Run the suite before every release. Goes wrong: a suite that runs on request is skipped under deadline.
- Decide on every red result: fix the app, fix the test or accept it with a reason. Goes wrong: red results left open train the team to ignore them.
How do automated UI tests run per role and device?
Each test case names the roles and devices it runs on, and the suite runs it once per pair. Kimai, the open-source time tracker we tested, has three roles: super admin, regular user and no login. We ran each on two devices, Desktop Chrome and an iPhone 13. That gives six Playwright projects, and a test case runs in the projects it names.
What does a first automated run find on a real web app?
A first run finds defects in the app and weak spots in the tests. In the first full run on Kimai, 35 of the 98 tracked rows were red: 26 on the iPhone 13 projects and 9 on Desktop Chrome. A row is one test case in one project, so the 98 rows hold 55 distinct test cases: 43 run in two projects and 12 in one. The report does not say how many red rows were app problems and how many were test problems.
In our run, a suite that covered one device would have hidden most of the red rows, because they sat on the phone.
One absence check caught a real defect. TC-WEB-001 and TC-WEB-002 log out and then press Back. The redirect to the login page worked. Back still showed the dashboard, because the dashboard response from Kimai has no no-store header. No change to the test satisfies the check, so the case is blocked and a person decides about the app.
TC-WEB-170 chooses list columns on the phone. It passed in 30 isolated repeats and timed out once in a full run. On the mobile profile Kimai posts the form and reloads the page, and a click before the reload landed on nothing. The test now waits for the reload.
Versions: Kimai 2.67.0, Playwright 1.49.0 and Chromium 131.0.6778.33. Method: we measured agent time and wall-clock time for the spec, the test cases, their review and fixes, and one parallel run of 66 test cases. The data comes from one app. The human figures are a model estimate, the 66 test cases leaned toward top priority, and we measured no upkeep. The full numbers are in the public report repo.
What can AI agents write for a test suite today?
In our run, AI agents wrote most of a suite from test cases that were written first. In the measured run, six agents worked through 66 test cases in 1h 48m. 55 landed as passing tests and 11 (about 17%) were blocked for a human decision: Kimai defects, criteria the app does not meet, and environment limits. An agent reads a written test case, drives the app in a browser and writes a Playwright test that holds the same steps and checks.
Playwright ships its own agents: a planner that explores the app and writes a test plan, a generator that turns the plan into tests, and a healer that repairs failing ones. The Playwright docs on test agents describe them. They arrived in Playwright 1.56, while the Kimai suite ran on 1.49.0.
For how agents write Playwright tests, read how AI agents write Playwright tests. For the wider question of agents in QA, read QA automation with AI agents.
Which test automation tools exist, and when does each fit?
The tools fall into three types: test automation frameworks that you code against, commercial suites, and AI agents that write framework tests. The table lists them without a ranking, because the right pick depends on your language, your team and the tests you already have.
| Tool | Type | Languages | Fits which team |
|---|---|---|---|
| Playwright | Open-source framework | JavaScript, TypeScript, Python, Java, .NET | Teams that code and want one tool for several browsers and phone profiles |
| Selenium | Open-source framework | Java, Python, C#, Ruby, JavaScript | Teams with an existing Selenium suite or a Java or C# stack |
| Cypress | Open-source framework | JavaScript, TypeScript | Front-end teams that write tests next to the app code |
| Commercial suites (Tosca, Ranorex) | Commercial suite | Model-based without code (Tosca); C# and VB.NET (Ranorex) | Teams with testers who don't code and a budget for licenses |
| AI agents on top of Playwright | AI agents on a framework | TypeScript tests (in the Kimai run) | Teams without testers that review what the agents write |
You can mix them. A team can keep its Selenium suite for old pages and write new tests in Playwright.
What did automating the Kimai backlog cost?
In the report's model for the whole 350-test-case backlog, the agent pipeline takes about 43 hours elapsed, including the time a human spends on unblocking, and about €2,155. A human QA engineer would need about 191 hours and about €10,445. These are model estimates, and we say so. The Kimai case study: hours and costs of 350 test cases shows the arithmetic.
What do flaky tests and upkeep cost after the first run?
A flaky test passes and fails on the same code. The Playwright docs on retries define it the same way. Track every test that needed a retry.
In the latest full run, Kimai had 0 flaky tests. TC-WEB-170 stays on the list as the one suspect, because it timed out once before its fix. Upkeep hours per release are not measured in the report, so this post gives no number. Count upkeep on your own releases.
To see a regression early, you need a record of which tests passed on which release. That is the subject of automated regression testing for web apps.
Which teams should wait before automating UI tests?
A team that ships a visibly unstable UI each week should wait. Its tests break faster than they pass, and every red result costs a fix that the next redesign erases. Such a team writes the test cases and runs them by hand before the next release. Once the screens settle, it automates the same test cases.
FAQ
Can test automation replace manual testing?
It takes over the repeated part of manual testing. Software is good at running the same checks the same way before every release. A person is still needed to explore a new feature, judge a new design and decide what a red result means. In the measured run, 11 of 66 test cases needed a human decision, so plan for both.
How long does it take to automate the tests of a web app?
For Kimai, about 91 minutes of agent time covered the spec, the test cases, the review and the fixes for 350 test cases. Running 66 test cases through six agents then took 1h 48m in the measured run. Human unblocking adds most of the elapsed time, about 43 hours for the whole backlog in the report's model. Your time depends on the size of the app and on how fast a person answers the blocked cases. Those 91 minutes and that run do not include any upkeep.
Do you need to code to start test automation?
You need to read code, and you can leave most of the writing to an agent. A framework such as Playwright expects code, and an AI agent can write it from your test cases. Some commercial suites, such as Tosca, offer model-based tests without code. Either way, you write the test cases in plain words first, and a person must review the result before you trust it.
How often should automated tests run?
Run them before every release at least, and on every change if the suite is fast. A suite that runs on request gets skipped under deadline. In the latest full run, the Kimai suite took about 6.1 minutes, so running it before each release is cheap. If your suite takes hours, split it by priority.
350 test cases for a real web app: what automating them cost and how long it took.
See the measured run on Kimai