Automated regression testing for web apps: a practical guide

By Ivan Rublev · Published · Updated · ~17 min read
Ivan Rublev builds qa-signoff, QA sign-off for client web apps, and ran the Kimai measurement. See the Kimai case study

Picture a team that ships a SaaS app without a regression suite. It hears about its breaks from customers: a billing change goes out on Friday, and by Monday a member cannot export a report. Automated regression testing re-runs tests by software after every change to confirm that what worked still works, and for a SaaS web app it covers each role and device before each release.

This guide is for product teams and software houses with UI-heavy apps, and for their quality assurance (QA) engineers. It shows how to build the suite, pick tools, track each test case across runs and gate releases, and what AI agents can write for you. The figures come from one measured run on Kimai, an open-source time tracker. The limits: it is one app, and the human-hours figure is a model estimate.

What is automated regression testing?

Automated regression testing re-runs a fixed set of tests by software after every change, to confirm that what worked still works. The International Software Testing Qualifications Board (ISTQB) glossary defines regression testing as change-related testing that detects defects introduced or uncovered in unchanged areas of the software.

Teams automate it for speed and repeatability. The latest full Kimai run re-ran about 55 distinct test cases in about 6 minutes. Manual testing stays the better tool for exploring a new feature and for judging how it feels. Automated regression testing covers the repeat checks.

What is meant by regression testing?

Regression testing checks that a change did not break something that already worked. The key word in the ISTQB definition is "unchanged": you test the areas around the new feature, because that is where nobody is looking.

A regression is a test that passed on the last run and fails on this one. The word matters later, because a regression points at the changes made since the last pass.

What is an automated regression test?

An automated regression test is one check that a tool runs without a person, with a fixed expected result. For a web app it takes the steps a user would take in a browser and compares what it sees with what the test case says must be true. A group of such tests that runs together is a regression test suite.

Retesting re-runs a test that failed, to confirm that a fix worked, while regression testing checks everything around the fix.

What is an example of a regression test?

Take a member who exports a report every Monday, as an illustration. A test logs in as a regular member, opens the reports page, clicks export and checks that a file downloads. It passed for months.

Then a billing change ships and the export stops working for the member role. The test fails, and the failure points at the release. The failure shows up in the test run before a customer files a ticket.

Why does regression testing for SaaS need to run on every release?

A software as a service (SaaS) app changes continuously, and every customer sits on the newest version the day it ships. There's no old version to fall back on and no installer to delay. A break reaches everyone at once. That is why regression testing for SaaS belongs to every release.

Shared code makes it worse. A permission check or a navigation component serves many pages, so a change in one place shifts behaviour in ten others. Nobody reviewing the diff sees all ten. Regression testing is one part of a wider QA process, and this guide covers that part.

Who usually does regression testing?

Who does regression testing depends on the team. In companies with QA staff, QA engineers own the suite and read its results. In teams without QA staff, developers run it, often in the build pipeline, and a tech lead decides what happens when a test fails.

Agencies add a third route: an outside team writes and runs the suite for a client. The section on who should write and run the suite compares the three routes.

Which roles and devices does a SaaS regression test cover?

A suite can be organised by code layer: unit, API, UI and end-to-end tests. A SaaS suite needs a second view, the user's. Ask which roles use the app (admin, member, guest, visitor) and which devices they use (desktop, phone), and make sure every role-by-device pair has checks.

Kimai shows the shape. In the measured run the roles were super admin and regular user, plus a visitor without a login. Each ran on Desktop Chrome and on an iPhone 13, which gives six Playwright projects. The same test case runs once per role and device it applies to, so a break that shows only for a guest on a phone still fails a test.

The suite also finds defects. The logout tests for the regular user and for the super admin were among the test cases the agents blocked, on both devices: after logout, the Back button showed the dashboard again. The suite found a defect. The tracked runs show no test going from passing to failing.

How do you build a regression test suite?

A regression test suite is the set of tests you re-run after every change. It is a product of its own: it has an owner and a maintenance cost, and it fails by growing until nobody trusts it. Build it small and extend it on purpose. Follow these steps in order, and note what goes wrong if you skip one.

  1. Pick risk first: login and access for every role. Start with long journeys instead and the suite is slow and says little about where a break is.
  2. Write the test case before the test. Skip it and the test has nothing to prove.
  3. Add one check per role and device. Skip it and a break that shows only for a guest on a phone passes.
  4. Run the suite on every change from the first week. Run it only before a release and the failures arrive in one pile.
  5. Update or delete tests when the app changes. Skip it and stale red tests teach people to ignore red.

What is a regression testing suite?

A regression testing suite is a named, versioned collection of tests with a clear scope, for example "everything a member can do in the app". A regression testing suite differs from a pile of test files in two ways. Each test traces back to a written test case, and the whole set runs from one command.

What goes into the suite first?

Start with risk: the checks whose failure would cost the most, such as login, access control, payment and data loss. Add one per role first, then widen by device.

Teams write long journey tests first because they feel valuable. We think that is the wrong order. A journey that crosses ten pages fails for many reasons, takes longest to repair and tells you least about where the break is. End-to-end journeys belong in a suite, after the short checks.

The Kimai backlog shows the pull. In the measured run, about 66 test cases were weighted toward top-priority checks and held about 9% journey-level cases. The other 284 hold about 3.5%. Write the short checks first (can this role sign in, can this role open that page) and the journeys last.

Why write the test cases before the tests?

A test needs a written expectation to prove. Write each test case in plain language first: what the user does and what must be true afterwards. Then write the test from it. The guide on how to write test cases covers the structure.

The order matters more when an agent writes the tests. An agent writes a neat test for a wrong expectation as readily as for a right one, so the expectation is the part a person must own.

How do you maintain a regression test suite when the app changes?

A suite that nobody updates turns red for the wrong reasons, and people stop reading it. Treat maintenance as part of every change: when a feature changes, change its test case, then its test. Delete tests for features that no longer exist, and keep the test data in step, because a test that depends on a removed record fails for the wrong reason.

Tooling helps. If every test traces to a test case, a changed test case flags its test for rewriting, and a test with no test case is flagged as an orphan. Without that link you find stale tests only when they fail.

Which regression testing tools fit a web app?

The right tool depends on the team more than on the app. Automated regression testing tools fall into four categories. The Watch-out column describes the category, and no vendor was tested for this guide.

CategoryExamplesFits whenWatch out for
Code-first frameworksPlaywright, Cypress, SeleniumYou have developers who can own test codeMaintenance sits with your team; flaky tests are yours to fix
Low-code and AI platformsKatalon and mablNon-developers must build and edit testsLock-in to the platform
Visual regression toolsPercy, ApplitoolsThe risk is how pages lookScreenshot diffs need a reviewer
Managed servicesOutsourced QA teams, test-as-a-serviceNobody in the team has the hoursCost per release; knowledge stays with the vendor

Visual regression tools compare screenshots and flag layout changes, which is a different problem from the one this guide covers.

End-to-end testing drives the whole app through the browser as a user would, and it is the layer these suites run at.

For web apps we use Playwright, because it runs one test across browsers and device profiles and ships with a test runner. The page on Playwright and AI: generate an end-to-end suite covers it in depth.

What is the best tool for automated regression testing?

The best tool is the one that fits your team, so choose the category first. Code-first fits teams with developers who have time to own test code. Teams where testers write the tests take low-code, and teams with no hours at all take a managed service.

Within the category, pick the tool your team can still read in a year.

Can AI do regression?

Yes, for the writing half of the job. An AI coding agent can write the Playwright test for each test case and repair it when it fails. A person keeps the reviewing half: whether the expectation is right and whether the release goes.

In the measured run on Kimai 2.67.0, with Playwright 1.49.0 and Chromium 131.0.6778.33, up to six Claude Code agents on Sonnet 5.5 processed 66 test cases and landed 55 as passing tests (350 test cases, 66 in a measured parallel run). 11 of the 66 test cases (about 17%) blocked, because of an app defect or a limit of the test setup. A person decided what to do about each one.

The split between agent work and review matters for regression testing. The agents are fast at the repeat work and need a person for the expectations, so the test cases need review before any code exists. Read QA automation with AI agents for the stage-by-stage account, and how Playwright test agents write the tests for the tooling.

How do you run automated regression testing in CI and gate releases?

Run the suite in continuous integration (CI) on every change and gate each release on its result. CI runs the same tests on a clean machine. The gate is a rule, written before the run, that turns results into PASS or BLOCKED. Each step names what goes wrong if you skip it.

  1. Run the fast part of the suite on every pull request. Skip it and breaks surface a week later, mixed with other changes.
  2. Run the whole suite before each release, on a fresh build. Run it on a developer's machine and the result depends on that machine.
  3. Record every result per test case, role and device. Keep only today's result and you can't tell a regression from a test that never passed.
  4. Mark tests that flip as flaky and count them. Retry them silently and the suite looks green while it loses signal.
  5. Apply the written gate and let one named owner read the result. Decide after the run and the loudest voice decides.

When should the suite run, and what gates a release?

Run the fast part on every change and the whole suite before each release. The Playwright 1.49.0 continuous integration guide shows how to run tests on a CI server. What gates a release is a decision made in advance: which failures stop it, and who may overrule the gate.

How do you track each test case across runs?

A pass or fail of today means little on its own. Keep a record per test case, role and device for every run, so you can see what passed last time. A regression is then easy to define: a test that passed before and fails now. List those first, each with the date it last passed, because the bug is most likely in the changes since that date.

Here is one worked example. qa-signoff Pro, a tool we built, runs the Playwright suite and writes one row per test case and Playwright project into a CSV, with ✔ or ❌ for each run, and the role and device sit in the row. It flags every regression with the date the test last passed and marks tests that flip as flaky. Any spreadsheet and a script can do the same.

How do you handle flaky tests without retrying them away?

A flaky test passes and fails on the same code. Retries hide it: the test goes red, passes on a retry and nobody looks. Playwright reports it openly: its Playwright 1.49.0 retries mark a test that failed on the first run and passed on a retry as flaky. Use that mark, count the flaky tests, and fix or quarantine them.

The Kimai runs show what to expect. Between the first and the second tracked run, 34 rows flipped from red to green and none flipped from green to red. One iPhone test stayed red until the third run: switching to the mobile profile makes Kimai reload the page, and a click before the reload landed on nothing and the test timed out. In the latest full run the flaky count was 0.

The Kimai flaky count comes from one app over a few runs. A suite that has run for days has not met its flaky tests yet.

What does a release gate that says PASS or BLOCKED look like?

A release gate turns the results into a decision. The one in qa-signoff Pro checks five things, and any failed check makes the report say BLOCKED:

  1. The pass rate is 80% or more.
  2. No regression: nothing that passed before fails now.
  3. No test whose test case changed and which was not rewritten.
  4. No orphaned test, with no test case behind it.
  5. No missing or misplaced test.

The report lists regressions first, because those are what the release owner must read. Choose your own threshold and write it down before the run. The latest full run on Kimai passed the gate with no failed row among about 98 tracked rows: 96 pass and 2 pass partly. A row is one test case in one project, so about 55 distinct test cases sit behind the 98 rows; two rows pass only partly, because the HTTPS-only check is skipped over HTTP.

The 98 rows cover about 55 of the 350 test cases, so the pass rate says little about the rest. The gate is part of qa-signoff Pro; see qa-signoff Pro pricing.

What is regression testing vs UAT?

Regression testing asks whether a change broke what worked, and runs after every change. User acceptance testing (UAT) asks whether the software does what the client agreed to, and runs once, when the build is ready to hand over. They have different owners and different moments, and a clean regression run comes first.

Regression testingUser acceptance testing
QuestionDid the change break something?Does the build meet the agreed criteria?
Who runs itQA, developers or agentsThe client's users
WhenAfter every changeOnce the build is ready to hand over
DecidesWhether the release is safe to shipWhether the client accepts
AutomationYes, that is the pointRarely; people judge fit

A clean regression run is the entry ticket to the client's acceptance test, and it runs again after every fix that acceptance triggers. See regression testing vs UAT, with a UAT plan template for the plan.

Who should write and run the suite: in-house, outsourced or agents?

You have three routes, and each trades money, speed, control and risk.

In-house. Your developers or QA engineers write the tests. You keep the knowledge and set the order, and you pay in hours that compete with feature work. It fits a team that already has someone who enjoys the job. By our rule of thumb, a team whose manual checklist takes under about 20 minutes and which releases monthly should not automate yet, because running that list by hand costs less than maintaining a suite.

Outsourced. An outside team writes and maintains the suite. You buy hours you do not have, and you pay per release or per month. The risk is that the knowledge of your app sits in the vendor's heads. The page on what outsourcing QA costs compared with automating it has the cost reasoning.

Agents. AI coding agents write the tests from test cases and a person reviews them. For the whole 350-test-case backlog on Kimai the agent pipeline needs about 43 hours against an estimated 191 hours for a Playwright agency, about 4.4 times faster. The agency figure is a model estimate and the pipeline figure is extrapolated from a measured run on one app, so read the ratio as a direction. QA automation with AI agents explains the pipeline, and the measured run on Kimai: 350 test cases written by agents has the run and its limits.

What are the most common regression testing mistakes?

  1. Starting with long journeys. They break often and tell you little. Write short checks first, per role.
  2. Writing tests with no test case. Nothing says what the test must prove, so a green run proves nothing. Write the test case first.
  3. Retrying flaky tests away. The suite looks green and you lose the signal. Mark and count them.
  4. Having no record across runs. Without the history you can't tell a regression from a test that never passed.
  5. Gating on a feeling. If the release rule is not written down before the run, the loudest voice decides.
  6. Never deleting tests. Tests for removed features fail for the wrong reason and train people to ignore red.
  7. Testing only one role or one device. A break often shows only for a guest, or only on a phone.

FAQ

Can you automate regression testing?

Yes: regression tests repeat the same checks after every change, which a machine does cheaply. Automate the stable, high-risk checks first: login and access for each role. Keep exploratory testing, which looks for problems nobody has written down, with a person. In the measured run on Kimai, agents landed about 55 of 66 test cases as passing Playwright tests, and people reviewed the expectations and the 11 blocked cases.

What is a regression test used for?

A regression test is used to confirm that a change did not break behaviour that already worked. Teams run it before a release to decide whether the build is safe to ship, and after a bug fix to confirm that the fix did not damage something nearby. A green run is the evidence the release owner reads. A red run lists what to look at first.

Can you give me an example of a test suite?

A small suite for a SaaS app holds one login check per role, one access check per protected page, the core flow of each role and a check that a logged-out visitor sees no protected content. Run it on desktop and on a phone. The Kimai run split its tests the same way: three roles on two devices, six Playwright projects. The latest full run took about 6 minutes.

How often should a SaaS team run its regression suite?

Run the fast subset on every change and the full suite before every release. A team that ships weekly runs the full suite weekly, and splits the suite instead of skipping the run when it is too slow. The latest full Kimai run, with about 55 distinct test cases, took about 6 minutes, so it needs no plan yet. A 350-test-case suite will.

Pick the next step by intent. A web application testing checklist covers what to test in a web app beyond regression.

Updated 13 November 2026: First publication.
The author sells qa-signoff Pro.

Get a regression record per test case and a PASS or BLOCKED gate before each release: qa-signoff Pro, €1,500 one-time for 5 seats.

See Pro prices