How to plan your regression suite so it doesn't rot
A regression suite only protects you if someone decides what goes in, what stays and what gets deleted.
Regression testing answers one question: does the system still do what it did yesterday? Run it in your CI pipeline and you get that answer on every change.
The trouble starts a few months later. Teams keep adding tests at every level, and nobody decides which ones stay. The suite grows, gets slower, and fills up with checks for functionality that no longer exists. Then people stop trusting it.
A regression suite doesn’t rot because testing is hard. It rots because nobody planned it.
Organize by functionality, not by story
In the teams I worked with, tests were managed in a test management tool, and the natural move was to attach each test to the story that created it. It didn’t last. Stories close. A year later nobody can tell which tests still protect something real. Organize the suite around what the system does instead, and a test has a home that outlives the story that produced it.
Five steps that keep a suite healthy
- Agree on the criteria for adding a test. Not every test deserves to run forever.
- Decide how to mark regression tests, with a tag or a saved query, so you can always pull the suite up on demand.
- Decide when tests get added: at the end of a feature, at the end of an iteration, or as part of your definition of Done.
- Let unit tests and TDD tests join the suite automatically through CI, with no manual step.
- Put testers and developers in one group that reviews the suite at regular intervals and decides what to keep, based on return on investment.
Step five is the one teams skip. Without it, you only ever add. When functionality is removed, its tests go too. A test that guards nothing still costs you run time and attention.
One thing to try tomorrow
Open your regression suite and look for tests that cover functionality your product no longer has. Delete them, and put a recurring review on the calendar so you don’t need a cleanup project next year.
I go deeper on building quality practices that last in my book Agile Testing Mastery.
Want the full playbook? Read The Art of Agile Metrics →
Need this done, not just read about? Work with me.
Related
Quality & measurement in the AI eraQuality used to mean "no bugs." It can't anymore.
When AI multiplies the amount of code a team produces, the old definition of quality, code that passed its tests and its review, can no longer be verified by people reading and testing it. DORA's 2024 research found that AI adoption, while it helps individual productivity, is associated with lower software delivery stability. Quality now has to be built in at the prompt, at the review of the whole output, and in layers of testing that used to be too expensive to run.
Quality & measurement in the AI eraWe set a bug limit. The bugs disappeared. The problems didn't.
When we limited the number of open bugs per team, the teams found creative ways to stay out of the report, starting with filing tasks instead of bugs. The number looked better while the product did not. Two rules came out of it: a metric nobody understands will be worked around, and every metric needs a process behind it that helps the team that misses it.
Quality & measurement in the AI eraThe metric you use to judge people is the reason your data lies
When a metric is used to evaluate people or teams, they start protecting the number instead of the reality it was meant to describe. In AI-assisted teams this shows up as healthy-looking burndown charts built on commitments sized for pre-AI capacity. The fix is to measure the system rather than the people, and to use metrics to ask questions instead of to hand out grades.