We set a bug limit. The bugs disappeared. The problems didn't.

At one point we set a rule: every team may have up to X open bugs.

It sounds reasonable. A bug backlog that grows without limit is a real problem, and a number everyone can see feels like a way to keep it in check.

The teams found a way to follow the rule. They started opening tasks instead of bugs, so the problem would never show up in the reports. And that was only one of several creative ways they found.

The report looked better. The product did not.

How I knew

It was easy for me to see. The number of bugs stayed about the same, and in some places it even went down, while the code base grew every day. More code, more changes, more people touching it, and the same number of bugs or fewer. I also know the people I work with. There was no chance that number showed a real picture.

That is the method, and it is simple: put the number next to what you know about reality. If the metric moves against everything else you can see, do not trust the metric. Trust your eyes and go look.

Why the number lied

Nobody here was a villain. Look at the rule from the team’s side. Before the rule, opening a bug was a normal part of work. After the rule, every bug a team opened was a mark against that team. The rule did not reduce the number of problems. It changed what it costs to report one.

When you punish the signal, you get less signal. The problems were still there, and now they lived under another name, in a place the report did not look.

Drucker, Parkinson, and a very old pattern

You may know the quote that is attributed to Peter Drucker: “You can’t manage what you can’t measure.” I agree with the spirit of it. But when the thing you measure is people, there is a catch: people who are measured will do what it takes to meet the goal. The measurement itself starts to set the rules of the game.

In my books I call this Parkinson’s law for measurements. Strictly speaking, Parkinson’s law is about work expanding to fill the time available for it, so this is a stretch of the original. The pattern itself is much older and much better documented. The social scientist Donald Campbell wrote in the 1970s that the more a quantitative indicator is used to make decisions, the more it will be pushed to distort the thing it was meant to monitor. The short version of it, the one Goodhart and Strathern are known for, is in my previous article, so I will not repeat it here.

Whatever you call it, it shows up on any team where a number decides how people are seen.

It goes in both directions

Gaming does not always mean hiding things. Sometimes it means doing too much of what is counted.

Measure a tester by the number of bugs opened per day, and you will get a lot of bugs. Nobody will take the time to check whether each one is real. Measure by the quality of the bug report, and the same person may spend the day documenting every small detail. It looks good on paper and adds nothing to the product.

In both cases, the person is not doing a bad job. They are doing exactly what the metric asked for, and that is the problem.

What I did when I found out

My first reaction was to remember something about human psychology: people do not like to be measured. And if people cheat a metric, I need to understand why. What do they think will happen to them if the number looks bad?

So I did not answer with a stricter rule. I talked with the teams. I wanted to understand what they were afraid of, and I wanted to show them how the metric is actually there to help them.

The metric stayed. What changed was how we communicated it. Before, it was a number with a limit. After, it was a number with a reason, and a conversation about what it is for.

It took time. But the teams started to align.

Two things I took from this

1. A metric is only a metric. If it has no value, and the people who are measured do not understand what it gives them, they will not be part of it. They will work around it. A number that comes from above, with no explanation of what it makes possible, is just a rule to get around.

Before I approve a metric, I ask a few plain questions, and these come from the book:

• Does it answer a specific question or serve a specific goal, or does it only look good to senior management?
• Is it easy enough to measure? If it is hard to measure or hard to understand, it will most likely fail to give the benefit you expected.
• Is it one piece of the puzzle? Even a good metric on its own does not give a balanced picture.
• Does it pay for itself? Think about the return before you add it.
• Does it show a trend for one team over time, instead of comparing different teams?
• Will the team embrace it? Teams are already under pressure, and a metric forced on them will not work.

2. Every metric needs a process behind it. This is the one I think people skip most. If a team does not meet the number, the team should be shown the plan, or given the tools, to deal with it. Otherwise the number works like a threat. And people protect themselves from threats, usually in creative ways.

A good process behind a metric answers one question before the number is published: if a team misses this, what happens next? If the honest answer is “it shows up in a report”, you have built a grade. If the answer is “we sit with the team and look at what is in their way”, you have built something that can actually improve things.

My own mistake

I made a mistake early on, and I want to be honest about it. I added a lot of simple metrics, and they never let us see the full picture. Piling up numbers is not the same as understanding.

I learned that with experience, not from a book. Today I never rely on a single metric. Every process I build has several metrics, and each one looks at a different aspect. The combination is what gives the most real picture I can get.

That also protects against the story in this article. A single number is easy to work around. It is much harder to improve several numbers that look at the same work from different sides without improving the work itself.

One thing to try tomorrow

Take one metric your teams report, and answer three questions:

1. What does this number let the team do? Write one sentence.
2. What happens to a team that misses it, and what help do they get? Write one more.
3. Look at the neighbors. If this count went down, did a similar count go up somewhere else, the way our bugs turned into tasks?

If you cannot write the first two sentences, the metric is not ready. If the third question has a yes, it is probably already being worked around. Do not answer with a stricter rule. Ask the team what they are afraid will happen if the number looks bad, and explain what the metric is there to do for them.

The Art of Agile Metrics goes deeper into how to choose metrics that make a difference and how to introduce them without losing the team. Read The Art of Agile Metrics.

Related: The metric you use to judge people is the reason your data lies