A Metric Misused or Misunderstood Is Worse than No Metric at All

At LNS Research we believe in the expression “you can’t control what you don’t measure.” With that, we've focused a significant part of our research on identifying the “Metrics That Matter.” And one of the topics of most interest to our Global Executive Council is benchmarking and understanding how to measure performance.

Follow @Dan_LNSResearch

In a previous post on a key metric used by many manufacturers, Overall Equipment Effectiveness, or OEE, I explained why OEE was an excellent tool but had its limitations. One point I made was that it was often misunderstood and then misused. This begs the broader question, though: What are the risks of misusing metrics? We'll explore the answer in this post.

The Danger of Metrics for Metrics Sake

Very early in my career I was the lead process control engineer at a brand new state-of-the-art sawmill that was just going through its start-up. This particular mill had a brand new downtime measurement system that would detect when a machine center was down and prompt the operator to enter a downtime cause code. To ensure that the system could capture control system failures it was decided to install redundant sensors to detect machine center stoppages.

One of the machine centers in a sawmill is a debarking station. In the debarking station, tree-length logs from the forest are run through a machine that scrapes the bark from the tree. One time a tree had an undetected soft spot which caught one of the rotating scrapers. The tree ended up thrashing around knocking all of the photocells, which controlled and synchronized the machine out of alignment as well as the downtime system photocells.

Amazingly, the mill manager insisted we fix the downtime system cells first so we knew how long the machine center was down instead of fixing the system so we could resume production. The metric had become king and we lost valuable production time for the sake of a metric that was going to be inaccurate since there was no way to capture the real reason we were down for part of the time–fixing the metric system!

The lesson here is not to let the measurements of the system degrade system performance just so you can gauge performance, since you aren’t really measuring what the system is truly capable of delivering.

Tweet this tip

The Danger of Miscalculated Metrics

As I pointed out in my OEE post, OEE was specifically designed to measure performance at a machine (or machine center), yet many organizations are trying to use it to measure an entire production line's or plant's performance. They do this by taking the total productivity of the plant and crossing it with the plants average availability and the aggregate quality number. These measurements are so broad that while the number is calculable, it most likely gives you no insight as to what is actually occurring in the plant.

You can calculate a total plant or line OEE but you should treat it as a network calculation much like you would calculate resistance in a complex series-parallel circuit or fluid flow in a complex hydraulic circuit. This way you can see what is happening at each element in the plant and begin to determine where problems actually begin as they may in reality be cascading and making things look worse than they really are.

The lesson here is that over-aggregation of metrics can mask or even exaggerate the initial problem in a process.

Tweet this tip

The Danger of People and Metrics

One of the greatest dangers with metrics is that people can often influence them, especially when those metrics are used to reward (or punish) those same people.

Cem Kamer is a noted author and researcher on the subject of software testing. In a report by Doug Hoffman which heavily cited Kamer’s work, particularly with the Los Altos Workshop on Software Testing, there were numerous examples where the metrics tended to match the predicted targets because people were manipulating the data.

One such metric was defect rates during testing. If the number was too high, the developers were “punished” for writing bad code and if the number was too low the testers were “punished” for not doing a good enough job of testing. With this lose-lose situation it is no wonder the defects found during testing exactly matched the target.

The lesson here is that if you reward or punish people based on a metric, and they have direct influence over that metric you can expect it the measurement to pretty closely match the target regardless of what the real situation is.

Tweet this tip

A Metric Misunderstood is Most Likely a Metric Misused

In his blog post on the 6 Myths of OEE, my colleague Greg Goodwin points out some misuses of the OEE metric. Unfortunately, most metrics suffer from the same risks as OEE. If they are misunderstood they are most likely misused.

Before organizations start managing by metrics and using those metrics as the primary means of managing the business, they need to understand exactly what those metrics mean, how they are calculated, and the risks associated with misusing those metrics.

As your driving to work and the speed limit sign says 60 MPH and your speedometer says you are only doing 40 MPH, it might be wise to consider that with all the snow on the road maybe things aren’t as slow as they seem.

Tweet this tip

Top 3 Ways to Misuse Metrics:

1. Have too many metrics – Most likely people will end up spending too much time measuring, taking them away from doing. Even if the metrics are all collected automatically, too many metrics will make it impossible to see the forest for the trees.

2. Choose arbitrary metrics – The metrics must match the process and actually provide meaningful insight into performance. If the metrics provide no insight as to what direction to take to improve, they are pretty meaningless.

3. Never change or adjust metrics – Business do not operate in a static world. At the very least you should adjust the target values of your metrics based on the changing environment. You should also be prepared to use different metrics if the environmental changes are significant enough. Twenty years ago when oil was cheap and climate change was virtually unknown, a process' carbon footprint was not a meaningful metric. Today that is not the situation.

Those are our thoughts on misusing and misunderstanding metrics. Tell us what you think in the comments section below.