Are Your Measures Being Gamed?

You might think your data and reports are robust and reliable, but your measures can be less than accurate reflections of the facts without much motivation or intention.  It also doesn’t take much effort to catch and gain a real sense of confidence.

Have you ever stood on the scale and jiggled around to get a more acceptable number?  When data is being collected, analyzed and communicated, it is subject to being jiggled with, adjusted or manipulated. Whenever we have an opinion on the outcome and a way to serve that opinion, the farther from the truth you should expect that number to be.

One of the contributing factors to the flop called Target Canada was bad data[1].

Business analysts responsible for the store’s supply chain were struggling with an inventory system that was new to everyone.  The data from it would be used to pack shipping containers, fill distribution centers and supply stores, but the data was missing or inaccurate.

Managers would pull reports off this system, then contact the appropriate business analyst to demand what was wrong. To avoid that kind of heat, these business analysts were able to game the system to make it look like their products were in stock.

“You cheat when the rules are flexible or not very clear and when you have a conflict of interest or reason to have a biased perception of reality,” says Dan Ariely of Duke University and author of The (Honest) Truth about Dishonesty.

It doesn’t take much creativity to game a metric, and the more creativity in a person’s job, the more flexible their moral boundaries[2].  Gaming metrics aren’t only the domain of those with bad intentions, or ill desires.  It can happen quite innocently. Information will get transformed as it is transferred from one person to another, intentionally or unintentionally.

When we are making changes, we need measures to ensure we are on track and making good progress, and that they are not easily tricked by our wishful toying around.  Whether your change is adopting a new habit or transforming an organization, you need to ensure that your data reflects the principles of great measures.

Measures are only as good as the conclusions you can draw from them.  Take for instance when you are driving.  You need to know if you should speed up or slow down to comply with limits and be safe for the conditions and congestion.  When your speedometer isn’t functioning, you might speed and get a ticket, but when your temperature gauge isn’t working, your car could overheat and leave you stranded.

You rely on more data than you realize.  An effective measurement system produces data that are accurate and precise.

Accuracy

Accuracy means the measure reflects the truth.

The reported data hits the bull’s eye.  There’s no bias, meaning it’s not always heavy and not always light.  It’s just truth, raw and bare.  Accuracy has two elements: stability and linearity.

Stability

Stability means accuracy over time.

No matter the day, the result, the crew on staff, or any other change that time can introduce, the measure is still accurate.  The system doesn’t report drift, shifts or step changes.

Scales tend to drift.  Over time, the zero reading on a scale might be more or less than that, based on the accumulation of grime and dirt, or other wear and tear.  You need to consider any effects of time on your apparatus.

Linearity

Linearity means accuracy over a range of values.

The measurement system produces data that is just as accurate at low numbers as it is in high numbers.  The range of linearity is known, as many measurement systems are not infinitely linear.

You know not to bother weighing your serving of cheese on the same scale you weigh your body weight, it doesn’t function at low weights (assuming you aren’t eating pounds of cheese).

Precision

Precision describes how close the measurements are to each other.

The easiest way to think of precision is to think of the number of decimal places.  An excess number of decimal places can imply precision, when in fact, it’s a product of calculation, such as a percent.

Precision is realistically valid.  Precision has three elements: discrimination, repeatability, and reproducibility.

Discrimination

Discrimination is the smallest readable unit of the measurement.

It can also be called sensitivity or resolution.  Think of the lines on a ruler.  Some mark only centimeters while others have millimeters also marked.  The millimeter ruler is more sensitive than the centimeter ruler.

Repeatability

Repeatability means that if you measure the same thing twice, you will get the same measure both times.

It can be further broken down into consistency and uniformity.  Consistency is repeatability over time, while uniformity is repeatability over different values.

Reproducibility

Reproducibility describes the precision of measurements of the same units by different people.  Any two people will arrive at the same measurement.

Parallax is one of the ways error happens.  If you are reading the line on a measuring cup, if you don’t look directly at it, but on an angle, you can easily misjudge the quantity.

Solid Data

Whatever you need to measure, if you follow these seven principles, you will generate solid, reliable data instead of being a pawn in someone’s game.

References

[1] (Castaldo, 2016)

[2] (Wickelgren, 2012)

Castaldo, J. (2016, January 21). What really happened at Target Canada. Retrieved from Macleans: http://www.macleans.ca/economy/business/what-really-happened-at-target-canada-the-retailers-last-days/

Wickelgren, I. (2012, November/December). Unveiling the Real Evil Genius. Scientific American Mind, pp. 26-27.

 

Please note: I reserve the right to delete comments that are offensive or off-topic.

Leave a Reply

Your email address will not be published. Required fields are marked *