Managing "Measuring"

Neil Raden's article, "The Limits of Measurement", bring us to the interesting problem of not being able to see the trees for the forest.

In this case, "performance" is the forest, and the trees are the events and inputs that make up successful explanations for the performance observed.

Measurement looks for data points and then it looks for patterns or "forms". We assume that the points themselves don't tell us what we need to know -- but when data collectively mean something different from the data points, we have to hope that the points are resolved into a focused picture and that the picture means something.

On the other hand, a preconceived pattern or form discourages finding the data points that don't fit. In measuring performance, this can turn into the difference between fact and myth.

I.

We have enough other problems getting accurate data without preconceptions clouding the view... Through interruptions and reprioritizations, ongoing change alters the routines of action that are considered necessary in day-to-day operations. So, by analogy, what if the key points keep moving? Chasing these moving targets, can the picture ever be focused? How far away do we have to get to see a "stable" picture?

Neil's article gives important reminders about the Heisenberg Uncertainty Principle, and they tell us that we have to at least estimate how much our data collection process -- e.g. measurement -- might either lack "accuracy" or exceed a useful amount of it. Otherwise, we risk simply accepting distortion that we created ourselves.

The distortion may be unintentional, like a manufacturing defect discovered only later. But worst case, you know the old saying, "statistics don't lie -- statisticians do."

II.

Furthermore, "accuracy" and "usefulness" are vulnerable not only to the measuring instrument and measurer; distortion can begin with the definition of what is being measured in the first place. These definitions are not absolutely right or good; they are debatable.

Comparing one organization's practice to another's yields proof of this. It's like two contestants in a recipe bakeoff. Two different organizations, pursuing the same outcomes, might have widely divergent definitions of what data comprise the key points that compose the bigger picture. Thus, on the surface, they might be trying to work with "the same" picture -- but they might disagree on the significance of any included point of measurement or of any omitted point.

The competitive ramifications of course make it all more interesting. Even when two organizations work from the same general framework, the diversity and uniqueness of their particulars could mean, as the saying goes, that "on any given Sunday, any team can beat any other team..." That is, the differences are not necessarily advantages, so we might decide that the definitions behind the difference are not the right ones after all...

III.

And finally, won't a different point of view on the data also result in a different picture? Strictly speaking, we'd have to expect a "definite maybe"... and the reason why is that we might not see the same data from a different point of view.

Since our management already leans on ideas like capability maturity, best practices, and other examples of empirically proven improvement, it seems that we already know about the need for measurements to differ according to modes of achievement as well as targets of achievement.

These improvement concepts are also ideas that encourage us to adopt standard answers to questions like "Are the right things being measured?" and "How do we know what the right things are?"

Being essentially scientific discoveries, these standards reach our management as a kind of Newtonian physics, setting an operational baseline and perspective. Our practice of it is then challenged by how sensitive our measuring instruments are. When our measurement sensitivity hits the point of diminishing returns, we want to go beyond. But how?

Beyond that is a different challenge: an alternative, more quantum-physics perspective of multiple concurrent points-of-view.

IV.

In our performance improvement efforts, organizations get pretty good at following prescriptions for action, and although allowing that our measurements interfere with the actions taken, the actions are not significantly varied from one time to the next. We're used to activity being measured within ranges of tolerance. As a result, we think of value from a distance that accommodates variance.

But let's look at this idea of "significance" ... If we bother to measure a prescribed action because it represents a certain value to be generated, we want the action to avoid getting caught in a cloud of varying kinds of measurement, each kind approximating the action from its respective point of descriptive view. Why? Because each perspective's description can virtually propose a different action, and until there is agreement amongst the perspectives then their collective measurement leaves us somewhat unsure about what action really happened.

For example, was a certain higher-than-expected expense a "detrimental cost" or a "beneficial investment"? Was a faster-than-expected delivery an opportunity or a disruption? Such varying views of "facts" lead to uncoordinated organizational responses and reactions that defeat the purpose of measuring things.

This emphasizes the fact that, while management likes precision, it really must have relevance even more -- regardless of precision. A fuzzy (or crude) picture of the right thing is more valuable than a clear picture of the wrong thing.

Having multiple stakeholders means having multiple points of view. This condition will show that interpretation sets the limits of measurement.

V.

A well-worn story related to this problem is the one of Six Blind Men and an Elephant. Touching the elephant, each man relies on his own perception of the part of the elephant he is nearest, and each part touched is very different from the others -- a tusk, an ear, a leg, etc. Consequently no one man figures out that what he is touching is an elephant, instead of a pole, a tent, a tree, etc.

That's an interesting story, but if our problem is to make a better elephant, then the story goes inside out. Instead, if all six men were first told that they were about to touch an elephant, would they each then be able to determine what part they were touching and assess the status of the part? Yes, the information-in-advance mitigates their individual blindnesses to a very important degree, but this works because they were not misdirected -- not told to expect something other than an elephant.

For this reason we can say that an organization's performance is driven more by agreements before-the-fact than it is by measurement after-the-fact. In fact, agreement should create the model from which measures are derived. But then, what we need measurement to do is to feed back and drive future agreement as well, across the inevitability of ongoing change... and this will depend on managing interpretation in the organization.

Posted by Malcolm Ryder at 9:39 PM | Comments (0) | TrackBack