« The Business Value of IT Operations | Main | Justifying Performance - Ambition versus Reality »

September 15, 2005

Assessments versus Measurement

In the growing body of literatures regarding performance improvement, the usual recommended starting line is the advice that "you can't manage what you can't measure."

But this allows a troublesome add-on thought, which is that your measurements are going to drive your management mainly in the fashion of "garbage in, garbage out"... The quality of input data is of course a risk, but what about whether the correct kind of data is being absorbed in the first place? Great data of the wrong type will lead nowhere fast, or to the wrong place too soon.

For that reason, picking the right measurements would need to be the logical starting point -- and updated thinking emphasizes that to do that there must be a model or a logic of what matters, not just of what "counts".

What's the difference between what matters and what counts? Observations probably count when they are able to show that they are usually associated with the subject. But showing how their association plays into the importance of the subject at hand is how they mean something. Getting from what counts to what matters means getting from findings to clues to evidence.

I.

As the initial concern of a management process, the notion of "measurement" most top of mind is usually metrics . These metrics are really particular ideas variously used like sculpting tools and fishing tools depending on the circumstances.

But the importance of a higher-level guiding model displaces measurement from that initial position. Metrics gets bumped even a little further away from the starting line by an additional issue as well -- the need to evaluate .

What is the difference between evaluation and measurement?

Typically, measurement determines a state, but evaluation determines the importance of the state. We might determine that it is 28 degrees outside, but having observed the time at which it was true, what does it mean to us? Are we sick? Is it July? Does the temperature matter? How?

But think of it another way as well -- as the difference between a subject and a topic. The subject is what is being talked about; and a topic is a way to talk about an aspect of the subject. Evaluation considers the subject; measurement considers a topic.

II.

Overall, evaluation emphasizes that you not only have to pick the right topics (measurements) for your subject, but you also have to pick the right subject -- which really means to not lose sight of it...

Our friends at the McKinsey Quarterly interviewed John S. Varley, the CEO of Barclay's Bank, who as the new guy in charge had plenty to say to his organization about the difference between what their financial topics appeared to mean versus what they really meant. The short version of the story is that Barclays had fiercely and successfully pursued enviably upward financial results, but largely at the cost of running the company down.

Noting that the organization was managerially unconscious of its "go for broke" mentality, Varley measured the financial performance as positive but he assessed the performance as negative. He saw that the way the big numbers were being hit was actually not sustainable -- in fact, was unhealthy -- for the company.

III.

Assessment takes the broad view of evaluation. It brings perspective to measurements by not allowing them as a topic to overwhelm the subject.

One of the greatest examples of this perspective is Muhammed Ali's famous "Rope-A-Dope" strategy against George Foreman in the 1974 Kinshasa Zaire Rumble in the Jungle. Scorekeeping might have shown Foreman obviously winning round after round on the punch counts, but as the CNN/SI writers put it, "by the eighth round, the 25-year-old champion was running on empty. Ali took advantage to knock out his exhausted opponent..." Ali's bigger picture of the fight, which he'd had at least since the second round, made him the winner.

Another interesting example of the bigger picture is in the Voter Referendum publications in California. Before voting takes place, each issue on the ballot is discussed in printed form with the following points of view:
- arguments for the issue
- rebuttals to the arguments for
- arguments against the issue
- rebuttals to the argument against

For many people, it's a real eye opener to see four different ways to consider the issue. Each point of view is a topic, and each topic is typically fairly persuasive on its own. Taking any one of them, a person might "measure" the ballot issue positively or negatively. But to assess the overall value of the ballot issue (the subject), the four points of view are compared against each other, and so a decision can be made "on balance".

In other words, that assessment model is the key to understanding the real importance of any of the subject's topics.

IV.

When we say that we can't manage what we can't measure, the appropriate response is not simply to start measuring things; the amazing ease of taking things out of context shows that measurement does not equal management.

Instead, by bringing a model to the situation -- one that can define and hold a coherent description of what matters at the subject level -- we get to better observations and decisions using measurements.

It's more fair to say that we can't manage what we can't model -- because without the model, we don't know what is the meaning of the measures and thus don't get a meaningful picture of what we're trying to manage.

All that said, why is it that most times the advice to companies is still to start with measurement?

V.

For one thing, all companies have measures. It's the common denominator. It seems to make sense to "make better use of what you already have..." Often the company doesn't really know another way to talk about its management.

Second, it's a reflection of the need to first actually confirm that a usable picture is obtainable. The real prod there is to investigate how the picture is being taken now. Generally, it's been found that omissions, confusion and inconsistency is significant there, preventing what might be considered a reasonable level of visibility, or producing what might be considered a picture with no apparent composition.

But an even deeper concern of that advice is adherence to the idea that "management" is mainly about efficient production -- one of the top target outcomes of measurement efforts. This is the Frederick Taylor school of scientific management. But the most important difference between 1911 when this school opened, and 2005 well after "hypercompetition" set in, is that the pace of change today is orders-of-magnitude greater and has changed the priorities for measurement -- from accounting for doing the work right, to accounting for doing the right work.

Especially in our era, ineffective efficiency is waste. And that's why deciding what to measure is more important than the measurement itself.

Punchline: in order to manage better, start with assessment. Organizations will find that the key representative concept behind an assessment is an objective.

Posted by Malcolm Ryder at September 15, 2005 12:07 PM

Trackback Pings

TrackBack URL for this entry:
http://www.malcolmryder.com/cgi-bin/mt-tb.cgi/129

Comments

Post a comment