" />

« October 2010 | Main | March 2011 »

November 29, 2010

Rational vs. Logical Measures: What Matters vs. What Counts

Managers always think about metrics, and metrics are always about distinguishing important things from unimportant ones. That said, metrics are famous for going awry. This occurs in a few high-impact ways:

  • Poor quality (completeness, accuracy)
  • Poor continuity (frequency, duration)
  • Poor semantics (definition, context)

While most missteps in measurement can be traced to one or more of those three issues, two additional and potentially dramatic problems complete the list.

The first is in not realizing that both signal and noise must be measured, even though the operational goal will always be to minimize noise. This requirement exists nonetheless because understanding noise is the key to removing it; we use our understanding of it to find it and counteract it, leaving less to obscure the signal. The most famous and complex of this problem is known as the Observer Effect, which is also associated with the Heisenberg Uncertainty Principle. In short, the measurer himself is the noise that must be mitigated.

The second is even more of a requirement: avoiding confusion between rational measurement and logical measurement. Far from synonymous, the rational and the logical have very specifically different jobs.

Logical measurement is done within an already established framework, and its purpose is to grade the characteristics and relationships of components that have been presented as the necessary parts of an already defined whole. Therefore, if you have a model for something, what logical measurement does is to test the integrity of the actual state of things in terms of the intended state of things. Essentially, managers pursue validations with logical measures, and confirmations established by counting grade points covers a huge percentage of the information needing to be processed.

Rational measurement has a completely different function. Relative to logical measurement, the purpose of rational measurement is to determine if there is a model that makes sense of the prevailing or expected conditions. Rational measurements, in other words, are not aimed at manipulating conditions to conform to an existing model. Instead, rational measurements may show that the model is irrelevant to the prevailing conditions. Rational measures can find a pattern in the available information, and the pattern may thereafter be refined into a model. The refinement may also conflict with or even obsolete some prior model that has usually been associated with the circumstances in which information was gathered. Rational measurement discovers what really matters regardless of presuppositions, as opposed to dealing with only what counts within a presupposition.

Scientific evolution is essentially rational, while scientific application is primarily logical. This observation is important to, for example, the notion of innovation, where value is generated through new combinations of ways to do things with effects to produce. "Newness" in methods or effects is born of rational measures, although the sustainability of the new is often critically dependent on logical measures that can generate a return on the investment made in the new.

Meanwhile, it will turn out that logical measures may disprove the sustainability of an existing model (including a new one), but they are unlikely to show what model should replace it. Rational measures will be the gateway to the even newer model.

We tend to think of rational measures as a part of what is called "creative problem-solving", as if that effort is something that occurs outside of the normal mandate for efficiency in management. Looking at the efficiency notion correctly, however, exposes that it is dominated by a concern for resource accounting within an accepted model, dedicated to sustainability. Naturally, this only makes sense in environments where being really good at staying the same is the best way to be appropriately effective. But the fundamental challenge to management is not to stay the same; instead, it is to stay effective -- and that means being able to change when appropriateness itself has changed.

The most obvious familiar example of that idea is the ultimately unpredictable series of changes that are made by a coach during a sporting contest. Success is predicated on things that reduce variables and simultaneously enhance adaptability. Management attends, correspondingly, to the risks of certain outcomes, and to the evidence that it it is time to make a change -- allowing existing capacity to be effective. The punchline may be that rational measurement is better at coming to grips with what is actually under the control of management. In situations where there is a strong competitor, a volatile environment, or both, it is les likely that a logical focus on efficiency will have meaning except under the guidance of a rational focus on effectiveness.

Rational measures are the ones that focus on the kind of change that makes or breaks the relevancy of models, as opposed to on variances that prove or disprove incidental compliance. From this, we can see that both strategy and innovation are essentially rational, not logical. Strategy and innovation must both look beyond what is internally logical and desirable within a model, and this will include considering much of what logic tries to ignore or discard.

 

 

Posted by Malcolm Ryder at 11:41 AM

November 10, 2010

The Form of Performance

For the uninitiated or the careless, Business is thick with disinformation.

The most rampant and disturbing example is the use of the term "transparent" to mean "visible" -- which of course is, well, silly, since transparent things are by definition quite hard to see.

Yes, yes, we know how this came about: things used to be hard to see until the obstructions became transparent. Removing obstructions has been hard and really useful. This doesn't make the current misuse of the term "transparent" less silly, but it points to a certain sloppy earnestness in business conversation, reflecting a need to be associated with some idea taken to be universally important. It may be that no one who is important in themselves needs to use the term "transparent" for any reason, having as they might a vocabulary that lets them correctly use words like "evident" or "visible".

Evidence can be deceiving, however. Take performance measures, for example. We have recently run across enthusiasm for the idea of "rubrics". Rubrics have the combined cool factor of secret decoder rings and great advice on technique. Simplified, a rubric is a list and a scope of target characteristics deemed significant. A rubric offers the chance for all who will use it to shoot for the same high grades, since the rubric says what to be like in order to get the high grade. It's like being given the answers to the test before the test is given. The road to success becomes highly ... visible.

So then, what's not to like? Wouldn't a rubric level the playing field by stating in advance how everyone is held to the same standards?

The answer is, why would it? A rubric on its own may be inherently flawless, but a rubric is pointless except in how it is used. As performance criteria, the terms of the rubric may have the virtue of consistency,capable of giving the same results from the same test over and over. But it may have the flaw of treating apples and oranges as if there was no difference between them. Terms that reveal a great orange may very well "reveal" a bad apple.

As a way to see this clearly, imagine two soccer teams, one conissting of high school players and one of college players. Each team has the same structure, made of the positions and roles that players can fill. A pretty straightforward rubric, used in evaluating performance, will point out that the college team is invariably better than the high school team. The post-game stats are going to look pretty bad for the high school team. But that measurement is dis-informative, because by ignoring the context of the respective teams, it fails to take into account that the high school team might be excellent amongst other high school teams, while its poor performance relative to a college team is trivial regardless of how factual it may be.

This becomes a potential issue when rewards are tied to the trivial. In an organization where performance evaluations are needed not only for assessment but for motivational purposes, rewarding trivial differences born of inappropriate comparisons is not only a good fake-out of management staff, but it is a good de-motivator of non-management staff.

Again, even at risk of being mis-used, a rubric has inherent virtues. But not only must it be context-sensitive to what it measures; also, the rubric itself exists in a larger functional context -- sharing influence with at least two other key instruments.

The other instruments are oldies but goodies: non-prize compensation (salary) and positions (titles). These go along with the key important offering of the rubric evaluation: recognition (reputation). High-performance in these three dimensions offers, respectively, the possibility of raises, promotions, and bonuses. But note that in this arrangement, rubrics should not determine salary or titles!

The relationship of compensation, position and recognition is this: they don't need each other to be respectively evaluated -- but each one may be seen as circumstantially influencing each other. The challenge for management, in understanding context,  is to identify the circumstances that relate the various dimensions, while not confusing them with each other.

Clarity requires that we have a fundamentalist view of the three issues. That is, what is the purpose of each, which then allows its assessment to reveal some form of excellence that should be rewarded?

Putting it bluntly, the effort spent on maintaining each dimension looks for an impact that would be seen as a Return On Investment. A good ROI merits a reward.

Salary -- this is about a slice of total resource capacity that should be competitively affordable to retain. Retaining the resource assures structural integrity in operations. In this dimension, high performance is about affecting organizational stability and potential.

Title -- this is about a definition of a boundary for a role. Setting the role boundary affects the potential interactions with and of the structural components. In this dimension, high-performance is about affecting organizational behaviors.

Reputation -- this is about attraction, in a magnetic sense. Cultivating attraction affects the willingness of parties to co-operate. In this dimension, high-performance is about affecting the real-time resourcefulness of the organization, which in effect generates its agility in its environment.

With those independent values given, what are the significant relationships between them?

  • Salary and Title together anchor organizational Availability.
  • Title and Reputation together anchor organizational Capability.
  • Reputation and Salary together anchor organizational Quality.

This perspective makes it somewhat obvious as to how a performance evaluation should look for critical contributions from the party being evaluated. Along with that, there is a reasonable expectation that raises, promotions, or bonuses recognize high performance (ROI) in the dimensions of compensation, position, and recognition, respectively. And earned Reputation, which associates with Rubrics, is a primary factor in quality... secondary in capability, and a non-factor in availability.

Even intuitively, context makes sense to us. We feel like rewarding someone who did something remarkable from a starting point of being lesser paid, junior level, or previously unnoticed. And naturally it is less surprising if something notable is done by a party that started out being higher paid, more senior level, or well-known -- mainly because those are conditions in which we already have greater expectations of something notable being done. Meanwhile, at any given time, some desired outcomes have higher priority than others and will therefore be linked to greater rewards.

But in the end, even the more accurate and relevant perspective above will not save someone from being mis-evaluated due to their having been put in a position unlikely to foster the high-end achievements.

So, what a rubric must do is find its place in the context of the evaluated party's opportunity to perform -- and clearly state what kind of ROI it is trying to both invoke and detect by looking for certain characteristics within that opportunity. Rewards should be about exceeding expectations that are reasonable within the opportunity.

 

 

 

Posted by Malcolm Ryder at 11:22 PM