August 2, 2008

Antiquity in the Garden of Good and Evil

Scientists unlock mystery of 2,000-year-old computer

As reported in 2006 by the CBC (i.e., Canada, yesterday) a then-recent study in the journal Nature had revealed the device known as the Antikythera Mechanism to be actually a complex means of tracking the movements of astronomical bodies for use in navigation.

As reported in 2008 by the American newspapers (i.e., USA TODAY), the same device, known as The Mean Sun Wheel, held 30 bronze gearwheels marked with instructions, allowing the user to link the cycles of the heavens to "the very mundane Greek games" (i.e., the Olympics).

Tthat particular usage, scholars believe, was primarily by wealthy sponsors of the games, for scheduling purposes -- putting the wheel in the same class of "information technology" as satellites, but not in the same class as television broadcasting, with which advertising spawned the Olympics Hype Cycle -- used for tracking the movements of earthly bodies -- nowadays far more important than a mundane thing like weather.

The only question here is already asked and answered: whether the 2,100 year old Mean Sun Wheel, given that it still works and all and doesn't even need batteries, can hold its own against a much bigger machine: marketers. Marketing's vanguard, the American press, took two years to find something in it that rated worth mentioning again. Maybe it's time to move to Canada.

Posted by Malcolm Ryder at 11:02 AM | Comments (0) | TrackBack

July 9, 2008

What's In Your Portfolio?

For providers (instead of consumers), Portfolio Management is a robust and widespread discipline that has meaning which crosses industries and departmental functions. In short, it organizes opportunities deemed to be beneficial into suites of categorized commitments that make the opportunity "actionable" . But portfolio management is most often associated with related efforts that represent either the authorizations of the action, the methods of the action, or the customer of the action -- in effect tracing the run from supply to demand. The efforts articulating this run are, respectively, programs, projects and solutions. One confusing aspect of the way these efforts are supported is that portfolios are mistakenly thought to be components (or "children") of programs and supersets (or "parents") of projects. In fact, that is an erroneous association: instead, as illustrated below, a portfolio is a model that relies on the other three efforts to be actualized. Further, it is the interoperations of these efforts that powers and stabilizes the portfolio.

Why is portfolio management often misplaced amongst these efforts? There are two predominant reasons. For one, practitioners of these efforts often mistake scorecards and dashboards for portfolios. And two, portfolios are often pursued under "performance" requirements (i.e., requirements to increase the rate of return on equity), whereas the actual purpose of a portfolio is to provide a model for the commitment to the opportunity, defining how value will be recognized, not how "value will be generated and captured".

The language that helps to understand where portfolios help goes like this: "what is the benefit of the investment model?" Obviously, one model could be modified or even discontinued and replaced, while still addressing the same apparent opportunity. At the least, this simply acknowledges that two competitors may chase the same prize in different ways, with both making progress (without predicting which one will prevail or even whether one necessarily must). But within the model, other key actions are generally positioned as catalysts or governors -- including things like identifying a distinctive market niche and specially producing for it, tracking the cost of scaling up for the demand level in that niche at a given quality benchmark, and exercizing policies to keep decisions and approvals predictable throughout changing circumstances -- all relative to a certain type of enabling stakeholder who is the primary beneficiary.

Posted by Malcolm Ryder at 9:21 PM

July 5, 2008

Beyond the Spin: Measure What You Give

Does your organization really measure what you give, or does it mainly spin what you measure?

Bruce MacEwen's industry-leading website Adam Smith, Esquire offers an opportunity to gaze into the abyss of metrics and walk away without jumping. In the article
"How High Quality Are Your Lawyers? (How Can You Tell?)"
a close reading shows contrasting business models contesting notions of "performance @ cost" and "value @ quality". In the competitive situation covered, one upstart model strategically goes after a chunk of the opponent's business by bringing customers the performance/cost equation, surprisingly leaving the traditionalist competitor to justify how pricing for that same chunk of business could rationally be based on value/quality. What makes this all interesting, notes MacEwen, is the idea that 99% of what the traditionalist does is what the upstart can steal away.

For those of us who fell out of the old hot habit of saying "disruptive innovation" once a month, this looks like news, but not new news. Still, there are some fresh perspectives worth bringing to this contest.

As seen in the diagram below, the different models above are easily distinguished by what they actually offer, making it inappropriate (for managers) and intellectually dishonest (to customers) for either of them to masquerade as the other. Customers buying into cost/performance are investing in the promise of efficiency, while those buying into value/quality are investing in the promise of reliability.

In MacEwen's article, we are sensitized to the problem that high-prestige value/quality law service firms institutionalize a significant unmanaged cost in the form of "available overachievers", against which these firms then build a hedge by charging premium prices beyond rational evidence of economy for the customer. But what is sold as the justification for this pricing? Their quality?

To be sure of avoiding management posturing, "quality" here must mean only one thing: adherence to the promised appropriateness of the deliverable versus the stated need. Consider that meaning against the question of what it takes to get quality: the value/quality firm proposes that by exceptional capability they eliminate the risk of not getting quality. Therefore, the key variable that this firm actually addresses is unpredictability in the customer's need. As an operational tactic, the value/quality firm hoards talent in order to avoid outsourcing and to presume agility.

But the cost/performance firm basically argues (by demonstration) that legal work requires only competency to sufficiently meet most stated needs -- not a matter of being exceptional but instead simply correct for the task, which eliminates unnecessary effort from the equation right off the bat. Of course this presumes a degree of predictability in scope of need -- and agreement on the scope becomes the main feature.

The discussion above intends no effort to offer a wisened critique of law firm strategy. That said, on the surface there are no truly important differences between marketing professional services in law versus other disciplines where subject matter expertise is the raw material and advice is the product.

Idiosyncracies in the legal services industry will of course provoke distinctive problems and solutions there, yet these are probably driven more by the state of mind of the customer - which is the underlying important difference because it is the competitive arena. Oversimplifying MacEwen's article, the difference between the value/quality firm and the cost/performance firm is that the former sells confidence while the latter sells credibility.

Are there spats? One accusing the other of con games, and the other accusing the first of being incredible? MacEwen's article says yes; but what is further interesting (per evidence of the illustration above) is the opportunity that both types of firms can objectively profile themselves on common ground (efficiency, capability, reliability and acceptability) -- and use those profiles to determine how to optimally segment and grow a shared market. When they don't do that, you can bet it isn't because the customers don't care.

Posted by Malcolm Ryder at 9:59 AM

June 18, 2008

When is "value" not valuable?

A wonderful discussion on Bruce MacEwen's website Adam Smith, Esquire included this challenging note from Paul Lippe about what logic is available to explain the connection between quality and value. While he questions "reputation" as an indication of warm fuzzies like "quality", he also kicks off his note citing the less fuzzy implication that better performance presumes to justify higher price:

"I'd be curious if anyone can come forth with any data to show that in fact (as opposed to in repute) more expensive law firms produce better results, e.g. can it be shown that the investment banks who had the largest losses on their mortgage portfolios were served by lower reputation law firms?

Once this conversation settles down, I will start a separate string (and perhaps a wiki to really pull something together) on what I consider to be the core issue: how can we develop a definition of VALUE in legal services that is meaningful and useful, and not simply measuring inputs like hours spent, diligence of lawyers, law school attended or reputation of the firm. With such a definition of value, I think we could expect that some lawyers' reputations and income would go up, but some would not."

Let's dig into that overall observation by making the undercurrents obvious.

1. "Value" is a label for the significant distinctions attributed to something. "Value" in professional services is 3-dimensional, at minimum. A certain method of co-operation with the customer interacts with a certain type of target outcome at a certain level of effective cost to the customer. The method, outcome, and customer cost are variables, each having a range of acceptability, which in turn allows some universe of acceptable overall impact to sprout from their combination. Now, from that dynamic, some professional service providers are great at being predictably consistent within a smaller universe (range of impacts) that the customer prefers. Some are great at being agile enough to cover a larger universe, keeping up with a customer who has more volatile preferences. And there are several other "flavors" of competency that a service provider may have. Ultimately the provider wants to be paid for the competency, and then be paid even more for a competitively greater level of competency. But the customer wants to pay for customer satisfaction, which is something different. And what mediates the balance of the two things is often just culture. I wouldn't choose to drive a perfectly good Tercel to the White House Christmas Ball, but I could; and I wouldn't choose to drive a Bentley to the 7-Eleven, but I could. In fact, I could use either car to get to either destination.



2. That's all well and good in theory, but in practice the realization of the potential value is hugely affected by the ability of the customer to appropriately and effectively align to it. (There is even plenty of historical evidence that customers sometimes buy based on how they wanna be seen, not based on how they really are.) That reality is the "forest". Relentless pursuit of profit is the bulldozer that strips the forest. Atomic metrical inputs like law schools and hours spent risk merely being "trees", where excessive attention obscures the view of the forest and therefore obscures the proper understanding of the value.
Profit and arbitrary metrics actually must not dominate an analysis of value. Instead, value, properly identified, can be correlated with profits and other interesting measures, and the correlations may be revealing or even exciting.


3. The final point from the above is that it is probably important to use rigor in discussing value, because "value" is not a reliable synonym for other things that deserve their own names, such as "competency" and "satisfaction", and "culture". It's important to know what is actually being taken into consideration and not gloss over things for convenience, because otherwise we find out too late that we're actually sitting on some key coordinate that does not allow us to "get there from here" (i.e., to the necessary value) on time. Meanwhile -- if we would like to elevate the discussion of value from the 3-D space of CustomerCost /Outcome/Method to the 3-D space of Competency/CustomerSat/Culture, while remembering to map the current coordinates in both spaces, well that's fine.

Posted by Malcolm Ryder at 12:17 PM

April 26, 2008

Inciting Insight

In the big picture of purposefulness versus accidents, management accommodates a coincidence of things that ought to make a difference, such as designs, events, and states. Information systems grind on each of these, going both deep and wide. But what matters is whether management orchestrates a convergence of those things. Is such orchestration forcing the issue, or is it simply the way these things actually turn out to have meaning?

It's customary to eschew information overload; but the key to their useful combination is not the specific information compared, rather how the available information is positioned in the overall scheme of interpretation. As seen in the picture above, intents and impacts which superficially represent "how things are going" will relate in terms of the "5 W's and How". Seeing the certain blending of factors here, it is easier to realize that most insights will be moments of correlation that are the prize for maintaining ordinary but diligent awareness in a variety of ways.

But just like money, insights are mainly worth the use to which they are put. So, whether this big picture describes the competency of an individual savvy person or of an enterprise, it tells something about being strategically capable but the goods are in the doing after the learning.

Posted by Malcolm Ryder at 6:15 PM

April 19, 2008

The Innovator's Real Dilemma

Jessica Stillman at the new BNET1 blog rounds up research from Accenture, the Conference Board, and Wharton to talk about why Fostering Innovation Stumps Executives ...

This is an interesting situation to ponder: making choices about how much to invest in innovation , versus in knowledge management and, separately, business intelligence as other paths to insight. Overall, what the organization is mainly after -- where the real money rests -- is the insight, whatever the path. But the current thinking about management priorities indicates that insight is pretty hard to come by, so lesser-beaten paths to it are also getting a lot of attention.

One challenge that surfaces, somewhat amusingly, is the presumed need to be innovative about how to foster innovation. For example, given that "innovation" is so easily approached as "creativity", it is not surprising that at places where real urgency comes from competition against either industry rivals or the budget, the idea of stimulating the worker's right brain with art experiences can gain some real traction.

But perhaps everything new is old again... The simplest way to assure that innovation is "fostered" is to provide
(1.) a clear statement of why the company will consider something to be "innovative" and...
(2.) a clear statement about what circumstances will cause the innovation to be rewarded in a way that directly benefits the individual(s) involved.

Generally, if company leadership can't get that much communication together and abide by it, then most other "fostering" efforts are essentially arbitrary.

Furthermore, this effort should not be confused at all with management's concern about how to measure the innovation's impact on the company's performance. The performance impact issue is not something that should be making innovation special. Any management team that rewards "performance impacts" with bonuses should simply add innovations to the mix of things that can be clearly accounted for as contributors to better performance. Meanwhile, innovation is about doing things differently to create opportunity; but execution is about doing things a certain way to hit performance targets.

This is where managers have to get real: if they will not reward innovators for being innovative, as opposed to making the reward conditional upon performance increases, then people will learn that innovation is not worth the effort at this organization. So in step (2.) above, the "circumstances" to be declared must start with something other than performance metrics.

Posted by Malcolm Ryder at 8:42 AM

March 23, 2008

Suddenly, It All Made Sense

Finally, that track that everything went off of, and where to get back on.
For the hi-res view, click here and go full screen or print.

Posted by Malcolm Ryder at 6:42 PM | Comments (0) | TrackBack

October 22, 2007

Dr. Cinderella or: How I Learned to Stop Worrying and Love (?) the BCS

In the entire known world of metrics, nothing is as important as the American college football rankings.

But if you believe the BCS rankings, then you probably believe in yesterday's weather reports, and you're probably not a gambler. Or, you believe some teams go to Hell, while other teams go somewhere else.

For the most part, it's really a given that most teams will go to hell at some point, and it is well known that when ranked teams go to hell, they go to a harsher hell than do most unranked teams.

All teams try their best to go to hell in the preseason, under less scrutiny. Whether it's preseason or not, good teams may go to hell only for a brief stretch of a practice, or in season perhaps during one quarter (probably the first or the third) of an actual game or two. You can see it on the coach's face: "Well, that just went all to hell."

But back to the BCS and metrics. If your team is ranked say number seven in the top twenty-five in the BCS list, does that mean that the teams ranked higher than your team are "better teams", and the ones ranked lower are "worse"?

Of course not. At best, it means that higher-ranked teams had a higher frequency of important beneficial results from the circumstances they met while operating. The key part of that explanation is the slippery part: "important".

With the formula used to calculate the BCS rankings, there is not nearly so much debate about what factors are important to each team, as there is debate about whether the factors important to Team X are also equally important to Team Y. In a measurement system like the BCS rankings, the point of the formula used is to impose it as an identical calculation on all teams. Whereas in gambling, the beauty of its complex logic is that determining the probable winner of a contest emphasizes what each contestant uniquely needs to have the best chance to win.

Another very interesting counterpoint to the BCS rankings would be Wall Street. Should we think of the BCS ranking as being analogous to the "stock price" of a team?

Well, even given the similar degree of analysis that goes into each contributing poll underlying the total BCS calculation, stock price is based on much the same idea that gambling is: given some degree of uncertainty, what is the probability that the contestant will actually generate the critical advantage it needs to have its unique formula for success prevail in its expected future circumstances? Unless contributing polls focus on that, there's not much reason to "put stock in" the composite ranking.

But so what, we still want to use the BCS ranking as an indicator of how probable Team X is to defeat Team Y in a hypothetical matchup. There, our desire reflects our interest in relying on "intelligence" -- on the power of facts to reveal truth (or at least repeatable or persistent patterns) -- just not nearly so much as we pretend it does. The determinant feature of this ambition is the matter of whose intelligence we choose to use, because we rigorously choose to not use other intelligence outside of that. And as we know from "experience", the flaw in this approach is that probable outcomes in sports are determined as much by what we don't know as they are by what we do know. And guess what, this limit on the BCS exposes that the real point of the BCS must be something else -- namely, not to find the best team but merely to promote an agenda of creating artificial interest ("need") in order to then make money by satisfying it.

On that note, it becomes apparent that an ideal, non-commercial performance ranking system (not the BCS) would not be a weekly list of mainly historical accuracy comparing apples and oranges to each other. Instead, it should be a monitoring system that takes the current probability of a team's next future win, and describes the importance of that probability relative to the importance of each other team's similarly described outlook.

Let's say that a largely uncelebrated team from a so-called "minor" league has a high probability of winning all of its games, and so does a high-profile team from a so-called major league. What is the relative importance of the minor team's perfection versus the importance of the major's? The real importance is not that we think the one team would probably beat the other in a playoff.

Instead, the real importance is in how each team's own league (or said differently, its market) can respond to the performance of the respective teams. The major league team can't do very much (that it likes) with the excellence of the minor league's team -- or at least not much more than the major league's scouts and recruiters have already done. But is there some important relationship between the minor league team's excellence and the major league team's league? How does the minor league team benefit (i.e., "play in") the major league market? Even the major league as a whole is lukewarm to the good minor league team. The minor league, however, can of course do a lot with its team's excellence.

The point would be that even if the minor team is in the BCS top ten along with the major team, but they don't actually play each other, it doesn't matter and it doesn't mean that the higher ranked team (whether major or minor) is the better team.

With that admission, it's easier to relax and understand what we do want the BCS for: we want it to suggest who should go to the playoffs to make the playoffs more interesting -- at which point the bookies (stockbrokers, traders, stakeholders) can take over and do their job.

Oh... wait... There are no playoffs...

Soooooo...

What's wrong with the BCS? Nothing, really, except that it's a gun pointing at a non-existent target.

Is your own measurement system loaded but pointless?

[Pick that Nit Dept. -- at least the BCS, or "Bowl Championship Series", isn't actually called the "National Championship Playoff". A series of bowl champions we can live with; why not? All ya gotta do is like bowls.]

Posted by Malcolm Ryder at 7:02 AM | Comments (0) | TrackBack

September 3, 2007

Performance Management meets Business Intelligence

If we assume that management prominently features Planning at the front end of the cycle of "management performance" (i.e., exercizing good competency in the discipline of "management")...

...and if we assume that planning uses intelligence in the form of research that provides indicators of the potential for future success and risk...

... then to establish that business intelligence (BI) is part of performance management (PM), it is unnecessary to go any further than the concept of forecasting. The important view of this involvement is that neither effort (BI nor PM) wholly includes or excludes the other; rather, they logically intersect, co-operatively.

BI manages the perception of the operational environment. PM manages investment in the operational dynamics.

Perhaps there will be comments from the readers on the idea that strategy manages the relationship of BI and PM for a target group of stakeholders...

Posted by Malcolm Ryder at 5:35 PM | Comments (0) | TrackBack

February 17, 2007

Split P Soup

Be careful what you ask for. This is the part everyone already knows to heed, but who actually does?

Here's a favorite view of the intensive efforts made to satisfy the customer: Forbell's "Splitting Peas for Split Pea Soup" printed in Old "Judge" Magazine. We love the "system" of production controls, and the implication that the overkill is necessary to get the soup right.

This worked for Andersen's Pea Soup Restaurant in old Buellton California; the "home of pea soup", they were very clear on what their customer wanted, and they did that one thing well.

But isn't that the exception? From what we read and hear, throughout many fields of effort, from projects to purchasing, failure or buyer's remorse is easily just as common as their opposites. Despite decades of programmatic attention to "improvement", things have only marginally improved when it comes to actually wanting what gets delivered.

This obviously suggests that the supply side of improvement is only part of the problem. The other part has to be on the demand side. But as the customer, we're supposed to be always right. So why do we not get what we want? Because we don't ask for it. We might only ask for what is "high priority", to try to reduce the risk; but what does "priority" really mean?

Most often, the problem with priorities is that the way they relate to the "want" isn't understood or isn't communicated. They wind up being not ambiguous but out of context, leaving it much more likely that other parties responding to them (as providers or stakeholders) will respond the wrong way or simply disagree.

To sort this out, we have to trace the "priority" back to what made it a priority, and set that out as an explicit part of the receiver's specification for the deliverable. As seen in the picture below, this will show four different aspects that may get evaluated when the deliverable arrives. The risk is that the provider and the receiver didn't agree, in the first place, on what mattered -- making the deliverable less tolerable, suitable, usable or whatever, having not met the unstated criteria... At the point of delivery, disagreements about whether the right thing was provided often seem to be about splitting hairs, and the reality is that the hair to be split is what was meant by "priority".

In this framework, it's clear that the key points to consider are neither indefinite nor synonymous. And that is why they are not interchangeable. Thus it is easy to get one or another of them right, only to find out that whatever wasn't addressed will cause a "failure" or buyer's remorse.

The sophisticated customer or likewise provider will recognize in advance that all these different aspects must be accounted for: each one either satisfied or ruled out, in an agreement between the supplier and the customer. For example: take television commercials for successful "diet programs" -- now they must try harder to pre-empt deal-breakers, because the key considerations of most potential customers are well acknowledged to "cover the bases"...

With an opportunity to identify issues in advance this way, there's much less reason to tolerate asking (or being asked) for the wrong thing. We see the path to getting it right.

Posted by Malcolm Ryder at 11:55 AM | Comments (0) | TrackBack

December 12, 2006

How Much Is Enough?

Our colleague Charles H. Green at Trusted Advisor helps to reinforce some strong themes found around here in his comment on the very recent Archestra post, "Performance Recap".

One of his points is that measurement is probably overrated as a prerequisite for management. Enjoy his examples as soon as you can, but here I'll add a few more points to continue both the recap and the line of thought.

How can we define measurement, management and performance simply enough to see their absolute difference from each other but just as easily see how they might relate?

A simple version of that is as follows:
- measurement is a form of description that intends to identify a relative state.
- performance is a describable effect , of an effort that intends to create progress towards a target state.
- management is a form of influence that intends to establish a relative orderliness.

Measurement doesn't cause orderliness; it merely can support orderliness by giving management some grounds for a model of orderliness to pursue. But management can easily use other sources to develop models. Interestingly, though, it is hard to imagine any source of a model that does not essentially require description, so the difference is most likely in the forms of description that are used to assert the basis of the model. Gossip, for example, can be posed as an alternative to measurement. Plenty of people model their efforts to influence things, by relying on gossip.

And performance -- well, it can be "good" or "bad" or somewhere along that spectrum. The point is that it is not necessarily either, and the notion of performance requires only the ability to identify where the effect of effort fits relative to the ambition associated with the effort. Interestingly, this helps to point out that the actual effectiveness of an effort might be strong, but if that "effectiveness" isn't of the right flavor, it won't in most cases be evaluated as "good performance"...

As for management: we most often work under the assumption that some kind of orderliness, in our environment and/or our effort, is more likely to result in progress from our effort. Yet in art, in learning (not "teaching"), and in play, we also know that this orderliness might be very slight and it might not even be from a desirable mode of influence -- to wit, "the method in the madness" -- yet progress is actually just as intentional, plentiful and expected as it is in any routinely "managed" work of other kinds.

What Charles Green (and much of the thought on Archestra as well) points out is that management can influence progress without being concerned about measurement.

On the other hand, it's quite difficult to use the notion of performance without using measurement. Here, the real issue is not whether measurement is involved, but rather what kind of measurement is involved.

Having responsibility for progress creates too much anxiety for us to get relaxed about having "unmeasured progress" -- and yet unmeasured progress happens all the time -- both with and without management. We just don't want to take the chance, normally, that desirable progress will occur without management, and then we use measurement to prove that the management effort being made is worth the trouble.

So, having reached that punchline, there's nothing fundamentally wrong with liking measurement. It's one way that we can try to increase the amount of time we spend being both lucky and smart. This doesn't mean that other ways, such as relationship building and collaboration, wouldn't be just as helpful or even moreso. That's also management at work, and it probably has a much longer history of practice and success. Let's call in some high-performance historians and check it out, when we can.

Posted by Malcolm Ryder at 11:38 AM | Comments (1) | TrackBack

August 25, 2006

What Matters versus What Counts

How do you decide the "most valuable"... the "best"... the "most significant" ??

Why Andre Agassi is on my list.

http://msn.foxsports.com/tennis/story/5892448?CMP=OTC-K9B140813162&ATT=199

When determining and understanding "value", it's all about what kind of difference the observed difference makes.

Posted by Malcolm Ryder at 1:44 PM | Comments (0) | TrackBack

June 26, 2006

Recognizing Progress: Effects versus Results

We know the old saying, "don't confuse activity with achievement." It warns us that making an effort doesn't necessarily mean we're making progress.

But one of the problems in recognizing progress is the need to know whether the conditions being generated by activity are beneficial or not.

To do that, there first must be an awareness that benefits may be unintentional as well as intentional. And meanwhile, we might get to the benefits in a planned or unplanned way.

This quickly catalogs four kinds of outcomes:
- intentional benefits from planned activity
- intentional benefits from unplanned activity
- unintentional benefits from planned activity
- unintentional benefits from unplanned activity

But along with activity and benefits, there's a third dimension too.

In management, we have to especially notice that planned or assigned activity is a mode of change, and that the circumstances (think "environment" or "context") of the activity can also change -- independently of the activity and especially during the activity.

With two sets of changes occurring -- activity and context -- the impact of each one on the other will shape the emergent conditions that are examined when we look for "progress".

When it comes to an assessment, history has shown that some combinations of conditions are far more associated with ultimate success than are others. This is why the "profile" of the conditions is so important to detect, not just a measure of an action or event.

This is a way of saying that success is relative to circumstances, so describing the circumstances adequately is more important than anything else in understanding whether an effort is being "effective" as opposed to its being ideally conclusive. For example, in a long race, the "patient tortoise" is more successful than the "impatient hare". The progress profile includes the awareness that the race is a long one... not just that the runner is fast or slow.

Here's another similar example. Imagine a motorized walkway running from point A to point B. If we decide that "progress" is to get from one point to the other, then the following problem occurs: walking on the walkway against the direction of its flow might "net out" to going nowhere. That seems to represent no progress.

But alternatively, if one must try to get to the opposite point even if walking against the flow, then going nowhere is better than going backwards -- so in making the effort, avoiding a likely loss of ground is a benefit to the cause and must be seen as making progress.

In the latter case, the benefit is clearly an "effect" -- meaning that it is an outcome contributing to the overall desired "result" although we don't yet have that final result. And we can see that the idea of "effectiveness" is most strongly associated with the way (and the fact) that we have predefined the requirement, not the goal.

Meanwhile, the effects of an effort are not always beneficial. We'll be getting effects from any effort, but they may not all make positive contributions to the desired result, and furthermore they may even be counter-productive.

The above perspective on things yields a description of the approach we need for understanding and communicating progress:
- We identify effects;
- We rate the impact of the effect(s); and...
- We measure the result

Thus there are three kinds of achievement to observe:
- producing the right kind of effects;
- gaining more beneficial impact from the effects; and...
- getting closer to the desired final state

Superficially, this mimics the structure of organizational responsibilities:
- operations (for the right effects)
- management (for the beneficial impacts)
- executive (for the target state)

But more importantly, there should be a strategy that guides the prioritization of efforts by telling what kind of progress is most critical, giving the most bang for the buck, at different times and places. Changing operational competencies is radically different from changing targets. Changing the wrong thing can be at minimum wasteful and at most catastrophic.

Postscript:
Extensive practical analysis of this issue -- including further distinctions between "activity", "achievement", and "progress" -- is available from Eliyahu Goldratt, who developed the Theory of Constraints.

Posted by Malcolm Ryder at 8:02 AM | Comments (0) | TrackBack

June 14, 2006

I.T. Been Berry Berry Good To Me

The best quote of the year so far: from Romano Prodi, former president of the European Commission running for prime minister of Italy against incumbent Silvio Berlusconi back in April. Said Prodi about Berlusconi, "you lean on numbers like alcoholics lean on lamp posts, not to be enlightened, but for support..."

Yeah. That's the spirit! Numbers should be fought over. A bunch of new ones from Forrester, in CIO Magazine, show IT organizations doing the same old same old, when it comes to providing value to the business. In this report, IT's top batting average of .380 against Business Pitching feels bad, but that's really hot in baseball.

Just Makes You Think: Oh yeah, that's right... this stuff is pretty hard...

For the sake of skirting copyright, I memorized the information so I can now fearlessly just "tell you what I remember"... OR you could go see for yourself online (or...the June 15th hardcopy includes the Forrester survey numbers chart).

I remember that Forrester said:

1 - Improving productivity or products/processes rates in the high .300's
2 - Optimizing cash flow or customer lifecycles hits the mid to low .300's
3 - Powering successful innovation in process collections or in product collections steps in the mid to low .200's. Wipe your shoes.

But what do these numbers mean?

First, the way Forrester set it up, the batting average actually represents the number of "IT Decision-Makers" who gave IT strong credit when given the chance to do so. For example, almost-but-not-quite four out of ten gave IT credit for improving productivity or likewise for improving products.

Now, looking at the numbers and what they are attached to, is it insane to say that the more complex a problem is, the less likely IT is going to make an obviously critical contribution?

We know the problem complexity rises as you go from the top of this list (e.g. productivity) to the bottom (e.g., innovation). Why?

Well, in each case, think of the number of variables that are reasonably under critical IT influence as they combine with each other. In order of appearance:

- There are simply fewer of them when it comes to improving productivity or improving products/processes. (We didn't say the tasks were any easier, we just said simpler.) That level of influence is a lot like construction work.

- But the next level -- optimizing cash flow or customer lifecycles -- is more like winning poker games. More participants trying to not be on your same page.

- And the remaining level -- innovating (reconceiving, not just improving) products or processes is more like herding cats. More participants who see your page but could care less.

Here's the main suggestion. As we drop down the list, and move from productivity to optimization to innovation, the necessary influence on the variety of participants who need to buy in becomes increasingly less a deliverable of IT.

Going along with that, we can still hope that IT can contribute to the necessary influence. But then of course, we must identify what kinds of influence are necessary, before we can understand whether IT can make a significant contribution.

What this calls for is a perspective in measurement that can recognize and understand the game-saving catches, the turning point singles, the continuity of getting players on base -- the things that go missing in games that are not winnable. Eveyone sees the home runs, but most of the time, most games are won by most players being good enough to allow a win. Their individual responsibility is to be good enough for the other players, not to win the game by themselves.

The Forrester numbers superficially tell the story a little differently, but let's decode them. Given what was argued just above, they suggest that very few of every ten IT Decision Makers understand the difference between how well IT does its part and how likely it is that the IT part will "cause" a win. This further suggests low understanding of what it takes to win at productivity, optimization and innovation. That is, statistically, some of the decision makers may have given 100% credit to IT for effectiveness, but those that gave little or no credit (due to lack of understanding) dragged the Forrester averages down.

Meanwhile, the reason why batting .300 in a game is good enough is because it's enough to allow the other parts of the operation to add up to a win. Batting .300 represents about 100% of reasonable expectations, not 30% of requirements. The decision-makers who gave IT high credit likely saw this.

So the numbers we see from Forrester are not what we really want to see. Instead, we want to see the explanations given by the decision makers who credit IT with high effectiveness, and compare those explanations to the ones given by the nay-sayers.

Post Script: Saturday Night Live was a hit before lots of IT people were old enough to tell a joke. If you're older than that, you might remember Chico Escuela played by Garrett Morris, who broke the line that we cloned for the title of this article, but who for all we know now works in IT somewhere as a decision maker.

Posted by Malcolm Ryder at 4:48 PM

April 23, 2006

Metrics Myths: Miles Per Gallon, or Gallons Per Mile?

A cautionary analogy of measuring value.

Miles Per Gallon (MPG) is an oft-used example of the "bang for the buck" flavor of value. But as we all know, the fine print that comes with it is pretty much of a whopper: mileage may vary. The fine print makes the measurement a lot less reliable until we know a lot more about an additional little factor waiting to be included -- the driver.

That is, with MPG, what we want to think of as the value it describes is often less than what we're really after: the reality is that the independent efficiency of the system it describes (the car) has a relationship with the driver that is the real source of whatever association we want to make between the system and effectiveness. What we're usually really after is effectiveness, and MPG gets us only part of the way there.

If we don't forget that the driver will be involved, miles per gallon is more obviously an indicator about a resource input and the efficiency of its consumption. Yet although we know better, we tend to use it as if we "make" miles with the resource (gas) and the resource must "efficiently" cause more miles to be made. That is, intuitively, the story we want to tell with the MPG measure is one of effective process production. The problem with this is that we don't really make miles with the resource -- instead, we acquire them. As a measure, MPG is actually more a story about "spending" the resource than about process effectiveness.

For process measurement about effectiveness, we have to look at not miles per gallon, but instead gallons per mile (GPM).

Why? Because the Mile is the target achievement unit of the travel (process). Going from point A to point B, we want to cover the necessary miles regardless of what the cost in gallons may be. The cost might change our enthusiasm for making the trip, but the cost does not change the distance to the target (of the trip). Consequently, GPM speaks to a production efficiency that might be indicated by the phrase "gallons may vary" and brings up effective resource consumption.

Summing up the above:
- MPG pertains to efficient resource consumption for effective process production
- GPM pertains to effective resource consumption for efficient process production

So, what's the management difference between MPG and GPM? It's the difference in what kind of value is determined.

Value through GPM is about deciding which type of resource should be committed.
- Gas, electric, or steam?
- People, process, or technology?
- etc.

Value through MPG is about deciding which resource of a given type should be committed, as in from what category should the given type of resource be drawn:
- Basic/premium? Generic/branded? New/legacy?
- Stored vs. just-in-time?
- Standard? Custom?
- etc.

Put that way, such choices for how to change value relate to each other in a fairly straightforward way. The picture below makes this more visible in conventional management terms:

As an example of how this plays out, consider a company's IT operations. According to the framework above, IT has four kinds of value contributions to offer to the business:
- Capacity: through IT Assets (resource efficiency)
- Outputs: through IT Systems composed of assets (process efficiency)
- Options: through IT Services running on the systems (resource effectiveness)
- Outcomes: through Applications leveraging the services (process effectiveness)

The overall discipline of managing the value would be about making choices in all four cases that systematically increase the likelihood of desired outcomes -- meaning the odds of generating "explicit actual value"... As demonstrated by the framework, it need not be confusing to direct the appropriate measurement of value contributions when the important perspectives are both distinct and logically related.

Posted by Malcolm Ryder at 10:04 PM | Comments (0)

January 15, 2006

Measuring process improvement

Events become notable and interesting to business primarily because of their outcomes.

While outcomes range across the spectrum of pursued to avoided, desirable to undesirable, or benefitsto damages, they generally all figure into operations by the same path. That is, the impact of the event is studied for whether that impact is (or can be) repeatable; whether the repetition is (or can be) intentional; and finally whether the intention is (or can be) manageable.

In that light, four areas of measurement are associated with all managed events, and the manageability is ultimately what distinguishes an operations event from an operations process.

1. The impact itself is compared against some standard of expectation or some qualifying definition.

2. The likelihood or frequency of the impact’s repetition is tracked against elapsed time and/or a range of different conditions.

3. The intent to repeat is proved against evidence of supporting commitments made.

4. The manageability of the intention is modeled in terms of responsibility and authority.

These four types of measurements – comparisons, trends, proofs and models – provide the terms for establishing both the bases of process development and the deltas of process improvement. Each type of measurement refers to some defining aspect of the operations environment, and that environment is therefore the context of the measurements.

Process improvement is usually thought of in terms of the difference between the earlier outcomes generated and the outcomes gained consequent to a modification of the process. But this is an erroneous attitude. Outcome improvements are of course highly important, but they are different from process improvements.

Instead, process improvements always assume that a target outcome is achievable, and the process improvements are concerned with how to engineer and assure the achievement.

Given the above, the difference between the initial circumstances of a process and the improved circumstances for the same process is the critical difference to be determined in a process improvement initiative. Effectively, the following is pursued by the initiative:
1 - establish definitions and standards
2 - establish continuous event monitoring from multiple perspectives
3 - establish commitments to prerequisites and/or causes underlying the dynamics of events deemed pertinent to goals
4 - establish models of proprietorship for the commitments

These factors make it more obvious that process improvement is essentially a matter of managing corresponding organizational changes. These changes include, for example, the strength, state and pace of an organization’s adoption of:
1 - taxonomies and vocabularies
2 - reporting and analyses
3 - approvals, allocations and priorities
4 - assignments and authorities

Given alignment of those changes, the organization acquires the capability to literally do things differently for the better, and the groundwork exists for shifting focus to production technique that will be evaluated as a quality factor in execution.

Improvements in technique are again a different matter from process improvement but they manifest the ability to reliably conduct the process on demand, and therefore point attention towards improved competency as opposed to improved capability.

Posted by Malcolm Ryder at 11:53 PM | Comments (0) | TrackBack

January 10, 2006

Evidence versus Truth, or Will the Witness Please Answer the Question!

From the land of "ignorance is no excuse" comes this picture of the fine line between being "accountable" and being "guilty".

At Federal Agency employee performance evaluation time, overwrought managers avoid confrontation and hassles with poor employees, but go on to solve the wrong problem, by simply giving them (only) a passing grade that makes all further debate moot. But then a new manager takes over and wants to get rid of the bad worker, which can't be done because the personnel folder doesn't have anything in it to support the claim of poor performance.

Thus proving once again that when it comes to measurement, like with jokes, what makes it work is not what you tell but how you tell it -- then the problem usually kicks in with the re-telling by someone else.

So much for a single version of the truth!

See the full gory story by Washington Post columnist Stephen Barr at the Federal Diary for Tuesday, January 10, 2006.

Posted by Malcolm Ryder at 1:11 PM | Comments (0) | TrackBack

December 5, 2005

Stats don't lie; but can they forget?

Our colleague Steven M. Kemp, at the department of psychology at UNC, reports:

I thought of a quick summary statement about the slogan, "You can't manage what you can't measure."

"You can't manage what you can't measure" is a terrific slogan for exhorting business folk to consider measuring what they are not currently measuring, or not even considering measuring.

Taken as a claim, it may be harmful if it encourages folks not to manage what they are not measuring. Measuring constrains the degree to which we can manage, but managing to the degree that we can is an imperative."

Co-authored with his brother Sid Kemp, Steve's book, Business Statistics Demystified at http://tinyurl.com/2lxyr, explores the measurement issue deeply, with an eye towards getting it right instead of just getting it done.

Posted by Malcolm Ryder at 5:49 PM | Comments (0) | TrackBack

Managing versus Measuring

We always say that we can't manage what we can't measure.

But what are you doing? Are you just measuring the management? Managing the measurement? Or, actually managing the item on which you're using the measurement and management processes?

I.

Management is essentially about determining whether the state of things needs adjusting and then intervening with adjustment actions that are appropriate. On their own, both environmental conditions and activities can be hugely indifferent to the particular desires of an organization, and to be routinely leveraged they must be perceived in consistent, non-random ways. Therefore, one can't manage what can't be described. Measurement is just one aspect of the fundamental management need for description, and it must be complemented by other forms. (See the illustration here and again near the end of this discussion.)

To understand this, it's important to immediately broaden our sensitivity about measurement -- away from "metrics", up to to the general functional problem of "description", and back down much closer to "definition".

Defining current states is a mandatory part of the management equation, but that doesn't amount to management. Instead, if the adjustments are the whole point of having management, then actually it's more critical that we define the adjustments. It's what kind of an adjustment, and how much of an adjustment, that drives the only difference we expect from what we'll later on still call "management"...

And just as all rectangles are not squares, not all definitions are measurements. Definitions are descriptions, and "measurements" are just a form of expression that descriptions can take.

But because measurements are so prolific, complicated and frequently debatable, we're concerned enough to ask the question, "can management be done without that form of description?"

II.

Let's say that you have a task to put a stone in a box, but the stone is initially too big -- which we determine simply from trial. How are you going to "manage to" get the stone into the box? Do you need measurement to get the stone certifiably "in the box"?

You could chip away at the stone to make it smaller, and keep trying after chipping it to put it in the box. Your action makes the stone different. Eventually you'll get the stone into the box.
- On the other hand, if you don't care how long it takes, how small the stone winds up, and other such things, you don't need measurement -- you just need the stone to finally fit into the box. But each attempt to fit the stone in the box is actually a measurement.
- And on the other hand, if it matters how long it takes, how big the stone finally is, etc., then of course you need even more measurement.

We tend to think of the latter case as being managed and the former not; but strictly speaking they are both managed. That is, in both cases, proactive adjustments are determined and made deliberately towards the target state until the target state is true. We see that measurement figures in differently but both times. Most importantly, the target state is true only when we agree that it is true, and we use the measurement to gauge the distance from the agreement.

So far, to act towards the target we still haven't necessarily invoked measurement, but to know that we're reaching the target we must have it.

Practically speaking, the key element is actually the agreement beforehand, which means that the terms of agreement are the key to describing the target state. These terms may be non-metrical, while still being highly differentiating. For example, objectives are often stated to represent the terms of agreement by which most comparisons will be made. Compatibility, not compliance, is usually the subject of this perspective on things.

But let's get back to measurement. Measurement provides a kind of additional certainty to definitions, by sharpening the definition's ability to express distinction -- typically in terms of "amounts". For example, even a qualitative binary distinction - true vs. false - can be considered almost purely measurement if the real question posed is "how much 'false' is there?" and the answer is "None."

Yet in that example, it's obvious that we must still know beforehand which conditions actually mean "True". Can we identify (define) the conditions without measurement?

Sure, if we can distinguish them in other ways.

First we should look at why we can do that, and then at why we should do that.

III.

In the example above, it can be argued that management didn't require measurement to make "stone in the box" become true. But measurement is required to confirm that it actually is finally true. So the point is that the success of the management is what we need measurement to expose.

Let's make that thought more concrete with another example. If a test case is looking for exactly a 3-foot length, any length tested will need to have been measured before we can say that it complies. Compliance is determined purely by comparison to some standard (in this case, something that we already know is 3 feet long). What makes the comparison work is that the standard has already been selected. Actually doing the comparison -- that is, simply applying the standard to the test item -- is an action that might be "managed" or accidental, but we can confirm the outcome only through measurement, and the confirmation is what tells us whether or not we need to try againto hit our target.

This is all adding up in a certain way:
- definition is aimed at identifying the distinction of the target conditions
- the difference is described and recognizable by terms of agreement
- compliance to those terms is largely detectible or provable through testing
- the testing would rely on a standard to help establish the degree of compliance to the terms
- more than one condition might be part of the standard

But there is another way to work in terms of a standard: procedure.

Procedure prescribes steps that are expected to produce compliance. The assumption built into a procedure is that during execution each step of the procedure should become true. Largely without measurement, we can simply follow the procedure. In this case, most measurement used is dedicated to determining how much execution complies with the procedure (supervision), and to confirming the effects of making adjustments (intervention) -- but the choice to use the procedure is far more significant: it means that given the responsibility to direct the course of events, we have agreed to the procedure's logic. Exercizing the logic, with whatever level of competence, is the most differentiating characteristic of what we are calling "management"...

IV.

So, as it turns out, the urgency about measurements is not really about manageability but instead is about predicting the effectiveness of the management, and about increasing the accountability of the management. In short, they are about making management scientific.

However, management need not be scientific in order to be management. What management really depends on even more than measurement is logic. Measurement should be a means of testing and tracking the logic, but logic must be derived even in the absence of desired measurements. This takes place in the form of assumptions and objectives. Management logic can form a closed loop or system when its assumptions and objectives are followed with ratings. (Current ratings influence future assumptions.)

In completing that loop we also get the broad outline of the framework for management description:
- Assumptions represent the key initial distinctions acknowledged in the management effort; this includes discovery of identities and definitions
- Objectives represent the point of view that finds comparative significance in what is done and monitored; this includes test criteria and terms of agreement
- Ratings represent the actual visibility on the value of conditions, achievable from foresight to hindsight; this includes standards and priorities.

With those anchoring observations, we can see the universe of description that management needs.

This framework intends to position the many modes and artifacts of description in a way that highlights where they make the key difference to management's overview of conditions. Against that, it sorts out and addresses the basic management concerns that drive decisions to intervene:
- preferred states;
- adjustment progress; and,
- the impact of changes.

In the illustration it is easy to see the range of formats for describing conditions, along with the fact that they are not necesarily metrical in nature. However, it is also evident that measurement plays some role in every area of description -- such as, by establishing trending ("track"), marking off milestones ("score"), or detailing cost/time limitations on requirements ("specify")...

V.

Our key management challenge is to conceptually arrive at definitions identifying one thing versus another, so that we can properly identify their relationships and leverage the relationships.

These definitions, when treated *as if* they are 'facts', are the normal basis of managerial logic. Communities of practice may develop and even enforce 'standards' of definition, but this ultimately does not prevent management from being practiced based on other standards or knowledges and in non-scientific yet still logical modes.

As for measurement, it has the task of identifying and comparing differences in an accounting mode. Three other issues ensue:
- to actually do the measuring;
- to measure the things that matter; and,
- to match the right measures with the right management.

Otherwise, measures simply become an unreliable tool being used for "solving the wrong problem."

Posted by Malcolm Ryder at 7:46 AM | Comments (0) | TrackBack

October 29, 2005

Managing "Measuring"

Neil Raden's article, "The Limits of Measurement", bring us to the interesting problem of not being able to see the trees for the forest.

In this case, "performance" is the forest, and the trees are the events and inputs that make up successful explanations for the performance observed.

Measurement looks for data points and then it looks for patterns or "forms". We assume that the points themselves don't tell us what we need to know -- but when data collectively mean something different from the data points, we have to hope that the points are resolved into a focused picture and that the picture means something.

On the other hand, a preconceived pattern or form discourages finding the data points that don't fit. In measuring performance, this can turn into the difference between fact and myth.

I.

We have enough other problems getting accurate data without preconceptions clouding the view... Through interruptions and reprioritizations, ongoing change alters the routines of action that are considered necessary in day-to-day operations. So, by analogy, what if the key points keep moving? Chasing these moving targets, can the picture ever be focused? How far away do we have to get to see a "stable" picture?

Neil's article gives important reminders about the Heisenberg Uncertainty Principle, and they tell us that we have to at least estimate how much our data collection process -- e.g. measurement -- might either lack "accuracy" or exceed a useful amount of it. Otherwise, we risk simply accepting distortion that we created ourselves.

The distortion may be unintentional, like a manufacturing defect discovered only later. But worst case, you know the old saying, "statistics don't lie -- statisticians do."

II.

Furthermore, "accuracy" and "usefulness" are vulnerable not only to the measuring instrument and measurer; distortion can begin with the definition of what is being measured in the first place. These definitions are not absolutely right or good; they are debatable.

Comparing one organization's practice to another's yields proof of this. It's like two contestants in a recipe bakeoff. Two different organizations, pursuing the same outcomes, might have widely divergent definitions of what data comprise the key points that compose the bigger picture. Thus, on the surface, they might be trying to work with "the same" picture -- but they might disagree on the significance of any included point of measurement or of any omitted point.

The competitive ramifications of course make it all more interesting. Even when two organizations work from the same general framework, the diversity and uniqueness of their particulars could mean, as the saying goes, that "on any given Sunday, any team can beat any other team..." That is, the differences are not necessarily advantages, so we might decide that the definitions behind the difference are not the right ones after all...

III.

And finally, won't a different point of view on the data also result in a different picture? Strictly speaking, we'd have to expect a "definite maybe"... and the reason why is that we might not see the same data from a different point of view.

Since our management already leans on ideas like capability maturity, best practices, and other examples of empirically proven improvement, it seems that we already know about the need for measurements to differ according to modes of achievement as well as targets of achievement.

These improvement concepts are also ideas that encourage us to adopt standard answers to questions like "Are the right things being measured?" and "How do we know what the right things are?"

Being essentially scientific discoveries, these standards reach our management as a kind of Newtonian physics, setting an operational baseline and perspective. Our practice of it is then challenged by how sensitive our measuring instruments are. When our measurement sensitivity hits the point of diminishing returns, we want to go beyond. But how?

Beyond that is a different challenge: an alternative, more quantum-physics perspective of multiple concurrent points-of-view.

IV.

In our performance improvement efforts, organizations get pretty good at following prescriptions for action, and although allowing that our measurements interfere with the actions taken, the actions are not significantly varied from one time to the next. We're used to activity being measured within ranges of tolerance. As a result, we think of value from a distance that accommodates variance.

But let's look at this idea of "significance" ... If we bother to measure a prescribed action because it represents a certain value to be generated, we want the action to avoid getting caught in a cloud of varying kinds of measurement, each kind approximating the action from its respective point of descriptive view. Why? Because each perspective's description can virtually propose a different action, and until there is agreement amongst the perspectives then their collective measurement leaves us somewhat unsure about what action really happened.

For example, was a certain higher-than-expected expense a "detrimental cost" or a "beneficial investment"? Was a faster-than-expected delivery an opportunity or a disruption? Such varying views of "facts" lead to uncoordinated organizational responses and reactions that defeat the purpose of measuring things.

This emphasizes the fact that, while management likes precision, it really must have relevance even more -- regardless of precision. A fuzzy (or crude) picture of the right thing is more valuable than a clear picture of the wrong thing.

Having multiple stakeholders means having multiple points of view. This condition will show that interpretation sets the limits of measurement.

V.

A well-worn story related to this problem is the one of Six Blind Men and an Elephant. Touching the elephant, each man relies on his own perception of the part of the elephant he is nearest, and each part touched is very different from the others -- a tusk, an ear, a leg, etc. Consequently no one man figures out that what he is touching is an elephant, instead of a pole, a tent, a tree, etc.

That's an interesting story, but if our problem is to make a better elephant, then the story goes inside out. Instead, if all six men were first told that they were about to touch an elephant, would they each then be able to determine what part they were touching and assess the status of the part? Yes, the information-in-advance mitigates their individual blindnesses to a very important degree, but this works because they were not misdirected -- not told to expect something other than an elephant.

For this reason we can say that an organization's performance is driven more by agreements before-the-fact than it is by measurement after-the-fact. In fact, agreement should create the model from which measures are derived. But then, what we need measurement to do is to feed back and drive future agreement as well, across the inevitability of ongoing change... and this will depend on managing interpretation in the organization.

Posted by Malcolm Ryder at 9:39 PM | Comments (0) | TrackBack

September 15, 2005

Assessments versus Measurement

In the growing body of literatures regarding performance improvement, the usual recommended starting line is the advice that "you can't manage what you can't measure."

But this allows a troublesome add-on thought, which is that your measurements are going to drive your management mainly in the fashion of "garbage in, garbage out"... The quality of input data is of course a risk, but what about whether the correct kind of data is being absorbed in the first place? Great data of the wrong type will lead nowhere fast, or to the wrong place too soon.

For that reason, picking the right measurements would need to be the logical starting point -- and updated thinking emphasizes that to do that there must be a model or a logic of what matters, not just of what "counts".

What's the difference between what matters and what counts? Observations probably count when they are able to show that they are usually associated with the subject. But showing how their association plays into the importance of the subject at hand is how they mean something. Getting from what counts to what matters means getting from findings to clues to evidence.

I.

As the initial concern of a management process, the notion of "measurement" most top of mind is usually metrics . These metrics are really particular ideas variously used like sculpting tools and fishing tools depending on the circumstances.

But the importance of a higher-level guiding model displaces measurement from that initial position. Metrics gets bumped even a little further away from the starting line by an additional issue as well -- the need to evaluate .

What is the difference between evaluation and measurement?

Typically, measurement determines a state, but evaluation determines the importance of the state. We might determine that it is 28 degrees outside, but having observed the time at which it was true, what does it mean to us? Are we sick? Is it July? Does the temperature matter? How?

But think of it another way as well -- as the difference between a subject and a topic. The subject is what is being talked about; and a topic is a way to talk about an aspect of the subject. Evaluation considers the subject; measurement considers a topic.

II.

Overall, evaluation emphasizes that you not only have to pick the right topics (measurements) for your subject, but you also have to pick the right subject -- which really means to not lose sight of it...

Our friends at the McKinsey Quarterly interviewed John S. Varley, the CEO of Barclay's Bank, who as the new guy in charge had plenty to say to his organization about the difference between what their financial topics appeared to mean versus what they really meant. The short version of the story is that Barclays had fiercely and successfully pursued enviably upward financial results, but largely at the cost of running the company down.

Noting that the organization was managerially unconscious of its "go for broke" mentality, Varley measured the financial performance as positive but he assessed the performance as negative. He saw that the way the big numbers were being hit was actually not sustainable -- in fact, was unhealthy -- for the company.

III.

Assessment takes the broad view of evaluation. It brings perspective to measurements by not allowing them as a topic to overwhelm the subject.

One of the greatest examples of this perspective is Muhammed Ali's famous "Rope-A-Dope" strategy against George Foreman in the 1974 Kinshasa Zaire Rumble in the Jungle. Scorekeeping might have shown Foreman obviously winning round after round on the punch counts, but as the CNN/SI writers put it, "by the eighth round, the 25-year-old champion was running on empty. Ali took advantage to knock out his exhausted opponent..." Ali's bigger picture of the fight, which he'd had at least since the second round, made him the winner.

Another interesting example of the bigger picture is in the Voter Referendum publications in California. Before voting takes place, each issue on the ballot is discussed in printed form with the following points of view:
- arguments for the issue
- rebuttals to the arguments for
- arguments against the issue
- rebuttals to the argument against

For many people, it's a real eye opener to see four different ways to consider the issue. Each point of view is a topic, and each topic is typically fairly persuasive on its own. Taking any one of them, a person might "measure" the ballot issue positively or negatively. But to assess the overall value of the ballot issue (the subject), the four points of view are compared against each other, and so a decision can be made "on balance".

In other words, that assessment model is the key to understanding the real importance of any of the subject's topics.

IV.

When we say that we can't manage what we can't measure, the appropriate response is not simply to start measuring things; the amazing ease of taking things out of context shows that measurement does not equal management.

Instead, by bringing a model to the situation -- one that can define and hold a coherent description of what matters at the subject level -- we get to better observations and decisions using measurements.

It's more fair to say that we can't manage what we can't model -- because without the model, we don't know what is the meaning of the measures and thus don't get a meaningful picture of what we're trying to manage.

All that said, why is it that most times the advice to companies is still to start with measurement?

V.

For one thing, all companies have measures. It's the common denominator. It seems to make sense to "make better use of what you already have..." Often the company doesn't really know another way to talk about its management.

Second, it's a reflection of the need to first actually confirm that a usable picture is obtainable. The real prod there is to investigate how the picture is being taken now. Generally, it's been found that omissions, confusion and inconsistency is significant there, preventing what might be considered a reasonable level of visibility, or producing what might be considered a picture with no apparent composition.

But an even deeper concern of that advice is adherence to the idea that "management" is mainly about efficient production -- one of the top target outcomes of measurement efforts. This is the Frederick Taylor school of scientific management. But the most important difference between 1911 when this school opened, and 2005 well after "hypercompetition" set in, is that the pace of change today is orders-of-magnitude greater and has changed the priorities for measurement -- from accounting for doing the work right, to accounting for doing the right work.

Especially in our era, ineffective efficiency is waste. And that's why deciding what to measure is more important than the measurement itself.

Punchline: in order to manage better, start with assessment. Organizations will find that the key representative concept behind an assessment is an objective.

Posted by Malcolm Ryder at 12:07 PM | Comments (0) | TrackBack

August 31, 2005

When Measurement doesn't help Management

Organizations increasingly manage the support, timing and priority of activities by using metrics. And the activity called "measurement" should not exempt from being managed, so naturally there should be measurements about measurement. This might be a hidden dimension of "management".

If we can't manage what we can't measure, and thus if measurement is a basic component of management, then the performance of our management effort is dependent on the performance of our measuring effort. But how dependent?

In what follows, let's look at how and why measurement is:
- critical but can still fail to support good management; and,
- potentially strategic to good management, if measurement itself is properly managed.

Measurement creates the formal definitions that will be used to identify and track states and events in operations and in the operating environment. So, what we come to "know" about our conditions can be deeply tied to the way our measurement allows us to recognize things.

In the classic hierarchy of data leading to information and information leading to knowledge, definitions generate data, or "facts", that are interpreted by a perspective into information, but information is not knowledge until it is assigned to a class of concepts. (This perspective-driven assignment of information is what we call "context".) Meanwhile, the definition of the concepts is not dependent on metrics, but rather on classification -- a different discipline.

Concepts is where management first stands beyond measurement. At this point, though, what management gains from measurement is the engineering, refinement and/or updating of prior knowledge into current knowledge.

The value of this knowledge then begins with applying it. The two main ways of applying the knowledge are:
- to distribute it; and,
- to use it as a decision-point.

For the purpose of "management", there are two distinguishing keys to applying the knowledge.

The first key is to remember that it is not "information" being applied, but "concepts". We're properly very leery of trying to manage without information; but measurement is only one source of information, and regardless of the source, information is not meaningful without conceptual associations. When information is received and processed, what should be further distributed for management's sake is the concepts. That is, in the next step, communication should drive the simultaneous, interlinked distribution of both the information and the meaning.

The second key to the management-oriented application of knowledge is to see that action is another way of propagating concepts. When actions are classified -- by purpose, impact, and propriety -- they can represent a model of operations that describes how the organization qualitatively distinguishes itself from others. Decisions typically select preferences that in turn select and promote actions. This is what "management" does with knowledge that drives opertional performance. Additionally, it reinforces or proposes the prevailing desired identity of the organization.

Because of what it represents, proposed or dictated action becomes a form of communication in itself. Starting with decisions, management must do a good job of forwarding both the ideas and the actions that matter.

This forwarding or promotion is the second point at which management stands beyond measurement. For many reasons, managers promote things according to preferences and tolerances, not just neutrally or automatically. Whether the driving reasons are psychological, political, tactical, or whatever, promotion can run independently of measurement.

However, when the promoted actions and ideas are then monitored and measured for their timing, compliance, impact, etc., management comes full circle as there will be new data generated about resulting states and events. Recognizing states and events can be facilitated by using the language of measurements, but not all states and events will have already been discovered before: unprecedented effects will occur and will need a way to be identified beyond the current terms of measurement. Research on possible effects should extend beyond the local or immediate management milieu, and measurements should be upgraded to accommodate the newly acknowledged effects.

In review of the above, looking at how definitions, concepts and promotion line up offers the picture below:

How good is your management at getting all the way around the circle? This question asks about the performance of management itself. Seeing that there are at least three major components in the cycle highlights the idea that measurement will be a prerequisite of good management but not necessarily a cause of good management.

The ideal situation would be that measurement is refined to the point where it is not just critical to management but actually strategic. This would mean that measurement should have both methodologies and deliverables that actually make management's overall effort the differentiating factor in generating the value of what is being managed.

We know that poor measurement can be a significant inhibitor of good management, because measurement is critical -- that is, it is a prerequisite whether it is helping or not.

By explicitly addressing tie-ins with knowledge and communications, measurement stages an opportunity to become strategic and not just critical.

Note: The discussion above focusses on characterizing the relationship of three types of "business information" (measurements, knowledge and communications) in a singular management lifecycle. One online resource of many in-depth papers more conventionally discussing measurement is the "TechRepublic" free-membership IT library. A good current example of the available content offered is the whitepaper The Metrics of IT: Management by Measurement from the Enterprise Computing Institute -- reached via TechRepublic membership at this URL: http://techrepublic.com.com/i/tr/downloads/home/metrics_of_it.pdf

Posted by Malcolm Ryder at 6:54 AM | Comments (0) | TrackBack

July 28, 2005

Managing without Metrics?

Metrics mania is for real, and not having it could be more of a problem than the problem we have with surviving the mania. Why? Most managers would not be able to communicate effectively without metrics, and with the pace of change in business, new metrics are necessary. The intense community-wide pursuit of more metrics means that there is always a better chance that the new ones you need are already discovered and available.

But management didn't grow from metrics; management adopted metrics. There is something deeper than metrics that characterises true management.

What is it?

Here is the really big notion that always drives my own sense of what "management" means.

Start with the idea that there is initially an item of some kind (an organism, process, object, etc.) going through changes and/or behaving in various ways, independently of your influence on it. Then, as soon as you have a particular objective for the behavior or changes of that item, you begin attempting to exert some logical influence on the changes or behavior to cause it to conform to your objective.

But you don't just take your best shot at that and see what happens.

You sustain attention and influence on the item's changes or behavior as much as is necessary to maintain and/or improve its conformity to your objective. This effort includes the possibility that you will at some point change your objective but attach it to the same item as before. Practically speaking, that may cause you to change the specifics of how you influence that item's changes and/or behavior.

As a result, you'll be busy with determining why things you've been doing are working well or why they are not -- and that is a two-way street: how manageable is something, and just how is it manageable?

The key to succeeding in your influence is to identify the characteristics, attributes, properties, etc. of the item that you can "engage" with the characteristics, attributes, properties, etc. of your mode of influence, such that your influence is *at least virtually* causal towards your objective.

I say "virtually" only because in practice the success of "the influence" is sometimes hard to confirm as being a coincidence versus a direct effect. I usually define that more precisely -- as confirming whether something is a prerequisite (exhibited by correlation) versus a cause (exhibited by testing). A prerequisite allows something else but does not necessarily cause it.

But here you can see why management is essentially dependent on a logic. The whole idea of management assumes that your influence on something is effective because it is "logical".

This means, furthermore, that obsessively measuring stuff outside of a logic of action to influence progress towards a designated objective (other than measurement itself) is NOT "management".

At the same time, the notion of "measurement" needs to be understood broadly. Here, the working definition of measurement is:

the determination of a significant distinction,
with the type of distinction being already named so that
comparison is straightforward.

That is, measurement determines that:
something is, to a discernable degree, of given state X,
and is therefore not of some other degree or some other state.

In the practices and measurements of a particular field, what becomes really crucial to management is to decide the reasons and standards for using the "degrees" and "states" being researched, referenced or communicated. From that, a logic of influence can be agreed and developed, and measurements appropriate to the logic can be cultivated.

Posted by Malcolm Ryder at 6:46 PM | Comments (0) | TrackBack

July 20, 2005

Managing versus Measuring: IT's Value to Business Performance

At CIO Decisions Magazine, Thorton May serves up results of an excellent survey of approaches being used to understand the value of IT to the business.

In general, the results indicate that the practicing managers of corporate IT range hugely in their established opportunity and intent for representing the way IT impacts business performance.

The survey does not investigate what lies behind the opportunities, so we don't get into the availability of time and tools that create the "business-perspective" visibility on IT. However, it is axiomatic that the way you look for something determines most of what you can see. In that regard, "formulas" that represent IT influence are more important than the tools and time used to apply them.

The four most important "takeaways" of Thornton's discussion are:
- IT is pervasive in the business the same way that management is pervasive, so it may be illogical to try to isolate "IT value" as a single global variable except in the various specific contexts (occasions) of IT usage.
- There is a difference between "value" and "performance"
- Measurements representing IT influence are not always numeric.
- Too much measurement is more bad than good.

One possible cause of the persistent confusion about value in IT is that the effort to connect IT Value to Business Performance doesn't make sense until after the connection of IT Performance to Business Value has been established.

If IT outputs are first associated with conditions that the business defines as important to the business, then a logical representation of "IT performance" exists. IT execution can be measured in those terms of performance, thus a picture emerges of when, how and why IT contributes to the enablement threshholds that the business agreed are needed for business functions to have their shot at meeting business goals.

Thus having the "business value" of IT performance defined, managing IT execution is a straightforward effort to have IT performance meet business needs. As a start, this is exactly what should be represented by a service level agreement (SLA).

Then, the performance of the business is nothing less than the results obtained from the business's execution of functions enabled by IT's performance. Presumably, the business has some functional targets to hit, which will be how business execution will be rated for its value to the business.

The biggest issue emerging from this is the need to separate the business's achieved capacity of enablement from the business's achieved competency of capacity utilization. IT can provide the capacity of enablement, but (contrary to the mythology of automation) IT simply cannot make the business use the enablement wisely.

If the business does not have working definitions of those two things, that furthermore unfailingly distinguish them from each other, there is no reason to believe that IT's influence on the business performance can be logically designed, tracked or analyzed. Naturally, this also means that no attempts to determine ROI are actually meaningful; instead, they are simply sophisticated statistical fictions prone to being evangelized or rejected by company politics.

The only exception to that assertion is the case of IT actually preventing a business function from being executed -- a huge issue driven by the way that the business relies on IT. In this case, we think in terms of a "negative contribution" -- one that is usually either unexpected or not officially tolerated. But in this case, what is important is to not suddenly have a double-standard of measurement methodology. If the contribution measurement philosophy is based on describing how IT relates to intended consequences, then the description must also neutrally cover how IT relates to unintended consequences. As we do this, it is important to neutrally separate IT's relation to desirable consequences and undesirable consequences.

This neutral and comprehensive perspective, which prevents arbitrariness in political tolerances, forms the basis of understanding otherwise neglected issues such as opportunity costs incurred by the influence of one IT implementation versus another. Since the business is responsible for deciding on those tradeoffs, it is clearly a matter of management and strategy more than of IT per se. When "negative contributions" occur, they must be defined and recognized -- as the results of:
- either bad decisions or bad enforcement that should not be politically written off;
- or bad luck (such as natural disasters or external attacks) that probably should be politically written off.

Posted by Malcolm Ryder at 8:11 AM | Comments (0) | TrackBack

June 19, 2005

Evaluation versus Value

Evaluation closes the loop of management activity that begins with idea definition and continues through design, implementation, operation and support.

Following and actually overlapping "support", "evaluation" determines when the effects of operation are complying with initial standards and objectives, and whether the effects are compatible with requirements that have emerged and become newly current during the elapsed time of the idea's actualization and employment.

For managers, that current compliance and future compatibility mean something primarily because of the initial justification for the idea being actualized. This justification is often simply called the "baseline", and one of the primary goals of evaluation is to generate information that supports decisions about how to respond to changes from the baseline. As a result, performance information feeds change management to protect value.

Normally, the justification is a description of "value" to be gained by realizing the idea. Compliance will mean that the terms of the original justification are being met, but compatibility to new requirements means that expectations about future value are still reasonably positive.

An evaluation that cannot include both perspectives -- that is, of original and current expectations -- is at least incomplete, if not downright suspect. For example:
- an evaluation based only on the (earlier) original expectations might describe work done properly to produce something whose intended impact is no longer possible or relevant by the time it is actually delivered.
- an evaluation based only on current expectations, ignoring the previous forethought behind how the current conditions developed, might mistake a presently mismatched organizational stance towards requirements for a lack of capabilities instead of a lack of direction.

Another problem in evaluations is confusion about what approach yields information of the right type of importance. For example, these three following approaches are distinctive enough in purpose and capability; they can and should be complementary but are unlikely to successfully do each other's job:

Assessment - a determination of what kind of value should be associated with something, not just how much something is worth. In determining what kind of value, the assessment process examines how something actually creates value, and/or how it fails to do so.

Measurement - a determination of the degree to which something has a given quality or property.

Analysis - a determination of something's constituent parts and especially their inter-relationships, explaining a particular characteristic of something.

As an example of why these must be correctly used, look at the common need that organizations have to conduct performance assessments, performance measurements, and performance analyses. Casual conversation might use these three terms interchangeably, but in actual practice they will not correctly provide the necessary decision support if misused.

A performance assessment should investigate, and report on, the likelihood that an operation can meet the demands of given types of requirements. Along with this, but regardless of what current requirements are the established ones of record, a performance assessment should expose what types of requirements the operation is most likely constitutionally fit to meet, based on knowledge about how such requirement types have been successfully met in the past. This is usually where notions such as best practices are referenced.

A performance measurement should investigate and report on what level of compliance the operation has achieved regarding specified target levels of impact for certain types of impact. Typically, benchmarks are referenced in measurements instead of elsewhere.

A performance analysis should investigate and report on why measurable performance levels have occurred, regardless of what the measured levels are; then based on the reasons why, the analysis should predict the levels and types of performance that are most likely from the same dynamics. Forecasting is usually expected here.

However, the three approaches are complementary:
- Assessments and Measurements can share categories
- Measurements and Analyses can share data
- Analyses and Assessments can share models in common.

Taken together, the three approaches provide a complete reference for describing the relationship of operational effects to the objectives that for managers represent creation of value.

A full evaluation that includes assessment, measurement and analysis will give a description that traces the logic of what is actually happening in a way that accounts for expectations. In that way, managerially practical comparison of original expectations to current expectations should be possible, identifying the underlying conditions that are shaping or being shaped by operations.

Posted by Malcolm Ryder at 7:41 AM | Comments (0) |