When measurement kills learning

by Art Kleiner and George Roth

This article appeared in the Journal for Quality and Participation, Vol 20, #5, Nov/Dec 1997. The draft here is the one we turned in, prior to the JQP editing process; so it is different than the published version.
JQP is published six times a year by the Association of Quality and Participation, 801-B West 8th Street, Suite 501, Cincinnati OH 45203-1607.

Suppose that you head up a department lets say, an engineering department and you recognize that the unspoken attitudes and working practices of your managers and employees are keeping your group from working effectively. So you invest several hundred thousand dollars in an in-depth consulting and training effort, complete with computer-simulations of your interoffice activity, to help people develop better capabilities and more entrepreneurial attitudes. You don't just bring in the consultants; you spend hours working with them, helping develop a learning-oriented approach that fits with your own business demands and the culture of your enterprise.

Sure enough, it pays off. After a few months, your department's new product development cycle time speeds up dramatically. Rival functional subgroup leaders report that they now feel free to talk about common problems, instead of hiding them from each other. Moreover, as the year moves on, a new kind of awareness builds in the group. People come to mutually understand each others' problems, and even sacrifice their own narrow interests to help other reach solutions. Together, they turn their attention to the systemic issues that have mutually held them back for years.

Everything makes bumpy but steady progress, and you feel a growing sense of satisfaction with your results. Then, at the end of the year, the corporate HR staff conducts its annual attitude and motivational study. Immersed in the new approach you've helped develop, you gleefully expect your improved performance to show up in the surveys. But instead, the learning team's motivations and awareness are marked "much worse" than those of other departments.

When you think about it, you realize that the disparate results make sense. After all, there is actually more dissatisfaction visible when people are learning. They are more aware of the gap between their aspirations and the corporate realities. They see the depth of the problems and issues surrounding them; they see that no individual will solve those problems acting alone, or even within one team's bailiwick. And they're more open and candid about expressing these views. In that light, this one report seems inconsequential; easy and safe to ignore.

Yet the attitude and motivational study is only the first of several negative pronouncements from outside your department. Other corporate studies, and the gossip of the "old hands" of the enterprise, join in denouncing the learning effort: While the group's production results are better than those of other groups, it still "looks worse" by conventional measures to the rest of the company. And those outside the innovative engineering department reject any insights that come from your group.

Meanwhile, inside your group, learning begins to atrophy as people respond to the survey results. Since cross-functional relationships will obviously go unrewarded, the engineers focus less on their own learning, and more on short-term performance. Instead, eager to look good for future assessments, they play up their own individual "best practices" and shut down riskier approaches. More importantly, the rest of the company's leaders, whom you were hoping could profit from your example and change the company as a whole, have lost any sense of value in the experience of your pilot enterprise.

You know that somewhere, buried in your team's experience, is a message that the organization as a whole needs to hear. But the organization lacks the capability to listen to itself.

This example may be fictional, but it is typical of the experience of innovators in a wide range of corporations. Throughout corporate America, people are trying new approaches to build organizational learning the ongoing development of peoples' collective capabilities, in the service of collectively desired results. Executives, in particular, have come to recognize that, with the right approach to collective learning, their enterprises can continually gain new talents and capabilities as they weather the vicissitudes of fate. Managers in middle levels, meanwhile, have embraced the "learning organization" idea because it encourages people to follow their own aspirations and, at the same time, boost organizational performance. Individual and organizational efforts reinforce each other people can reclaim a spirit of community and personal involvement as they make business decisions that increase corporate profitability. For these reasons, most types of corporate change continuous improvement, total quality, reengineering, organizational development, "complexity" systems, and a host of others have been recast as "learning" initiatives.

For many managers, however, a critical question persists: "How do we know that learning is taking place?" Executives authorize millions of dollars for organizational learning, reengineering, re-invention, or quality efforts which involve hundreds or thousands of people in new types of training and in-depth process improvements. Then the leaders grapple with the problem of assessing their investment. As professor/writer David Garvin has pointed out, the expression "If you can't measure it, you can't manage it" should also apply to organizational learning. But with something as intrinsic, subjective and tacit as learning, judging the value of a learning effort is itself a new kind of activity for corporations. It will take new kinds of attention to tbe methods of gathering data about learning efforts, the ways you judge success and failure, and even the ways you celebrate and reward yourselves. Until then, your efforts to measure learning, to analyze its effects, and to replicate its success will tend to stop, dead in its tracks, the very learning that you are trying to nurture and grow.

To be sure, measurement doesn't always kill learning; but it seems to happen more often than most managers expect. There seems to be some kind of equivalent to the "Heisenberg Principle" of physics in which observing a phenomenon alters the phenomenon itself at work whenever people are surveyed or questioned about the value of their new initiative. The basic underlying cause, we believe, is the way that managers allow measurement to stand in for implicit judgments that never get questioned or examined. This manifests itself in at least three different ways.

First, when typical assessments are conducted, the results are presented back as a set of "answers" to a sponsor of the project. Whoever (sponsor or evaluator) has the most creditable report may well see his or her judgment prevail in determining whether the activity is worthwhile or not. To "win" at this game means to have your position be persuasive to have numbers that seem convincing instead of to consider the pros and cons of all the various "positions" together. The more that people get caught up in the need to lock in their positions, the more they forget about experimenting and gaining new knowledge through open-minded inquiry. After all, when you're struggling for tactical advantage, genuine learning can compromise your efforts; because learning cannot take place unless you are willing to admit that your theories and methods might be incomplete or wrong.

Second, there's an irresistible pressure on people being evaluated. As they become aware of being measured and judged, they "perform" for the evaluation. They try to intuit and satisfy assessment criteria instead of continuing to focus on improving their capabilities. The intrinsic aspiration which drives learning is supplanted by drive to look successful, the pressure for extrinsic rewards, and the fear of reprisal for not "measuring up."

Third, even when assessment experts recognize these problems, their analytical stance limits their abilities to inquire into the "softer" sort of changes which lead to "harder," more visible results. Performance gains brought by the innovation often dissipate as the assessment process diverts peoples' attention away from their original enthusiasm.

During the last few years, we have come to recognize all three of these dynamics in a variety of locales. One of us (George Roth) has studied reengineering efforts, where it regularly pops up as people try to assess their ambiguous results. The other (Art Kleiner), in his book The Age of Heretics (1996; Doubleday) -- a history of the forerunners of the corporate change movement -- has chronicled a series of influential change efforts, dating back to the early 1960s, that foundered on the shoals of attempts to evaluate them. In 1994, our understanding of the problem came into greater focus, when we began to develop a new kind of corporate communication tool called "learning histories." These are in-depth stories of corporate change efforts, designed to help other parts of the company learn from their experience. In many of the learning histories we have worked on, the problem of assessing change takes center stage.

One prominent learning history, for instance, chronicles the fate of the "Epsilon" automobile launch team, at a company we call "AutoCo." (All identities in a learning history are disguised, so that we can tell a candid story with all the difficulties and dilemmas intact.) This document tells the story of a 200-member team which initiated a number of groundbreaking innovations. They set up "learning labs" and other types of in-depth, reflective activities, developed in collaboration with MIT's Center for Organizational Learning. In these meetings, people learned to combine engineering and "human communication" skills to solve difficult systemic problems that had held up production in many other car launches. As a direct result of this combination of "soft" and "hard" learning (or so the team leaders believed), the team surpassed many production records, producing a car ahead of schedule and below cost. Yet, in the end, the team leaders were offered positions lower than they had expected, the car did not sell well, and the project suffered an ambiguous reputation within the company.

From the moment they began, the project leaders of the Epsilon car launch had been caught in an impossible situation a story line out of Joseph Heller's novel Catch-22. Heller's protagonist Yossarian pretended to be crazy to be discharged; but no one who wanted to escape could possibly be considered crazy, and therefore his requests were ignored. The Epsilon leaders put on a front of being "normal" managers, in order to influence the company but no one who wanted to change the rules they changed could possibly be considered normal, and therefore they were ignored. Or so they felt.

One quote from a launch team leader shows the tightrope they were walking whenever they tried to demonstrate the value of their new approach:

If I were 35 years old and worried about getting promoted, I couldn't have taken those risks. I guess I'm old enough and I've been around long enough, and some of the things I was doing with MIT were changing my mindset. The change wasn't with them. The change came from within me.

At first, there were reasons to believe that the rest of the company was paying attention to Epsilon's superior performance. But gradually, it became clear that senior executives of the company had begun to discount the results."I've had a unique ability for the last 10 or 15 years," said one key executive, "to say what's ready... [and I'm] right 95% of the time. The patient is terminal." Another executive compared the team members to disciples in a cult. And ultimately, a potentially healthy signal requests for component changes proliferating early enough to spark inexpensive solutions led outsiders to conclude that the project was "out of control." To cope with outside criticism, the team leaders started to hide their innovative practices from view, which meant that the team never managed to follow through on its early potential.

Almost exactly the same sort of outcome had happened 25 years earlier, in one of the most celebrated modern corporate change stories. In 1971, a Gaines dog food plant was opened in Topeka Kansas set up around sociotechnical, participative, team-management principles. Workers were paid for their skills and knowledge; team leaders rotated and teams managed the parameters of their own work; and performance, at least in terms of cost and quality, outclassed any other plant in the General Foods system. Years before the quality movement took hold in America, Topeka employees pursued an ethic of continuous improvement. To be sure, other plants paid more meticulous attention to collecting data on quality control and pricing data that they sent back to General Foods corporate headquarters in White Plains, New York. But at Topeka, collection of data for faraway corporate staff was a secondary goal. The teams of operators concentrated on gathering the data that they needed themselves, to improving their performance on the line. And senior management, at first, kept hands off the new plant. One vice president gave his permission for the new approach by saying, literally, "You are free to fail."

As long as the plant operated in isolation, it continued to innovate. But gradually, word began to spread, elsewhere in the General Foods system: "Topeka can't make its numbers." Before long, people from Topeka found it difficult to get jobs elsewhere in the General Foods system; and the two managers most responsible for starting this plant, Ed Dulworth and Lyman Ketchum, were forced to leave General Foods under suspicion of incompetence. The next manager at Topeka was told to "cut out this missionary crap." He couldn't stop the team-based system from continuing indeed, it outlived General Foods, which sold the plant to Anderson-Clayton in 1979 (before its own takeover by Kraft). But the pressure to meet conventional standards of measurement was felt increasingly strongly. We cannot say for sure that this new set of controls was responsible, but the plant's innovation effectively entered a state of suspended animation. It did not decline; but it did not improve. Under three more corporate owners, Topeka operated in much the same way, through the mid-1990s, that it had in the mid-1970s.

"Don't try to change the boss' minds," many heretical managers decide, "until you can back up your argument with results." And, indeed, the leaders at both the Epsilon car launch and the Topeka dog food plant followed this course. Unable to see any other way to communicate, the leaders of these local "organizational learning" efforts tried to convince senior managers they were "right" using conventional progam metric data. The frustration they felt is evident in both stories. Years later, Ed Dulworth would remember other General Foods managers telling him, "My boss isn't really interested in performance." Managers didn't rise through good performance; they rose by manipulating the results to make themselves look good.

As for the leaders of the Epsilon team, they never felt they had a choice. They had to find a way to engage senior management. Given the fierce time pressures on people at that level, they felt they had no choice but to produce evidence that would force senior managers to take them, and their evidence, seriously: But the evidence never emerged, until it was too late or, at least, when it emerged, senior management didn't recognize it as compelling evidence for changing the nature of product management at AutoCo. As one of the corporate staff people familiar with the project pointed out:

[The team leaders] were saying to themselves, "We did everything right. Everybody at AutoCo should be pleased. When they see the results, they'll knock the doors down trying to learn how to do what we did. And I'm sure I shared in some of that attitude myself.

But in retrospect, that was a naive approach to the world. If we expected results and teamwork, in themselves, to communicate our message to the rest of the company, then we were setting ourselves up for disappointment. And we have to be careful not to blame everyone else for not recognizing us in the way we hoped they would.

Assessment versus measurement versus evaluation

Why should assessment cause such problems? In part, because the very concept of "assessment" is emotionally loaded. It derives from the Latin root assessare to impose a tax, or to set a rate and it often seems to invoke a feeling of being persecuted by an auditor. People who contemplate assessment report palpable fear the word draws forth a strong, gut-level memory of being evaluated and measured in school.

But at its best, "assessment," is simply the comparison of reality to expectations. Organizations need assessment to build judgment: about significance They need to develop a sense of "What we sought to do," versus "what we did. The problem occus when. "assessment" is conceived of as "measurement." Certainly, the benefits of measurement are undeniable: they allow the comparison of performance across a large number of teams, projects, processes, and activities. As educator Robert Gahagan has noted,

We can but not with numbers alone. Significance requires words to make sure that everyone understands each others' interpretations of the numbers. As consultant/writer Fred Kofman points out, accounting measurement is a form of language, and language determines what we perceive. The way in which corporations count "beans" indicates which type of "beans" are valued, and which are not, in a way so subtle that it determines the subconscious focus of peoples' attention. After a learning effort, people may see the "beans" differently. A rigid measurement scheme, like that of financial accounting, might not recognize the effort associated with people's learning, or people may limit their learning in order to comply with the perceptions a measurement system enforces.

That's why we prefer to use the term "evaluation." Evaluation involves values and valuing, deriving from the Old French evaluer, "to value." To evaluate means to determine the worth of, to find the amount or value of, to appraise.

Judgment, in this context, is subjective and true evaluation embraces subjectivity. To evaluate, we must be willing to do the difficult work of looking underneath our judgments, including our judgments of people. Evaluation is rare in corporations precisely because managers often shrink from expressing their subjective judgments. But the problem is not with subjectivity itself; the problem lies with finding a venue through which subjectivity can be expressed constructively.

A framework for initiating evaluation

To start with, managers need to recognize that evaluation and assessment themselves can be activities suffused with learning if they are set up to build the collective insight of the people responsible for the new project, and others in the organization as well. To accomplish this, every stage of a learning effort requires a deliberate attempt to engage people, in a way that helps involve them in one of the most critical questions of any initiative: How do we know that we're doing well?

Diagram of the stages of a learning or corporate change effort, used as a diagnostic for knowing when to raise questions about the criteria for measurement and evaluation. The black arrows show the progress of a learning effort -- from a new intervention to developing skills, to deliberate changes at a "pilot team" level, and on upward -- eventually to impact the business organization as a whole. Each "level" throws off different measures of success: Tests of skill, behavioral indicators, or noticeable results. At each stage, the thick grey dotted arrow shows where to raise inquiry about the evaluations that will be needed later. Thus, for instance, when starting the learning process, it's important to go up one level to leaders of the pilot team and ask: What behavioral indicators will represent success?

This diagram shows the progress of a learning effort, in a way that helps people recognize the effectiveness of timing. Imagine that you have started such an effort: A program not just to increase quality or performance, but to systematically increase the capabilities of the people at work. This will require new types of conversation and thinking; it may even require some intervention, such as a "learning laboratory" or other practice field where people are rewarded for experimenting with their work.

At first, when you're still at the bottom rung of the ladder, only a few people may be involved in the intervention. They may take a training program or a course; they may engage in their own investigation. How can you tell that anything is going on? You can't except through the evidence that people are developing new skills. Thus, you (as the evaluator) are likely to want to test those skills, through surveys, examinations, and informal assessments (like asking people if they feel more capable when they walk out of a training session.)

New skills, in themselves, don't help collective purpose very much. To aid in mutual accomplishment, they need to be used. But already, at this level, it's necessary to raise the question: "How will we judge our success? What criteria will we set to know we're doing well?"

At the same time, you need to draw in the people at the next higher level: "Boss, we don't know exactly how this will turn out. We have our own ideas, of course, and we are monitoring our results very closely. But we need to know, before we go any further: In this unprecedented effort, how would you define achievement? What would have to happen for you to judge this effort a success?"

Understandably, this is not an easy statement to make in many companies. Telling a boss that you don't know something, as you embark on a new project, is risky for both of you. It indicates that you might well make a mistake or create an embarrassing situation. Not only is your boss associated with you, he is partly responsible for your results. When that effort is an experimental approach with potential implications for others in the organization, he knows that everyone involved will be exposed to a high degree of public scrutiny. By asking for his judgment about your efforts, you are overtly asking him to take on part of the risk and responsibility for your success or failure. You are saying, in effect, that being "free to fail" is not good enough.

Organizational learning theorist Chris Argyris, of the Harvard Business School, has identified three levels of thinking and actionthat limit learning in many organizations. All three levels come into play in this situation. First, mistakes are covered up. Next, the act of covering up mistakes is undiscussable. Finally, as Argyris' research has shown, the undiscussability of covering up mistakes is itself undiscussable. Most senior managers don't need to be told about about these three levels; they "sense" the looming difficulties, when asked to be supportive of a learning initiative. They are often unable to articulate this sense or their reasoning, and are uncomfortable with substantive discussions around learning issues. They have a tacit knowledge of the norms and shared assumptions which govern behaviors. They know that the process of learning  making mistakes, initial incompetence and possible embarrassment - is itself undiscussable in most organizations. And they generally don't have or can't take the time to learn the skills needed to safely address what they inately sense as "dangerous" situations.

Thus, the ostensibly prudent action for a boss is to avoid any of this risk altogether. "I'll just quietly let them learn as long as they don't get too threateningly visible." As a result, though the senior leaders may be conceptually supportive, they may also send mixed signals. They speak of commitment to learning, but their behavior at meetings shows that they expect the new efforts to fail.

At the earliest stages, however, it is easiest to break this pattern by drawing in the boss to consider how this new effort might succeed. What frustrations and challenges exist at the boss' level, which this new project could address? What dangers and opportunities does he or she see, which are not as obvious within its midst? To get at these questions, you need to lay the groundwork for anticipating the most visible outcomes: Behavioral changes in the company's practices and culture. People, by this time next quarter or next year, will probably act differently: What types of different actions, in everyone's view, would represent a step forward? A two-hour conversation can lay the groundwork for the boss to feel genuine responsibility in the effort, as not just the objective "judge," or the benign paternalistic parent, but a sincerely engaged individual.

Six months or more might pass, with conversations every couple of months to reconsider the chosen criteria (and map your progress). At some point, you will move up to the next rung of the ladder: to "pilot unit" activity. Now, there is some middle-range group of people, perhaps 25-300 people, which has begun to accomplish miracles. (Anyway, they think they're miracles.) The emphasis within the group has shifted, from marvelling at the skills they've learned to marvelling at the results they can produce. Now, it's vital to raise the question of criteria for success again not just with the boss, but with people outside the effort, who will be inevitably watching, and judging, its progress.

Note that you are not asking them in advance for approval? You are making a sincere inquiry about their judgment: What will constitute success, as this innovation reaches more broadly into the fabric of the organization? Now, at long last, you are ready to talk about results. "If this learning and change effort has been successful," you can say, "what results might we expect to see?"

Techniques for raising these questions

The framework we present here is only a starting point a plan for proceeding over time. You will also need a vehicle, a way to bring reflective evaluation, with all of its ramifications, to the forefront of a team's activity and conversation. Thus, in developing the learning history -- a form of side-by-side story-telling and commentary -- we have tried to create a method of assessment which frees people from the tyranny of a predetermined measurement and evaluation scheme. The learning history process does not deny the value of measurement, or the existence of measurement schemes in most organizations. Indeed, in learning history projects, all three types of judgment are made explicit:

We offer measurement of significant improvements if we can describe how and why the measurement systems themselves were developed, and what impact the measurement had on the people being measured.

 We provide assessments, made by people throughout the organization, with a clear link between the assessment and the direct observable data.

We present overall evaluations, always with enough context that the reader (and interviewee) knows, beyond a doubt, that the evaluation stems directly from the sense and meaning that people make of their own experiences and work.

We have learned, through our work with learning histories, that stories themselves are a kind of data particularly well-suited to capturing the dynamically complex nature of corporate learning. Corporate change efforts unfold over time; they involve large numbers of people, separated not just by space, but by time. Events that take place in California in June may have great influences on what happens in the Hague in December. Stories told by participants capture these elaborate chains of influence, while traditional measures from surveys and questionnaires provide only snapshots of static events. Even large-scale surveys aggregate variations in perspective, translating everyone's experience into rankings and hierarchies of numbers. They thus blur the intangible awareness that people get by reading the distinctions found "between the lines" of a person's story. With multiple stories woven together, people can make sense of complex, ambiguous processes by relating other peoples' perceptions to their own. If a picture is worth a thousand words, a good story is worth a thousand pictures.

The greatest value of the learning history (and from other new forms of organizational evaluation and reflection) come not from the artifacts, but from the conversations that result. That, in the end, should be the purpose of evaluation and assessment: Not to set people's efforts up on a scorecard, but to spur a mutual sense of the future. What did we really try to accomplish this time? How did we do? And how should we judge our success or failure, in our next endeavor? To ask these questions, early on, will help any learning effort reach its goals more effectively.

About the authors:

Art Kleiner and George Roth are the founders of Reflection Learning Associates, an organization created to develop and foster the learning history practice. Kleiner is the author of The Age of Heretics (Doubleday/Currency, 1996) a history of the thinkers and practitioners who sparked the modern organizational change movement. Roth is a lecturer at the MIT Sloan School of Management, and director of a joint research project between MIT and Ford Motor Company. Both are coauthors, with Peter Senge, Charlotte Roberts, Rick Ross, and Bryan Smith, of The Fifth Discipline Challenge, a book in progress by the same team that produced The Fifth Discipline Fieldbook. For more information about learning histories, see the RLA website at: http://www.fieldbook.com/rlearning.htm.