This article was originally published on July 12, 2010 on the website of The Hauser Center for Nonprofit Organizations at Harvard University.
Several powerful donors have concluded that nonprofits make inadequate use of impact assessment tools. They are backing up their arguments with an implicit threat: measure in particular ways or you don’t get the money. Wise nonprofit leaders know that the problems they work on are not susceptible to simple measurement. They know that the kind of formal impact measures some donors expect and management consulting firms prescribe are hard to come by honestly. They collect various data all the time to inform their judgment and decision-making and to spur learning. Now, data collection (to donor-specified standards) is increasingly used for accountability purposes.
This may have the effect of reducing the degrees of freedom nonprofit leaders have to innovate and to pursue promising but risky ideas (without the fear that failure to prove one idea will poison their chances to learn from that failure and try something else another day). As former Ford Foundation President Susan Berresford argues, insisting that grantees demonstrate measurable, short-term impact can have the effect of “miniaturizing ambition” for doing risky but potentially break-through work.
People who impose these restrictions confuse use of prescribed tools or achievement of certain outcomes as evidence of good management. Sometimes they are. But, in and of themselves, they hardly constitute an impressive tool kit of good management practice.
The good judgment of experienced managers, deeply immersed in the complex social dynamics of the communities in which they work, is a formidable and essential resource in assessing impacts. Experience and tested judgment also come into play in shaping a picture of the complex variety of social factors that might explain, for instance, why some poor children and not others attend school, or what mix of interventions are most likely to keep kids out of trouble with the police.
Effective nonprofit managers get information from a variety of sources: formal studies, observation of trends in behavior, feedback from partners and clients. They also draw on deep reserves of knowledge of the local social context, of cultural norms and values, and on the ability to empathize, to look at the world through the eyes of others.
These sources of knowledge are particularly important in shaping untested but potentially innovative, breakthrough approaches to social change. Effective leaders first and foremost seek to explain how a given problem is responding to a given set of interventions. Data help describe what is happening, but the interpretative powers of managers are essential to meaningful explanation.
One of my favorite examples (see working paper here ) of the kinds of insights that arise from observation, judgment and experience is the particular knowledge that Muhammad Yunus gained from walking through poor communities around Chittagong University in Bangladesh on his daily walk to work. His knowledge of rural Bangladeshi society, combined with his advanced training and powers of intuition, spawned his ideas on social lending, or what became known as micro-finance.
The invention of micro-finance demonstrates that breakthrough innovations, and even simple adjustments to well-established programs, are spawned by a variety of sources and intellectual attributes: data, data intelligently interpreted, knowledge of the local and comparative contexts, and good judgment. All four of these factors are essential to shaping development breakthroughs. Donors should give greater weight to the latter three over the first in considering funding proposals.
A recently published book on the use of applied mathematics to help understand messy, hard-to-measure problems speaks to the importance of experience and judgment in making sense of limited data. The book is “Street-Fighting Mathematics: the Art of Educated Guessing and Opportunistic Problem Solving,” by Dr. Sanjoy Mahajan. Dr. Mahajan is associate director of MIT’s Teaching and Learning Laboratory and the book grew out of a course by the same name that Dr. Mahajan taught for several years at MIT.
The basic premise of his approach, set out in the books first sentence, is that “Too much rigor teaches rigor mortis: the fear of making an unjustified leap even when it lands on the correct result.” Many real-world problems are not easily described with the kind of precision that professional mathematicians insist upon. This is due to the limitations of data, the costs of collecting and analyzing data, and the inherent difficulties of giving mathematical expression to the complexity of human behavior. In the face of these obstacles, mathematicians tend to do one of two things: insist on finding the true proof, even in the face of huge methodological constraints (rigor mortis) or give up.
Mahajan counsels a third-way: using mathematical reasoning to find a good-enough, approximate and usually valid and useful answer; or as Dr. Mahajan so adeptly puts it, “When the going gets tough, the tough lower their standards.” His book describes six tools for better understanding complex problems with limited data, including picture proofs, lumping, and reasoning by analogy.
There is wisdom in Dr. Mahajan’s core argument that is relevant to current debates about the place of impact assessment in program management. Many problems, especially problems of social analysis, present huge problems of description and accurate measurement. W