alt

This article was originally published  on July 12, 2010 on the website of The Hauser Center for Nonprofit Organizations at Harvard University.

Several powerful donors have concluded that nonprofits make inadequate use of impact assessment tools.  They are backing up their arguments with an implicit threat: measure in particular ways or you don’t get the money.  Wise nonprofit leaders know that the problems they work on are not susceptible to simple measurement.  They know that the kind of formal impact measures some donors expect and management consulting firms prescribe are hard to come by honestly.  They collect various data all the time to inform their judgment and decision-making and to spur learning. Now, data collection (to donor-specified standards) is increasingly used for accountability purposes.

This may have the effect of reducing the degrees of freedom nonprofit leaders have to innovate and to pursue promising but risky ideas (without the fear that failure to prove one idea will poison their chances to learn from that failure and try something else another day).   As former Ford Foundation President Susan Berresford argues, insisting that grantees demonstrate measurable, short-term impact can have the effect of “miniaturizing ambition” for doing risky but potentially break-through work.

People who impose these restrictions confuse use of prescribed tools or achievement of certain outcomes as evidence of good management.  Sometimes they are. But, in and of themselves, they hardly constitute an impressive tool kit of good management practice.

The good judgment of experienced managers, deeply immersed in the complex social dynamics of the communities in which they work, is a formidable and essential resource in assessing impacts.  Experience and tested judgment also come into play in shaping a picture of the complex variety of social factors that might explain, for instance, why some poor children and not others attend school, or what mix of interventions are most likely to keep kids out of trouble with the police.

Effective nonprofit managers get information from a variety of sources: formal studies, observation of trends in behavior, feedback from partners and clients. They also draw on deep reserves of knowledge of the local social context, of cultural norms and values, and on the ability to empathize, to look at the world through the eyes of others.

These sources of knowledge are particularly important in shaping untested but potentially innovative, breakthrough approaches to social change. Effective leaders first and foremost seek to explain how a given problem is responding to a given set of interventions.  Data help describe what is happening, but the interpretative powers of managers are essential to meaningful explanation.

One of my favorite examples (see working paper here ) of the kinds of insights that arise from observation, judgment and experience is the particular knowledge that Muhammad Yunus gained from walking through poor communities around Chittagong University in Bangladesh on his daily walk to work.  His knowledge of rural Bangladeshi society, combined with his advanced training and powers of intuition, spawned his ideas on social lending, or what became known as micro-finance.

  SUBSCRIBE | Click Here to subscribe to THE NONPROFIT QUARTERLY for just $49

The invention of micro-finance demonstrates that breakthrough innovations, and even simple adjustments to well-established programs, are spawned by a variety of sources and intellectual attributes:  data, data intelligently interpreted, knowledge of the local and comparative contexts, and good judgment.  All four of these factors are essential to shaping development breakthroughs.  Donors should give greater weight to the latter three over the first in considering funding proposals.

A recently published book on the use of applied mathematics to help understand messy, hard-to-measure problems speaks to the importance of experience and judgment in making sense of limited data.  The book is “Street-Fighting Mathematics: the Art of Educated Guessing and Opportunistic Problem Solving,” by Dr. Sanjoy Mahajan.  Dr. Mahajan is associate director of MIT’s Teaching and Learning Laboratory and the book grew out of a course by the same name that Dr. Mahajan taught for several years at MIT.

The basic premise of his approach, set out in the books first sentence, is that “Too much rigor teaches rigor mortis: the fear of making an unjustified leap even when it lands on the correct result.”  Many real-world problems are not easily described with the kind of precision that professional mathematicians insist upon. This is due to the limitations of data, the costs of collecting and analyzing data, and the inherent difficulties of giving mathematical expression to the complexity of human behavior. In the face of these obstacles, mathematicians tend to do one of two things: insist on finding the true proof, even in the face of huge methodological constraints (rigor mortis) or give up.

Mahajan counsels a third-way: using mathematical reasoning to find a good-enough, approximate and usually valid and useful answer; or as Dr. Mahajan so adeptly puts it, “When the going gets tough, the tough lower their standards.” His book describes six tools for better understanding complex problems with limited data, including picture proofs, lumping, and reasoning by analogy.

There is wisdom in Dr. Mahajan’s core argument that is relevant to current debates about the place of impact assessment in program management.  Many problems, especially problems of social analysis, present huge problems of description and accurate measurement.  We can learn much of what we need to know by tracking a few data points, but knowledge of the underlying social forces and personal motivations that frame the decisions people make is essential to specifying what should be measured and interpreting findings wisely.

My concerns about the emphasis some donors give to evaluation and impact assessment lie not in their lack of value, but in a skewing of perspective.  I want to sum up with a few thoughts on getting the perspective in better balance.

* Knowledge of the local context and the insights spawned by that knowledge are hard won and accumulated over many years. External donors and many of their staff too often don’t possess such knowledge.  For large Western donors, reliance on data and impact measures can be a crutch, a substitute for the knowledge of local context they don’t have.

* Lack of knowledge of context contributes to an overreliance on one-size-fits-all interventions based on experience from elsewhere, resulting in poorly-adapted local project design.   An obvious remedy is to place greater trust in the leadership and judgment of people who live and work close to the problems; local educators, entrepreneurs, civil society leaders.

* Evaluation is first and foremost a learning tool, of greatest value as an aid to the judgment of program leaders and managers. The work of donors also stands to benefit from the knowledge that grantees gain in assessing changes within the communities they work and progress in pursuing particular goals.

* Of greatest relevance to predicting the merits and eventual success of a proposed grantee initiative are the wisdom, experience, judgment and reputation of the grantee organization and its leadership and staff.   These are the important qualities that should be considered when contemplating a grant.  (William Duggan’s book, “Strategic Intuition,” examines the qualities of leadership and management that spawn systemic impacts.)

* Donors who insist on short-term measurable impact should stay away from funding work that seeks breakthroughs on complex, long-intractable problems.

Steven Lawry is the Senior Research Fellow at the Hauser Center for Nonprofit Organizations at Harvard University.