Measure for Measure: The Nonprofit Impact Conversation Continues



July 31, 2013;


“As an accountant who makes a living helping nonprofit organizations, I believe in the power of timely, accurate financial and non-financial information, once analyzed and interpreted, to help power an organization toward superior results,” says Eric Fraint, president and founder of Your Part-Time Controller, in a recent blog post.

But as an accountant, Mr. Fraint also believes there is no single metric to measure how effectively a nonprofit delivers on its mission. Searching for such a metric, he argues, is as likely to be as fruitless as searching for the Holy Grail. He also notes that, even when data are available to demonstrate outcomes, reasonable people very well might disagree on how to interpret such data. He illustrates this point by citing a March 2013 TED Talk by Princeton bioethicist Peter Singer suggesting that a charity that could cure 400 to 2,000 people in Africa of the infectious eye disease trachoma with $40,000 could be considered hundreds or thousands of times more effective than one that spends the same amount to train a guide dog for one sight-impaired individual. (Apples and oranges, to be sure.)

So the conversation continues about how the nonprofit sector can best tell its stories and demonstrate its impact. Back in June, it was big news—in NPQ and elsewhere—when the “overhead myth” was debunked and the conversation at long last began to shift away from measuring impact by calculating the ratio of dollars spent on programs versus those spent on management and fundraising. The sector and its funders long ago acknowledged that inputs, outputs, and outcomes are very different types of data, and while each has a story to tell, only the latter really begins to tease out the true impact of an organization’s work. And even that can be difficult to do, as explained in another recent NPQ Newswire, which notes that even when outcomes can be measured accurately, they still won’t show up in the financial statements, and they cannot always be reduced to dollars and cents.

The bright, elusive butterfly of impact is even trickier to pin down in some nonprofit categories than in others. Within the arts sector, for example, a book published last year, Counting New Beans: Intrinsic Impact and the Value of Art, which grew out of research done with a number of theater companies across the U.S., describes how over time the theater community (and presumably other performing arts organizations) has learned to talk about impact in terms of the numbers of tickets sold and the ripple effects patrons have on a local economy—by dining out, paying for parking, or hiring a babysitter. The problem, of course, is that none of these measures gets at the true impact of a live theater experience. As the Counting New Beans website notes, “We make art because we believe it makes better human beings. We make art because we believe it makes being human better. So why do we spend so much energy quantifying the economics of what we do, and so little time quantifying the impact?”

This research has sparked new conversations in the arts community about how to get at the intrinsic value of an arts experience, conversations about shared experiences and community building and storytelling and diversity. The fact remains there are no simple answers—in the arts or across the range of nonprofit organizations—about how to the measure the intrinsic impact of an organization’s work. But we must keep asking the questions.—Eileen Cunniffe

  • Mat Despard

    If the question keeps getting phrased as “how can nonprofits measure their impact?” then we will continue to throw up our hands. There are two fundamental problems with how the issue of impact is typically framed: 1) it assumes that the only way to assess impact is by using quantitative methods; and 2) that there must be a standard set of “metrics” upon which we can judge nonprofit impact. I recommend the following: 1) ask questions about impact relative to specific fields of practice (e.g. affordable housing, mental health) – not to the sector as a whole; 2) regard the evidence that most nonprofits can produce regarding their impact as being merely suggestive (not conclusive as this requires a body of evidence generated by multiple randomized experiments) that they are improving lives in the way that they intended; 3) encourage nonprofits to collect qualitative data as well (e.g. focus groups, semi-structured interviews) to understand impact – particularly given the complexity of human behavior in social environments; and 4) reward nonprofits not on the basis of demonstrated impact (i.e. because actual conclusive evidence based on rigorous social scientific standards is beyond the scope of what most nonprofits can generate), but on their use of evaluation and commitment to learning concerning a) the social problems they are addressing; b) their use of best available evidence to plan programs; and c) the degree to which they seem to be making a difference based on suggestive evidence.

  • Farron Levy

    Tracking changes in appreciation, understanding, or even “resonance and wonder” among consumers of art can be a matter of surveys, interviews, or focus groups – a little representative sampling can go a long way towards estimating the overall impact across a full population. Some references of organizations that have done this:

  • Karen

    I think Mat is right – standardized and quantitative measures are problematic. For most community benefit organizations, impact is tied to vision – what are we aiming to accomplish? If the ultimate goal was to travel from New York to Vancouver, many indicators would tell us whether we were making success towards achieving our goal…how many miles have we come, how many are still ahead, etc. But if we only pay attention to the traditional “measurable data” we might miss important things like direction (are we on the right road?) or environment (gee, it’s hard to drive with the sun in your eyes). When we think about all the things that are necessary in order to get to our destination (conditions for success) it helps us realize that MANY factors combined will get us there, or not. And, importantly, there are significant risks inherent in not being mindful about ALL those factors. The fact that we have become very good at measuring specific things has largely been driven by a demand for that information. The conversations now taking place about monitoring progress towards vision, about impact on community, about progressive stewardship, these will undoubtedly help to shift our thinking along with that of donors, funding partners, and the public. We all want fuel efficiency, but that alone will not be close to enough. The rich diversity of change we are seeking (all that impact, in all those communities) isn’t static or standardized nor do I think there is one optimal “speed” for making sure we can actually SEE the changes we’re making along the way.

  • Mavis Finnamore

    The main problem as I see it, in determining the impact and value of Art in Human society, is the fact that it’s most valuable contributions are internal , not visible with the naked eye, and they can only be inferred by the actions and beliefs of those exposed to these artistic traditions. When I take my child to the art gallery, I want him to see very graphically, how other people think and see the world around them. It may be very different from the way he thinks, but I hope some of the artists’ ideas will be added in their original form or modified by my child and added to his reservoir of ideas. In this way, his choice of ideas is enriched, and like the brain itself, various multiple combinations, folding back on his original thoughts, can produce new information and insights. On another level, I have encouraged both my children to read widely and often, for both enjoyment and to gain knowledge of the world and its people. I especially encourage life stories, as they provide examples again of different ways of thinking and handling life’s problems. And they often promote values of curiosity, empathy, and concern for others, which I want to reinforce in them.They can also cause one to examine our own behaviour and goals, as well as the larger goals of society as a whole.
    So when I think of the Arts, I see them as necessary components in gathering and transforming information from all our senses, to produce new and ever enlarging creative entities,in both science and arts,within both individuals and indeed all human society. Western societies, with their history of freedom of thought, speech and actions,has traditionally promoted artistic freedom and supported many artists. I believe this fertile mix of ideas and support is a major reason for the continuing economic success of the major democracies today. And I believe the Arts will continue to be a major influence on the success of human societies to come.