Want Charities to be Evaluated Based on Impact? Be Careful What You Wish For

Print Share on LinkedIn More

 

Apple Orange

In the nonprofit sector, disdain for accounting-based measures of overhead as a means of evaluating charitable activities now appears to be universal. At the same time, though, donors continue to rely heavily on those measures, which only amplifies the frustration for those who dislike accounting-based evaluation. Of course, there’s much to dislike about overhead measures. Accounting functional classification of expenses (classification between programs, fundraising, and administrative costs) is only an estimate of where resources are going, which means it is not perfect, and recent cases of cost allocations gone awry are among the many circumstances where the intent of accounting standards and their implementation diverge notably. Even if it were a perfect reflection of where resources are being used, the functional expense classification afforded by accounting standards stops short of providing any evidence to how effectively they are being used.

For these reasons, the passion with which many hate accounting measures is often matched by their ardent belief in the promise of impact-based performance measurement. At the risk of sounding like an accounting apologist, I believe the nonprofit sector should be careful what it wishes for when it comes to pushing for evaluating charities based on impact measures rather than accounting measures. I fear that the same voices expressing hatred for accounting may one day redirect their ire toward impact measures, pining for the “good old days” when accounting reigned. As I see it, there are three key things about impact measures that could make the flaws in accounting measures pale in comparison.

Impact measures are less reliable

Measuring impact may give donors information about performance that’s more relevant, but this information will inevitably be less reliable. Such a tradeoff is a key consideration in all types of performance measurement. However, it seems those in the nonprofit sector have focused on getting the most relevant measure while ignoring that tradeoff with reliability.

For all their faults, accounting measures are constructed to be verifiable. Being transactions-based and reliant on a common measurement basis (dollars), the accounting paper trail permits routine auditing by a third-party. Verifiability and independent auditing make accounting-based measures relatively reliable. True, the accounting can only tell us where resources went, not what was gained for society from that allocation of resources. If one cares about reliability, though, it is hard to imagine a systematic way of verifying any claims of societal gains. Take the case of a literacy organization. Whereas accounting measures stop at telling us how much in resources was devoted to literacy efforts, an impact measure could tell us how many individuals were taught to read by an organization. However, imagine trying to verify such claims in an objective and systematic fashion. In other words, the relevance benefit of measuring literacy outcomes inevitably entails a sacrifice of reliability.

These tradeoffs have consistently played out in development of measures in the for-profit sector. For instance, consider fair-value measurements of investments as governed by Statement of Financial Accounting Standards (SFAS) 157. In an attempt to provide financial statement readers with better information about the value of a firm’s investments, SFAS 157 requires organizations to develop measures of the value of these investments even if they are not actively traded in markets. The result of the standard was the use of economic models of value, particularly for investments in private equity and real assets. Academic research has confirmed that efforts to develop these “mark-to-model” measures has both increased the extent to which investors have relied on them and, at the same time, made accounting measures of investments less precise. While some have praised the efforts to increase relevance, others have blamed the lack of reliability on perpetuating problems with valuation and even cultivating the financial crisis. It is this issue of sacrificing reliability that nonprofit leaders must grapple if they want to fully embrace impact measures.

Impact measures are less easily compared

One overlooked feature of accounting measures for nonprofits (such as the program expense ratio and fundraising efficiency ratio) is that they offer donors comparability across many types and sizes of organizations. Though such comparability is not perfect, it does offer a good starting point for potential donors. It is hard to imagine such comparisons in a world of impact measures. For one, it is often hard to come to agreement on the relevant metrics for different organizations. Even when the relevant measures for two different organizations are apparent, how does a donor compare them? Trying to decide between donating to a literacy group or a homeless shelter based on the number of people taught to read by one and the number of people housed by another is a tortuous exercise in mental accounting. Even two organizations with the same goal of different sizes become hard to compare solely on impact grounds. If one organization is larger than another, we would naturally expect it to have a greater impact. That alone does not mean it is more deserving of donor funds.

Again, these issues have already played out in the for-profit sector. Despite the many complaints about accounting and the concurrent development of alternative qualitative metrics and key performance indicators for internal users, there remains a heavy reliance on accounting-based metrics (e.g., return-on-assets and return-on-equity) by outside investors. True, outside investors may find key performance indicators informative, but they highly value measures that can be compared across a broad swath of firms and industries. As it turns out, accounting metrics usually trump others in this regard. So while the voices complaining about the flaws of accounting are certainly loud, everyone also eagerly awaits quarterly earnings announcements. Perhaps the inherent lack of comparability portends a similar future for nonprofit impact measures.

Impact measures are less controllable

The primary enthusiasm for impact measures is that they capture the ultimate outcomes of charitable activities, that about which we care most. However, the “controllability principle,” a viewpoint now widely accepted both by economic theory and by performance measurement specialists, argues that we should not evaluate individuals or groups based on outcomes per se, but instead based on the aspects of outcomes over which they have control. The theory advocates that using controllable measures is particularly important when those being measured are averse to taking risks, a phenomenon omnipresent in the nonprofit sector.

With this in mind, consider again the question of accounting vs. impact measures. The accounting-based program expense ratio does not tell us perfectly about a nonprofit’s effectiveness, but it is certainly controllable. It is up to the discretion of a nonprofit’s management about where to devote its resources, and that is primarily what is measured (even if imperfectly) by the functional classification of expenses. Program outcomes are another matter altogether. One literacy organization may have greater success in teaching people to read than another because it had students who were on the cusp of reading and able to focus exclusively on learning, whereas another organization’s students were far from ready and facing a variety of other problems. Is it fair to say the organization that had more “teachable” students was better?

Even worse is the concern that organizations evaluated based on these impact measures will seek out circumstances that are more apt to lead to successes. As an example, a fixation on survival rate measures among cancer hospitals is the backdrop for the accusation that the Cancer Treatment Centers of America screens patients and turns away those who are less likely to be treatable. Since nonprofits serve as a societal safety net, it becomes all the more important that they be rewarded rather than punished for seeking out the difficult cases.

Again, a for-profit analog to the problem is instructive. Firms that focus solely on outcome-based performance measurement often see a concomitant decrease in research and development (R&D) spending. Though R&D has a substantial upside, there is also the risk of failure. While investors may be willing to take such risks, many being evaluated by outcomes are not, fearing they have little control over R&D successes. To overcome this disincentive to invest in R&D, firms have either tried to make decision-makers less risk averse (say, by limiting their downside via stock options or stock appreciation rights) or to instead reward them for that which they control. The latter choice, which seems more germane to nonprofit performance measurement, entails rewarding R&D spending rather than R&D success. Much the same thinking applies to nonprofits; while rewarding them for spending on programs is only an input measure, it is much more controllable than the output measure of those programs’ success.

Taken together, these concerns about reliability, comparability, and controllability of impact measures suggest the staying power of accounting measures amidst a barrage of criticism is no accident. I realize this defense of accounting measures exhibits the flavor of looking for lost keys near the streetlight because it’s easier to see there. But it does seem like a good place to start. After all, I am not suggesting measuring impact is not worth the trouble. What I am saying is that the notable weaknesses of impact measures are the very things at which accounting excels. For this reason, impact measures are better viewed as a complement to, not substitute for, accounting measures.

  • Paul Brest

    I agree with all of Brian Mittendorf’s wise cautions about impact measures, but accounting measures are simply no substitute. In the for-profit sector, accounting standards measure the very impacts that investors care about: the financial bottom line. Most accounting measures in the nonprofit sector have little relationship to an organization’s performance or impact. Indeed, much-touted measures, such as administrative costs, are sometimes inversely related to performance. Relying on these measures is not looking for the lost keys under the lamppost, but feeling in the dark for a venomous snake.
    The starting point for getting it right is to ask organizations to be publicly clear about their goals and how they propose to measure their progress. As they begin to put out their actual measures, the sector must rely on independent groups and critics to address the very real issues that Brian raises.

  • Justin Pollock

    Both are correct and neither is right!
    As an organization, a nonprofit must be evaluated on its effectiveness (the impact it has) AND its efficiency (how it uses its limited resources). Any organizational evaluation must look at both variables or you only get half the story. Our challenge is not abandoning one measure for the other, it is getting people to understand what each measure actually provides feedback about and then bridging the gap between them.

  • Julie Miner

    I would like to see staff to volunteer ratio added to the accounting-based measures of overhead used to evaluate charitable activities. I’ve worked for social service non-porifts and a science museum and sadly the museum comes out looking better. This is because Girl Scouts and a literacy program for children in Title 1 schools utilize thousands of volunteers to deliver the program and the museum pays emplyees to deliver program. They are all great organizations, but the Girl Scouts and SMART are much more dependent on donations than the museum that collects admission. Girl Scouts have an average of 151 volunteers for every staff member, at SMART there are 147 volunteers per staff member, and at the science musuem there are 2 volunteers per staff. They are all great organizations, but SMART and Girl Scouts are much more dependent on donations than the museum, that collects admission fees. When you utilize volunteers to that extent, half of the employees are categorized as program (recruit, train, supervise, and retain volunteers) and the other half is administration (finance, HR, IT) and fundraising.

  • Lee Weingrad

    A well written article. But I have a few issues with it. We are a small independent charity that has an asymmetric strategy for dealing with a major disease among China’s ultra-poor: infant and maternal mortality. We have a small administration (2, plus part time accounting). We work with email via online portals to raise money and with volunteer help, have annual fundraisers. In addition we are always looking for grant money. Our overhead is in some sense of the word beyond the 10% comfort zone of NGOs. Our 59 project health workers are embedded in their farming and nomadic communities. They are incentivized for every procedure and referral they do. We’ve eliminated maternal mortality in our catchment.

    By a performance-based evaluation of our projects we come up much more efficiently than much larger NGOs who don’t have our flexibility and our lack of adherence to more orthodox NGO administration, public health models and business models. They are not bad people, these large international nonprofit corporations, but with large marketing budgets, they have resources to attract much more money, than we do. But I’ve noticed that our ability to get community and government buy-in for our work is much greater, where support for the existing public health structure via hardware donations or mass standardized training and testing is not an issue.

    So why should we, with greater success, and more efficiency be measured by the same financial standards as these larger less efficient nonprofit companies? My own gut feeling is that habit alone rewards what amounts to conventional warfare in public health, because the evaluators come from that corporate culture. We, on the other hand are more outside the box, using an NGO analogue of asymmetric warfare to fight the enemy maternal and child mortality and morbidity.

  • Mikala

    This is a long, deeply personal, and slightly foul-mouthed essay that lays out some potential pitfalls related to “impact-based” evaluation:

    http://kiddewoodward.blogspot.com/2011/11/why-hell-not.html

    I thought some of your readers might find it interesting.

  • BBOW

    Although I don’t think that the overhead ratio should be the sole measure of efficiency and effectiveness, I don’t know that it’s use, per se is the real issue. It is the unrealistic expectation of what the ratio should be and the one-size-fits all approach. Nonprofits have been told over and over that the lower it is the better. In fact, the opposite is true (of course there are always exceptions). Studies show that those who spend more are more effective. And, as noted, there are multiple legitimate variables that affect overhead rates. I recently came across the quote below and thought, finally, someone is starting to get it::

    SeriousGivers.Org (SGO) which provides information about nonprofits, cautions potential donors to investigate further if an organization, “reports spending more than 80% on programs, that means it is spending less than 20% on administration and fundraising. While an organization might be proud of minimizing administrative and fundraising costs, we believe that well-run organizations must spend meaningful resources on administration and fundraising.” http://seriousgivers.org/find-a-charity/charity-data-faqs/#spendingprograms.

    Keep in mind that the more emphasis that gets placed on measuring outcomes, the more overhead costs will increase as well.

  • Steven Nardizzi, CEO, Wounded Warrior Project

    Brian,

    This article far from paints you as an accounting apologist and is one of the most thoughtful defenses of accounting based measures of charity effectiveness I have ever read. I appreciate the sincerity and honesty of your conviction on this issue. That having been said, your conclusions are incorrect.

    As the CEO of a large nonprofit, I value the role accountants and auditors play in testing internal controls and providing boards, donors, and the general public assurance that charities are free of fraud, waste and abuse. But you go too far when you suggest accounting standards, including overhead measures, are a better test for charity effectiveness than impact measurements.

    You state that there are “three key things about impact measures that could make the flaws in accounting measures pale in comparison.” Reliability, comparability, and controllability. Yet these are the very things found lacking in the application of accounting standards that contributed to the collapse of the US economy as reported, for example, in the Harvard Business Journal and Wall Street Journal:.

    http://blogs.law.harvard.edu/corpgov/2012/03/02/the-role-of-accounting-in-the-financial-crisis/;

    http://online.wsj.com/article/SB10001424052748703814804576036094165907626.html

    This excerpt from the Wall Street Journal is particularly illustrative, “Every big auditing firm had clients that blew up or required huge government bailouts or engaged in questionable practices leading up to the crisis … Companies rely on auditors to bless their valuation methods and there’s so much subjectivity.”

    The lack of reliability, comparability, and controllability of accounting standards for nonprofits is “equally” (change to certainly) well documented. From accounting and auditing discrepancies in valuing donated goods that may mislead donors (http://www.forbes.com/sites/williampbarrett/2011/11/30/donated-pills-makes-some-charities-look-too- good-on-paper/2/) to the subjectivity of accounting standards on joint costs you’ve previously commented upon in your own blog (http://countingoncharity.blogspot.com/2013/02/cost-allocation-and-american-heart.html).

    In the end, your premise that accounting standards are the appropriate measure of charity effectiveness is flawed for three reasons. First, you posit that “impact measures are better viewed as a complement to, not substitute for, accounting measures.” However reliance on arbitrary accounting measures, including overhead ratios, may actually reduce charitable impact by forcing charities to underinvest in the staff, infrastructure, and, yes, fundraising, necessary to improve and scale their impact. At least one recent study shows just that, http://www.guardian.co.uk/voluntary-sector-network/2013/may/02/good-charities-admin-costs-research.

    Second, charitable work can’t be measured and evaluated by numbers. You state that “(e)ven two organizations with the same goal of different sizes become hard to compare solely on impact grounds. If one organization is larger than another, we would naturally expect it to have a greater impact.” Impact is not truly measured by the numbers served, but by fulfillment of mission. The fundamental difference in our perspectives is that the real question is not about the numbers served or dollars spent, but about the outcomes achieved. Did you alleviate hunger, stop animal abuse, protect the environment, or cure cancer? Answer those first, then ask if you did so for one or one thousand constituents, or how much was spent on overhead.

    The one point we firmly agree on is that evaluating program effectiveness is hard. I know, because at the charity I work for, Wounded Warrior Project, we have some of the most rigorous internal and independent impact measures in the industry, You can find them all on the links under the mission section of our website, here: http://www.woundedwarriorproject.org/mission.aspx.

    We all do charities, our donors, and the beneficiaries of our good works a disservice when we focus on accounting ratios and give ourselves a pass at accounting for what truly matters, the effectiveness of the important work we do.

    Steven Nardizzi
    CEO
    Wounded Warrior Project

  • Keenan Wellar

    I think Justin Pollock has it right. A charitable organization can have a remarkable balance sheet but might be making an entirely useless or even detrimental contribution to the community. Or they might be very passionately delivering some outstanding outcomes, but find themselves on the verge of bankruptcy due to a lack of financial oversight.

    You cannot have one (impact measurement and accounting measurement) without the other.

    To me the more common and greater tragedy is organizations that have lovely balance sheets that are based on single-funder 30 year old models of programmatic service delivery. Those making no effort to change (and clearly a socially relevant organization needs to change as the world around it changes) simply have no external motivation to do so, because no one is paying attention to their outcomes. The client is the cheque, and the financial audit shows they are doing just fine.

    From time to time stories come to light about genuine financial management issues, but I believe those problems are the far less frequent of the two evils – the biggest problem of all being resources directed to the wrong outcomes. The financial oversight of charitable organizations in Canada has improved significantly in recent years, but we are still in a fumbling preliminary stage of outcomes measurement.

  • Thomas Matteo

    Brian, very interesting paper my friend, I learned something. My name is a Thomas a Stormy a Matteo, who served with your Dad in a Vietnam as. Platoon Commander and Platoon Sgt.

    My nickname is Stormy, as you Dad calls me! I realize that this is not the proper forum to discuss this, but please know I am here for Karen and your Dad. He is very dear to me and Will whatever it takes to support him.

    Stay well young man and know your very was an excellent Marine!

    Semper Fi
    Thomas Matteo
    706-889-1812
    561-2739499