• Paul Brest

    I agree with all of Brian Mittendorf’s wise cautions about impact measures, but accounting measures are simply no substitute. In the for-profit sector, accounting standards measure the very impacts that investors care about: the financial bottom line. Most accounting measures in the nonprofit sector have little relationship to an organization’s performance or impact. Indeed, much-touted measures, such as administrative costs, are sometimes inversely related to performance. Relying on these measures is not looking for the lost keys under the lamppost, but feeling in the dark for a venomous snake.
    The starting point for getting it right is to ask organizations to be publicly clear about their goals and how they propose to measure their progress. As they begin to put out their actual measures, the sector must rely on independent groups and critics to address the very real issues that Brian raises.

  • Justin Pollock

    Both are correct and neither is right!
    As an organization, a nonprofit must be evaluated on its effectiveness (the impact it has) AND its efficiency (how it uses its limited resources). Any organizational evaluation must look at both variables or you only get half the story. Our challenge is not abandoning one measure for the other, it is getting people to understand what each measure actually provides feedback about and then bridging the gap between them.

  • Julie Miner

    I would like to see staff to volunteer ratio added to the accounting-based measures of overhead used to evaluate charitable activities. I’ve worked for social service non-porifts and a science museum and sadly the museum comes out looking better. This is because Girl Scouts and a literacy program for children in Title 1 schools utilize thousands of volunteers to deliver the program and the museum pays emplyees to deliver program. They are all great organizations, but the Girl Scouts and SMART are much more dependent on donations than the museum that collects admission. Girl Scouts have an average of 151 volunteers for every staff member, at SMART there are 147 volunteers per staff member, and at the science musuem there are 2 volunteers per staff. They are all great organizations, but SMART and Girl Scouts are much more dependent on donations than the museum, that collects admission fees. When you utilize volunteers to that extent, half of the employees are categorized as program (recruit, train, supervise, and retain volunteers) and the other half is administration (finance, HR, IT) and fundraising.

  • Lee Weingrad

    A well written article. But I have a few issues with it. We are a small independent charity that has an asymmetric strategy for dealing with a major disease among China’s ultra-poor: infant and maternal mortality. We have a small administration (2, plus part time accounting). We work with email via online portals to raise money and with volunteer help, have annual fundraisers. In addition we are always looking for grant money. Our overhead is in some sense of the word beyond the 10% comfort zone of NGOs. Our 59 project health workers are embedded in their farming and nomadic communities. They are incentivized for every procedure and referral they do. We’ve eliminated maternal mortality in our catchment.

    By a performance-based evaluation of our projects we come up much more efficiently than much larger NGOs who don’t have our flexibility and our lack of adherence to more orthodox NGO administration, public health models and business models. They are not bad people, these large international nonprofit corporations, but with large marketing budgets, they have resources to attract much more money, than we do. But I’ve noticed that our ability to get community and government buy-in for our work is much greater, where support for the existing public health structure via hardware donations or mass standardized training and testing is not an issue.

    So why should we, with greater success, and more efficiency be measured by the same financial standards as these larger less efficient nonprofit companies? My own gut feeling is that habit alone rewards what amounts to conventional warfare in public health, because the evaluators come from that corporate culture. We, on the other hand are more outside the box, using an NGO analogue of asymmetric warfare to fight the enemy maternal and child mortality and morbidity.

  • Mikala

    This is a long, deeply personal, and slightly foul-mouthed essay that lays out some potential pitfalls related to “impact-based” evaluation:

    http://kiddewoodward.blogspot.com/2011/11/why-hell-not.html

    I thought some of your readers might find it interesting.

  • BBOW

    Although I don’t think that the overhead ratio should be the sole measure of efficiency and effectiveness, I don’t know that it’s use, per se is the real issue. It is the unrealistic expectation of what the ratio should be and the one-size-fits all approach. Nonprofits have been told over and over that the lower it is the better. In fact, the opposite is true (of course there are always exceptions). Studies show that those who spend more are more effective. And, as noted, there are multiple legitimate variables that affect overhead rates. I recently came across the quote below and thought, finally, someone is starting to get it::

    SeriousGivers.Org (SGO) which provides information about nonprofits, cautions potential donors to investigate further if an organization, “reports spending more than 80% on programs, that means it is spending less than 20% on administration and fundraising. While an organization might be proud of minimizing administrative and fundraising costs, we believe that well-run organizations must spend meaningful resources on administration and fundraising.” http://seriousgivers.org/find-a-charity/charity-data-faqs/#spendingprograms.

    Keep in mind that the more emphasis that gets placed on measuring outcomes, the more overhead costs will increase as well.

  • Steven Nardizzi, CEO, Wounded Warrior Project

    Brian,

    This article far from paints you as an accounting apologist and is one of the most thoughtful defenses of accounting based measures of charity effectiveness I have ever read. I appreciate the sincerity and honesty of your conviction on this issue. That having been said, your conclusions are incorrect.

    As the CEO of a large nonprofit, I value the role accountants and auditors play in testing internal controls and providing boards, donors, and the general public assurance that charities are free of fraud, waste and abuse. But you go too far when you suggest accounting standards, including overhead measures, are a better test for charity effectiveness than impact measurements.

    You state that there are “three key things about impact measures that could make the flaws in accounting measures pale in comparison.” Reliability, comparability, and controllability. Yet these are the very things found lacking in the application of accounting standards that contributed to the collapse of the US economy as reported, for example, in the Harvard Business Journal and Wall Street Journal:.

    http://blogs.law.harvard.edu/corpgov/2012/03/02/the-role-of-accounting-in-the-financial-crisis/;

    http://online.wsj.com/article/SB10001424052748703814804576036094165907626.html

    This excerpt from the Wall Street Journal is particularly illustrative, “Every big auditing firm had clients that blew up or required huge government bailouts or engaged in questionable practices leading up to the crisis … Companies rely on auditors to bless their valuation methods and there’s so much subjectivity.”

    The lack of reliability, comparability, and controllability of accounting standards for nonprofits is “equally” (change to certainly) well documented. From accounting and auditing discrepancies in valuing donated goods that may mislead donors (http://www.forbes.com/sites/williampbarrett/2011/11/30/donated-pills-makes-some-charities-look-too- good-on-paper/2/) to the subjectivity of accounting standards on joint costs you’ve previously commented upon in your own blog (http://countingoncharity.blogspot.com/2013/02/cost-allocation-and-american-heart.html).

    In the end, your premise that accounting standards are the appropriate measure of charity effectiveness is flawed for three reasons. First, you posit that “impact measures are better viewed as a complement to, not substitute for, accounting measures.” However reliance on arbitrary accounting measures, including overhead ratios, may actually reduce charitable impact by forcing charities to underinvest in the staff, infrastructure, and, yes, fundraising, necessary to improve and scale their impact. At least one recent study shows just that, http://www.guardian.co.uk/voluntary-sector-network/2013/may/02/good-charities-admin-costs-research.

    Second, charitable work can’t be measured and evaluated by numbers. You state that “(e)ven two organizations with the same goal of different sizes become hard to compare solely on impact grounds. If one organization is larger than another, we would naturally expect it to have a greater impact.” Impact is not truly measured by the numbers served, but by fulfillment of mission. The fundamental difference in our perspectives is that the real question is not about the numbers served or dollars spent, but about the outcomes achieved. Did you alleviate hunger, stop animal abuse, protect the environment, or cure cancer? Answer those first, then ask if you did so for one or one thousand constituents, or how much was spent on overhead.

    The one point we firmly agree on is that evaluating program effectiveness is hard. I know, because at the charity I work for, Wounded Warrior Project, we have some of the most rigorous internal and independent impact measures in the industry, You can find them all on the links under the mission section of our website, here: http://www.woundedwarriorproject.org/mission.aspx.

    We all do charities, our donors, and the beneficiaries of our good works a disservice when we focus on accounting ratios and give ourselves a pass at accounting for what truly matters, the effectiveness of the important work we do.

    Steven Nardizzi
    CEO
    Wounded Warrior Project

  • Keenan Wellar

    I think Justin Pollock has it right. A charitable organization can have a remarkable balance sheet but might be making an entirely useless or even detrimental contribution to the community. Or they might be very passionately delivering some outstanding outcomes, but find themselves on the verge of bankruptcy due to a lack of financial oversight.

    You cannot have one (impact measurement and accounting measurement) without the other.

    To me the more common and greater tragedy is organizations that have lovely balance sheets that are based on single-funder 30 year old models of programmatic service delivery. Those making no effort to change (and clearly a socially relevant organization needs to change as the world around it changes) simply have no external motivation to do so, because no one is paying attention to their outcomes. The client is the cheque, and the financial audit shows they are doing just fine.

    From time to time stories come to light about genuine financial management issues, but I believe those problems are the far less frequent of the two evils – the biggest problem of all being resources directed to the wrong outcomes. The financial oversight of charitable organizations in Canada has improved significantly in recent years, but we are still in a fumbling preliminary stage of outcomes measurement.

  • Thomas Matteo

    Brian, very interesting paper my friend, I learned something. My name is a Thomas a Stormy a Matteo, who served with your Dad in a Vietnam as. Platoon Commander and Platoon Sgt.

    My nickname is Stormy, as you Dad calls me! I realize that this is not the proper forum to discuss this, but please know I am here for Karen and your Dad. He is very dear to me and Will whatever it takes to support him.

    Stay well young man and know your very was an excellent Marine!

    Semper Fi
    Thomas Matteo
    706-889-1812
    561-2739499