When “Impact Data” Goes Wrong: Good People Giving Bad Numbers

Print Share on LinkedIn More


Disappearing data

November 22, 2014; Daily News Journal (Murfreesboro, TN)

According to an article by the Daily News Journal, the United Way of Rutherford and Cannon Counties (UWRCC) in Tennessee is “more efficient than other nonprofits.” The article outlines several areas where the UWRCC claims impact and effectiveness. Unfortunately, a number of these claims appear to be flawed and unsubstantiated.

Outlined below are those claims, and the problems that each of them hold:

“The UWRCC’s fundraising efficiency rating was nearly nine times higher than other nonprofit entities of similar scope.”—There are a couple challenges with this claim. While the UWRCC claims it is a human service agency, in truth it is akin more to a community foundation. In its latest public 990 (2013), UWRCC obtained approximately $2.8M in revenue and granted out just over $1.8M, which would mostly make UWRCC a grantmaking institution. Comparing UWRCC to a traditional human service agency is like comparing a coffee shop to an auto repair service, although the UWRCC is more like an auto repair service with a coffeemaker because it does provide some direct programming. Therefore, its fundraising apparatus is different and should not be compared to a traditional human service effort.

For argument’s sake, let’s say we do regard the UWRCC as a human service agency similar to others in their community. To that end, our analysis shows that their fundraising efficiency ratio (FER) is not nine times better than others, even in their own backyard of Murfreesboro. Fundraising efficiency ratios are a common metric to showcase how much funds are being used to fundraise versus how much is actually going to the programs of the organization. Charity Navigator, according to its website, calculates a charity’s fundraising efficiency by “determining how much it spends to generate $1 in charitable contributions. In other words, we divide a charity’s fundraising expenses by the total contributions it receives.” If we use Charity Navigator’s metric, then on the last public 990, UWRCC’s ratio shows that it spends four cents to raise a dollar ($2.7M in contributions to $101K in fundraising expenses), which is a very good ratio. But in looking at other human service organizations with budgets over a million dollars, we found that the Rutherford Habitat for Humanity used nine cents to raise a dollar and Journeys in Community Living used 16 cents to raise a dollar. It appears that UWRCC may have inflated its prowess in this area.

“The UWRCC’s administrative expenses were 13.8 percent, resulting in 86.2 percent that was invested in the community through programming.”—If the UWRCC regarded itself as a grantmaking institution, then there would not be a major challenge here—but if they consider themselves a service provider, then this statistic is flawed. In order for this statistic to be truly valuable, the UWRCC would need to calculate the costs of the direct services it provides—the Volunteer Income Tax Assistance (VITA) Program, 2-1-1, FamilyWize Prescription Discount Program, and other direct services—and then recalculate these costs. The numbers might be quite different.

Would it be such a problem if the numbers were different, however? As we hear in the current discussion on overhead being led by GuideStar and Charity Navigator, focusing on overhead rates is a misguided venture. In my firm’s own analysis of nonprofits on the East Coast, we have found that nonprofits with overhead rates below 15 percent had weaker organizational capacity than did organizations with overhead rates of 19 to 26 percent. UWRCC should spend more time discussing its impact and not exacerbating a misguided notion of low overhead rates to its donors.

“We are able to deliver $11 of value in impact for every one dollar we spend on administrative costs (leveraging investments to create the largest impact on our community.”—Additionally, UWRCC stated that “89 percent of students enrolled in a United Way partner program achieved basic, proficient or advanced levels in reading, and 82 percent of formerly homeless individuals participating in a United Way partner program were able to secure affordable housing.” These statistics are the most problematic ones, for a few reasons. The first challenge is that the UWRCC has made a data claim before the impact could be achieved; the UWRCC does not own the eventual impact result, and therefore ought not claim the impact for themselves.

Let’s start with the first challenge. The measure of $11 dollars of value in impact for every dollar spent is a data tool most commonly referred to as Social Return on Investment, or SROI. SROI as a data tool was developed in Europe, is increasingly being used in the U.S. and, if done correctly, is a complicated metric that can provide a financial value on social good. The computing of this statistic should not be taken lightly, which UWRCC has done. According to the UK’s New Economics Foundation, the process to getting to an SROI takes six specific steps, among them “finding data to show whether outcomes have happened” and “having collected evidence on outcomes and monetized them, those aspects of change that would not have happened anyway (deadweight) or are not as a result of other factors (attribution) are isolated.”

The challenge with UWRCC is that the outcomes of the examples they use will not be seen for some time. For example, looking at the affordable housing example, those in the homelessness space will tell you that securing affordable housing for a client is just one step of the many steps needed to ensure self-sufficiency. Therefore, the evidence of social impact for those obtaining the housing might not be seen for a few years. Attributing SROI before seeing the results is flawed in its approach.

Second, and more concerning, is that if UWRCC is the grantmaker in the relationship to its partners, then it is an ethical gray area for them to claim the social impact of the work of its partners, which are local nonprofits. In a review of UWRCC’s 990, a majority of their grants fall below $60K, and any executive director will tell you that a $60K grant is a small but probably important part of the organization’s social impact. For UWRCC to claim the “11 to 1” social impact result based on its small role in the eventual social impact is over-leveraging their part of the relationship while minimizing the work of those partners doing the work on the ground.

In conclusion, United Ways historically are responsible for raising dollars locally and disbursing it to those who serve the most needy. More and more United Ways are also providing direct services on the ground. United Ways are an important part of community life and in maintaining the local social safety net. To this end, creating momentum so that United Ways can raise funding for the community is important and is probably the reason behind UWRCC’s rush to showcase its impact data, running the risk of stating data inaccurately. Creating fundraising momentum, however, should not lead to misguiding donors while also minimizing the true impact of other organizations in the community.—John Brothers

  • Judy Levine

    Even before the issue of the claiming credit for outcomes years in the future, is the fact that this UW is not achieving these outcomes on its own.

    Most likely, the programs this UW seems to be claiming outcomes for are probably not entirely UW-funded. So are they including the individual charities’ expense of raising the additional dollars for those programs, and the actual costs of running the programs, in their ROI? There’s something wildly off here, where the UW is claiming all the credit but, I would assume, only putting out 20-50% of the dollars necessary to produce those outcomes.

  • Meagan Flippin

    Mr. Brothers,

    Are you aware that the data and information you cited in your article (2013) is from a fiscal year different than that of the annual report released (2014) by the United Way of Rutherford and Cannon Counties?

    Meagan Flippin
    President and CEO
    United Way of Rutherford and Cannon Counties

  • Meagan Flippin

    Mr. Brothers,

    I am reposting my comment as it was removed earlier.

    Are you aware that the basis of your article is on a fiscal year (2013) different than that which was referenced in the Daily News Journal article (2014)?

    Meagan Flippin
    President and CEO
    United Way of Rutherford and Cannon Counties

  • Rick Cohen

    Dear Ms Flippin: No one removed your earlier comment. As I am the person in charge of approving comments, I hadn’t yet gotten to reviewing and approving the comment.

    Rick Cohen

  • John Brothers

    Ms. Flippin,

    Thanks for responding. As I stated in the article, the data I used was the latest public 990s, which is 2013. Your latest audit, which is featured on your site does not include any data that could be examined. I think the only data that could be impacted by your comment below is the fundraising efficiency ratio, which does use 2013 data. But, I feel confident that my argument would be the same and would hold up as solidly as it does now. The UWRCC would have to more than double it’s efficiency ratio from 2013 to 2014 (with other local nonprofits remaining constant) in order to meet the criteria you established in the DNJ article.

    My recommendation would be that in the future that UWRCC issue a data sheet that outlines how you arrived at the data and post it online, with the link as a note in your public document. As a former nonprofit CEO, I learned the hard way on this as well. I can show you examples, if you need them. Just e-mail me at john@quidooconsulting.com

    I wish you well in your important work.


    Dr. John Brothers