We recently spoke with Carlos Martinez, executive director of La Alianza Hispana, who shared what, in his view, had been an empowering experience. He was exuberant as he told us about an opportunity he had to advocate for his organization’s constituents from a position of knowledge and power. During a contract negotiation with a Massachusetts state agency, the funder asked Carlos about his organization’s use of program outcomes. Because his organization, a community-based, multi-service agency, has fully embraced the importance of measuring its work and its impact on its constituents, Carlos was able to clearly articulate the meaning and importance of measuring benefits achieved by program participants. He was also able to describe the ways that his organization could measure such outcomes. Unexpectedly, Carlos found himself in the unusual position of educating the funder (who was confused about what program outcomes actually are) and explaining just how his organization planned to measure program outcomes in order to improve its programs and better serve its constituents.
Carlos Martinez’s experience illustrates but one of the many benefits nonprofit organizations can attain when they strategically adopt a program-outcome measurement mind-set and diligently set about designing a system by which to implement it. In this article, we’ll elaborate on the potential benefits and describe the overall framework for measuring program outcomes that the United Way is using with its affiliated agencies.
Based on our experience as trainers and consultants in the use of the United Way’s outcome measurement initiative, and on our more general experience as consultants with nonprofit organizations, we show that taking leadership for and ownership of program outcome measurement makes sense from managerial, strategic, and philosophical points of view.
That the current funding environment emphasizes greater accountability is no surprise to leaders and managers of nonprofit organizations. Often, these calls for increased accountability from the funding community are received by nonprofits as just one more imposition on their already over-burdened and understaffed organizations. However, adopting an outcome measurement philosophy can potentially be highly beneficial for your organization. It can:
* Help you stay mission-focused.
* Demonstrate (and/or reaffirm) what your board and staff may already know, albeit intuitively and anecdotally, about the good work your agency is doing.
* Help you design and implement more effective program planning, program management, and program implementation.
* Improve your board’s decision-making process by providing relevant program information that helps your agency decide what programs are working and how these forward your agency’s mission.
* Help you advocate more effectively for increased funding and policy changes that affect your programs and your constituents.
* Allow you to take a leadership role by enhancing your field’s collective knowledge about what constitutes relevant program outcomes.
* Help you be more accountable to funders, the public at large, and, most important, your own program participants and constituents, by ensuring that your agency is providing constantly renewed, relevant, and appropriate programs and activities.
Designing and implementing an outcome measurement system can be a daunting and ambitious task. However, the United Way has developed a model that we believe greatly simplifies the job. It is a model that places the responsibility of identifying and measuring meaningful and relevant outcomes in the hands of nonprofits themselves.
The United Way of America initiated a collaboration between the Urban Institute and a group of community-based nonprofits, consultants, researchers, and foundations to assess current, traditional program evaluation models. This group concluded that existing models did not adapt well to community-based nonprofit organizations that had limited resources. In response to this finding, they designed a new, step-by-step, practical approach to measuring program outcomes.
In the spring of 1997, the United Way of Massachusetts Bay initiated a comprehensive approach for focusing on outcome measurement–one that emphasizes building the capacity of its affiliated organizations to measure program outcomes. In collaboration with participating organizations, consultants, and researchers, the United Way tailored the national model to local needs.
The United Way model has four distinctive features:
* It focuses on benefits to your constituents.
* It places the responsibility for identifying appropriate outcomes into the hands of nonprofit organizations.
* It is a participatory, inclusive model, based on the active participation of constituents, staff, board, and key stakeholders.
* It is practical, accessible, and affordable to nonprofit organizations.
The Model Focuses on Outcomes Rather Than Outputs
Many funders, particularly government funders, require organizations to track “outputs” using numerical units of service resulting from program activities, such as numbers of participants served, numbers of educational materials distributed, or numbers of counseling sessions conducted. This tracking methodology offers only limited benefits to nonprofits, and particularly to their constituencies.
Conversely, the United Way model helps organizations to focus on the more meaningful aspects of their programs. For example, what benefits or changes occur for individuals or constituencies during or after their participation in program activities? Measured outcomes include changes in individual or constituent knowledge, attitudes, skills, or behaviors.
Traditional program evaluation models are often based on the premise that those closest to the work cannot accurately evaluate it; rather an outside, neutral evaluator can better identify and measure “standard” outcomes. But who knows the constituents and programs better than nonprofits themselves?
An important underlying assumption of the United Way model is that organizations can and should determine what outcomes are most meaningful for them to measure and how they want to measure them. Another underlying premise is that the constituency should be involved in both defining meaningful outcomes and in participating in the outcome measurement system. This process helps organizations align more closely with their constituencies.
Furthermore, program staff working at nonprofit human service organizations have experience in and knowledge of their target populations and so can also prove valuable in identifying and measuring program outcomes that are relevant to their particular realities. Getting ahead of the game and designing an outcome measurement system positions nonprofits in pioneering and leadership roles.
Given the diverse constituency that most nonprofits serve, placing responsibility for identifying outcomes in the hands of the organizations themselves makes more sense. The reality is that contextual factors, such as geography, types of populations served, and other sociodemographic factors condition the extent to which programs work (or do not work). To this end, we believe that nonprofits do have the capacity to determine what outcomes are important to them.
Most program evaluation models are complex, academic, expensive, and overwhelming for community-based organizations to implement. Having been trained in traditional program evaluation research methods in our doctoral programs, we believe that this new model is more accessible, more user-friendly, and more effective for community-based nonprofit organizations.
Many traditional models require hiring outside researchers to design and conduct evaluations. Most nonprofits cannot afford the time, staff, or funds that these traditional models require. And as mentioned earlier, traditional models are built on the premise that organizations are too biased to evaluate their own work and that such evaluations represent a conflict of interest, thus outcome measurement ultimately becomes another “project” for already overburdened organizations.
The new model solves both of these problems. It allows organizations to collect sufficient information to improve programs and provide more effective benefits for its constituency without an overwhelming investment of time or money. And it is built on the premise that the most effective ways to measure outcomes are well-integrated with ongoing program development and implementation work. The measurement system becomes a feedback-loop that continually informs your program decisions.
The new model is easy to learn and can be applied even by individuals who do not have a background in research methodology. According to Don Buchholtz, director of the United Way of Massachusetts Bay Outcome Measurement Project, “the strength of this model is that it is simple, step-by-step, logical, and straightforward. It’s intended for people who do not have a background in program evaluation.”
With training and practice applying the model in real situations, and moderate consulting assistance, even your organization can develop the capacity to lead and design the implementation of an outcome measurement system. It adapts well to different types of nonprofit organizations, whether they are focused on human services, environmental issues, community organizing, or political advocacy.
Use of this model can help your group become a learning organization–an organization that uses feedback on an ongoing basis to improve its own effectiveness. The model emphasizes developing and implementing the measurement system in an inclusive, collaborative manner. We usually recommend that your team be composed of a cross-section of organizational personnel. Because this team both designs and implements the system, we believe that those closest to the work should be intimately involved in both the planning and implementation of the outcome measurement system.
Sign up for our free newsletters
Subscribe to NPQ's newsletters to have our top stories delivered directly to your inbox.
By signing up, you agree to our privacy policy and terms of use, and to receive messages from NPQ and our partners.
The model itself is participatory in that it is constantly evolving–it is continually being improved on and transformed through ongoing testing and input from participating organizations, consultants, local academic experts, and other community resources.
Despite our general enthusiasm for this model, we do recognize that there are limitations in terms of the types of conclusions that can drawn from outcome measurement systems. Since this is not a research model, and does not represent program impact research, it does not use random sampling or sophisticated statistical methods. Because this model does not enable you to separate your program’s influence from other factors that might affect its participants, it does not allow you to prove that a program caused certain outcomes and, because it is a fairly new model that has not been tested over the long term, uncertainties exist as to its benefits over time.
One of the limitations of the United Way’s outcome measurement model is that it generally focuses on distinct, individual programs. For organizations that specialize in one program area, where activities are very similar, one comprehensive model can be developed. However, multi-service organizations that provide a wide array of services may be faced with the challenge of having to develop different models for each program.
The model, however, is now adapting to these initial challenges. A consultation model has been developed to help organizations design outcome measurement systems that not only integrate with their other programs, but also help organizations consider the implications of their outcomes for systemwide organizational change. In addition, the consultation model includes bringing together organizations with similar missions to facilitate peer support and joint efforts in this project.
Despite some limitations, we do believe that this model can help organizations draw meaningful conclusions with regard to outcomes and that this information can ultimately improve an organization’s programming.
The trend is clear–more and more funders are asking nonprofits to measure their outcomes. However, since funders have different and often conflicting definitions of outcomes, this trend has caused a great deal of confusion and frustration for nonprofits. We believe that nonprofits and funders should agree to common definitions and acceptable models for outcome measurement. We recommend that they explore the possibility of adopting the United Way model, not only for the reasons already discussed in this article, but to support the large number of nonprofits who are already using this model.
In support of this conclusion, we’d like to share the experiences of two organizations who are implementing the model and describe the early benefits their efforts have yielded.
This community-based organization in Salem, Massachusetts, provides a range of services and shelter for battered women and their children. HAWC has been evaluating the Nurturing Program, a twelve-week program that helps abused women improve their parenting skills while in the emergency shelter. They designed a system that measured the women’s increased knowledge of parenting techniques and how their attitudes toward some parenting issues (such as physical punishment) had changed based on participation in the program. In the future, HAWC plans to measure actual changes in their parenting behavior rather than just knowledge of those techniques. In other words, did the parents make use of the new techniques learned in the program?
The initial findings clarified what the women were learning, indicated which sessions were most effective and which needed to be rethought, and ultimately confirmed that the training program was making a real difference for the constituency. Based on these findings, HAWC was able to make some significant improvements to their program. They streamlined the curriculum from twelve to eight sessions to accommodate the varied arrivals and departures of their clients, thus allowing more women to participate. They also revised the curriculum so that it covers more relevant issues to battered women and now offer instruction in a variety of languages and presentation styles to accommodate linguistic needs and learning styles.
HAWC is currently examining how specific factors influence their outcomes. For example, what effect do women’s educational levels and their ability to speak English have? Based on their findings, HAWC will further tailor their strategies.
By improving their programs through outcome measurements, HAWC is better able to align itself to the ideals of its mission. The outcome measurement process also helped HAWC assess staff capacity issues and reorganize staffing for the training program. Finally, they are actively using their findings to advocate for additional funding for this and other programs.
Accountability to Constituents Using Outcome and Constituent-Driven Planning
The trend toward greater accountability is often viewed as just one more attempt by the funding community to keep nonprofits accountable to funders. However, because the United Way model’s focus is on the benefits to constituents, the model shifts the concern for greater accountability from funders to constituents.
PASG, a community-based organization that provides services to people with HIV and AIDS in the Provincetown, Massachusetts, area, is a good example of an organization that utilizes a constituent-driven outcome measurement system. PASG wanted to assess the rapid changes within the field of HIV and AIDS to assess the impact of such changes on its clients and programs and to evaluate current program outcomes in light of such changes.
Through the use of client surveys and focus groups conducted over a one-year period, PASG solicited opinions from their constituency regarding useful outcomes. They also evaluated the specific benefits their clients received in order to make informed decisions regarding current and new programs so that they could better align the services with their clients’ needs. As a result, PASG:
* Expanded its program scope. Rather than continuing to be primarily a service provider, it will now provide opportunities for peer and social support. For example, PASG will now shift its emphasis from the home-delivery of meals to a model that encourages congregate meals (facilitating more peer support and greater educational opportunities). PASG also expanded its transportation program so that instead of primarily providing rides to medical appointments (and thus being “illness-focused”), it will also provide opportunities for peer-support events and wellness activities.
* Reorganized its staffing configuration and is moving toward team-management structures in order to provide more effective, better-coordinated services.
* Improved its board’s decision-making process for long-term planning. For example, as it learned how clients benefited from a focus on wellness (Living with AIDS), PASG was able to make critical, strategic decisions regarding future programming that better addressed clients’ newly articulated needs.
* Used its findings to more closely align with its clients and mission. It developed ongoing mechanisms for program assessment, while increasing client participation in program revision and planning. This helped PASG build its capacity for strategic thinking within the organization. The organization also plans to publicize its findings and strategic plan to demonstrate its effectiveness and responsiveness to clients.
Outcome measurement systems demonstrably help an organization improve its effectiveness–they simply make managerial sense. They also make strategic sense: they can help your organization advocate for increased funding and position your group as a leader in defining what program outcomes are relevant within your field.
When your organization is empowered to make a case based on clearly articulated programmatic frameworks, accompanied by evidence on how your communities, your constituents, and your program participants are directly benefiting from your programs, you (like our enthusiastic Carlos Martinez) can advocate from a position of power, using language that transcends mere rhetoric.
Most important, an outcome measurement system based on the United Way model clearly makes philosophical sense–it can help your organization stay true to its mission and reaffirm your staff and board’s ability to contribute to the organization in meaningful ways.
Most of us in the nonprofit community choose to work for social change out of our own personal desire to be involved in creating a more just and equitable society. We owe it to ourselves and, most of all, to the people we purport to serve, to learn whether or not we really are making a difference. We are obligated to constantly refine, refresh, and improve the work we do for the benefit of our constituents and communities. Outcome measurement can help us achieve those goals.
Judy Freiwirth, Psy.D., has consulted to a wide variety of nonprofit organizations, particularly community-based groups, for the past 25 years. Her practice includes all major areas of organization development including program evaluation, strategic planning, board development, team building, diversity work, quality improvement, and helping organizations become learning organizations.
As lead consultant, she helped design the United Way of Massachusetts Bay’s training curriculum and consulting model and is training and supervising a team of 12 consultants who, in turn, will train and consult to more than 100 United Way affiliates.
Elena Letona, Ph.D., is the associate director of Centro Presente, a grassroots organization serving the Central American immigrant community in Boston. Previously, she managed a variety of programs in the nonprofit and government agencies on a local and statewide level. In addition to her experience as a trainer of the United Way’s initiative in outcome measurement, she has taught courses in human resource management, program planning, and implementation and evaluation.
