During the last few years, the pressure for increased accountability, continuous improvement, and service innovation has sparked an “evaluation revolution” in nonprofit organizations. Nonprofits are responding to the challenge by defining their intended outcomes and then measuring their results in ways that tell accurate “performance stories” and identify new opportunities for programming and collaboration. Consider these three scenarios for a moment:

Scenario #1: A large nonprofit grantmaker must obtain evaluation data from 300 organizations to show donors that their money is producing real results in areas of vital community concern. It also needs evaluation information to improve its strategic grantmaking and take advantage of new fundraising opportunities.

Scenario #2: A small nonprofit with few staff and even less money needs evaluation data to improve its programs, work more effectively with its partners, and meet demanding new reporting requirements.

Scenario #3: A government funder needs evaluation data from nonprofit agencies to demonstrate accountability for the expenditure of public money, address high-priority policy areas, and encourage the use of good practices.

These three apparently different scenarios have several elements in common: there is a pressing demand for accurate evaluation information; evaluation is necessary for learning as well as accountability; evaluation is essential for strengthening programs and forging new directions; and there are limited resources for evaluation. The challenge is obvious—how do nonprofit organizations with limited resources develop an evaluation response that meets these demands and seizes these opportunities?

Many nonprofits are turning to technology in response to this challenge. Technology offers nonprofit organizations the opportunity to design and implement more effective evaluations for improving program performance while doing so at lower cost. What do I mean by technology? In the evaluation arena, the term refers to information and communications technology (ICT) that signifies the close relationship between computers and telecommunications. ICT has two immediate benefits for the evaluation process:

• Enhancing the use of familiar methods (surveys, interviews) to achieve better, less expensive evaluations.

• Accessing the wellspring of technology innovation to improve service delivery, strengthen communities, and catalyze meaningful change.

Although many nonprofits are already using technology, most have not moved much beyond the basics of e-mail, teleconferencing, and accessing Web sites. They have not reaped the benefits of integrating technology into the evaluation process. With technology changing so rapidly, one purpose of this article is to present a “mental map” of the technology landscape, so that nonprofits can appraise and select the right tools to meet their evaluation needs. Most nonprofits start by using technology to implement “better, faster, cheaper” evaluations, but as they become more experienced they begin to use technology in unforeseen ways. So another important purpose of this article is to build awareness of how evaluation technology can help nonprofits innovate, by becoming aware of new approaches for meeting persistent community needs and building community strengths that cannot be addressed (or even envisioned) without technology.

Technology in Action

A survey of the technology landscape shows that nonprofits are using technology successfully in four major aspects of the evaluation process—data collection and analysis, collaboration, knowledge mobilization, and evaluation capacity building. Technology is rapidly expanding the tools for doing evaluations, broadening participation and collaboration, fashioning highly effective channels for mobilizing evaluation information into focused action and social change, and building the capacity of nonprofits to conduct their own evaluations. The following sections examine each of these four themes more closely.

Data Collection and Analysis. During the recent Asian tsunami disaster, we all witnessed the power of technology to help organizations such as Doctors Without Borders (Médecins Sans Frontières) and the Red Cross obtain an unprecedented level of financial support and disaster relief from around the globe. In fact, “click-and-donate” was so powerful that we saw some nonprofits do something unprecedented—they were overwhelmed with donations and asked them to stop. Just as the tsunami disaster showed plainly the power of technology to transform philanthropy, in a less dramatic but equally significant manner, technology is beginning to transform nonprofits through new tools for collecting and analyzing evaluation data. These electronic tools give nonprofits new ways of understanding client needs, improving services, catalyzing community change, and deepening relationships with stakeholders and donors alike.

Electronic surveys are an important example of how technology can impact nonprofit evaluation. The client satisfaction questionnaire is arguably the most frequently used evaluation tool by nonprofit organizations. Instead of costly mail or telephone surveys, nonprofits are increasingly turning to client feedback surveys administered either by e-mail (questionnaire sent by e-mail) or the Internet (questionnaire posted on a Web site). There are several advantages to e-mail or the Internet surveys: quicker response times (days instead of weeks), greater convenience for clients (complete the questionnaire at any time of the day or night and submit it instantly), and sharply reduced costs (after initial setup, costs are virtually the same for 100 or 1,000 questionnaires). If that were not enough, many inexpensive online survey packages enter data from completed questionnaires automatically and then employ analysis templates to generate nearly instantaneous feedback, nicely formatted with professionally-designed tables and charts. And did I mention the sharply reduced costs?

There are some clouds floating in the sky of this electronic evaluation paradise. Online surveys are not without their limitations. Survey expert Don Dillman, author of Mail and Internet Surveys: The Tailored Design Method (Wiley: 2000), points out that good survey fundamentals must be applied to Internet surveys to avoid potential sources of error. He makes clear that poorly designed surveys are likely to fail, whether we administer them by mail, telephone, face-to-face, or the Internet. Further, electronic surveys bring with them their own unique problems and require the right knowledge and skills to deal with them. Common problems include electronic surveys that are too long, look confusing on computer screens, or do not reach the target group because they do not have access to the Internet.

Nonprofit evaluation today also employs a bevy of other data collection tools, both quantitative (numeric) and qualitative (narrative, audio/visual). In addition to electronic surveys, nonprofits are obtaining individual and group feedback from clients and service delivery partners through online discussion forums, real-time chat sessions, and Internet videoconferencing. These qualitative approaches provide in-depth data that help explain how program activities are contributing to client outcomes and long-term impacts. Nonprofits are collecting interview data and digital photos/

video clips using the new generation of powerful and low-cost cellphones, to paint a more accurate picture of the relationship between program activities and outcomes. We saw the power of these devices during the bombings in London during the summer of 2005. Cellphone-enabled “citizen reporters” posted their eye-witness accounts and photos/video clips from the London subway to public Web sites for instant access by first-responder teams, mass media and relatives alike. Staff, volunteers, and program participants are now using the same “popular technology” to document important program events and collect data that provide compelling evidence of program successes and shortcomings. Because of their low cost and convenience, these methods can provide timely evaluation feedback to strengthen program design and delivery. Instead of evaluations being done only after a program is completed, technology can provide a rich “diary of development” or “moving picture” that tracks program progress and guides program activities to ensure positive outcomes.

Collaboration. In nonprofits, program design and delivery are becoming increasingly collaborative and complex. Partnerships, cooperative ventures, and service delivery networks with multiple delivery locations are becoming the norm. In keeping with these changes in nonprofits, evaluation is evolving as well. Contemporary evaluation practice is stakeholder-based and participatory. Managers, staff, service delivery partners, and program participants alike are expected to participate fully in the evaluation process.

As much as the new models of evaluation prize inclusiveness and collaboration, stakeholder involvement can be difficult because it is time-consuming and expensive, especially if travel is required. To meet these challenges, evaluators are turning to electronic collaboration tools to complement face-to-face meetings and teleconferences. Tools for electronic collaboration provide the means to do effective group work. The most basic e-collaboration tools, usually provided at low cost through your Internet Service Provider, are listservs and discussion forums. These tools permit evaluators and stakeholders to collaborate more effectively by organizing messages into “threads” according to topic, providing a secure repository for evaluation documents, and by supplying tools for conducting secret ballots on important issues.

Beyond the basics, the more advanced collaborative tools can enhance the entire evaluation process through their scheduling, calendaring, approval tracking, and other project management features. Nonprofits have a wide choice of full-featured collaborative tools, ranging from sophisticated enterprise-level solutions (e.g., e-Room, Sharepoint), to lower cost solutions (e.g, WebOffice), to open-source software (e.g, Plone) that is available at no/low cost but requires some programming expertise to implement well. These advanced e-collaboration tools also reduce the potential for error and workload of generating multiple versions of evaluation reports by offering sophisticated document management functions, such as version control and document routing, so that documents are disseminated to the right audiences at the right time. These advanced collaboration tools can help nonprofits manage complex evaluation projects that involve multiple partners and/or multiple sites. In short, e-collaboration tools enable nonprofits to lighten their workloads, reduce costs, and manage projects more effectively, while improving the overall evaluation process through broader stakeholder participation and better project management.

Knowledge Mobilization. One of the most important emerging applications of technology for nonprofits is knowledge mobilization, that is, using evaluation findings to mobilize action and drive change. It is a response to one of the major challenges facing nonprofits today—how to use knowledge resources in a strategic way for improving services, advocating change, building communities, and transforming society. Knowledge mobilization goes far beyond the usual practice of simply disseminating evaluation findings in a written report or brief presentation. Knowledge mobilization stresses an exchange of information among stakeholders throughout the evaluation process, shared-decision making, democratic deliberation and validation of evaluation findings, and then using evaluation findings in strategic and proactive ways to influence decision-makers and improve the lives of those that nonprofits serve. Let’s look at a concrete example of how technology is being used for knowledge mobilization by the nonprofit sector.

The Growing Roots…Strengthening Neighbourhoods Program of the Hamilton Community Foundation (HCF), in Hamilton, Ontario, is a five-year pilot project that strives to strengthen four challenged neighborhoods and contribute to improved quality of life of the residents. The program seeks to build assets by creating opportunities for residents to gain leadership skills and increase the organizational and leadership capacities in the neighborhoods. In a recent issue of Harvard’s Evaluation Exchange publication (Fall, 2005), Betty Muggah and I provide details of the evaluation of this program. In a nutshell, the evaluation began with a participatory assets-mapping process to deepen residents’ understanding of their assets and strengths (see G.P. Green and A. Haines, Asset Building and Community Development, Sage: 2001). Small groups of residents worked with the program coordinator to portray neighborhood assets on hand-drawn maps. The groups met and discussed the assets maps in a community forum. Next, Geographic Information System (GIS) software was used to map the assets digitally and integrate them with demographic and other statistical data. To collect program outcome data from neighborhood residents, the evaluation employed photovoice, a technique combining photography and storytelling that has proven very effective for participatory evaluation and community building. After analysis, the various data streams were integrated in a multimedia presentation that combined graphical data, photographs, and both quantitative and qualitative data into a computer slide show. The multimedia presentation served as a focal point for a community meeting attended by hundreds of residents and other stakeholders. After the presentation, the residents publicly discussed the findings and the lessons learned from the evaluation, other stakeholders heard about the issues affecting the neighborhoods, and everyone deliberated together about how this knowledge might be transferred to other challenged neighborhoods in Hamilton and be used to influence public policy. Because of the ease of incorporating digital photographs, graphical data, and narrative into multimedia presentations, technology made the entire knowledge mobilization process not only feasible, but practical and highly effective too.

Evaluation Capacity Building. Evaluation capacity building is a hot topic in the evaluation field— how do nonprofits develop the capacity to provide accurate evaluation information, especially evaluation that will lead to improved program delivery, organizational learning, and accountability for results? Technology is helping to build the capacity for evaluation and forging the infrastructure for supporting evaluators and contributing to their professional development. For example, many nonprofits have already made use of the “evaluation toolkits” freely available online, such as the widely-respected program evaluation and logic model handbooks that may be downloaded from the W.K. Kellogg Foundation’s Web site (www.wkkf.org). Other nonprofits are developing capacity by making use of the evaluation topical interest groups, discussion groups, article databases, and links to evaluation resources available through the professional evaluation associations, such as the American Evaluation Association (www.eval.org) and the Canadian Evaluation Society (www

.evaluationcanada.ca).

At the grassroots level, nonprofit programs are also building evaluation capacity through short half-day courses and mentoring that transfer self-evaluation skills to program staff. For example, when designing programs, staff are learning to search online databases to examine the relative strengths and weaknesses of a proposed model of change. This leads to better-designed and more effective programs by building on the experience of others—all before spending any money. Managers and staff are learning to document and analyze program outcomes through the use of cost-effective technologies such as online surveys, digital photographs and videos, and online focus groups. They are learning how to share findings from evaluations on Web pages and debate them in discussion forums and virtual conferences. In these applications, technology not only helps to build evaluation capacity, but also helps to empower staff, while building a bridge between evaluation and action.

Challenges and Next Steps

Does technology have the power to transform evaluation in nonprofits? I think that the answer is clear: nonprofits alone have the power to use technology to transform evaluation into an integral part of nonprofit management and a valued tool for organizational and community development and change. History shows that technology is rarely adopted unless it meets a need. Therefore a key question to ask is: what will technology do for evaluation in my nonprofit and what aspects of evaluation practice will it facilitate or replace?

A related challenge is that within the next ten years, technology is expected to transform the way nonprofits, no matter how large or how small, deliver their services, relate to clients and the public, and govern themselves. As Joni Podolsky notes in Wired for Good: Strategic Technology Planning for Nonprofits (Wiley: 2003), there is still time to derive maximum benefit from technology by developing plans that will align technology with your mission and strategic goals, help you select the right technology tools, and chart your way through the implementation process. It is important to integrate evaluation into this process—evaluation has a key role in guiding your nonprofit along the migration path and in helping you overcome obstacles as they arise.

Harnessing the power of technology for evaluation depends largely on appreciating the pros and cons of technological tools and using this knowledge to shape technology to our own ends. The inherent flexibility of electronic tools should encourage us not to fixate on the hardware and software but to direct our attention to improving the evaluation process, making it more inclusive and transparent, building truly collaborative evaluation efforts, removing the drudgery from data collection and focusing more on data analysis and use, and vastly increasing the reach and impact of evaluative information for the betterment of all.

Arnold Love is a program evaluation consultant who has taught program evaluation at the Nonprofit Management and Leadership Program, Schulich School of Business, York University, for more than 20 years. He specializes in the evaluation of nonprofit and public programs and in building evaluation capacity in organizations worldwide.