This article is written from an evaluator’s point of view and although the examples focus on community-based organizations, they illustrate broader lessons about accountability, evaluation and participation that should prove useful to any nonprofit leader. As you read this piece, we suggest that you think about which elements of the practice described here can inform your own work.
Many community-based organizations (CBOs) experience two very different sets of demands. On the one hand, they want to connect more closely with the communities they represent and serve. Honest engagement with the community includes both understanding how residents view their situations and involving them in determining, evaluating, and then re-determining program goals. This is a dynamic and progressive process. On the other hand, CBOs need to show progress toward measurable outcomes because funding is increasingly contingent on specifying such outcomes before work gets started. This presents a conflict between openness and closure, although both point to the need for systematicways of gathering information. It is our belief that when CBOs establish participatory research and evaluation as a core part of their practice, they protect the quality of their work with their communities in a way that builds credibility, power, and influence with other stakeholders, including funders. Perhaps most importantly, it builds effectiveness.
This article examines ways that CBOs and community residents can become partners in knowledge enterprise. The examples cited here come primarily from our work in New York City, but the challenges and lessons learned have broader applicability. Since 1990, the Housing Environments Research Group (HERG) has worked with dozens of CBOs and surveyed over 8,000 residents of inner city neighborhoods searching with them for “practical truths” (Montero, 2002) that support action.
At the simplest level, if the projects and programs undertaken by CBOs embody a wish for change, then community development evaluations are designed to determine if the wish has come true. But as we seek to understand both the wish and the extent to which it has become reality, questions arise about the different perspectives and interests that define “the wish” and drive the actions taken in its name.
Tensions between the CBO’s interests and perspectives inevitably arise. For example, a CBO may want to understand who lives in the community and which housing programs work best for which residents. At the same time, the CBO may have a complicated set of interests to protect; for instance a CBO may own some of the housing and have funders who want evidence of successful outcomes. This may cause a cer tain amount of ambivalence within the CBO whereby it may wish to understand how to act more effectively, but may also fear the consequences of surfacing information that suggests that mistakes have been made. Consequently, it is not uncommon for a group of CBOs working in the same neighborhood to differ widely in their vision of what is in the community’s best interests as end results. This may result from such things as the influence of funders, or the personal preferences and skill bases of organizational leadership, or even the CBO’s own institution-building interests. Given that these interests exist and are often quite powerful, how and to what extent a CBO is rooted in a community becomes a critical question. Is the organization in question committed to the involvement of resident s in co-designing their neighborhood’s future?
When the answer to that question is “yes,” and the wishes of the neighborhood are paramount in what a CBO decides to do, there is a whole practice that should, but does not always, follow. Neighborhood residents come from different backgrounds, live in var ied ways, and have divergent strengths, weaknesses, hopes and goals. (Saegert and Glunt, 1990). Understanding these divergent interests—and forging common ones—are an important and continuous part of community development. Evaluation is as much about the ongoing definition of a collective wish as it is about an assessment of progress toward a goal.
We, as evaluators inspired by the works of Antonio Gramsci and Paulo Freire, engage ourselves, CBOs, and community residents in a process of understanding the dimensions of the current situation and engaging in critical thinking about our shared reality. This has a political character to it—a character of resistance.
Gramsci’s concept of hegemony describes how all parts of society are immersed in dominant discourses reinforced by powerful institutions. This means that people living in poverty may sometimes agree to the wishes of the dominant group as if it were their own. Paulo Freire (1972) refers to this molding of consciousness as “prescription.” Participation of less powerful groups in action-reflection processes help develop, articulate and activate an alternate world view by identifying shared perceptions and tracing the cause-and-effect factors (Speer and Hughey, 1995). This provides a basis for independent wishes, collectively articulated.
Instead of measuring community life against standards set externally, participatory research can allow communities to find their personal and shared realities and desires that can be both negotiated and contested. The outcome is not to uncover a stable reality, but to generate dynamic knowledge that can be used to discover, debate, and fulfill the wishes of the community.
Community evaluations pass through different stages, each shedding new light on the following questions: Whose story about the situation gets told? How do the relationships between involved parties affect the evaluation process? How do these relationships influence what can actually be learned? The answers to these questions unfold over time, giving rise to each other and leading to reinterpretations of reality that fold back into one another. We characterize these moments as including:
- Moments of meeting
- Moments of surprise and consolidation
- Moments of constrained conversations
- Moments of reflection and interpretation
- Moments of formal representation of reality
- Moments of challenge for future action
In our practice as evaluators, at the beginning of all evaluations we meet with the funder, the CBOs, and others who will be integral to the process. This moment rarely marks a new beginning, but is usually inserted into prior relationships. Any CBO beginning a similar effort should have much the same type of meeting. Participants usually know each other through a mix of relationships and reputations, but their knowledge of each other is incomplete and fragmented. After studying low-income housing in New York for 20 years, we often meet old friends, exfunders, past opponents of policy fights, new directors of programs we’ve studied since their beginnings, people whose children we have seen grow up, and those who have gone from partners in research to positions in government agencies and foundations. Prior promises, shared efforts, hopes and fears all hang in the air as we negotiate the following:
- Agreements between the parties about the scope and purpose of the work; division of responsibilities; cost; products and time frame (even though all will likely change).
- The character of the relationships within the group. (Who listens to whom? Which areas are subject to compromise, and which are not?)
We enter this stage wanting to honor CBO residents and staff as privileged knowers of their own lives, while at the same time trying to advance understanding of current conditions and the consequences of CBO activities. Benitez (2002) has used Gramsci’s terminology to describe CBO staff because they perform the functions Gramsci identifies as the occupations of “intellectuals”: organizing, administering, directing, educating and/or leading others. Both evaluators and CBO staff can act as traditional intellectuals or organic intellectuals. Organic intellectuals represent the common sense of a particular place and social world that is more likely to emerge when evaluations promote a cycle of action and reflection.
Those who fund evaluation play a critical role in enabling or constraining the degree to which evaluations allow for the formation of organic intelligence. The amount of trust that develops between the funder, the CBO, the evaluator, and community residents helps to determine whether community goals and experiences can be expressed in the evaluation. Communication and frequent interaction are required to promote trust and shared understandings. Interaction and communication take time, and time is usually bounded by the dollars available to pay for it, as well as by competition from other activities and demands.
Communication also requires that those who are engaged in the process work through ideas together over time. Barriers to dialogue arise when the people involved in the conversation are too transient. Turnover among key funders, evaluators, and CBO staff can disrupt a process. Such changes are inevitable but steps can be taken to minimize discontinuity of personnel. For example, in one three-year evaluation, we assigned an evaluation team member as a contact person for each site. In addition, we designated a research assistant team leader for the project who could back up the site’s assigned researcher and provide continuity of communication. The lead researcher also communicated regularly with CBO staff and the foundation, as well as with the research team, to promote continuous dialogue and to assure that what was learned from the dialogue was reflected in formal representations of the evaluation. In other words we created enough redundancy so that even though the team’s composition changed over three years, we had introduced and engaged new people before the old members withdrew.
Some minimum of shared cultural knowledge greatly enhances the chances for good communication. Thus the ethnic and racial composition of the team, as well as members’ previous experience, is very important. In order to build up the basis for communication that dialogue requires, we create diverse teams who work together intensely and spend as much time in the community as possible. Middle- and upper-class researchers of European origin can function well in these situations, but not if they predominate completely, and not unless they engage in actively addressing differences in perspective. The same is true, of course, for anyone, but it is particularly important when a dominant paradigm might be imposed on a process.
Our core team has a fundamental diversity, which extends deeper than simply racial identity. To give but two examples: we have native-born Spanish speakers as well as non-natives who achieved fluency through school and time spent abroad; we have team members from rigorous social science backgrounds as well as from the arts and music. The loose weave of our combined personalities and life-experiences allows neighborhood residents greater options for interaction, and more surface area on which to choose a point to engage us.
Shared social goals foster the ability to communicate over the long run, a position that runs counter to the traditional claim of objectivity and neutrality of the researcher. The point of evaluation is to learn about the conditions in which communities and individuals live, and the nature of actions that improve them. If all parties to a dialogue share some common understandings of neighborhood conditions, and the desire for shared goals, then the picture of the world they develop can become an “actionable” reality.
We’ve often encountered breakthrough moments—both for us and for the community —early in the evaluation process, when an unexpected piece of hard data is revealed. This data is usually related to the community’s physical environment, and often tells both the evaluators and the CBO something that we did not already know about the community, thus providing an occasion for interpreting information together. Our preliminary steps usually involve exploring the environment intensively from as many different angles as possible. The research team experiences moments of surprise during the revelation of information— at this stage, usually physical or demographic —that gives us a sense of strong acquaintance with the community. For the community and the CBOs, the moments of surprise often involve finding a new, more intelligible, holistic perspective from which they can view themselves.
We find it useful to present information using photos, maps, numbers, and graphs in order to provide a not-yet-linguistically-processed puzzle for discussion. This guards against the evaluator or facilitator of the process having too much ownership of the analysis. Presenting the information graphically also allows us to start linking the physical dimensions of communities with their sociocultural context. It helps us understand, for example, the ways in which building conditions are linked to the residents’ willingness to participate in a community organization. Our use of census data and neighborhood and housing inventories along with detailed maps have served to promote shared reflections about the neighborhood and how it is changing. Residents can reflect upon and share their own knowledge of the neighborhood’s characteristics with the evaluation team, and both can reconstruct their knowledge, assimilate the new information given to them, refute misinformation, and identify ambiguities. This productive dialogue between the evaluator and the community forms a reciprocal relationship, fundamental to the work we aim to do.
Capturing the diverse experiences of community residents, especially those who are not closely associated with the CBO, is challenging. Community surveys provide a good opportunity to reach a large number and variety of residents, albeit in a constrained format. The quantitative, “scientific” forms of knowledge that surveys produce can be hard for community residents to understand and use. However, such data can provide residents with both meaningful information and tools to interpret it if the whole survey process is viewed as an opportunity for dialogue and shared reflection.
The formal, constrained nature of a survey presents an opportunity for all participants to clarify perspectives and communicate them to a widening group of relevant others. When we generate survey items, drawing from existing research on community development, we explain why they may be relevant. Questions are reviewed with CBO staff and, if possible, with community representatives and/or an advisory board of key stakeholders. As we discuss each item, we have the opportunity to explain other research that we believe is related to the situation being studied. The community representatives should then challenge or confirm our understandings, as well as present their own ideas. Important questions derived from practical knowledge are added and the academic irrelevancy quotient is reduced. Community “secrets”—illegal immigrants, drug problems, shady real-estate speculation, etc.—surface in discussions of what questions people will and will not answer. The aim is for the final questions and their answers to be deeply interesting to both the CBO staff and the neighborhood residents.
Viewed from a theoretical perspective, the community survey begins to make actionable the wish that motivates community development by trying to make its components measurable. For CBOs, this is often a moment of dialogue and conversation where tensions between s tandardized ideas and grounded knowledge of the community emerge. We strive to avoid inhibiting opportunities for thinking through the developing agenda for action.
Community residents are enlisted to do surveys, or to help gain access to other residents. The first step to engaging the larger community in a meaningful, if limited, dialogue is to identify key people in the community and establish relationships of trust. Paying surveyors contributes to the project legitimacy, but is not enough. For example, when we asked a community surveyor about her experience, she said:
“Participating in the process of a community survey gave me the opportunity to talk to people—especially young people and seniors—about their problems and concerns. From participating in the survey process I learned that a lot of people in my community are having problems related to housing and they are trying to find help, get their buildings straightened out, and looking for information on how to improve their buildings. This survey was very significant in this community because it let people know that people out there want to know about our living conditions, they want to hear our voices; our voices count and someone is out there listening.”
Sign up for our free newsletters
Subscribe to NPQ's newsletters to have our top stories delivered directly to your inbox.
By signing up, you agree to our privacy policy and terms of use, and to receive messages from NPQ and our partners.
Gaining trust requires learning about community social organization and neighborhood culture. While facilitating the process of a community survey, two members of our team became active participants in the summer internship program sponsored by a local CBO. By participating in the daily activities of this program, we developed a more trusting relationship with the community and got to know the leaders in the community. Good relationships with these leaders were critical for the success of the survey.
The portion of the dialogue represented by the formal survey data is a small segment of the widening conversation. We have seen door-to-door surveying rearrange the geography of neighborhood relationships in many heartening ways. The presence of community surveyors throughout the neighborhood contributes to awareness of the CBO, the project, and the problems being discussed. As more people become involved in the conversation, new issues arise. Hidden populations are discovered, such as squatters or new immigrant groups. We also begin to discover who will not talk with surveyors, and occasionally why. As the limitations of the survey method reveal themselves, ethnographic and qualitative methods are needed. Thus, as the survey progresses, the research team must be in the community frequently, and they must work closely with surveyors.
Resident surveyors often volunteer because of their commitment to improving their community. As they broaden their circle of acquaintances, they gain legitimacy and knowledge as community representat ives. Sur veying becomes a vehicle not only for gathering information, but also for analysis of the community, for broadening participation, and for taking action. For example, community surveyors from one evaluation recounted bringing the conversations they had while surveying back to meetings for interpretation and debate. When they encountered residents with pressing problems, they tried to solve them. Surveyors also brought their discoveries to the attention of the CBO, and engaged staff in thinking about new approaches to their work.
When survey data are translated into numbers, community representatives often disengage. The gap between the complex, open-ended experience of living in the community and the apparent closure of numerical data can undermine the process of shared reflection and interpretation. But quantitative information can also provide an object for all to examine, puzzle about, question, and interpret. When we bring the numbers back to community representatives, qualitative data is essential to “revitalize” the moribund sections of quantitative data. The numbers feed the community with information about itself, and the interviews, meetings, and engagement in every-day life keep the interpretation of numbers open, allowing for new ideas and directions to evolve.
Great conversations often grow out of shared examination of the numbers of people choosing one answer over another. Community representatives and CBOs often see numerical data as evidence that their wish has or hasn’t come true. They begin to explain what they think the numbers mean and question whether the data capture certain realities. We as researchers do the same thing. By discussing interpretations with community members, we recognize that the process is saturated with values and interests. If we listen well, we can broaden our own visions and clarify the questions that the community wants answered through more sophisticated statistical analysis. For example, if the data are read as indicating that the wish did not come true, an explanation is often offered such as “a lot of neighborhood people do not have contact with us because they believe in solving their own problems within the family.” That explanation cannot be taken at face value but becomes a hypothesis to test.
CBOs both fear and need formal representations of their work. Charts and written reports affect funding and the organization’s reputation, for better or worse. But often the formal representations seem to have little to do with the experience of community life, and the successes and failures of everyday work in community development. The research process we describe aims to produce a formal representation that rings true to the community. But the negotiation of a written report is a particularly crucial moment in the relationships surrounding an evaluation.
Clarity is paramount. Graphic illustrations and writing in plain English help explain quantitative findings, but it is impor tant not to mislead the reader (for example, it could be that the people whom CBOs relate to are more likely to participate anyway). Statistical analyses show to what degree different factors contribute to an outcome. While such analyses are often difficult to explain verbally, they can be more easily grasped when presented as graphic illustrations (e.g. bar and pie charts).
Mapping data also helps community residents understand what is going on, and in fact puts them in a position to inform the researcher about the causes of findings. For example, residents will have a lot of information about why— even though housing conditions are widely perceived as improving—a group of buildings on one block report conditions as worse than five years before. The map information also helps CBOs target their activities.
We review final reports with CBO staff, community representatives, and funders. If disagreements occur, we attempt to reanalyze data to test the alternative ideas. We correct problems that we can verify, sometimes gathering new data. In the end, the negotiation of a final report tests the trust and communication that has developed among researchers, funders, CBOs and residents, revealing the extent of reciprocity, commitment and mutual interest, and giving rise to a new phase.
Open meetings for the community give life to a report that might otherwise gather dust. The discussions are often freer and more specific, quite different from discussions during the survey and interview phases. Their liveliness reflects both increased social engagement and the expanded ability of residents—challenged and invigorated by the report’s information—to examine, formulate and act on their needs afresh.
When a community study succeeds in engaging residents, CBOs, and funders in shared reflection, it prepares the ground for future action. Such studies often mark the end of a funding commitment, yet promote a flood of new ideas. Funders hope that community-development efforts will become self-sustaining by their end. But when new ideas and relationships form through the process of evaluation, financial support for follow-up activities is needed.
For example, a study for a community revolving-loan fund challenged the assumptions behind their funding strategy. Contrary to expectations, findings revealed generally good physical condition of a below-market-rate housing stock. But rising sales prices and monthly costs threatened to block access to the stock by low-income households. The research helped community members see their own buildings in the physical, economic, and social context of the larger housing market and to realize the shared nature of their needs and problems. At a community meeting on the report (Saegert, Benitez, Eizenberg, Extein, Hsieh, and Chang, 2003), residents identified their own conflicts and asked for help in confronting their problems. After deliberation, the loan fund made a grant to meet this request. Thus the process of action and reflection, sparked by the study, continued.
By the end of an evaluation, the community’s “wish” and the community, itself, have evolved. The evaluation sharpens particular definitions of the wish and its fulfillment, as it also sets new puzzles and illuminates wishes that have been unaddressed. Side conversations and new relationships begin. Act ive suppor t from large numbers of community residents can often help CBOs win battles with powerful public and private institutions. The energy set loose by the unfinished nature of the “wish” provides a basis for CBOs to develop new programs, to demonstrate the need for new funding, and most of all to root their efforts more strongly in the felt desires of the community.
We also believe that the knowledge and strategies communities generate to promote a better future for themselves resonate beyond the boundaries of the neighborhood, contributing to a public wish for a more just society. The vibrancy of research findings grounded in neighborhood residents’ lived experience can contribute to the public’s “common sense” assumptions about inner city communities that go beyond the derogatory stereotypes of both popular and scholarly media. In the also diverse larger society, many people can identify their own goals with the fuller picture of the lives of residents of disadvantaged communities, and their efforts, achievements, and desires.
1. We wish to acknowledge Gary Winkel’s invaluable contributions to our work; the responsive, appropriate, and up-to-date approach to statistical analysis we use would be impossible without him.
References
Benitez, Lymari. 2002. Deconstructing hegemony: Applying Gramsci and Friere to brownfield redevelopment in the South Bronx. Paper presented at 2002 Conference of the Association for the Study of People and Places, Laoruna, Spain.
Bourdieu, Pierre. 2000. Pascalian meditations. Stanford University Press.
Freire, Paulo. 1972. Cultural action for freedom. Ringwood, Victoria, Australia: Penguin Books.
Gramsci, Antonio Ed. 1978. Selections from political writings. New York: International Publishers.
Montero, Maritza. 2002. On the construction of reality and truth. Towards an epistemology of community social psychology. American Journal of Community Psychology, 30(4): 571-584.
Saegert, Susan, Lymari Benitez, Efrat Eizenberg, Melissa Extein, Tsai-shiou Hsieh, and Chung Chang. 2003. Limited equity co-ops as bulwarks against gentrification. Presented at the Urban Affairs Association Conference, Cleveland Ohio, March 28, 2003.
Saegert, Susan, and Eric K. Glunt. 1990. Community development corporations and community behavior: Implications from environmental psychology. Working paper, Community Development Research Center, New School for Social Research, New York.
Speer, P. W. and J. Hughey. 1995. Community organizing: An ecological route to empowerment and power. American Journal of Community Psychology 23(5):729-748.
