logo
    • Magazine
    • Membership
    • Donate
  • Racial Justice
  • Economic Justice
    • Collections
  • Climate Justice
  • Health Justice
  • Leadership
  • CONTENT TYPES
  • Subscribe
  • Webinars
    • Upcoming Webinars
    • Complimentary Webinars
    • Premium On-Demand Webinars
  • Membership
  • Submissions

Michael Quinn Patton’s Top 10 Developments in Qualitative Evaluation for the Last Decade

Michael Quinn Patton
February 25, 2014
Share
Tweet
Share
Email
Print

Flower

To help put together this list, I also consulted the three qualitative colleagues listed below (and they, in turn, with their colleagues and students) about their sense of the major trends—though they bear no responsibility for the final top-ten list I’ve constructed.

  • Sharon Rallis, incoming editor of the American Journal of Evaluation and co-author of two qualitative books
  • Leslie Goodyear and Jennifer Jewiss, co-chairs of the AEA Qualitative Topical Interest Group (TIG) and editors of the forthcoming book, Qualitative Inquiry in the Practice of Evaluation.

So, here are my top ten developments in qualitative evaluation inquiry over the last decade:

 

10. Powerful qualitative software

Software features and capabilities have expanded greatly but the learning curve remains steep. There is also confusion that qualitative software actually analyzes data: it doesn’t. Software is a data management tool. Human beings still have to organize, interpret, and make meaning from the data.

9. Social media as a qualitative tool increasingly used for both data collection and sharing findings

The rise of social media in every aspect of modern life is a hallmark of the information age. Qualitative inquiry is already being profoundly influenced by social media opportunities as social media becomes a tool both for data collection and communicating results.

8. Ethical challenges abound

The in-depth, engaged, interactive, and interpersonal nature of qualitative fieldwork increases dramatically the challenges of creating and following appropriate ethical standards. Institutional Review Boards are struggling to determine how to apply traditional standardized ethical research standards to qualitative designs that are emergent, naturalistic, and dynamic. Issues include:

  • Anticipating impact on participants
  • Confidentiality with small sample sizes
  • Appropriate compensation—particularly in circumstances where their involvement is more than simple data collection. Where does the principle of reciprocity lead us?
  • Lack of qualitative expertise and experience on review boards.

7. Mixed methods

The former qualitative-quantitative debate has realized rapprochement around the triangulated value of mixed methods. However, in actual implementation, mixed methods manifest more as parallel play (like two-year olds not yet able to play together) than as genuinely integrated inquiry and analysis.

6. Data visualization

Qualitative inquiry is rapidly incorporating a variety of data visualization techniques and tools. The two most recent issues of New Directions for Evaluation (Issues 139 and 140, Fall and Winter 2013) are devoted to data visualization. The year 2013 saw the organization of a Data Visualization TIG. Dr. Stephanie Evergreen has led AEA in pioneering data visualization as a key competency for evaluators.

 

5. Qualitative inquiry developments driven by evaluation practice and users’ demands

Historically, qualitative inquiry has been driven by epistemological, ontological, paradigmatic, and philosophical traditions and debates. Evaluation has brought to qualitative inquiry generally a pragmatic, utilitarian orientation. The intersection of evaluation and qualitative inquiry is being shaped by:

  • Short, tight timelines
  • Large multi-site studies
  • Demands for speed

However, qualitative evaluation is only realizing a fraction of the potential contributions of in-depth qualitative field methods:

  • Observation is vastly underutilized
  • Interviewing dominates—and shorter interviews, at that

4. Qualitative evaluation is being used intentionally as an intervention; a high degree of process use

Because qualitative data gathering methods and reporting approaches can be made accessible to non-academics (a.k.a., ordinary people), and because engaging in qualitative evaluation involves learning in multiple ways at multiple levels, process use is increased through evaluation approaches that often incorporate some aspects of qualitative inquiry, such as:

  • Participatory or collaborative evaluation
  • Feminist evaluation
  • Empowerment evaluation
  • Transformative evaluation
  • Developmental evaluation

These terms mean different things to different people, but share a commitment to involving the people in the setting being studied as co-inquirers, at least to some important extent, though the degree and nature of the involvement vary widely.

3. Increased value for deep contextual understanding enhances demand for and appreciation of qualitative evaluation

  • The ascendance of realist evaluation has made contextual sensitivity paramount.
  • “Contextual Intelligence” is being understood as a “A Critical Competency” (Matthew R. Kutz & Anita Bamford-Wade, 2013, E:CO)
  • Evidence-based effective principles require contextual adaptation (as opposed to the high fidelity emphasis of best practice models)

2. The qualitative evaluator is the instrument; experience, expertise, and cultural competence matter

The focus in quantitative methods is on the validity and reliability of the data collection instruments and analytic procedures.  In qualitative inquiry, the experiences and capabilities of the qualitative evaluator as a person remain central to credibility: who does the work matters. In this regard, the song lyric of country music legend Waylon Jennings is germane and insightful:

Old age and treachery
Always overcomes youth and skill

1. Increased purposeful sampling options

Purposeful sampling involves selecting information-rich cases to study, cases that by their nature and substance will illuminate the evaluation question being investigated. Perhaps nothing better illustrates the difference between quantitative and qualitative methods than the different logics that support sampling approaches. The third edition (2002) of Qualitative Research and Evaluation Methods discussed 16 purposeful sampling options. The new edition presents and discusses 35 options. This development of more nuanced and targeted purposeful sampling applications captures, for me, the key to the increased utility of qualitative evaluation methods over the last decade. In the end, whatever conclusions we draw and judgments we make depend on what we have sampled. To be more strategically purposeful about sampling is to be more strategically purposeful about evaluation.

Looking Ahead: Four Challenges and Opportunities

  1. Building qualitative inquiry capacity: Global efforts are underway to strengthen evaluation capacity. Strengthening quality inquiry capacity needs to be part of that effort.
  1. Increasing interest in and attracting resources to do serious, triangulated, in-depth qualitative, multi-method evaluations: As noted in the top ten trends, observation and in-depth fieldwork are underutilized. Interviewing and short site visits dominate.
  1. Deepening evaluators’ commitment to inquire seriously into unintended consequences and take emergence (complexity) seriously: Lip service and rhetoric give the appearance of attending to unintended consequences, but most evaluation designs devote the entire budget to assessing planned implementation and goal attainment. The kind of open-ended fieldwork needed to turn up actual consequences and emergent dynamics remains rare.
  1. Cumulative-longitudinal integration at the case and context levels: Long-term, in-depth case studies, with purposeful sampling that is sufficiently diverse to capture contextual variations, remains an ideal too rarely realized in practice.

 

Share
Tweet
Share
Email
Print
About the author
Michael Quinn Patton

Michael Quinn Patton is the former president of the American Evaluation Association. He has authored six evaluation books, including Utilization-Focused Evaluation, Qualitative Research & Evaluation Methods, and Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.

More about: Culture ChangeManagement and LeadershipOrganizational culture and its driversSocial MediaTechnology

Become a member

Support independent journalism and knowledge creation for civil society. Become a member of Nonprofit Quarterly.

Members receive unlimited access to our archived and upcoming digital content. NPQ is the leading journal in the nonprofit sector written by social change experts. Gain access to our exclusive library of online courses led by thought leaders and educators providing contextualized information to help nonprofit practitioners make sense of changing conditions and improve infra-structure in their organizations.

Join Today
logo logo logo logo logo
See comments

Spring-2023-sidebar-subscribe
You might also like
Hierarchy and Justice
Cyndi Suarez
Why Tech Nonprofits Are Building Digital Tools for Racial Equity and Justice
Sakshi Udavant
Technology for Whom? Owning Our Platforms
Minsun Ji
A Better Way to Do Good: The Case for Comprehensive Community Care
Erin Mulligan Nelson
Grave Work
Phil Jones
How to Make Better Decisions Online
Josh Lerner and Rose Longhurst

NPQ Webinars

April 27th, 2 pm ET

Liberatory Decision-Making

How to Facilitate and Engage in Healthy Decision-making Processes

Register Now
You might also like
Hierarchy and Justice
Cyndi Suarez
Why Tech Nonprofits Are Building Digital Tools for Racial...
Sakshi Udavant
Technology for Whom? Owning Our Platforms
Minsun Ji

Like what you see?

Subscribe to the NPQ newsletter to have our top stories delivered directly to your inbox.

See our newsletters

By signing up, you agree to our privacy policy and terms of use, and to receive messages from NPQ and our partners.

NPQ-Spring-2023-cover

Independent & in your mailbox.

Subscribe today and get a full year of NPQ for just $59.

subscribe
  • About
  • Contact
  • Advertise
  • Copyright
  • Careers

We are using cookies to give you the best experience on our website.

 

Non Profit News | Nonprofit Quarterly
Powered by  GDPR Cookie Compliance
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Strictly Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.