To help put together this list, I also consulted the three qualitative colleagues listed below (and they, in turn, with their colleagues and students) about their sense of the major trends—though they bear no responsibility for the final top-ten list I’ve constructed.
- Sharon Rallis, incoming editor of the American Journal of Evaluation and co-author of two qualitative books
- Leslie Goodyear and Jennifer Jewiss, co-chairs of the AEA Qualitative Topical Interest Group (TIG) and editors of the forthcoming book, Qualitative Inquiry in the Practice of Evaluation.
So, here are my top ten developments in qualitative evaluation inquiry over the last decade:
10. Powerful qualitative software
Software features and capabilities have expanded greatly but the learning curve remains steep. There is also confusion that qualitative software actually analyzes data: it doesn’t. Software is a data management tool. Human beings still have to organize, interpret, and make meaning from the data.
9. Social media as a qualitative tool increasingly used for both data collection and sharing findings
The rise of social media in every aspect of modern life is a hallmark of the information age. Qualitative inquiry is already being profoundly influenced by social media opportunities as social media becomes a tool both for data collection and communicating results.
8. Ethical challenges abound
The in-depth, engaged, interactive, and interpersonal nature of qualitative fieldwork increases dramatically the challenges of creating and following appropriate ethical standards. Institutional Review Boards are struggling to determine how to apply traditional standardized ethical research standards to qualitative designs that are emergent, naturalistic, and dynamic. Issues include:
- Anticipating impact on participants
- Confidentiality with small sample sizes
- Appropriate compensation—particularly in circumstances where their involvement is more than simple data collection. Where does the principle of reciprocity lead us?
- Lack of qualitative expertise and experience on review boards.
7. Mixed methods
The former qualitative-quantitative debate has realized rapprochement around the triangulated value of mixed methods. However, in actual implementation, mixed methods manifest more as parallel play (like two-year olds not yet able to play together) than as genuinely integrated inquiry and analysis.
6. Data visualization
Qualitative inquiry is rapidly incorporating a variety of data visualization techniques and tools. The two most recent issues of New Directions for Evaluation (Issues 139 and 140, Fall and Winter 2013) are devoted to data visualization. The year 2013 saw the organization of a Data Visualization TIG. Dr. Stephanie Evergreen has led AEA in pioneering data visualization as a key competency for evaluators.
5. Qualitative inquiry developments driven by evaluation practice and users’ demands
Historically, qualitative inquiry has been driven by epistemological, ontological, paradigmatic, and philosophical traditions and debates. Evaluation has brought to qualitative inquiry generally a pragmatic, utilitarian orientation. The intersection of evaluation and qualitative inquiry is being shaped by:
- Short, tight timelines
- Large multi-site studies
- Demands for speed
However, qualitative evaluation is only realizing a fraction of the potential contributions of in-depth qualitative field methods:
- Observation is vastly underutilized
- Interviewing dominates—and shorter interviews, at that
4. Qualitative evaluation is being used intentionally as an intervention; a high degree of process use
Because qualitative data gathering methods and reporting approaches can be made accessible to non-academics (a.k.a., ordinary people), and because engaging in qualitative evaluation involves learning in multiple ways at multiple levels, process use is increased through evaluation approaches that often incorporate some aspects of qualitative inquiry, such as:
- Participatory or collaborative evaluation
- Feminist evaluation
- Empowerment evaluation
- Transformative evaluation
- Developmental evaluation
These terms mean different things to different people, but share a commitment to involving the people in the setting being studied as co-inquirers, at least to some important extent, though the degree and nature of the involvement vary widely.
3. Increased value for deep contextual understanding enhances demand for and appreciation of qualitative evaluation
- The ascendance of realist evaluation has made contextual sensitivity paramount.
- “Contextual Intelligence” is being understood as a “A Critical Competency” (Matthew R. Kutz & Anita Bamford-Wade, 2013, E:CO)
- Evidence-based effective principles require contextual adaptation (as opposed to the high fidelity emphasis of best practice models)
2. The qualitative evaluator is the instrument; experience, expertise, and cultural competence matter
The focus in quantitative methods is on the validity and reliability of the data collection instruments and analytic procedures. In qualitative inquiry, the experiences and capabilities of the qualitative evaluator as a person remain central to credibility: who does the work matters. In this regard, the song lyric of country music legend Waylon Jennings is germane and insightful:
Old age and treachery
Always overcomes youth and skill
1. Increased purposeful sampling options
Purposeful sampling involves selecting information-rich cases to study, cases that by their nature and substance will illuminate the evaluation question being investigated. Perhaps nothing better illustrates the difference between quantitative and qualitative methods than the different logics that support sampling approaches. The third edition (2002) of Qualitative Research and Evaluation Methods discussed 16 purposeful sampling options. The new edition presents and discusses 35 options. This development of more nuanced and targeted purposeful sampling applications captures, for me, the key to the increased utility of qualitative evaluation methods over the last decade. In the end, whatever conclusions we draw and judgments we make depend on what we have sampled. To be more strategically purposeful about sampling is to be more strategically purposeful about evaluation.
Looking Ahead: Four Challenges and Opportunities
- Building qualitative inquiry capacity: Global efforts are underway to strengthen evaluation capacity. Strengthening quality inquiry capacity needs to be part of that effort.
- Increasing interest in and attracting resources to do serious, triangulated, in-depth qualitative, multi-method evaluations: As noted in the top ten trends, observation and in-depth fieldwork are underutilized. Interviewing and short site visits dominate.
- Deepening evaluators’ commitment to inquire seriously into unintended consequences and take emergence (complexity) seriously: Lip service and rhetoric give the appearance of attending to unintended consequences, but most evaluation designs devote the entire budget to assessing planned implementation and goal attainment. The kind of open-ended fieldwork needed to turn up actual consequences and emergent dynamics remains rare.
- Cumulative-longitudinal integration at the case and context levels: Long-term, in-depth case studies, with purposeful sampling that is sufficiently diverse to capture contextual variations, remains an ideal too rarely realized in practice.
