Qualitative Research Methods In Program Evaluation .

1y ago
654.96 KB
46 Pages
Last View : 6d ago
Last Download : 10m ago
Upload by : Mara Blakely

QUALITATIVE RESEARCH METHODS IN PROGRAM EVALUATION:CONSIDERATIONS FOR FEDERAL STAFFOffice of Data, Analysis, Research & EvaluationAdministration on Children, Youth & FamiliesMay 2016

Table of ContentsEXECUTIVE SUMMARY . iINTRODUCTION . 1PART ONE: DECIDING WHEN TO USE QUALITATIVE METHODS. 3What are Qualitative Methods? . 3The Role of Qualitative Methods in Program Evaluation . 4TEXT BOX 1: The Role of Qualitative Methods in Program Evaluation . 6Summary . 7PART TWO: GETTING STARTED . 8Researcher Experience . 8Questions to ask about researcher credibility . 9Budgeting and Time Planning . 9Questions to ask about time and cost estimates . 11Summary . 11PART THREE: A BRIEF PRIMER ON QUALITATIVE METHODS . 12Research Design: The Structure of the Study . 12Questions to ask about research design . 13Sampling: What Gets Measured . 14Sampling Strategies in Qualitative Research . 14Sample Size. 15Questions to ask about sampling. 16Qualitative Data Collection: Methods and Other Considerations. 16Interviews and Focus Groups. 17Asking Open-Ended Questions . 18Capturing Interview Data . 19Qualitative Observation . 19Document Review . 20On-Site Data Collection and Researcher Bias . 20A Few Words About Data Collection By Program Staff . 22Questions to ask about data collection . 23Qualitative Data Analysis . 24TEXT BOX 4 Overview of the Qualitative Data Analysis Procedures . 25Inter-Rater Reliability . 26Questions to ask about qualitative data analysis . 26

Summary . 27PART FOUR: THE CREDIBILITY OF QUALITATIVE FINDINGS . 28Internal Validity: Ruling Out Alternative Explanations . 28Rival Conclusions . 29Contrasting and Disconfirming/Negative Cases . 30Triangulation . 31Getting Feedback from Participants/Informants . 32Questions to ask about internal validity . 32External Validity: Generalizing the Findings . 33Sample-to-population extrapolation. 34Case-to-case transfer . 35Analytic generalization . 35Summary . 36Questions to ask about generalizability . 36CONCLUSION . 37REFERENCES . 38

EXECUTIVE SUMMARYQualitative research methods can play a powerful role in program evaluation, but theyfrequently are misunderstood and poorly implemented, giving rise to the idea that they are justnot as rigorous and credible as quantitative methods. That is not true, but qualitative methods areless standardized than are quantitative methods, and that makes determining their appropriate useand assessing their quality more difficult for federal staff overseeing program evaluation projectsthat include a qualitative component.This document was written to support federal program officers, particularly those whohave not had formal research training, as they develop and oversee projects that include programevaluations that use qualitative methods. Divided into four parts, this paper begins with thedecision to include qualitative methods (or not) in a contemplated project. This first sectiondefines qualitative methods, distinguishes them from quantitative research methods, and outlinesthe roles qualitative approaches can play in program evaluation. This portion of the documentwill be particularly useful as a project or a funding announcement is being developed.The second section of the paper takes up a couple of early considerations, once thedecision to incorporate qualitative methods has been made: How to identify an evaluator with therequisite experience in these methods and addressing some time and cost considerations that areparticular to qualitative research. These points will be helpful both in drafting the fundingannouncement and in reviewing applications. The third section of the paper provides a briefoverview of qualitative research methods addressing research design, sampling, data collection,and data analysis. The idea is to provide some familiarity with concepts and terminology thatwill help federal staff communicate more effectively with evaluators as the project progresses;this section also may be of use in evaluating proposals that include program evaluation.Finally, the document takes up how to assess the credibility of qualitative findings andconclusions, addressing both internal and external validity. Although the goal of obtainingcredible findings drives any research project from its inception, it may be easier to follow thediscussion of credibility if the reader has received some grounding in qualitative researchmethods. Throughout this document are questions federal staff can ask themselves or theirevaluators to aid decision-making as well as to press evaluators to be explicit about theirmethodological choices, time-planning, and budgeting.

INTRODUCTIONThis document has been prepared for staff charged with overseeing federally-fundedprogram evaluations that include the collection and analysis of qualitative data. The informationis organized and presented in four parts to allow you to get quickly to the information you need,depending on your level of expertise and/or the project's stage of work — developing the fundingannouncement, reviewing proposals or evaluation plans, overseeing the work as it progresses,and approving final reports and other products.Not all federal staff have formal research training; the intent of this document is to equipyou with a basic knowledge of qualitative methods and when they may be most useful, how todetermine the quality of the findings, and to help you consider the trade-offs involved in decidingwhen to use these methods. It also should help dispel the oft-expressed idea that qualitativemethods are not as rigorous as quantitative research. That simply is not true, although it hasbeen the case the qualitative research has not always been well-executed, and poor executioncertainly compromises the utility and credibility of the findings. As noted in ACF’s EvaluationPolicy, “Rigor is not restricted to impact evaluations, but is also necessary in implementation orprocess evaluations, descriptive studies, outcome evaluations, and formative evaluations; and inboth qualitative and quantitative approaches.”1 This document will give you some grounding inwhat good qualitative research looks like and how much careful, rigorous work goes into a wellexecuted project. This document also suggests questions you can ask of evaluators to help youin your decision-making, as well as to press them to be explicit about their methodologicalchoices and to deliver a usable, credible product.The document is organized as follows: Part One lays the foundation by defining exactlywhat qualitative methods are, the kinds of questions that can best be addressed by this approach,and the role qualitative methods can play in program evaluation. Just as quantitative methods arewell suited for some types of questions, qualitative methods are particularly well suited to other,specific types of questions. Therefore, the first step is to consider what questions the study needsto answer and whether qualitative methods are appropriate for answering them. The informationin this section should be helpful for deciding whether or not to include a request for qualitativeresearch in a funding announcement, as well as for wording the announcement and reviewingproposals.Part Two takes up a couple of early-stage steps, once the decision to incorporatequalitative methods has been made: the importance of engaging an evaluator with specificexperience in using qualitative methods and a consideration of the time and budget implicationsof the decision to use these approaches. This information should be of use for the developmentof the funding announcement and in reviewing applications.Part Three provides a high-level overview of qualitative research methods, includingresearch design, sampling, data collection, and data analysis. It also covers methodologicalconsiderations attendant upon research fieldwork: researcher bias and data collection byprogram staff. This section will familiarize you with terminology and concepts, as well asprovide a sense of what, exactly, researchers do when they collect and analyze qualitative data.The goal is to equip you to communicate effectively with evaluators and project directors. This1Administration for Children and Families, Evaluation Policy. /acf-evaluation-policy1

section should be useful at the proposal review stage, when approving evaluation plans, and inmonitoring work as it progresses.Part Four dives a bit deeper into method — particularly analysis and interpretation — todiscuss what determines the validity and credibility of qualitative findings and conclusions. Thegoal of generating valid, credible findings should guide the project at all stages of the work, sothe presence of this section at the end of the document in no way suggests that it applies only atthe conclusion of a project. Not at all. From the selection of an experienced investigator, thoughthe implementation of an appropriate research design and sampling strategy, and the carefulanalysis of the data, credibility and validity should be the project's guide-stars. That said, thisfinal section will also be useful during the later stages of the project for thinking about thecredibility of the findings and how the conclusions can be extended to other settings. However,it is too late in the game to find out, only as the final report is written, that the study's findingscannot be relied upon. Therefore, looking this section over, even as a project gets started, willhelp you to start thinking about how to ensure a quality product.2

PART ONE: DECIDING WHEN TO USE QUALITATIVE METHODSReferences to program evaluation methods frequently include the phrase "qualitative andquantitative methods," as if the mention of one method demands the inclusion of the other.Although methodological diversity in evaluation is widely accepted, and even recommended bysome observers, it remains necessary to consider carefully the goals of any given programevaluation and to select the approach most suitable for answering the questions at hand, ratherthan reflexively calling for both (Patton, 2002; Schorr & Farrow, 2011). Depending on theresearch questions to be answered, the best approach may be quantitative, qualitative, or somecombination of the two.What follows is intended to help you decide what benefits, if any, qualitative data canprovide to a given project. Even though qualitative data often are collected under less structuredresearch designs and on a comparatively small sample of subjects, the enormous amount of datagenerated — and the time and expertise needed to collect, organize, and analyze those data —means that qualitative studies are at least as expensive, and can be more costly, than quantitativestudies (Morse, 2003).2 Therefore, it is essential to be clear about when qualitative techniquesare called for and to be prepared to fund the project adequately to ensure credible, high qualitydata.The following discussion begins by defining exactly what qualitative methods are andhow they differ from quantitative research. Next, the particular strengths of qualitative methodswithin program evaluation are discussed.What are Qualitative Methods?Let us start by clarifying exactly what qualitative methods are — and what they are not.The broad term "methods" is used to apply to the collection, analysis, interpretation, andpresentation of research data. This brief will address methods used with qualitative data as thesediffer from those used for quantitative data. Typically gathered in the field, that is, the settingbeing studied, qualitative data used for program evaluation are obtained from three sources(Patton, 2002): In-depth interviews that use open-ended questions: "Interviews" include both one-onone interviews and focus groups. "Open-ended" questions are those that allow theinformants to express responses using their own words. These questions may beembedded in interviews that are structured, unstructured, or semi-structured; the openendedness is what makes the interview qualitative in nature.32The next section of this document touches briefly on time and budget matters related to qualitative methods.It is important not to conflate question type with interview structure: Structured, semi-structured, and unstructuredinterviews, which are defined in the data collection section, are characterized by how rigidly the interviewer has toadhere to a pre-defined interview protocol. These terms do not apply to the type of questions (i.e., open-ended orforced choice) included in those interviews. That said, qualitative interviews, structured or otherwise, may include alimited number of forced-choice questions.33

Direct observation yields detailed descriptions of the activities, actions, and behaviors ofindividuals; interpersonal interactions; settings; and organizational processes andprocedures.Document analysis may include the full range of organizational, programmatic, orclinical records, including public reports, memoranda, policy documents,correspondence, and the like.Quantitative methods may also use some of these data collection approaches; thedifference between quantitative and qualitative is in how the data are captured and expressed. Inquantitative research, data are expressed numerically. In contrast, qualitative data most often arein the form of words — interview or focus group transcripts, observational field notes, orexcerpts from documents (Miles & Huberman, 1994; National Research Council and Institute ofMedicine, 2002). Analysis of such data consists of extracting themes, patterns, categories, andcase examples (Patton, 2002; Hood, 2006). The purpose of qualitative analysis is to understandhow people involved with the program being studied understand, think about, make sense of, andmanage situations in their lives and environment and/or to describe the social or environmentalcontexts within which a program is implemented.Despite the differences between qualitative and quantitative methods, the line betweenthe two is less distinct than it may seem. For example, although the presentation of qualitativefindings relies primarily on words and focuses on patterns and themes, quantitative concepts alsomay be expressed. The presentation of qualitative findings, in addition to in-text descriptions,may include counts of how many respondents expressed a particular theme, sometimes presentedin a table or matrix, and the report text often will include words like "often" or "rarely," whichexpress quantitative concepts (Secrest & Sidani, 1995). However, quantitative terms andconcepts serve mainly to organize and summarize qualitative findings. The focus on detaileddescription expressed in words and analyzed for meaning is what characterizes qualitativeresearch methods.The Role of Qualitative Methods in Program EvaluationIn contrast to quantitative methods, which ask variations of "how much/many" questions,qualitative methods focus more on "how" and "why" types of questions (James Bell Associates,2009). Qualitative inquiry places a priority on people's lived experience and the meanings theyascribe to their experiences (Miles & Huberman, 1994). Data often are collected in the settingsunder study, and they aim for rich description of complex ideas or processes, albeit typicallyacross a limited number of individuals or settings. This approach stands in contrast toquantitative methods, which explore variables that can be captured or represented in numericalform, often across large samples and/or multiple points in time.Although qualitative methods may be used in both formative and summative evaluations,as a practical matter, they tend to be more heavily relied upon in formative evaluations.Summative evaluations — that is, t

what qualitative methods are, the kinds of questions that can best be addressed by this approach, and the role qualitative methods can play in program evaluation. Just as quantitative methods are well suited for some types of questions, qualitative methods are particularly well suited to other, specific types of questions.

Related Documents:

1. Explain what qualitative methods can add to program evaluation and identify situations/reasons when qualitative methods may be appropriate. 2. Identify different types of qualitative evaluation data collection and analysis and list steps involved in doing them. 3. Apply best-practices for qualitative methods in relation to program evaluation.

ter on qualitative research methods (Morrow & Smith, 2000), a comprehensive introduction to qualitative research methods that drew from the larger body of qualitative methodological literature, particularly in education. Given the diversity and comprehensiveness of the qualitative writings in education and the location of many counseling

qualitative data. (Note that pure qualitative research will follow all of the paradigm characteristics of qualitative research shown in the right column of Table 2.1.) Mixed research – research that involves the mixing of quantitative and qualitative methods or paradigm characteristics. The mixing of

The relationship between qualitative, quantitative and mixed methods research. The importance of the research question in an analysis. The need for methodological rigour in qualitative research. 1.1 Qualitative, Quantitative – A Few Clarifications What do the terms ‘qualitative data’ and ‘quantitative data’ mean? While the

Qualitative Analysis of Anions 1 Experiment 10 Qualitative Analysis of Anions Pre-Lab Assignment Before coming to lab: Read the lab thoroughly. Answer the pre-lab questions that appear at the end of this lab exercise. The questions should be answered on a separate (new) page of File Size: 343KBPage Count: 16Explore further(PDF) Experiment Report: Analysis of Anions and Cations .www.academia.eduExperiment 7 Qualitative Analysis: Anionswww.csus.eduLab Experiment #8: Qualitative Analysis of Common Anions .www.youtube.comQualitative Analysis of Anions - Odinitywww.odinity.comLab 13 Qualitative Analysis of Cations and Anionsdoctortang.comRecommended to you b

methods are used more often than qualitative methods in criminology and criminal justice. Importantly, quantitative and qualitative methods differ in several ways. The present study contributes to the literature by presenting a theoretical treatment of quantitative and qualitative research. The study begins by presenting quantitative and .

Research Practice Guide. 2 Code for America’s Qualitative Research Practice Guide is a statement from our qualitative research team of how we approach qualitative research, why we believe research is critical to the effective delivery of government services, and how you can engage with our research practice.

Quantitative and Qualitative Research Methods . might depend on the qualitative methods used in ethnographic research. Some re - search questions might require the application of different research methods. This approach is called a mixed-method approach (or triangulation).

Qualitative Research Design January 2015 Margaret R. Roller such as social media and mobile, as well as the evolving stature of Qualitative Research Design: Selected Articles from Research Design Review . The Sage handbook of qualitative research. T

"Qualitative Research Methods" (2020). Certificate Participants who attend all workshop sessions will receive an electronic Certificate of Participation. Interactive Teaching Format "The teaching / learning methods were the most valuable - they kept me engaged the entire time." WORKSHOP PARTICIPANT, UK. "The teaching method was

Qualitative Research from Start to Finish SECOND EDITION Robert K. Yin THE GUILFORD PRESS New York London . Brief Contents PUT 1 Understanding Qualitative Research CHAPTER 1 What Is Qualitative Research— 3 and Why Might You Consider Doing Such Research?

lished nursing and midwifery research where different research approaches have been used. Chapter 3 discusses methods for collecting qualitative data, including sampling strategies and methods for data analysis. Chapter 4 con-cludes the section on qualitative research, considering issues of quality and ethics in qualitative research.

2 qualitative-quantitative research but also under pressure to consider mixed methods—a potential way to think about integrating both paradigms (or sets of methods) within a study. Chapter 1 includes the history of qualitative, quantitative, and mixed methods research the typical purposes and

Spotlight on Women's Health Qualitative Methods in Rapid Turn-Around Health Services Research Alison B. Hamilton, PhD, MPH Research Health Scientist & Lead, Qualitative Methods Group HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy VA Greater Los Angeles Healthcare System . December 11, 2013

Qualitative Analysis of Group I Cations – The Silver Group 2 temperature of the solution increases from 20 C to 100 C. The solubilities of AgCl and Hg 2Cl 2 increases very little over this temperature range. Thus, PbCl 2 can be separated from the other two chlorides by adding hot water.File Size: 287KBPage Count: 9Explore furtherExperiment 12: Qualitative Analysis of Cationswww.bc.edu6: Qualitative Analysis of Group I Ions (Experiment .chem.libretexts.org18.9: Qualitative Cation Analysis - Chemistry LibreTextschem.libretexts.orgSeparation and identification of cationswww.periodni.comOxidation numbers calculatorwww.periodni.comRecommended to you b

Philosophical Assumptions Underlying Qualitative Paradigms 14 Ontology 15 Epistemology 16 . The Role of Ethics in Qualitative Research 49 Summary 52. viii Contents ' PART II: MAJOR QUALITATIVE TRADITIONS IN LEADERSHIP RESEARCH 55 . Contributions to the Study of Le

and qualitative research and methods in a single study to understand a research problem. To utilize this design effectively, you must understand both quantitative and qualitative research. Philosophical Approaches Creswell , J. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4thed.).

Handbook of qualitative research methods in marketing/[edited by] Russell W. Belk. p. cm. Includes bibliographical references and index. 1. Marketing research—Methodology. 2. Consumers— Research—Methodology. 3. Qualitative research—Methodology. I. Belk, Russell W. HF5415.2.H288 20

Appendix An Annotated Bibliography of Qualitative Research Methods Resources 339 Saldaña, J., & Omasta, M. (2018). Qualitative research: Analyzing life. Thousand Oaks, CA: Sage. An introductory textbook that utilizes data a

IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. 33, NO. 9, SEPTEMBER 2014 1277 Placement for Binary-Weighted Capacitive Array in SAR ADC Using Multiple Weighting Methods Yongfu Li, Student Member, IEEE, Zhe Zhang, Student Member, IEEE, Dingjuan Chua, Student Member, IEEE, and Yong Lian, Fellow, IEEE Abstract—The overall accuracy and linearity of a matching .