Field Guide For Evaluation

Results and MeasurementField Guide forEvaluation:How to Develop anEffective Terms ofReferenceJune 2014

Field Guide for Evaluation:How to Develop an EffectiveTerms of ReferencePact, Inc.Washington, DCCopyright 2014 by Pact, Inc.This work is licensed under the Creative Commons Attribution-ShareAlike 4.0International License.To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/or send a letter to Creative Commons, 444 Castro Street, Suite 900, MountainView, California, 94041, USA.

NFPAUNICEFUSAIDVCTantenatal cliniccooperative agreementUnited States Centers for Disease Control and PreventionCatholic Relief Servicescounseling and testingenumeration areaexpression of interestexecutive summaryfocus group discussionfiscal yeargeographic information systemInternational Fund for Agricultural DevelopmentInternational Program for Development Evaluation Traininginternal review boardkey informant interviewmonitoring and evaluationmonitoring, evaluation, research, and learningmost significant changenongovernmental organizationOverseas Development InstituteOrganization for Economic Co-operation and Developmentorphans and vulnerable childrenparticipatory learning and actionperformance monitoring and evaluation planproject yearrandomized controlled trialSouth African Demographic and Health Surveysimple random sampletechnical assistanceterms of referenceUnited Nations Population FundUnited Nations Children’s Emergency FundUnited States Agency for International Developmentvoluntary counseling and testing [for HIV]

ForewordOver time, evaluation has become increasingly important at Pact. An updated evaluation policyreleased in 2014 guides our programs in the requirements and expectations for evaluation.Module 3: Field Guide for Evaluation: How to Develop an Effective Terms of Reference, finalizedin 2014 by our Results and Measurement team, provides a basic overview of evaluation conceptsand gives those who commission and manage evaluations an overview of best practices forplanning, implementing, and managing the overall process of an evaluation.A slide set accompanying the module provides an opportunity to engage in practical exercises totest the skills outlined in this text.How to Use This ModuleA practical guide to managing and leading evaluation efforts, this module is designed for use inevaluations within your organization and among your partners. Increasing local capacity toconduct and manage good program evaluation is a key goal of the Results and Measurementteam at Pact. The copyright under a Creative Commons agreement encourages you to use,remix, and adapt all Pact materials as you see fit, with attribution to Pact.Each chapter’s learning objectives and exercises relate to sections of Pact’s evaluation protocol,and each chapter builds on the previous one. By the end of the module, if you have worked yourway through the exercises, you should have a comprehensive, written plan for your evaluation—that is, a full terms of reference (TOR). In addition, a TOR template can be found in Appendix 1(page 90). The shaded boxes that introduce and give an overview of each chapter refer torelevant sections of this template.The closing pages of each chapter link you to the wealth of further resources available on theInternet. I encourage all practitioners to use these resources to keep your knowledge ofevaluation up to date and growing, in order to deepen your practice and bring your increasingskills to the programs serving the communities we care about.Kerry BruceSenior Director, Global Health and Measurementrm@pactworld.orgPAGE 4

AcknowledgementsThis module was inspired by Dr. Rita Sonko and her team in South Africa. Sensing a generalapprehension of managing and participating in evaluation, they developed, with their partners,a similar training module: Key Considerations for Managing Evaluations, Brief ReferenceGuide.The early draft of this manual was written by Mollie Williams, pulling material and structurefrom Dr. Sonko’s work and enriching it with addition of ideas from the literature on evaluationfrom both best practices and academia.Alison Koler and Jade Lamb both worked to finalize the content and format of the document,bringing it into its final shape and form. The slide set accompanying this module was developedby Alison Koler, Jade Lamb, and Joris Vandelanotte; Samuel Kudhlande from Pact Swazilandprovided review and feedback on these slides. This curriculum was pilot tested with PactSwaziland, its partners, and staff from Pact Tanzania and Pact Zimbabwe; valuable feedbackfrom this pilot has been incorporated into this current document.Copyediting and formatting expertise was provided by Karen Cure.PAGE 5

ContentsAcronyms. 3Foreword . 4Acknowledgements .5Contents . 6Tables .10Figures .10Chapter 1: What Is Evaluation? . 11Defining Evaluation . 11Systematic . 12Specific . 12Versatile. 12Evaluation versus Research . 12Why Evaluate? . 13Types of Evaluation. 13Internal and External Evaluation . 14Involving Stakeholders in Evaluation. 14Participatory Evaluation . 15Common Pitfalls . 15Monitoring and Evaluation. 15Barriers to Evaluation . 16Summary . 16Resources . 17EXERCISES . 18#1: Why Should Your Organization Invest in Evaluation? . 18#2: Barriers To Evaluation . 18Chapter 2: Evaluation Purpose and Questions . 19The Terms of Reference . 19Focusing the Evaluation . 20Steps to Focus the Evaluation . 20Logic Models . 20Key Assumptions . 22Stakeholder Engagement . 22Writing an Evaluation Purpose Statement . 23Evaluation Questions . 23Evaluation Questions Versus Evaluation Purpose . 23PAGE 6

Steps to Develop Evaluation Questions . 24Types of Evaluation Questions . 24Descriptive Questions . 24Normative Questions .25Cause–Effect Questions .25An Optimal Mix .25Summary . 26Resources . 26EXERCISES .27#3: Stakeholder Analysis.27#4: Writing Evaluation Purpose Statement .27#5: Prioritizing Evaluation Questions.27Prioritizing Evaluation Questions . 28Chapter 3: Overview of Evaluation Design and Methods . 29What is Evaluation Design? . 29Overview of Approaches . 29Quantitative Approaches . 29Experimental Design . 30Quasi-experimental Design . 30Nonexperimental/Observational Design . 32Qualitative Approaches. 32Mixed Approaches . 33Choosing a Design. 34What Is Good Basic Design? .35Designing Survey Instruments and Interview Guides . 36Summary . 36Resources . 36EXERCISES . 38#6: Developing a Planning Matrix . 38Template for an Evaluation Planning Matrix . 39Chapter 4: Data Sources and Collection Methods. 40Considering Data Collection Methods . 40Existing Records . 40Additional Information on Using Existing Data . 41Surveys . 41Additional Information on Developing and Conducting Surveys . 41Direct Measurement . 42Additional Information on Direct Measurements . 42Observation . 42Additional Information on Collecting Data by Observation . 42Key-Informant Interviews . 43Additional Information on Conducting Key Informant Interviews . 43PAGE 7

Focus Group Discussions . 43Additional Information on Conducting Focus Group Discussions . 44Most Significant Change . 44Additional Information on Conducting Most Significant Change Evaluation. 44Outcome Mapping .45Additional Information on Conducting Outcome Mapping .45Mapping .45Participatory Mapping .45Transect Walk.45Additional Information on Mapping. 46Selecting the Right Approach and Methods .47Ethical Review .47Summary .47Resources . 48EXERCISE . 49#7: Evaluation Design via Round Robin Conversations . 49Chapter 5: Sampling . 50Why Sampling Matters . 50Units of Analysis . 50Definition . 51Probability Sampling . 51Sampling Frames. 51Sample Size .52Sampling Methods .52Common Pitfalls to Avoid in Probability Sampling .55Nonprobability Sampling Techniques .55Purposive Sampling .55Sample Size for Nonprobability Sampling .56Choosing a Sampling Strategy .56Summary . 57Resources . 57EXERCISE . 58#8: Choosing an Appropriate Sample . 58Chapter 6: Basic Data Analysis.59What Is Data Analysis? .59Planning for Data Analysis .59Analyzing Quantitative Data . 60Prepare the Quantitative Data . 60Conduct the Analysis . 61Present the Quantitative Data . 63Analyzing Qualitative Data . 64Prepare the Qualitative Data . 64PAGE 8

Code and Analyze the Qualitative Data . 64Present the Qualitative Data .65Summary .65Resources .65EXERCISES . 66#9: Quantitative Data Analysis . 66#10: Qualitative Data Analysis . 66Chapter 7: Using and Communicating

The early draft of this manual was written by Mollie Williams, pulling material and structure from Dr. Sonko’s work and enriching it with addition of ideas from the literature on evaluation from both best practices and academia. Alison Koler and Jade Lamb both worked to finalize the content and format of the document,