HOW DO WE KNOW IF IMPACT HAS OCCURRED?
HOW DO WE KNOWIF IMPACT HASOCCURRED?Understanding and usingevidence of impactimpactmanagementproject.com1 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
INTRODUCTIONContentsINTRODUCTION.2CONTEXT SETTING.3Why does evidence matter?.3What is evidence of impact?.3EXISTING EVIDENCE.4Collecting existing evidence.4Using existing evidence.4NEW EVIDENCE.5Collecting our own evidence .5MAXIMISING THE VALUE OF EVIDENCE.7Choosing the best methodology.7Sharing evidence.7The Impact Management Project is a collaborativeeffort by over 700 organisations, from differentcontext and countries, to agree on sharedfundamentals for how we talk about, measureand manage impact - and therefore our goals andperformance.Shared fundamentals for describing the effects thatdifferent businesses - or portfolios of businesses have on people and planet help different actors in avalue chain figure out what information they need tomanage those effects. Collecting evidence of impactis an important step towards figuring out whichmaterial positive and negative effects are occurring,before making decisions to improve the experienceof people and planet.Over the last six months, Nesta has been workingwith the Impact Management Project to explore howwe think about evidence of impact. This documentprovides guidance on both using existing evidence ofimpact, and building your own evidence base.This report is the product of a series of discussionsamong a wide range of practitioners as part of theImpact Management Project. The report has beenco-authored by Nesta and the Impact ManagementProject Team. Please direct any feedback or furtherenquiries about this report to:team@impactmanagementproject.comMORE INFORMATION.8AcknowledgementsThe Impact Management Project is deeplygrateful to all project participants, who providedexcellent insight into the topic of evidence and toNesta co-author Lucy Heady.2 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
CONTEXT SETTINGWhy does evidence matter?What is evidence of impact?The assertion that we should use evidenceto guide our decisions seems so obviousas to be hardly worth saying. Why wouldn’twe use what we already know to guide ourdecision-making? Scratch the surface,however, and things immediately seem lessclear. Different disciplines prioritise differenttypes of evidence. Techniques from one fieldmight be imported to a new context withoutappreciating important differences. And whendifferent types of evidence disagree, or there isvery little evidence that anything works, it canseem easier and cheaper to abandon evidencealtogether.We define ‘evidence of impact’ as:But without evidence we are doomed to repeatour mistakes. Ideas with strong intuitiveappeal, such as scaring children off crimeby having them spend time in prisons orputting teenagers off motherhood by givingthem artificial crying babies, turn out to beactively harmful1. Achieving social impact ishard; we must never assume that somethingworks because the logic is appealing. Bybuilding the evidence base, across contextsand with different approaches, and learningand adapting based on what we find, we canachieve positive impact, and are more likely toattract investment.“The available body of facts or information thatcan be used to judge to what extent (or not)impact has occurred.”2We collect lots of different types of informationthrough the impact management process,only some of which is used to understand theexperience of people and planet, and is therefore‘impact evidence’. Impact evidence helps usunderstand what impact has occurred, and enablesus to reduce our level of evidence risk.3When using evidence to make judgements we takeinto account how good the evidence is likely to be.There are many attributes which help us determinewhether we can have strong grounds for believinginformation about impact constitutes ‘good’evidence. Frameworks like Standards of Evidencehelp us assess what level of confidence we have,based on the type of evidence we collect.Using and collecting evidence of impactneed not require a PhD in social science or amillion-dollar budget. The norms below set outprinciples for interpreting existing evidenceand collecting new evidence in a cost-effectiveway. Wherever possible we have signposted tomore detailed resources that demonstrate thepractical application of these principles.1For example: see the evaluation of the juvenile delinquency prevention programme ‘Scared straight’2Based on Oxford English Dictionary definition, adapted to be specific to whether impact (positive or negative) has occurred.There are many other types of evidence used to make decisions within the impact management process - such as expertopinion - but these do not constitute evidence of impact33 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
EXISTING EVIDENCECollecting existing evidence When setting our impact goals and selecting ordesigning delivery models (e.g. a business orprogramme), we often draw on existing evidence.To do this we consider what available evidenceexists across all of the dimensions of impact.ACCURACYFor example, when evaluating an existingenterprise, perhaps as a prospective investor,we might consider how far data exists acrossall the dimensions (Figure 1, below) and whatthis assessment tells us about an enterprise’sability to meet a set of goals. This helps us workout where gaps, or weak evidence, exist in theexisting evidence base for one or more group ofpeople (or the planet) affected, and where newevidence may be needed. Does the evidence conform to establishedcriteria for quality and rigour?4 Is the evidence the product of external reviewor an independent assessment?5RELEVANCE Does the evidence directly correspond to thesame impact goals across all the dimensions ofimpact? Are there any gaps?When examining an existing evidence base, themore individual studies (or sets of information) thatdemonstrate these features and share the samefindings, the stronger the evidence base.6Using existing evidenceHowever, existing evidence will rarely tell useverything we need to know about the likely impactof a delivery model in a given context. We thereforeoften need to collect our own evidence to re-affirmwhat we know, or fill in evidence gaps. Resourcessuch as 3ie’s Evidence Gap Maps help us identifywhere these evidence gaps are likely to be.There are a number of considerations which helpassess whether existing evidence of impact maybe ‘good’ enough:COMPLETENESS Has data been collected from a range ofrelevant stakeholders?Is the data set complete and comprehensive?Figure 1: The dimensions of impactDATA1ANALYSISType of outcome(s)Importance of ivenegativeoutcome(s) outcome(s)outcome(s)WHAT2HOW MUCHSmall scaleLarge scaleShort-termLong-termSlowly3Demographic dataEnvironmental dataGeographic data4Benchmarked performanceacross who, what, how much5Risk factors,e.g. evidence riskQuicklyWell-servedUnderservedMuch worsethan what islikely tooccurMuch betterthan what islikely tooccurWHOCONTRIBUTIONLow riskHigh riskRISK4See: USAID data quality checklist5See: Nesta standards of evidence; Levels of evidence in Medicine; What Works?See: DFID’s ‘Assessing Strength of Evidence’ guide64 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
NEW EVIDENCECollecting our own evidenceIn some cases, the existing evidence of a linkbetween the activity and the outcome is sostrong that activity or output data can serveas a proxy for outcomes, for example thenumber of children receiving vaccinationsagainst measles. When selecting our ownindicators or using proxies, we are at riskof enforcing causality assumptions. Wherean enterprise is taking a new approach,or the evidence base linking outputs tooutcomes is inconclusive, non-substantiveor incomparable, these proxy measures willnot constitute good evidence of impact, eventhough they may provide useful evidence thatsome parts of a theory of change are valid.Collecting our own evidence better enablesus to substantiate or refute existing evidence,and further understand what impact is beingdelivered, in order to improve the experienceof people and planet. We may also need tounderstand whether this experience changesafter engagement with an enterprise hasended. All of the evidence we gather informs thedecisions we make about our strategy and thegoals we set and re-set.The type of evidence collected will vary accordingto what level of evidence is necessary andproportional for an enterprise to understandtheir effects on people and planet. There are twobroad dimensions along which we can think aboutproportionality: 1. What type of data we collect: the amount ofconfidence we have in our data is often relatedto the type of data we use. Direct measurement of change in effectsenables a greater level of certainty.For example, change in nutrition status orreduction in CO2 emissions.However, sometimes these measures arenot possible, very difficult or very expensiveto collect for every demographic andoutcome type. This could be due to thetimescale within which effects are realised,customer confidentiality, a lack of appropriatemeasurement techniques or simply cost. Inthese cases, it is common to collect data onproxies for the final effect, for example homeimprovements might be a proxy for increasedincomes or hospital waiting lists might be aproxy for a well-functioning health service. Activity or output data is often used as aproxy as it is typically much easier to collectthis type of data.For example, counting a child that hasrecieved a vaccination against measles as aproxy for that child not developing measles.5 IMPACT MANAGEMENT PROJECTWe also recognise that people and planetwill likely experience other impact too, bothpositive and negative and therefore draw onexisting information about the material effectsthat comparable approaches delivered incomparable contexts to help us judge othereffects we might want either to mitigate (ifnegative) or increase deliberately (if positive).For example, by consulting existing researchon the effects of offshore wind farms, wemight learn that some marine animalsexperience negative impact due to the noiseand regular disturbance. This informationhelps us to dedicate resource to trying tocollect new information to uncover andmitigate these negative effects. Feedback from people experiencing theeffects through a questionnaire or surveyingmethodology is the most direct and sometimesmost reliable indicator that any change ineffect is occurring. This method is mostreliable where there is existing evidence thatpeople’s feedback is strongly correlated withthe desired effect. This data may be a proxy(e.g. ‘did you take your medicine on time?’), ormay be the best metric to indicate what effectis occurring across any of the dimensions (e.g.‘how far has your quality of life improved?’is a good indicator of depth for a wellbeingoutcome).impactmanagementproject.com
NEW EVIDENCE cont’d2. How we collect it: the amount of confidencewe have in our data is also informed by themethodology we use. At a minimum, we collect data (directmeasures or proxies) before engagement withan enterprise and after that engagement.However, this approach still leaves potentialbias as any change detected is unlikely to becaused only by the engagement, for exampleif some people would have improved theirsituation anyway or if there is some otherengagement causing change.For example, maternal health clinics havetracked that 470 mothers received treatment in2008, 89% of whom safely delivered a healthychild and 97% of whom reported satisfactionwith the service. The greatest confidence is often obtainedthrough an external evaluation that removesselection bias, considers attribution and/oruncovers other effects.For example, an evaluation shows that, of allof the treatments available, mothers recievingsuplemental nutrition were most likely to givebirth to a healthy-weight child.Evaluations that use a control group, suchas Randomised Control Trials or PropensityScore Matching, help us establish causality.While innovative approaches are bringing thecosts of such techniques down, they do oftenstill require a higher financial investment andmore technical expertise.For example, a survey of a similar group ofmothers in a neighbouring region reported a65% safe, healthy delivery rate in the sameperiod, while the maternal mortality rate is43% below the regional average. Being clear about how context affects thedelivery of these effects enables consistentreplication.For example, there are multiple driversof positive maternal health outcomes –including access to good nutrition and asanitary delivery environment – and peoplein different contexts will be underserved indifferent ways in relation to this outcome.This will influence the effectiveness and/orapplicability of pre- and post-natal care.Standards of Evidence are helpful frameworksfor assessing the confidence we have in ourevidence based on the metholodgy used.Figure 2: An example Standards of Evidence8Puttick & Ludlow (2013) ‘Nesta: Standards of Evidence’. These Standards were based on the Maryland Scientific MethodsScale, and the Project Oracle Standards of Evidence.86 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
MAXIMISING THE VALUE OFEVIDENCEChoosing the best metholodogyIt is not always appropriate to use the mostresource-intensive and complex methodology forcollecting evidence. Considering our confidencein existing evidence, and the level of impact riskparties are willing to take, is a helpful startingpoint.It is often sensible to use less expensiveapproaches to test whether it is plausible thatimpact has occurred while a strategy is still beingdeveloped, and then invest in something morerigorous once there is more confidence that thestrategy is of sufficient quality.Collecting, analysing and communicatingevidence all come at a cost and so it is importantthat the evidence we collect is both useful andcollected in a cost-effective way.The following considerations can help us to collectuseful evidence:1. Is there a clear purpose for the evidence?This might be: improving the experience of people orplanet increasing publicly available evidence aboutwhich strategies work (or don’t) in relation toa specific population and/or outcome reducing risk and uncertainty to increase thelikelihood that impact is being delivered asexpected2. Is the value of the evidence likely to beproportionate to its cost? The most valuableevidence is likely to be:3. Is the methodology appropriate for the typeof evidence sought?9 Draw on the existingevidence base to learn about: gaps in existing evidence, and why theyexist likely risks involved (e.g. ethical, cultural orpractical) what data-collection methods are oftenused in this context4. Who will benefit most from the evidence?This should inform:the standards of evidence employed who carries the cost10Sharing evidenceSharing evidence maximises its value by enablingmore enterprises to learn and improve theirdelivery models. How well evidence is used isdirectly related to how it is shared.To maximise the value of evidence, we shouldadhere to the following principles when reportingevidence: TRANSPARENT: Reports should includewhat was tested, time-stamped results,methodology, data quality, context andstakeholders. TIMELY: Evidence should be shared as soonas possible to make it useful to others, wherepossible considering the timeline of decisionsthe data may influence. COMPREHENSIVE: Reports should makeclear how well the evidence can say whatwould have happened in the absence ofthe enterprise, report any negative findingsand any assumptions involved in arriving atestimates of impact. CONTEXTUALISED: New evidence shouldalways be presented in the context of otherexisting evidence and our original goals. Thisallows us to see where results differ and whatnew lessons have been learned. evidence that helps us manage our mostmaterial effects on people and planet focused on building the evidence base wherethere is greatest uncertainty See: IDInsight decision-focused evaluationWhere a resource-intensive methodology is being employed to answer questions that will help build the field, it is less likelythat it is appropriate for the enterprise to carry the cost alone.9107 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
MORE INFORMATIONSuggested resourcesIdentifying good evidence Bond principles for assessing the quality of evidence DFID Assessing the Strength of Evidence guide Nesta Standards of Evidence USAID Data Quality Checklist Evidence-Based Medicine resources Alliance for Useful Evidence Using Research Evidence ImpactMatters Impact Audit StandardFinding existing evidence Dartington Social Research Unit Beam Exchange Evidence Map 3ie Evidence Gap Maps What Works Centre papers Campbell Collaboration LibraryChoosing a methodology IDInsight Decision-Focused Evaluation CDI Review of evaluation methodologies IPA Research resources SSIR How Impact Investors Measure Impact OHRC Collecting Data Ethically8 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
This work is licensed under the Creative CommonsAttribution-NoDerivatives 4.0 InternationalLicense, that allows the copying and distribution ofthis material as long as no changes are made andcredit is given to the authors.The Impact Management Project, September 2017THE IMPACT MANAGEMENT PROJECT9 IMPACT MANAGEMENT PROJECTimpactmanagementproject.com
However, existing evidence will rarely tell us everything we need to know about the likely impact of a delivery model in a given context. We therefore often need to collect our own evidence to re-affirm what we know, or fill in evidence gaps. Resources such as 3ie’s Evidence Gap Maps help us identify where these evidence gaps are likely to be.
contractor's accounting system for accumulating and billing costs under government contracts. ̶ Aligns with DFARS criteria. ̶Involves testing of transactions. ̶If the previous post award accounting system audit is more than 12 months old, and/or the contractor's accounting system has changed, the auditor should perform a full audit.
1 Pamela Rao. Office of HIV/AIDS. USAID Concept Note . Everybody’s Business 2007 4 Know your Epidemic, Know your System, Know your Response various governmental and nongovernmental agencies, groups which have access to health care and the means by
Understand what an operating system is and the role it plays. Know and identify Windows 7 desktop components Know and apply how to open and explore the control panel. Know and apply how to personalize the desktop. Know and identify the components of a window. Know and apply how to manage multiple windows and multiple monitors. Know and apply how to change the date .
Understand what is an operating system and the role it plays. Know and identify Windows 7 desktop components Know and apply how to open and explore the control panel. Know and apply how to personalize the desktop. Know and identify the components of a window. Know and apply how to manage multiple windows and multiple monitors. Know and apply how to change the date .
machinery analysis concepts and to show the reader what is needed to perform an actual analysis on specific machinery. Rule 1 Know what you know and don’t pretend to know what you don’t know! Often, a situation arises where the answer is not obvious or not contained within the analysis data. At this point “I don’t know” is the best .
Know Coding and Billing Basics Have a current CPT manual or LCD. Have a current ICD-10 manual or access. Have a copy of the AHA ICD-10 Guidelines for 2016. Have a current HCPCS book. Know your bilateral indicator codes. Know your global days (zero, 10, 90). Know modifier rules. Know ICD-
Get out of your own way and be genuinely you “Faking it” makes everyone uncomfortable Learn and grow, don’t copy “You know what you know what you know” Don’t try to project or act in your interview (I know they try to coach otherwise) It’s OK to say, “I don’t know, but
Books in the Know Series Know the Creeds and Councils, Justin S. Holcomb Know the Heretics, Justin S. Holcomb Know Why You Believe, K. Scott Oliphint 9780310525974_KnowWhyBelieve_int_SC.indd 2
