Evidence Synthesis For Policy - Royal Society

CHAPTER ONEPolicy questions arerarely answered by asingle study, or evenby a single discipline.How does evidence synthesis benefit policy?Synthesis techniques draw together alarge amount of information and turn it intoaccessible, usable knowledge that benefitssociety. If made freely available through openaccess publication, that knowledge becomesa global public good, furthering insight andpromoting evidence-informed policy anddebate across national borders.Every year there are over two million newacademic publications18. However, primaryresearch can be difficult to navigate, even foracademics, and it is generally inaccessibleto those outside academia. Moreover,policy questions are rarely answered by asingle study, or even by a single discipline.Decision-making and public debate arebest served if policymakers have accessto all the relevant evidence relating to aparticular issue. This involves an importantstep – evidence synthesis – between researchbeing conducted and decisions being taken.Indeed, an accurate, concise and unbiasedsynthesis of the evidence is arguably one ofthe most valuable contributions the academiccommunity can offer policymakers19.Policy options can be viewed through multiplelenses. For example, different stakeholders mayhold different personal and political values, thepolicy objectives themselves may be contested,and there may be questions about the extentto which an ‘ideal’ solution can actually bedelivered on the ground20,21,22. In such complexand contested situations, evidence synthesiscan help ensure that debate and decisions areinformed by the current best evidence.While this document focuses on evidencesynthesis for policy, synthesis is valuablein many different spheres of public life.Recognising this, the Royal Society hasbegun publishing judicial ‘primers’23 toassist the judiciary in their understandingof scientific topics and to ensure the bestscientific evidence is available to the courts.The Academy of Medical Sciences hasrecommended further development and use of‘NHS Choices’24,25 as a source of synthesisedevidence for patients making individualdecisions around medical treatments.18. ee Jinha, AE. 2010 Article 50 Million: An estimate of the number of scholarly articles in existence. LearnedSPublishing 23, 258-263, and around 2m new articles per year are published on Scopus. See Scopus: Access anduse Support Center https://service.elsevier.com/app/answers/detail/a id/11274/kw/articles/c/10547/supporthub/scopus/ (accessed 3 March 2018).19.Whitty CJM. 2015 What makes an academic paper useful for health policy? BMC Medicine 13:301.20. F or an accessible discussion piece on the role of opinions in decision-making see Sutherland WJ, Burgman M. 2015Policy Advice: Use experts wisely. Nature 526, 317-318.21. he UK government regularly conducts feasibility studies to inform policymaking – for an example see TruckTPlatooning: UK road trial feasibility study, latooning-uk-road-trialfeasibility-study (accessed 3 March 2018).22. I nnovation: managing risk, not avoiding it. Government Chief Scientific Adviser’s annual report 2014.See ion-managing-risk-not-avoiding-it (accessed 3 March 2018).23. T he Royal Society: Science and the law. See ce-and-law(accessed 3 March 2018).24. T he Academy of Medical Sciences. 2017 Enhancing the use of scientific evidence to judge the potential benefits andharms of medicines. See https://acmedsci.ac.uk/file-download/44970096 (accessed 3 March 2018).25. NHS choices. See https://www.nhs.uk/pages/home.aspx (accessed 3 March 2018).8EVIDENCE SYNTHESIS FOR POLICY

CHAPTER ONEWhat are the current challenges?Despite examples of good practice andsuccessful working relationships betweenevidence synthesis providers, brokersand policymakers, there remain significantchallenges with both the supply of, anddemand for, evidence synthesis26.Although ‘supply’ and ‘demand’ are usefulorganising concepts, the reality is more ofa spectrum than a dichotomy. Supply-sideand demand-side challenges can reinforceone another, and the means of overcomingthem (outlined in chapter 3) generally requirecollaboration and co-production ratherthan action solely by synthesis providers orsynthesis users.SupplySome academic disciplines lack a cultureof evidence synthesis, which can mean thatsynthesis is either not done at all or done inways that introduce bias.Even where a culture of evidence synthesisexists, synthesis relies on the availability ofhigh-quality primary research relevant to thepolicy question. Gaps in the primary researchand poor-quality evidence both severely limitthe utility of the resulting synthesised evidence.In addition, the UK Research ExcellenceFramework (REF)27 and research fundingsystem primarily value and reward originalresearch, which can deter researchers fromundertaking seemingly low-prestige synthesiseven where good primary research exists.Another challenge is that synthesisedevidence is often full of technical jargon andpresented in a style that is inaccessible topolicymakers. Evidence will never be theonly factor being considered by policymakersand synthesised evidence will be quicklydiscarded if it is deemed irrelevant orinaccessible. Compounding this is the factthat some synthesis articles are currentlyhidden behind journal pay walls or are lostamong the myriad of other publications. Suchlimited accessibility can mean that even ifpolicymakers are motivated to seek out anduse synthesised evidence this is not always astraightforward task.Even when synthesisedevidence is freelyavailable and writtenin plain language,it may not be availablewithin policy-relevanttimeframes.Even when synthesised evidence is freelyavailable and written in plain language, itmay not be available within policy-relevanttimeframes. While rapid synthesis carries ahigher risk of bias, in public policy it may bethat an 80% complete synthesis providedbefore a decision is made is considerablymore valuable than a more comprehensiveversion that arrives the day after, providedthe limitations imposed by doing it quicklyare made clear28. In the long run, habituallysynthesising evidence to provide answers toenduring questions could reduce the needfor more rapid approaches which carry agreater risk of bias.26. For further discussion of these challenges see op. cit. note 19.27. REF (Research Excellence Framework) 2021. See http://www.ref.ac.uk/ (accessed 3 March 2018).28. Op. cit note 19.EVIDENCE SYNTHESIS FOR POLICY9

CHAPTER ONECASE STUDY 3 fotoVoyager.Defra Evidence StatementsDefra’s29 Evidence Statements are neutral,succinct summaries of evidence on adefined topic of policy relevance. Thestatements are produced over a period ofthree months in collaboration with the policycustomer. The topics covered are guided byDefra’s research priorities, as outlined in thedepartmental Areas of Research Interest30.Policy officials work in a fast-paced andresource-limited environment. Pre-prepared,neutral Evidence Statements are an efficientway to keep officials and ministers informedof the state of the current evidence.By producing these statements in-house,the department’s existing data and scientificevidence can be considered alongsidethose from external sources.A consistent method for producing andupdating the Evidence Statements overtime ensures rigour, accessibility andconfidence in the findings. Each statementis completed according to a pre-definedmethodology, adapted from the systematicreview methods used in healthcare.The statements demonstrate how agovernment department can build anin-house repository of synthesised evidencerelevant to its policy needs.29. D epartment of Environment, Food and Rural Affairs ment-forenvironment-food-rural-affairs (accessed 11 May, 2018).30. U K Government. Defra group areas of research interest. See roup-areas-of-research-interest (accessed 3 March 2018). 10EVIDENCE SYNTHESIS FOR POLICY

CHAPTER ONEDemandPolicymakers can be unwilling or unable totake account of existing evidence for a varietyof reasons.In some government departments and publicsector bodies, particularly where the profileof evidence synthesis is low, policymakersmay not be aware of the breadth of availableevidence on a topic, or of the most up-todate techniques for combining and criticallyappraising this evidence.Time pressures, competing policy priorities,limited internal skills and external networkscan also contribute to synthesis beingoverlooked. In addition, lack of communicationand understanding between governmentdepartments and the research community cancreate an unintended disconnect betweenthe questions policymakers are dealing withand the availability of research that has thepotential to provide insight.31.Why now?In recent months the Royal Society and Academyof Medical Sciences have become aware ofincreasing, and increasingly high-profile, callsfor evidence synthesis to inform policy. At ajoint Royal Society and Defra conference inMarch 2017 31 (see case study 3) Mark Walport,the then Government Chief Scientific Adviser(GCSA), described how policymakers oftenneed succinct summaries of the entire evidencebase on a topic, and that the main challenge forresearchers and brokerage organisations32 ismaking these summaries available in a usefultimeframe. Acting GCSA, Chris Whitty, reinforcedthis message at the academies’ first evidencesynthesis meeting in July 2017.In September 2016, the Academy of MedicalSciences published a report on Improving thehealth of the public by 2040, which discussedthe role of evidence synthesis in improvingpublic health interventions33. This issue wasdiscussed further in a report on how we canbest use scientific evidence, published inJune 2017, which focused on synthesis tosupport decision-making in medicine butincluded recommendations with implicationsacross the sciences34. he Royal Society and the Department for the Environment, Food & Rural Affairs. 2017 Science for Defra: excellenceTin the application of evidence. See https://royalsociety.org/ onference-report.pdf (accessed 3 March 2018).32. By ‘brokerage organisations’ we mean organisations and initiatives that connect policymakers with researchers.33. T he Academy of Medical Sciences. 2016 Improving the health of the public by 2040. See https://acmedsci.ac.uk/download?f file&i 37428 (accessed 3 March 2018).34. T he Academy of Medical Sciences: How can we all best use scientific evidence? See se-evidence (accessed 3 March 2018).EVIDENCE SYNTHESIS FOR POLICY11

CHAPTER ONEMore information isavailable digitally thanbefore, and more of itis open access. Thiscreates both risksand opportunities.The Areas of Research Interest (ARIs)35 nowpublished by several government departmentssummarise departmental research needs,highlighting the topics on which synthesisedevidence would be most welcome. The ARIsprovide a useful resource for academicsand others, and a useful starting point forgreater collaboration between governmentdepartments and external researchers.The establishment of UK Research andInnovation (UKRI)36 also provides a potentialopportunity to put in place mechanisms tosupport evidence synthesis as an importantcomplement to primary research. Similarly,discussions on the shape of future ResearchExcellence Frameworks (REFs)37 shouldexplore how best to recognise and incentiviseexcellence in both subject specific andinterdisciplinary evidence synthesis.Another prompt is that more information isavailable digitally than before, and more of itis open access. This creates both risks andopportunities.Digital technologies and social media platformscan mean that false information circulatesand gains traction rapidly, and that unusualor exceptional cases receive unwarrantedattention. There is a risk that public debate andpolicy decisions are based on concerns fromthe electorate inspired by misrepresented oroverhyped information. The consequencesof this have been reported in numerousmedical debates, including prescription criteriafor statins, and associations between theMeasles, Mumps and Rubella (MMR) vaccineand autism38,39. In this context, it is increasinglyimportant that policy and debate are informedby the current best evidence and that, wherepossible, this evidence is available to all.Digital information and technologies also bringthe opportunity to increase researchers’ abilityto synthesise evidence. Machine learningand automation could potentially improvethe speed, breadth and rigour of evidencesynthesis. For example, by carefully automatingaspects of the synthesis process such as theidentification of relevant articles, it could bepossible to achieve a more thorough synthesiswithin shorter policy-relevant timeframes.Finally, some forms of synthesis may becomeincreasingly valuable to the conduct of researchitself. With more researchers and more articlespublished than before, the challenge of enablingnew research to build on what has alreadybeen carried out is increasingly acute. In anydiscipline, establishing what is already known(through rigorous synthesis) before undertakingnew research should be a fundamental aspectof the research cycle. There is a continuing needfor funders to base their funding decisions, andresearchers their proposals, on proven evidencegaps; for research to be conducted and reportedin a way which accumulates bodies of evidence;and for editors to recognise the importance ofsynthesising evidence alongside the importanceof seeking novelty.35. U K Government. Collection: Areas of Research Interest. See research-interest (accessed 3 March 2018).36. UK Research and Innovation. See https://www.ukri.org/ (accessed 3 March 2018).37. Op. cit. note 27.38. Op. cit. note 24.39. T he Academy of Medical Sciences. 2017 Enhancing the use of scientific evidence to judge the potential benefitsand harms of medicines, online annex A: Detailed case studies. See ssed 3 March 2018).12EVIDENCE SYNTHESIS FOR POLICY

CHAPTER TWOPrinciples for good evidencesynthesis for policyHere we present a set of principles, groupedinto four categories, for good evidencesynthesis for policy (Figure 1). In developingthese principles we have focused on thefundamental features that should apply toany evidence synthesis regardless of thetimeframe, topic or methods used – fromsystematic reviews to rapid syntheses, acrossdisciplines and policy areas. As such, many ofthe principles will be familiar to those alreadyinvolved in synthesis.The principles aim to make it easy for thosenew to synthesis – whether policymakersor researchers – to identify, use, conduct orcommission good synthesis. If the principlesare followed, policymakers should have a clearunderstanding of how synthesised evidencehas been generated and reviewed, andshould appreciate the complexities, areas ofcontention and limitations.EVIDENCE SYNTHESIS FOR POLICYFor their part, researchers less familiar withsynthesis should be able to rapidly designprocesses that will add the greatest value topublic debate. Only then can policymakerseffectively integrate the evidence withthe other factors on which they base theirdecisions.The principles aim tomake it easy for thosenew to synthesis –whether policymakersor researchers – toidentify, use, conductThe principles do not detail how to conducta systematic review or any other type ofsynthesis. For this, we recommend consultingevidence synthesis experts to buildknowledge and capacity.or commission goodsynthesis.13

CHAPTER TWOFIGURE 1Principles for good evidence synthesis for policy.InclusiveRigorous Involves policymakers andis relevant and useful to them. Uses the most comprehensivefeasible body of evidence. Considers many types and sourcesof evidence. Recognises and minimises bias. Is independently reviewed as partof a quality assurance process. Uses a range of skills and people.Principlesfor goodevidencesynthesisfor policyAccessible Is written in plain language. Is available in a suitable timeframe. Is freely available online.Transparent Clearly describes the research question,methods, sources of evidence and qualityassurance process. Communicates complexities and areasof contention. Acknowledges assumptions, limitations anduncertainties, including any evidence gaps. Declares personal, political and organisationalinterests and manages any conflicts.14EVIDENCE SYNTHESIS FOR POLICY

CHAPTER TWOONETHREEInclusiveTransparentEvidence synthesis that involves policymakersthroughout – from the design of the research questionto the interpretation of findings – is most likely to yieldsignificant policy insights. Keeping the process inclusivemakes it more likely that it will identify the full range ofrelevant evidence types, sources and expertise. Teams ofcontributors should have a mix of skills in synthesis andcould include some or all of the following: policymakers,practitioners, subject experts, statisticians, experts indatabases and search terms, objective writers (usuallynon-subject experts), and independent reviewers. Inpractice, policymakers may be less involved during partsof the process if the aim is to scan the horizon for futurepriorities or to synthesise evidence on a topic that is yetto attract major policy interest.Synthesised evidence that is transparent is likely to bemore credible, replicable and useful. A clearly describedstudy design should include the search terms used,the databases and other evidence sources consideredand when they were accessed, and the criteria thatdetermine which studies are and are not included andwhy. Such m

case for evidence synthesis. It then proposes a set of principles that defines the fundamental features of good synthesis. Finally, it proposes changes to the research and policy landscapes that would create a more effective 'marketplace' for synthesis: one in which policymakers and commentators reach out