The Impact On Healthcare, Policy And Practice From 36 Multi-project .

Hanney et al. Health Research Policy and Systems (2017) 15:26an initial scan of the available studies we identified fourimpact categories that were broadly compatible with, butnot necessarily identical to, the impact categories in thewidely-used Payback Framework [14, 15] and the CanadianAcademy of Health Sciences adaptation of that framework[10]. The categories were impact on health policy or on ahealthcare organisation, informing practice or clinicianbehaviour, a combined category covering policy andclinician impact, and impact on health gain, patient benefit,improved care or other benefits to the healthcare system.Studies were included if they had presented findings inone or more of these categories in a way that couldallow standardised comparison across programmes. Insome cases, the studies presented findings solely interms of the numbers of projects that had claimed orbeen shown to have had impact in a particular category.These had to be standardised and presented as percentages. Each study was given the same weight in the analysis, irrespective of the number of individual projectscovered by the study. For each of the four categories ofimpacts we then calculated the median figure for thosestudies showing the percentage of projects that hadclaimed to make an impact in that category. We alsopresented the full range of percentages in each category.We extracted data on methods and conceptual frameworks for assessment of research impact described in eachstudy, and on categories of factors considered by the authors to be relevant for the level of impact achieved. Inidentifying the latter, our approach was informed by arange of international research literature, in particular the1983 analysis by Kogan and Henkel of the importance ofresearchers and potential users working together in a collaborative approach, the role of research brokers, and thepresence of bodies that are ready to receive and use theresearch findings [16, 17]. Other papers on these and related themes that influenced our approach to the analysisincluded literature related to North and Central America[18–21], Africa [22], the European Union [23], and theUnited Kingdom [6, 14, 24], as well as international studies and reviews [25–31].ResultsThirty-six studies met the inclusion criteria for this analysis [6, 32–66]. These were highly diverse in terms ofthe location of the research, nature and size of the funder’s research programme or portfolio, the fields of research and modes of funding, time between completionof the programme and impact assessment, the methods(and sometimes conceptual frameworks) used to assessthe impact, and levels of impact achieved. A brief summary of each study is provided in Table 1.The studies came from 11 different countries, plus aEuropean Union study and one covering various locationsin Africa. The number of projects supplying data to thePage 3 of 21studies ranged from just eight in a study of an occupational therapy research programme in the UnitedKingdom [59], to 22 operational research projects inGuatemala [35], 153 projects in a range of programmeswithin the portfolio of the Australian National BreastCancer Foundation [38], and 178 projects from the HongKong Health and Health Services Research Fund [51].In terms of the methods used to gather data about theprojects in a programme, 21 of the 36 studies surveyedthe researchers, usually just each project’s Principal orChief Investigator (PIs), either as the sole source of dataor combined with other methods such as documentary review, interviews and case studies. Six studies relied exclusively, or primarily, on documentary review and deskanalysis. In at least three studies, interviewing all PIs wasthe main method or key starting point used to identify further interviewees. The picture is complicated because somestudies used one approach, usually surveys, to gain information about all projects, and then supplemented that withother approaches for selected projects on which case studies were additionally conducted, and often involved interviews with PIs. In total, over a third of the studies involvedinterviews with stakeholders, again sometimes in combination with documentary review. Many studies drew on arange of methods, but two examples illustrate a particularlywide range of methods. In the case of Brambila et al. [35] inGuatemala, this included site visits which were used to support key informant interviews. Hera’s [46] assessment of theimpact of the Africa Health Systems Initiative Support toAfrican Research Partnerships involved a range of methods.These included documentary review, and programme levelinterviews. Project level information was obtained fromworkshops for six projects and from a total of 12 interviewsfor the remaining four projects. In addition, they used participant observation of an end-of-programme workshop, atwhich they also presented some preliminary findings. Inthis instance, while the early timing of the assessmentmeant that it was unable to capture all the impact, the programme’s interactive approach led to some policy impactduring the time the projects were underway.In 20 of the 36 studies, the various methods used wereorganised according to a named conceptual framework(see Hanney et al. [6] and Raftery et al. [9] for a summary of all these frameworks); 16 of the 36 studies drewpartly or wholly on the Payback Framework [15]. Aseries of existing named frameworks each informed oneof the 36 studies, and included the Research ImpactFramework [24], applied by Caddell et al. [37]; theCanadian Academy of Health Sciences framework [10],applied by Adam et al. [32]; the Banzi Research Impactmodel [11], applied by Milat et al. [53]; and the BeckerMedical Library model [67], applied by Sainty [59].In addition, various studies were identified as drawing, at least to some degree, on particular approaches,

Semi-directive interviews withstakeholders affected by therecommendations (14); casestudies used surveys inhospitals to examine impact ofthe recommendations (13)No framework stated, butapproach to scoring impactfollowed earlier studies of theCETS in Quebec reported byJacob et al. [47, 48]Key informant (KI) interviews;document review; site visits tohealth centres and nongovernmental organisationsimplementing operational research interventions; scored 22projects (out of 44 conductedbetween 1988 and 2001) onindicators: 14 process; 11 impact; 6 contextAlberta Heritage Fund forMedical Research – HealthresearchFrench Committee for theAssessment and Disseminationof Technological Innovations(CEDIT) – Health technologyassessment (HTA)Population Council –Programme of OperationResearch projects inreproductive health inGuatemalaAlberta Heritage Fund forMedical Research, 2003 [33];Alberta, CanadaBodeau-Livinec et al., 2006[34]; FranceBrambila et al., 2007 [35];GuatemalaSurvey to PIs (100, 50responded, 50%); interviewswith decision makers andusersVersion of Payback FrameworkBibliometric analysis; surveysto researchers(99, 70 responded, 71%);interviews – researchers (15),decision-makers (8); in-depthcase study of translationpathwaysCanadian Academy of HealthSciences frameworkCatalan Agency for HealthInformation, Assessment andQuality – Clinical and healthservices researchAdam et al., 2012 [32];Catalonia, SpainMethods for assessing healthresearch impact/concepts andtechniquesProgramme/specialityAuthor, date, locationTable 1 Thirty-six impact assessment studies: methods, frameworks, findings, factors linked to impact achievedHighlighted how impact canarise from a long-term approach and the several 5-yearcycles of funding “allowed forthe accumulation of evidence inaddition to the development ofcollaborative ties between researchers and practitioners,which ultimately resulted inMain factor fosteringcompliance withrecommendations “appears tobe a system of regulation” ([34],p.166) Reviewed other studies:“All these experiences togetherwith our own work suggest thatthe impact of HTA on practicesand introduction of newtechnologies is higher the morecircumscribed is the target ofthe recommendation” ([34], p.167)Widespread interest, “used asdecision-making tools by administrative staff and as a negotiating instrument by doctorsin their dealings with management.ten of thirteen recommendations had an impact onthe introduction of technologyin health establishments” ([34],p. 161); 7 considerable,3 moderate: total 77%Of the 22, 13 projectsintervention effective inimproving results, threeinterventions not effective; in14 studies implementingagency acted on results; nineinterventions scaled up insame organisation; fiveadopted by anotherorganisation in Guatemala;Research teams with decisionmakers or users more successful than those withoutInteractions and participationof healthcare and policydecision-makers in the projectswere crucial to achieving impact; the study showed thatthe Agency achieved the aimof filling a gap in local knowledge needs; study provideduseful lessons for informingthe funding agency’s subsequent action; the studies “provide reasons to advocate fororiented research to fill specificknowledge gaps” ([32], p. 327)Factors associated with level ofimpact; comments onmethods and use of thefindings49% impact on policy; 39%changed behaviour; 40%health sector benefitsOverall, 40 principalinvestigators (PIs) (of the 70)gave 50 examples of changes;examples included 12organisational changes of thecentre/institution; two publichealth management; twolegal/regulatory (some PIsmight have given more thanone of these: therefore, totalfor organisational/management/policy changes:possibly 17–23%, and 20%figure used in this analysis); 29of the 70 (41%): changedclinical practiceImpacts foundHanney et al. Health Research Policy and Systems (2017) 15:26Page 4 of 21

NHS North Thames Region –Wide-ranging responsivemode R&D programmeIWK Health Centre, Halifax,Canada, Research OperatingGrants (small grants) – Womenand children’s healthNational Breast CancerFoundation – Wide range ofprogrammesEuropean Union FrameworkProgrammes 5, 6, and 7 –Public health projectsBuxton et al., 1999 [36];United KingdomCaddell et al., 2010 [37];CanadaDonovan et al., 2014[38];AustraliaExpert Panel for HealthDirectorate of the EuropeanCommission’s ResearchInnovation Directorate General,2013 [39]; European UnionDocumentary review: all 70completed projects; 120ongoing; KI interviews withparticularly successful andunderperforming projects (16);Documentary analysis,bibliometrics, survey of PIs(242, 153 responded, 63%), 16case studies, cross-caseanalysisPayback FrameworkOnline questionnaire to PIsand co-investigators (Co-Is)(64, 39 responded, 61%)Research Impact Framework:adaptedQuestionnaires to PIs (164, 115responded, 70%) and somebibliometric analysis for allprojects and case studies (19);case studies includedinterviews with researchersand some usersBenefit scoring system basedon two criteria (importance ofthe research to the changes,and level at which the changewas made) was used to scorequestionnaire responses aboutthe impacts and re-score theimpact from each study onwhich a case study conductedPayback FrameworkDeveloped an approachinvolving process, impact andcontextual factors; drew onliterature such as Weiss [18]and interactive approachesUsed documentary review,therefore for completedprojects had data about wholeset; however, “Extensive followup of the post-project impact ofBasic research – more impacton knowledge and drugdevelopment; applied research– greater impact in otherpayback categories; manyprojects had only recentlybeen completed – moreimpact expected; in launchingthe report the charityhighlighted how it wasinforming their fundingstrategy [92]10% impact on policy – 29%expected to do so; 11%contributed to productdevelopment; 14% impact onpractice/behaviour – 39%expected to do soAppendix 1: only 6 out of the70 completed projects did notachieve the primary intendedoutput; 42% took actions toengage or informAn association betweenpresenting at conferences andpractice impacts; authors stresslink between research andexcellence in healthcare: “It isessential that academic healthcentres engage actively inensuring that a culture ofresearch inquiry is maintained”([37], p. 4)The survey/case studycomparison suggests “greaterdetail and depth of the casestudies often leads to asomewhat different judgementof payback, but there is noevidence of a systematic underassessment of payback from thequestionnaire approach, nor,generally, of greatly exaggeratedclaims being made by researchers in the self-completedquestionnaires” ([36], p. 196)changes to the service deliveryenvironments” ([35], p. 242)16% policy impact: 8% inhealth centre, 8% beyond; 32%said resulted in a change inclinical practice; 55% informedclinical practice by providingbroader clinical understandingand increased awareness(average of 43% for practiceimpact); 46% improved qualityof care41% impact on policy; 43%change in practitioner/manager behaviour; 37% ledto benefits to health andhealth servicesome studies led to policychanges, mainly at theprogramme level (total 64%impact in combined policyand practice category)Table 1 Thirty-six impact assessment studies: methods, frameworks, findings, factors linked to impact achieved (Continued)Hanney et al. Health Research Policy and Systems (2017) 15:26Page 5 of 21

NHS Northern and YorkshireRegion – Health ServicesResearch (HSR) (two otherprogrammes not includedhere)Agency for HealthcareResearch and Quality –Integrated delivery systemsresearch networkRobert Wood JohnsonFoundation – Active livingresearchFerguson et al., 1998 [40];United KingdomGold & Taylor, 2007 [41];United States of AmericaGutman et al., 2009 [42];United States of AmericaSuccess factors: responsivenessof project work to deliverysystem needs, ongoingfunding, development of toolsthat helped users see theiroperational relevanceOnly 16% of grants had beencompleted prior to the year ofthe evaluation; someapproaches “worked well,including developing amultifaceted, ongoing,interactive relationship withadvocacy and policymakerorganizations” ([42], p. S32);grantees who completed bothinterviews and surveysgenerally gave similarresponses, but researchersincluded in the randomsample of interviewees gavehigher percentage of policyimpact than researcherssurveyed; questions slightlydifferent in the interviews thanin the surveysChanges in operations; “Of the50 completed projects studied,30 had an operational effect oruse” [41] (Operational effect oruse is a broad term: so the60% put into our combinedimpact category)Generally thought to be tooearly for much policy impact,but 25% of survey, 43% ofinterviewees reported a policyimpact; however, policyimpact in survey could befrom active living research ingeneral, not just the specificprogramme, and couldinclude: “a specific interactionwith policymakers (e.g.testifying, meeting withpolicymakers, policymakerbriefings, etc.) or direct evidenceof the research findings in awritten policy” ([42], p. S33)Documentary review ofprogramme as a whole andindividual projects (50);descriptive interviews (85); fourcase studies, additionalinterviewsNo explicit frameworkdescribedA retrospective, in-depth, descriptive study utilising multiple methods; quantitativedata derived primarily from aweb-based survey of granteeinvestigators (PIs, Co-PIs), ofthe 74 projects: 68 responsesanalysed; qualitative data from88 interviews with KIsThe conceptual model used inthe programme “was used toguide the evaluation” ([42], p.S23).Aspects of Weiss's model usedfor analysing policycontributionscompleted projects was notpossible” ([39] p. 9)Comprehensive coverage of aprogramme without requiringadditional data from theresearchers; however, alsoshows the limitations of suchan approach in capturing laterimpactsThis was part of a wideranalysis, but in all three areasthe projects were reactive;particularly difficult to make animpact with Primary andCommunity Care researchpolicymakers; 4 (6%) projectschange of policy, 22%expected to do so; 7 (10%)impact on health practitioners;6 (9%) impact on healthservice delivery and 6 (9%)impact on health; 1 beneficialimpact on small/medium-sizedenterpriseFive HSR projects (16%) had apolicy impact, i.e. “Betterinformed commissioning andcontracting” ([40] p. 17); 5(16%) led to a change in NHSpractice, i.e. “More effectivetreatment, screening ormanagement for patients” ([40],p. 16)Desk analysis (bibliometrics),surveys to gather quantitativeand qualitative data sent to allPIs and Co-Is in all three programmes: but only HSR projects asked about policy, sojust the 32 HSR responses analysed hereRefer to Payback Framework;no attempt to develop owndata extraction formconstructed based on thecategories from the PaybackFramework, with each of themain categories broken downinto a series of specificquestionsPayback FrameworkTable 1 Thirty-six impact assessment studies: methods, frameworks, findings, factors linked to impact achieved (Continued)Hanney et al. Health Research Policy and Systems (2017) 15:26Page 6 of 21

Multiple methods: literaturereview, funder documents,survey all PIs of projectsbetween 1993 and 2003 (204,133 responses, 65%), casestudies with interviews (16)Payback FrameworkSurvey of all PIs (153, 96responses, 59%), documents,case studies (14) involvinginterviews and someexpanding the approach tocover role of chairs and centrePayback FrameworkNational Health Service (NHS)– HTA programmeAsthma UK – All programmesof Asthma researchAfrica Health Systems InitiativeSupport to African ResearchPartnershipsHanney et al., 2007 [6];United KingdomHanney et al., 2013 [45];United KingdomHera, 2014 [46]; AfricaDocumentary review;interviews at programme level;project level information – forsix projects, workshops, for theremaining four a total of 12interviews; participantInterviews with thoserequesting the 20 brief HTAnotes (i.e. reviews); checks onquality of the reports madeusing desk analysis andcomments from expertsNo framework describedCanadian province (not stated)– HTA brief tech notesHailey et al., 2000 [44]; CanadaLooked at technologies (20)covered by HTA reports fromthe panel up to end of 1988.Little provided on methods –presumably desk analysis, juststates comparingrecommendations,assessments and policyactivitiesNo framework describedNational Health TechnologyAdvisory Panel – HTA reportsHailey et al., 1990 [43];AustraliaPolicy impact was createdduring the research process: 7out of 10 projects reportedpolicy impact already, “Thepolicy dialogue is not yet13% impact on policy; 17%product development; 6%health gain; but case studiesreveal some importantexamples of influence onguidelines, some potentiallymajor breakthroughs inasthma therapies,establishment of pioneeringcollaborative research centreTechnology AssessmentReports (TARs) produced forthe National Institute forHealth and Clinical Excellence(NICE): 96% impact on policy,60% on clinician behaviour;primary and secondary HTAresearch: 60% impact onpolicy, 31% on behaviourAverage for programme: 73%impact on policy, 42% onbehaviour; case studiesshowed large diversity inlevels and forms of impactsand the way in which theyarise14 (70%) had influence onpolicy and other decisionsOut of the first 20technologies covered by HTAreports there had beensignificant impact in 11 andprobable influence in three:70% in totalTable 1 Thirty-six impact assessment studies: methods, frameworks, findings, factors linked to impact achieved (Continued)“Research teams who startedthe policy dialogue early andmaintained it throughout thestudy, and teams that engagedwith decision-makers at locallevel, district and national levelsMany types of research andmodes of funding – long-termfunding of chairs led to

between completion and impact assessment, methods used to assess impact, and level of impact identified. Thirty-one studies reported on policy impact, 17 on clinic ian behaviour or informing clinical practice, three on a combined category such as policy and clinician impact, and 12 on wider elements of impact (health gain, patient benefit,