Stata: A Key Strategic Statistical Tool Of Choice In Major Impact .

Stata: A key strategic statistical tool of choice in majorimpact evaluations of socioeconomic programsGwinyai NyakuengamaIndependent Advisor2017 Oceania Stata Users Group MeetingAustralian National University, Canberra29 September 2017

Stata – a key strategic statistical tool-of-choice in impact evaluationsThis presentation: Explains program impact evaluations (PiEs), evaluation designsand program-logic. Discusses strategic thinking behind PiEs, particularly the art andimportance of choosing appropriate data tools. Explains why Stata is internationally a highly regarded, state-ofthe-art, strategic tool used in impact program evaluations.Presentation Outline

Stata – a key strategic statistical tool-of-choice in impact evaluationsAims of PiEs? To provide accurate and timely, quantitative evidence to governments onwhether or not, a national program has worked; Why, how, when andwhere? To identify any unintended consequences To identify areas for improvementEvidence-based policy advice

Stata – a key strategic statistical tool-of-choice in impact evaluationsWhy are PiEs important in evidence-based policy advice?Good government – withstand parliamentary and media scrutiny: Accountability for millions of tax-payer dollars spent on programs Transparent government decisions Improve and fund future national programsEvidence-based policy advice

Stata – a key strategic statistical tool-of-choice in impact evaluationsHow are PiEs conducted? Evaluation frame to assess program appropriateness, effectivenessand efficiency – this is the evaluation program logic.Evidence-based policy advice

Stata – a key strategic statistical tool-of-choice in impact evaluationsGood framework for PiEs? Clear statement of the public policy problem the program addressed Correct experimental designs to collect credible evidence Credible and appropriate data sources - administrative and survey Carefully tailored robust statistical methods Correct statistical tools to collect data, assess the evidence and reportCollect, evaluate and report quality evidence

Stata – a key strategic statistical tool-of-choice in impact evaluationsKey strategic evaluation questions? Accountability – evaluation is accountable to stakeholders Propriety – evaluation is ethical Utility – evaluation meets information needs of stakeholders Feasibility – evaluation is viable and pragmatic Accuracy / Quality – evaluation findings are considered correctCollect, evaluate and report quality evidence

Stata – a key strategic statistical tool-of-choice in impact evaluationsWhy are evaluation designs very important? Ultimately, quality of government policies rests on quality of theevidence produced from impact programs evaluations. PiEs demonstrate if policy has made a quantifiable difference, interms of cause and effect. Greatest difficulty is in generating controls / counterfactual: Natural experiments are unethical; Controlling internal validity issues: Confounders; Selection bias;Program contamination, Spill-overs and Impact heterogeneity Evaluation designs are star-rated GOLD, SILVER and BRONCE.Evaluation Designs – show me the evidence

Stata – a key strategic statistical tool-of-choice in impact evaluationsHow is Stata used to turn data into evidence? Data Planning: Develop PiEs design, research questions & analysis Data Collection: Survey instruments & Panel data; Data Management: Extraction Transform, Load (ETL) Data Analyses: Broad suite of statistical procedures; Visualisations; Maps; Data Reporting: LaTEX; Publication-ready graphic & tables; Automation Data and Code Sharing and Publishing/Reporting: PDF; ado-files etcStata data stages

Stata – a key strategic statistical tool-of-choice in impact evaluationsStata – when efficient, timely, comprehensive and accurate evidence matters Handles very large and complex datasets from many socio-economic sectors and in real-time; Provides robust, rigorous and reliable statistics and data visualisations in impact programevaluations: Counterfactual bias controls:o GOLD standard: Randomised Controlled Trials (RCT)o SILVER standard: Difference in Difference; Regression Discontinuity; Propensity ScoreMatching; Instrument Variables, Bandwidth Matching; Quantile Regression and DataEnvelopment Analysis.o BRONCE standard: Qualitative comparisons – with or no controlsStata strategic value

Stata – a key strategic statistical tool-of-choice in impact evaluationsRandomised Control Trials (RCT)Stata - Advanced statistical procedures for impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsRegression Discontinuity Design (RDD)Stata - Advanced statistical procedures for impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsRegression Discontinuity Design (RDD)Stata - Advanced statistical procedures for impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsDifference in Difference (DiD)Stata - Advanced statistical procedures for impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsPropensity Score Matching (PSM)Stata - Advanced statistical procedures for impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsAustralian Government impact program evaluations that used StataStata in national socio-economic impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsOther International Government PiEs that used StataStata in national socio-economic impact program evaluations

Stata – a key strategic statistical tool-of-choice in impact evaluationsA selection of published national PiEs using STATAAuthorAustralian Gov. DEEWR (2008)John Haisken-DeNew (2013)Moran et al. (2015)Rogers et al. (2015)Rogers et al. (2016)The Melbourne Institute (2017)OECD (2004)Pu and Gibson (2016)World Bank (2011)Rutkowski et al (2013)Sun et al. (2017)Grill et al. (2014)Wolszczak-Derlacz and Parteka (2011)Galarraga et al. (2010)Buddelmeyer & Skoufias (2005)Green et al. (2014)Sianesi (2016)Nunn & Qian (2014)Hawkins et al. (2014)Inter-university Consortium for Politicaland Social Research (2009)Patel et al. (2015)Sahr (2010)Lance et al. (2014)Djebbari and Lopera (2011)Langbein and Felbinger (2014)Bell and Gianni (2015)Sikora (2016)Study TitleCountryWebsiteWe l fa re to Work Eva l ua ti on Re les/welfaretoworkevaluationreport.pdfA gra phi ca l us e r i nterfa ce i n s ta ta for e xtra cti ng da ta from the LSAC a nd LSICDo Aus tra l i a n Ca thol i c a nd Inde pe nde nt Pri ma ry School s Produce Be tte r Aca de mi c Outcome s tha nGove rnme nt School s /commwkpapers/340/Choos i ng a ppropri a te de s i gns a nd me thods for i mpa ct e va l ua ti onInve s ti ga ti ng the i mpa ct of NAPLAN on s tude nt, pa re nt a nd tea che r e moti ona l di s tre s s i ni nde pe nde nt s chool sThe Hous e hol d, Income a nd La bour Dyna mi cs i n Aus tra l i a Surve y: Se l e cted Fi ndi ngs from Wa ve s 1to melbourneinstitute.unimelb.edu.au/ data/assets/pdf file/0010/2437426/HILDA-SR-med-res.pdfEva l ua ti ng Loca l Economi c a nd Empl oyme nt De ve l opme ntThe e ffe cts of i ntera cti on be twe e n l oca ti on of bi rth a nd l oca ti on of s tudy on i mmi gra nt worke rs 'wa ge s i n Ca na g/canada17/slides/Canada17 Pu poster.pdfImpa ct e va l ua ti on i n pra cti DOFFICE/Resources/5485726-1295455628620/Impact Evaluation in Practice.pdfHa ndbook of Interna ti ona l La rge -Sca l e As s e s s me nt - /EdPsy587/MLchapter submit.pdfEva l ua ti on of the pe rforma nce of na ti ona l he a l th s ys tems i n 2004-2011: An a na l ys i s of 173 countri e sExpl oi ti ng TIMSS a nd PIRLS combi ne d da ta : mul ti va ri a te mul ti l e ve l mode l l i ng of s tude nta chi e ve me nt173 id /pdf/1409.2642.pdfEffi ci e ncy of Europe a n publ i c hi ghe r e duca ti on i ns ti tuti ons : a two-s ta ge mul ti country a pproa chHe a l th i ns ura nce for the poor: i mpa ct on ca ta s trophi c a nd out-of-pocke t he a l th e xpe ndi ture s i nMe xi articles/PMC2888946/The Progre s a oportuni da de s progra m of Me xi co a nd i ts Impa ct Eva l ua ti onCos t-Effe cti ve ne s s of Col l a bora ti ve Ca re for De pre s s i on i n UK Pri ma ry Ca re : Economi c Eva l ua ti on of aRa ndomi s e d Control l e d Tri a l / plosone/article?id 10.1371/journal.pone.0104225Pol i cy e va l ua ti on me thodsUKhttps://www.ifs.org.uk/events/1292Food a i d a nd ci vi l confl i ctEva l ua ti ng the i mpa ct of the Ba by-Fri e ndl y Hos pi ta l Ini ti a ti ve on bre a s t-fe e di ng ra tes : a mul ti -s ta tea na l ys i sUSA & es/faidconf 20130806 final PMC4163534/Na ti ona l Supported Work Eva l ua ti on Study, 1975-1979: Publ i c Us e Fi l e s (ICPSR 7865)Rol e -Mode l i ng Cos t-Cons ci ous Ca re —A Na ti ona l Eva l ua ti on of Pe rce pti ons of Fa cul ty a t Te a chi ngHos pi ta l s i n the Uni ted Sta /articles/PMC4539317/Sta ta ma ke s a di ffe re nce a t the He a l th Pol i cy Ins ti tute of Ohi oHow Do We Know i f a Progra m Ma de a Di ffe re nce ? A Gui de to Sta ti s ti ca l Me thods for Progra mImpa ct Eva l ua ti rces/publications/ms-14-87-enImpa ct e va l ua ti on us i ng uation-using-stataPubl i c Progra m Eva l ua ti on: A Sta ti s ti ca l Gui /MEI-3-2012-Public-Program-Evaluation- -validity.pdfThe ory a nd Pra cti ce us i ng e SummerSchool TStat2015 UK.pdfAdol e s ce nt i ntere s t i n s ci e nce ca re e rs i n Europe : Tre nds be twe e n 2006 a nd 2015TBATBASelect national socio-economic impact program evaluations with Stata

Stata – a key strategic statistical tool-of-choice in impact evaluationsStata is a tool of choice for SMART Impact Program Evaluation Specific – strategic and specific target for improvement Measurability – quantifiable indictor of progress, especially of thecounterfactual Achievability – state what results can realistically be achieved Realistic – attainability of standards Time – specify when the result(s) can be achieved Budget – within the allocated budgetCollect, evaluate and report quality evidence

Stata – a key strategic statistical tool-of-choice in impact evaluationsStata – Strategic PiEs Data Capabilities Impact Evaluation Capabilities – Self-contained, powerful data management, analysis, reporting & mapso Meets ISO standards & US FDA regulator compliance requirements Intuitive & interactive user-interface – Easy to learn, use and teach Simple & consistent structure – Easy to learn, use and teach Expandability & maintainability – Easy and efficient, in-program installation of updates; Java API expandability Well-documented – Examples for every procedure Resources – Stata: Journal, News, Press, Blog, Training and Video Tutorials Community support – Stata conference, Statalist, User comments Portability – Windows, Mac, Linux/Unix and configurations (standalone or networked) Interoperability – Works with other tools (e.g. SAS, R, Python, Tableau, MapInfo, Spotfire, RapidMiner)Stata strategic capabilities

Stata – a key strategic statistical tool-of-choice in impact evaluationsStata – Comprehensive program impact evaluations analytics Data Typeso Panel datao Survey data Unmatched Treatment effects/Causal inference estimatorso Propensity-score matchingo Inverse-probability weights (IPW)o Covariate matchingo Regression adjustment Sophisticated Counterfactual Testingo Weighted regressiono Experimental method (Gold standard)o Augmented IPW (AIPW)o IPW with regression adjustment Random Control Trialso Doubly robust methodso Quasi-Experimental methods (Silver standard)o eTregresso Non-parametric synthetic controls Regression Discontinuity Difference in Difference Instrument Variables Big Data, Machine Learning and Predictive modelso Parallel processing (e.g. ftools)o Classifications, Regressions and ClustersStata - Advanced statistical procedures for impact program evaluations