Applying Propensity Score And Mediation Analysis To .

ApplyingPropensity Score and Mediation Analysisto Program and Policy EvaluationTuesday, September 16, 2014Kristin Rankin, PhDkrankin@uic.eduAmanda Bennett, PhD Deb Rosenberg, PhDacaven3@uic.edudrose@uic.eduDivision of Epidemiology and BiostatisticsSchool of Public Health, U. of IL at Chicago (UIC-SPH)

2Agenda for the Day8:00-8:40Overview of Methodological Issues8:40-12:00Propensity Score Analysiswith a break at approx. 10:1512:00-1:00LUNCH1:00-4:20Mediation Analysiswith a break at approx. 2:454:20-5:00Summary and Final CommentsDiscussion throughout

ApplyingPropensity Score and Mediation Analysisto Program and Policy EvaluationOverview of Methodological IssuesDeb Rosenberg

4The Epidemiologic Context forEvaluating Programs and PoliciesPrograms and policies are intended to interrupt acausal sequence that leads to an adverse outcome orto promote a causal sequence leading to a positiveoutcome.The mechanism(s) of action are likely to be complexand may be aimed at differing points along a causalpathway

5The Epidemiologic Context for EvaluatingPrograms and PoliciesThe mechanisms of action and evaluatingeffectiveness are particularly difficult when lookingat public health programs that have multiplecomponents delivered in varying ways and withvarying intensity. Some examples: prenatal caremedical homehome visitingMedicaid expansionworkplace breastfeedingWhat are the hypothesesabout these programsand policies?How do they operatealong a causal pathway?

6The Epidemiologic Context for EvaluatingPrograms and PoliciesIs there an association between a program and anoutcome? How much of that association operatesthrough affecting a specific factor and how muchoperates through other pathways?Program / Policy(Exposure)Other Factor(Mediator)OutcomeHere the program takes on the role of an exposure("treatment") along the causal pathway

7The Epidemiologic Context for EvaluatingPrograms and PoliciesIs there an association between a factor and anoutcome? How much of that association operatesthrough access to a specific program and how muchoperates through other pathways?Other Factor(Exposure)Program / Policy(Mediator)OutcomeHere the program takes on the role of a mediatoralong the causal pathway.

8Methodological Approaches to TestHypotheses about Programs or PoliciesThe Counterfactual FrameworkThe only way we would truly know if a program, policy,or other "exposure" causes an outcome is if we couldconsider each individual under both conditions ofexperiencing the program or policy and of notexperiencing the program or policy.Since this is never possible, we aim to choose studydesigns and methodological approaches that can get usas close as feasible to what the counterfactual resultswould be.

9Methodological Approaches to TestHypotheses about Programs or PoliciesStudy Design and Sample Selection Experimental design with random assignment,subjects are recruited to study and then randomlyassigned to receive the treatment / program /exposure Quasi-experimental design—norandomization, but some control over the programand subject selection Observational design—no randomization andno specific control over the program

10Methodological Approaches to TestHypotheses about Programs or PoliciesIt is often not feasible or ethical to conduct arandomized controlled trial for estimating the causaleffect of a program on health outcomes. In addition,even quasi-experimental designs may not be possiblefor public health programs.In observational studies, study bias—in particularselection bias—and confounding cannot beaddressed prior to analysis, so methods are neededin order to obtain a precise estimate of programeffect or its mechanisms of action.

11Bias and ConfoundingBiases—selection bias and information bias(misclassification, reporting) are related to an aspect ofthe analytic process itself: participant recruitment /retention, data collection or data analysis, resulting inan invalid estimate of effect.Confounding—the distortion of the estimate of effectby another factor not explicitly related to an aspect ofthe analytic process.

12Selection BiasFor evaluation of programs and policies, selection biasis particularly important because it results in a non—random sample for use in testing hypotheses.Specific selection forces may operate singly or incombination to produce bias in programparticipation—that is, program participation reflectsa systematic, non-random process.Depending on which selection forces are operating,the effect of a program might be either overestimated or under-estimated

13Selection BiasOver-estimation of program effect:1. Favorable selection: those at low risk, with betteraccess to care generally, potentially more healthconscious are over-represented in the program2. Estrangement selection: those at high risk, withlife circumstances which generally distance themfrom the health care system are underrepresented in the programMethodological Note: Selection Bias in Prenatal Care Use by Medicaid RecipientsJanice F. Bell, MN, MPH1, and Frederick J. Zimmerman, PhD,Maternal and Child Health Journal, Vol. 7, No. 4, December 2003 ( C 2003)

14Selection BiasUnderestimation of program effect:3. Adverse selection: those at high risk spurred bysymptoms, prior experience, or family historyare over-represented in the program4. Confidence selection: those at low risk who donot perceive a need based on prior experience,general good health status are under-representedin the program

15Group Equivalence: Selection Bias,Confounding, and the Counterfactual IdealAddressing selection bias and confounding—defining comparison groupsThe counterfactual a priori achieves equivalence, aseach individual is compared to his/herself. Statisticalapproaches aim to approach this at the population level. In experimental studies, the process of randomassignment achieves equivalence, on average,between groups In non-experimental studies, use matching oradjustment to achieve on average equivalence.

Bias and Confounding

17Regression Modeling ApproachesTraditional multivariable regression involvesspecifying a single model for estimating theassociation between a program or policy—the"exposure"—and an outcome, addressing potentialselection, effect modification and/or confounding:Outcome Program covariates, inc. inter. termsThe number and complexity of covariate adjustmentwill be constrained by sample size

18Regression Modeling ApproachesPropensity Score Analysis and Mediation Analysisuse more complex regression approaches than theusual linear modelEach method uses a multi-step process, employingtwo regression equations

19Regression Modeling ApproachesPropensity Score Analysis A tool for approximating a randomized trial andreducing selection bias in observational studies On average, individuals with the same propensityscore are balanced on a wide array of covariates,achieving close equivalence between the exposedand unexposed groups (participants and nonparticipants in a program).

20Regression Modeling ApproachesMulti-step process forPropensity Score Analysis:1. Model I: Program pool of covariates(using entire sample to compute Pscores)2. Use matching, stratification, or weighting3. Check covariate balance4. Model II: Outcome Program(using only individuals successfully matched onpropensity scores, or using entire samplestratified or weighted by propensity score)

21Regression Modeling ApproachesMediation Analysis: Explicitly addresses questions of causal processesand a chain of events; often addresses selectionforces as part of that chain Provides a conceptual framework andaccompanying analytic methods Decomposes the total effect of an exposure on anoutcome (TE) into a natural direct effect (NDE) anda natural indirect effect (NIE) which involves apathway through a specific mediator

22Regression Modeling ApproachesMediation analysis focuses on the role the programor policy plays as part of the causal pathway to theoutcome.The goal is to identify both direct and indirectpathways, either from the program through anotherfactor to the outcome or direct and indirect pathwaysfrom a factor through the program to an outcomeThe direct and indirect pathways estimatecounterfactual results

23Regression Modeling ApproachesMulti-step process for Mediation Analysis:1. Model I: Outcome Exposure Mediator E*M pool of covariates2. Model II: Mediator Exposure pool of covariates3. Combine the coefficients from these models tocalculate Natural Direct and Natural Indirect Effects

24Examples and InterpretationAmong children with asthma:MedicalHomeReduced UnmetHealthcare NeedsFewer ER visitsNote the directionality of all three variablesThe associations between a medical home and fewer ER visits,a medical home and unmet healthcare needs, and unmethealthcare needs and fewer ER visits may all be of interest.The primary association is between the medical home—theprogram—and fewer ER visits among children with asthma.Other covariates farther back in the causal chain may beimposing selection bias on this association—which childrenreceive care in the medical home model may not be random.

25Examples and InterpretationWith medical home—the "program"—as the exposure /treatment and reduced unmet healthcare needs as themediator, we may want to ask questions like the following:1. What is the effect of the medical home on the number ofER visits among children with asthma through anymechanism – through all pathways?2. What is the effect of the medical home on the number ofER visits among children with asthma if these children allhad the same prevalence of unmet health care needs?3. As one specific mechanism of the medical home, howimportant is reducing unmet healthcare needs indecreasing ER visits among asthmatic children?

26Examples and InterpretationTypical regression: asthmatic children1. Estimate the association between having amedical home and number of ER visitsEstimate the association using the full sample, after adjustingfor a small set of covariates, but not adjusting for unmet needssince hypothesized to be in the causal pathway.2. Estimate the association between unmet needsand the number of ER visitsEstimate the association using the full sample, after adjustingfor a small set of covariates, including the medical home

27Examples and InterpretationPropensity Score Analysis: asthmatic children1. Estimate the association between having amedical home and number of ER visits Estimate the association using only those childrenindividually matched on their propensity for having amedical home, or using the full sample, stratifying orweighting by the propensity score. The propensity score will include a wide array of covariates,but will not include unmet needs. By more complete control for selection forces andconfounding, the estimated association between having amedical home and ER visit should be close to unbiased.

28Examples and InterpretationMediation Analysis:2. Estimate the association between having a medicalhome and number of ER visitsEstimate the association using the full sample, adjusting for asmall set of covariates and considering reduced unmethealthcare needs as a mediator.By then calculating the Natural Direct Effect and NaturalIndirect Effect, the results will inform an understanding ofthe extent to which the medical home does and does notoperate through reduction in unmet healthcare needs.

29Examples and InterpretationMediation Analysis: asthmatic childrenEstimate the Natural Direct Effect: the associationbetween a medical home and ER visits while fixing theprevalence of reduced unmet needs to that seen amongasthmatic children without a medical home.The NDE indicates the magnitude of the association betweenhaving a medical home and ER visits that exists even withequal prevalence of unmet needs. This reflects the extent towhich the medical home operates through pathways otherthan reducing unmet needs (has other mechanisms of action).