Missing Data And Sensitivity Analyses In Randomized Studies

Missing Data and Sensitivity Analysesin Randomized StudiesDaniel ScharfsteinJohns Hopkins Universitydscharf@jhu.eduNovember 12, 2013ScharfsteinGSK Presentation

Missing Data MattersIIMissing outcome data are a widespread problem inrandomized trialsWe reviewed all randomized trials reporting five majorpatient-reported outcomes published in five leadinggeneral medical journals between January 1, 2008 andMarch 14, 2015III83.5% reported percentages greater than 10%,46.1% reported percentages greater than 20%23.1% reported percentages greater than 30%.ScharfsteinGSK Presentation

Missing Data MattersIIIWhile unbiased estimates of treatment effects can beobtained from trials with no missing data, this is nolonger true when data are missing on some patients.The essential problem is that inference about treatmenteffects relies on unverifiable assumptions about the natureof the mechanism that generates the missing data.While we usually know the reasons for missing data, wedo not know the distribution of outcomes for patientswith missing data, how it compares to that of patientswith observed data and whether differences in thesedistributions can be explained by the observed data.ScharfsteinGSK Presentation

Robert Temple and Bob O’Neil (FDA)I”During almost 30 years of review experience, the issue ofmissing data in . clinical trials has been a major concernbecause of the potential impact on the inferences thatcan be drawn . when data are missing . the analysisand interpretation of the study pose a challenge and theconclusions become more tenuous as the extent of’missingness’ increases.”ScharfsteinGSK Presentation

OutlineIIRecommendations from 2010 NRC ReportSensitivity AnalysisScharfsteinGSK Presentation

2010 NRC ReportIThe report, commissioned by the FDA, provides 18recommendationsScharfsteinGSK Presentation

Recommendation 1IIIThe trial protocol should explicitly define (a) theobjective(s) of the trial; (b) the associated primaryoutcome or outcomes; (c) how, when, and on whom theoutcome or outcomes will be measured; and (d) themeasures of intervention effects, that is, the causalestimands of primary interest.These measures should be meaningful for all studyparticipants, and estimable with minimal assumptions.The protocol should address the potential impact andtreatment of missing data.ScharfsteinGSK Presentation

EstimandsI1. (Difference in) Outcome Improvement for allRandomized PatientsIIIIITTInterpreted as a treatment policy.Parallel group, randomized trial in which outcome iscollected on all patients, regardless of treatmentadherence.2. (Difference in) Outcome Improvement in ToleratorsIIActive treatment run-in phase, followed by placebowashout, followed by randomizationOutcome data collected on all patients.ScharfsteinGSK Presentation

EstimandsI3. (Difference in) Outcome Improvement If All PatientsTolerated and AdheredIIIParallel group, randomized trial in which all patients areprovided adjunctive or supportive care to insuretolerability and adherence.Outcome data collected on all patients.4. (Difference in) Area Under the Outcome Curve DuringAdherenceIISimultaneously quantifies the effect of treatment onboth the outcome and the duration of tolerability oradherence in all patients.No need to collect outcome data after treatmentdiscontinuation.ScharfsteinGSK Presentation

EstimandsI5. (Difference in) Outcome Improvement DuringAdherence to TreatmentIISimultaneously quantifies the effect of treatment onboth the outcome and the duration of tolerability oradherence in all patients.No need to collect outcome data after treatmentdiscontinuation.ScharfsteinGSK Presentation

EstimandsIIEstimands 1, 4 and 5 may be influenced by bothpharmacological efficacy and tolerance and adherence.They have the potential to be be misinterpreted.Estimand 5 does not distinguish between highly effectivebut toxic treatments from a non-toxic treatment withgradual improvement over time.ScharfsteinGSK Presentation

Recommendation 2Investigators, sponsors, and regulators should design clinicaltrials consistent with the goal of maximizing the number ofparticipants who are maintained on the protocol-specifiedintervention until the outcome data are collected.ScharfsteinGSK Presentation

Ideas to Limit Missing DataIIITarget a population which is not adequately served bycurrent treatments, and hence has an incentive to remainin the study.Include a run-in period where all patients are assigned tothe active treatment, after which only individuals whotolerated and adhered to therapy are randomized to atreatment.Allow flexible dosing that accommodates individualdifferences in efficacy and tolerability, reducing thefrequency of dropout for lack of efficacy or tolerability.ScharfsteinGSK Presentation

Ideas to Limit Missing DataIIIIConsider add-on designs, where the study treatment (orplacebo) is added to an existing treatment, typically witha different mechanism of action known from previousstudies to be effective.Shorten the follow-up period for the primary outcome.Allow rescue medications, designated as components of atreatment regimen in the study protocol.For assessing long-term efficacy, where dropouts are likely,consider randomized withdrawal designs so onlyparticipants who have remained on therapy arerandomized (to continue or withdraw to placebo)ScharfsteinGSK Presentation

Ideas to Limit Missing DataIAvoid outcome measures that are likely to lead tosubstantial missing data; in some cases it may beappropriate to consider time to use of rescue treatment asan outcome measure, or discontinuation of studytreatment as a form of treatment failure.ScharfsteinGSK Presentation

Recommendation 3Trial sponsors should continue to collect information on keyoutcomes on participants who discontinue theirprotocol-specified intervention in the course of the study,except in those cases for which a compelling cost-benefitanalysis argues otherwise, and this information should berecorded and used in the analysis.ScharfsteinGSK Presentation

Recommendation 9Statistical methods for handling missing data should bespecified by clinical trial sponsors in study protocols, and theirassociated assumptions stated in a way that can beunderstood by clinicians.ScharfsteinGSK Presentation

Recommendation 10Single imputation methods like last observation carried forwardand baseline observation carried forward should not be used asthe primary approach to the treatment of missing data unlessthe assumptions that underlie them are scientifically justified.ScharfsteinGSK Presentation

Recommendation 11Parametric models in general, and random effects models inparticular, should be used with caution, with all theirassumptions clearly spelled out and justified. Models relyingon parametric assumptions should be accompanied bygoodness-of-fit procedures.ScharfsteinGSK Presentation

Recommendation 12It is important that the primary analysis of the data from aclinical trial should account for the uncertainty attributable tomissing data, so that under the stated missing dataassumptions the associated significance tests have valid type Ierror rates and the confidence intervals have the nominalcoverage properties.I For inverse probability weighting and maximum likelihoodmethods, this can be accomplished by appropriatecomputation of standard errors, using either asymptoticresults or the bootstrap.I For imputation, it s necessary to use appropriate rules formultiply imputing missing responses and combiningresults across imputed datasets because single imputationdoes not account for all sources of variability.ScharfsteinGSK Presentation

Recommendation 13Weighted generalized estimating equations methods should bemore widely used in settings when missing at random can bewell justified and a stable weight model can be determined, asa possibly useful alternative to parametric modeling.ScharfsteinGSK Presentation

Recommendation 14IIWhen substantial missing data are anticipated, auxiliaryinformation should be collected that is believed to beassociated with reasons for missing values and with theoutcomes of interest. This could improve the primaryanalysis through use of a more appropriate missing atrandom model or help to carry out sensitivity analyses toassess the impact of missing data on estimates oftreatment differences.Investigators should seriously consider following up all ora random sample of trial dropouts, who have notwithdrawn consent, to ask them to indicate why theydropped out of the study, and, if they are willing, tocollect outcome measurements from them.ScharfsteinGSK Presentation

Recommendation 15Sensitivity analyses should be part of the primary reporting offindings from clinical trials. Examining sensitivity to theassumptions about the missing data mechanism should be amandatory component of reporting.ScharfsteinGSK Presentation

ICH, EMEA and Sensitivity AnalysisII1998 International Conference of Harmonization (ICH)Guidance document (E9) entitled ”Statistical Principles inClinical Trials” states: ”it is important to evaluate therobustness of the results to various limitations of the data,assumptions, and analytic approaches to data analysis”European Medicines Agency 2009 draft ”Guideline onMissing Data in Confirmatory Clinical Trials” states ”[i]nall submissions with non-negligible amounts of missingdata sensitivity analyses should be presented as supportto the main analysis.”ScharfsteinGSK Presentation

PCORI and Sensitivity AnalysisIIIIn 2012, Li et al. issued the report ”Minimal Standards inthe Prevention and Handling of Missing Data inObservational and Experimental Patient CenteredOutcomes Research”This report, commissioned by PCORI, provides 10standards targeted at (1) design, (2) conduct, (3) analysisand (4) reporting.Standard 8 echoes the NRC report, statingIExamining sensitivity to the assumptions about themissing data mechanism (i.e., sensitivity analysis) shouldbe a mandatory component of the study protocol,analysis, and reporting.ScharfsteinGSK Presentation

Schizophrenia Clinical TrialIIIIMulti-center, randomized clinical trial to assess the safetyand efficacy of a test drug (81 subjects) relative toplacebo (78 subjects) for individuals suffering from acuteschizophrenia.The primary instrument used to assess the severity ofsymptoms was the positive and negative syndrome scale(PANSS). Higher scores worse.Measurements were scheduled to be collected at baseline,day 4 after baseline, and weeks 1, 2, 3, and 4 afterbaseline.One goal was to compare the two treatment groups withrespect the mean PANSS score at week 4 (6th timepoint).ScharfsteinGSK Presentation

(9) (4) 80 120 (10) (7)(6) (4) (54)4060(58)140160 100100 (4)80 Mean PANSS by Last Observation(3) 60140120 40Mean PANSS by Last Observation160Problem: Missing Data123456Figure: Placebo1234Figure: TestScharfsteinGSK Presentation56

Fundamental IssueIIIEven with infinite data, we cannot learn about thetreatment-specific mean PANSS score at week 4.We don’t know the distribution of PANSS scores forindividuals who have dropped out prior to week 4.Need to make assumptions!ScharfsteinGSK Presentation

Sensitivity AnalysisThe set of possible assumptions about the missing datamechanism is very large and cannot be fully explored. Thereare different approaches to sensitivity analysis:IIIAd-hocLocalGlobalScharfsteinGSK Presentation

Ad-hoc Sensitivity AnalysisIAnalyzing data using a few different analytic methods,such as last or baseline observation carried forward,complete or available-case analysis, mixed models ormultiple imputation, and evaluate whether the resultinginferences are consistent.ScharfsteinGSK Presentation

Local Sensitivity AnalysisISpecify a reasonable benchmark assumption (e.g., missingat random) and evaluate the robustness of the resultswithin a small neighborhood of this assumption.ScharfsteinGSK Presentation