Systematic Uncertainties: Principle And Practice
Systematic Uncertainties:Principle and PracticeOutline1. Introduction to Systematic Uncertainties2. Taxonomy and Case Studies3. Issues Around Systematics4. The Statistics of Systematics5. SummaryPekka K. Sinervo,F.R.S.C.Rosi & Max Varon Visiting ProfessorWeizmann Institute of Science&Department of PhysicsUniversity of Toronto18 Dec 08Weizmann Institute of Science1
Introduction Systematic uncertainties play key role in physicsmeasurements– Few formal definitions exist, much “oral tradition”– “Know” they are different from statistical uncertaintiesRandom Uncertainties Arise from stochasticfluctuationsUncorrelated with previousmeasurementsWell-developed theoryExamples measurement resolutionfinite statisticsrandom variations in systemSystematic Uncertainties Due to uncertainties in theapparatus or modelUsually correlated withprevious measurementsLimited theoretical frameworkExamples calibrations uncertaintiesdetector acceptancepoorly-known theoreticalparametersWeizmann Institute of Science2
Literature Summary Increasing literature on the topic of “systematics”A representative list:––––––––––R.D.Cousins & V.L. Highland, NIM A320, 331 (1992).C. Guinti, Phys. Rev. D 59 (1999), 113009.G. Feldman, “Multiple measurements and parameters in the unified approach,”presented at the FNAL workshop on Confidence Limits (Mar 2000).R. J. Barlow, “Systematic Errors, Fact and Fiction,” hep-ex/0207026 (Jun 2002), andseveral other presentations in the Durham conference.G. Zech, “Frequentist and Bayesian Confidence Limits,” Eur. Phys. J, C4:12 (2002).R. J. Barlow, “Asymmetric Systematic Errors,” hep-ph/0306138 (June 2003).A. G. Kim et al., “Effects of Systematic Uncertainties on the Determination ofCosmological Parameters,” astro-ph/0304509 (April 2003).J. Conrad et al., “Including Systematic Uncertainties in Confidence Interval Constructionfor Poisson Statistics,” Phys. Rev. D 67 (2003), 012002G.C.Hill, “Comment on “Including Systematic Uncertainties in Confidence IntervalConstruction for Poisson Statistics”,” Phys. Rev. D 67 (2003), 118101.G. Punzi, “Including Systematic Uncertainties in Confidence Limits”, CDF Note inpreparation.Weizmann Institute of Science3
I. Case Study #1: W Boson CrossSection Rate of W boson production– Count candidates Ns Nb– Estimate backgroundNb & signal efficiency ε" ( N c # N b ) ( L)– Measurement reported as!" 2.64 0.01 (stat) 0.18 (syst) nb– Uncertainties are!!" stat # " 0stat 1/N c" syst # " 0syst22( N b /N b ) ( % /%) ( L /L)2Weizmann Institute of Science4
Definitions are Relative Efficiency uncertainty estimated using Zboson decays– Count up number of Z candidates NZcand Can identify using charged tracks Count up number reconstructed NZreconrecon" NZcand # " %NZNZrecon(NZcand& NZN Z cand– Redefine uncertainties2" stat # " 0 1/N c ( % /%)–22" syst # " 0 ( N b /N b ) ( L /L)!!recon)Lessons: Some systematic uncertaintiesare really “random” Good to know this Uncorrelated Know how they scale May wish to redefine Call these“CLASS 1” SystematicsWeizmann Institute of Science5
Top Mass Good Example Top mass uncertainty in template analysis– Statistical uncertainty from shape ofreconstructed mass distribution andstatistics of sample– Systematic uncertainty coming from jetenergy scale (JES) Determined by calibration studies,dominated by modelling uncertainties 5% systematic uncertainty Latest techniques determine JESuncertainty from dijet mass peak (W- jj)–Turn JES uncertainty into a largelystatistical one– Introduce other smaller systematicsM top 171.8 1.9(stat JES) 1.0 (syst) GeV/c 2 171.9 2.1 GeV/c 2Weizmann Institute of Science!6
Case Study #2: BackgroundUncertainty Look at same W cross section analysis– Estimate of Nb dominated by QCD backgrounds Candidate event– Have non-isolated leptons– Less missing energy Assume that isolationand MET uncorrelatedHave to estimate theuncertainty on NbQCD– No direct measurementhas been made to verify the model– Estimates using Monte Carlo modelling have largeuncertaintiesWeizmann Institute of Science7
Estimation of Uncertainty Fundamentally different class of uncertainty– Assumed a model for data interpretation– Uncertainty in NbQCD depends on accuracy of model– Use “informed judgment” to place bounds on one’signorance Vary the model assumption to estimate robustness Compare with other methods of estimation Difficult to quantify in consistent manner– Largest possible variation? Asymmetric?– Estimate a “1 σ” interval?– Take " # ?12Lessons: Some systematic uncertaintiesreflect ignorance of one’s data Cannot be constrained byobservations Call these“CLASS 2” SystematicsWeizmann Institute of Science!8
Case Study #3: Boomerang CMBAnalysis Boomerang is one of severalCMB probes– Mapped CMB anisoptropy– Data constrain models of theearly universe Analysis chain:– Produce a power spectrum forthe CMB spatial anisotropy Remove instrumental effects through a complexsignal processing algorithm– Interpret data in context of many models withunknown parametersWeizmann Institute of Science9
Incorporation of ModelUncertainties Power spectrum extractionincludes all instrumentaleffects– Effective size of beam– Variations in data-takingprocedures Use these data to extract7 cosmological parameters– Take Bayesian approach Family of theoretical models defined by 7 parameters Define a 6-D grid (6.4M points), and calculate likelihoodfunction for eachWeizmann Institute of Science10
Marginalize Posterior Probabilities Perform a Bayesian“averaging” over a gridof parameter values– Marginalize w.r.t. theother parameters NB: instrumentaluncertainies includedin approximate manner– Chose various priorsin the parameters Comments:– Purely Bayesian analysis withno frequentist analogue– Provides path for inclusion ofadditional data (eg. WMAP)Lessons: Some systematic uncertaintiesreflect paradigm uncertainties No relevant concept of afrequentist ensemble Call these“CLASS 3” SystematicsWeizmann Institute of Science11
Proposed Taxonomy forSystematic Uncertainties Three “classes” of systematic uncertainties– Uncertainties that can be constrained by ancillarymeasurements– Uncertainties arising from model assumptions orproblems with the data that are poorly understood– Uncertainties in the underlying models Estimation of Class 1 uncertainties straightforward– Class 2 and 3 uncertainties present unique challenges– In many cases, have nothing to do with statisticaluncertainties Driven by our desire to make inferences from the datausing specific modelsWeizmann Institute of Science12
II. Estimation Techniques No formal guidance on how to define a systematicuncertainty– Can identify a possible source of uncertainty– Many different approaches to estimate their magnitude# Determine maximum effect Δ" ?2General rule:#" ?– Maintain consistency with definition of12statistical intervals– Field is pretty glued to 68% confidence intervals– Recommend attempting to reflect that! in magnitudes ofsystematic uncertainties– Avoid tendency to be “conservative”Weizmann Institute of Science13
Estimate of BackgroundUncertainty in Case Study #2 Look at correlation of Isolation and MET– Background estimateincreases as isolation“cut” is raised– Difficult to measure oraccurately model Background comesprimarily from veryrare jet events withunusual properties Very model-dependent Assume a systematic uncertainty representingthe observed variation– Authors argue this is a “conservative” choiceWeizmann Institute of Science14
Cross-Checks Vs Systematics R. Barlow makes the point in Durham(PhysStat02)– A cross-check for robustness is not an invitation tointroduce a systematic uncertainty Most cross-checks confirm that interval or limit is robust,– They are usually not designed to measure a systematicuncertainty More generally, a systematic uncertainty should– Be based on a hypothesis or model with clearly statedassumptions– Be estimated using a well-defined methodology– Be introduced a posteriori only when all else has failedWeizmann Institute of Science15
III. Statistics of SystematicUncertainties Goal has been to incorporate systematic uncertaintiesinto measurements in coherent manner– Increasing awareness of need for consistent practice Frequentists: interval estimation increasingly sophisticated– Neyman construction, ordering strategies, coverage properties Bayesians: understanding of priors and use of posteriors– Objective vs subjective approaches, marginalization/conditioning– Systematic uncertainties threaten to dominate as precisionand sensitivity of experiments increase There are a number of approaches widely used– Summarize and give a few examples– Place it in context of traditional statistical conceptsWeizmann Institute of Science16
Formal Statement of the Problem Have a set of observations xi, i 1,n– Associated probability distribution function (pdf) andlikelihood functionp x " # L " p x "(i)( ) i (i)Depends on unknown random parameter θHave some additional uncertainty in pdf!– Introduce a second unknown parameter λ L (", # ) p( x i ", # )i In some cases, one can identify statistic yj thatprovides information about λ!L (", # ) p( x i , y j ", # )i, j– Can treat λ as a “nuisance parameter”!Weizmann Institute of Science17
Bayesian Approach Identify a prior π(λ) for the “nuisance parameter” λ– Typically, parametrize as either a Gaussian pdf or a flatdistribution within a range (“tophat”)– Can then define Bayesian posteriorL (", # ) ( #) d" d#– Can marginalize over possible values of λ Use marginalized posterior to set Bayesian credibilityintervals,estimate parameters, etc.! Theoretically straightforward .– Issues come down to choice of priors for both θ, λ No widely-adopted single choice Results have to be reported and compared carefully toensure consistent treatmentWeizmann Institute of Science18
Frequentist Approach Start with a pdf for data p( x i, y j ", #)– In principle, this would describe frequencydistributions of data in multi-dimensional space– Challenge is take accountof nuisance parameter!– Consider a toy modelp( x, y µ," ) G( x # (µ " ),1) G( y # " ,s) ! Parameter s is Gaussianwidth for νLikelihood function (x 10, y 5)– Shows the correlation– Effect of unknown νWeizmann Institute of Science19
Formal Methods to EliminateNuisance Parameters Number of formal methods exist to eliminatenuisance parameters– Of limited applicability given the restrictions– Our “toy example” is one such case Replace x with t x-y and parameter ν with2µsv'" # 1 s2&)22tss p( t, y µ,# ') G t % µ, 1 s G( y % # ' 2,2 1 s'1 s *()Factorized pdf and can now integrate over ν’ Note that pdf for µ has larger width, as expected– In practice, one often loses information using thistechnique !Weizmann Institute of Science20
Alternative Techniques forTreating Nuisance Parameters Project Neyman volumes onto parameter ofinterest– “Conservative interval”– Typically over-covers,possibly badly Choose best estimate ofnuisance parameter– Known as “profile method”– Coverage propertiesFrom G. Zechrequire definition of ensemble– Can possible under-cover when parameters stronglycorrelated Feldman-Cousins intervals tend to over-cover slightly(private communication)Weizmann Institute of Science21
Example: Solar Neutrino GlobalAnalysis Many experiments have measured solar neutrino flux– Gallex, SuperKamiokande, SNO, Homestake, SAGE, etc.– Standard Solar Model (SSM) describes ν spectrum– Numerous “global analyses” that synthesize these Fogli et al. have detailed one such analysis– 81 observables from these experiments– Characterize systematic uncertainties through 31 parameters 12 describing SSM spectrum 11 (SK) and 7 (SNO) systematic uncertainties Perform a χ2 analysis– Look at χ2 to set limits on parametersHep-ph/0206162, 18 Jun 2002Weizmann Institute of Science22
Formulation of χ2 In formulating χ2, linearize effects of the systematicuncertainties on data and theory comparison" 2pull- '/ ) R exp t % R theor % & (c k )nnn k/N )# min{ } /&)un/n 1 )/ )(.0*22,K2,2, & k 2,, k 1 22 1Uncertainties un for each observable– Introduce “random” pull ξk for each systematic Coefficients ckn to parameterize effect on nth observable Minimize χ2 with respect to ξk Look at contours of equal Δ χ2 !Weizmann Institute of Science23
Solar Neutrino Results Can look at “pulls” at χ2minimum– Have reasonable distribution– Demonstrates consistency ofmodel with the variousmeasurements– Can also separate Agreement with experiments Agreement with systematicuncertaintiesWeizmann Institute of Science24
Pull Distributions for Systematics Pull distributions for ξkalso informative– Unreasonably small variations– Estimates are globally tooconservative?– Choice of central valuesaffected by data Note this is NOT ablind analysis But it gives us someconfidence that intervalsare realisticWeizmann Institute of Science25
Typical Solar Neutrino Contours Can look at probabilitycontours– Assume standard χ2 form– Probably very smallprobability contours haverelatively largeuncertaintiesWeizmann Institute of Science26
Hybrid Techniques A popular technique (Cousins-Highland) does an“averaging” of the pdf– Assume a pdf for nuisance parameter g(λ)– “Average” the pdf for data xp CH ( x " ) #% p( x ", ) g( ) d – Argue this approximates an ensemble where Each measurement uses an apparatus that differs in!parameter λ– The pdf g(λ) describes the frequency distribution Resulting distribution for x reflects variations in λIntuitively appealingSee, for example, J. Conrad et al.– But fundamentally a Bayesian approach– Coverage is not well-definedWeizmann Institute of Science27
Summary HEP & Astrophysics becoming increasingly“systematic” about systematics– Recommend classification to facilitate understanding Creates more consistent framework for definitions Better indicates where to improve experiments– Avoid some of the common analysis mistakes Make consistent estimation of uncertainties Don’t confuse cross-checks with systematic uncertainties Systematics naturally treated in Bayesian framework– Choice of priors still somewhat challenging Frequentist treatments are less well-understood– Challenge to avoid loss of information– Approximate methods exist, but probably leave the “truefrequentist” unsatisfiedWeizmann Institute of Science28
Weizmann Institute of Science 3 Literature Summary Increasing literature on the topic of “systematics” A representative list: – R.D.Cousins & V.L. Highland, NIM A320, 331 (1992). – C. Guinti, Phys. Rev. D 59 (1999), 113009. – G. Feldman, “Mult
resolution of an instrument, the Poisson uctations associated with measurements involving nite sample sizes and random variations in the system one is ex-amining. Systematic uncertainties, on the other hand, arise from uncertainties associated with the nat
outcrops, trenches, boreholes, geophysics (surface and down hole), and excavations are not sufficient to provide a complete picture of the ro ck mass of interest. Uncertainties are inherent when dealing with rock masses; uncertainties in the material itself, uncertainties in data collection and testing and uncertainties in model prediction.
effective compliance practices . principle ii – sponsored program management. principle iii-financial administration in the form of principle iv-instit. utional. infrastructure . principle v-assessments and audits principle vi-integrity and protections . regulations . principle vii-intellectual property . management . principle viii-export .
2. The Sources of Evangelical Systematic Theology 3. The Structure of Evangelical Systematic Theology 4. The Setting of Evangelical Systematic Theology 5. The Satisfaction of Evangelical Systematic Theology Study 1: The Nature of Systematic Theology & the Doctrine of Revelation "God is most glorified in us as we are most satisfied in him." John .
Principle 1: Identify activities and assess risks 15 Principle 2: Select standards 18 PHASE TWO: Design the CMS 21 Principle 3: Develop procedures and methods 21 Principle 4: Decide how to meet the standards 23 Principle 5: Establish requirements for training, development and assessment 25 Principle 6: Ma
CONTENTS Foreword and Acknowledgments 1 Abbreviations 3 Preamble 4 Core Principles and Compliance Assessment 8 PRINCIPLE 1 Public Policy Objectives 8 PRINCIPLE 2 Mandate and Powers 11 PRINCIPLE 3 Governance 15 PRINCIPLE 4 Relationships with Other Safety-Net Participants 19 PRINCIPLE 5 Cross-Border Issues 22 PRINCIPLE 6 Deposit Insurer’s Role in Contingency Planning
Handling People Principle 1: Don't criticize, condemn, or complain. Principle 2: Give honest and sincere appreciation Principle 3: Arouse in the person an eager want Six Ways to Make People Like You Principle 1: person. Become generally interested in others Principle 2: Smile Principle 3: Remember that a person's name is to that person the
3.2.2 Principle 2: You should be objectiveError! Bookmark not defined. 3.2.3 Principle 3: You should do no harm 3.2.4 Error! Bookmark not defined.Principle 4: You should treat people with dignity . Error! Bookmark not defined. 3.2.5 Principle 5: You should build on the community's strengths 3.2.6 Principle 6: You should empower
