Bayesian Statistics - An Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theorySolution for a second testISuppose that we have a second positive test resultp( D)p(D)p( D)p(D) p( D̄)p(D̄)0.992 0.02 0.992 0.02 0.012 0.98 0.9950p(D ) Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theorySolution for a second testISuppose that we have a second positive test resultp( D)p(D)p( D)p(D) p( D̄)p(D̄)0.992 0.02 0.992 0.02 0.012 0.98 0.9950p(D ) IThus although the test is very accurate we need two positiveresults before we can say with confidence that the person hasthe disease.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryDensity form of Bayes TheoremILet X , θ be two continuous r.v.’s (possibly multivariate).f (θ x) f (θ)f (x θ)f (θ)f (x θ) Rf (x)f (θ0 )f (x θ0 )dθ0Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryDensity form of Bayes TheoremILet X , θ be two continuous r.v.’s (possibly multivariate).f (θ x) If (θ)f (x θ)f (θ)f (x θ) Rf (x)f (θ0 )f (x θ0 )dθ0Posterior Likelihood PriorDr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryDensity form of Bayes TheoremILet X , θ be two continuous r.v.’s (possibly multivariate).f (θ x) If (θ)f (x θ)f (θ)f (x θ) Rf (x)f (θ0 )f (x θ0 )dθ0Posterior Likelihood PriorIp(θ x) p(x θ)p(θ)Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryThe Likelihood PrincipleITo illustrate the difference between the classical and Bayesianapproaches we start with an example.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryThe Likelihood PrincipleIITo illustrate the difference between the classical and Bayesianapproaches we start with an example.Suppose we toss a drawing pin and get 9 ‘ups’ and 3 ‘downs’.We denote ‘up’ by U and ‘down’ by D. Is the pin unbiased?Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryThe Likelihood PrincipleIIITo illustrate the difference between the classical and Bayesianapproaches we start with an example.Suppose we toss a drawing pin and get 9 ‘ups’ and 3 ‘downs’.We denote ‘up’ by U and ‘down’ by D. Is the pin unbiased?Classically we might test H0 : p 12 versus H1 : p 12 wherep p(U). The probability of the observed result or somethingmore extreme (tail area) if H0 is true is 12121212121 32102which level.2994096 7.3%. Thus we would accept H0 at the 5%Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryHowever this assumes that we did the experiment bydeciding to toss the drawing pin 12 times.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryHowever this assumes that we did the experiment bydeciding to toss the drawing pin 12 times.What if we decided to toss the pin until we achieved 3 D’s?Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryHowever this assumes that we did the experiment bydeciding to toss the drawing pin 12 times.What if we decided to toss the pin until we achieved 3 D’s?Now the probability of the observed result or somethingmore extreme if H0 is true is 12 13 14121131111 ···222222We may calculate the probability of the complement of thisevent by 11 10 31019121 ··· 222222It follows that the P-value isreject H0 at the 5% level.Dr Lawrence Pettit1344096 3.25%. So we wouldBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIn order to perform a significance test we are required tospecify a sample space i.e. the space of all possibleoutcomes.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIIn order to perform a significance test we are required tospecify a sample space i.e. the space of all possibleoutcomes.IPossibilities for the drawing pin are:(i) {(u, d) : u d 12},(ii) {(u, d) : d 3},or if I carry on tossing the pin until my coffee is ready, sothat there is a random stopping point(iii) {all (u, d)}.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryThe Bayesian analysis of this problem is somewhat different.Let θ be the chance that the pin lands up.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIThe Bayesian analysis of this problem is somewhat different.Let θ be the chance that the pin lands up.Iθ is a “long run frequency” of U’s. It is an objective propertyof the pin. It does not depend on You.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIThe Bayesian analysis of this problem is somewhat different.Let θ be the chance that the pin lands up.Iθ is a “long run frequency” of U’s. It is an objective propertyof the pin. It does not depend on You.IYou have beliefs about θ which you express in the form of aprobability density function (pdf) p(θ). You use Bayestheorem to update your beliefs.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIThe Bayesian analysis of this problem is somewhat different.Let θ be the chance that the pin lands up.Iθ is a “long run frequency” of U’s. It is an objective propertyof the pin. It does not depend on You.IYou have beliefs about θ which you express in the form of aprobability density function (pdf) p(θ). You use Bayestheorem to update your beliefs.Ip(θ data) p(data θ)p(θ)θ9 (1 θ)3 p(θ)The sampling rule is irrelevant.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIn deriving the posterior distribution the only contributionfrom the data is through the likelihood p(data θ). Thus aBayesian inference, which will depend only on the posteriordistribution, obeys the likelihood principle.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIIn deriving the posterior distribution the only contributionfrom the data is through the likelihood p(data θ). Thus aBayesian inference, which will depend only on the posteriordistribution, obeys the likelihood principle.IThis roughly says that if two experiments give the samelikelihoods then the inferences we make should be the same ineach case.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIIn deriving the posterior distribution the only contributionfrom the data is through the likelihood p(data θ). Thus aBayesian inference, which will depend only on the posteriordistribution, obeys the likelihood principle.IThis roughly says that if two experiments give the samelikelihoods then the inferences we make should be the same ineach case.IClassical hypothesis tests or confidence intervals violate thelikelihood principle.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryWe need to give a prior distribution for θ.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryWe need to give a prior distribution for θ.Suppose we takep(θ) Γ(a b) a 1θ (1 θ)b 1 a, b 0Γ(a)Γ(b)ie a beta distribution with mean a/(a b) and varianceab1.a ba ba b 1We shall write this distribution as Be(a, b).Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryWe need to give a prior distribution for θ.Suppose we takep(θ) Γ(a b) a 1θ (1 θ)b 1 a, b 0Γ(a)Γ(b)ie a beta distribution with mean a/(a b) and varianceab1.a ba ba b 1IWe shall write this distribution as Be(a, b).It then follows thatp(θ data) θ9 a 1 (1 θ)3 b 1 .That is Be(9 a, 3 b). Thus if we take a beta prior for θ weshall obtain a beta posterior.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryThe choice a b 1 gives a uniform distribution as the prior,ie we think any value of θ is equally likely. More realisticallyfor a pin we might take a b 2, reflecting a belief that wethink θ is more likely to be near 0.5 than 0 or 1 but not beingvery sure.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIThe choice a b 1 gives a uniform distribution as the prior,ie we think any value of θ is equally likely. More realisticallyfor a pin we might take a b 2, reflecting a belief that wethink θ is more likely to be near 0.5 than 0 or 1 but not beingvery sure.IOthers might choose an asymmetric prior, perhaps arguingthat a pin with a very long point would very likely land downso a pin with any point would land down more often than up. Iam not convinced by this argument but it shows that differentpeople do have different beliefs and one of the advantages ofthe Bayesian approach is that the analysis can reflect these.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIf we were throwing a coin rather than a pin then we wouldalmost certainly choose a different prior. We have much moreexperience throwing coins than pins and are much more surethat θ, the chance the coin will land heads, is close to 0.5.Thus we might take a b 50 say.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIf we were throwing a coin rather than a pin then we wouldalmost certainly choose a different prior. We have much moreexperience throwing coins than pins and are much more surethat θ, the chance the coin will land heads, is close to 0.5.Thus we might take a b 50 say.The posteriors we get for the pin and the coin will be verydifferent. For the pin the posterior mean will be 11/16 0.6875 and the posterior variance11/16 5/16 1/17 0.0126. For the coin the posteriormean will be 59/112 0.527 which is close to 0.5.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryIf we were throwing a coin rather than a pin then we wouldalmost certainly choose a different prior. We have much moreexperience throwing coins than pins and are much more surethat θ, the chance the coin will land heads, is close to 0.5.Thus we might take a b 50 say.The posteriors we get for the pin and the coin will be verydifferent. For the pin the posterior mean will be 11/16 0.6875 and the posterior variance11/16 5/16 1/17 0.0126. For the coin the posteriormean will be 59/112 0.527 which is close to 0.5.The classical unbiased estimate is 9/12 0.75 if the numberof throws is fixed, or 9/11 0.818 if we continue until wehave three ‘failures’. The classical answers are the samewhether for pins or coins, ignoring the extra information thatwe have.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryPlot for pinsDr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryPlot for coinsDr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryBy taking mixtures of conjugate priors we can represent morerealistic beliefs.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryBy taking mixtures of conjugate priors we can represent morerealistic beliefs.As an example consider the result when a coin is spun on itsedge. Experience has shown that when spinning a coin theproportion of heads is more likely to be 1/3 or 2/3 than 1/2.Therefore a bimodal prior seems appropriate. Since spinningcoins will be Bernoulli trials the beta distribution will beconjugate.Dr Lawrence PettitBayesian Statistics - an Introduction

Bayesian StatisticsStochastic Simulation - Gibbs samplingIIIWhat is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theoryBy taking mixtures of conjugate priors we can represent morerealistic beliefs.As an example consider the result when a coin is spun on itsedge. Experience has shown tha

Bayesian Statistics Stochastic Simulation - Gibbs sampling Bayesian Statistics - an Introduction Dr Lawrence Pettit School of Mathematical Sciences, Queen Mary, University of London July 22, 2008 Dr Lawrence Pettit Bayesian Statistics - an Introduction