Lecture 2 Bayesian Linear Regression

Advanced Probabilistic Machine LearningLecture 2 – Bayesian linear regressionNiklas WahlströmDivision of Systems and ControlDepartment of Information TechnologyUppsala talog/nikwa7781 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Summary of lecture 1 (I/IV)Conditional probability is defined asp(x y) p(x, y)p(y)where p(y) 6 0Marginalization is defined asp(x) XZp(x, y) or p(x) p(x, y)dyyyMuch of the probability theory can be derived from these two rules.Bayes’ theorem is derived by using the def. of conditional probabilitytwicep(x y) 2 / 38niklas.wahlstrom@it.uu.sep(y x)p(x)p(y)Bayesian linear regression

Summary of lecture 1 (II/IV)In this course we solve problems using Bayes’ theoremp(θ D) p(D θ)p(θ)p(D) D : observed data θ : parameters of some model explaining the data p(θ): prior belief of parameters before we collected any data p(θ D): posterior belief of parameters after inferring data p(D θ): likelihood of the data in view of the parameters p(D): The marginal likelihood3 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Summary of lecture 1 (III/IV)If we view the quantities as functions of θ , we can disregard thenormalization constant p(y).p(θ D) p(D θ) p(θ) {z } {z } {z}posteriorlikelihood priorConjugate prior A prior ensuring that the posterior and the prior belong to the same probability distribution family.Example: Beta-BinomialBeta (µ; a , b ) Bin (m; N, µ) Beta (µ; a, b) {zposterior} {zlikelihood} {zpriora a m}b b N mBeta distribution is a conjugate prior to the binomial likelihood.4 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Summary of lecture 1 (IV/IV)LikelihoodfunctionPriorp(µ)p(m µ)p(µ m)Beta (µ; a, b)a 1, b 1Bin (m; N, µ)m 1, N 1Beta (µ; a , b )a 2, b 12210Posterior 00.5µ1210 00.5µ11000.5µ1Assume you get N 1 data point, of which m 1 is head, D {1}.5 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Summary of lecture 1 (IV/IV)LikelihoodfunctionPriorp(µ)p(m µ)p(µ m)Beta (µ; a, b)a 1, b 1Bin (m; N, µ)m 4, N 5Beta (µ; a , b )a 5, b 22210Posterior 00.5µ1210 00.5µ11000.5µ1Assume you get N 5 data points, of which m 4 are heads,D {1, 0, 1, 1, 1}.5 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Congenitalrmatosis dyskeratosisBenignSupervised machine learningB-cellLearninga model from labeled data.T-cellLabels e.g. mat,mirror, boat, . . .AngiosarcomaDermalcarcinomaEpidermalBasal cellcarcinomaUnseen data?LearningMerkel cellalgorithmMalignantMalignantmaTraining dataCutaneouslymphomaPredicting output of newdata based on this model.ModelpredictionModelSquamouscellcarcinomaHow do we rephrase supervised machine learning as a within theonomy and example testexample images from two disease classes. These test imagprobabilistic methodology?ructured taxonomy of skindifficulty of malignant versus benign discernment for theeases and is organized basedcritical classification tasks we consider: epidermal lesionsd indicates malignant,lesions and melanocytic lesions visualized with a dermos6 / 38niklas.wahlstrom@it.uu.selinear regressiononditionsthat can be either.images reprinted with permission fromBayesianthe EdinburghD

Supervised machine learning –Probabilistic perspectiveGiven: Data of inputs & outputs D {(x1 , y1 ), . . . , (xN , yN )}.Task: Predict the output y for a new unseen input x .Solution:1. Likelihood Define the likelihoodp(y θ, X)2. Prior Define the prior p(θ) xT1 X . ,xTNy1 y . yN3. Learning Do inference by applying Bayes’ theoremp(θ y, X) p(y θ, X)p(θ)4. Prediction Compute predictive distribution by marginalizingZp(y x , y, X) 7 / 38niklas.wahlstrom@it.uu.sep(y θ, x )p(θ y, X)dθBayesian linear regression

Example: Linear regression model Recall the linear regression from lecture 2 in the SML course Now we introduce a prior over the parameter wLinear regression modelyn wT xn εn ,εn N (0, σ 2 ),n 1, ., Nw p(w).Present assumptions:1. yn – observed random variable.2. w – unknown deterministic3. xn – known deterministic variable.4. εn – unknown random variable.5. σ – known deterministic variable.8 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Linear regression: Maximum likelihoodTwo equivalent ways of expressing the linear regression model:21. yn wT xn εn , εn N (0, σ )T22. p(yn w) N yn ; w xn , σ .The likelihood p(y w) is given byp(y w) NYp(yn w) n 1NY N yn ; w T xn , σ 2n 12 N y; Xw, σ IN . xT1 X . ,xTNy1 y . yNThe solution if found by maximizingthe likelihoodŵ arg max p(y w)w9 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Example: Linear regression model Recall the linear regression from lecture 2 in the SML course Now we introduce a prior over the parameter wBayesian linear regression modelyn wT xn εn ,εn N (0, σ 2 ),n 1, ., Nw p(w).Present assumptions:1. yn – observed random variable.2. w – unknown random variable. (difference from SML)3. xn – known deterministic variable.4. εn – unknown random variable.5. σ – known deterministic variable.10 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Bayesian linear regression modelRemember Bayes’ theoremp(w y) p(y w)p(w)p(y) Prior distribution: p(w) describes the knowledge we haveabout w before observing any data. Likelihood: p(y w) described how “likely” the observed data isfor a particular parameter value. Posterior distribution: p(w y) summarize all our knowledgeabout w from the observed data and the model.In Bayesian linear regression we use a Gaussian distribution as priorp(w) N (w; m0 , Σ0 )11 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Scalar Gaussian (Normal) distributionFor a scalar variable x, the Gaussian distribution can be written on theform2(x µ) 1e 2σ2N x; µ, σ 2 2 2πσ{z } µ is the mean (expected value σ is the standard deviationyof the distribution) σ 2 is the variance Z is the normalizationconstantWhat if x is a vector x x112 / 38niklas.wahlstrom@it.uu.se0.0 0.2 0.4 0.6 0.8 1.0Zµ 1, σ2 0.5µ 6, σ2 0.2µ 3, σ2 4 402xx2 · · · xD46 T8 10?Bayesian linear regression

Multivariate GaussianFor a D -dimensional vector x, the multivariate Gaussian distributioncan be written on the form N (x; µ, Σ) 1 1 exp (x µ)T Σ 1 (x µ) .D/2 {z}2(2π)det Σ {z}quadratic formZ µ is the mean vector Σ is the covariance matrix Z is the normalization constantGaussian equadratic form13 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Multivariate Gaussian0.3Value of PDFValue of PDF0.20.150.10.050.20.105055500x2-5 Σ 14 / 38niklas.wahlstrom@it.uu.se-51 00 1x10x2 0-5 Σ -5x1 1 0.40.4 0.5Bayesian linear regression

Partitioned Gaussian – marginalizationPartition the Gaussian random vector x N (µ, Σ), where x Rninto two sets of random variables xa Rna and xb Rnb , xax ,xb µaµ ,µb Σaa Σab.Σ Σba ΣbbTask: Compute the marginal distribution p(xa ),Zp(xa ) 15 / 38niklas.wahlstrom@it.uu.sep(xa , xb )dxb .Bayesian linear regression

Partitioned Gaussian – marginalizationTheorem 1 (Marginalization)Partition the Gaussian random vector x N (µ, Σ) according to xax ,xb µ µa,µb Σaa ΣabΣ .Σba ΣbbThe marginal distribution p(xa ) is then given byp(xa ) N (xa ; µa , Σaa ) .16 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Partitioned Gaussian – marginalizationxa17 / 38niklas.wahlstrom@it.uu.sexbBayesian linear regression

Partitioned Gaussian – Theoremsp(xa , xb )Thm 1p(xa )p(xb )18 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Partitioned Gaussian – conditioningTheorem 2 (Conditioning)Partition the Gaussian random vector x N (µ, Σ) according to xax ,xb µ µa,µb Σaa Σab.Σ Σba ΣbbThe conditional distribution p(xa xb ) is then given by p(xa xb ) N xa ; µa b , Σa b ,µa b µa Σab Σ 1bb (xb µb ),Σa b Σaa Σab Σ 1bb Σba .19 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Partitioned Gaussian – conditioningxa20 / 38niklas.wahlstrom@it.uu.sexbBayesian linear regression

Partitioned Gaussian – Theoremsp(xa , xb )Thm 1Thm 221 / 38p(xa )p(xb xa )p(xb )p(xa xb )niklas.wahlstrom@it.uu.seBayesian linear regression

Affine transformation of multivar. GaussWe can also do the opposite:compute p(xa , xb ) based on p(xb xa ) and p(xa )Theorem 3 (Affine transformation)Assume that xa , as well as xb conditioned on xa , are Gaussiandistributed according top(xa ) N (xa ; µa , Σa ) , p(xb xa ) N xb ; Mxa , Σb a .Then the joint distribution of xa and xb is xaµap(xa , xb ) N;,RxbMµawith22 / 38niklas.wahlstrom@it.uu.seΣaΣa MTMΣa Σb a MΣa MT R Bayesian linear regression

Partitioned Gaussian – Theoremsp(xa , xb )Thm 1Thm 2Thm 323 / 38p(xa )p(xb xa )p(xb )p(xa xb )niklas.wahlstrom@it.uu.seBayesian linear regression

Bayesian linear regression modelBayesian linear regression model:y n w T x n εn ,εn N (0, β 1 ),w p(w).β σ 2 is calledthe precisionThe probabilistic model is given by: p(y w) N y; Xw, β 1 IN ,p(w) N (w; m0 , S0 ) ,likelihoodprior distributionTask: Compute the posterior distribution: p(w y).24 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Partitioned Gaussian – Theoremsp(xa , xb )Thm 1Thm 2Thm 3p(xa )p(xb xa )Col 1 Thm 3 Thm 2p(xb )25 / 38niklas.wahlstrom@it.uu.sep(xa xb )Bayesian linear regression

Affine transformation of multivar. GaussBy combining Theorem 3 and Theorem 2 we getCorollary 1 (Affine transformation – conditional)Assume that xa , as well as xb conditioned on xa , are Gaussiandistributed according top(xa ) N (xa ; µa , Σa ) , p(xb xa ) N xb ; Mxa b, Σb a .Then the conditional distribution of xa given xb is p(xa xb ) N xa ; µa b , Σa b ,with T 1µa b Σa b Σ 1µ MΣ(x b),aab a b 1T 1Σa b Σ 1.a M Σb a M26 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

Bayesian linear regressionThe probabilistic model is given by: p(y w) N y; Xw, β 1 IN ,p(w) N (w; m0 , S0 ) ,likelihoodprior distributionTask: Compute the posterior distribution: p(w y).Solution: Identifyxa w,xb y,With Corollary 1 we get the posterior distributionp(w y) N (w; mN , SN )whereTmN SN (S 10 m0 βX y), 1TS 1N S0 βX X,27 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

ex) Bayesian linear regressionConsider the problem of fitting a straight line to noisy measurements.Let the model be (yn R, xn R)yn w0 w1 xn εn , {z }wT xwheren 1, . . . , N.n 1xn ,xn w0w w1εn N (0, β 1 ),β 52 . Furthermore, let the prior be Tp(w) N w 0 0 , α 1 I2 ,whereα 2.28 / 38niklas.wahlstrom@it.uu.seBayesian linear regression

ex) Bayesian linear regressionPlot of the situation before any data arrives.10.80.60.4yy0.2w10 0.2 0.4 0.6 0.8w0Prior, T 1p(w) N w 0 0 , I2229 / 38niklas.wahlstrom@it.uu.se 1 1 0.50xx0.51Example of a few realizations fromthe prior.Bayesian linear regression

ex) Bayesian linear regressionPlot of the situation after one measurement has arrived.w1w1w0Priorw1w0Likelihoodp(w) N (w m0 , S0 )p(y1 w) N (y1 w0 w1 x1 , β 1 )1Posterior/prior,w0p(w y1 ) N (w m1 , S1 ) ,m1 βS1 XT y1 ,S1 (αI2 βXT X) 1 .0.80.60.4y0.2y0Example of a few realizations from the posterior and thefirst measurement (black circle). 0.2 0.4 0.6 0.8 1 130 / 38 0.50xxniklas.wahlstrom@it.uu.se0.51Bayesian linear regression

ex) Bayesian linear regressionPlot of the situation after two measurements have arrived.w1w1w0Priorw1w0Likelihoodp(w y1 ) N (w m1 , S1 ) p(y2 w) N (y2 w0 w1 x2 , β 1 )1Posterior/prior,w0p(w y2 ) N (w m2 , S2 ) ,m2 βS2 XT y,S2 (αI2 βXT X) 1 .0.80.60.4y0.2y0Example of a few realizations from the posterior and thefirst measurement (black circle). 0.2 0.4 0.6 0.8 1 130 / 38 0.50xxniklas.wahlstrom@it.uu.se0.51Bayesian linear regression

ex) Bayesian linear regressionPlot of the situation after 30 measurements have arrived.w1w1w0Priorw1w0Likelihoodp(w y2 ) N (w m2 , S2 ) p(y3 w) N (y3 w0 w1 x3 , β 1 )1Posterior/prior,w0p(w y3 ) N (w m3 , S3 ) ,m3 βS3 XT y,S3 (αI2 βXT X) 1 .0.80.60.4y0.2y0Example of a few realizations from the posterior and thefirst measurement (black circle). 0.2 0.4 0.6 0.8 1 130 / 38 0.50xxniklas.wahlstrom@it.uu.se0.51Bayesian linear regression