The Multivariate Gaussian Distribution

The Multivariate Gaussian DistributionChuong B. DoOctober 10, 2008 T A vector-valued random variable X X1 · · · Xn is said to have a multivariatenormal (or Gaussian) distribution with mean µ Rn and covariance matrix Σ Sn 1if its probability density function2 is given by 11T 1p(x; µ, Σ) exp (x µ) Σ (x µ) .(2π)n/2 Σ 1/22We write this as X N (µ, Σ). In these notes, we describe multivariate Gaussians and someof their basic properties.1Relationship to univariate GaussiansRecall that the density function of a univariate normal (or Gaussian) distribution isgiven by 1122p(x; µ, σ ) exp 2 (x µ) .2σ2πσHere, the argument of the exponential function, 2σ1 2 (x µ)2 , is a quadratic function of thevariable x. Furthermore, the parabola points downwards, as the coefficient of the quadratic1term is negative. The coefficient in front, 2πσ, is a constant that does not depend on x;hence, we can think of it as simply a “normalization factor” used to ensure that Z 112 exp 2 (x µ) 1.2σ2πσ Recall from the section notes on linear algebra that Sn is the space of symmetric positive definite n nmatrices, defined as Sn A Rn n : A AT and xT Ax 0 for all x Rn such that x 6 0 .12In these notes, we use the notation p( ) to denote density functions, instead of fX ( ) (as in the sectionnotes on probability theory).1

0500 500123456789 5 1010 10Figure 1: The figure on the left shows a univariate Gaussian density for a single variable X.The figure on the right shows a multivariate Gaussian density over two variables X1 and X2 .In the case of the multivariate Gaussian density, the argument of the exponential function, 21 (x µ)T Σ 1 (x µ), is a quadratic form in the vector variable x. Since Σ is positivedefinite, and since the inverse of any positive definite matrix is also positive definite, thenfor any non-zero vector z, z T Σ 1 z 0. This implies that for any vector x 6 µ,(x µ)T Σ 1 (x µ) 01 (x µ)T Σ 1 (x µ) 0.2Like in the univariate case, you can think of the argument of the exponential function asbeing a downward opening quadratic bowl. The coefficient in front (i.e., (2π)n/21 Σ 1/2 ) has aneven more complicated form than in the univariate case. However, it still does not dependon x, and hence it is again simply a normalization factor used to ensure that Z Z Z 11T 1···exp (x µ) Σ (x µ) dx1 dx2 · · · dxn 1.(2π)n/2 Σ 1/2 2 2The covariance matrixThe concept of the covariance matrix is vital to understanding multivariate Gaussiandistributions. Recall that for a pair of random variables X and Y , their covariance isdefined asCov[X, Y ] E[(X E[X])(Y E[Y ])] E[XY ] E[X]E[Y ].When working with multiple variables, the covariance matrix provides a succinct way tosummarize the covariances of all pairs of variables. In particular, the covariance matrix,which we usually denote as Σ, is the n n matrix whose (i, j)th entry is Cov[Xi , Xj ].2

The following proposition (whose proof is provided in the Appendix A.1) gives an alternative way to characterize the covariance matrix of a random vector X:Proposition 1. For any random vector X with mean µ and covariance matrix Σ,Σ E[(X µ)(X µ)T ] E[XX T ] µµT .(1)In the definition of multivariate Gaussians, we required that the covariance matrix Σbe symmetric positive definite (i.e., Σ Sn ). Why does this restriction exist? As seenin the following proposition, the covariance matrix of any random vector must always besymmetric positive semidefinite:Proposition 2. Suppose that Σ is the covariance matrix corresponding to some randomvector X. Then Σ is symmetric positive semidefinite.Proof. The symmetry of Σ follows immediately from its definition. Next, for any vectorz Rn , observe thatz T Σz n XnXi 1 j 1n XnXi 1 j 1n XnXi 1 j 1"(Σij zi zj )(2)(Cov[Xi , Xj ] · zi zj )(E[(Xi E[Xi ])(Xj E[Xj ])] · zi zj )#n XnX(Xi E[Xi ])(Xj E[Xj ]) · zi zj . E(3)i 1 j 1Here, (2) follows from the formula for expanding a quadratic form (see section notes on linearalgebra), and (3) follows by linearity of expectations (see probability notes).observe that the quantity inside the brackets is of the formP ToP complete the Tproof,2xxzz (xz) 0 (see problem set #1). Therefore, the quantity inside theij i j i jexpectation is always nonnegative, and hence the expectation itself must be nonnegative.We conclude that z T Σz 0.From the above proposition it follows that Σ must be symmetric positive semidefinite inorder for it to be a valid covariance matrix. However, in order for Σ 1 to exist (as required inthe definition of the multivariate Gaussian density), then Σ must be invertible and hence fullrank. Since any full rank symmetric positive semidefinite matrix is necessarily symmetricpositive definite, it follows that Σ must be symmetric positive definite.3

3The diagonal covariance matrix caseTo get an intuition for what a multivariate Gaussian is, consider the simple case where n 2,and where the covariance matrix Σ is diagonal, i.e., 2 σ1 0µ1x1Σ µ x 0 σ22µ2x2In this case, the multivariate Gaussian density has the form, T 2 1 !11 x1 µ 1σ1 0x1 µ1p(x; µ, Σ) exp 1/20 σ22x2 µ22 x2 µ 2σ12 02π0 σ22# ! T " 1 0211 x1 µ1x µ11σ1, exp 0 σ12 x2 µ22π(σ12 · σ22 0 · 0)1/22 x2 µ22where we have relied on the explicit formula for the determinant of a 2 2 matrix3 , and thefact that the inverse of a diagonal matrix is simply found by taking the reciprocal of eachdiagonal entry. Continuing,#! T " 1(x µ)211 x1 µ 111σ1p(x; µ, Σ) exp 1(x2 µ2 )2πσ1 σ22 x2 µ 2σ22 11122exp 2 (x1 µ1 ) 2 (x2 µ2 ) 2πσ1 σ22σ12σ2 111122 exp 2 (x1 µ1 ) · exp 2 (x2 µ2 ) .2σ12σ22πσ12πσ2The last equation we recognize to simply be the product of two independent Gaussian densities, one with mean µ1 and variance σ12 , and the other with mean µ2 and variance σ22 .More generally, one can show that an n-dimensional Gaussian with mean µ Rn anddiagonal covariance matrix Σ diag(σ12 , σ22 , . . . , σn2 ) is the same as a collection of n independent Gaussian random variables with mean µi and variance σi2 , respectively.4IsocontoursAnother way to understand a multivariate Gaussian conceptually is to understand the shapeof its isocontours. For a function f : R2 R, an isocontour is a set of the form x R2 : f (x) c .for some c R.4a b ad bc.c d4Isocontours are often also known as level curves. More generally, a level set of a function f : Rn R,is a set of the form x R2 : f (x) c for some c R.3Namely,4

4.1Shape of isocontoursWhat do the isocontours of a multivariate Gaussian look like? As before, let’s consider thecase where n 2, and Σ is diagonal, i.e., 2 σ1 0µ1x1Σ µ x 0 σ22µ2x2As we showed in the last section, 11122p(x; µ, Σ) exp 2 (x1 µ1 ) 2 (x2 µ2 ) .2πσ1 σ22σ12σ2(4)Now, let’s consider the level set consisting of all points where p(x; µ, Σ) c for some constantc R. In particular, consider the set of all x1 , x2 R such that 11122c exp 2 (x1 µ1 ) 2 (x2 µ2 )2πσ1 σ22σ12σ2 11222πcσ1 σ2 exp 2 (x1 µ1 ) 2 (x2 µ2 )2σ12σ211log(2πcσ1 σ2 ) 2 (x1 µ1 )2 2 (x2 µ2 )22σ12σ2 111log 2 (x1 µ1 )2 2 (x2 µ2 )22πcσ1 σ22σ12σ22(x2 µ2 )2(x1 µ1 ) . 1 2σ12 log 2πcσ11 σ22σ22 log 2πcσ11 σ2Definingr1 s2σ12log 12πcσ1 σ2 r2 s2σ22log 1,2πcσ1 σ2it follows that1 x1 µ1r1 2 x2 µ2r2 2.(5)Equation (5) should be familiar to you from high school analytic geometry: it is the equationof an axis-aligned ellipse, with center (µ1 , µ2 ), where the x1 axis has length 2r1 and thex2 axis has length 2r2 !4.2Length of axesTo get a better understanding of how the shape of the level curves vary as a function ofthe variances of the multivariate Gaussian distribution, suppose that we are interested in5

8866442200 2 2 4 6 6 4 20246810 4 412 20246810Figure 2:The figure on the left shows a heatmap indicating values of the density function for an3and diagonal covariance matrix Σ axis-aligned multivariate Gaussian with mean µ 2 25 0. Notice that the Gaussian is centered at (3, 2), and that the isocontours are all0 9elliptically shaped with major/minor axis lengths in a 5:3 ratio. The figure on the rightshows a heatmap indicating valuesof the density function for a non axis-aligned multivariate 10 53. Here, the ellipses areand covariance matrix Σ Gaussian with mean µ 5 52again centered at (3, 2), but now the major and minor axes have been rotated via a lineartransformation.6

the values of r1 and r2 at which c is equal to a fraction 1/e of the peak height of Gaussiandensity.First, observe that maximum of Equation (4) occurs where x1 µ1 and x2 µ2 . Substituting these values into Equation (4), we see that the peak height of the Gaussian densityis 2πσ11 σ2 . Second, we substitute c 1e 2πσ11 σ2 into the equations for r1 and r2 to obtainv uuur1 t2σ12 log v uuur2 t2σ22 log 12πσ1 σ2 ·1e 12πσ1 σ212πσ1 σ2 ·1e 12πσ1 σ2 σ1 2 σ2 2.From this, it follows that the axis length needed to reach a fraction 1/e of the peak height ofthe Gaussian density in the ith dimension grows in proportion to the standard deviation σi .Intuitively, this again makes sense: the smaller the variance of some random variable xi , themore “tightly” peaked the Gaussian distribution in that dimension, and hence the smallerthe radius ri .4.3Non-diagonal case, higher dimensionsClearly, the above derivations rely on the assumption that Σ is a diagonal matrix. However,in the non-diagonal case, it turns out that the picture is not all that different. Insteadof being an axis-aligned ellipse, the isocontours turn out to be simply rotated ellipses.Furthermore, in the n-dimensional case, the level sets form geometrical structures known asellipsoids in Rn .5Linear transformation interpretationIn the last few sections, we focused primarily on providing an intuition for how multivariateGaussians with diagonal covariance matrices behaved. In particular, we found that an ndimensional multivariate Gaussian with diagonal covariance matrix could be viewed simplyas a collection of n independent Gaussian-distributed random variables with means and variances µi and σi2 , respectvely. In this section, we dig a little deeper and provide a quantitativeinterpretation of multivariate Gaussians when the covariance matrix is not diagonal.The key result of this section is the following theorem (see proof in Appendix A.2).Theorem 1. Let X N (µ, Σ) for some µ Rn and Σ Sn . Then, there exists a matrixB Rn n such that if we define Z B 1 (X µ), then Z N (0, I).7

To understand the meaning of this theorem, note that if Z N (0, I), then using theanalysis from Section 4, Z can be thought of as a collection of n independent standard normalrandom variables (i.e., Zi N (0, 1)). Furthermore, if Z B 1 (X µ) then X BZ µfollows from simple algebra.Consequently, the theorem states that any random variable X with a multivariate Gaussian distribution can be interpreted as the result of applying a linear transformation (X BZ µ) to some collection of n independent standard normal random variables (Z).8