Algebra Review De Ning Review Of Linear Algebra For Statistics

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsReview of Linear Algebra for StatisticsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesBrian Vegetabile2017 Statistics BootcampDepartment of StatisticsUniversity of California, IrvineSimple LinearRegressionReferencesSeptember 22nd, 20161 / 35

06 - LinearAlgebraReviewOverview IDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatrices We wrap up the math topics by reviewing some linearalgebra concepts Linear algebra will become an important tool for you as astatisticianOperations ofMatrices You’ll be using matrix operations most of the year, butSimple LinearRegressionthe main necessity for linear algebra will come in STAT200C.References2 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferencesOverview II Here are a few good references for reviewingundergraduate linear algebra in general Introduction to Linear Algebra by Gilbert Strang Gilbert Strang’s Lectures on YouTube(https://www.youtube.com/watch?v ZK3O402wf1c) Linear Algebra and it’s Applications by David Lay Linear Algebra by Friedberg, Insel, Spence (Upper divisiontext) Graduate Level Linear Algebra References for Statistics Matrix Algebra from a Statisticians Perspective by DavidHarville Appendix of Linear Regression Analysis by George Seberand Alan Lee Appendix of Applied Linear Regression by SanfordWeisberg3 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferencesMotivation I A familiarity with matrices will allow you to expand thetypes of statistics you can do. Consider the multivariate normal distributionX (X1 , X2 , . . . , Xn )T 1T 1f (x) pexp (x µ) Σ (x µ)2(2π)n Σ 1which is said to be “non-degenerate” when Σ ispositive-definite. Additionally, x is a real-valued n-dimensional columnvector and Σ is the determinant of Σ To investigate many of the properties of this distributionwe’ll need matrix algebra4 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMotivation II We’ll specifically use this distribution to explore linearregression Let Y be a random variable which has some mean µwhich we measure under error , , specificallyMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionY µ We will focus on linear models whereµ β0 β1 x1 · · · βp 1 xp 1Referenceswhere x are explanatory variables and each βj is unknownand to be estimated5 / 35

06 - LinearAlgebraReviewMotivation III If we consider a random sample of n observations we willDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionhave Y1 Y2 . .Yn x10x20.x11x21.x1,p 1x2,p 1.xn0 xn1 . . .xn,p 1 β0β1. 1 2. nβp 1 Or more simply writtenY Xβ References We will eventually show that Y Nn (Xβ, Σ). Matrix algebra will play a very important role throughoutunderstanding linear algebra6 / 35

06 - LinearAlgebraReviewDefining a MatrixDefiningMatricesBasic MatrixOperations A rectangular array of real numbers is called a matrix. Special Typesof Matrices MatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferencesa11a21.a12a22.a1na2n.am1 am2 . . .amn A matrix with m rows and n columns is referred to as anm n matrix Matrices will often be denoted by boldface letters X. Additionally we can denote a matrix X {aij }7 / 35

06 - LinearAlgebraReviewBasic Matrix Operations IDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferences Scalar Multiplication: Consider a matrix A and a scalar k,thenkA k{aij } {kaij } Matrix Addition: Consider two matrices A and B, if theyare both of dimension m n then we define additionbetween these two matrices. Specifically A B is them n matrix {aij bij } for all pairs i, j. Matrix addition is commutative and associative Additionally matrices having the same number of rows andcolumns are said to be conformal for addition (orsubtraction).8 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferencesBasic Matrix Operations II Matrix Multiplication: Let A {aij } represent an m nmatrix and B {bij } a p q matrix. When n p (whenA has the same number of columns as B has rows), thenthe matrix product AB is defined to be the m q matrixwhose ij th element isnXaik bkj ai1 b1j ai2 b2j · · · ain bnjk 1 The formation AB is called the premultiplication of B byA or the postmultiplication of A by B. When n 6 p then the matrix product AB is undefined. Two n n matrices A and B are said to commute ifAB BA9 / 35

06 - LinearAlgebraReviewBasic Matrix Operations IIIDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatrices Matrix Transpose: The transpose of an m n matrix A,to be denoted AT or A0 is the n m matrix whose ij thelement is the jith element of A. For any matrix A, (A0 )0 A For any two matrices A and B which are conformal foraddition(A B)0 A0 B0Operations ofMatricesSimple LinearRegressionReferences Finally any two matrices A and B for which the productis defined,(AB)0 B0 A010 / 35

06 - LinearAlgebraReviewDefiningMatricesVectors A matrix with only one columnBasic MatrixOperations Special Typesof Matrices MatrixInversionSimple LinearRegressionReferences amProperties ofMatricesOperations ofMatricesa1a2.is called an m-dimensional column vector A matrix with only one row is called a row vector Vectors will often be denoted by lower case bold symbolsx. Clearly the transpose of an m-dimensional column vectoris an m-dimensional row vector11 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionSquare Matrices One of the most important types of matrices in all ofstatistics is the square matrix A matrix having the same number of rows as it doescolumns is called a square matrix An n n square matrix is said to have order n.Properties ofMatrices Operations ofMatrices Simple LinearRegressionReferencesa11a21.a12a22.a1na2n.an1 an2 . . .ann The set of terms {aii } are called the diagonal elements ofthe square matrix and the terms {aii }, i 6 j are theoff-diagonal terms12 / 35

06 - LinearAlgebraReviewSymmetric MatricesDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegression A matrix A is said to be symmetric is A0 A Thus a symmetric matrix is a square matrix where theij th element equals the jith element. 540 4 10 2 0 23References13 / 35

06 - LinearAlgebraReviewDiagonal MatrixDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferences A diagonal matrix is a square matrix whose off-diagonalelements are zero, that is d1 0 . . . 0 d2 . . . . . . .0 0 .00. dn The effect of premultiplying an m n matrix A by am m diagonal matrix D, DA is to multiply eachelement of the ith row of A by the element dii .14 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferencesIdentity Matrix Often the most useful diagonal matrix is the identitymatrix In where the subscript n denotes the dimension ofthe identity matrix (n n). That is, 1 0 . 0 0 1 . 0 In . . . . . . . 0 0 .1often the subscript n is dropped. An important property isIA AI A15 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionMatrix Inversion I For any scalar c there is a number called the inverse of c,say d such that the product of cd 1. For example, ifc 3, then d 1/c 1/3, and the inverseof 3 is 1/3. This can be extended to square matricesProperties ofMatricesDefinition (Matrix Inverse)Operations ofMatricesAn n n square matrix A is called invertible (also nonsingularand non-degenerate) if there exists an n n square matrix Bsuch thatAB BA InSimple LinearRegressionReferencesIf this is the case, then the matrix B is uniquely determined byA and is called the inverse of A denoted A 116 / 35

06 - LinearAlgebraReviewMatrix Inversion IIDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatrices The collection of matrices that have an inverse are calledfull rank, invertible, or nonsingular. A square matrix that is not invertible, is of less than fullrank or singular. The identity matrix is its own inverse (In ) 1 In .Simple LinearRegressionReferences17 / 35

06 - LinearAlgebraReviewDefiningMatricesInverting a 2 2 Matrix. I Consider the following matrixBasic MatrixOperations Special Typesof MatricesMatrixInversionProperties ofMatricesA the inverse of A denoted A 1 isOperations ofMatricesASimple LinearRegressionReferencesa11 a12a21 a22 11 A a22 a12 a21 a11 where the determinant of A, A a11 a22 a12 a21 By our previous definitions we should have thatAA 1 I18 / 35

06 - LinearAlgebraReviewInverting a 2 2 Matrix. IIDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferencesAA 1 1a11 a12a22 a12 a21 a22 a21 a11a11 a22 a12 a21 1a11 a22 a12 a21 a11 a12 a12 a aa aa aa11 a22 a12 a2121 2222 2121 a12 a22 1 0 0 1 This satisfies our requirement19 / 35

06 - LinearAlgebraReviewDefiningMatricesOrthogonality Two vectors a and b (of the same length), are orthogonalifa0 b 0Basic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatrices An r c matrix Q has orthonormal columns if itscolumns, viewed as a set c r different r 1 vectors, areorthogonal and in addition have length 1. This is equivalent toOperations ofMatricesQ0 Q ISimple LinearRegressionReferences Additionally a square matrix A is orthogonal ifA0 A AA0 Iso A 1 A0 .20 / 35

06 - LinearAlgebraReviewLinear Dependence and Rank IDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferences Consider an n p matrix X with columns given by thevectors x1 , x2 , . . . , xp (we only consider the case whenp n.) We say that x1 , x2 , . . . , xp are linearly dependent if wecan find multipliers a1 , . . . , ap not all equal to 0, suchthatpXa i xi 0i 121 / 35

06 - LinearAlgebraReviewLinear Dependence and Rank IIDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesOperations ofMatricesSimple LinearRegressionReferences If no such multipliers exist, then we say the vectors arelinearly independent, and the matrix is full-rank. In general the rank of a matrix is the maximum number ofxi which form a linearly independent set. The matrix X0 X is a p p matrix. If X has rank p, so does X0 X. Full Rank matrices always have an inverse Square matrices less than full rank never have an inverse22 / 35

06 - LinearAlgebraReviewDefiningMatricesBasic MatrixOperationsSpecial Typesof MatricesMatrixInversionProperties ofMatricesMore Properties of Matrices IDefinition (Positive-Semidefinite Matrix)A symmetric matrix A is said to be positive-semidefinite(p.s.d) if and only ifx0 Ax 0for all xOperations ofMatricesDefinition (Positive-Definite Matrix)Simple LinearRegressionA symmetric matrix A is said to be positive-definite (p.d.) ifReferencesx0 Ax 0for all x, x 6 0. Note that a matrix that is p.d. is also p.s.d.23 / 35