15.075 Notes, Multiple Linear Regression

Chapter 11 : Multiple Linear RegressionWe have:height weight . . . ageperson 1:person 2::x11x21x12x22.x1kx2kamount oflemonade purchasedy1y2where we assumeYi β0 β1 xi1 β2 xi2 · · · βk xik tifor i 1, . . . , n and ti N (0, σ 2 ). The xi· ’s are not random.Is there any way we can fit something that isn’t linear? Like a polynomial?We can do least squares to find β̂0 , β̂1 , . . . , β̂k : Minimize Q where:(yi (β0 β1 xi1 β2 xi2 · · · βk xik ))2 .Q iSolve it the same way as we did in Chapter 10: set Q/ βj 0 for all j. In thiscase, we’ll let the computer solve it for us. So now we have all the β̂j ’s.To assess the goodness of fit, again define:(yi ŷi )2 where ŷi β̂0 β̂1 xi1 β̂2 xi2 · · · β̂k xikSSE iand compare with:(yi ȳ)2 .SST iAgain, SSR SST- SSE.The coefficient of “multiple” determination is :r2 SSRSSE 1 .SSTSST1(1)

This time, by convention,SSE.SSTThe square root is only positive, since it is not meaningful to assign an associationbetween y and multiple x’s.r 1 For hypothesis testing, we’ll need to know:1. Each of the coefficients obeys:βˆj N (βj , σ 2 Vjj )where Vjj is the j’th diagonal entry of V (X ' X) 1 , j 0, 1, · · · , k2. Because we don’t know σ 2 , we use SE(βˆj ) s Vjjwhere s2 SSEn (k 1)We could do the hypothesis tests on each βj :H0j : βj βj0H1j : βj βj0 .Reject H0j when βˆj βj0 tj tn (k 1),α/2SE(βˆj )and thus if βj0 0:H0j : βj 0H1j : βj 0.Reject H0j when tj βˆj tn (k 1),α/2 .SE(βˆj )2

Or we could test all βj ’s simultaneously:H0 : β1 β2 · · · βk 0H1 : βi 0 for at least one i.Reject H0 when F fk,n (k 1),α where:M SRF M SESSRkSSEn (k 1)n2i 1 (yˆi ȳ) kn2i 1 (yi yˆi ).n (k 1)Both the numerator and the denominator look like sample variances so youM SRcould see the intuition why MSE has an F-distribution.Equivalently:M SRF M SESSRr2 SST(?)k (1 r2k)SSTSSEn (k 1)n (k 1) r2 (n k 1)k(1 r2 )Where did the (?) step come from?Note: The F-test above does not tell you which βj s are nonzero.But then how do you do that?Note: Beware of multicollinearity, meaning that some of the factors in themodel can be determined from the others (i.e. they are linearly dependent).Example: for savings, income, expenditure wheresavings income - expenditure.This makes computation numerically unstable and βˆj are not statistically signif icant. To avoid this, use only income and expenditure, not savings. (Or savingsand income, not expenditure, etc.)3

Corresponding ANOVA regression tableSource of variation sum of squaresRegressionSSRErrorSSETotalSSTd.f.Mean SquarekMSR n (k 1) MSE SSRkFF pMSRMSEp-valueSSEn (k 1)n 1We can also put the hypothesis tests for the individual βj ’s in a table:predictorSEβˆ0SE(βˆ0 )βˆ1SE(βˆ1 ).βˆkSE(βˆk )t-statisticp-valuet βˆ0SE(βˆ0 )p-valuet βˆ1SE(βˆ1 )p-value.t 4βˆkSE(βˆk ).p-value

MIT OpenCourseWarehttp://ocw.mit.edu15.075J / ESD.07J Statistical Thinking and Data AnalysisFall 2011For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

Note: The F-test above does not tell you which β . 15.075 Notes, Multiple Linear Regression Author: 15.075 Faculty and Staff Created Date: 12/5/2011 11:52:35 PM .