Multiple Linear Regression (MLR) Handouts
Multiple Linear Regression (MLR) HandoutsYibi Huang Data and ModelsLeast Square Estimate, Fitted Values, ResidualsSum of SquaresDo Regression in RInterpretation of Regression Coefficientst-Tests on Individual Regression CoefficientsF -Tests on Multiple Regression Coefficients/Goodness-of-FitMLR - 1
Data for Multiple Linear RegressionMultiple linear regression is a generalized form of simple linearregression, in which the data contains multiple explanatoryvariables.SLRMLRxyx1x2 . . . xpycase 1: x1 y1x11 x12 . . . x1p y1case 2: x2 y2x21 x22 . . . x2p y2.case n:IIIxnynxn1xn2.xnpynFor SLR, we observe pairs of variables.For MLR, we observe rows of variables.Each row (or pair) is called a case, a record, or a data pointyi is the response (or dependent variable) of the ithobservationThere are p explanatory variables (or covariates, predictors,independent variables), and xik is the value of the explanatoryvariable xk of the ith caseMLR - 2
Multiple Linear Regression Modelsyi β0 β1 xi1 . . . βp xip εiwhere εi ’s are i.i.d. N(0, σ 2 )In the model above,Iεi ’s (errors, or noise) are i.i.d. N(0, σ 2 )IParameters include:β0 intercept;βk regression coefficients (slope) for the kthexplanatory variable, k 1, . . . , pσ 2 Var(εi ) is the variance of errorsIObserved (known): yi , xi1 , xi2 , . . . , xipUnknown: β0 , β1 , . . . , βp , σ 2 , εi ’sIRandom variables: εi ’s, yi ’sConstants (nonrandom): βk ’s, σ 2 , xik ’sMLR - 3
QuestionsIWhat are the mean, the variance, and the distribution of yi ?IWe assume εi ’s are independent. Are yi ’s independent?MLR - 4
I 8 10regression line6 an arbitrarystraight line 4y 2Recall for SLR, the leastsquares estimates (βb0 , βb1 )for (β0 , β1 ) is the interceptand slope of the straight linewith the minimum sum ofsquared vertical distance tothePn data pointsbb 2i 1 (yi β0 β1 xi ) 0I14Fitting the Model — Least Squares Method01234x5678MLR is just like SLR. The least squares estimates(βb0 , βb1 , . . . , βbp ) for β0 , . . . , βp is the intercept and slopes ofthe (hyper)plane with the minimum sum of squared verticaldistance to the data pointsnX(yi βb0 βb1 xi1 . . . βbp xip )2i 1MLR - 5
Solving the Least Squares Problem (1)From now on, we use the “hat” symbol to differentiate theestimated coefficient βbj from the actual unknown coefficient βj .To find the (βb0 , βb1 , . . . , βbp ) that minimizeL(βb0 , βb1 , . . . , βbp ) nX(yi βb0 βb1 xi1 . . . βbp xip )2i 1one can set the derivatives of L with respect to βbj to 0nX L 2(yi βb0 βb1 xi1 . . . βbp xip ) βb0 L 2 βbki 1nXxik (yi βb0 βb1 xi1 . . . βbp xip ), k 1, 2, . . . , pi 1and then equate them to 0. This results in a system of (p 1)equations in (p 1) unknowns.MLR - 6
Solving the Least Squares Problem (2)The least square estimate (βb0 , βb1 , . . . , βbp ) is the solution to thefollowing system of equations, called normal equations.Pnβb0 βb1 ni 1 xi1PP2βb0 ni 1 xi1 βb1 ni 1 xi1PP · · · βbp ni 1 xip ni 1 yiPP · · · βbp ni 1 xi1 xip ni 1 xi1 yi.PnPnPPbbβ0 i 1 xik β1 i 1 xik xi1 · · · βbp ni 1 xik xip ni 1 xik yi.PnPnPPbbβ0 i 1 xip β1 i 1 xip xi1 · · · βbp ni 1 xip2 ni 1 xip yiIIDon’t worry about solving the equations.R and many other softwares can do the computation for us.In general, βbj 6 βj , but they will be close under someconditionsMLR - 7
Fitted ValuesThe fitted value or predicted value:ybi βb0 βb1 xi1 . . . βbp xipIAgain, the “hat” symbol is used to differentiate the fittedvalue ybi from the actual observed value yi .MLR - 8
ResidualsIOne cannot directly compute the errorsεi yi β0 β1 xi1 . . . βp xipsince the coefficients β0 , β1 , . . . , βp are unknown.IThe errors εi can be estimated by the residuals ei defined as:residual ei observed yi predicted yi yi ybi yi (βb0 βb1 xi1 . . . βbp xip ) β0 β1 xi1 . . . βp xip εi βb0 βb1 xi1 . . . βbp xipIei 6 εi in general since βbj 6 βjIGraphical explanationMLR - 9
Properties of ResidualsRecall the least square estimate (βb0 , βb1 , . . . , βbp ) satisfies theequationsnXi 1nX(yi βb0 βb1 xi1 . . . βbp xip ) 0 and{z} yi byi ei residualz} {xik (yi βb0 βb1 xi1 . . . βbp xip ) 0, k 1, 2, . . . , p.i 1Thus the residuals ei have the propertiesXnXnxik ei 0, k 1, 2, . . . , p.ei 0 , i 1 i 1{z}{z}Residuals add up to 0.Residuals are orthogonal to covariates.The two properties also imply that the residuals are uncorrelatedwith each of the p covariates. (proof: HW1 bonus).MLR - 10
Sum of SquaresObserve thatyi y (byi y ) (yi ybi )Squaring up both sides we getyi y )2 (yi ybi )2 2(byi y )(yi ybi )(yi y )2 (bSumming up over all the cases i 1, 2, . . . , n, we getSSTSSRSSEz} { z} { z} {nnnXXX(yi y )2 (byi y )2 (yi ybi )2 {z }i 1i 1i 1 2 einX(byi y )(yi ybi ) i 1{z}We’ll shortly show this is 0.MLR - 11
WhyPnyii 1 (bnX y )(yi ybi ) 0?i 1(byi y )(yi ybi ) {z }nXnX eiybi ei i 1 nXy eii 1(βb0 βb1 xi1 . . . βbp xip )ei i 1 βb0nXy eii 1nXei βb1nXxi1 ei . . . βbpnXxip ei ynXeii 1i 1i 1i 1 {z } {z } {z } {z } 0 0 0 0 0inPnwhich we used the properties of residuals thati 1 xik ei 0 for all k 1, . . . , p.MLR - 12Pni 1 ei 0 and
Interpretation of Sum of Squares einnn z } {XXX22(yi y ) (byi y ) (yi ybi )2 i 1 {zSST} i 1 {zSSR} i 1 {zSSE}ISST total sum of squaresI total variability of yI depends on the response y only, not on the form of themodelISSR regression sum of squaresI variability of y explained by x , . . . , x1pISSE error (residual) sum of squaresPn2I minβ0 ,β1 ,.,βpi 1 (yi β0 β1 xi1 · · · βp xip )I variability of y not explained by x’sMLR - 13
Nested ModelsWe say Model 1 is nested in Model 2 if Model 1 is a special caseof Model 2 (and hence Model 2 is an extension of Model 1).E.g., for the 4 models below,Model A : Y β0 β1 X1 β2 X2 β3 X3 εModel B : Y β0 β1 X1 β2 X2 εModel C : Y β0 β1 X1 β3 X3 εModel D : Y β0 β1 (X1 X2 ) εIB is nested in A . . . . . . . . . . . since A reduces to B when β3 0IC is also nested in A . . . . . . . since A reduces to C when β2 0ID is nested in B . . . . . . . . . since B reduces to D when β1 β2IB and C are NOT nested in either wayID is NOT nested in CMLR - 14
Nested Relationship is TransitiveIf Model 1 is nested in Model 2, and Model 2 is nested in Model 3,then Model 1 is also nested in Model 3.For example, for models in the previous slide,D is nested in B, and B is nested in A,implies D is also nested in A, which is clearly true because ModelA reduces to Model D whenβ1 β2 , and β3 0.When two models are nested (Model 1 is nested in Model 2),Ithe smaller model (Model 1) is called the reduced model,andIthe more general model (Model 2) is called the full model.MLR - 15
SST of Nested ModelsQuestion: Compare the SST for Model A and the SST for ModelB. Which one is larger? Or are they equal?What about the SST for Model C? For Model D?MLR - 16
SSE of Nested ModelsWhen a reduced model is nested in a full model, then(i) SSEreduced SSEfull , and (ii) SSRreduced SSRfull .Proof. We will prove (i) forI the full model yi β0 β1 xi1 β2 xi2 β3 xi3 εi andI the reduced model yi β0 β1 xi1 β3 xi3 εi .The proofs for other nested models are similar.SSEfull minβ0 ,β1 ,β2 ,β3nX(y1 β0 β1 xi1 β2 xi2 β3 xi3 )2i 1nX(y1 β0 β1 xi1 β3 xi3 )2 minβ0 ,β1 ,β3i 1 SSEreducedPart (ii) follows directly from (i), the identity SST SSR SSE ,and the fact that all MLR models of the same data set have acommon SSTMLR - 17
Degrees of FreedomIt can be shown that if the MLR modelyi β0 β1 xi1 . . . βp xip εi , εi ’s i.i.d. N(0, σ 2 ) is true thenSSE χ2n p 1 ,σ2If we further assume that β1 β2 · · · βp 0, thenSST χ2n 1 ,σ2SSR χ2pσ2and SSR is independent of SSE.Note the degrees of freedom of the 3 chi-square distributionsdfT n 1,dfR p,dfE n p 1break down similarlydfT dfR dfEjust like SST SSR SSE .MLR - 18
Why the Degrees of Freedom for Errors is dfE n p 1?The n residuals e1 , . . . , en cannot all vary freely.There are p 1 constraints:nXi 1ei 0andnXxki ei 0 for k 1, . . . , p.i 1So only n (p 1) of them can be freely varying.The p 1 constraints comes from the p 1 coefficients β0 , . . . , βpin the model, and each contributes one constraint β k 0.MLR - 19
Mean Square Error (MSE) — Estimate of σ 2The mean squares is the sum of squares divided by its degrees offreedom:SSTSST sample variance of Y ,dfTn 1SSRSSRMSR ,dfRpSSESSEMSE σb2dfEn p 1MST II χ2n p 1 and that the mean of a χ2kFrom the fact SSEσ2distribution is k, we know that MSE is an unbiasedestimator for σ 2 .Though SSE always decreases as we add terms to the model,adding unimportant terms may increases MSE.MLR - 20
Multiple R-SquaredMultiple R 2 , also called the coefficient of determination, isdefined asSSESSR 1 R2 SSTSST proportion of variability in y explained by x1 , . . . , xpwhich measures the strength of the linear relationship between yand the p covariates.I 0 R2 12 is the square of the correlation coefficientI For SLR, R 2 rxybetween X and Y (Proof: HW1).So multiple R 2 is a generalization of the correlationcoefficient.I When two models are nested, thenR 2 (reduced model) R 2 (full model).IIs large R 2 always preferable?MLR - 21
Adjusted R-SquaredBecause R 2 always increases as we add terms to the model, somepeople prefer to use an adjusted R 2 defined as2 1 RadjIIISSE /dfESSE /(n p 1) 1 SST /dfTSST /(n 1)n 1 1 (1 R 2 ).n p 1p2 R2 1 Radjn p 12 can be negativeUnlike R 2 , Radj 2 does not always increase as more variables are added toRadjthe model.2 may decrease.In fact, if unnecessary terms are added, RadjMLR - 22
Example: Housing PricePrice BDR FLR FP RMS ST LOT BTH CON GAR LOC532 967 05 0 39 1.51 0.00552 815 15 0 33 1.01 2.00563 900 05 1 35 1.51 1.00583 1007 06 1 24 1.50 2.00643 1100 17 0 50 1.51 1.50444 897 07 0 25 2.00 1.00495 1400 08 0 30 1.00 1.00703 2261 06 0 29 1.00 2.00724 1290 08 1 33 1.51 1.50824 2104 09 0 40 2.51 1.00858 2240 1 12 1 50 3.00 2.00452 641 05 0 25 1.00 0.01473 862 06 0 25 1.01 0.01494 1043 07 0 30 1.50 0.01564 1325 08 0 50 1.50 0.01602 782 05 1 25 1.00 0.01623 1126 07 1 30 2.01 0.01644 1226 08 0 37 2.00 2.01.502 691 06 0 30 1.00 2.00653 1023 07 1 30 2.01 1.00MLR - 23PriceBDRFLRFPRMSST Selling price in 1000Number of bedroomsFloor space in sq. ft.Number of fireplacesNumber of roomsStorm windows(1 if present, 0 if absent)LOT Front footage of lot in feetBTH Number of bathroomsCON Construction(1 if frame, 0 if brick)GAR Garage size(0 no garage,1 one-car garage, etc.)LOC Location(1 if property is in zone A,0 otherwise)
How to Do Regression Using R? housing read.table("housing.dat",h TRUE)# to load the data lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing)Call:lm(formula Price FLR RMS BDR GAR LOT ST CON LOC, data LOC6.95701GAR5.07873The lm() command above asks R to fit the modelPrice β0 β1 FLR β2 RMS β3 BDR β4 GAR β5 LOT β6 ST β7 CON β8 LOC εand R gives us the regression equationPrice 12.480 0.017FLR 2.383RMS 4.544BDR 5.079GAR 0.382LOT 9.828ST 4.865CON 6.957LOCMLR - 24
More R Commands lm1 lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) summary(lm1)# Regression output with more details# including multiple R-squared,# and the estimate of sigma lm1 coef lm1 fitted lm1 res# show the estimated beta’s# show the fitted values# show the residualsGuess what we will get in R if we type the following commands? sum(lm1 res)mean(lm1 res)cor(lm1 res, housing FLR)cor(lm1 res, housing RMS)cor(lm1 res, housing BDR)MLR - 25
lm1 lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) summary(lm1)Call:lm(formula Price FLR RMS BDR GAR LOT ST CON LOC, data housing)Residuals:Min1Q Median-6.020 -2.129 -0.2133Q2.147Max6.492Coefficients:Estimate Std. Error t value Pr( t )(Intercept) 12.4797994.4439852.808 0.012094 *FLR0.0170380.0027516.195 9.8e-06 ***RMS2.3826381.4182901.680 0.111251BDR-4.5435501.781145 -2.551 0.020671 *GAR5.0787291.2096924.198 0.000604 ***LOT0.3824110.1068323.580 0.002309 **ST9.8275721.9292325.094 9.0e-05 ***CON4.8650711.8907182.573 0.019746 *LOC6.9570072.0440843.403 0.003382 **--Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 4.021 on 17 degrees of freedomMultiple R-squared: 0.9305,Adjusted R-squared: 0.8979F-statistic: 28.47 on 8 and 17 DF, p-value: 2.25e-08MLR - 26
Residual standard error: 4.021 on 17 degrees of freedomMultiple R-squared: 0.9305,Adjusted R-squared: 0.8979F-statistic: 28.47 on 8 and 17 DF, p-value: 2.25e-08 IResidual standard error I“on 17 degrees of freedom” becausedfE n p 1 26 8 1 17IMultiple R-squared 1 I2 1 Adjusted R-squared RadjIThe F-statistic 28.47 is for testingMSE σbSSESSTSSE /(n p 1)SST /n 1H0 : β1 β2 · · · βp 0 v.s.Ha : not all βj 0, j 1, . . . , p.We will soon explain the meaning of this F -statistics.MLR - 27
Price 12.480 0.017FLR 2.383RMS 4.544BDR 5.079GAR 0.382LOT 9.828ST 4.865CON 6.957LOCThe regression equation tells us:Ian extra square foot in floor area increase the price by 17 ,Ian additional room by . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2383 ,Ian additional bedroom by . . . . . . . . . . . . . . . . . . . . . . . 4544 ,Ian additional space in the garage by . . . . . . . . . . . . . . 5079 ,Ian extra foot in front footage by . . . . . . . . . . . . . . . . .Iusing storm windows by . . . . . . . . . . . . . . . . . . . . . . . . . . 9828 .Iconstructing in brick rather than in frame by . . . . . . 4865 ,Ilocating in zone A rather than other area by . . . . . . . 6957 . 382 ,Question:Why an additional bedroom makes a house less valuable?MLR - 28
Interpretation of Regression CoefficientsIβ0 : intercept, the mean value of the response y when allcovariates xj ’ are zero.I May not make practical sensee.g., in the housing price model, β0 doesn’t makepractical sense since there is no such housing unit with 0floor space.Iβj : regression coefficient for xj , is the mean change in theresponse y when xj is increased by one unit holding all othercovariates constantI Interpretation of β depends on the presence of otherjcovariates in the modele.g., the meaning of the 2 β1 ’s in the following 2 modelsare differentModel 1 : yi β0 β1 xi1 β2 xi2 β3 xi3 εiModel 2 : yi β0 β1 xi1 εi .MLR - 29
What’s Wrong?# Model 1 lmBDRonly lm(Price BDR, data housing) lmBDRonly coef(Intercept)BDR43.4873653.920578The regression coefficient for BDR is 3.92 in the Model 1 abovebut 4.54 in the Model 2 below.# Model 2 lm1 lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) lm1 coef(Intercept)FLRRMSBDRGAR12.47979941 0.01703833 2.38263831 -4.54355024 5.07872939LOTSTCONLOC0.38241085 9.82757237 4.86507085 6.95700689Considering BDR alone, house prices increase with BDR.However, when other covariates (FLR, RMS, etc) are taken intoaccount, house prices decrease with BDR.Does this make sense?MLR - 30
One Sample t-Test (Review)Given a sample of size n, Y1 , Y2 , . . . , Yn , from some (normal)population with unknown mean µ and unknown variance σ 2 .Want to testH0 : µ µ0 v.s. Ha : µ 6 µ0The t-statistic ist Y µ0Y µ0 s/ nSE(Y )swhere s Pn Y )2n 1i 1 (YiIf the population is normal, the t-statistic has a t-distribution withn 1 degrees of freedomMLR - 31
t-Tests on Individual Regression CoefficientsFor a MLR model Yi β0 β1 Xi1 . . . βp Xip εi , to test thehypotheses,H0 : βj c v.s. Ha : βj 6 cthe t-statistic ist βbj cSE(βbj )in which SE(βbj ) is the standard error for βbj .IGeneral formula for SE(βbj ) is a bit complicate butunimportant in STAT222 and hence is omittedR can compute SE(βbj ) for usIFormula for SE(βbj ) for a few special cases will be given laterIThis t-statistic also has a t-distribution with n p 1 degrees offreedomMLR - 32
lm1 lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) summary(lm1)Call:lm(formula Price FLR RMS BDR GAR LOT ST CON LOC, data housing)Residuals:Min1Q Median-6.020 -2.129 -0.2133Q2.147Max6.492Coefficients:Estimate Std. Error t value Pr( t )(Intercept) 12.4797994.4439852.808 0.012094 *FLR0.0170380.0027516.195 9.8e-06 ***RMS2.3826381.4182901.680 0.111251BDR-4.5435501.781145 -2.551 0.020671 *GAR5.0787291.2096924.198 0.000604 ***LOT0.3824110.1068323.580 0.002309 **ST9.8275721.9292325.094 9.0e-05 ***CON4.8650711.8907182.573 0.019746 *LOC6.9570072.0440843.403 0.003382 **--Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 4.021 on 17 degrees of freedomMultiple R-squared: 0.9305,Adjusted R-squared: 0.8979F-statistic: 28.47 on 8 and 17 DF, p-value: 2.25e-08MLR - 33
Coefficients:Estimate Std. Error t value Pr( t )(Intercept) 12.4797994.4439852.808 0.012094 *FLR0.0170380.0027516.195 9.8e-06 ***RMS2.3826381.4182901.680 0.111251.(some rows are omitted)LOC6.9570072.0440843.403 0.003382 **Ithe first column gives variable namesIIthe column Estimate gives the LS estimate βbj ’s for βj ’sthe column Std. Error gives SE(βbj ), the standard error of βbjIthe column t value gives t-value Icolumn Pr( t ) gives the P-value for testing H0 : βj 0 v.s.Ha : βj 6 0.βbjSE (βbj )E.g., for RMS, we see βbRMS 2.38 and SE(βbRMS ) 1.42, and thet-value for RMS βRMS 2.381.42 1.68. The P-value 0.111 isSE (βbRMS )the 2-sided P-value for testing H0 : βRMS 0bMLR - 34
Types of Tests for MLRThe t-test can only tests for a single coefficient. One may alsowant to test for multiple coefficients.E.g., for the following MLR model with 6 covariatesY β0 β1 X1 . . . β6 X6 ε,one may want to test whether1. all the regression coefficients are equal to zeroH0 : β1 β2 · · · β6 0 v.s.Ha : at least one of β1 . . . , β6 is not 02. some subset of the regression coefficients are equal to zeroe.g. H0 : β2 β3 β5 0 v.s.Ha : at least one of β2 , β3 , β5 6 03. some of the regression coefficients equal to each othere.g. H0 : β2 β3 β5 v.s.Ha : β2 , β3 and β5 are not all equal4. some β’s satisfy certain specified linear constraintse.g. H0 : β2 β3 β5 v.s. Ha : β2 6 β3 β5MLR - 35
Tests on Multiple Coefficients are Model ComparisonsEach of the four tests in the previous slide can be viewed as acomparison of 2 models, a full model and a reduced model.ITesting β1 β2 . . . β6 0Full :Y β0 β1 X1 . . . β6 X6 εReduced :Y β0 ε (All covariates are redundant)ITesting β2 β3 β5 0Full :Y β0 β1 X1 β2 X2 β3 X3 β4 X4 β5 X5 β6 X6 εReduced :Y β0 β1 X1 β4 X4 β6 X6 ε (X2 , X3 , X5 are redundant)ITesting β2 β3 β5Full :Y β0 β1 X1 β2 X2 β3 X3 β4 X4 β5 X5 β6 X6 εReduced :Y β0 β1 X1 β2 (X2 X3 X5 ) β4 X4 β6 X6 ε(Y depends on X2 , X3 , X5 only through their sum X2 X3 X5 )MLR - 36
Tests on Multiple Coefficients are Model Comparisons (2)ITesting β2 β3 β5Full : Y β0 β1 X1 β2 X2 β3 X3 β4 X4 β5 X5 β6 X6 εReduced : Y β0 β1 X1 (β3 β5 )X2 β3 X3 β4 X4 β5 X5 β6 X6 ε β0 β1 X1 β3 (X2 X3 ) β4 X4 β5 (X2 X5 ) β6 X6 εObserved the reduced model is always nested in the full model.MLR - 37
General Framework for Testing Nested ModelsH0 : the reduced model is true v.s.Ha : the full model is trueIAs the reduced model is nested in the full model, it is alwaystrue thatSSEreduced SSEfullITrade-off between Simplicity and PrecisionI Full model fits the data better (with smaller SSE) but ismore complicateI Reduced model doesn’t fit as well but is simpler.How to choose between the full and the reduced models?I If SSEreduced SSEfull , one can sacrifice a bit ofprecision in exchange for simplicityI If SSEreduced SSEfull , it costs a great reduction inprecision in exchange for simplicity. The full model ispreferred.IMLR - 38
The F -StatisticF (SSEreduced SSEfull )/(dfreduced dffull )SSEfull /dffullISSEreduced SSEfull is the reduction in SSE from replacingthe reduced model with the full model.Idffull /dfreduced is the dfE for the full/reduced model.IThe denominator is the MSE of the full modelIF 0 since SSEreduced SSEfull 0IThe smaller the F -statistic, the more we favor the reducedmodelIUnder H0 , the F -statistic has an F -distribution withdfreduced dffull and dffull degrees of freedom.MLR - 39
Testing All Coefficients Equal ZeroTesting the hypothesesH0 : β1 · · · βp 0 v.s. Ha : not all β1 . . . , βp 0is a test to evaluate the overall significance of a model.Full :Y β0 β1 X1 . . . βp Xp εReduced :Y β0 ε (all covariates are unnecessary)IThe LS estimate for β0 in the reduced model is βb0 Y , soSSEreducednXX (Yi βb0 )2 (Yi Y )2 SSTfulli 1IIidfreduced dfEreduced n 1,because the reduced model has only one coefficient β0dffull dfEfull n p 1.MLR - 40
Testing All Coefficients Equal ZeroSo(SSEreduced SSEfull )/(dfreduced dffull )SSEfull /dffull(SSTfull SSEfull )/[n 1 (n p 1)] SSEfull /(n p 1)SSRfull /p .SSEfull /(n p 1)Moreover, F Fp,n p 1 under H0 : β1 β2 · · · βp 0.In R, the F statistic and p-value are displayed in the last line of theoutput of the summary() command.F lm1 lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) summary(lm1). (output omitted)Residual standard error: 4.021 on 17 degrees of freedomMultiple R-squared: 0.9305,Adjusted R-squared: 0.8979F-statistic: 28.47 on 8 and 17 DF, p-value: 2.25e-08MLR - 41
ANOVA and the F -TestThe test of all coefficients equal zero is often summarized in anANOVA table.SourcedfSum ofSquaresRegressiondfR pSSRErrordfE n p 1SSETotaldfT n 1SSTMeanSquaresFMSRSSRF MSR dfRMSESSEMSE dfEANOVA is the shorthand for analysis of variance.It decomposes the total variation in the response (SST) intoseparate pieces that correspond to different sources of variation,like SST SSR SSE in the regression setting.Throughout STAT222, we will introduce several other ANOVAtables.MLR - 42
Testing Some Coefficients Equal to ZeroE.g., for the housing price data, we may want to test if we caneliminate RMS and CON from the model,i.e., H0 : βRMS βCON 0. lm1 lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) lm3 lm(Price FLR BDR GAR LOT ST LOC, data housing) anova(lm3,lm1)Analysis of Variance TableModel 1: Price FLR BDR GAR LOT ST LOCModel 2: Price FLR RMS BDR GAR LOT ST CON LOCRes.DfRSS Df Sum of SqF Pr( F)119 472.03217 274.84 2197.19 6.0985 0.01008 *--Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Note SSE is called RSS (residual sum of square) in R.MLR - 43
Testing Equality of Coefficients (1)Full model : Y β0 β1 X1 β2 X2 β3 X3 β4 X4 εExample 1: want to test H0 : β1 β4 , then the reduced model isY β0 β1 X1 β2 X2 β3 X3 β1 X4 ε β0 β1 (X1 X4 ) β2 X2 β3 X3 ε1.2.3.4.Make a new variable W X1 X4Fit the reduced model by regressing Y on W , X2 , and X3Find SSEreduced and dfreduced dffull 1Can be done in R as follows: lmfull lm(Y X1 X2 X3 X4) lmreduced lm(Y I(X1 X4) X2 X3) anova(lmreduced,lmfull)The line lmreduced lm(Y I(X1 X4) X2 X3) isequivalent to W X1 X4 lmreduced lm(Y W X2 X3)MLR - 44
Testing Equality of Coefficients (2)Example 2: want to test H0 : β1 β2 β3 , then the reducedmodel isY β0 β1 X1 β1 X2 β1 X3 β4 X4 ε β0 β1 (X1 X2 X3 ) β4 X4 ε1. Make a new variable W X1 X2 X32. Fit the reduced model by regressing Y on W and X43. Find SSEreduced and dfreduced dffull 24. Can be done in R as follows lmfull lm(Y X1 X2 X3 X4) lmreduced lm(Y I(X1 X2 X3) X4) anova(lmreduced, lmfull)MLR - 45
Testing Equality of Coefficients (3)Example 3: want to test H0 : β1 β2 and β3 β4 , then thereduced model isY β0 β1 X1 β1 X2 β3 X3 β3 X4 ε β0 β1 (X1 X2 ) β3 (X3 X4 ) ε1. Make new variables W1 X1 X2 , W2 X3 X42. Fit the reduced model by regressing Y on W1 and W23. Find SSEreduced and dfreduced dffull 24. Can be done in R as follows lmfull lm(Y X1 X2 X3 X4) lmreduced lm(Y I(X1 X2) I(X3 X4)) anova(lmreduced, lmfull)MLR - 46
Testing Coefficients under Constraints (1)Again say the full model isFull model : Y β0 β1 X1 β2 X2 β3 X3 β4 X4 εExample 4: If H0 : β2 β3 β4 , then the reduced model isY β0 β1 X1 (β3 β4 )X2 β3 X3 β4 X4 ε β0 β1 X1 β3 (X2 X3 ) β4 (X2 X4 ) ε1. Make new variables W1 X2 X3 , W2 X2 X42. Fit the reduced model by regressing Y on X1 , W1 and W23. Find SSEreduced and dfreduced dffull 14. Can be done in R as follows lmfull lm(Y X1 X2 X3 X4) lmreduced lm(Y X1 I(X2 X3) I(X2 X4)) anova(lmreduced, lmfull)MLR - 47
Testing Coefficients under Constraints (2)Example 5: If we think β2 2β1 , then the reduced model isY β0 β1 X1 β2 X2 β3 X3 β4 X4 ε β0 β1 X1 2β1 X2 β3 X3 β4 X4 ε β0 β1 (X1 2X2 ) β3 X3 β4 X4 ε1. Make a new variable W X1 2X22. Fit the reduced model by regressing Y on W , X3 and X43. Find SSEreduced and dfreduced dffull 14. Can be done in R as follows lmfull lm(Y X1 X2 X3 X4) lmreduced lm(Y I(X1 2*X2) X3 X4) anova(lmreduced, lmfull)MLR - 48
Relationship Between t-tests and F -tests (Optional)The F -test can also test for a single coefficient, and the result isequivalent to the t-test. E.g., if one wants to test a singlecoefficient β3 0 in the modelY β0 β1 X1 β2 X2 β3 X3 β4 X4 εone way is to do a t-test using the commandsummary(lm(Y X1 X2 X3 X4)) and read the t-statistic andP-value for X3 . Alternatively, one can also be viewed as a modelcomparison betweenFull model :Y β0 β1 X1 β2 X2 β3 X3 β4 X4 εReduced model :Y β0 β1 X1 β2 X2 β4 X4 ε anova(lm(Y X1 X2 X3 X4), lm(Y X1 X2 X4))One can show that the F -statistic (t-statistic)2 and the P-valuesare the same, and thus the two tests are equivalent.The proof involves complicate matrix algebra and is hence omitted.MLR - 49
Consider again the modelPrice β0 β1 FLR β2 RMS β3 BDR β4 GAR β5 LOT β6 ST β7 CON β8 LOC εfor the housing price data and want to test βBDR 0.From the output on Slide 33, we see the t-statistic for testβBDR 0 is 2.551 with P-value 0.020671.If using an F -test, lmfull lm(Price FLR RMS BDR GAR LOT ST CON LOC, data housing) lmreduced lm(Price FLR RMS GAR LOT ST CON LOC, data housing) anova(lmreduced,lmfull)Analysis of Variance Table12Res.DfRSS Df Sum of SqF Pr( F)18 380.0417 274.84 1105.2 6.5072 0.02067 *we see t 2 ( 2.551)2 6.507601 6.5072 F (the subtledifference is due to rounding), and the P-value is also 0.02067.MLR - 50
Multiple Linear Regression (MLR) Handouts Yibi Huang Data and Models Least Square Estimate, Fitted Values, Residuals Sum of Squares Do Regression in R Interpretation of Regression Coe cients t-Tests on Individual Regression Coe cients F-Tests
1947 NAID 1726771 / MLR Number A1 1001AC 454 boxes 1948 NAID 1726775 / MLR Number A1 1001AD 433 boxes 1949 NAID 1726778 / MLR Number A1 1001AE 471 boxes 1950 NAID 1726787 / MLR Number A1 1001AF 451 boxes 1951 NAID 1726792 / MLR Number A1 1001AG 417 boxes 1952 NAID 1726794 / MLR Number A1 1001AH 450 boxes
Loss Ratio Reporting System (MLR module) has been identified as the system of record to support the collection of the MLR data. The MLR data will be collected using Excel templates (MLR Annual Reporting Form). The submission window for the 2017 reporting year will open on June 30, 2018. Submission
independent variables. Many other procedures can also fit regression models, but they focus on more specialized forms of regression, such as robust regression, generalized linear regression, nonlinear regression, nonparametric regression, quantile regression, regression modeling of survey data, regression modeling of
Helping you navigate the changes A summary of MLR 2017 The MLR 2017 introduces a number of new and updated requirements on entities which fall within scope of the MLR 2017 ("Relevant Persons"). In addition, the Joint Money Laundering Steering Group ("JMLSG") has updated its AML/CTF guidance and we encourage all regulated entities to ensure
LINEAR REGRESSION 12-2.1 Test for Significance of Regression 12-2.2 Tests on Individual Regression Coefficients and Subsets of Coefficients 12-3 CONFIDENCE INTERVALS IN MULTIPLE LINEAR REGRESSION 12-3.1 Confidence Intervals on Individual Regression Coefficients 12-3.2 Confidence Interval
Standard assumptions for the multiple regression model Assumption MLR.1 (Linear in parameters) Assumption MLR.2 (Random sampling) In the population, the relation-ship between y and the expla-natory variables is linear The data is a random sample drawn from the population
Multiple Linear Regression Linear relationship developed from more than 1 predictor variable Simple linear regression: y b m*x y β 0 β 1 * x 1 Multiple linear regression: y β 0 β 1 *x 1 β 2 *x 2 β n *x n β i is a parameter estimate used to generate the linear curve Simple linear model: β 1 is the slope of the line
OF ARCHAEOLOGICAL ILLUSTRATORS & SURVEYORS LSS OCCASIONAL PAPER No. 3 AAI&S TECHNICAL PAPER No. 9 1988. THE ILLUSTRATION OF LITHIC ARTEFACTS: A GUIDE TO DRAWING STONE TOOLS FOR SPECIALIST REPORTS by Hazel Martingell and Alan Saville ASSOCIATION OF ARCHAEOLOGICAL ILLUSTRATORS & SURVEYORS THE LITHIC STUDIES SOCIETY NORTHAMPTON 1988 ISBN 0 9513246 0 8 ISSN 0950-9208. 1 Introduction This booklet .
