Robust Regression Modeling With STATA Lecture Notes

Robust RegressionModelingwith STATAlecture notesRobert A. Yaffee, Ph.D.Statistics, Social Science, and Mapping GroupAcademic Computing ServicesOffice:75 Third Avenue, Level C-3Phone: 212-998-3402Email: yaffee@nyu.edu1

What does Robust mean?1.Definitions differ in scope and content. In the mostgeneral construction: Robust models pertains tostable and reliable models.2. Strictly speaking:Threats to stability and reliability includeinfluential outliersInfluential outliers played havoc with statisticalestimation. Since 1960, many robusttechniques of estimation have developed thathave been resistant to the effects of suchoutliers.SAS Proc Robustreg in Version 9deals with these.S-Plus robust library inStata rreg, prais, and arimamodels3.Broadly speaking: HeteroskedasticityHeteroskedastically consistent varianceestimatorsStata regress y x1 x2, robust4.Non-normal residuals1. Nonparametric Regression modelsStata qreg, rreg2. Bootstrapped Regression1. bstrap2. bsqreg2

Outline1.Regression modeling preliminaries1. Tests for misspecification1. Outlier influence2. Testing for normality3. Testing for heterskedasticity4. Autocorrelation of residuals2. Robust Techniques1. Robust Regression2. Median or quantile regression3. Regression with robust standard errors4. Robust autoregression models3. Validation and cross-validation1. Resampling2. Sample splitting4. Comparison of STATA with SPLUS and SAS3

Preliminary Testing: Prior tolinear regression modeling, use amatrix graph to confirm linearityof relationshipsgraph y x1 x2, 244.215.819.14

The independent variablesappear to be linearlyrelated with yWe try to keep the models simple. If therelationships are linear then we model them withlinear models. If the relationships are nonlinear,then we model them with nonlinear ornonparametric models.5

Theory of RegressionAnalysisWhat is linear regressionAnalysis?Finding the relationship between adependent and an independentvariable.Y a bx eGraphically, this can be done witha simple Cartesian graph6

The MultipleRegression FormulaY a bx eY is the dependent variablea is the interceptb is the regression coefficientx is the predictor variable7

Graphical Decompositionof EffectsDecomposition of Effectsyi y Total Effect{Y}}Yiŷ a bxyi yˆi errorŷ y regression effectYXX8

Derivation of the Intercepty a bx ee y a bxn ei 1 inn y aii 1i 1in b xii 1nBecause by definition ei 0i 10 nn y ai 1ii 1nnni 1i 1i 1in b xii 1 ai yi b xinni 1i 1na yi b xia y bx9

Derivation of theRegression CoefficientGiven : yi a b xi eiei yi a b xin eii 1n (y ii 1n a b xi )n2e i 2y a bx() iii 1i 1n ei 2ni 1i 1 2 xi ( yi ) 2b xi xii 1 b0nnni 1i 1 2 xi ( yi ) 2b xi xinb x yi 1ni2x ii 1i10

If we recall that the formula forthe correlation coefficient canbe expressed as follows:11

n r i 1xi yi ( x ) ( y )nn2i 1i2ii 1w herex xi xy yi ynbj xii 1n xyi2i 1from which it can be seen that the regression coefficient b,is a function of r.bj r *sd ysd x12

Extending the bivariate to the multivariateCase13

β yx . x 12β yx . x 21ryx1 ryx2 rx1x21 r2x1 x2ryx2 ryx1 rx1x21 r*2x1 x2*sd ysd xsd ysd x(6)(7)It is also easy to extend the bivariate interceptto the multivariate case as follows.a Y b1 x1 b2 x2 (8)14

Linear MultipleRegression Suppose that we have thefollowing data set.15

Stata OLS regressionmodel syntaxWe now see that the significance levels reveal that x1 and x2are both statistically significant. The R2 and adjusted R2have not been significantly reduced, indicating that this model stillfits well. Therefore, we leave the interaction term pruned from themodel.What are the assumptions of multiple linear regression analysis?16

Regression modelingand the assumptions1. What are the assumptions?1. linearity2. Heteroskedasticity3. No influential outliers in smallsamples4. No multicollinearity5. No autocorrelation of residuals6. Fixed independent variables-nomeasurement error7. Normality of residuals17

Testing the model formispecification androbustnessLinearitymatrix graphs shown aboveMulticollinearityvifMisspecification testsheteroskedasticity testsrvfplothettestresidual autocorrelation testscorrgramoutlier detectiontabulation of standardized residualsinfluence assessmentresidual normality testssktestSpecification tests (not covered in this lecture)18

Misspecification tests We need to test the residualsfor normality. We can save the residuals inSTATA, by issuing a commandthat creates them, after wehave run the regressioncommand. The command to generate theresiduals is predict resid, residuals19

Generation of the regression residuals20

Generation ofstandardized residuals Predict rstd, rstandard21

Generation ofstudentized residuals Predict rstud, rstudent22

Testing the Residualsfor Normality1. We use a Smirnov-Kolmogorovtest.2. The command for the test is:sktest residThis tests the cumulative distribution of the residuals against that ofthe theoretical normal distribution with a chi-square testTo determine whether there is a statistically significant difference.The null hypothesis is that there is no difference. When the probabilityis less than .05, we must reject the null hypothesis and infer that23the residuals are non-normally distributed.

Testing the Residualsfor heteroskedasticity1. We may graph the standardized orstudentized residuals against thepredicted scores to obtain a graphicalindication of heteroskedasticity.2. The Cook-Weisberg test is used to testthe residuals for heteroskedasticity.24

A Graphical test ofheteroskedasticity:rvfplot, border yline(0)This displays any problematic patterns that might suggestheteroskedasticity. But it doesn’t tell us which residuals areoutliers.25

Cook-Weisberg TestVar (ei ) σ 2 exp( zt )whereei error in regression modelz x βˆ or variable list supplied by userThe test is whether t 0hettest estimates the model ei 2 α zi t ν iSS of model2 χ 2 where p number of parametersit forms a score test S h0 : S df p26

Cook-Weisberg testsyntax1. The command for this test is:hettest residAn insignificant result indicates lack of heteroskedasticity.That is, an such a result indicates the presence of equal varianceof the residuals along the predicted line. This condition isotherwise known as homoskedasticity.27

Testing the residuals forAutocorrelation1. One can use the command,dwstat, after the regression toobtain the Durbin-Watson dstatistic to test for first-orderautocorrelation.2. There is a better way.Generate a casenumvariable: Gen casenum n28

Create a timedependent series29

Run the Ljung-Box Q statisticwhich tests previous lags forautocorrelation and partialautocorrelationThe STATA command is :corrgram residThe significance of the AC (Autocorrelation) and PAC(Partial autocorrelation) is shown in the Prob column.None of these residuals has any significant autocorrelation.30

One can runAutoregression in theevent of autocorrelationThis can be done withnewey y x1 x2 x3 lag(1) timeprais y x1 x2 x331

Outlier detection Outlier detection involves thedetermination whether the residual(error predicted – actual) is anextreme negative or positive value. We may plot the residual versusthe fitted plot to determine whicherrors are large, after running theregression. The command syntax was alreadydemonstrated with the graph onpage 16: rvfplot, border yline(0)32

Create StandardizedResiduals A standardized residual is onedivided by its standard deviation.yˆi yiresid standardized swhere s std dev of residuals33

Standardized residualspredict residstd, rstandardlist residstdtabulate residstd34

Limits of StandardizedResidualsIf the standardized residualshave values in excess of 3.5and -3.5, they are outliers.If the absolute values are lessthan 3.5, as these are, thenthere are no outliersWhile outliers by themselvesonly distort mean predictionwhen the sample size is smallenough, it is important togauge the influence of outliers.35

Outlier Influence Suppose we had a differentdata set with two outliers. We tabulate the standardizedresiduals and obtain thefollowing output:36

Outlier a does not distortthe regression line butoutlier b does.baY a bxOutlier a has bad leverage and outlier adoes not.37

In this data set, we have two outliers. One is negative and theother is positive.38

Studentized Residuals Alternatively, we could formstudentized residuals. These aredistributed as a t distribution withdf n-p-1, though they are notquite independent. Therefore, wecan approximately determine ifthey are statistically significant ornot. Belsley et al. (1980)recommended the use ofstudentized residuals.39

Studentized Residualei seis 2 (i ) (1 hi )whereei s studentized residuals( i ) standard deviation where ith obs is deletedhi leverage statisticThese are useful in estimating the statistical significanceof a particular observation, of which a dummy variableindicator is formed. The t value of the studentized residualwill indicate whether or not that observation is a significantoutlier.The command to generate studentized residuals, called rstudt is:predict rstudt, rstudent40

Influence of Outliers1. Leverage is measured by thediagonal components of the hatmatrix.2. The hat matrix comes from theformula for the regression of Y.Yˆ X β X '( X ' X ) 1 X ' Ywhere X '( X ' X ) 1 X ' the hat matrix, HTherefore,Yˆ HY41

Leverage and the Hatmatrix1.2.3.4.5.6.The hat matrix transforms Y into thepredicted scores.The diagonals of the hat matrix indicatewhich values will be outliers or not.The diagonals are therefore measures ofleverage.Leverage is bounded by two limits: 1/n and1. The closer the leverage is to unity, themore leverage the value has.The trace of the hat matrix the number ofvariables in the model.When the leverage 2p/n then there is highleverage according to Belsley et al. (1980)cited in Long, J.F. Modern Methods ofData Analysis (p.262). For smaller samples,Vellman and Welsch (1981) suggested that3p/n is the criterion.42

Cook’s D1. Another measure of influence.2. This is a popular one. Theformula for it is: 1 hi ei 2Cook ' s Di 2 p 1 hi s (1 hi ) Cook and Weisberg(1982) suggested that values ofD that exceeded 50% of the F distribution (df p, n-p)are large.43

Using Cook’s D inSTATA Predict cook, cooksd Finding the influential outliers List cook, if cook 4/n Belsley suggests 4/(n-k-1) as a cutoff44

Graphical Exploration ofOutlier Influence Graph cook residstd, xlab ylabThe two influential outliers can be found easily herein the upper right.45

DFbeta One can use the DFbetas toascertain the magnitude ofinfluence that an observation hason a particular parameter estimateif that observation is deleted.DFbeta j b j b(i ) j u j u2j(1 h j )where u j residuals ofregression of x on remaining xs.46

Obtaining DFbetas inSTATA47

Robust statistical optionswhen assumptions areviolated1.Nonlinearity1.2.2.Influential Outliers1.2.3.2.Regression with Huber/White/Sandwichvariance-covariance estimatorsRegress y x1 x2, robustResidual autocorrelation correction1.2.5.Robust regression with robust weight functionsrreg y x1 x2Heteroskedasticity of residuals1.4.Transformation to linearityNonlinear regressionAutoregression withprais y x1 x2, robustnewey-west regressionNonnormality of residuals1.2.Quantile regression: qreg y x1 x2Bootstrapping the regression coefficients48

Nonlinearity:Transformations to linearity1. When the equation is notintrinsically nonlinear, thedependent variable orindependent variable may betransformed to effect alinearization of the relationship.2. Semi-log, translog, Box-Cox, orpower transformations may beused for these purposes.1. Boxcox regression permitsdetermines the optimal parametersfor many of these transformations.49

Fix for Nonlinear functionalform: Nonlinear RegressionAnalysisExamples of 2 exponential growth curve models, the firstof which we estimate with our data.nl exp2 y xestimates Y b1b2xnl exp3 y xestimates y b0 b1b2x50