Weisberg Applied Linear Regression - University Of

Preface to the Fourth EditionThis is a textbook to help you learn about applied linear regression. The bookhas been in print for more than 30 years, in a period of rapid change in statistical methodology and particularly in statistical computing. This fourth editionis a thorough rewriting of the book to reflect the needs of current students. Asin previous editions, the overriding theme of the book is to help you learn todo data analysis using linear regression. Linear regression is a excellent modelfor learning about data analysis, both because it is important on its own andit provides a framework for understanding other methods of analysis.This edition of the book includes the majority of the topics in previous editions, although much of the material has been rearranged. New methodologyand examples have been added throughout. Even more emphasis is placed on graphics. The first two editions stressedgraphics for diagnostic methods (Chapter 9) and the third edition addedgraphics for understanding data before any analysis is done (Chapter 1).In this edition, effects plots are stressed to summarize the fit of a model.Many applied analyses are based on understanding and interpretingparameters. This edition puts much greater emphasis on parameters, withpart of Chapters 2–3 and all of Chapters 4–5 devoted to this importanttopic.Chapter 6 contains a greatly expanded treatment of testing and modelcomparison using both likelihood ratio and Wald tests. The usefulnessand limitations of testing are stressed.Chapter 7 is about the variance assumption in linear models. The discussion of weighted least squares has been been expanded to coverproblems of ecological regressions, sample surveys, and other cases.Alternatives such as the bootstrap and heteroskedasticity correctionshave been added or expanded.Diagnostic methods using transformations (Chapter 8) and residuals andrelated quantities (Chapter 9) that were the heart of the earlier editionshave been maintained in this new edition.xv

xvi preface to the fourth editionThe discussion of variable selection in Chapter 10 has been updated fromthe third edition. It is designed to help you understand the key problemsin variable selection. In recent years, this topic has morphed into the areaof machine learning and the goal of this chapter is to show connectionsand provide references.As in the third edition, brief introductions to nonlinear regression(Chapter 11) and to logistic regression (Chapter 12) are included, withPoisson regression added in Chapter 12.Using This BookThe website for this book is http://z.umn.edu/alr4ed.As with previous editions, this book is not tied to any particular computerprogram. A primer for using the free R package (R Core Team, 2013) for thematerial covered in the book is available from the website. The primer canalso be accessed directly from within R as you are working. An optional published companion book about R is Fox and Weisberg (2011).All the data files used are available from the website and in an R packagecalled alr4 that you can download for free. Solutions for odd-numberedproblems, all using R, are available on the website for the book1. You cannotlearn to do data analysis without working problems.Some advanced topics are introduced to help you recognize when a problemthat looks like linear regression is actually a little different. Detailed methodology is not always presented, but references at the same level as this bookare presented. The bibliography, also available with clickable links on thebook’s website, has been greatly expanded and updated.Mathematical LevelThe mathematical level of this book is roughly the same as the level of previous editions. Matrix representation of data is used, particularly in the derivation of the methodology in Chapters 3–4. Derivations are less frequent in laterchapters, and so the necessary mathematics is less. Calculus is generally notrequired, except for an occasional use of a derivative. The discussions requiringcalculus can be skipped without much loss.ACKNOWLEDGMENTSThanks are due to Jeff Witmer, Yuhong Yang, Brad Price, and Brad’s Stat 5302students at the University of Minnesota. New examples were provided byApril Bleske-Rechek, Tom Burk, and Steve Taff. Work with John Fox over thelast few years has greatly influenced my writing.For help with previous editions, thanks are due to Charles Anderson, DonPereira, Christopher Bingham, Morton Brown, Cathy Campbell, Dennis Cook,1All solutions are available to instructors using the book in a course; see the website for details.

preface to the fourth editionxviiStephen Fienberg, James Frane, Seymour Geisser, John Hartigan, DavidHinkley, Alan Izenman, Soren Johansen, Kenneth Koehler, David Lane,Michael Lavine, Kinley Larntz, Gary Oehlert, Katherine St. Clair, Keija Shan,John Rice, Donald Rubin, Joe Shih, Pete Stewart, Stephen Stigler, DouglasTiffany, Carol Weisberg, and Howard Weisberg.Finally, I am grateful to Stephen Quigley at Wiley for asking me to do anew edition. I have been working on versions of this book since 1976, and eachnew edition has pleased me more that the one before it. I hope it pleasesyou, too.Sanford WeisbergSt. Paul, MinnesotaSeptember 2013

CHAPTER 1Scatterplots and RegressionRegression is the study of dependence. It is used to answer interesting questions about how one or more predictors influence a response. Here are a fewtypical questions that may be answered using regression: Are daughters taller than their mothers?Does changing class size affect success of students?Can we predict the time of the next eruption of Old Faithful Geyser fromthe length of the most recent eruption?Do changes in diet result in changes in cholesterol level, and if so, do theresults depend on other characteristics such as age, sex, and amount ofexercise?Do countries with higher per person income have lower birth rates thancountries with lower income?Are highway design characteristics associated with highway accidentrates? Can accident rates be lowered by changing designcharacteristics?Is water usage increasing over time?Do conservation easements on agricultural property lower land value?In most of this book, we study the important instance of regression methodology called linear regression. This method is the most commonly used inregression, and virtually all other regression methods build upon an understanding of how linear regression works.As with most statistical analyses, the goal of regression is to summarizeobserved data as simply, usefully, and elegantly as possible. A theory may beavailable in some problems that specifies how the response varies as the valuesApplied Linear Regression, Fourth Edition. Sanford Weisberg. 2014 John Wiley & Sons, Inc. Published 2014 by John Wiley & Sons, Inc.1

2chapter 1 scatterplots and regressionof the predictors change. If theory is lacking, we may need to use the data tohelp us decide on how to proceed. In either case, an essential first step inregression analysis is to draw appropriate graphs of the data.We begin in this chapter with the fundamental graphical tools for studyingdependence. In regression problems with one predictor and one response, thescatterplot of the response versus the predictor is the starting point for regression analysis. In problems with many predictors, several simple graphs will berequired at the beginning of an analysis. A scatterplot matrix is a convenientway to organize looking at many scatterplots at once. We will look at severalexamples to introduce the main tools for looking at scatterplots and scatterplotmatrices and extracting information from them. We will also introduce notation that will be used throughout the book.1.1SCATTERPLOTSWe begin with a regression problem with one predictor, which we will generically call X, and one response variable, which we will call Y.1 Data consist ofvalues (xi, yi), i 1, . . . , n, of (X, Y) observed on each of n units or cases. Inany particular problem, both X and Y will have other names that will be displayed in this book using typewriter font, such as temperature orconcentration, that are more descriptive of the data that are to be analyzed. The goal of regression is to understand how the values of Y change asX is varied over its range of possible values. A first look at how Y changes asX is varied is available from a scatterplot.Inheritance of HeightOne of the first uses of regression was to study inheritance of traits fromgeneration to generation. During the period 1893–1898, Karl Pearson (1857–1936) organized the collection of n 1375 heights of mothers in the UnitedKingdom under the age of 65 and one of their adult daughters over the ageof 18. Pearson and Lee (1903) published the data, and we shall use these datato examine inheritance. The data are given in the data file Heights.2Our interest is in inheritance from the mother to the daughter, so weview the mother’s height, called mheight, as the predictor variable andthe daughter’s height, dheight, as the response variable. Do taller motherstend to have taller daughters? Do shorter mothers tend to have shorterdaughters?A scatterplot of dheight versus mheight helps us answer these questions.The scatterplot is a graph of each of the n points with the response dheighton the vertical axis and predictor mheight on the horizontal axis. This plot is1In some disciplines, predictors are called independent variables, and the response is called adependent variable, terms not used in this book.2See Appendix A.1 for instructions for getting data files from the Internet.

1.13scatterplots(b)75757070Rounded dheightJittered dheight(a)65605565605555606570Jittered mheight755560657075Rounded mheightFigure 1.1 Scatterplot of mothers’ and daughters’ heights in the Pearson and Lee data. The original data have been jittered to avoid overplotting in (a). Plot (b) shows the original data, so eachpoint in the plot refers to one or more mother–daughter pairs.shown in Figure 1.1a. For regression problems with one predictor X and aresponse Y, we call the scatterplot of Y versus X a summary graph.Here are some important characteristics of this scatterplot:1. The range of heights appears to be about the same for mothers and fordaughters. Because of this, we draw the plot so that the lengths of thehorizontal and vertical axes are the same, and the scales are the same. Ifall mothers and daughters pairs had exactly the same height, then all thepoints would fall exactly on a 45 -line. Some computer programs fordrawing a scatterplot are not smart enough to figure out that the lengthsof the axes should be the same, so you might need to resize the plot orto draw it several times.2. The original data that went into this scatterplot were rounded so eachof the heights was given to the nearest inch. The original data are plottedin Figure 1.1b. This plot exhibits substantial overplotting with manypoints at exactly the same location. This is undesirable because one pointon the plot can correspond to many cases. The easiest solution is to usejittering, in which a small uniform random number is added to each value.In Figure 1.1a, we used a uniform random number on the range from 0.5 to 0.5, so the jittered values would round to the numbers given inthe original source.3. One important function of the scatterplot is to decide if we might reasonably assume that the response on the vertical axis is independent of thepredictor on the horizontal axis. This is clearly not the case here sinceas we move across Figure 1.1a from left to right, the scatter of points is

4chapter 1 scatterplots and regression75 70 dheight 65 60 555560657075mheightFigure 1.2 Scatterplot showing only pairs with mother’s height that rounds to 58, 64, or 68 inches.different for each value of the predictor. What we mean by this is shownin Figure 1.2, in which we show only points corresponding to mother–daughter pairs with mheight rounding to either 58, 64, or 68 inches. Wesee that within each of these three strips or slices, the number of pointsis different, and the mean of dheight is increasing from left to right.The vertical variability in dheight seems to be more or less the samefor each of the fixed values of mheight.4. In Figure 1.1a the scatter of points appears to be more or less ellipticallyshaped, with the major axis of the ellipse tilted upward, and with morepoints near the center of the ellipse rather than on the edges. We will seein Section 1.4 that summary graphs that look like this one suggest the useof the simple linear regression model that will be discussed in Chapter 2.5. Scatterplots are also important for finding separated points. Horizontalseparation would occur for a value on the horizontal axis mheight thatis either unusually small or unusually large relative to the other valuesof mheight. Vertical separation would occur for a daughter withdheight either relatively large or small compared with the other daughters with about the same value for mheight.These two types of separated points have different names and rolesin a regression problem. Extreme values on the left and right of thehorizontal axis are points that are likely to be important in fitting regression models and are called leverage points. The separated points on thevertical axis, here unusually tall or short daughters give their mother’sheight, are potentially outliers, cases that are somehow different from

1.15scatterplotsthe others in the data. Outliers are more easily discovered in residualplots, as illustrated in the next example.While the data in Figure 1.1a do include a few tall and a few shortmothers and a few tall and short daughters, given the height of themothers, none appears worthy of special treatment, mostly because in asample size this large, we expect to see some fairly unusual mother–daughter pairs.Forbes’s DataIn an 1857 article, the Scottish physicist James D. Forbes (1809–1868) discusseda series of experiments that he had done concerning the relationship betweenatmospheric pressure and the boiling point of water. He knew that altitudecould be determined from atmospheric pressure, measured with a barometer,with lower pressures corresponding to higher altitudes. Barometers in themiddle of the nineteenth century were fragile instruments, and Forbes wondered if a simpler measurement of the boiling point of water could substitutefor a direct reading of barometric pressure. Forbes collected data in the Alpsand in Scotland. He measured at each location the atmospheric pressure presin inches of mercury with a barometer and boiling point bp in degrees Fahrenheit using a thermometer. Boiling point measurements were adjusted forthe difference between the ambient air temperature when he took the measurements and a standard temperature. The data for n 17 locales are reproduced in the file Forbes.The scatterplot of pres versus bp is shown in Figure 1.3a. The generalappearance of this plot is very different from the summary graph for theheights data. First, the sample size is only 17, as compared with over 1,300 forthe heights data. Second, apart from one point, all the points fall almost exactlyon a smooth curve. This means that the variability in pressure for a givenboiling point is extremely 0 0.2195200205Boiling point210195200205Boiling pointFigure 1.3 Forbes data: (a) pres versus bp; (b) residuals versus bp.210

6chapter 1 scatterplots and regressionThe points in Figure 1.3a appear to fall very close to the straight lineshown on the plot, and so we might be encouraged to think that the meanof pressure given boiling point could be modeled by a straight line. Lookclosely at the graph, and you will see that there is a small systematic deviationfrom the straight line: apart from the one point that does not fit at all,the points in the middle of the graph fall below the line, and those at thehighest and lowest boiling points fall above the line. This is much easier tosee in Figure 1.3b, which is obtained by removing the linear trend fromFigure 1.3a, so the plotted points on the vertical axis are given for each valueof bp byresidual pres point on the lineThis allows us to gain resolution in the plot since the range on the verticalaxis in Figure 1.3a is about 10 inches of mercury while the range in Figure 1.3bis about 0.8 inches of mercury. To get the same resolution in Figure 1.3a, wewould need a graph that is 10/0.8 12.5 as big as Figure 1.3b. Again ignoringthe one point that clearly does not match the others, the curvature in the plotis clearly visible in Figure 1.3b.While there is nothing at all wrong with curvature, the methods we will bestudying in this book work best when the plot can be summarized by a straightline. Sometimes we can get a straight line by transforming one or both of theplotted quantities. Forbes had a physical theory that suggested that log(pres)is linearly related to bp. Forbes (1857) contains what may be the first publishedsummary graph based on his physical model. His figure is redrawn in Figure1.4. Following Forbes, we use base-ten common logs in this example, althoughin most of the examples in this book we will use natural logarithms. The .0500.000195200205Boiling point210195200205210Boiling pointFigure 1.4 (a) Scatterplot of Forbes’s data. The line shown is the ols line for the regression oflog(pres) on bp. (b) Residuals versus bp.

1.17scatterplotsof base has no material effect on the appearance of the graph or on fittedregression models, but interpretation of parameters can depend on the choiceof base.The key feature of Figure 1.4a is that apart from one point, the data appearto fall very close to the straight line shown on the figure, and the residual plotin Figure 1.4b confirms that the deviations from the straight line are not systematic the way they were in Figure 1.3b. All this is evidence that the straightline is a reasonable summary of these data.Length at Age for Smallmouth BassThe smallmouth bass is a favorite game fish in inland lakes. Many smallmouthbass populations are managed through stocking, fishing regulations, and othermeans, with a goal to maintain a healthy population.One tool in the study of fish populations is to understand the growth patternof fish such as the dependence of a measure of size like fish length on age ofthe fish. Managers could compare these relationships between different populations that are managed differently to learn how management impacts fishgrowth.Figure 1.5 displays the Length at capture in mm versus Age at capture forn 439 smallmouth bass measured in West Bearskin Lake in NortheasternMinnesota in 1991. Only fish of age 8 or less are included in this graph. Thedata were provided by the Minnesota Department of Natural Resources andare given in the file wblake. Similar to trees, the scales of many fish specieshave annular rings, and these can be counted to determine the age of a fish.350300Length2502001501005012345678AgeFigure 1.5 Length (mm) versus Age for West Bearskin Lake smallmouth bass. The solid lineshown was estimated using ordinary least squares or ols. The dashed line joins the averageobserved length at each age.

8chapter 1 scatterplots and regressionThese data are cross-sectional, meaning that all the obser

Weisberg, Sanford, 1947– Applied linear regression / Sanford Weisberg, School of Statistics, University of Minnesota, Minneapolis, MN.—Fourth edition. pages cm Includes bibliographical references and index. ISBN 978-1-118-38608-8 (hardback) 1. Regression analysis. I. Title. QA278.2.W44 2014 519.5′36–dc23 2014026538