Chapter 11: SIMPLE LINEAR REGRESSION AND

Chapter 11: SIMPLE LINEAR REGRESSIONAND CORRELATIONPart 1: Simple Linear Regression (SLR)IntroductionSections 11-1 and 11-2Abrasion Loss vs. HardnessPrice of clock vs. Age of clock2200Price Sold at Age of Clock (yrs)1175

Regression is a method for studying therelationship between two or morequantitative variables Simple linear regression (SLR):One quantitative dependent variable- response variable- dependent variable-YOne quantitative independent variable- explanatory variable- predictor variable-X Multiple linear regression:One quantitative dependent variableMany quantitative independent variables– You’ll see this in STAT:3200/IE:3760Applied Linear Regression, if you take it.2

SLR Examples:– predict salary from years of experience– estimate effect of lead exposure on schooltesting performance– predict force at which a metal alloy rodbends based on iron content3

Example: Health dataVariables:Percent of Obese IndividualsPercent of Active IndividualsData from CDC. Units are regions of U.S. in 2014.353025Percent obese123456.PercentObesity 0.426.350.94045505560Percent Active465

3025Percent obese35A scatterplot or scatter diagram can give us ageneral idea of the relationship between obesity and activity.404550556065Percent ActiveThe points are plotted as the pairs (xi, yi)for i 1, . . . , 25Inspection suggests a linear relationship between obesity and activity (i.e. a straight linewould go through the bulk of the points, andthe points would look randomly scattered aroundthis line).5

Simple Linear RegressionThe model The basic modelYi β0 β1xi i– Yi is the observed response or dependentvariable for observation i– xi is the observed predictor, regressor, explanatory variable, independent variable,covariate– i is the error term– i are iid N (0, σ 2)(iid means independently and identically distributed)6

– So, E[Yi xi] β0 β1xi 0 β0 β1xiThe conditional mean (i.e. the expectedvalue of Yi given xi, or after conditioningon xi) is “β0 β1xi” (a point on the estimated line).– Or, as another notation, E[Y x] µY x– The random scatter around the mean (i.e.around the line) follows a N (0, σ 2) distribution.7

Example: Consider the model that regresses Oxygen purity on Hydrocarbon levelin a distillation process with.β0 75 and β1 15For each xi there is a different Oxygen purity mean (which is the center of a normaldistribution of Oxygen purity values).Plugging in xi to (75 15xi) gives you theconditional mean at xi.8

The conditional mean for x 1:E[Y x] 75 15 · 1 90The conditional mean for x 1.25:E[Y x] 75 15 · 1.25 93.759

These values that randomly scatter around aconditional mean are called errors.The random error of observation i is denotedas i. The errors around a conditional meanare normally distributed, centered at 0, andhave a variance of σ 2 or i N (0, σ ).Here, we assume all the conditional distributions of the errors are the same, so we’reusing a constant variance model.V [Yi xi] V (β0 β1xi i) V ( i) σ 210

The model can also be written as:Yi xi N (β0 β1xi , σ 2) {z }Conditionalmean– mean of Y given x is β0 β1x (known asconditional mean)– β0 β1xi is the mean value of all theY ’s for the given value of xiThe regression line itself represents all theconditional means.All the observed points will not fall on theline, there is some random noise around themean (we model this part with an error term).Usually, we will not know β0, β1, or σ 2 sowe will estimate them from the data.11

Some interpretation of parameters:– β0 is conditional mean when x 0– β1 is the slope, also stated as the changein mean of Y per 1 unit change in x– σ 2 is the variability of responses about theconditional mean12

Simple Linear RegressionAssumptions Key assumptions– linear relationship exists between Y and x*we say the relationship between Y andx is linear if the means of the conditionaldistributions of Y x lie on a straight line– independent errors(this essentially equates to independentobservations in the case of SLR)– constant variance of errors– normally distributed errors13

Simple Linear RegressionEstimationWe wish to use the sample data to estimate thepopulation parameters: the slope β1 and theintercept β0 Least squares estimation– To choose the ‘best fitting line’ using leastsquares estimation, we minimize the sumof the squared vertical distances of eachpoint to the fitted line.14

– We let ‘hats’ denote predicted values orestimates of parameters, so we have:ŷi βˆ0 βˆ1xiwhere ŷi is the estimated conditional meanfor xi,βˆ0 is the estimator for β0,and βˆ1 is the estimator for β1– We wish to choose βˆ0 and βˆ1 such that weminimize the sum of the squared verticaldistances of eachfitted line,Pn point to thei.e. minimize i 1(yi ŷi)2– Or minimize the function g:Pnˆˆg(β0, β1) i 1(yi yˆi)2 Pnˆ0 βˆ1xi))2(y (βii 115

– This vertical distance of a point from thefitted line is called a residual. The residual for observation i is denoted ei andei yi ŷi– So, in least squares estimation, we wishto minimize the sum of the squaredresiduals (or error sum of squares SSE ).– To minimize Pg(βˆ0, βˆ1) ni 1(yi (βˆ0 βˆ1xi))2we take the derivative of g with respect toβˆ0 and βˆ1, set equal to zero, and solve.nX g 2(yi (βˆ0 βˆ1xi)) 0 βˆ0 g 2 βˆ1i 1nX(yi (βˆ0 βˆ1xi))xi 0i 116

Simplifying the above gives:nβˆ0 βˆ1βˆ0nXxi βˆ1nXxi nXi 1nXi 1nXi 1i 1i 1(x2i ) yiy i xiAnd these two equations are known asthe least squares normal equations.Solving the normal equations gets us ourestimators βˆ0 and βˆ1.17

Simple Linear RegressionEstimation– Estimate of the slope:Pn(xi x̄)(yi ȳ) Sxyi 1Pn β̂1 2Sxxi 1(xi x̄)– Estimate of the Y -intercept:β̂0 ȳ β̂1x̄the point (x̄, ȳ) will always be on theleast squares lineAlternative formulas for β̂0 and β̂1 are alsogiven in the book.18

Example: Cigarette data(Nicotine vs. Tar content)1.52.0 1.0Nic 0.5 051015202530Tarn 25Least squares estimates from software:β̂0 0.1309 andβ̂1 0.0610Summary statistics:Pnx̄ 12.216Pnȳ 0.8764i 1 xi 305.4i 1 yi 21.9119

Pni 1(yi ȳ)(xi x̄) 47.01844Pn2 770.4336(x x̄)i 1 iPnPn2 22.2105yi 1 i2 4501.2xi 1 iUsing the previous formulas and the summary statistics.Sxy 47.01844ˆβ1 0.061029Sxx 770.4336andβˆ0 ȳ β̂1x̄ 0.8764 0.061029(12.216) 0.130870(Same estimates as software)20

Simple Linear RegressionEstimating σ 2 One of the assumptions of simple linear regression is that the variance for each of theconditional distributions of Y x is the sameat all x-values (i.e. constant variance). In this case, it makes sense to pool all theobserved error information (in the residuals)to come up with a common estimate for σ 221

Recall the model:iidYi β0 β1xi i with i N (0, σ 2)– We use the error sum of squares (SSE )to estimate σ 2.Pn2(y ŷ)SSiE i 1 i M SEσˆ2 n 2n 2 SSE Perror sum of squares ni 1(yi ŷi)2 M SE is the mean squared error E[M SE] E[σˆ2] σ 2p ˆ2 σ̂ σ M SE22(Unbiasedestimator)

‘2’ is subtracted from n in the denominator because we’ve used 2 degrees offreedom for estimating the slope and intercept (i.e. there were 2 parameters estimated when modeling the conditionalmean) When we estimated σ 2 in a single normalwe dividePn population,2 by (n 1) because(y ŷ)ii 1 iwe only estimated 1 mean structure parameter which was µ, now we’re estimate two parameters for our mean structure, β0 and β1.23

Chapter 11: SIMPLE LINEAR REGRESSION AND CORRELATION Part 1: Simple Linear Regression (SLR) Introduction Sections 11-1 and 11-2 Abrasion Loss vs. Hardness Price of clock vs. Age of clock 1000 1400 1800 2200 125 150 175 Age of Clock (yrs) n o ti c