Chapter 12. Simple Linear Regression And Correlation
Chapter 12. Simple LinearRegression and Correlation12.1 The Simple Linear Regression Model12.2 Fitting the Regression Line12.3 Inferences on the Slope Rarameter β112.4 Inferences on the Regression Line12.5 Prediction Intervals for Future Response Values12.6 The Analysis of Variance Table12.7 Residual Analysis12.8 Variable Transformations12.9 Correlation Analysis12.10 Supplementary ProblemsNIPRL1
12.1 The Simple Linear Regression Model12.1.1 Model Definition and Assumptions(1/5) With the simple linear regression modelyi β0 β1xi εithe observed value of the dependent variable yi is composed of alinear function β0 β1xi of the explanatory variable xi, together with anerror term εi. The error terms ε1, ,εn are generally taken to beindependent observations from a N(0,σ2) distribution, for some errorvariance σ2. This implies that the values y1, ,yn are observationsfrom the independent random variablesYi N (β0 β1xi, σ2)as illustrated in Figure 12.1NIPRL2
12.1.1 Model Definition and Assumptions(2/5)NIPRL3
12.1.1 Model Definition and Assumptions(3/5)The parameter β0 is known as the intercept parameter, and theparameter β0 is known as the intercept parameter,parameter and theparameter β1 is known as the slope parameter.parameter A third unknownparameter, the errorvariance σ2, can alsobe estimated from thedata set. As illustratedin Figure 12.2, the datavalues (xi , yi ) lie closerto the liney β0 β1xas the error variance σ2decreases.NIPRL4
12.1.1 Model Definition and Assumptions(4/5) The slope parameter β1 is of particular interest since it indicates howthe expected value of the dependent variable depends upon theexplanatory variable x, as shown in Figure 12.3 The data set shown in Figure 12.4 exhibits aquadratic (or at least nonlinear) relationshipbetween the two variables, and it would makeno sense to fit a straight line to the data set.NIPRL5
12.1.1 Model Definition and Assumptions(5/5) Simple Linear Regression ModelThe simple linear regression modelyi β0 β1xi εifit a straight line through a set of paired data observations(x1,y1), ,(xn, yn). The error terms ε1, ,εn are taken to beindependent observations from a N(0,σ2) distribution. The threeunknown parameters, the intercept parameter β0 , the slopeparameter β1, and the error variance σ2, are estimated from the dataset.NIPRL6
12.1.2 Examples(1/2) Example 3 : Car Plant Electricity UsageThe manager of a car plant wishes to investigate how the plant’selectricity usage depends upon the plant’s production.The linear modely β 0 β1 xwill allow a month’s electricalusage to be estimated as afunction of the month’s production.NIPRL7
12.1.2 Examples(2/2)NIPRL8
12.2 Fitting the Regression Line12.2.1 Parameter Estimation(1/4)The regression line y β 0 β1 x is fitted to the data points ( x1 , y1 ),K , ( xn , yn )by finding the line that is "closest" to the data points in some sense.As Figure 12.14 illustrates, the fitted line is chosen to be the line that minimizesthe sum of the squares of these vertical deviationsQ in 1 ( yi ( β 0 β1 xi )) 2and this is referred to asthe least squares fit.NIPRL9
12.2.1 Parameter Estimation(2/4)With normally distributed error terms, βˆ0 and βˆ1 are maximumlikelihood estimates.( Q ) The joint density of the error terms ε1 ,K , ε n isnn ε i2 1 i2 1σ 2. e 2πσ This likelihood is maximized by minizing ε i2 ( yi ( β 0 β1 xi )) 2 Q Q in 1 2( yi ( β 0 β1 xi )) and β 0 Q in 1 2 xi ( yi ( β 0 β1 xi )) β1 the normal equations yi nβˆ0 βˆ1 in 1 xi and in 1 xi yi βˆ0 in 1 xi βˆ1 in 1 xi2NIPRL10
12.2.1 Parameter Estimation(3/4)n in 1 xi yi ( in 1 xi )( in 1 yi ) S XYβ1 nnn i 1xi2 ( i 1xi ) 2S XXand then in 1 yi in 1 xiβ0 β1 y β 1xnnwhereS XX in 1 ( xi x ) 2 in 1 xi2 nx 2 ni 1( in 1 xi ) 2x n2iand( in 1 xi )( in 1 yi )S XY ( xi x )( yi y ) xi yi nxy xi yi nFor a specific value of the explanatory variable x* , this equationni 1ni 1ni 1provides a fitted value yˆ x* β 0 β 1 x* for the dependent variable y, asillustrated in Figure 12.15.NIPRL11
12.2.1 Parameter Estimation(4/4)The error variance σ 2 can be estimated by considering thedeviations between the observed data values yi and their fittedvalues yi . Specifically, the sum of squares for error SSE isdefined to be the sum of the squares of these deviationsSSE in 1 ( yi yi ) 2 in 1 ( yi ( β 0 β 1 xi )) 2 in 1 yi2 β 0 in 1 yi β1 in 1 xi yiand the estimate of the error variance is2SSEσ n 2NIPRL12
12.2.2 Examples(1/5) Example 3 : Car Plant Electricity UsageFor this example n 12 and12 xi 4.51 L 4.20 58.62i 112 yi 2.48 L 2.53 34.152i 4.512 L 4.202 291.2310i 112 xi 112222y 2.48 L 2.53 98.6967 ii 112 x yii (4.51 2.48) L (4.20 2.53) 169.2532i 1NIPRL13
12.2.2 Examples(2/5)NIPRL14
12.2.2 Examples(3/5)The estimates of the slope parameter and the intercept parameter :nβ1 nnn xi yi ( xi )( yi )i 1i 1i 1nni 1i 1n xi2 ( xi ) 2(12 169.2532) (58.62 34.15) 0.498832(12 291.2310) 58.6234.1558.62β 0 y β1 x (0.49883 ) 0.40901212 The fitted regression line :y β 0 β1 x 0.409 0.499 x y 0.409 (0.499 5.5) 3.15355.5NIPRL15
12.2.2 Examples(4/5)Using the model for production values x outside this range is knownas extrapolation and may give inaccurate results.NIPRL16
12.2.2 Examples(5/5)n2σ y2ii 1nni 1i 1 β 0 yi β1 xi yin 298.6967 (0.4090 34.15) (0.49883 169.2532) 0.029910 σ 0.0299 0.1729NIPRL17
12.3 Inferences on the Slope Parameter β112.3.1 Inference Procedures(1/4)Inferences on the Slope Parameter β1βˆ1Ν( β1 ,σ2S XX).A two-sided confidence interval with a confidence level 1 α for the slopeparameter in a simple linear regression model isβ1 ( β1 tα / 2,n 2 s.e.( β1 ),β1 tα / 2,n 2 s.e.( β1 ))which isβ1 ( β1 σ tα / 2,n 2S XX,β1 σ tα / 2,n 2S XX)One-sided 1 α confidence level confidence intervals areβ1 ( ,NIPRLβ1 σ tα ,n 2S XX) and β1 ( β1 σ tα ,n 2S XX, )18
12.3.1 Inference Procedures(2/4) The two-sided hypothesesH 0 : β1 b1 versus H A : β1 b1for a fixed value b1 of interest are tested with the t -statistict β1 b1σ S XXThe p-value isp-value 2 P( X t )where the random variable X has a t -distribution with n 2 degrees of freedom.A size α test rejects the null hypothesis if t tα / 2,n 2 .NIPRL19
12.3.1 Inference Procedures(3/4) The one-sided hypothesesH 0 : β1 b1 versus H A : β1 b1have a p-valuep -value P( X t )and a size α test rejects the null hypothesis if t tα ,n 2 . The one-sided hypothesesH 0 : β1 b1 versus H A : β1 b1have a p-valuep-value P ( X t )and a size α test rejects the null hypothesis if t tα ,n 2 .Slki Lab.NIPRL20
12.3.1 Inference Procedures(4/4) An interesting point to notice is that for a fixed value of the errorvariance σ2, the variance of the slope parameter estimate decreasesas the value of SXX increases. This happens as the values of theexplanatoryvariable xi become morespread out, as illustratedin Figure 12.30. This resultis intuitively reasonablesince a greater spreadin the values xi providesa greater “leverage” forfitting the regression line,and therefore the slopeparameter estimate β 1should be more accurate.NIPRL21
12.3.2 Examples(1/2) Example 3 : Car Plant Electricity Usage1212S XX x 2i( xi )212i 1 s.e.( β1 ) 58.622 291.2310 4.872312i 1σS XX 0.1729 0.07834.8723The t -statistic for testing H 0 : β1 0tβ1( )s.e. β1 0.49883 6.370.0783The two-sided p -valuep value 2 P ( X 6.37)NIPRL022
12.3.2 Examples(2/2)With t0.005,10 3.169, a 99% two-sided confidence interval for theslope parameterβ1 ( β1 critical point s.e.( β1 ), β1 critical point s.e.( β1 )) ( 0.49883 3.169 0.0783,0.49883 3.169 0.0783) ( 0.251, 0.747 )NIPRL23
12.4 Inferences on the Regression Line12.4.1 Inference Procedures(1/2)Inferences on the Expected Value of the Dependent VariableA 1 α confidence level two-sided confidence interval for β 0 β1 x* , theexpected value of the dependent variable for a particular value x* of the explanatory variable, isβ 0 β1 x* ( β 0 β1 x* tα / 2,n 1 s.e.( β 0 β1 x* ),β 0 β1 x* tα / 2,n 2 s.e.( β 0 β1 x* ))where1 ( x* x ) 2s.e.( β 0 β1 x ) σ nS XX*NIPRL24
12.4.1 Inference Procedures(2/2)One-sided confidence intervals areβ 0 β1 x* ( , β 0 β1 x* tα ,n 2 s.e.( β 0 β1 x* ))andβ 0 β1 x* ( β 0 β1 x* tα ,n 1 s.e.( β 0 β1 x* ), )Hypothesis tests on β 0 β1 x* can be performed by comparing the t -statistict ( β 0 β1 x* ) ( β 0 β1 x* )s.e.( β 0 β1 x* )with a t -distribution with n 2 degrees of freedom.NIPRL25
12.4.2 Examples(1/2) Example 3 : Car Plant Electricity Usage1 (x x)1 ( x* 4.885) 2s.e.( β 0 β1 x ) σ 0.1729 nS XX124.87232**With t0.025,10 2.228, a 95% confidence interval for β 0 β1 x*1 ( x* 4.885) 2β 0 β1 x (0.409 0.499 x 2.228 0.1729 ,124.8723**1 ( x* 4.885) 20.409 0.499 x 2.228 0.179 )4.872312*At x* 5β 0 5β1 (0.409 (0.499 5) 0.113, 0.409 (0.499 5) 0.113) (2.79,3.02)NIPRL26
12.4.2 Examples(2/2)NIPRL27
12.5 Prediction Intervals for Future Response Values12.5.1 Inference Procedures(1/2) Prediction Intervals for Future Response ValuesA 1 α confidence level two-sided prediction interval for y x* , a future valueof the dependent variable for a particular value x* of the explanatory variable,is1 ( x* x ) 2y x* ( β 0 β1 x tα / 2, n 1σ 1 ,nS XX*1 ( x* x ) 2β 0 β1 x tα / 2,n 2 σ 1 )nS XX*NIPRL28
12.5.1 Inference Procedures(2/2)One-sided confidence intervals arey x* ( ,1 ( x* x ) 2β 0 β1 x tα ,n 2 σ 1 )nS XX*and*21(x x)y x* ( β 0 β1 x* tα ,n 1σ 1 , )nS XXNIPRL29
12.5.2 Examples(1/2) Example 3 : Car Plant Electricity UsageWith t0.025,10 2.228, a 95% confidence interval for y x*13 ( x* 4.885) 2y x* (0.409 0.499 x 2.228 0.1729 ,124.8723**213(x 4.885)0.409 0.499 x* 2.228 0.179 ) 124.8723At x* 5y 5 (0.409 (0.499 5) 0.401, 0.409 (0.499 5) 0.401) (2.50,3.30)NIPRL30
12.5.2 Examples(2/2)NIPRL31
12.6 The Analysis of Variance Table12.6.1 Sum of Squares Decomposition(1/5)NIPRL32
12.6.1 Sum of Squares Decomposition(2/5)NIPRL33
12.6.1 Sum of Squares Decomposition(3/5)SourceDegrees of freedomSum of squaresRegressionError1N-2SSRSSETotaln-1σMean squaresF-statisticp-valueMSR SSR MSE SSE/(n MSE SSE/(n-2)F MSR/MSEP( F1,n1,n-2 F )2F I G U R E 12.41Analysis of variance table for simplelinear regression analysisNIPRL34
12.6.1 Sum of Squares Decomposition(4/5)NIPRL35
12.6.1 Sum of Squares Decomposition(5/5)Coefficient of Determination R2 The total variability in the dependent variable, the total sum of squaresSST in 1 ( yi y ) 2can be partitioned into the variability explained by the regression line,the regression sum of squaresSSR in 1 ( yi y ) 2and the variability about the regression line, the error sum of squaresSSE in 1 ( yi yi ) 2 . The proportion of the total variability accounted for by the regression line isthe coefficient of determinationSSRSSE1R2 1 SSTSST 1 SSESSRwhich takes a value between zero and one.NIPRL36
12.6.2 Examples(1/1) Example 3 : Car Plant Electricity UsageMSR 1.2124F 40.53MSE 0.0299SSR 1.2124R 0.802SST 1.51152NIPRL37
12.7 Residual Analysis12.7.1 Residual Analysis Methods(1/7) The residuals are defined to beei yi yi , 1 i nso that they are the differences between the observed values of thedependent variable andyithe corresponding fitted values .yi A property of the residuals in 1 ei 0 Residual analysis can be used toNIPRL––––Identify data points that are outliers,outliersCheck whether the fitted model is appropriate,appropriateCheck whether the error variance is constant,constant andCheck whether the error terms are normally distributed.38
12.7.1 Residual Analysis Methods(2/7) A nice random scatter plot such as the one in Figure 12.45 there are no indications of any problems with the regressionanalysis Any patterns in the residual plot or any residuals with a largeabsolute value alert the experimenter to possible problems with thefitted regression model.NIPRL39
12.7.1 Residual Analysis Methods(3/7) A data point (xi, yi ) can be considered to be an outlier if it does not appearto predict well by the fitted model.Residuals of outliers have a large absolute value, as indicated in Figureeiis used instead of12.46. Note in the figure thatei .sˆ[For your interest only]2Var (ei ) (1-NIPRL1 ( xi - x ) 2)s .nS XX40
12.7.1 Residual Analysis Methods(4/7) If the residual plot shows positiveand negative residuals groupedtogether as in Figure 12.47, thena linear model is not appropriate.As Figure 12.47 indicates, anonlinear model is needed forsuch a data set.NIPRL41
12.7.1 Residual Analysis Methods(5/7) If the residual plot shows a “funnelshape” as in Figure 12.48, so thatthe size of the residuals dependsupon the value of the explanatoryvariable x, then the assumption ofa constant error variance σ2 is notvalid.NIPRL42
12.7.1 Residual Analysis Methods(6/7) A normal probability plot ( a normal score plot) of the residuals The normal score of the i th smallest residual3 i – 18Φ 1 n 4 – Check whether the error terms εi appear to be normally distributed.The main body of the points in a normal probability plot lieapproximately on a straight line as in Figure 12.49 is reasonableThe form such as in Figure 12.50 indicates that the distribution is notnormalNIPRL43
12.7.1 Residual Analysis Methods(7/7)NIPRL44
12.7.2 Examples(1/2) Example : Nile River FlowrateNIPRL45
12.7.2 Examples(2/2)x 3.88 y 0.470 (0.836 3.88) 2.775 ei yi yi 4.01 2.77 1.24ei1.24 3.750.1092σx 6.13ei yi yi 5.67 ( 0.470 (0.836 6.13)) 1.021.02 3.070.1092σeiNIPRL46
12.8 Variable Transformations12.8.1 Intrinsically Linear Models(1/4)NIPRL47
12.8.1 Intrinsically Linear Models(2/4)NIPRL48
12.8.1 Intrinsically Linear Models(3/4)NIPRL49
12.8.1 Intrinsically Linear Models(4/4)NIPRL50
12.8.2 Examples(1/5) Example : Roadway Base AggregatesNIPRL51
12.8.2 Examples(2/5)NIPRL52
12.8.2 Examples(3/5)NIPRL53
12.8.2 Examples(4/5)NIPRL54
12.8.2 Examples(5/5)NIPRL55
12.9 Correlation Analysis12.9.1 The Sample Correlation CoefficientSample Correlation CoefficientThe sample correlation coefficient r for a set of paired data observations( xi , yi ) isr S XYS XX SYY in 1 ( xi x )( yi y ) in 1 ( xi x ) 2 in 1 ( yi y ) 2 in 1 xi yi nxy in 1 xi2 nx 2 in 1 yi2 ny 2It measures the strength of linear association between two variables and canbe thought of as an estimate of the correlation ρ between the two associatedrandom variable X and Y .NIPRL56
Under the assumption that the X and Y random variables have a bivariatenormal distribution, a test of the null hypothesisH0 : ρ 0can be performed by comparing the t -statistict r n 21 r2with a t -distribution with n 2 degrees of freedom. In a regression framework,this test is equivalent to testing H 0 : β1 0.NIPRL57
NIPRL58
NIPRL59
12.9.2 Examples(1/1) Example : Nile River Flowrater R 2 0.871 0.933NIPRL60
Chapter 12. Simple Linear Regression and Correlation 12.1 The Simple Linear Regression Model 12.2 Fitting the Regression Line 12.3 Inferences on the Slope Rarameter ββββ1111 NIPRL 1 12.4 Inferences on the Regression Line 12.5 Prediction Intervals for Future Response Values 1
independent variables. Many other procedures can also fit regression models, but they focus on more specialized forms of regression, such as robust regression, generalized linear regression, nonlinear regression, nonparametric regression, quantile regression, regression modeling of survey data, regression modeling of
Chapter 7 Simple linear regression and correlation Department of Statistics and Operations Research November 24, 2019. Plan 1 Correlation 2 Simple linear regression. Plan 1 Correlation 2 Simple linear regression. De nition The measure of linear association ˆbetween two variables X and Y is estimated by the s
Multiple Linear Regression Linear relationship developed from more than 1 predictor variable Simple linear regression: y b m*x y β 0 β 1 * x 1 Multiple linear regression: y β 0 β 1 *x 1 β 2 *x 2 β n *x n β i is a parameter estimate used to generate the linear curve Simple linear model: β 1 is the slope of the line
Probability & Bayesian Inference CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition J. Elder 3 Linear Regression Topics What is linear regression? Example: polynomial curve fitting Other basis families Solving linear regression problems Regularized regression Multiple linear regression
Its simplicity and flexibility makes linear regression one of the most important and widely used statistical prediction methods. There are papers, books, and sequences of courses devoted to linear regression. 1.1Fitting a regression We fit a linear regression to covariate/response data. Each data point is a pair .x;y/, where
Part One: Heir of Ash Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8 Chapter 9 Chapter 10 Chapter 11 Chapter 12 Chapter 13 Chapter 14 Chapter 15 Chapter 16 Chapter 17 Chapter 18 Chapter 19 Chapter 20 Chapter 21 Chapter 22 Chapter 23 Chapter 24 Chapter 25 Chapter 26 Chapter 27 Chapter 28 Chapter 29 Chapter 30 .
LINEAR REGRESSION 12-2.1 Test for Significance of Regression 12-2.2 Tests on Individual Regression Coefficients and Subsets of Coefficients 12-3 CONFIDENCE INTERVALS IN MULTIPLE LINEAR REGRESSION 12-3.1 Confidence Intervals on Individual Regression Coefficients 12-3.2 Confidence Interval
pihak di bawah koordinasi Kementerian Pendidikan dan Kebudayaan, dan dipergunakan dalam tahap awal penerapan Kurikulum 2013. Buku ini merupakan “dokumen hidup” yang senantiasa diperbaiki, diperbaharui, dan dimutakhirkan sesuai dengan dinamika kebutuhan dan perubahan zaman. Masukan dari berbagai kalangan diharapkan dapat meningkatkan kualitas buku ini. Kontributor Naskah : Suyono . Penelaah .
