Section 4.2 Fitting Curves And Surfaces By LeastSurfaces By Least Squares

Section 4.2 Fitting Curves andSurfaces by Least Squares

4 2 1 Curve Fitting4.2.1In many cases the relationship of y to x is not ag linestraight

To fit a curve to the data one can Fit a nonlinear function directly to the data. . Rescale, transform x or y to make therelationshipl i hi lilinear. Fit a ppolynomialyfunction to the data.

For a uniform fluid– lightg should decayy as an exponentialpfunction ofdepth– Given data on depthdepth, x,x and light intensityintensity, y,yTheh datad shouldh ld llookk liklike an exponentiali l ffunction.iy aeb1 x

1. Direct FittingWe could fit the equation directly byminimizingi i ii (y ae )If the residuals are normal with equalq variancethenib1 x i 2– this is an optimal way to gogo.We could use a "nonlinear regression"programpgto accomplishpthis.

This can also be accomplish with the "Solver" add‐in inExcel.We won't be doing such curves this way in Stat 3411.But veryy often in situations such as exponentiallypycurved data the variances are not constant. One way to handle nonconstant variance is"weighted regression".– We give larger weight to residuals where variance is smalland weare more sure of where the fitted line should go. We won't be doing this in Stat 3411.

2. TransformingOften with an exponential relationship– ththe llarger valueslhhave more varianceiandd– the smaller values have smaller variances.In this case it can work better to rescale the datay ael ( y ) ln(ln(l (a ) b1 x1b1 xynew y ′ b0 b1 x1Often values in the log scale have fairly constant variances.

Original Scale302520Y 15105002468101214161820XA residual plot gives us a magnified view of the increasing variance and curvature.Original 8This residual plot indicates 2 problemswith this linear least squares fit.The relationship is not linear‐Indicateddd byb theh curvature in thehresidual plotThe variance is not constant‐SoS leastl t squares iisn't't ththe bbesttapproach even if we handle thenonlinearity.

Let us try to transform the data.(A log transformation)

Ln Scale3.532.52Ln(Y)151.510.5005101520XFit a line to these data. Here it makes sense for all residuals to count equally.qy

Then back‐transform to the original scaleOriginal Scale302520Y 1510500246810X1214161820

Extrapolation Predicted values are less reliable if you arepg outside the rangeg of x values inextrapolatingthe data set.

Extrapolations are more reliable if one has anqfrom a pphysicalymodel.equation For example physical models predict that in auniform fluid light will decay exponentiallywith depth.

CE4505SurfaceWaterQuality Engineering http://www.cee.mtu.edu/ mtauer/classes/ce4505/lecture3.htm

3. PolynomialsWhen we have no theory to guide us, we canoften fit the curve in the range of observed xvalues with a polynomial function.For example a cubic polynomial would bey b0 b1 x b2 x 2 b3 x 2 ThisThi iis lilinear ffunctionti ffor ththe ththree variablesi blx1 xx1 x 2x3 x 3y b0 b1 x1 b2 x 2 b3 x3 Excel and other programs fit these sorts ofpolynomial models.

In Excel Add new columns for x2 and x3 Run a regression using 3 columns for the xvariablesi bl– x , x2 , and x3– All three columns need to be next to each other inExcel.

Given theGih fifittedd ffunction,iwe want to checkh k fforan adequate fit by plotting the data along withthe fitted function.Notice that replicationis useful for assessinghow well thefunctional form fitsthe data.With replication herewe can tell that thequadratic polynomialis under‐fitting the yvalues at x 2.

Residual PlotsWe wouldld plotl residualsid l iin theh same wayssuggested in section 4.1––––Versus predicted valuesVersus xIn run orderVersus other potentially influential variables, e.g.technician– Normal Plot of residuals

Section 4.2.2 Surface Fitting by Least SquaresIn many situations the response variable, y, isaffected byy more than one x variable.For example we could have (see problem 21 inthe Exercises)y armor strengthgxl thicknessx2 Brinell hardnessThe simplest model to fit in this case is a linearmodel in both variablesy b0 b1 x1 b2 x 2

In this case we are fitting a plane to the 3‐Dpoints.This sort of linear model with more than one xvariable is called "multiple linear regression"

For this function fitted y values for a fixedvalue of x1 follow parallel lines when plottedagainsti t x2.

y i b0 b1 x1i b2 x 2iAgain, we find b0 , b1 , b2 to minimize the sum of squaredqdeviations of the fittedvalues, the residual or error sum of squaressquares. min (yi b0 b1 x1i b2 x2i )2This minimization can be solved with explicit matrixformulas.Many programs including Excel have the capability.

There can be any number of x variables, forpexampley b0 b1 x1 b2 x 2 b3 x 3

It is also possible to introduce curvature intoone or more off ththe variablesi blFor examplepy b0 b1 x1 b2 x 2 b3 x 3 b4 x 22 b5 x 32 b6 x 2 x 3

Product terms (called interactions) in themodel gigivee nonparallel contocontours.rs

ResidualsResiduals are as always ei yi yˆi .R2 is still the percent of variation explained by theregression. How better do we do– using the regression equation versus– using the mean to predict all values22ˆ()()y y y y i i i 100R2 2 ( yi y )R2 is also the square of the correlation between yi and yˆi .

To check how well the model fits we plot residuals in the same ways as earlier––––Versus predicted valuesVersus xIn run orderVersus other potentially influential variables, e.g.technician– Normal Plot of residuals In addition we would plot the residuals– Versus each x variable Versus x1Versus x2Versus x3An so on

Final Choice of ModelWe have many choices for our final model including Transformations of manyy kinds PolynomialsOur final choice is guided by 3 considerations Sensible engineering models and experience How well the model fits the data.data The simplicity of the model.

1. Sensible engineering models– Whenever possible use models that make intuitivephysical sense– Good mathematical models based on physicsmakes extrapolation much less dangerous.– HavingH i physicallyh i ll iinterpretablett bl estimatedti t dparameters

2. Obviously, the model needs to fit the datareasonablyy well.– AlwaysAlplotl tddatat andd residualsid l iin iinformativeftiways.

3. Theh simplicityi li i off theh model.d lA very complex model relative to the amountof data– Can fit the data well– But not predict new points well,well particularlyextrapolated points.– "Overfitting" the modelLearn more in Stat 5511 Regression Analysis

Overfitting Polynomial120100Y8060402000123456XThe fourth order ppolynomialyin ppink fits the data exactly, but likely would not work well for predicting for x 0.5 or x 1.5The linear fit in blue likely predicts new points better.Oftenftransformingfto a llog scalel allowsllsimplerl modelsd l to ffit thehdata reasonably well.

ExtrapolatingWith multiple x variables we need to bepg beyondythe regiongcareful about extrapolatingof x variables for the observed data.We cancan'tt necessarily tell that a combination ofX1 and X2 is unusual just by seeing– separately– where the new value of Xl falls amongstg themeasured Xl values– where the new value of X2 falls amongst themeasured X2 values

Some Study Questions: You run a regression model fitting y with x1, x2, and x3. What fiveresidual plots would you need to check in order see if the data arefit well by the model? WhatWh t isi extrapolation?tl ti ? WhWhy iis extrapolationtl ti potentiallyt ti ll risky?ik ? What can we do to reduce ppotential risks in extrapolating?pg We run a regression predicting y armor strength from xl thickness and x2 Brinell hardness and obtain R2 80%80%. Explain inwords for a manager who has not seen R2 values what it means forR2 to be 80%. What is the problem with fitting a complex model to a small dataset in order to fit the data points very exactly?

The following residual plot is for predicting product lifetimes from using x1 and redictedWhat 3 problems with our fitting method and results are indicated by thisresidual plot?What might you do to try finding a better fit to the data?

421CurveFitting4.2.1 Curve Fitting In many cases the relationship of y to x is not a straight line. . with this linear least squares fit. The relationship is not linear ddbh h-2 0 2 4 0 2 4 6 8 10 12 14 16 18 . Section 4.2.2 Surface Fitting by Least Squares In many situations the response variable, y, is