ERROR ANALYSIS 2: LEAST-SQUARES FITTING

ERROR ANALYSIS 2: LEAST-SQUARES FITTINGINTRODUCTIONThis activity is a “user’s guide” to least-squares fitting and to determining the goodness of your fits. It doesn’tderive many results. There are good textbooks listed in the references.LEARNING GOALSAt the end of the activity you will be able to:1.2.3.4.5.Explain why we minimize the sum of squares to get the best fitCarry out a least-squares minimization graphicallyPlot residuals to visually inspect the goodness of a fitBe able to interpret the uncertainty in fit parameters that Mathematica’s fit routines outputBe able to compute 𝜒 " for a fit and use it to determine if a fit is “good”WHY DO WE MINIMIZE THE SUM OF SQUARES?Question: Why do we call it “least-squares” fitting?Answer: Because the best fit is determined by minimizing the weighted sum of squares of the deviation betweenthe data and the fit. Properly speaking this “sum of squares” is called “chi-squared” and is given by4𝜒" '561"(𝑦 𝑦(𝑥' , 𝑎, 𝑏, 𝑐, )3𝜎'" '(1)where there are 𝑁 data points (𝑥' , 𝑦' ), and the fit function is given by 𝑦(𝑥' , 𝑎, 𝑏, 𝑐, ) where 𝑎, 𝑏, etc. are the fitparameters.Question: What assumptions are made for the method to be valid?Answer: The two assumptions are(1) Gaussian distributed. The random fluctuations in each data point 𝑦' are Gaussian distributed withstandard deviation 𝜎' .(2) Uncorrelated. The random fluctuations in any one data point are uncorrelated with those in another datapoint.Question: Why does minimizing the sum of squares give us the best fit?Answer: Given those two assumptions, the fit that minimizes the sum of squares is the most likely function toproduce the observed data. This can be proven using a little calculus and probability. A more detailed explanationis found in Taylor’s Introduction to Error Analysis Sec. 5.5 “Justification of the Mean as Best Estimate” or Bevingtonand Robinson’s Data Reduction Sec. 4.1 “Method of Least-Squares".

MINIMIZING 𝜒 " GRAPHICALLY:Question 1Graphically minimizing 𝝌𝟐You will rarely minimize 𝜒 " graphically in a lab. However, this exercise will help you betterunderstand what the fitting routines in Mathematica actually do to find the best fit.a. Import and plot the data set from profile data without errors activity 2.csv. It wasgenerated by inserting a razor blade into path of a laser beam and measuring thephotodetector voltage of the laser light. The x column is the microcrometer (razor)position in meters and the y column is the photodetector voltage in Volts. Import["profile data without errors activity 2.csv",”CSV”]b. Define the same fit function many of you used in the Gaussian laser beams lab:𝑦(𝑥, 𝑎, 𝑏, 𝑐, 𝑤) 𝑎 Erf ? "A(𝑥 𝑏)B 𝑐c. Reduce the fit to two free parameters. This step is only necessary because it is hard tovisualize more than 3 dimensions. Assume 𝑎D'E (𝑉GHI 𝑉G'J )/2 1.4375 and 𝑐D'E (𝑉GHI 𝑉G'J )/2 1.45195. These were determined by averaging the first 6 data pointsto get 𝑉G'J and the last 5 to get 𝑉GHI .d. Use Equation 1 to write an expression for 𝝌𝟐 in terms of your 𝑤 and 𝑐 parameters, andthe 𝑥 (position) data, 𝑦 (voltage) data. Since you don’t have any estimate for theuncertainties 𝜎' , do what Mathematica does, and assume they are all unity so 𝜎' 1.e. Make a contour plot of 𝜒 " (𝑤, 𝑏) and tweak the plot range until you see the minimum.Just like with NonlinearModelFit, it will help to have a good initial guess for your fitparameters. You can iteratively improve the plot range your plot to zoom in on theparameter values that minimize 𝜒 " , you should get a plot kind of like: f. Graphically determine the best fit parameters to 3 significant digits.g. Compare with the best fit result from NonlinearModelFit (allow all 4 parameters tovary). Do the fits agree for those three digits of precision?UNCERTAINTY IN THE FIT PARAMETERS

Question: Where does the uncertainty in the fit parameters come from?.Answer: The optimal fit parameters depend on the data points (𝑥' , 𝑦' ). The uncertainty 𝜎' in the 𝑦' means there isa propagated uncertainty in the calculation of the fit parameters. The error propagation calculation is explained indetail in the references, especially Bevington and Robinson.Question: How does Mathematica calculate the uncertainty in the fit parameters when no error estimate for the𝜎' is provided?Answer: Mathematica (and other programs) estimate the uncertainty in the data 𝜎S" using the “residuals” of thebest fit:4𝜎S"1" (𝑦' 𝑦(𝑥' , 𝑎U , 𝑏U , 𝑐U , )3𝑁 𝑛'56(2)where there are 𝑁 data points 𝑦' and the best fit value at each point is given by 𝑦, which depends on 𝑥' and the 𝑛best fit parameters 𝑎U , 𝑏U , 𝑐U , . It is very similar to how you would estimate the standard deviation of arepeated measurement, which for comparison’s sake is given by4𝜎S"1 (𝑦' 𝑦V)"𝑁 1'56Question 2(3)Estimating the uncertainty in the dataa. Use Equation 2 and your best fit parameters to estimate 𝜎S" , the random error of eachdata point given by your data.b. Compare your result with Mathematica’s NonlinearModelFit, which can also outputthis estimate of the random error. If nlm is the NonlinearModelFit output, theestimate of 𝜎S" is given by nlm[“EstimatedVariance”]c. Do the estimates agree? Why or why not?

GOODNESS OF FITThis section covers two ways to analyze if a fit is good.1.2.Plotting the residualsDoing a 𝜒 " testPLOTTING THE FIT RESIDUALSThe first step is to look at the residuals. The residuals 𝑟' are defined as the difference between the data and the fit.𝑟' 𝑦' 𝑦(𝑥' , 𝑎, 𝑏, 𝑐, )Question 3Making a plot of the residuals and interpreting ita. Make a ListPlot of the residuals. If nlm is the NonlinearModelFit output, thelist of residuals is given by nlm[“FitResiduals”]b. Since we didn’t provide any estimates of the uncertainties, Mathematica assumed theuncertainty of every point is the same. Based on the plot of residuals, was this a goodassumption?c. Do the residuals look randomly scattered about zero or do you notice any systematic errorsources?d. Is the distribution of residuals scattered evenly around 0? Or is there a particular range ofx values where the residuals are larger than others?e. What is the most likely source of the large uncertainty as the beam is cut near the centerof the beam?“CHI BY EYE” – EYEBALLING THE GOODNESS OF FITQuestion: If I have a good fit, should every data point lie within an error bar?Answer: No. Most should, but we wouldn’t expect every data point to lie within an error bar. If the uncertainty isGaussian distributed with a standard deviation 𝜎' for each data point 𝑦' , then we expect roughly 68% of the datapoints to lie within their error bar. This is because 68% of the probability in a Gaussian distribution lies within onestandard deviation of the mean."𝜒 " AND 𝜒XYZFOR TESTING THE “GOODNESS OF A FIT”This section answers the question “What should 𝜒 " be for a good fit?”Suppose the only uncertainty in the data is statistical (i.e., random) error, with a known standard deviation 𝜎' , thenon average each term in the sum is1"(𝑦 𝑦(𝑥' , 𝑎, 𝑏, 𝑐, )3 1𝜎'" 'and the full 𝜒 " sum of squares is approximately(4)

4"𝜒 '561")3 𝑁 𝑛" (𝑦' 𝑦(𝑥' , 𝑎, 𝑏, 𝑐, 𝜎'(5)So a good fit has"𝜒XYZ Question 4𝜒" 1𝑁 𝑛Fact: To find the goodness of fit test, you must first estimate the uncertainties on the datapoints that you are fitting.How would you explain the reason for this in your own words?Question 5Choosing a strategy to estimate the uncertaintyConsidering your answers to question 3, especially 3e, which method would give you the bestestimate of the uncertainty for each data point? Eyeballing the fluctuations in each data point. Taking 𝑁 measurements at each razor position and then going to the next position. Taking the entire data set 𝑁 timesWEIGHTED BEST FITS IN MATHEMATICAWhen you have estimated the uncertainty 𝜎' of each data point 𝑦' you would like to use this information whenfitting to correctly evaluate the 𝜒 " expression in Equation 1. The points with high uncertainty contribute lessinformation when choosing the best fit parameters. If you have a list of uncertaintiesσlist { σ1, σ2, σ3, }then the weights for the fit areweightslist 1/ σlist2 {1/ σ12, 1/ σ22, }Add the Weights- weightslist option to the LinearModelFit or NonlinearModelFit. b,c},x, Weights- weightslist](6)

Question 6a. Import the data set for a beam width measurement with uncertainties from theprofile data with errors activity 2.csv. The first column is razor position in meters, thesecond column is photodetector output voltage, and the third column is the uncertaintyon the photodetector output voltage. Import["profile data with errors activity 2.csv",,”CSV”]b. Do a weighted fit using the same fit function as in question 1. Use the uncertaintyestimates in the third column to calculate the weights.c. Calculate 𝜒 " . You can obtain 𝜒 " from the fit returned by Mathematica. Supposing the fitwas called nlmError, use nlmError[“ANOVATable”]. For the curious, ANOVAstands for ANalysis Of VAriance.d. How close is the reduced chi-squared to 1?e. The “chi-squared test”. This part helps us understand if the value of 𝜒 " is statisticallylikely or not. The following graph gives the probability of exceeding a particular value of𝜒 " for 𝜈 𝑁 𝑛 22 degrees of freedom. It can be calculated using the CumulativeDensity Function (CDF) for the chi-squared distribution. Use the graph to estimate thelikelihood this value of 𝜒 " occurred by chance.ν 22; (*Degrees of freedom*)Plot[1 - CDF[ChiSquareDistribution[ν], x* ν], {x, 0, 2}]

WHY IS IT OFTEN BAD TO OVERESTIMATE UNCERTAINTIES?Question 7Why can overestimating the uncertainty make your fit appear good (i.e., 4 J 1)?Overestimating the uncertainties makes the fit seem good (according to a 𝜒 " test), even when it might beobviously a bad fit. It is best to do the 𝜒 " test using an honest estimate of your uncertainties. If the 𝜒 " is largerthan expected (𝜒 " 𝑁 𝑛), then you should consider both the possibility of systematic error sources and thequality of your estimates of the uncertainties. On the other hand, if the 𝜒 " test is good (𝜒 " 𝑁 𝑛), then itshows you have a good handle on the model of your system, and your sources of uncertainty. Finally, if 𝜒 " is muchless than 𝑁 𝑛, this likely indicates overestimated uncertainties.REFERENCES1.2.Taylor, J. R. (1997). An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements(p. 327). University Science Books. This is the standard undergraduate text for measurement anduncertainty.Bevington, P. R., & Robinson, K. D. (2003). Data Reduction and Error Analysis for the Physical SciencesThird Edition (3rd ed.). New York: McGraw-Hill. Great for advanced undergrad error analysis. Professionalphysicists use it too.

ERROR ANALYSIS 2: LEAST-SQUARES FITTING INTRODUCTION This activity is a “user’s guide” to least-squares fitting and to determining the goodness of your fits.