System Regularities In Design Of Experiments And Their .

System Regularities in Design of Experiments andTheir ApplicationbyXiang LiB.S. and M.S., Engineering MechanicsTsinghua University, 2000, 2002Submitted to the Department of Mechanical Engineeringin Partial Fulfillment of the Requirements for the Degree ofDoctor of Philosophy in Mechanical Engineeringat theMassachusetts Institute of TechnologyJune, 2006 2006 Massachusetts Institute of Technology. All rights reserved.Signature of Author .Department of Mechanical EngineeringMay 9, 2006Certified by . Daniel D. FreyAssistant Professor of Mechanical Engineering and Engineering SystemsThesis SupervisorAccepted by . .Lallit AnandChairman, Department Committee on Graduate Students

System Regularities in Design of Experiments andTheir ApplicationbyXiang LiSubmitted to the Department of Mechanical Engineering on May 9, 2006,in partial fulfillment of the requirements for the Degree ofDoctor of Philosophy in Mechanical EngineeringABSTRACTThis dissertation documents a meta-analysis of 113 data sets from published factorialexperiments. The study quantifies regularities observed among main effects and multi-factorinteractions. Such regularities are critical to efficient planning and analysis of experiments, andto robust design of engineering systems. Three previously observed properties are analyzed –effect sparsity, hierarchy, and heredity. A new regularity on effect synergism is introduced andshown to be statistically significant. It is shown that a preponderance of active two-factorinteraction effects are synergistic, meaning that when main effects are used to increase thesystem response, the interactions provide an additional increase and that when main effects areused to decrease the response, the interactions generally counteract the main effects.Based on the investigation of system regularities, a new strategy is proposed for evaluating andcomparing the effectiveness of robust parameter design methods. A hierarchical probabilitymodel is used to capture assumptions about robust design scenarios. A process is presentedemploying this model to evaluate robust design methods. This process is then used to explorethree topics of debate in robust design: 1) the relative effectiveness of crossed versus combinedarrays; 2) the comparative advantages of signal-to-noise ratios versus response modeling foranalysis of crossed arrays; and 3) the use of adaptive versus “one shot” methods for robustdesign. For the particular scenarios studied, it is shown that crossed arrays are preferred tocombined arrays regardless of the criterion used in selection of the combined array. It is shownthat when analyzing the data from crossed arrays, signal-to-noise ratios generally providesuperior performance; although that response modeling should be used when three-factorinteractions are absent. Most significantly, it is shown that using an adaptive inner array designcrossed with an orthogonal outer array resulted in far more improvement on average than otheralternatives.Thesis Supervisor: Daniel D. FreyTitle: Assistant Professor of Mechanical Engineering and Engineering Systems2

AcknowledgmentsI would like to thank a number of people who contributed in many ways in making this thesispossible. First, I am grateful to my advisor, Professor Daniel D. Frey, for giving me theopportunity to work on such a challenging topic.With deep research insights and richengineering knowledge, he has patiently guided me to explore the complicated topic ofeffectively applying experimental designs to engineering systems and to conquer a variety ofdifficulties along the way. In addition, he always gave me the freedom to pursue my interestsand always offered me great opportunities to widen my perspective. I also wish to thank mydoctoral committee members, Professor Warren P. Seering and Professor Roy E. Welsch, fortheir invaluable advice and inspiring discussions on this research.Thanks to the staff members of MIT who have been in great assistance to this work. I wouldespecially like to thank Ms. Leslie Regan, Ms. Joan Kravit, Ms. Maggie Sullivan, Mr. Jason Pring,and Ms. Danielle Guichard-Ashbrook. I also owe a debt of gratitude to Mr. Steven Schondorf forbeing my mentor and giving me continuous support.I would like to sincerely acknowledge the encouragement from my colleagues in the RobustDesign Group: Rajesh Jugulum, Yiben Lin, Hungjen Wang, Chad Forster, Nandan Sudarsanam,Danielle Zurovcik, Jagmeet Singh, and Troy Savoie, who have been wonderful friends andresearch partners.Finally, I would like to express my deepest thanks to my parents and my sister for their loveand encouragement. This work could not have been done without their support. In addition, myclose friends have always kept me sane and motivated: Elaine, Chen, Shuang, Dave, Bolero,Huajie, and Hao – thanks for everything.3

Contents1. Introduction.111.1 Overview. 111.2 Motivation. 131.2.1 Debate on Adaptive Experimentation. 141.2.2 Debate on Robust Design Methods. 151.3 Research Objectives. 161.4 Research Roadmap. 171.5 Organization of the Dissertation . 192. Regularities in Data from Experiments .212.1 Overview. 212.2 Design of Experiments. 222.2.1 A Historical Review. 232.2.2 A Technical Review. 242.2.3 Adaptive One-Factor-at-A-Time (OFAT) Experiments . 272.3 Introduction to System Regularities. 292.3.1 Effect Sparsity. 302.3.2 Effect Hierarchy. 332.3.3 Effect Heredity. 352.4 DOE Models Incorporating System Regularities . 362.4.1 The General Linear Model. 362.4.2 The Relaxed Weak Heredity Model . 372.4.3 The Hierarchical Probability Model . 392.5 Effects of System Regularities on DOE Methods. 422.6 Summary . 493. Verification and Quantification of System Regularities.503.1 Overview. 503.2 Objectives and Methods. 513.2.1 The General Linear Model Revisited. 523.2.2 The Lenth Method for Effect Analysis . 523.2.3 Method for Quantifying Effect Sparsity . 563.2.4 Method for Quantifying Effect Hierarchy . 573.2.5 Method for Quantifying Effect heredity . 584

3.3 The Set of Experimental Data. 593.4 An Illustrative Example for a Single Data Set. 623.5 Results of Meta-Analysis of 133 Data Sets . 683.6 Quantification of the Standard Deviation c . 713.7 Conclusions and Discussion . 733.8 Summary . 764. Model-Based Validation and Comparison of Robust Parameter Design Methods.774.1 Overview. 774.2 The Concept of Validation. 784.3 Robust Design Method and Methodology Evaluation. 804.4 Objectives and Methods. 884.4.1 The Model and the System Regularities . 894.4.2 Instantiate Multiple Response Surfaces . 944.4.3 Simulate the RPD Methods. 954.4.4 Evaluate the Primany Variable . 954.4.5 Analyze and Present the Data . 964.5 Case Study I – Adaptive OFAT in Robust Design . 974.5.1 Selected Robust Design Methods . 974.5.2 Discussion of the Case Study I Results. 1054.5.3 Suggestions for Future Research Following Case study I . 1084.6 Case Study II – Compounding Noise Strategy in Robust Design . 1104.6.1 Select Parameters of the Hierarchical Probability Model. 1104.6.2 Robust Design Methods to Be Evaluated . 1134.6.3 Results of Case Study II. 1154.6.4 Discussion of Case Study II. 1174.7 Concluding Remarks of the Case Studies. 1184.8 A Broader Discussion in Relation to Complex Engineering Systems. 1204.9 Summary . 1235. Asymmetric Synergistic Interaction Structure. 1265.1 Overview. 1265.2 Define ASIS . 1275.3 Quantifying ASIS. 1285.3.1 The Set of Experimental Data. 1285.3.2 Method for Quantifying Asymmetric Synergistic Interaction Structure . 1295.4 Wet-Clutch Example Revisited. 1305.5 Results of Meta-Analysis on ASIS . 1335.6 Additional Investigation of the Log Transformation . 1345.7 Conclusions and Discussion . 1365.8 Summary . 1385

6. Conclusions. 1396.1 Major Contributions. 1396.2 Summary of My Work . 1416.3 Future Work . 143Appendix I. References of the Experimental Data Sets . 144Appendix II. List of the Responses Subjected to Meta-Analysis . 150Bibliography. 1516

List of FiguresFigure 1 Objectives of my research work .16Figure 2 Research roadmap .18Figure 3 One-Factor-at-A-Time experiments .28Figure 4 A fractional factorial 2 3 1 design and its projections into 2 2 designs. .32Figure 5 Effect hierarchy and heredity among main effects and interactions in a system withfour factors A, B, C, and D. The font size represents the size of the effects. .33Figure 6 DOE methods comparison with parameter setting 1.47Figure 7 DOE methods comparison with parameter setting 2.47Figure 8 Bar graph of the Lenth method applied to the Laser-printed paper experiment .54Figure 9 Normal probability plot of the Laser-printed paper experiment .56Figure 10 Statistics on fields of the engineering experiment database.59Figure 11 A wet clutch pack .63Figure 12 Effect analysis using the Lenth method for the wet-clutch experiment .65Figure 13 Box plot of absolute values for main effects, two-factor interactions, and threefactor interactions.70Figure 14 A general model for robust design .81Figure 15 Adjusting design variables to reduce response variance .82Figure 16 A cross array for robust design.84Figure 17 The adaptive one factor at a time method crossed with a resolution III outer array.This method is denoted as aOFAT 2 3III 1 in this case study. Noise factors a, b,and c are varied according to a factorial design and control factors D, E, and F areexplored sequentially to seek lower variance in the observed response.1017

Figure 18 The distribution of factor effects from real systems.112Figure 19 The distribution of factor effects from 1000 simulated systems sampled from thefitted weak heredity model with c 15 .112Figure 20 Summary of my work.1428

List of TablesTable 1 A full factorial 23 design.26Table 2 A fractional factorial design 27-4 .27Table 3 A One-Factor-at-A-Time design.28Table 4 A Revised-One-Factor-at-A-Time design .44Table 5 The relation between model parameters and system regularities .45Table 6 Parameters used in the DOE method comparison example.46Table 7 Factors in the laser-printed paper experiment.55Table 8 Summary of the set of 113 responses and the potential effects therein.61Table 9 The main effects from the wet clutch case study.66Table 10 The active two-factor interactions from the wet clutch case study.66Table 11 The Main Effects from the Clutch Case Study Using a Log Transform. .67Table 12 The active 2fi’s from the wet clutch case study with a log transformation. .68Table 13 Percentage of potential effects in 113 experiments those were active asdetermined by the Lenth method. .69Table 14 The conditional probabilities of observing active effects based on meta-analysisof 113 experiments.71Table 15 Model variants and associated sets of parameters .93Table 16 Cost indicators for the various methods considered in this case study.103Table 17 Mean percent reduction in transmitted variance achieved by various methodsapplied to various models. .103Table 18 Inter-quartile range of percent reduction in transmitted variance achieved byvarious methods applied to various models.104Table 19 Sets of model parameters considered in the case study .1109

Table 20 Additional model parameters for each set considered in the case study .111Table 21 Expected values of percent reduction in standard deviation for various robustdesign methods and system parameter sets.

Based on the investigation of system regularities, a new strategy is proposed for evaluating and comparing the effectiveness of robust parameter design methods. A hierarchical probability model is used to capture assumptions about robust design scenarios. A process is presented employing this model to evaluate robust design methods. This process is then used to explore three topics of debate .