Experimentation In Software Engineering
Experimentation in SoftwareEngineeringOporto May, 2003 E. Manso U. de Valladolid1.2.1IntroductionType of Studies2.1 Experimental Studies3.Experimentation Process3.1 Definition3.2 Planing3.3 Operation3.4 Analysis and Interpretation3.5 Conclusions4.Conclusions E. Manso U. de Valladolid2n1
1. Introductionn“Software Engineering means application ofsystematic, disciplined, quantifiable approach todevelopment, operation and maintenance ofsoftware” [IEEE90]Software ProcessProcessSoftwareSystematic andand disciplineddisciplined approachapproachnn SystematicQuantificationnn Quantificationnn E. Manso U. de Valladolid31. IntroductionResourcesProduct IdeaSoftware ProcessComplexitySoftwareProcessTo improve thesoftware ProcessSoftware ProductHumaninvolvement“Experimentation providesprovides aa systematic,systematic, ableandcontrolledwayofevaluatinghumanquantifiable and controlled way of evaluating humanbasedactivities”Wholin2000based activities” Wholin 2000 E. Manso U. de Valladolid4n2
IntroductionExperimentation in SoftwareEngineeringnnZelkowitz (1997)(1997) conclusionsconclusions overover 612612 sthatincludeexperimentationOnly the 10% of papers that include shave controlled experimentation methodsTichy (1995)(1995) conclusionsconclusions overover 400400 andandThetheyrequiredempiricalvalidationthey required empirical validation E. Manso U. de Valladolid5IntroductionExperimentation in SoftwareEngineeringnn¿Why inin softwaresoftware engineeringengineering aa lotlot ofof assertsasserts aren’taren’t¿Whyvalidated?validated?nnnnnew sciencescienceItIt isis aa newThey needneed toto obtainobtain goodgood quantitativequantitative datadata toto makemakeTheyvalidations, butbut itit oftenoften isis hardhardvalidations,The wayway thatthat cancan convertconvert softwaresoftware engineeringengineering claimsclaimsTheinto validatedvalidated factsfacts itit isis thethe experimentalexperimental methodmethodinto E. Manso U. de Valladolid6n3
Introduction¿ ¿Why Software Engineer don’t useExperimentation?Scientific method is notsuitableThe software engineers have to observethe phenomenon, to formulatehypothesis and to contrast themThe level ofThe software engineers don’t contrastexperimentation is enough their claims as much as other scientistThe experiments areexpensiveIt is possible to do a significantexperiment that is not expensiveThe shows are enoughThe shows don’t prove nothingThe technology changesspeedilyIf yesterday you said an important claimthat today is not important, that isbecause it does not well defined E. Manso U. de Valladolid7Introduction¿Why Software Engineer don’t useExperimentation?nnThe softwaresoftware engineersengineers thinkthink thatthatThennnnnnThe scientificscientific methodmethod isis notnot necessarynecessary inin softwaresoftwareTheengineeringengineeringn¿How testingtesting thethe ideasideas againstagainst realreal world?world?n¿HowThere isis notnot aa backgroundbackground ofof statisticalstatistical gnanexperimentor totoso it is very difficult to design an experiment oranalyse thethe experimentalexperimental resultsresultsanalyseThere isis notnot enoughenough cultureculture andand bibliographybibliography al software engineering E. Manso U. de Valladolid8n4
Introduction¿Why Software Engineer don’t useExperimentation?nnThe experimentationexperimentation inin SoftwareSoftware EngineeringEngineering isis moremoreThedifficult thanthan inin otherother sciences,sciences, becausebecause itit isisdifficultnecessaryalotofvariablesnecessary a lot of variablesnnnn¿It isisaavalidvalidraison?raison?¿ItTo publishpublish aa experimentalexperimental studystudy ofof SoftwareSoftwareToEngineering isis moremore difficultdifficult thanthan inin otherother calstudiesthatareFurthermore, the empirical studies that arereplications eraera notnot asas importantimportant asas newnew studies.studies.replicationsnnBut otherother sciencessciences havehave twotwo sides:sides: TheoryTheory andand PracticePractice andandButbotharerelatedboth are related E. Manso U. de Valladolid9IntroductionSoftware EngineeringA Laboratory ScienceResearcher’s role: To understand the nature ofProcesses and Products in the context of oner’s role: To build improve systems,using knowledge(Basili) E. Manso U. de Valladolid10n5
IntroductionIn ConclusionWe knowknow thatthat SoftwareSoftware Nature:Nature:Wennnnnnnndevelopment notnot productionproductionItItisis developmentThe disciplinediscipline technologiestechnologies areare human-basedhuman-basedTheThere areare aa largelarge numbernumber ofof variablesvariablesthatthat causecause differencesdifferences ààThere¿How measuremeasure theirtheir effects?effects?¿HowSoftware EngineeringEngineering needsneeds moremore onfirmTheoriesTheoriesandand “Conventional“Conventional McCabe’scyclomaticcyclomatic racyaccuracyofofModelsModelsTonnTo ValidateValidate ¿Isthethenumbernumberofofmethodsmethodsaa ?complexity?¿Is E. Manso U. de Valladolid11Type of studies2. Research MethodsnnThe analyticanalytic iveresultsDevelop and derive resultspossible, verifyverify thethe resultsresults empiricallyempiricallyIfIf possible,ChemistryPhisiqueMathematiquesSoftware E.nnThe engineeringengineering andand empiricalempirical methodsmethodsThe(scientific methodl)methodl)(scientificnnnnnnnnObserving thethe worldworldObservingProposingamodelor otherother solutionssolutionsProposing a model orMeasuringandanalysingMeasuring and analysingValidating oror invalidatinginvalidating thethe proposedproposed modelmodelValidating E. Manso U. de Valladolid12n6
Type of studiesResearch ParadigmsnnQuantitative ResearchResearchQuantitativennnnnnControlled Verification orientedorientedVerificationnnQualitative ResearchResearchQualitativennNaturalistic andand servationsSubjectiveSubjectivennDiscovery orientedorientedDiscoverynnStudynn StudyAn actact toto testtest aa hypothesishypothesis oror discoverdiscover somethingsomethingAnCan includeinclude quantitativequantitative andand qualitativequalitative researchresearchnn Cannn E. Manso U. de Valladolid13Type of studiesResearch ParadigmsQualitative ResearchResearchQualitativevvWe study,study, usingusing aa meeting,meeting, thethe reasonreason becausebecause newthe productivity when a team have used aa newlanguage.languagevvThis wouldwould bebe aa QualitativeQualitative studystudy aboutabout thinksthinks s logic and human reasoning .The analysisanalysis willwill bebe aboutabout thethe wordswords whichwhich cancan bebeTheorganized inin orderorder toto thethe researcherresearcher cancan test,test, compare,compare,organizedanalyze andand identifyidentify patronspatrons.analyzevv E. Manso U. de Valladolid14n7
Type of studiesResearch ParadigmsQuantitative ResearchResearchQuantitativevvWe study,study, usingusing aa quantitativequantitative study,study, thethe reasonreasonWebecause increaseincrease thethe productivityproductivity whenwhen aa teamteam havehavebecauseused aa newnew languagelanguage.usedvvmust toto definedefine thethe hypothesis,hypothesis, toto planplan thethe process,process,II mustto selectselect thethe independentindependent andand dependentdependent variables,variables,toand toto controlcontrol extraneousextraneous factors.factors.andThe analysisanalysis willwill bebe aboutabout thethe numericnumeric on,usingobserved as result of experiment execution, usingstatistical techniquestechniques toto testtest thethe hypothesis.hypothesis.statisticalvv E. Manso U. de Valladolid15Type of studiesEmpirical StrategiesEmpirical StrategiesStrategiesEmpiricalObserved tisticsReal WorldWorldRealStatistical InferenceInferenceStatistical E. Manso U. de Valladolid16n8
Type of studiesEmpirical StrategiesFirst Level:Level: HypothesisHypothesis (Model)(Model) àà ControlledControlledFirstexperiments inin laboratory,laboratory, withwith ilitySecond Level:Level: HypothesisHypothesis (Model)à(Model)à inin aaSecondreal environment,environment, usingusing observationalobservationalrealstudies (case(case studies)studies)studiesThirth Level:Level: ModelModel appliedapplied inin allall realreal process,process,Thirthwe mustmust mademade aa historicalhistorical filefile (surveys).(surveys). InIn thethewefutur wewe havehave toto testtest thethe ModelModel withwith thisthis file.file.futur E. Manso U. de Valladolid17Type of studiesEmpirical StrategiesnDepending of the degree of control over QuestionnairesnnnnRetrospectiveRetrospectiveTheThe teamteam arearennnnnnCase StudynnnnnnDataData collectioncollectionnn ToTo avoidavoid stical AnalysisAnalysisConclusionsConclusionsnn GeneralizationGeneralization isis difficultdifficult E. Manso U. de ValladolidnnnnnnObservationalObservationalnn WithWith littlelittle controlcontrolTheThe teamteam cativeExploratoryExploratoryToTo comparecompareToestablishTo establish relationshiprelationshipInIn aa specificspecific time-spacetime-space18n9
Type of studiesEmpirical StrategiesnDepending of the degree of control over datannExperimentnnnnIsIs aa ProcessProcessnn StatisticalStatistical AnalysisAnalysisItIt isis possiblepossible replicationreplicationnnnnToTo confirmconfirmToTo generalizegeneralizennnnControlledControlled ProcessProcessTheThe teamteam arearennnnnnnnToTo ConfirmConfirm TheoriesTheories andand“Conventional“Conventional Wisdom”Wisdom”ToTo ExploreExplore RelationshipsRelationshipsToTo EvaluateEvaluate thethe accuracyaccuracyofof ModelsModelsToTo ValidateValidate MeasuresMeasures E. Manso U. de Valladolid19Type of studiesEmpirical StrategiesExploratory SurveyWhy the developers think that a technique A is betterthan other B?Case study (Relationship)We want to build a model to predict the number offaults in testing, in a enterpriseExperimentWe want to compare two inspection methods, in alaboratory environment, that is, selecting variablesand controlling extraneous factors, E. Manso U. de Valladolid20n10
Type of studiesEmpirical Strategies Factors§ Execution ControlHow much the researcher control the studie?§ Measurement ControlThe degree to wich the researcher can decide uponwich measures to be collected¿In a survey?§ Investigation Costrelated with the factors above§ Easy Replicationinvolves repeating the investigation under identicalconditions, in another population E. Manso U. de Valladolid21Type of studiesEmpirical StrategiesComparisonFactorSurveyCase ediumHighHighLowHigh E. Manso U. de Valladolid22n11
Type of studiesExperimental Studies(Another Classification)Driven by hypothesisnControlled experimentnnTo demonstrate feasibility in smallQuasi-experimentsTo simulate the effects of the treatment variables in arealistic environmentn E. Manso U. de Valladolid23Type of studiesObservational Studies(Another Classification)Driven byunderstanding# of sitesOneMore than oneVariable ScopesA priori defined No a priori definedDeductions:Deduction: verbalMathematicalpropositionsformal logicCase studyField study E. Manso U. de ValladolidCase qualitativestudyField qualitativestudy24n12
Type of studiesExperimental StudiesControlled ExperimentControlled V.CmmCC11.C.XnnXX11.XExtraneous Factors:Bias and highVariability.YrrYY11.YProcessIndependent V.Dependent V.rkkrr11.rTreatmentsTreatmentsRandomized V. E. Manso U. de ValladolidTypes of studies25Experimental StudiesBasic ConceptsnnIndependent variablesvariables (factor,(factor, state,state, predictand)predictand)IndependentnnDependent variablesvariables (response,(response, predictor)predictor)DependentnnnnWhich wewe cancan controlcontrol andand changechange inin thethe experimentexperimentWhichThey measuremeasure thethe effecteffect ofof thethe treatmentstreatments andand appearappear inin thetheTheyHypothesis testtestHypothesisnnControlled variablesvariablesControllednnRandomized variablesvariablesRandomizednnConfounded variablesvariablesConfoundednnnnTo convertinnnThey cancan bebe controlledcontrolled byby thethe designdesignTheyThey areare consideredconsidered asas randomrandom errorerror inin thethe designdesignTheyThey aren’taren’t controlledcontrolled andand changechange togethertogether withwith aa independentindependentTheyvariablevariable E. Manso U. de Valladolid26n13
Types of studiesExperimental StudiesBasic ConceptsnnnnnnnnnnTreatment: eacheach combinationcombination ofof thethe levelslevels ofof fthereisonlyone,eachlevel willwill bebeindependent variables. If there is only one, each levelatreatment.a treatment.Population ofof subjects:subjects: wewe cancan generalizegeneralize thethe resultsresults overover thethePopulationpopulationpopulationSample: subjectssubjects selectedselected fromfrom thethe populationpopulation (( ¿subjects¿subjectsSample:selection?à planning)planning)selection?àObjects: objectsobjects ofof thethe study:study: products,products, process,process, heGoaldefinitiontemplate)models, etc. (Is a part of the Goal definition template)Experiment: setset ofof trialstrials (treatment(treatment subjectsubject object) object)Experiment: E. Manso U. de Valladolid27Types of studiesExperimental StudiesExperiment ExampleAnalyze a new design tool and a old design tool, for thepurpose of to compare their impact with respect to productivity,from the point of view of developers, in the context of theuniversity students.¿ Dependent Variable?Productivity ¿measurement?¿¿ IndependentIndependent Variables?Variables?¿Extraneous Factor?Population? Objects? Sample? E. Manso U. de ValladolidNew Tool / Old ToolEnvironmentEnvironmentProduct TypesTypesProduct¿Any More?More?¿Any28n14
Types of studiesExperimental StudiesExperiment ExampleAnalyze the object oriented design method vs. processmethod, for the purpose of to evaluate with respect toquality, from the point of view of developers, in the contextof the university students.Quality¿ Dependent Variable?¿¿ IndependentIndependent Variables?Variables?Development methodOO vs. Process orientedExperienceExperienceType ofof productproductType¿Extraneous Factor?Environment.Environment. E. Manso U. de ValladolidTypes of studies29Experimental StudiesExperimentPopulation:Objectsn Studentsn Toysn Professionalsn Real productExperience: 44 levelslevelsExperience:sampleProduct Types:Types: oneone typetypeProduct¿Control of ExtraneousEnvironment: oneone typetypeEnvironment:Factor?¿Any More?More?¿Any¿Experiment¿Experiment Validation?Validation? ¿¿ ConclusionsConclusions Generalization?Generalization? E. Manso U. de Valladolid30n15
Experimentation Process3. Experimentation onDiscussion &&DiscussionConclusionsConclusionsAnalysis &&AnalysisInterpretationInterpretationSummaryPilot studystudyPilotDataThreatsThreatsSummary E. Manso U. de Valladolid31Experimentation ProcessDefinition3.1 Experiment Definition¿Why?Goal definition templatenExperiment definition:Analyze object of the study n The PBR and checklist techniquesFor the purpose of purpose n EvaluationWith respect to quality focus n Effectiveness and efficiencyn The researcherFrom the point of view ofn perspective In the context of context nnn E. Manso U. de ValladolidnM. Sc and Ph. D students32n16
Experimentation ProcessPlanning3.2 Experiment Experiment ctsselectionExperimentDesign E. Manso U. de Valladolid33Experimentation ProcessPlanning. ContextContext SelectionOff-line vs. On-lineReduces the riskProduces extra costsStudents vs.ProffesionalReduces the costsEasier to controlContext generalization?Toy vs. RealproblemReduces the costs & timeContext generalization?Specific vs.General E. Manso U. de ValladolidReduces the costs & timeContext generalization?.34n17
Experimentation ProcessPlanning. ContextExperiment sili)#OnesubjectsperMoreobjectthan oneOneMore than oneSingle object studyMulti-object variationstudyMulti-test withinobject studyBlocked subject object study E. Manso U. de ValladolidExperimentation Process35Planning. ContextExperiment ContextConclusion“Be suresure toto specifyspecify asas muchmuch ofof thethe industrialindustrial contextcontextC1. “Beas possible.possible. InIn particular,particular, clearlyclearly definedefine thethe entities,entities,asattributes, andand measuresmeasures thatthat areare capturingcapturing l information”It isis necessarynecessary ininItn ObservationalObservational andandnnExperimentalstudiesn Experimental studiesC2. “If a specific hypothesis is being tested, state it clearlyprior to performing the study and discuss the theory fromwhich it is derived, so that its implication are apparent” E. Manso U. de Valladolid36n18
Experimentation ProcessPlanning. ContextExperiment ContextConclusionC3. “If the research is exploratory, state clearly and,prior to data analysis what questions theinvestigation is intended to address and how it willaddress them”C4. “Describe research that is similar to, or has abearing on, the current research and how current workrelates it” E. Manso U. de ValladolidExperimentation Process37Planning. HypothesisHypothesis FormulationDerived fromfrom ExperimentExperiment definition:definition: oneone oror moremore HH00DerivedGoal definitiondefinition templatetemplateGoalAnalyze TheThe PBRPBR andand checklistchecklist techniques(CKL)techniques(CKL)AnalyzeFor thethe purposepurpose ofof EvaluationEvaluation WithWith respectrespect totonn Forefficiency andand effectivenesseffectivenessefficiencyFrom thethe pointpoint ofof viewview ofof TheThe researcherresearchernn FromnInthecontextofM.ScandPh.D studentsstudentsn In the context of M. Sc and Ph. DnnPBR efficiencyefficiency CKLCKL efficiencyefficiencyHH0101:: PBRPBR effectivenesseffectiveness CKLCKL effectivenesseffectivenessHH0202:: PBR E. Manso U. de Valladolid38n19
Experimentation ProcessPlanning. HypothesisHypothesis FormulationThe observedobserved vehiclevehicle isis aa carcarHH00:: TheH1: TheThe observedobserved vehiclevehicle isis notnot aa carcar ààH1:Critical AreaArea (C.A(C.A.) .) {#wheels{#wheels 55 oror #wheels#wheels 3} 3}CriticalIf wewe observeobserve 33 oror lessless wheelswheels oror 55 oror moremore wheelswheels weweIfreject HH00 àà ¿error?¿error?rejectα p(number of wheels 4/ car )If wewe observeobserve 44 wheelswheels wewe don’tdon’t rejectreject HH00Ifà¿error?¿error?àβ p(number of wheels 4/ not car ) E. Manso U. de Valladolid39Experimentation ProcessPlanning. HypothesisHypothesis TestingDerived from Experiment definition: one or more H 0Null HypothesisHypothesis tmenttreatmenteffect)effect)HH00:: NullH1: AlternativeAlternative HypothesisHypothesis àà CriticalCritical AreaArea (C.A.)(C.A.)H1:ReallyWedecide.Non reject H0(Non significant result)Reject H0(Significant result) E. Manso U. de ValladolidH0 is trueH1 is true1- αβ Error P( C.A./H1)α Error (significancelevel) P(C.A./H0 )Test Power P(C.A./H1)40n20
Experimentation ProcessPlanning. HypothesisHypothesis TestingHH00:: NullNull HypothesisHypothesisWeWe needneed toto selectselect aa “random“random measure”measure” (m)(m) ofof thethe effecteffectofof treatment:treatment: thethe estimateestimatennnnTimeTime toto understandunderstand aa age of defects detected inin aa documentdocumentParametricParametric TestTest àà thethe distributiondistribution patternpattern ofof mm isisknowledgedknowledgednnnnnnTimeTime isis N(µ,σ)N(µ,σ)1/2))PercentagePercentage isis B(n,p)B(n,p) (aprox.(aprox. N(N(p,p, (p*(1-p)(p*(1-p) 1/2NonNon ParametricParametric TestTest àà thethe distributiondistribution patternpattern ofofmm isis acknowledgedacknowledged E. Manso U. de Valladolid41Experimentation ProcessPlanning. HypothesisHypothesis Test: Performance1.1.2.2.3.3.4.4.5.5.6.6.7.7.To define Hypothesis H00 and H11To select the suitable estimateTo determine the error α (usually 0,05 or 0,01)Using 1, 2 and 3 to determine the Critical Area (C.A.)Using n, H00 and H11, and the C.A. to determine ββTo reject H00 or not from the observed (estimation)value of estimateα11 0.05α22 0.10α 33 0.01 E. Manso U. de Valladolidàààβ β1β β2 β1β β3 β142n21
Experimentation ProcessPlanning. HypothesisHypothesis Testingnn1- ββ (Test(Test Power):Power): probabilityprobability thatthat thethe testtest willwill revealreveal1true patternpattern ifif HH0isis falsefalseaa falsecancanbebeunknownunknownàà ¿¿11- ββ?The0We shouldshould choosechoose aa testtest withwith asas highhigh powerpower asasWepossible (increasing(increasing n,n, forfor example)example)possible1- ββ dependsdepends onon α,α, samplesample sizesize (n)(n) andand effecteffect sizesize11- ββ isis betterbetter whenwhen wewe havehave testtest parametricparametric1- E. Manso U. de Valladolid43Experimentation ProcessPlanning. HypothesisHypothesis TestingT X estimate with known pattern N(µ, σΤ) when Ho is trueHo: µ 0Ho: µ 0f(T)0,060,050,040,03H1 : µ µoMean,Std. dev.0,715,7-12,0,7µ0 -12.0µ1 15α α0µ1 25µ0 0α α0β β0β β00,02µ 15σΤ 70,010-47-27 E. Manso U. de Valladolidβ0-7µo 0 13σΤ 733α053 TC.A.44n22
Experimentation ProcessPlanning. HypothesisHypothesis TestingConclusionsD1. “Identify the population from which the subjectsand objects are drawn ”D2.”Define the process by which the subjects andobjects were selected”n TheThe conclusionconclusion maymay bebe usefuluseful ifif thethe samplesample arearenrepresentativerepresentativen WeWe mustmust toto excludeexclude thethe studentsstudents withwith aa lotlot ofofnexperience inin thethe experiment.experiment. TheyThey areare notnotexperiencerepresentative.representative. E. Manso U. de ValladolidExperimentation Process45Planning. HypothesisHypothesis TestingConclusionsD3. “Define the process by subjects and objects areassigned to treatmentsD4. “Restrict yourself to simple study designs or, atleast, to designs that are fully analyses in the statisticalliterature. If you are not using a well-documenteddesign and analysis method, you should consult astatistician to see whether yours is the most effectivedesign for what you want to accomplish” E. Manso U. de Valladolid46n23
Experimentation ProcessPlanning. HypothesisHypothesis TestingConclusionsD5. “Define the experiment unit”n IfIf youyou areare evaluatingevaluating teamsteams butbut youyou getget measuresmeasuresnfrom eacheach teamteam membermember ¿what¿what itit isis thethe experimentalexperimentalfromunit? àà teamteamunit?D6. “For formal experiments, perform a pre-experimentor precalculation to identify or estimate the minimumrequired sample size”n TheThe samplesample sizesize determinedetermine thethe testtest powerpowern E. Manso U. de ValladolidExperiment Process47Planning. VariablesVariables SelectionnIndependent variablesnDependent variablesnnWhich we can control and change in the experimentThey measure the effect of the treatments and appear in theHypothesis testnControlled variablesnRandomized variablesnnThey can be controlled by the designThey are considered as random error in the designConfounded variablesnnTo convertinThey aren’t controlled and change together with a independentvariable E. Manso U. de Valladolid48n24
Experiment ProcessPlanning. SubjectsSubjects Selectionnn¿How to select the subjects?nnnnCan be probability or non-probabilityn Simple random sampling, systematic sampling nn Convenience sampling, quota sampling n¿Size of the sample?nnIf there is a large variability, a larger size we needThe Sample from the Population must be representative E. Manso U. de ValladolidExperiment Process49Planning. DesignExperiment heexperimentconclusions of the experimentRelevantnQuestionsRepeated measuresnBlocking RandomizationnCrossed designNested designn¿How many independent variables arethere?n Only one à Simple experimentsn More than one à Factorial experiments¿How many treatments per subject?¿How “to control” extraneous factors?¿How “to combine” the independent variableslevels? à # treatmentsThe answersanswers willwill dependdepend onon thethe validityvalidity ThreatsThreats wewe wantwant toto controlcontrolThe E. Manso U. de Valladolid50n25
Experiment ProcessPlanning. DesignGeneral Design PrinciplesRandomization Blocking BalancingRandomizationisis usedused totoRandomizationnnnnnnAssure thethe observationsobservations areare fromfrom independentindependent randomrandomAssurevariablesvariablesAllocate objects,objects, subjectssubjects andand inin whichwhich orderorder thethe testtest areareAllocateperformedperformedAverage outout thethe effecteffect ofof aa extraneousextraneous factorfactorAverageBlockingBlockingnnnnnnBlocking subjectssubjects isis usedused toto eliminateeliminate thethe undesiredundesired effecteffect ininBlockingthe comparisoncomparison amongamong thethe treatmentstreatments ofof aa extraneousextraneous factorfactorthethat wewe areare notnot interestedinterested ininthatnWithinablocktheundesiredeffect isis thethe same,same, andand wewen Within a block the undesired effectcan studystudy thethe effecteffect ofof treatmentstreatments onon thatthat blockblockcanBlocking increasesincreases thethe precisionprecision ofof thethe experimentexperimentBlockingBlocking treatmentstreatments isis usedused toto reducereduce dede amountamount ofofBlockingtreatments forfor subjectsubjecttreatments E. Manso U. de Valladolid51Experiment ProcessPlanning. DesignGeneral Design PrinciplesRandomization Blocking BalancingBalancingBalancingnnnnThe numbernumber ofof subjectssubjects perper treatmenttreatment isis thethe ointof viewview ofofIt is not necessary, but is desirable from the point ofstatisticalanalysisofthedata.statistical analysis of the data.Treatment1 Treatment2 Treatment3 ect1414Subject1212Randomized and Balanced E. Manso U. de Valladolid52n26
Experiment ProcessPlanning. DesignGeneral Design PrinciplesThe principalprincipal ClaimsClaims ofof thethe experimentexperiment designdesignTheare::arennnnnnTo reducereduce thethe variabilityvariabilityToTo controlcontrol extraneousextraneous factorsfactorsToTo reducereduce thethe differentdifferent threatsthreats toto dity as much as possible E. Manso U. de Valladolid53Experiment ProcessPlanning. DesignExperiment DesignA TaxonomySimpleBetween Subjects(BS)Within Subjects(WS)Random groupsRandom blocks (matched groups)RandomizationCounterbalancing (2 levels)Latin squares (3 or more levels)BS x BSFactorialComplete(2 factors)BS x WSWS x WSComplete confoundingPartial confoundingPartial Fractional E. Manso U. de Valladolid54n27
Experiment ProcessPlanning. DesignSimple Design Between SubjectsCharacteristicsnnEach subjectsubject hashas onlyonly oneone TreatmentTreatmentEachnnThreats toto internalinternal validity:validity: SelectionSelection isis thethe principalprincipal threat,threat,Threatsnnis thethe effecteffect ofof naturalnatural variationvariation inin humanhuman performance.performance.itit isTo avoidavoid thisthis threatthreat::TonnnnRandomization: thethe subjectssubjects areare assignedassigned toto thetheRandomization:treatment randomlyrandomlytreatmentBlocking: WeWe havehave subjectsubject inin eacheach blockblock withwith thetheBlocking:same valuevalue inin thethe blockedblocked variable.variable. WeWe assignassignsamerandomly allall treatmentstreatments inin eacheach blockblockrandomly E. Manso U. de Valladolid55Experiment ProcessPlanning. DesignSimple Design Between SubjectsStatistical HypothesisThe mostmost c
Introduction n “Software Engineering means application of systematic, disciplined, quantifiable approach to development, operation and maintenance of . n The experimentation in Software Engineering is more difficult than in o
investigating whether radical experimentation is sustainable in the long run. This work draws on several strands of literatures, concerned with experimentation, social learn-ing, and innovation. The literature on multi-armed bandits and strategic experimentation typically considers a xed set of independent arms.
Anytime-Anywhere Engineering Experimentation Abstract: This work examines the delivery of quality, hands-on engineering experimentation in a home or remote environment. The course was designed to develop experimental skills in engineering measurement methods, based on electronic instrumentation and computer-based data acquisition
experimentation, A/B testing, software architecture, causal infer-ence, science-centric ACM Reference format: iasGerostathopou-los, Matthew Wardrop, Tobias Mao, and Colin McFarland. 2020. Engineering for a Science-Centric Experimentation Platform. In Proceedings of Software
Experimentation in Software Engineering Software Engineering: the state of the art . Experimentation in Software Engineering: An Introduction Claes Wohlin et al., Kluwer Academic Publishers,
both teaching and experimentation was essential, and adjustments in the curriculum were needed more than ever. Therefore, in this paper we present an adaptive and on-demand education framework for engineering students, thereby enabling remote experimentation and adjustments of exercise content to enhance students' learning experience.
4. Ethics in engineering: 4.1 Purpose and concepts of engineering ethics, 4.2 Engineering as social experimentation, 4.3 Types of enquiry, 4.4 Issues in engineering ethics,. 5. Engineer's responsibility and Safety: 5.1 Safety, Risk, Underestimating the Risk, Over-estimating the Risk, Risk-Benefit analysis, 5.2 Cause of an accident and
however, was not a jump but rather an evolution over a period of years. This development occurs through phases and we illustrate this process by creating the "Experimentation Evolution Model". With this model, we describe the steps to take while evolving data-driven development practices towards continuous experimentation at scale.
15th AMC ! 8 1999 5 Problems 17, 18, and 19 refer to the following: Cookies For a Crowd At Central Middle School the 108 students who take the AMC! 8 meet in the evening to talk about prob-lems and eat an average of two cookies apiece. Walter and Gretel are baking Bonnie’s Best Bar Cookies this year. Their recipe, which makes a pan of 15 cookies, list these items: 11 2 cups of our, 2 eggs .
