Agile Test Automation Strategy
Agile Test Automation StrategyFor Anyone and Everyone!Gerard MeszarosAgile2012ATAS@gerardm.comMuch Ado About Agile 20111Copyright 2011 Gerard Meszaros
My Background Software developer80’s Development manager----- Project Manager Software architect90’s OOA/OOD Mentor----00’sEmbeddedTelecom Requirements (Use Case) Mentor XP/TDD MentorI.T. Agile PM Mentor Test Automation Consultant & Trainer Lean/Agile Coach/ConsultantProduct & I.T.Much Ado About Agile 20112Gerard MeszarosATAS2012@gerardm.comCopyright 2011 Gerard Meszaros
Agenda Motivation– The Agile Test Problem– The Fragile Test Problem Approaches to Test Automation Test Automation StrategyRough timings for Agile Test Automation StrategyMuch Ado About Agile 20113Slide ##TopicTime Slides Start EndMotivation11.2829Exercise 1 - Automation Motivation101 10 10Intro to Automation75 11 15Exercise 2 - Why not Record & Playback?101 16 16Why Automated Tests are Fragile8.46 17 22How Agile Automation Changes Things9.87 24 30Intro to Example-Driven Development75 32 36Managing Scope vs Detail in Examples15.411 38 48How to specify workflows8.46 50 55Exercise 3 - Workflow Tests (Keyword-Driven)151 56 56Using Data-Driven Tests to specify business rules8.46 55 60Exercise 4 - Business Rules Test (Data-Driven)151 61 61How Tests Interact With the SUT75 62 66Test-Driven Architecture5.64 67 70Legacy Systems (if time permits)19.614 71 84The Role of Unit Tests8.46 85 90Test Automation Strategy1410 91 100180.297Copyright 2011 Gerard MeszarosTime per slide:1.4 # of
Product Owner Goal Goal: Maximize business value t OwnerQuality is Assumed; Not ManagedMuch Ado About Agile 20114Copyright 2011 Gerard Meszaros
Why Quality Often Sucks Iron Triangle of Software Engineering:ResourcesQualityTimeFunctionality What about Quality?You can fix any three; the fourth is the outcomeMuch Ado About Agile 20115Copyright 2011 Gerard Meszaros
Why Quality Often Sucks Iron Triangle of Software Engineering:ResourcesQualityTimeIn Agile, we“Pin” qualityusingautomatedtestsFunctionality What about Quality?You can fix any three; the fourth is the outcomeMuch Ado About Agile 20116Copyright 2011 Gerard Meszaros
Speaking of Quality,would you . ask your doctor to reduce the costof the operation . by skipping the sterile technique ?Test Automation is like hand washing:Improved results but an upfront cost.Much Ado About Agile 20117Copyright 2011 Gerard Meszaros
Minimizing Cost of ProductTotal cost includes: developing the software verifying the newly built functionality verifying old functionality still works fixing any bugs found Verifying noting was broken by fixesAgile Test Automation can reduce thecost of all of these activities.Much Ado About Agile 20118Copyright 2011 Gerard Meszaros
Incremental DevelopmentInitialConceptNFVersion 1.0EvolvedConceptNFVersion 1.x Type NF bugs: New Func. is wrong Type RB bugs: New bugs in old func.(Regression Bugs)Much Ado About Agile 20119Version 1.0RBCopyright 2011 Gerard Meszaros
Exercise 1 Time to test our little application Oh, new build, please retest! Another build, please retest!Much Ado About Agile 201110Copyright 2011 Gerard Meszaros
The Agile Test ProblemRequirementsMuch Ado About Agile 2011Development11TestingCopyright 2011 Gerard Meszaros
The Agile Test evelopmentTesting As development increments reduce induration, testing needs to be reducedaccordinglyMuch Ado About Agile 201112Copyright 2011 Gerard Meszaros
The Agile Test uirementsDevelopmentTesting. and traditional approaches to testing nolonger workMuch Ado About Agile 201113Copyright 2011 Gerard Meszaros
Anatomy of an Automated TestPreconditions(State)Other SystemGiven.TestSetup1&2 May berepeatedTestTeardownTest Scenario Name-----------------------Our SystemPreconditions---------------------1. Do SomethingTest2. Check Something----------------------Clean UpInterfaceBusinessLogicDatabaseContainer ServicesOther SystemWhen .Then .Much Ado About Agile 2011Other SystemAdapter14Copyright 2011 Gerard Meszaros
(C)OTS Record&Playback User executes tests manually; tool records as tests Tool replays tests later without user interventionThe tests areare code/datainterpretedby the tsExpectedOutputsTest Script RepositoryTestScript 1TestScript 2TestScript nTestResultsScript nResultTest Result RepositoryMuch Ado About Agile 201115Copyright 2011 Gerard Meszaros
Exercise 2 Record & Playback Test Automation– Please record a test against the System Under Test– Then, run the test to make sure it works New build has been delivered– Please run the test against new buildMuch Ado About Agile 201116Copyright 2011 Gerard Meszaros
Agenda Motivation– The Agile Test Problem– The Fragile Test Problem Changing the Role of Test Automation Approaches to Test Automation Test Automation StrategyMuch Ado About Agile 201117Copyright 2011 Gerard Meszaros
The Fragile Test ProblemWhat, when changed,may break our testsaccidentally:– Behavior SensitivityOther SystemOther SystemOur System» Business logic– Interface Sensitivity» User or system– Data SensitivityInterfaceTest» Database contentsBusinessLogicDatabaseContainer Services– Context Sensitivity» Other system stateOther SystemIn Agile, these are all changing all the time!Much Ado About Agile 201118Copyright 2011 Gerard Meszaros
Interface Sensitivity Tests must interact withthe SUT through someinterface Any changes to interfacemay cause tests to fail.Other SystemOther SystemWindowOur System– User Interfaces:Title» Renamed/deleted windows ormessages» New/renamed/deleted fields» New/renamed/deleted data valuesin listsCaption FieldInterfaceLink ButtonData GridWindow– Machine-Machine Interfaces:» Renamed/deleted functions in API» Renamed/deleted messages» New/changed/deleted functionparameters or message fieldsMuch Ado About Agile 2011BusinessLogicDatabaseContainer ServicesOther SystemE.g.: Move tax fieldto new popup window19Copyright 2011 Gerard Meszaros
Behavior Sensitivity Tests must verify thebehavior of the system.Other System– Behavior also involved in testset up & tear down Any changes to businesslogic may cause tests tofail.– New/renamed/deleted states– New/changed/removedbusiness rules– Changes to businessalgorithms– Additional data requirementsMuch Ado About Agile 201120Other SystemOur orithmRuleDatabaseObjectContainerServicesOther SystemE.g.: Change fromGST PST to HSTCopyright 2011 Gerard Meszaros
Data Sensitivity All tests depend on“test data” which are:– Preconditions of test– Often stored in databases– May be in other systems Changing the contentsof the database maycause tests to fail.– Added/changed/deletedrecords– Changed SchemaOther SystemOther SystemOur Container ServicesOther SystemE.g.: Change customer’sbilling termsMuch Ado About Agile 201121Copyright 2011 Gerard Meszaros
Context Sensitivity Tests may depend on inputsfrom another system– State stored outside theapplication being tested– Logic which may changeindependently of our systemOther SystemCustomerXOtherSystemOur System Changing the state of thecontext may cause tests tofail.– State of the containerInterfaceBusinessLogicContainer ServicesContainer ServicesDate» e.g. time/date– State of related systemsDatabaseTimeSecuritySystemOtherSystem» Availability, data contentsUser: XPermissions: noneE.g.: Run test in ashorter/longer monthMuch Ado About Agile 201122Copyright 2011 Gerard Meszaros
Agenda Motivation Changing the Role of Test Automation– From Defect Detection to Defect Prevention– Different Tests for Different Purposes Approaches to Test Automation Test Automation StrategyMuch Ado About Agile 201123Copyright 2011 Gerard Meszaros
The Role of Automation in Agile Provide a Safety Net for Change & Innovation– Provide rapid feedback to reduce cost of fixing defects.» On demand (Developer) and event driven (CI build)– Rapid feedback enables experimentation» Don’t have to choose between Quick and Safe Guide Development of the Product– Provide executable examples of what “done” looks like Support Manual Testing– Remove the repetitive drudgery so testers can focus onhigh value activity by:– Automating entire tests, or by– automating the steps that can be automated.Much Ado About Agile 201124Copyright 2011 Gerard Meszaros
How is Agile Test Automation Different? We automate the tests for a different reason– Defect Prevention vs. Detection– To communicate requirements– To “Pin” the functionality once it’s built We automate the tests a different way– Many different kinds of tests» E.g. We don’t rely solely on GUI-based automation– Using tools that support collaboration & communication» in addition to confirmation We plan the automation based on ROI– Goal isn’t: 100% automation– Goal is: To maximize benefit while minimizing costMuch Ado About Agile 201125Copyright 2011 Gerard Meszaros
Traditional Role of TestingCritique ProductBusinessFacingAcceptance TestsRegression TestsTechnologyUnit TestsFacingComponent TestsInspectionto finddefects isWasteShigeo ShingoCo-inventor ofToyota Production SystemMuch Ado About Agile 2011Usability TestsExploratory TestsReportCardFunctionality BUsabilityCScalability AResponseBAvailability CProperty Tests(Response Time,Security, Scalability)Inspectionto preventdefects isessential26 Marrick and Mary PoppendieckCopyright 2011 Gerard MeszarosQuadrants courtesy of Brian
Changing the Role of TestingRequirementsBusinessFacingCritique ProductDefine ProductAcceptance TestsRegression TestsUnit TestsTechnologyComponent TestsFacingSoftwareDesignMuch Ado About Agile 2011Preventanticipatabledefects fromhappeningReportCardUsability TestsExploratory TestsFunctionality BUsabilityCScalability AResponseBAvailability CProperty Tests(Response Time,Security, Scalability)Find nonanticipatableDefects, ASAP!27 Marrick and Mary PoppendieckCopyright 2011 Gerard MeszarosQuadrants courtesy of Brian
Changing the Role of TestingRequirementsBusinessFacingDefine ProductAcceptance TestsRegression TestsUnit TestsTechnologyComponent TestsFacingSoftwareDesignMuch Ado About Agile 2011Critique ProductUsability TestsExploratory TestsReportCardFunctionality BUsabilityCScalability AResponseBAvailability CProperty Tests(Response Time,Security, Scalability)For effective prevention:1. Tests must be available beforedevelopment2. Developers must be able to run testsbefore check-in28 Marrick and Mary PoppendieckCopyright 2011 Gerard MeszarosQuadrants courtesy of Brian
Reducing the Cost to Fix DefectsCost to understandand fix a defect goesup with the time ittakes to discover it. Why? We can remember where we put the newlyinserted defect because1. We know what code we were working on2. The design of the code is still fresh in our minds We may have to change less code– Because we wrote less codebased on the defect29Copyright 2011 Gerard MeszarosMuch Ado About Agile 2011
Continuous Acceptance Testing! Defines what “Done Looks Like”– Several to many tests per User Story / Feature Tests executed as soon developer says “It’sReady”– End-of-iteration: OK– Mid-iteration : BetterWrite StoryTest Build Code Test Code Test StoryWrite StoryTest Build Code Test Code Test Story30Much Ado About Agile 201130Copyright 2011 Gerard Meszaros
Continuous Readiness Assessment! Defines what “Done Looks Like”ReadinessAssessment– Several to many tests per User Story / Feature Executed by developers during development– To make sure all cases are implemented– To make sure it works before showing to business Tests executed as soon developer says “It’sReady”– End-of-iteration: OK– Mid-iteration : BetterWrite StoryTest Build Code Test Code Test StoryWrite StoryTest Build Code Test Code Test Story31Much Ado About Agile 201131Copyright 2011 Gerard Meszaros
Prevention: - Building the Right ProductWhat the customer thought they wantedWhat the customer actually asked forWhat the customer realized they actually neededWhat development thought the customer asked forWhat development actually builtWhat testing thought the customer asked forWhat testing actually tested forMuch Ado About Agile 201132Copyright 2011 Gerard Meszaros
Building the Right Product How do we eliminatethe waste caused bybuilding the wrongproduct?– Missed requirements?– Misunderstoodrequirements?– Unneeded functionality?Much Ado About Agile 201133Copyright 2011 Gerard Meszaros
Building the Right Product How do we eliminatethe waste caused bybuilding the wrongproduct?– Missed requirements?– Misunderstoodrequirements?Much Ado About Agile 201134Copyright 2011 Gerard Meszaros
Example-Driven Development A.K.A.– Acceptance Test Driven Development– Behaviour-Driven Development– Executable Specification– StoryTest-Driven Development Concrete examples flesh out requirements Testers flush out missed scenarios.before development startsMuch Ado About Agile 201135Copyright 2011 Gerard Meszaros
Life Cycle of an Example / TestStoryExamplesMake ConcreteBy Adding xamplesMuch Ado About Agile 201136Copyright 2011 Gerard Meszaros
Agenda Motivation Changing the Role of Test Automation– From Defect Detection to Defect Prevention– Different Tests for Different Purposes Approaches to Test Automation Test Automation StrategyMuch Ado About Agile 201137Copyright 2011 Gerard Meszaros
Test Automation Pyramid Large numbers of verysmall unit testsExploratory Tests– Ensures integrity of code Smaller number offunctional tests for majorcomponents– Verify integration of units Even fewer tests for theentire application &workflowSystemTestsComponentTestsUnit Tests– Ensure application(s) supportusers’ requirements Tools to support effectiveexploratory testingMuch Ado About Agile 2011Pyramid originally38 proposed by Mike CohnCopyright 2011 Gerard Meszaros
Behavior Specification at Right Level Specify broad scope at minimum detail– E.g. Use least detail when specifying workflow Specify most detailed req’ts at narrowest �� E.g. Don’t use workflow when specifying business rulesToo much detailUnmaintainableMake examples /tests easy tounderstand andeasy to writeComponentBusinessTestsRulesToo vagueBroadMuch Ado About Agile 2011UnitTestsUnit TestsScopeNarrow39Copyright 2011 Gerard Meszaros
Example: MegaBank Requirements Notify user of transactions against theiraccounts. User can configure threshold amount fornotification based on any/all of account,transaction type or region, charge category Notification can be sent via e-mail, voice-mailor SMS/IM User can suspend notifications indefinitely orfor a defined period of time.Much Ado About Agile 201140Copyright 2011 Gerard Meszaros
Example:Mega Bank Use CasesConfigure NotificationThresholdSuspend NotificationAccountHolderResume NotificationProcess TransactionTransactionSettlementMuch Ado About Agile 201141Copyright 2011 Gerard Meszaros
Example:Specifying Notification WorkflowUse Case:ManageNotificationThresholdsUse Case:ProcessTransactionBroad Scope; Minimum Detail;No mention of User Interface!Much Ado About Agile 201142Check outputof Use Case:ProcessTransactionCopyright 2011 Gerard Meszaros
Alternate form of Workflow Test:Given Bobma has account 1003592877And BobMa sets notification threshold to 10,000 for all transactionsWhen the bank processes debit for 15,000 toaccount 1003592877And the bank processes debit for 9,000 toaccount 1003592877And the bank processes debit for 11,000 toaccount 1003592877Then bobma receives notification for debit15,000 to account 1003592877And bobma receives notification for debit 11,000to account 1003592877Much Ado About Agile 201143Copyright 2011 Gerard Meszaros
Example:Specifying Suspension WorkflowUse Case:ManageNotificationThresholdsUse Case:ProcessTransactionUse Case:SuspendNotificationUse Case:ResumeNotificationUse Case:ViewNotificationsMuch Ado About Agile 201144Copyright 2011 Gerard Meszaros
Example:GUI for Manage Notifications Tx User Interfaceimplies specificfunctionality:– List of accounts– Ability to makechanges tonotifications– List of activenotifications This functionalitycan be testedindependently of UIMuch Ado About Agile 201145Copyright 2011 Gerard Meszaros
Example:Single Transaction TestUse Case:ManageNotificationsData to be shown onManage Accounts TabSide effect of AddingA NotificationMedium Detail; Medium ScopeStill no mention of User Interface!Much Ado About Agile 201146Data to be shownon ManageNotifications TabCopyright 2011 Gerard Meszaros
Example:Business Rule SpecsThreshold per Charge TypeConfigurationProcess TransactionMuch Ado About Agile 2011High Detail; Narrow ScopeCompletely ignores UI!47Copyright 2011 Gerard Meszaros
Changing Level of Abstraction/DetailLow Need to Reduce Detail or Reduce ScopeToo vague(Rarely esToo much detailUnmaintainableBroadMuch Ado About Agile 201148ScopeNarrowCopyright 2011 Gerard Meszaros
Agenda Motivation Changing the Role of Test Automation Approaches to Test Automation– Test Preparation Approach– Test Definition Language– Test Execution Interface Test Automation StrategyMuch Ado About Agile 201149Copyright 2011 Gerard Meszaros
Why is Automation ApproachImportant? Common Failure Mode:–Choose tools, then try to make them work–Wrong tools can prevent achieving goals Better Approach:–Choose automation approach to achieve goals–Then, select tools to support itMuch Ado About Agile 201150Copyright 2011 Gerard Meszaros
Common Approaches to faceRaw UIAdapterAPITest DataGlobal, StaticPer RunPer TestHow Set xtureTest Scenario ------------Our SystemRaw UI1. Do Something Via2. Check Something----------------------- APIClean UpMuch Ado About Agile ntainer Services51Copyright 2011 Gerard Meszaros
(C)OTS Record&PlaybackTestInterfaceRaw UI #AdapterAPINotes:* Keywords, if used, tend to be very low level: GotoWindowNamed: name SelectFieldNamed: name EnterText: text (Not the same as true Keyword-Driven testing)# Most COTS Tools operate at UI or HTTPinterface; many open-source tools do so as wellMuch Ado About Agile parationRecordedRefactoredHand-writtenTest DataGlobal, StaticPer RunPer TestPoor OKExample DrivenXWorkflowSystemBusiness iness RulesXUnitXGoodXXCopyright 2011 Gerard Meszaros
Keyword-Driven Tests The tests are expressed in domain-specific vocabulary. The tests are read & executed by a test interpreter writtenby techies.definitionexecutionPrepared like HandCoded Tests butwith a much morelimited vocabulary.Much Ado About Agile 2011Keyword53Copyright 2011 Gerard Meszaros
Keyword-Driven TestsTestLanguageCodeKeywordDataTestInterfaceRaw UI *AdapterAPINewNotes: While the Keyword Interpreter may goagainst the Raw UI, it is better to delegateto an adapter if no API is and-writtenMuch Ado About Agile 201154Test DataGlobal, StaticPer RunPer TestPoor OK GoodExample DrivenXWorkflowXSystemXBusiness iness RulesxUnitxCopyright 2011 Gerard Meszaros
Sample Keyword-Driven Test(e.g.
Support Manual Testing –Remove the repetitive drudgery so testers can focus on high value activity by: –Automating entire tests, or by . Software Design For effective prevention: 1. Tests must be available before development 2. Developers must be able to run tests before check-in.
1. The need for an agile way of working 6 2. The need for an agile way of working 9 3. Agile Core Values - Agile Project Management Vs. 10 Agile Event Management 4. Agile principles 12 _Agile Principles of Agile Project Management 13 _Agile Principles of VOK DAMS Agile Event Management 14 5. Agile Methods 16 _Scrum in Short 16 _Kanban in Short 18
AGILE TESTING For agile testers, test engineers, test managers, developers, technical leads. Ensure the production of effective and valuable software. Agile Fundamentals Agile Programming Agile Software Design Agile Fundamentals Agile Testing Agile Test Automation ICP CERTIFIED PROFESSIONAL ICP CERTIFIED PROFESSIONAL ICP-PRG CERTIFIED .
1.1 Purpose of the Agile Extension to the BABOK Guide1 1.2 What is Agile Business Analysis?2 1.3 Structure6 Chapter 2:The Agile Mindset 2.1 What is an Agile Mindset?7 2.2 The Agile Mindset, Methodologies, and Frameworks8 2.3 Applying the Agile Mindset9 2.4 Agile Extension and the Agile Ma
Agile Estimating and Planning by Mike Cohn Agile Game Development with Scrum by Clinton Keith Agile Product Ownership by Roman Pichler Agile Project Management with Scrum by Ken Schwaber Agile Retrospectives by Esther Derby and Diana Larsen Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin and .
Agile World View "Agility" has manydimensions other than IT It ranges from leadership to technological agility Today's focus is on organizational & enterprise agility Agile Leaders Agile Organization Change Agile Acquisition & Contracting Agile Strategic Planning Agile Capability Analysis Agile Program Management Agile Tech.
The most popular agile methodologies include: extreme programming (XP), Scrum, Crystal, Dynamic Sys-tems Development (DSDM), Lean Development, and Feature Driven Development (FDD). All Agile methods share a common vision and core values of the Agile Manifesto. Agile Methods: Some well-known agile software development methods include: Agile .
1. Agile methods are undisciplined and not measurable. 2. Agile methods have no project management. 3. Agile methods apply only to software development. 4. Agile methods have no documentation. 5. Agile methods have no requirements. 6. Agile methods only work with small colocated teams.-7. Agile methods do not include planning. 8.
The Agile Customer . 9/6/2012 6 Agile Development Team Agile Analyst . 9/6/2012 7 Agile Programmer Agile Tester . 9/6/2012 8 Agile Manager Agile Usability Designer . 9/6/2012 9 Kicking off a project The Inception Deck –Ten questions you’d be crazy not to ask before starting any
