Colm OhEocha Agile Testing at Eurostar2014

25/11/2014 Preven-ng Defects over Finding Failures Topics on Agile Tes-ng Colm O’hEocha, AgileInnova-on Topics We’ll Cover • 
• 
• 
• 
• 
• 
A (very) brief introduc-on to agile Cost of Delay – how delayed test creates waste Agile Testers Role – Requirements to Tests Test First – developing tests before implementa-on Test Automa-on – key points Exploratory Tes-ng – its all in the mind www.eurostarconferences.com 1 25/11/2014 COMPLEXITY & UNCERTAINTY Chao-c & Emergent • 
Requirements are Vague & Liable to Change • 
• 
• 
• 
Business/
System Requirements We don’t know what we don’t know Answers understood only in retrospect Probe and see what happens Experimental Management – reinforce desirable paWerns Poli%cs, Childs Birthday Party Complex • 
Simple We know Answer Self-­‐
Evident Command & Control Manufacturing • 
• 
Requirements are Clear & Stable • 
• 
Well understood & Predictable • 
• 
• 
• 
We know what we don’t know Mul-ple Right Answers Experts know Answers Value of making decision might outweigh wai-ng for right answer So3ware Development Solu-on/Technology Design & Implementa-on Many possible solu-ons, es-mates ‘finger in the air’ www.eurostarconferences.com Empirical Process Control OPEN-LOOP
Deterministic- Predictive
Set Target Execute CLOSED-LOOP
Empirical - Adaptive
Set Ini-al Target Adapt Execute Inspect AIM & FIRE FIRE & AIM www.eurostarconferences.com 2 25/11/2014 Scrum Cycle 1-­‐4 WEEK SPRINT Product Backlog Review Retrospec-ve Test, Develop, Test 1 DAY Product Backlog Refinement (Con-nuous) Daily stand-­‐
up mee-ng Sprint Planning Mee-ng Sprint Backlog VISION + TEAM (poten-ally releasable) PRODUCT + TEAM www.eurostarconferences.com From Plan-­‐Driven to Scrum NEED Requirements + RESOURCES Design Implementa-on Verifica-on PRODUCT + RESOURCES Produc-on PRODUCT + IMPROVED TEAM VISION + TEAM www.eurostarconferences.com Poten-ally Releasable Product Increments 3 25/11/2014 AGILE MANIFESTO + 1 “We are uncovering beWer ways of developing socware by doing it and helping others do it. Through this work we have come to value: • 
• 
• 
• 
• 
– Individuals and interac-ons over processes and tools – Working socware over comprehensive documenta-on – Customer collabora-on over contract nego-a-on – Responding to change over following a plan – Preven%ng defects over finding failures “That is, while there is value in the items on the right, we value the items on the lec more.” www.eurostarconferences.com 9.00 Cost of Delay Economies of Speed www.eurostarconferences.com 4 25/11/2014 Evolu-on: WaTerFall, Sprint+1, Mini-­‐Waterfall, Agile Problems with Non-­‐
Agile approaches: • 
• 
• 
Too late to fix bugs at end of sprint Unevenness in workload (Code/
Test) Cost of Delay www.eurostarconferences.com What is ‘Cost of Delay’? •  The difference in the value of doing something sooner vs. doing it later E.g. –  Releasing a Product this month vs. next month –  Delivering a feature 6 months acer it was specified –  Valida-ng a perceived user need before we build the feature vs. acer –  Fixing a bug today vs. in two weeks -me www.eurostarconferences.com 5 25/11/2014 What is ‘Cost of Delay’? •  The difference in the value of doing something sooner vs. doing it later E.g. Delayed Feature
Delayed Bug Fix
€ Benefits / Week Delayed Product
Finite benefits horizon, and reduced peak due to late delivery www.eurostarconferences.com Cost of Delay in Tes-ng SETUP V1 & V2 CREATE MERGE LOG RECREATE, FIND,FIX SETUP V1 & V2 SETUP V1 REVIEW FIND SETUP V2 & V3 LOG RECREATE, FIND,FIX SETUP V2 & V3 REVIEW FIND Time To Find/Fix/Retest (2 Sprints) SETUP V3 & V4 RECREATE, FIND,FIX LOG LOG SETUP V3 & V4 REVIEW FIND REVIEW FIND www.eurostarconferences.com 6 25/11/2014 Test & Debug – Typical Ac-vi-es (Re)Create Environments of differing versions for Test, Debug, Retest Log the Bug in a -cke-ng system, incl. how to recreate Manage the bug (review mee-ng, priori-se, assign) Find the Bug (among all the changes made since it was injected) Fix the bug (that I created weeks ago) Merge fix with all required branches (incl. managing any environment dependencies) Find and Fix Regression Issues Retest Regression Fixes Merge Regression Fixes with any branches created since the bug was injected Remembering where I was in the code, and what I was doing Retes-ng a failure found by another tester, as they are now on holidays … www.eurostarconferences.com 9.15 Exercise: Es-ma-ng the Cost of Delay On day 1, Susan injects a defect into her code. 3 days later, a3er adding further func%onality, she merges her code to the V2.23 branch. Jim, the tester, is busy retes%ng some fixes from the last release, and its not %ll day 13 he discovers the failure caused by Susan’s bug. He logs the incident and assigns it to the test manager, who, 5 days later, schedules it for debug. Another 5 days pass before Susan opens up and starts working on the %cket. Other developers have been working on this same code recently and she’s been working on V1.24, so it takes her a full day to find and fix the bug. 6 days later, Jim retests and finds the defect has been fixed. He closes the %cket. In groups of 5/6: • Calculate the Cost of Delay in the above scenario in man hours. What ac-vi-es would be typical. Include (re)crea-ng environments, logging bug details, regression tes-ng, merging code, managing branches, reviewing, priori-sing, assigning -ckets, etc. See the Worksheet. • Which ac-vi-es could be avoided if the en-re code/test/fix/retest cycle took just 1 day, rather than 30 days? • Assuming a ‘man hour’ costs €100, what’s the cost of the 30 day delay? www.eurostarconferences.com 9.35 7 25/11/2014 www.eurostarconferences.com User Stories Late Elabora-on and Specifica-on by Example www.eurostarconferences.com 8 25/11/2014 What are User Stories? •  User Centric – what’s important to your customer •  Story – The Power of Narra-ve –  We understand, remember and pay much more aWen-on to stories than facts –  Collabora-ng on Stories helps us focus on the users and business value •  User Stories help us define –  The Actor/Persona/Role (Who) –  The Ac-on/Func-onality (What) –  The Result/Benefit/Goal (Why) A brief statement of intent that describes something the system needs to do for the user www.eurostarconferences.com What are User Stories? •  User Centric – what’s important to your customer •  Story – The Power of Narra-ve • 
–  We understand, remember and pay much ore aWen-on to stories than ,m
on ’
a
s
facts ver t the n
o
o
–  Collabora-ng on Stories helps u Cs focus n, n on the users and business value or a
o
er f ificad
l
o
User Stories help us ed-­‐hefine ec
t! c to Sp utpu
a
l
P
o
–  The Actor/Persona/Role A ‘ PUT (Who) IN
–  The Ac-on/Func-onality (What) –  The Result/Benefit/Goal (Why) A brief statement of intent that describes something the system needs to do for the user www.eurostarconferences.com 9 25/11/2014 Valuable ‘Slices’ of Func-onality •  Stories should represent a ver-cal slice through the system and should be completed in one itera-on – and can therefore be tested in the same itera%on www.eurostarconferences.com User Story Example High Roaming Spend Warning (Data) As a billpay customer with data roaming enabled I want to be made aware if I’m approaching my threshhold so that I can decide whether to incur out-­‐of-­‐plan charges Acceptance Criteria: •  Ini-al warning at 80% threshhold. •  Follow Up at 95% threshhold and op-on to cap at threshhold. •  Warnings issued by SMS and email (where email address validated) •  If 80% threshhold achieved in first 25% of billing cycle, send ‘possible fraud alert’ to customer service queue. CONVERSATION (Customer, Tester, Developer): •  How many warnings? •  How should we issue the warning? •  What text do we use in the warnings? •  Do we force the user to acknowledge the warning? •  Can the user configure warning points? •  If they go above their threshhold, do we issue periodic reminders/
warnings? www.eurostarconferences.com 10 25/11/2014 A Specifica-on www.eurostarconferences.com User Story As a HomeOwner, I want to regularly trim my lawn so its neat and -dy. www.eurostarconferences.com 23 11 25/11/2014 Build Innova-on In As a <role> Problem Space so that <result> Innova-on Space Customers End Users Domain Experts Product Owners Uncertainty Ambiguity Conversa-on Social Objects Solu-on Space Developers Testers Architects UI/UX Designers I want to <ac-on> www.eurostarconferences.com Specifica-on By Example 24 Business Goal Deriving scope from goals Scope (User Stories) Specify collabora-vely Key Examples Refine the spec Specifica-on with Examples Automate Literally Executable Specifica-on Validate Frequently Living Documenta-on www.eurostarconferences.com 12 25/11/2014 What are Specifica-ons By Example? • 
• 
• 
• 
• 
• 
• 
Thin slices of system behaviour that deliver business value described as concrete examples that are poten-ally automatable without transla-on to create executable specifica-ons captured in live documenta-on. www.eurostarconferences.com Advantages of SBE •  Realis-c Examples contain Precise Informa-on & require Precise Answers •  Concrete examples s-mulate discussion & shared understanding •  They focus on business func-onality •  Edge cases flush out unseen requirements – but we can’t be exhaus-ve Tes%ng: Crea%ng Examples Checking: Running Examples www.eurostarconferences.com 13 25/11/2014 Using Examples to Elaborate and Illustrate User Stories Given <a precondi-on> When <an ac-on happens> Then <the following is expected> •  Each example describes a scenario or path through the func-onality in a user story. •  Examples can define several pre-­‐condi-ons and post-­‐condi-ons (mul-ple Given and Then sec-ons) for one or more ac-ons (When clauses). www.eurostarconferences.com A User Story with Mul-ple Examples User Story: In order to aWract customers As an online bookseller I want to give purchasers of 5 books or over free delivery Example: Given I have ordered 6 books When I go to online checkout Then I expect free delivery Given Order Size 5 When I go to online checkout 4 Then Delivery Free At cost 100 Free www.eurostarconferences.com 14 25/11/2014 Worked Example -­‐ Customer Type User Story: In order to retain customers As an online bookseller I want to reward returning customers with free delivery on orders of 3 books or more Example: Given I am a returning customer Given When I go to the Then And I order 3 books checkout When I go to the checkout Customer Order Delivery Type Size Then I expect free delivery New 1 At cost Repeat 2 At cost Repeat 3 Free Repeat 100 Free New 3 At cost www.eurostarconferences.com Trade System Example Source: Cian Bracken, AgileTour Dublin 2014 www.eurostarconferences.com 15 25/11/2014 Trade System Example Source: Cian Bracken, AgileTour Dublin 2014 Book is not adding any value here as the Primary Book aWribute is driving the behaviour User story is not formaWed correctly – it has two value statements In order to be testable and unambiguous, the content of each cell in the table needs to represent only one value www.eurostarconferences.com User Stories & Examples Edge C(ases
Examples Depth)
Typical Use E(Depth)
xcep-on
Examples Typical Use E(Depth)
xcep-on
Examples Typical Use (EDepth)
xample
Examples ‘SPINE’ Example
Examples (Depth)
Typical Use (EDepth)
xample
Examples Typical Use E(Depth)
xcep-on
Examples Typical Use E(Depth)
xcep-on
Examples Edge C(ases
Examples Depth)
User Story (Breadth) ’describe, demonstrate and develop’ www.eurostarconferences.com 16 25/11/2014 Examples vs. Scripts & Specifica-ons 1.  Set Thold 1=80%, THold2=95%, Cap=N, In-­‐
Plan Limit 1GB, Usage 799MB 2.  Create Billing Request for 1MB and post to the users account 3.  Check that Request is granted and Warning 1 Issued •  On reaching Thold1, Billing Requests are granted and Warning 1 issued. www.eurostarconferences.com Subjec-ve and difficult to automate. 10:30 Exercise -­‐ SBE User Story: As a billpay customer with data roaming enabled I want to be made aware if I’m approaching my Data Limit so that I can decide whether to incur out-­‐of-­‐plan charges Example: Given my data limit is 1GB, Warning Threshhold 900MB, current usage is 999MB When I aWempt to download data Then I get a warning ‘You will incur extra charges acer a further 1MB’ Given In-­‐Plan Data Limit Warning Threshold Data Usage 1GB 900MB 999MB When Then I go to download data Ac-on (s) You will incur extra charges acer a further 1MB In groups of 5/6, Create a table of Examples to Test this User Story (See Worksheet) www.eurostarconferences.com 17 25/11/2014 www.eurostarconferences.com Test First Whole Team Approach www.eurostarconferences.com 18 25/11/2014 Ever Heard This? •  When I see what it looks like, I’ll decide how to test it! •  How could I know how to test something un-l I’ve seen how its been implemented? Specify Design Code Test www.eurostarconferences.com Test First Tradi-onal Sequence Specify Design Code Test Test First Test Incremental Specify Design Code Test Specify Design Code Specify Design Specify Code Design Specify Test Code Design Specify Test Code Design Specify Test Test Test Code Design Incremental Test First Agile – All at Once Test Code Specify Code Design www.eurostarconferences.com 19 25/11/2014 Ever Heard This? •  When I see what it looks like, I’ll decide how to test it! •  How could I know how to test something un-l I’ve seen how its been implemented? Developers Interpreta-on Code Requirement (User Story) Testers Interpreta-on ≠ Tests www.eurostarconferences.com Ever Heard This? •  When I see what it looks like, I’ll decide how to test it! •  How could I know how to test something un-l I’ve seen how its been implemented? Requirement (User Story) Collabora-on (3 Amigos – Customer, Developer, Tester) results in a Shared Understanding ‘Single Source of Truth’: Tests, then Code www.eurostarconferences.com 20 25/11/2014 Test Driven Development www.eurostarconferences.com What’s your experiences with TDD? –  Do you write a new test before you create a new class or method? –  Do you refactor acer you get it working? –  Do your tests some-mes break when you refactor? –  Is that what you expected to happen? www.eurostarconferences.com 21 25/11/2014 extremeprogramming.org on Unit Tests •  TDD is usually synonymous with UNIT tes-ng •  “you should test all classes in the system. Trivial geWer and seWer methods are usually omiWed” •  “Unit tests enable refactoring…Acer each small change the unit tests can verify that a change in structure did not introduce a change in func-onality” www.eurostarconferences.com Refactoring – A Defini-on Mar'n Fowler •  noun: a change made to the internal structure of so3ware to make it easier to understand and cheaper to modify without changing its observable behavior •  verb: to restructure so3ware by applying a series of refactoring's without changing its observable behavior www.eurostarconferences.com 22 25/11/2014 What’s a ‘Unit’ in Unit Tes-ng? •  What Units are we talking about? –  Objects –  Private Methods –  External Interfaces –  APIs/Services }
}
Implementa-on OBJ4 OBJ2 OBJ1 Observable Behaviour OBJ3 •  Refactoring Changes Implementa-on OBJ5 –  For TDD a ‘Unit’ is not a unit of implementa-on –  It’s a unit of ‘Observable Behaviour’ www.eurostarconferences.com The TDD Cycle Step Ac-on Objec-ve RED Write a Test reflec-ng the behavior required – it Fails Learn – about the problem to be solved, the func-onality to be built GREEN Write the code to make the test pass – as fast (cheap) as you can Learn – about the solu-on needed to pass the tests – including excep-ons, edge cases, unreasonable use,… Design and re-­‐implement the code in the simplest, most parsimonious and most elegant way you can Implement – the cleanest, best designed code possible -­‐ based on what you’ve learned about the problem and the required solu-on (kludge, copy/paste, spaghe] code, whatever) REFACTOR www.eurostarconferences.com 23 25/11/2014 Some Objec-ves of TDD •  Firstly its about Learning –  Understanding the requirement/problem –  Understanding what func-onality is needed •  Second its about Development – 
– 
– 
– 
Implement the simplest thing that would work Allows us ‘refactor’ safely – reduced cost of change Enables ‘Emergent Design’ – design when we understand the problem Encourages callable, testable, loosely coupled code •  Thirdly its about Tes-ng –  We test Early – even before we implement –  We test Ocen (if we automate) – fast, on-­‐going feedback –  Reduces Tes-ng Cost of Delay: Prevent Defects over Finding Failures www.eurostarconferences.com •  Never write a single line of code unless you have a failing test •  Eliminate Duplica%on Some Objec-ves of TDD •  Firstly its about Learning T
es- the requirement/problem –  Understanding Con ng awnhat –  Understanding d func-onality is needed D
c
othat t
ch o the simplest pm
–  Implement would entw ork ther Ac-thing v
i
e
–  Allows us ‘refactor’ – reduced tos afely s Cocost oaf crhange e In
m
ax–i m
n-nwe understand ter-­‐ the problem –  Enables ‘Emergent Design’ design when u
i
o
s
e Lceoupled code usly depend
–  Encourages callable, testable, loosely arni
Thirdly its about Tes-ng ng feeding ent, off –  We test Early – even before we implement urreDevelopment eve
•  Second ea its about n
l
• 
–  We test Ocen (if we automate) – fast, on-­‐going feedback –  Reduces Tes-ng Cost of Delay: Prevent Defects over Finding Failures www.eurostarconferences.com •  Never write a single line of code unless you have a failing test •  Eliminate Duplica%on 24 25/11/2014 TDD -­‐ Summary •  TDD is Outside-­‐In: Create tests to test func-onality, not implementa-on (BB) – Beck –  Unit Tests test a ‘Unit’ of implementa-on •  Verifies the code implements the developers design –  TDD tests a ‘Unit’ of func-onality/behaviour •  Verifies the code implements the behaviour required •  BDD is the new ATDD is the new TDD •  TDD is about LearningèDevelopingèTes-ng www.eurostarconferences.com Test First and You In Groups of 5/6, discuss barriers and possible solu-ons to implemen-ng Test First in your organisa-on: –  Ge{ng testers involved before/concurrently with developers •  Gaining a shared understanding of requirements –  Ge{ng early agreement of acceptance criteria for each requirement •  Defining Test Criteria before deciding on the implementa-on –  Crea-ng Concrete, precise examples as ‘executable requirements’ and tes-ng using these (and automa-ng them) –  Including ‘Refactoring Time’ in the schedule 10 mins discussion, 10 mins sharing barriers/solu-ons www.eurostarconferences.com 25 25/11/2014 Test Automa-on Autonoma-on www.eurostarconferences.com Test Automa-on – An ‘An--­‐Paeern’ www.eurostarconferences.com 26 25/11/2014 The Automa-on Pyramid Manual Tests e.g. exploratory Things to Consider: •  Cost to Produce (TDD/BDD Collateral) •  Cost to Maintain GUI layer •  Ability to Test First/Early e.g. Selenium •  Quick to Run/Reduced EXE Overlap •  Ease of Pinpoin-ng Defect API/Service layer Automate at feature/
workflow level Automate at story level Acceptance Tests e.g. Fitnesse, Cucumber Automate at design level Unit/Component layer Developer Tests e.g. JUnit Derived from Mike Cohn www.eurostarconferences.com Based on Mike Cohn Checking vs. Testing
•  Checking •  is the process of making evalua%ons by applying algorithmic decision rules to specific observa%ons of a product. •  human or machine •  determinis%c •  a type of tes%ng Source: James Bach blog •  Tes-ng •  is the process of evalua%ng a product by learning about it through experimenta%on, which includes to some degree: ques%oning, study, modeling, observa%on and inference. •  a human endeavor •  subjec%ve •  includes checking www.eurostarconferences.com 27 25/11/2014 Warnings on Automa-on •  Tests find bugs, automa-on doesn’t •  Automa-on good for: –  Checking: Confirma-on, Machine decidable, determinis-c –  Regression, test prepara-on, non-­‐func-onal •  Can be overused – “if you have a hammer everything looks like a nail”. •  Understand Costs & Benefits (create, maintain, run, interpret, etc.) www.eurostarconferences.com Exploratory Tes-ng The Spy that came in from the Cold www.eurostarconferences.com 28 25/11/2014 Exploratory Testing
Hendrickson: a style of testing in which you explore the software
while simultaneously designing and executing tests, using
feedback from the last test to inform the next
Kaner: Simultaneous learning, design and execution, with an
emphasis on learning.
Cunninghan: “Because the program always runs, it is always ready
to be explored.”
www.eurostarconferences.com Why do exploratory tes-ng? •  Advantages: –  FAST: •  Minimal prepara-on (e.g. a charter/objec-ve, -mebox) •  Test Cases/Procedure not recorded, only incidents –  FOCUSED •  On a par-cular area to achieve coverage/confidence •  Risk based focus – re-­‐evaluated as tes-ng proceeds –  PRODUCTIVE •  Finds more defects per unit of effort than other methods •  Disadvantages: –  Un-­‐repeatable and not suitable for regression –  May require a skilled tester to execute, with knowledge of the product/domain –  Need to avoid tendency to revert to ‘unplanned’ tes-ng www.eurostarconferences.com 29 25/11/2014 Example: Session based tes-ng • 
• 
1-­‐2 hour sessions each star-ng with a charter and ending with a report/debrief Charter – iden-fies the goal of the session – 
– 
– 
– 
– 
– 
• 
What to test? How long for? Areas to explore Risky areas? Bug types to be inves-gated? Extreme values/tricky situa-ons? In general exploratory can help you get familiar with product & target hotspots – 
– 
– 
– 
– 
– 
Try extremes, anything you like to break product When you find a bug, explore around it Look for paWerns, clues to other bugs Document the test that caused the failure; document other tests that cover 'interes-ng' situa-ons Report/Log an incident Move on to next interes-ng area www.eurostarconferences.com Exercise: Exploratory Tes-ng •  Individually : •  Download coin flip free (iHandy) app if you have smart phone (or sit with someone who has) •  Start exploratory tes-ng to find bugs (Record on worksheet) •  Acer 5-­‐10 mins discuss what you did with the person beside you and see if you can come up with any more tests/bugs •  (15 mins total) •  Discussion on results • 
• 
What bugs did you find? How did you find them? What guided your ac-ons? Thanks to Ann-­‐Marie CharreW for the idea of this exercise www.eurostarconferences.com 30 25/11/2014 How agile changes test •  Whole Team Approach to Quality •  QA means ‘Quality Assistance’, ‘Ques-on Asker’ •  Specialised Skills, not Demarcated Responsibility (Competency vs. Role) •  Be Proac-ve: Can’t ‘Test In’ Quality, need to “Build In’ •  Design, Coding and Tes-ng are one process (‘All at Once’) •  Testers need to understand ‘the system’ – e.g. architecture •  ‘T Shaped Testers’: e.g. SQL, Environments, Perf Tes-ng, Data Prep, scrip-ng •  Role of Testers is to Help Prevent, not to Detect www.eurostarconferences.com Q&A – 
– 
– 
– 
Training Consul-ng Coaching Assessments www.agileinnova-on.eu www.eurostarconferences.com 31