Combinatorial Test Design

Combinatorial Test Design
Juergen Heymann
August 2013
The Problem of Testing
©  2012 SAP AG. All rights reserved.
2
"Combinatorial Test Design"
… is a 'black box' test technique
views function of software from the outside
combines several test design techniques
to deal well with the problem of 'all combinations'
©  2012 SAP AG. All rights reserved.
3
Step 1: Finding the Use Cases / Test Cases
Assume you have a certain scope of functionality to test…
Step 1. Find 'use cases' / test cases
Ÿ  Ask "What can I do here?" ('user action')
Ÿ  Each action triggers a behavior / function
p1,p2,p3
–  'Enter' / action-buttons but also menu items,
tool bar clicks, …
–  API: call of a service / function / method / …
Ÿ  Each action / 'use case' becomes a test
case
F8
menu1
p4
tool3
What can I do here?
This should give you a complete list of possible actions / test cases that
cover the test object – and have to be tested (test = setup-action-verify)
©  2012 SAP AG. All rights reserved.
4
Step 2: Find the parameters of the test / action
Step 2. Find parameters of the test case
Differentiate between
Ø  Dummy Data: just needed to run tests, not varied
in testing, e.g. customer address, article name
Ø  Test Model Relevant Parameters: affect the
behavior of code under test (CUT)
Test Relevant Parameters can be found in
Ø  Direct Input e.g. parameters in API, on screen
Ø 
Application Configuration
Ø 
Object Data in DB
Exec
Find parameters
This gives you the set of parameters that are relevant for this test case
©  2012 SAP AG. All rights reserved.
5
Parameter Oriented Test Design
These 4 test design methods helps us to cover the parameter
space in a smart way to find more defects with less effort!
The methods focus on functional correctness.
©  2012 SAP AG. All rights reserved.
6
1: Identify Equivalence Classes
Example: Insurance Handling
©  2012 SAP AG. All rights reserved.
7
2: Analyze Boundary Values
… to be considered in edge cases
©  2012 SAP AG. All rights reserved.
8
Handling Multiple Parameters
©  2012 SAP AG. All rights reserved.
9
3: Decision Tables
Example: Account Opening
©  2012 SAP AG. All rights reserved.
10
But …
Many functions/test cases have more than 4 parameters
Let’s focus on a method that supports this case very well …
©  2012 SAP AG. All rights reserved.
11
4: All-Pairs Testing
©  2012 SAP AG. All rights reserved.
12
Use All-Pairs Testing
… to test big parameter spaces efficiently
©  2012 SAP AG. All rights reserved.
13
An All-Pairs Example
©  2012 SAP AG. All rights reserved.
14
All-Pairs Tool Support
You cannot generate 'minimal pair coverage' by hand!
Ÿ  We use the PICT engine from Microsoft + XLS as wrapper for convenience
Tool DEMO
©  2012 SAP AG. All rights reserved.
15
A Case Example: Combining all Techniques
EPOS Mix&Match
All these fields are irrelevant
for model (à 'other test data')
Single field errors need to be
tested, e.g. date in the future etc.
BUT: Error checks are outside
of All-Pairs model scope
First set of relevant parameters:
MMCriteria: Unique, Match, Any
CompleteMM: 0,1
ProratedMM: 0,1
AllowRetMM: 0,1
©  2012 SAP AG. All rights reserved.
16
Combining all techniques 2
ItemKey: must exist, but is
single-field error condition
è ignore in model
EligibilityRule1: ItemKeyExisting, NoDiscToAll, AllItems, Cosmetics, VenderSelection, UnitChargeDiscount
Operation1: Greater|GreaterEQ, EQ, LessEQ|Less
# aliasing : we consider Greater|GreaterEQ 'almost the same' / equivalent; they are iterated but generate no addtl. pair coverage
Qty : 2
# Qty: states how many items of the same type (item key) need to be bought before the special price / … applies.
# Qty=2 covers the case that the program identifies at least 2 separate items to belong to the 'special' (as 'condition items').
# You need the proper test data to 'trigger' this quantity
Discount1: DollarOff, PctOff, SellPrice
# Amount: ignored for model, just use any test data; error checks: amount off > price, pct off > 100, Sell price > orig price
ApplyTo1: Item, Group
# constraint: LowestPriceItem only used with EligibilityRules; item/group only with item
LowestPriceItems1: 0,1
# count semantics?
Count1: 0, 1, 10
©  2012 SAP AG. All rights reserved.
17
All-Pairs Summary
All-Pairs in Practice
Ÿ  Some advanced modeling concepts needed: Conditionals, Alias Groups
Ÿ  Models generate 'strange cases' where one sometimes must think hard
what is supposed to happen
ROI
Ÿ  On average, All-Pairs found 4x the defects-per-test-effort compared to
'manual test design'
Links
Ÿ  Overview of AllPairs / Pairwise Tools (incl. PICT):
http://www.pairwise.org/tools.asp
©  2012 SAP AG. All rights reserved.
18
Thank you
Contact information:
Juergen Heymann
[email protected]
Decision Tables 5
Redundancy and Inconsistency
ECP
BVA
DT
AP
State
Decision Tables can be Redundant and/or Inconsistent
Redundant := More than 1 rule defined for the same set of condition values
Inconsistent := Inconsistent actions defined for the same set of condition values
Ÿ  Needs more than 1 rule defined for same set of condition values
Redundancy: # of actually
defined cases is greater
than 'Full case count'
Tool shows redundancy:
both rules cover n,y,n case
Inconsistency:
Inconsistent actions
specified for n,y,n
If you avoid redundancy (tool shows this) you avoid inconsistency
©  2012 SAP AG. All rights reserved.
20
All-Pairs Testing Exercise 2:
Test FINDSTR (a command line tool)
ECP
BVA
DT
AP
State
Develop All-Pairs test model for FINDSTR
but ignore regular expressions for 'strings' to be searched for
Searches for strings in files.
FINDSTR [/B] [/E] [/L] [/R] [/S] [/I] [/X] [/V] [/N] [/M] [/O] [/P] [/F:file]
[/C:string] [/G:file] [/D:dir list] [/A:color attributes] [/OFF[LINE]]
strings [[drive:][path]filename[ ...]]
/B
/E
/L
/R
/S
Matches pattern if at the beginning of a line.
Matches pattern if at the end of a line.
Uses search strings literally.
Uses search strings as regular expressions.
Searches for matching files in the current directory and all
subdirectories.
/I
Specifies that the search is not to be case-sensitive.
/X
Prints lines that match exactly.
/V
Prints only lines that do not contain a match.
/N
Prints the line number before each line that matches.
/M
Prints only the filename if a file contains a match.
/O
Prints character offset before each matching line.
/P
Skip files with non-printable characters.
/OFF[LINE] Do not skip files with offline attribute set.
/A:attr
Specifies color attribute with two hex digits. See "color /?"
/F:file
Reads file list from the specified file(/ stands for console).
/C:string Uses specified string as a literal search string.
/G:file
Gets search strings from the specified file(/ stands for console).
/D:dir
Search a semicolon delimited list of directories
strings
Text to be searched for.
[drive:][path]filename
Specifies a file or files to search.
©  2012 SAP AG. All rights reserved.
21
All-Pairs Exercise 2: FINDSTR (Windows)
Commented solution (Microsoft)
All-pairs model for FINDSTR parameters
Note: regular expression arguments (for strings to be found) not shown here
Trailing commas do not generate 'empty' value
This is a sample solution from Microsoft
©  2012 SAP AG. All rights reserved.
22
All-Pairs Exercise 2: FINDSTR (Windows)
Approach2: Strict outside view
All-pairs model for FINDSTR parameters
Note: We do consider FINDSTR strictly from the outside, i.e. we model whatever the user can
enter as options – not what we think the software should do
Only when options are truly mutually exclusive (you get error message), we put them as
alternatives in the model (à parameter with alternative values, not constraints)
This is a sample solution from us (SAP)
©  2012 SAP AG. All rights reserved.
23
© 2013 SAP AG. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose
without the express permission of SAP AG. The information contained herein may be
changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary
software components of other software vendors.
Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered
trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x,
System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power
Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA,
pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP,
RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli,
Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation.
Linux is the registered trademark of Linus Torvalds in the United States and other countries.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered
trademarks of Adobe Systems Incorporated in the United States and other countries.
Oracle and Java are registered trademarks of Oracle and its affiliates.
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps,
Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync,
Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are
trademarks or registered trademarks of Google Inc.
INTERMEC is a registered trademark of Intermec Technologies Corporation.
Wi-Fi is a registered trademark of Wi-Fi Alliance.
Bluetooth is a registered trademark of Bluetooth SIG Inc.
Motorola is a registered trademark of Motorola Trademark Holdings LLC.
Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.
SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer,
StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as
their respective logos are trademarks or registered trademarks of SAP AG in Germany and
other countries.
Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal
Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services
mentioned herein as well as their respective logos are trademarks or registered trademarks
of Business Objects Software Ltd. Business Objects is an SAP company.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin
are trademarks or registered trademarks of Citrix Systems Inc.
Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase
products and services mentioned herein as well as their respective logos are trademarks or
registered trademarks of Sybase Inc. Sybase is an SAP company.
HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®,
World Wide Web Consortium, Massachusetts Institute of Technology.
Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered trademarks of
Crossgate AG in Germany and other countries. Crossgate is an SAP company.
Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C,
Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc.
All other product and service names mentioned are the trademarks of their respective
companies. Data contained in this document serves informational purposes only. National
product specifications may vary.
IOS is a registered trademark of Cisco Systems Inc.
RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry
Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App
World are trademarks or registered trademarks of Research in Motion Limited.
©  2012 SAP AG. All rights reserved.
The information in this document is proprietary to SAP. No part of this document may be
reproduced, copied, or transmitted in any form or for any purpose without the express prior
written permission of SAP AG.
24
© 2013 SAP AG. Alle Rechte vorbehalten.
Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem
Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung
durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne
vorherige Ankündigung geändert werden.
Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps,
Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync,
Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik und Android sind
Marken oder eingetragene Marken von Google Inc.
Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können
Softwarekomponenten auch anderer Softwarehersteller enthalten.
INTERMEC ist eine eingetragene Marke der Intermec Technologies Corporation.
Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight und Visual Studio sind
eingetragene Marken der Microsoft Corporation.
Bluetooth ist eine eingetragene Marke von Bluetooth SIG Inc.
IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x,
System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power
Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA,
pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP,
RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli,
Informix und Smarter Planet sind Marken oder eingetragene Marken der IBM Corporation.
Linux ist eine eingetragene Marke von Linus Torvalds in den USA und anderen Ländern.
Adobe, das Adobe-Logo, Acrobat, PostScript und Reader sind Marken oder eingetragene
Marken von Adobe Systems Incorporated in den USA und/oder anderen Ländern.
Oracle und Java sind eingetragene Marken von Oracle und/oder ihrer Tochtergesellschaften.
UNIX, X/Open, OSF/1 und Motif sind eingetragene Marken der Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame und MultiWin
sind Marken oder eingetragene Marken von Citrix Systems, Inc.
HTML, XML, XHTML und W3C sind Marken oder eingetragene Marken des W3C®,
World Wide Web Consortium, Massachusetts Institute of Technology.
Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C,
Retina, Safari, Siri und Xcode sind Marken oder eingetragene Marken der Apple Inc.
IOS ist eine eingetragene Marke von Cisco Systems Inc.
RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry
Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook und BlackBerry App
World sind Marken oder eingetragene Marken von Research in Motion Limited.
©  2012 SAP AG. All rights reserved.
Wi-Fi ist eine eingetragene Marke der Wi-Fi Alliance.
Motorola ist eine eingetragene Marke von Motorola Trademark Holdings, LLC.
Computop ist eine eingetragene Marke der Computop Wirtschaftsinformatik GmbH.
SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer,
StreamWork, SAP HANA und weitere im Text erwähnte SAP-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der
SAP AG in Deutschland und anderen Ländern.
Business Objects und das Business-Objects-Logo, BusinessObjects, Crystal Reports,
Crystal Decisions, Web Intelligence, Xcelsius und andere im Text erwähnte BusinessObjects-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken
oder eingetragene Marken der Business Objects Software Ltd. Business Objects ist ein
Unternehmen der SAP AG.
Sybase und Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere und weitere im Text
erwähnte Sybase-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind
Marken oder eingetragene Marken der Sybase Inc. Sybase ist ein Unternehmen der
SAP AG.
Crossgate, m@gic EDDY, B2B 360°, B2B 360° Services sind eingetragene Marken
der Crossgate AG in Deutschland und anderen Ländern. Crossgate ist ein Unternehmen der
SAP AG.
Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen
Firmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen.
Die in dieser Publikation enthaltene Information ist Eigentum der SAP. Weitergabe und
Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und
in welcher Form auch immer, nur mit ausdrücklicher schriftlicher Genehmigung durch
SAP AG gestattet.
25