Combining model-based testing and machine learning

Transcription

Combining Model-Based Testing
and Machine Learning
Roland GROZ
LIG, Université de Grenoble,
France
TAROT Summer School 2009
Acknowledgments

Muzammil Shahbaz
Keqin Li (now with SAP Research)
 Alexandre Petrenko (CRIM, Canada)


Doron Peled (seminal paper BBC), David Lee,
Khaled El Fakih…
2
Outline
MBT and software engineering trends
 Machine Learning for test automata
 Integration: MBT and ML, integration
testing
 Experiments and case studies
 Research directions

3
Traditional software cycle
Specification - Validation
Customer
Vendor
Implementation
Assembling fully controlled components
4
MBT in software development
Acceptance
tests
Model
System tests
Design
Integration tests
Unit tests
Implementation
5
Challenge: CBSE with 3rd party
Specification - Validation
Customer
Vendor
Implementation
Assemblingill controlled components
6
Component Based Software Engineering
Requirement
Analysis
High Level
System
Design
Components
Components
selection &
selection
Integration
Integrated System





Rapid Development
Reuse Components
Reduce cost
Flexibility
Ease of integration
7
Issues
Interactions
Behaviors
Validation
Understanding
the System of
Black Box Components
is a challenge
Integrated System
How do I perform system behavioral
analysis?
How do I identify integration problems ?
8
Soft. Engineering trends

MDE & MBT
 Growing
trend in some industries (e.g. embedded)
 Derive design, code and tests (MBT)
 Models = 1st class citizens

CBSE
 Dominant
growing trend
 Absence of (formal) models
 Pb maintaining spec <-> model
9
MDE & MBT in the reverse

MDE assumption
 Start
from model, formal spec
 Models = 1st class citizens

Test Driven Development (XP, Agile…)
 Tests
are spec: 1st class citizens
 Formal models ? No way !

Proposed approach
 Derive
models from tests, & combine with MBT
 Reconcile Test-Driven (or code-driven) dvt
with Models
10
Technical Goals

Reverse Engineering

Understanding the behaviors of the black box components



by deriving the formal models of the components/system
Can also serve documentation purposes (tests for doc)
System Validation


Being able to derive new systematic tests
Analyzing the system for anomalies / compositional problems

by developing a framework for integration testing of the system of
black box components
11
Approach
observes
Integrated
System

Learner
Tester
tests
(Validation
Techniques)
learn
s
Model(s)
(Finite State Machine)
es
deriv
What do we achieve?


Models (to understand the system more "formally")
System is validated for anomalies
12
Reverse engineer
Model
Requirements
Component
Sollicitations
generate
Scenario 1
Scenario 2
Partial, incremental and approximate characterization
13
13
Objections

Answers
Model is derived from
bugged components
 Derived
tests will
consider bug=feature

Incremental: stopping
criterion ?
14
Machine learning
15
Various types of Machine
Learning

Artificial Intelligence (& datamining)
 Ability
to infer rules, recognize patterns
 Learning from samples
 E.g. neural networks

Two major techniques
 Statistical
(bayesian) inference from collection of
data -> e.g. Weka tool in testing
 Grammatical inference of language
from theoretical computer science
16
Learning languages from samples
• Finding a minimum DFA (Deterministic Finite
Automaton) is NP-HARD
– Complexity of automaton identification from given data. [E. Gold 78]
• Even a DFA with no. of states polynomially larger
than the no. of states of the minimum is NPComplete
– The minimum consistent DFA problem cannot be approximated within any
polynomial. [Pitt & Warmuth 93]
• Probably Approximately Correct (PAC)
– A theory of the learnable. [L.G. Valiant 84]
Le
e
iv
s
s
Pa
g
n
i
a rn
17
17
Background Work

Passive Learning



"Learning from given positive/negative samples"
NP-Complete
Active Learning


"Learning from Queries" (Regular Inference)
Angluin's Algorithm L* [Angluin 87]


Learns Deterministic Finite Automaton (DFA)
in polynomial time
Applied in




Dana Angluin
Yale University
Black Box Checking [Peled 99]
Learning and Testing Telecom Systems [Steffen 03]
Protocol Testing [Shu & Lee 08]
…
18
Concept of the Regular Inference
(Angluin's Algorithm L*)
Input Alphabet Σ
ip
sh Σ *
r
be m
m
ro
m e es f
i
er
u
q
Black Box
Machine
pt
ce
c
a
The Algorithm L*
t
ec
j
e
/r
eq
uiv
(D alen
FA
c
co e qu
nje
e
ctu ries
r e)
co " y
un es
te " o
re
xa r
m
p le
Oracle
Final Minimum
DFA Conjecture
Assumptions:
 The input alphabet Σ is known
 Machine can be reset
Complexity : O( |Σ | m n² )
 |Σ | : the size of the input alphabet
 n : the number of states in the actual machine
19
 m : the length of the longest counterexample
Our Context of Inference
Test Strategies and heuristics
Input Alphabet Σ



Components having I/O behaviors
I/O are structurally complex
(parameters)
ip *
h
s
Σ sets
Formidable size bofer input
m
m f ro
me ies
er
u
q
System of
Communicating
Black Box
Components
Black Box
Machine
Learned
Models can be used to
eq
uiv tests to find
generate
(D que alen
FA
c
discrepancies
r
c ies e
on
The Algorithm L*
pt
e
c
ac
ct
e
j
/ re
Enhanced State Machine Models
Mealy Machines
jec
t ur
e)
co " y
un es
te " o
re
xa r
m
p le
Oracle
Final DFA
Conjecture
Parameterized Machines
More adequate for complex systems
DFAs may result in transition blow up
20
Preliminaries

Mealy Machine: M = (Q, I, O, δ, λ, q0)







Q : set of states
I : set of input symbols
O : set of output symbols
δ : transition function
λ : output function
q0 : initial state
Input Enabled

dom(δ) = dom(λ) = Q × I



Q = {q0, q1, q2, q3}
I = {a,b}
O = {x,y}
21
Mealy Machine Inference Algorithm
The Algorithm LM*
Input set I
ut m I *
p
t
ou s fro
ie
er
u
q
Black Box
Mealy Machine
The Algorithm LM*
t
tpu
u
o
s
ing
r
t
s
eq
uiv
al
(M
q
ea uer enc e
ly c ies
on
jec
t ur
e)
co " y
un es
te " o
re
xa r
m
ple
Oracle
Final Mealy
Conjecture
Assumptions:
 The input set I is known
 Machine can be reset
 For each input, the corresponding output is observable
22
Basic principles of Angluin’s algorithm (mod.)
Discriminating
sequences
a
b
x
x
y
x
b
x
x
aa
y
x
ab
x
x
ε
S (span seq)
for States
a
S•I
transitions
I={a, b}
Build queries
row.col
submit row.col ->
record output <for col
tr1: a/x
Conjecture
tr2: b/x
[a]
[ε ]
tr4: b/x
Observation Table
•ε is an empty string
tr3: a/y
Black Box Mealy Machine
Component
23
Mealy Machine Inference Algorithm LM* (1/5)
I={a, b}
EM
SM
a
b
ε
x
x
a
y
x
b
x
x
SM • I
Observation Table (SM,EM,TM)
Concepts:
 Closed
 Consistency
Black Box Mealy Machine Component
Output Queries:
s•e, s ∈ (SM∪ SM • I), e ∈ EM
•
= a/ x
24
Concept: Closed
I={a, b}
EM
SM
SM
a
b
ε
x
x
a
y
x
b
x
x
aa
y
x
ab
x
x
SM • I
Concepts:
 Closed : All the rows in SM • I must be equivalent to the rows in SM


Same behaviour = known state
Consistency
25
Making Conjecture
tr
tr1
SM
SM • I
tr2
3
tr4
I={a, b}
EM
a
b
ε
x
x
a
y
x
b
x
x
aa
y
x
x
x
ab
Counterexample:
ababbaa
component's response: x x x x x x y
conjecture's
response:
x Mx,TxM)x x x x
Observation
Table (SM,E
tr1: a/x
[ε ]
tr2: b/x
[a]
tr4: b/x
tr3: a/y
26
Processing Counterexamples
E
M
SM
SM • I
a
b
ε
x
x
a
y
b
a
b
ε
x
x
x
a
y
x
x
x
ab
x
x
aa
y
x
aba
x
x
ab
x
x
abab
ababb
x
x
x
x
ababba
x
x
ababbaa
y
x
aa
y
x
b
x
x
abb
x
x
abaa
x
x
ababa
ababbb
y
x
x
x
Counterexample: a b a b b a a
Method:
Add all the prefixes of the counterexample to SM
27
Concept: Consistency
a
a
b
ε
x
x
xy
a
a
y
x
yy
ab
x
x
xx
aba
x
x
xx
abab
ababb
x
x
xy
x
x
xx
ababba
x
x
xy
ababbaa
y
x
yy
aa
y
x
yy
b
x
x
xx
abb
x
x
xx
abaa
x
x
xy
ababa
ababbb
y
x
x
x
yy
xy
Concepts:
 Closed
 Consistency : All the successor rows of
the equivalent rows must also be
equivalent
Consistency check can be avoided
If all rows in SM are inequivalent
The rows in SM become equivalent
due to the method of processing
counterexamples in the table
28
New Method for Processing Counterexamples
(The Algorithm LM+)
Counterexample
All rows remain inequivalent
(inconsistency never occurs)
ab abbaa
a
b
ε
x
x
a
y
b
Add all the
suffixes to EM
a
b
aa
baa
bbaa abbaa
ε
x
x
xy
xxx
xxxy
xxxxx
x
a
y
x
yy
xxx
xxxx
yxxxx
x
x
b
x
x
xx
xxy
xxxx
xxxxy
aa
y
x
aa
y
x
xx
xxx
xxxy
xxxxx
ab
x
x
ab
x
x
yy
xxx
xxxx
yxxxx
before processing counterexample
after processing counterexample
29
Comparison of the two Methods
Total Output Queries in LM+ : 64
Final Observation Table (SM,EM,TM)
according to LM+
Total Output Queries in LM* : 86
Final Observation Table (SM,EM,TM)
according to LM*
30
Comparison of the two Methods
Total Output Queries in LM+ : 64
Output Queries in LM* : 86
Complexity of LTotal
*:
M
O( |I|² n m + |I| m n²)
Complexity of LM+:
O( |I|² n + |I| m n²)
I : the size of the input set
 n : the number of states in the actual machine
Observation
Table (SM,EM,TM)
 m
: the length of the longest counterexample

according to LM+
according to LM*
31
Experiments with
Edinburgh Concurrency Workbench
Examples
Size of
Input
set
No. of
States
No. of
No. of
Output
Output
Queries for Queries for
LM*
LM+
Reduction
Factor
6000
5000
R
3
6
48
48
0
ABP-Lossy
3
11
764
340
1,22
Peterson2
3
11
910
374
1,43
Small
6
11
462
392
0,18
VM
6
11
836
392
1,13
Buff3
3
12
680
369
1,24
Shed2
6
13
824
790
0,04
ABP-Safe
3
19
2336
764
2,1
TMR1
6
19
1396
1728
-0,19
R
Peterson2
ABP-Lossy
VMnew
4
29
3696
1404
0,86
CSPROT
6
44
4864
3094
0,67
4000
3000
Output
2000 Queries
1000
0
Small
VM
Buff3 Shed2
TMR1 VMnew
CSPROT
ABP-Safe
Examples
LM *
LM +
32
System Architecture



System of communicating Mealy Machine Components
Components are deterministic and input-enabled
System has External and Internal i/o interfaces
External interface is controllable
 External and Internal interfaces are observable


Single Message in Transit and Slow Environment
33
Learning & Testing Framework
Step 2(a): Compose Models
Step 3: Refine Models
Step 1: Learn Models
Product
[problem as
counterexample]
Step 2(b): Analyze Product
[compositional
problem]
Learned Models
No
Compositional
Problems
[problem
confirmed]
Product
Step 4:
Generate Tests from Product
Step 2(c):
Confirm Problem on System
Terminate
[no discrepancy]
Discrepancy trace
Step 5:
Resolve Discrepancy
(exception, crash, out of memory,...?)
[error found]
[discrepancy as
counterexample
]
34
Contributions
1.
Mealy Machine Inference

Improvements in the basic adaptation
from the Angluin's algorithm
1.
Parameterized Machine Inference
1.
Framework of Learning and Testing of integrated
systems of black box Mealy Machine components
2.
Tool & Case Studies (provided by Orange Labs)
35
The tool RALT
(Rich Automata Learning and Testing)
36
Case Studies
1.
Concurrency Workbench
2.
Air Gourmet
3.
Nokia 6131, N93, Sony Ericsson W300i

1.
Real
Systems
Experimented with Media Player
Domotics (Orange Labs' Smart Home Project)

Devices: Dimmable Light, TV, Multimedia Systems)
37
Air Gourmet
Goal: Learning & Testing the System
Mealy
Machines
Components
Size of No. of States No. of Errors
Input Set
Error Types
Passenger Check-in
3
4
0
-
Flight Reservation
5
7
3
NPE, IIE,
Date Parsing Exception
Meals Manager
4
8
4
NPE, IIE, IAE
NPE: Null Pointer Exception, IIE: Invalid Input Exception, IAE: Illegal Input Exception
38
Nokia 6131
Goal: Learning the behaviors of the Media Player
Music P1.wav
Music P2.wav
{ Play, Pause, Stop }
RALT
39
Nokia 6131 / N93 / Sony Ericsson W300i
Goal: Learning the behaviors of the Media Player
p1.start(p1.wav) / p1.STARTED
p1.stop() / p1.stopped
q1
q0
p2.start(p2.wav) / Exp
ED
p1.CLOS
/
()
e
s
lo
p1.c
q3
p2.stop() / p2.STOPPED
p2.close
/ p2
Nokia 6131
(PFSM)
.CLOSED
q2
p1.start(p1.wav) / Exp
q0
q1
p1.close() / p1.CLOSED
p2.close() / p2.CLOSED
Nokia 93 /
Sony Ericsson W300i
(PFSM)
40
Domotics
Goal: Learning the interactions of the devices
Acer AT3705-MGW
ProSyst Dimmable Light
Device
Philips
Streamium400i
Size of Input Set
No. of States
Time (minutes)
Light (ProSyst)
4
6
4
Media Renderer (Philips)
5
5
16
Domotics (Interaction Model)
9
16
30
41
Conclusion

Reverse Engineering Enhanced State Models


Improved Mealy machine inference

Framework for Learning & Testing of
Integrated Systems of Black Box Mealy machines

The tool RALT that implements the reverse engineering
and integration testing framework

Experiments with real systems in the context of Orange
Labs.
42
Perspectives

Work in Progress
 Approach
for detecting sporadic errors
 Learning Nondeterministic Automata

Future Work
 Test
Generation for Model Refinements
 Framework for PFSM models
43
Behind the Curtain
44
DoCoMo: A Motivational Example in Orange Labs
We
b
uploads score on the
Docomo's server
(Web Connection)
Hidden Behavior:
(Game)
User's scores are uploaded
to the server through web
Hidden Interaction:
The Game component interacts with
the Web component for connection
45
Mealy Machine Quotients

Let Φ be a set of strings from I then


the states s1 and s2 are Φ-equivalent if they produce same
outputs for all the strings in Φ
A quotient based upon Φ-equivalence is called Φ-quotient
Φ = {a, b, ab, ba, bb, bba}
Mealy Machine M
q0 and q2 are Φ-Equivalent
q1 and q3 are Φ-Equivalent
Φ-quotient of M
46
Relation between
the Conjecture and the Black Box Machine
EM-Quotient
EM
SM
Conjecture from the
SM • I
Closed (and Consistent)
Black Box Mealy Machine
47
Objections
Answers
48
Contributions
1.

1.
1.
2.
49
Parameterized Finite State Machine (PFSM)
50
PFSM Algorithm LP* (a view)
I = {a,b}, DI = N ∪ {⊥}
EP
b
a
SP
R
ε
(⊥,s ⊗ ⊥)
(1,s ⊗ ⊥) (6,t ⊗ 6)
a
(⊥,s ⊗ ⊥)
(1,s ⊗ ⊥) (6, t ⊗ 36)
b⊗ 1
(⊥,s ⊗ ⊥)
b⊗ 6
(⊥,s ⊗ ⊥)
(1,s ⊗ ⊥) (6, t ⊗ 36)
(1,s ⊗ ⊥) (6, t ⊗ 6)
aa
ab ⊗ ⊥ 1
(⊥,s ⊗ ⊥)
(1,s ⊗ ⊥) (6, t ⊗ 36)
(1,s ⊗ ⊥) (6, t ⊗ 36)
ab ⊗ ⊥ 6
(⊥,s ⊗ ⊥)
(1,s ⊗ ⊥) (6, t ⊗ 6)
(⊥,s ⊗ ⊥)
u(x≤ 1)/s
Observation Table (SP,R,EP,TP)
tr1: a/s
[ε ]
tr3: b({6})/t({6})
Black Box PFSM Component
tr2: b({1})/s
tr6: b({6})/t({36})
[a]
tr4: a/s
tr5: b({1})/s
51
Contributions
1.

1.
1.
2.
52

Combining model-based testing and machine learning

Transcription

Similar documents

Angiosperm Latin Name: Cyperus involucratus Common Name

Angiosperm Latin Name: Callisia fragrans Common Name: Basket

Bromeliaceae

Anechoic Recordings - Audio Group

151st Edition

Charles Robert Darwin 1809-1882

For advance bookings or queries please call our

SiteDircet 2016-04-06

3.02 RL O`Mealy Biography

Tropical Plant Identifications

Infinite Edge ParOOon Models for Overlapping Community