IEMAS-aided exploration of sensitivity analysis methods - PL-Grid

IEMAS-aided exploration of sensitivity analysis
methods implemented in MATLAB and R
Włodzimierz Funika1,2, Paulina Żak1,2, Grzegorz Łaganowski1,2
University of Science and Technology, al. Mickiewicza 30, 30-059 Kraków,
2ACK Cyfronet AGH, ul. Nawojki 11, 30-950 Kraków
1AGH
CGW’15 26-28.10.2015 Kraków
Agenda
• Sensitivity Analysis: what it is about
• IEMAS
• Scalarm platform
• Ready-made tools in R and MATLAB
• Experiments and results
• Conclusions and future work
2
Sensitivity Analysis
3
1. Purpose
2. Core
methodology
IEMAS as the test model
Immunological Evolutionary Multi-agent System for
given optimization problem
"We know that they work, but we do not know why”
T. Back, U. Hammel and H.-P. Schwefel
https://www.age.agh.edu.pl/
4
Scalarm as data-farming platform
5
Visualisation methods in Scalarm
6
Search for tools
Main criteria:
1. Freeware
2. Ease of use
3. Computation results coming from external
platform
4. Built-in visualization methods
7
Ready-made tools in R and
MATLAB
1. “Sensitivity” package in R
2. SAFE Toolbox in MATLAB
(SAFE Toolbox due to courtesy of)
F. Pianosi, F. Sarrazin, Th. Wagener
http://bristol.ac.uk/cabot/resources/safe-toolbox/
8
Research of methods
Required features:
Main:
Method should allow for initial grading of
impact of parameters.
Secondary:
●
●
●
●
independent from monotonicity of model
accepts discrete values
capability of computing variety of sample sizes
computionally non intensive
9
Short review of methods
● Morris method (EET)
○ Global screening method,
○ One-step-at-a-time method (OAT)
● Sobol method (VBSA)
○ Variance based method,
○ Main effect (first-order),Total effect (total order)’
● FAST
○ Fourier series to represent the model in the frequency
domain, poor for discrete inputs (Saltelli et al., 2000)
10
Experiment specifications and
parametric space
10 Inputs:
● reproduction minimum = [ 300 - 1000 ]
● newborn energy = [ 0 - 1000 ]
● transferred energy = [ 0 - 1000 ]
● amount of iterations = [ 0 - 10 ]
● immunological time span = [ 1 - 1000 ]
● bite transfer = [1 - 200]
● mahalanobis = [0.8 - 5]
● immunological maturity = [1 - 200]
● good agent energy = [1 - 1000]
● evaluation method = “rastrigin” | “schwefel”
3 outputs:
● time elapsed
● iemas fitness
● fitness calls
11
Package “sensitivity” in R in use
1/4
Morris
OAT design
1000 samples
µ - average direction of
dependency of output and
feature
µ* - strength of dependency
σ - the degree of nonlinearity
of dependency of output and
feature
12
Package “sensitivity” in R in use
2/4
Sobol
Normal
1380 samples
main indices - influence of only one feature
while others being fixed
total indices - summary of main and
interaction indices
interaction indices - take into account
interactions of feature with others.
13
Package “sensitivity” in R in use
3/4
Sobol
Martinez
330 samples
main indices - influence of only one feature
while others being fixed
total indices - summary of main and
interaction indices
interaction indices - take into account
interactions of feature with others.
14
Package “sensitivity” in R in use
4/4
Sobol
Mara
330 samples
main indices - influence of only one feature
while others being fixed
total indices - summary of main and
interaction indices
interaction indices - take into account
interactions of feature with others.
15
SAFE Toolbox in MATLAB in use
1/5
Morris method (EET)
Sample size: 550
16
SAFE Toolbox in MATLAB in use
2/5
Morris method (EET)
Sample size: 836
17
SAFE Toolbox in MATLAB in use
3/5
Sobol method (VBSA)
Sample size: 1200
18
SAFE Toolbox in MATLAB in use
4/5
Sobol method (VBSA)
Sample size: 1200
19
SAFE Toolbox in MATLAB in use
5/5
FAST
Sample size: 2641
20
Comparison of implementations
Method
impl./factor
Simplicity
21
Visualization
Consistency
Depth
Sampling
size
Morris - R
3
4
4
2
large
Morris - SAFE
4
5
4
3
medium
Sobol - R
2
2
3
5
medium
Sobol - SAFE
5
N/A
3
4
large
FAST
2
N/A
2
3
large
Visualization - Are charts are clear and easy to interpret?
Consistency - Are results the same in multiple simulations?
Simplicity - How much effort does it take to adapt external model and set fitting
attributes?
Depth - How much information is provided with results about interactions between
parameters?
Sample size - What size of experiment did we deal with?
Conclusions
22
Our Choice:
SAFE implementation
of Morris method
Reasons:





estimates first order effects
easy to use
efficient
can measure the level of nonlinearity of dependency
clear visualization
Future work 1/2
Use of “sensitivity” package in Scalarm
with visualization in HighCharts
23
Future work 2/2
Morris visualization
24
25
Thank you for
attention!
Questions
26