parameter tuning of svm by bio

Comments

Transcription

parameter tuning of svm by bio
PARAMETER TUNING OF SVM BY
BIO-INSPIRED SEARCH
ALGORITHMS
Ph.D.Candidate: André Luis Debiaso Rossi
Advisor: André C.P.L.F. de Carvalho
Institute of Mathematics and Computer Science –University of Sao Paulo
Co- advisor: Carlos Soares
Faculdade de Economia – University of Porto
November /2010
:
Apoio
Outline
2







Introduction
Machine learning and classification
Support Vector Machines and parameter
tuning
Bio-inspired algorithms
Experimental Method
Experiments
Final remarks
Introduction
3

Machine Learning (ML)
 Area
based on Computational Intelligence, Statistics,
neuroscience, etc.
 The
goal is to design systems that are able to
automatically acquire knowledge and improve with
their experience
 Applications:

Speech Recognition, Bioinformatics, Robot motion,
Medicine, Management, Engineering, Science, Pattern
Recognition
Supervised Learning: Training Data
4
Outlook
Temperature
Humidity
Wind
Play Tennis
Sunny
Hot
High
Weak
No
Sunny
Hot
High
Strong
No
Overcast
Hot
High
Weak
Yes
Rain
Mild
High
Weak
Yes
Rain
Cool
Normal
Weak
Yes
Rain
Cool
Normal
Strong
No
Overcast
Cool
Normal
Strong
Yes
Sunny
Mild
High
Weak
No
Sunny
Cool
Normal
Weak
Yes
Rain
Mild
Normal
Weak
Yes
Overcast
Hot
Normal
Weak
Yes
Rain
Mild
High
Strong
No
Supervised Learning
5
Training
data
Learning
algorithm
Model
New examples:
Outlook
Temperature
Humidity
Wind
Play Tennis
Sunny
Mild
Normal
Weak
?
Overcast
Hot
High
Strong
?
Sunny
Mild
High
Strong
?
Machine Learning: parameters
6

Machine Learning (ML) Techniques
 Parameter
values have to be set by user
 Artificial
Neural Networks (ANN): learning rate,
number of nodes in the hidden layer, etc
 Most
of the ML techniques are sensitive to the
choice of its parameter values (Kohavi and John,
1995)
 Parameter
values  performance
Machine Learning: parameters
7



Default parameter values
Parameter values should be set for a specific
data set (Kohavi and John, 1995)
Parameter tuning
 Exhaustive
search methods or by trial-and-error
 Increase of the use of bio-inspired search
algorithms
Related Work
8

There are studies using different bio-inspired search
algorithms to tune the parameters of ML techniques:
 Genetic
Algorithms for ANN (Leung et al., 2003)
 Particle
Swarm Optimization for SVM (Souza et
al., 2006)

Genetic Algorithms for Decision Trees (Stiglic and Kokol,
2006)

Genetic Algorithms for SVM parameter tuning (Lorena and
Carvalho, 2008)
Aim
9

To compare the influence of the tuning process
in the SVM performance using differents bioinspired search algorithms

To compare bio-inspired algorithms and
baseline methods for the tuning process in
SVM applied to classification problems
SUPPORT VECTOR
MACHINES
Support Vector Machines
11

Classification problems: SVM looks for the
hyperplane that separates data from two
different classes maximizing the separation
margin
Support Vector Machines
12
f ( x) 
 y   ( x )   ( x)  b
xi SV
i
i
i
Support Vector Machines
13
K ( xi , x j )   ( xi )   ( x j )
2
K ( xi , x j )  exp( xi  x j )
SVM Parameter Tuning
14


Let SVMγ be the model induced using the
parameters γ=(Cγ,σγ)
Let yj be the real class and ŷj be the class
predicted by the SVMγ for the j-th example
NE
1
errˆor ( )  1 
NE
 f (E )
1 if
f (E j )  
0 if
yˆ j  y j
yˆ j  y j
j 1
j
Bio-inspired Search Algorithms
15


Design of algorithms that reproduce, in an
elementary way, some biological mechanisms
and processes
Bio-inspired search algorithms
(metaheuristics)
 Ant
Colony Optimization (ACO)
 Particle Swarm Optimization (PSO)
 Genetic Algorithms (GA)
 Clonal Selection Algorithm (CSA)
EXPERIMENTAL
METHOD
Parameter Tuning Process by a
Bio-inspired Search Algorithm
17
Test of the Parameter Values
Found by the Search Algorithms
18
Sampling Method for SVM
Parameter Tuning
19
Discretized Search Space
20

The learning algorithm is performed NO X NI
times for each pair of parameter values
 It

is computationally expensive!
Discretized search space (fraction of 0.5):
{-5, -4.5, …., 14.5, 15} - (41 values)
 σ: {-15, -14.5, ...., 2.5, 3} - (37 values)
 C:

1517 combinations of C and σ
Baseline Approaches
21

Random
 It
generates s pairs of random values for the
parameters
 s is approximately the number of solutions
evaluated by the bio-inspired algorithms

Default values of the LibSVM library
 C=1
 σ=
1/|A|
 |A|
is the number of attributes of the data
Experimental Setup
22


Support Vector Machines with Radial Basis
Function (RBF) kernel
Search space of the parameters:
 Cost
(C): [-5,15]
 Radius of the RBF kernel (σ): [-15,3]

Codification
 Exponents
 Individual
values of base two:
that has par=(3, -5): C=23 e σ =2-5
Experimental Setup
23





Number of iterations: 50
Number of individuals: 5
The algorithms could test approximately 250
solutions
Thus, the random method generates 250
different solutions
The search algorithms and the random method
are performed 10 times for each training set
RESULTS
Experimental Results
25

Pearson correlation between the validation and
test error rates
Data set
Correlation
Colon
0.630
Glioma
0.681
Leukemia
0.922
Pancreas
0.905
Leukemia2
0.959
Lung
0.665
Australian
0.972
Pima
0.876
Segment
0.998
Vehicle
0.987
26
Discrete Space Error Rates
27
Error rates for the Australian data set - fold 1
Final Remarks
28

The default parameter values are robust.
However, these values were not suitable for
some data sets
 Many
combination of values result in low error
rates

The parameter values of the bio-inspired
algorithms were fixed
 The
tuning of these parameters could improve
their results
References
29

Rossi, A. L. D. & Carvalho, A. C. P. L. F. (2008). Bio-inspired
optimization techniques for SVM parameter tuning. In Proceedings of
10th Brazilian Symposium on Neural Networks, pag. 435-440. IEEE
Computer Society.

Rossi, A. L. D., Carvalho, A. C. P. L. F., & Soares, C. (2008). Bio-inspired
parameter tunning of MLP networks for gene expression analysis. In
Proceedings of 8th International Conference on Hybrid Intelligent
Systems, pag. 57-62. IEEE Computer Society.

Kohavi, R. & John, G. H. (1995). Automatic parameter selection by
minimizing estimated error. In Prieditis, A. & Russel, S., editors, Proceedings
of the Twelfth International Conference on Machine Learning, pag. 304-312,
San Francisco, CA. Morgan Kaufmann.

Leung, F. H. F., Lam, H. K., Ling, S. H., & Tam, P. K. S. (2003). Tuning of the
structure and parameters of a neural network using an improved genetic
algorithm. IEEE Transactions on Neural Networks, 14(1):79-88.

Hoste, V. & Daelemans, W. (2005). Comparing learning approaches to
coreference resolution. There is more to it than 'bias'. In Proceedings of the
Workshop on Meta-Learning, pag. 20-27.
References
30

Lorena, A. C. & Carvalho, A. C. P. L. F. (2006). Multiclass SVM design and
parameter selection with genetic algorithms. In Proceedings of the Ninth
Brazilian Symposium on Neural Networks, pg. 23. IEEE Computer Society.

Castro, L. N. & Von-Zuben, F. (2002). Learning and optimization using the
clonal selection principle. IEEE Transactions on Evolutionary Computation,
6(3):239-251.

Huang, C.-L. & Wang, C.-J. (2006). A GA-based feature selection and
parameters optimization for support vector machines. Expert Systems with
Applications, 31(2):231-240.

Gao, L., Zhou, C., Gao, H.-B., & Shi, Y.-R. (2006). Credit scoring model
based on neural network with particle swarm optimization. In Proceedings of
the Second International Conference on Advances in Natural Computation,
pag. 76-79. Springer-Verlag.

Souza, B. F., Carvalho, A. C. P. L. F., Calvo, R., & Ishii, R. P. (2006).
Multiclass SVM model selection using particle swarm optimization. In
Proceedings of the Sixth International Conference on Hybrid Intelligent
Systems, pag. 31, Washington, DC, USA. IEEE Computer Society.
References
31

Stiglic, G. & Kokol, P. (2006). Evolutionary tuning of combined multiple
models. In Gabrys, B., Howlett, R. J., & Jain, L. C., editors, KES (2),
volume 4252 of Lecture Notes in Computer Science, pag. 1297{1304.
Springer.

Elshamy, W., Emara, H. M., & Bahgat, A. (2007). Clubs-based particle
swarm optimization. In IEEE Swarm Intelligence Symposium, pag. 289-296.

Dorigo, M., 1992. Optimization, Learning and Natural Algorithms (in Italian).
Ph.D. thesis, Dipartimento di Elettronica, Politecnico di Milano, Italy.

Demsar, J. (2006). Statistical comparisons of classiers over multiple data
sets. Journal of Machine Learning Research, 7:1-30.

Nadeau, C. & Bengio, Y. (2003). Inference for the generalization error.
Machine Learning, 52(3):239-281.

Kennedy, J. & Eberhart, R. (1995). Particle swarm optimization. In
Proceedings of the IEEE International Conference on Neural Networks,
pag. 1942-1948, Perth, Australia.

Similar documents