More on kernel functions

10/14/13
Kernels
Definition: A function k(x, z) that can be expressed as a dot
product in some feature space is called a kernel.
More on kernel functions
In other words, k(x, z) is a kernel if there exists
such that
k(x, z) = (x)| (z)
: X 7! F
1
Example
Consider:
(x) =
(x21 ,
p
Kernels for arbitrary domains
Claim: Let A, B be sets over a finite domain, then the function
2x1 x2 , x22 )|
and compute the dot product
is a kernel.
p
p
(x)| (z) = (x21 , 2x1 x2 , x22 )| (z12 , 2z1 z2 , z22 )
= (x z)
K(A, B) = |A \ B|
Need to construct a mapping such that
= x21 z12 + 2x1 x2 z1 z2 + x22 z22
|
2
(A)| (B) = |A \ B|
2
3
4
1
10/14/13
Kernels for DNA and protein sequences
Standard kernel functions
You want to classify protein/DNA sequences.
The linear kernel
Feature space:
|
k(x, z) = x z
Each sequence has different length.
Original features
Homogeneous polynomial kernel
Represent a sequence in a fixed-dimensional space: e.g.
substrings of length k (this is called the spectrum kernel).
k(x, z) = (x| z)d
What is the dimensionality as a function of k?
All monomials of
degree d
Polynomial kernel
k(x, z) = (x| z + 1)d
Extensions of the spectrum kernel: allowing mismatches and
gaps.
All monomials of
degree less than d
Gaussian kernel
k(x, z) = exp(
||x
z||2 )
Infinite dimensional
How do we even know that the Gaussian is a valid kernel?
5
6
Some tricks for constructing kernels
Some tricks for constructing kernels
Let K(x,z) be a kernel function
Let K(x,z) be a kernel function
If a > 0 then aK(x,z) is a kernel
If a > 0 then aK(x,z) is a kernel
What can we say if a < 0?
q 
7
We know that a kernel matrix is positive semi definite, and
therefore has positive eigenvalues. Multiplying a PSD matrix
by a negative number makes is negative semi definite, and
therefore not a kernel matrix.
8
2
10/14/13
Some tricks for constructing kernels
Some tricks for constructing kernels
Sums of kernels are kernels:
Sums of kernels are kernels:
Let K1 and K2 be kernel functions then K1 + K2 is a kernel
Let K1 and K2 be kernel functions then K1 + K2 is a kernel
What is the feature map that shows this?
Feature map: concatenation of the underlying feature maps
9
Some tricks for constructing kernels
10
Some tricks for constructing kernels
Products of kernels are kernels:
Products of kernels are kernels:
Let K1 and K2 be kernel functions then K1 K2 is a kernel
Let K1 and K2 be kernel functions then K1 K2 is a kernel
Construct a feature map that contains all products of pairs of
features
11
12
3
10/14/13
The cosine kernel
The cosine kernel
If K(x,z) is a kernel then
is a kernel
K 0 (x, z) = p
If K(x,z) is a kernel then
K(x, z)
K(x, x)K(z, z)
is a kernel
K 0 (x, z) = p
K(x, z)
K(x, x)K(z, z)
This kernel is equivalent to normalizing each example to have
unit norm in the feature space associated with the kernel.
This kernel is the cosine in the feature space associated with
the kernel K:
cos(x, z) =
(x)| (z)
(x)| (z)
=p
|| (x)|| || (z)||
(x)| (x) (z)| (z)
13
14
Infinite sums of kernels
Theorem: A function
Infinite sums of kernels
K(x, z) = K(x| z)
With a series expansion
K(t) =
1
X
Theorem: A function
With a series expansion
an t
n
K(t) =
n=0
is a kernel iff
an
K(x, z) = K(x| z)
1
X
a n tn
n=0
0 for all n.
is a kernel iff
Corollary:
15
an
0 for all n.
K(x, z) = exp (2 x| z)
is a kernel
16
4
10/14/13
Infinite sums of kernels
Theorem: A function
K(x, z) = K(x| z)
With a series expansion
K(t) =
1
X
a n tn
n=0
is a kernel iff
an
0 for all n.
Corollary:
K(x, z) = exp (2 x| z)
Corollary:
k(x, z) = exp(
exp(
||x
z||2 ) = exp (
||x
is a kernel
z||2 )
(x| x
is a kernel
z| z) + 2 (x| z))
i.e. the Gaussian kernel is the cosine kernel of the exponential
K(x, z)
kernel
0
K (x, z) = p
K(x, x)K(z, z)
17
5