Largest eigenvalues of the sample covariance matrix

Transcription

Largest eigenvalues of the sample covariance matrix
for p-variate time series with heavy-tails
Richard A. Davis, Columbia University
Thomas Mikosch, University of Copenhagen
Oliver Pfaffel, Munich Re
September 8–11, 2013
EVT2013 Extremes in Vimeiro Today
In Honour of Maria Ivette Gomes
Vimeiro, Portugal
Davis (Columbia University)
Extremes in Vimeiro Today
1
Motivation
Large dimensional data sets appear in many quantitative fields like
finance, environmental sciences, wireless communications, fMRI, and
genetics.
Structure in this data can often be analyzed via sample covariances.
PCA is used to transform data to a new set of variables, the principal
components, ordered s.t. the first few retain most of the variation of
the data.
This suggests the need for an eigenvalue decomposition of the sample
covariance matrix.
2
Game Plan
The Setup
Background
Main result for linear time series with linear dependence between
rows
Corollaries and applications
Elements of the proof I
Elements of the proof II
Extensions to more general linear structure
Random coefficient models and hidden Markov models
Extension to nonlinear models–stochastic volatility and GARCH(1,1)
3
The Setup
Data matrix: A p × n matrix X consisting of n observations of a
p-dimensional time series, i.e.,

 X11 X12 · · ·
 X
 21 X22 · · ·
X =  .
..
..
 ..
.
.

Xp1 Xp2 · · ·
X1n
X2n
..
.
Xpn




 .


Sample covariance matrix: the p × p sample covariance matrix
(normalized) is given by
 n
p
X

ˆ

XX = nΓ(0) = 
Xit Xjt 
T
t =1
.
i ,j =1
Objective: study the ordered eigenvalues
λ(1) ≥ λ(2) ≥ . . . ≥ λ(p )
of the p × p sample covariance matrix XX T .
4
The Setup-continued
Data matrix and sample covariance matrix:

 X11 X12 · · ·
 X
 21 X22 · · ·
X =  .
..
..
 ..
.
.

Xp1 Xp2 · · ·
X1n
X2n
..
.
Xpn




ˆ 0)
 and XX T = nΓ(


Note that if the rows are independent and identically distributed
ergodic time series (with mean 0 and variance 1), then for p fixed,
P
ˆ 0) →
Γ(
Ip .
Relation to PCA: λ(1) is the empirical variance of the first principal
component, λ(2) of the second, and so on.
5
Known results for the largest eigenvalue
Assume the entries of X are iid Gaussian (with mean zero and
variance one)
For n → ∞ and fixed p, Anderson [1963] proved that
r
n λ(1)
d
− 1 → N(0, 1) .
2 n
!
Johnstone [2001] showed that for p , n → ∞ s.t. p /n → γ ∈ (0, ∞)
√
n+
q
3
√1
n
√
+



λ(1)
 d
 √
−
1
 → Tracy-Widom distribution
2
√

n+ p

p 
√1
p
6
Our objective
The assumption of Gaussianity in Johnstone’s result can be relaxed
to a moment condition (c.f. Four Moment Theorem by Tao and
¨ Johansson, Pech
´ e,
´ Schlein,
Vu [2011]; and work by Erdos,
Soshnikov, Yau and others).
BUT: in applications one often has neither independent observations,
nor Gaussianity or even the existence of sufficient moments.
This lead us to consider heavy-tailed random matrices with dependent
entries.
7
Setting for our results
Suppose X = (Xit )i ,t , i = 1, . . . , p, t = 1, . . . , n, with
Xit =
∞ X
∞
X
cj θk Zi −k ,t −j .
j =0 k =0
The noise (Zi ,t ) is iid with regularly varying tails of index α ∈ (0, 4)
(infinite fourth moment), i.e.,
n P (|Z11 | > an x ) → x −α as n → ∞, for x > 0,
(an = L (n)n1/α ) and
lim
x →∞
P (Z11 > x )
= p+
P (|Z11 | > x )
and
lim
x →∞
P (Z11 ≤ −x )
= 1 − p+
P (|Z11 | > x )
|cj |δ < ∞,
0
k =0 |θk | < ∞ for some δ < min{1, α} and δ < min{1, α/2} .
Summability assumptions on the coefficients:
P∞
δ0
P∞
j =0
8
Setting for our results (cont)
Let λ1 , . . . , λp be the eigenvalues of



if α ∈ (0, 2),
XX T ,


T
T
XX − E (XX ), if α ∈ (2, 4).
Let (Ds ) be the iid sequence given by
(n)
Ds = Ds
=
n X
∞
X
cl2 Zs2,t −l .
t =1 l =0
Note:
1
The Ds play a key role in determining the asymptotic properties of the
ordered eigenvalues λ(1) ≥ · · · ≥ λ(p ) .
2
Ds is the partial sum of a linear process
noise is regularly varying with index α/2.
P∞
2
l =0 cl Zs ,t −l ,
t ∈ Z, whose
9
Theorem (First main result)
Let p = pn → ∞ be a sequence satisfying certain growth conditions (to be
specified later) and suppose k = kp → ∞ is any sequence such that
P
2
k 2 = o (p ). Set Θ = ∞
i =0 θi .
1
If α ∈ (0, 2), then
−2
anp
2
P
max λ(i ) − D(i ) Θ ∨ max λ(i ) → 0 ,
i =1,...,k
n → ∞.
k +1≤i ≤p
If α ∈ (2, 4), then
P
−2
anp
max e
λ(i ) − (D`i − ED ) Θ ∨ max |λ(i ) | → 0 ,
i =1,...,k
k +1≤i ≤p
n → ∞.
˜(1) , . . . , λ
˜(p ) are the ordered eigenvalues (λi ) according to their
Note: 1) λ
absolute values.
2) The `i ’s are defined via
˜ `1 ≥ · · · ≥ D
˜ `p ,
D
˜ s = |Ds − ED | .
where D
10
Theorem (Point process convergence)
Under the conditions of the previous theorem, we have the point process
convergence,
Np :=
p
X
d
anp
−2 λ → N =
i
∞
X
C ΘΓ−2/α ,
i =1
i =1
i
where Γi = E1 + . . . + Ei is the cumulative sum of iid standard (i.e., mean
one) exponentially distributed rv’s,
C=
∞
X
cj2 and Θ =
j =0
∞
X
θk2 .
k =0
(N is a Poisson process with E (N (dx )) = (C Θ)α/2 α/2x −α/2−1 dx.)
11
The largest eigenvalues
The theorem implies the joint convergence of the m-largest
eigenvalues
d
−2
λ(1) , . . . , λ(m) → C Θ Γ−1 2/α , . . . , Γ−m2/α .
anp
(1)
For independent entries this was shown by Soshnikov [2006] for
´ e´ [2009] for 2 ≤ α < 4.
α < 2, and by Auffinger, Ben Arous and Pech
12
eigenvalues
d
−2
λ(1) , . . . , λ(m) → C Θ Γ−1 2/α , . . . , Γ−m2/α .
anp
(1)
´ e´ [2009] for 2 ≤ α < 4.
d
2
(λ(1) /(anp
C Θ))−α/2 → Γ1
12
eigenvalues
d
−2
λ(1) , . . . , λ(m) → C Θ Γ−1 2/α , . . . , Γ−m2/α .
anp
(1)
´ e´ [2009] for 2 ≤ α < 4.
d
2
(λ(1) /(anp
C Θ))−α/2 → Γ1
By the continuous mapping theorem,
λ(1)
d
λ(1) + · · · + λ(m)
→
Γ−1 2/α
Γ−1 2/α + · · · + Γ−m2/α
,
n → ∞.
12
eigenvalues
d
−2
λ(1) , . . . , λ(m) → C Θ Γ−1 2/α , . . . , Γ−m2/α .
anp
(1)
´ e´ [2009] for 2 ≤ α < 4.
d
2
(λ(1) /(anp
C Θ))−α/2 → Γ1
By the continuous mapping theorem,
λ(1)
d
λ(1) + · · · + λ(m)
→
Γ−1 2/α
Γ−1 2/α + · · · + Γ−m2/α
,
n → ∞.
In fact, we have more!!
12
Under the conditions of the theorem, the following limit results hold.
1
If α ∈ (0, 2), then
Γ−2/α
d
→ P∞ 1 −2/α ,
λ1 + · · · + λp
i =1 Γi
λ(1)
n → ∞,
where the limit is independent of (θk ) and (cl ).
2
If α ∈ (2, 4) then
λ(1)
d
λ1 + · · · + λp
→
Γ−1 2/α
,
ξα/2
n → ∞,
and the limit is independent of (θk ) and (cl ). The LHS is the
proportion of the sample variance explained by the variance of the
largest principal component.
13
Application–consistency
For α ∈ (2, 4),
d
−2
anp
λ(1) → C ΘΓ−1 2/α ,
where λ(1) is the largest eigenvalue of Xn XnT − E (Xn XnT ).
Now λ(1) /n is the largest eigenvalue of n−1 Xn XnT − Γn (0), where
Γn (0) = [E (Xit Xjt )]pi,j =1 ,
Since n−1 Xn XnT − Γn (0) → 0 element-wise, one might expect consistency
of the eigenvalues, i.e.,
λ(1) /n → 0 .
−2 → 0 or when p = o (n (α−2)/2 ) .
But this is true if and only if nanp
14
QQ−Plot via ratios of partial sums of the largest eigenvalue
Normalized partial sum of the transf. largest ev
1
0.9
Xt ∼ Pareto(α = 1.3) iid
0.8
0.7
n=p=1000
Number of replications: 200
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
i/100
15
1
0.9
0.8
0.7
Xt + 0.7Xt−1 = Zt
Zt ∼ Pareto(α = 1.3) iid
n=p=1000
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
i/100
16
Growth conditions on pn
Typical entry in XX T involves sums of terms involving squares Zs2 and
cross-products Zs1 Zs2 with s1 , s2 .
17
From Embrechts and Goldie (1980),
P (|Z1 | > x ) = L1 (x )x −α
and P (|Z1 Z2 | > x ) = L2 (x )x −α
where L1 and L2 are SV functions.
17
P (|Z1 | > x ) = L1 (x )x −α
and P (|Z1 Z2 | > x ) = L2 (x )x −α
Precise conditions on pn rely on the asymptotic relationship between
L1 and L2 .
17
P (|Z1 | > x ) = L1 (x )x −α
and P (|Z1 Z2 | > x ) = L2 (x )x −α
Precise conditions on pn rely on the asymptotic relationship between
L1 and L2 .
General conditions:
2
For α ∈ (0, 1), lim supn→∞ p [n p P (|Z1 Z2 | > anp
)] = 0 .
For α ∈ (1, 2), there exists γ ∈ (α, 2) but arbitrarily close to α such
2
that lim supn→∞ p γ [n p P (|Z1 Z2 | > anp
)] = 0 .
For α ∈ (2, 4), there exists γ ∈ (α, 4) arbitrarily close to α such that
2
lim supn→∞ nγ/2−1 p γ [n p P (|Z1 Z2 | > anp
)] = ∞ .
17
Case P (Z1 > x ) ∼ cx −α : Here L2 (x ) = C log(x ).
For α ∈ (0, 2),
pn = O (nβ ) ,
for any β > 0 .
Can allow for a touch faster growth rate (pn = O (exp{cn }), where
cn2 /n → 0 in the α ∈ (0, 1) case.
For α ∈ (2, 4),
pn = O (nβ ) ,
β ∈ (0, (4 − α)/[2(α − 1)]) .
This excludes the case pn ∼ cn.
18
Elements of the proof I:
Special case: Xi ,t = θ0 Zi ,t + θ1 Zi −1,t
n
X
Xit2
=
t =1
n
X
Xit Xi +1,t
t =1
n
X
θ02 Zi2,t + θ12 Zi2−1,t +2θ0 θ1
Zi ,t Zi −1,t
| {z }
|
{z
}
t =1
t =1
tail index α
tail index α/2
=
and
n
X
θ02 Di
= θ0 θ1
+
θ12 Di −1
n
X
+
2
nop (anp
)
2
Zi2,t + nop (anp
)
t =1
2
= θ0 θ1 Di + nop (anp
)
19
Elements of the proof I:
Special case: Xi ,t = θ0 Zi ,t + θ1 Zi −1,t
n
X
Xit2
=
t =1
n
X
n
X
θ02 Zi2,t + θ12 Zi2−1,t +2θ0 θ1
Zi ,t Zi −1,t
| {z }
|
{z
}
t =1
t =1
tail index α
tail index α/2
=
and
n
X
Xit Xi +1,t
θ02 Di
= θ0 θ1
+
θ12 Di −1
n
X
+
2
nop (anp
)
2
Zi2,t + nop (anp
)
t =1
t =1
2
= θ0 θ1 Di + nop (anp
)
XTi Xi
XTi+1 Xi
XTi+1 Xi
XTi+1 Xi +1
θ02 θ0 θ1
θ0 θ1 θ12
!
!
≈
+
0 0
0 θ02
!
Di +
θ12 0
0
0
!
Di −1
Di +1
19
The covariance matrix can be approximated by
T
XX =
p
X
2
Di Mi + op (anp
),
i =1
where Mi is the p × p matrix consisting of all zeros except for a 2 × 2 matrix,
θ02 θ0 θ1
θ0 θ1 θ12
M=
!
,
whose NW corner is pinned to the i th position on the diagonal. For
example,
 2
 θ0 θ0 θ1 0 · · ·

2
 θ0 θ1 θ1 0 · · ·
 0
0
0 ···
M1 = 
 .
..
.. . .
 ..
.
.
.

0
0
0
0






.. 
. 

0
0
0
0





M2 = 



0
0
0 θ02
0 θ0 θ1
0
..
.
..
.
θ0 θ1
θ12
..
.
0
0
0






.. 
. 

···
···
···
..
.
0
0
0
0
0
20
Denote the order statistics of the Di ’s by D(1) ≥ D(2) ≥ · · · ≥ D(p ) and write
DLi = D(i ) .
Then,
XX T =
Pp
i =1
2
DLi MLi + op (anp
) in the sense that
−2
anp
kXX T −
p
X
P
DLi MLi k2 → 0 ,
i =1
where
q
kA k2 =
largest eigenvalue of AA T (operator 2-norm).
21
For k → ∞ sufficiently slow,
k
X
P
−2 XX T −
DLi MLi → 0 .
anp
i =1
2
Since the Ds are iid, (L1 , . . . , Lp ) is a random permutation of (1, . . . , p )
and hence the set Ak = {|Li − Li −1 | > 1, i = 1, . . . , k } has probability
converging to 1 provided k 2 = o (p ).
22
k
X
P
−2 XX T −
DLi MLi → 0 .
anp
i =1
2
On the set Ak , the matrix ki=1 DLi MLi is block diagonal with nonzero
eigenvalues DLi Θ, i = 1, . . . , k . Here we used the fact that MLi is a
rank 1 matrix with nonzero ev equal to Θ = θ02 + θ12 .
P
22
k
X
P
−2 XX T −
DLi MLi → 0 .
anp
i =1
2
On the set Ak , the matrix ki=1 DLi MLi is block diagonal with nonzero
eigenvalues DLi Θ, i = 1, . . . , k . Here we used the fact that MLi is a
rank 1 matrix with nonzero ev equal to Θ = θ02 + θ12 .
P
By Weyl’s inequality
k
X
P
−2
−2 XX T −
anp
max |λ(i ) − DLi Θ| ≤ anp
DLi MLi → 0 .
i =1,...,k
i =1
2
22
Elements of the proof II (cont)
(n)
Large deviations: Ds
=
P∞
Pn
t =1
2 2
l =0 cl Zs ,t −l
.
∞
P (D > x ) X
1
α |
c
|
−
sup j → 0,
2
x >bn nP (Z > x )
1
j =0
where bn /an2 → ∞.
Classical EVT plus large deviations implies:
p
X
i =1
anp
−2 λ ∼
i
p
X
d
anp
−2 Θ D → N =
i
i =1
∞
X
i =1
ΘC Γ−2/α .
i
23
Important tool: kA k2 =
2-norm).
p
largest eigenvalue of AA T (operator
Define D ∈ Rp ×p by Dii = (XX T )ii and Dij = 0 for i , j. Then
P
−2 anp
XX T − D 2 → 0 as p , n → ∞.
By Weyl’s inequality
n
X
T
P
−2 2 −2 2 → 0 as p , n → ∞
anp
λ
−
max
X
≤
a
XX
−
D
(
1
)
np
it 1 ≤i ≤p
t =1
and likewise for λ(2) , λ(3) , . . .
Hence, we “only” have to derive the extremal behavior of the diagonal
P
elements ( nt=1 Xit2 )i of XX T .
24
1
0.9
0.8
0.7
Xit = Zit + θi Zi,t−1
Zit ∼ Pareto(α = 1.3) iid
θi = 0.8θi−1 + ξi
ξi ∼ Uniform(−1, 1) iid
n=p=1000
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
i/100
25
Preview example:
Xi ,t
= Zi ,t − Zi ,t −1 − 2Zi −1,t + 2Zi −1,t −1
26
Preview example:
Xi ,t
= Zi ,t − Zi ,t −1 − 2Zi −1,t + 2Zi −1,t −1
Then,
Np :=
p
X
i =1
d
anp
−2 λ → N =
i
∞ X
8Γ−2/α + 2Γ−2/α .
i =1
i
i
26
Preview example:
= Zi ,t − Zi ,t −1 − 2Zi −1,t + 2Zi −1,t −1
Xi ,t
Then,
Np :=
p
X
i =1
d
anp
−2 λ → N =
i
∞ X
8Γ−2/α + 2Γ−2/α .
i =1
i
i
Results:
d
−2/α
−2 λ
anp
(1) → 8Γ1
26
Preview example:
= Zi ,t − Zi ,t −1 − 2Zi −1,t + 2Zi −1,t −1
Xi ,t
Then,
Np :=
p
X
i =1
d
anp
−2 λ → N =
i
∞ X
8Γ−2/α + 2Γ−2/α .
i =1
i
i
Results:
d
−2/α
−2 λ
anp
(1) → 8Γ1
d
−2 (λ , λ ) → (8Γ−2/α , 2Γ−2/α ∨ 8Γ−2/α )
anp
(1) (2)
1
1
2
26
Preview example:
= Zi ,t − Zi ,t −1 − 2Zi −1,t + 2Zi −1,t −1
Xi ,t
Then,
Np :=
p
X
d
anp
−2 λ → N =
i
i =1
∞ X
8Γ−2/α + 2Γ−2/α .
i
i =1
i
Results:
d
−2/α
−2 λ
anp
(1) → 8Γ1
d
−2 (λ , λ ) → (8Γ−2/α , 2Γ−2/α ∨ 8Γ−2/α )
anp
(1) (2)
1
1
2
λ(1)
λ1 + · · · + λp
d
→
−2/α
Γ1
8
,
P
∞
10
Γ−2/α
i =1
n → ∞.
i
26
Suppose X = (Xit )i ,t , i = 1, . . . , p, t = 1, . . . , n, with
Xit =
m
∞ X
X
h (k , l )Zi −k ,t −l ,
j =0 k =0
Summability assumptions on the coefficients:
m
X
|h (k , l )|δ < ∞ for some δ < min{1, α}
k =0
and
∞ X
∞
X
( |h (k , l )|)α/2 < ∞,
for k = 0, . . . , m.
t =0 l =t
27
Set hi = (hi0 , hi1 , . . .)0 and define the matrix H = (h0 , . . . , hm ). Let
M = H0H .
i.e., the (i , j )th entry of M is
Mij = h0i hj =
∞
X
hil hjl ,
i , j = 0, . . . , m .
l =0
By construction, M is symmetric and non-negative definite and has
ordered eigenvalues
vm ≤ · · · ≤ v1 .
Let r be the rank of M so that vr > 0 while vr +1 = 0 if r < m.
This matrix plays the role of the M matrix in previous example.
28
For previous numerical example,
1 −1 0 0 · · ·
−2 2 0 0 · · ·
T
H =
so that
T
M=H H=
2 0
0 8
!
!
which has eigenvalues 2 and 8.
Theorem (Second main result)
Np :=
p
X
i =1
d
anp
−2 λ → N =
i
r X
∞
X
j =1 i =1
vj Γ−2/α .
i
29
Random coefficient models–dependence between rows
X = (Xit ) with
∞
X
Xit =
cj (θi )Zi ,t −j
j =−∞
where cj (·) is a family of functions s.t.
supθ |cj (θ)| ≤ c˜j with
∞
X
|c˜j |δ < ∞, δ < min{1, α}, and
j =−∞
(θi ) is a stationary ergodic sequence independent of (Zit )i ,t .
Conditionally on (θi ), each process (Xit )t in the rows of X has a
different set of coefficients (cj (θi ))j .
Unconditionally, the row processes are identically distributed with
regularly varying tail probabilities.
Rows of X are dependent if the θi ’s are dependent.
30
Random coefficient models
Theorem (Random coefficient models)
Suppose pn , n → ∞ such that
lim sup
n→∞
pn
<∞
nβ
for some β > 0 satisfying
1
2
β < ∞ if α ∈ (0, 1], and
n
o
−α 1
β < max 2α−
,
1 2 if α ∈ (1, 2).
Then, conditionally on (θi ) as well as unconditionally,
p
X
d
→N=
anp
−2 λ
(i )
i =1
∞
X
i =1
P
α/2 2/α
2
Γi−2/α E ∞
j =−∞ cj (θ1 )
.
31
The largest eigenvalue
In particular we have
α/2 −1
 

∞

 λ(1) −α/2 d  X

2




cj (θ1 )  Γ1
→ E 2

anp

j =−∞
= constant × standard exponential
as p , n → ∞
Also a good approximation for finite p and n?
32
Hidden Markov models
Suppose that (θi ) is either
an irreducible Markov chain on a countable state space Θ, or
positive Harris in the sense of Meyn and Tweedie [2009].
If (θi ) has a stationary probability distribution π then, conditionally on
(θi ) as well as unconditionally,
p
X
i =1
d
anp
→N=
−2 λ
(i )
∞
X
i =1
Γ−i 2/α
2/α
R P
| j cj2 (θ)|α/2 π(d θ)
.
33
Stochastic volatility models—special case
Suppose the rows are independent copies of the SV process given by
Xt = σt Zt
where (Zt ) is iid RV(α) and (ln σ2t ) is a purely nondeterministic stationary
Gaussian process (this can be weakened), independent of (Zt ).
Theorem
Suppose pn , n → ∞ such that
lim sup
n→∞
pn
< ∞ , for some β > 0 satisfying
nβ
1
β < ∞ if α ∈ (0, 1), and
2
β<
2−α
α−1
if α ∈ (1, 2).
Then, we have the point process convergence,
Np :=
p
X
d
anp
→N=
−2 λ
(i )
i =1
∞
X
i =1
Γ−2/α .
i
34
Stochastic volatility models—special case
Point process convergence:
Np :=
p
X
d
anp
→N=
−2 λ
(i )
∞
X
Γ−2/α .
i =1
i =1
i
Remarks:
Proof uses a large deviation result of Davis and Hsing (1995); see
also Mikosch and Wintenberger (2012).
Likely that we can weaken the restriction on β
Similar results hold for GARCH processes if Xt is RV(α) with
α ∈ (0, 2).
35
References
Richard A. Davis, Oliver Pfaffel and Robert Stelzer
Limit Theory for the largest eigenvalues of sample covariance matrices with
heavy-tails.
Stoch. Proc. Appl. 24, 2014, 18–50.
Ian M. Johnstone
On the Distribution of the Largest Eigenvalue in Principal Components
Analysis.
Ann. Statist., 2001, 29, 295-327.
Alexander Soshnikov
Poisson Statistics for the Largest Eigenvalues in Random Matrix Ensembles.
Lect. Notes in Phys., 2006, 690, 351-364.
´
´ e´
Antonio Auffinger, Gerard
Ben Arous, and Sandrine Pech
Poisson convergence for the largest eigenvalues of heavy tailed random
matrices.
Ann. Inst. H. Poincare´ Probab. Statist., 2009, 45, 589-610.
36
Happy Birthday and Best Wishes Ivette!!!
37

Largest eigenvalues of the sample covariance matrix

Transcription

Similar documents

LiKE it - Katherine B. Weissman

GÖRAN SÖLLSCHER Discography

Technetics Group Columbia Visitor`s Guide

Subscription april-VOL-143.indd

september 2016

SIGASCOT – GOTS FELLOWSHIP 2013

Architecture and Ideology

Emilia Romagna

Exploring Disciplinarity Conceptualizations

quick guide - 2nd EURON Summer School on Perception and