Document 6535635

OUTLIERS IN SAMPLE SURVEYS
P.D. Ghangurde, Statistics Canada, Ottawa, KIA 0T6, Canada
KEYWORDS:
estimation
1.
Variance-inflation,
outlier
2.
robust
Variance-inflation Model
Consider a linear model
Introduction
Y = X
8 + e ,
n×1
n×p p×l
n×1
In the literature on regression analysis several
approaches for detection and treatment of outliers
have been developed. In addition to methods based on
the
mean-shift
and
variance-inflation
models,
estimators based on order statistics such as trimmed
and Winsorized means and M-estimators based on
robust regression methods are available. Regression
diagnostics provide methods for critical examination
of models and measures of influence of individual
outliers and groups of outliers on estimates of
parameters (see Beckman and Cook (1983); Cook and
Weisberg (1982)).
The objective of this paper is to develop outlier
robust estimators for sample surveys, based on
variance-inflation m o d e l . T h i s model is a simple
extension of superpopulation model often implicitly
assumed for the traditional design-based estimators
and more explicitly used in the prediction approach.
Although these estimators are obtained as optimal
estimators of parameters of this model, the results on
the bias and variance of these estimators and optimal
weight reduction for outliers, presented in this paper,
are in the framework of finite population sampling.
These outlier robust estimators are not model
dependent and have not been evaluated by prediction
approach.
Outlier robust estimators in finite
population sampling based on robust regression and
prediction approach have b e e n investigated by
Chambers (1986).
The problem of outliers has been considered in
the literature on finite population sampling in the
context of estimation of mean or total, usually
assuming no auxiliary information in estimation.
Estimators obtained by methods based on order
statistics, s u c h as Winsorization and trimming, and
weight reduction, have been investigated by assuming
simple random sampling (see e.g. Fuller (1970); Ernst
(1980)).
It seems that it is not possible to extend
methods based on order statistics to sample designs
involving stratification and different sampling ratios
and non-response rates between strata and unequal
probabilities of selection, which result in unequal
design weights.
Sample surveys are often periodic
with rotation samples designed for estimation of
changes. Moreover, estimates are needed at several
levels such as stratum, group of strata and domains.
Because of these features of sample surveys,
estimators based on reduction of weights of outliers
are more convenient for use in practice.
In Section 2 we introduce the model in which
variance is a function of an auxiliary variable x
and
assume that outliers have inflated variances.
The
optimal estimators of parameter 8
are derived by
assuming k outliers (l.<k<n) in a random sample of
size n. In Section 3 conditional mean square error of
these estimators and optimal weight reduction have
been derived by assuming simple random sampling
from a finite population.
In concluding remarks in
Section 4, comments have b e e n made on possible
extensions of outlier robust estimation and the
problem of estimation of unit
variances and
covariances of x and y for outliers and non-outliers.
(2.1)
2
2
where e ~ (0, o W),
unknown,W is a diagonal
variance-covariance matrix with elements wi
depending on x i , i = i , 2 . . . . .
n, Y is a n-vector of
responses of y, X is a design matrix of p
auxiliary
variables each with n observations assumed fixed, 8
is
a p-vector of regression coefficients and e is an error
term .
Under the model assuming p=1, wi--x g
i'
of sample means y/x
n
1( z Yi/xi ) are
and
the
?=~i=i
mean
best
ratio
of
linear
ratios
unbiased
estimators of 8 for g=l and 2 respectively.
The
model is appropriate for categorical variables in
socio-economic surveys. It is known that values of g
for many variables lie in the interval [1,2] and more
often closer to i than to 2.
In practice in multipurpose sample surveys with several y-variables and
an auxiliary variable x possibly used for stratification,
ratio estimation, although less than optimal for
variables with g>1, is often used for all y-variables
for convenience of uniform weighting method.
We now consider the variance-inflation model for
k outliers (1.<k<n) which are the last k s a m p l e units,
without loss of generality. Thus
Y = X e + e ,
n×l
n×p p×l
n×l
(2.2)
where e ~ (0, 02 W(k)) and W(k) is
a diagonal
variance-covariance matrix with elements wi , i = i ,
2.....
n-k and wi/w, i = ( n - k + l ) . . . . .
n; w
is
unknown constant (0<w.<1).
This model is a simple
extention of the variance-inflation model considered
by Pregibon (1981), Cook, Holschuh and Weisberg
(1982) and Thompson (1985).
We consider ^expression for the best linear
unbiased estimator 8 i ( w ) o f 8 under (2.2) for the ease
of one outlier, the i th sample unit. Thus
I
^
^
Bi (w) = ~
where for
^
(X' W-Ix) - I Xi ( y i - Y i ) ( l - w )
wi[l-
p = 1 , (X'N-Ix)-I
,
(Z-w) V i i ]
and
I
Xi
(2.3)
are scalars
= (X'W-IX)-I(x'N-Iy) is the estimator of 8
under(2.1) and Vii = w-i i Xi(X,W-ix )- IX,i is the ith
diagonal
element
of
variance-covariance matrix
V = V(Xs), called leverage of Xi.
Also, 0.<Vii.<l
whenp=1. For large values of Xi, Vii is close to i,
which makes contribution of i to 8i (w)
very
large.
The second term on the right hand side• of (2.3) shows
736
change due to variance-inflation of i th
sample
Thus influence of both residual (yi-Yi)
and
unit.
leverage
Vii is r e d u c e d due to f a c t o r
(l-w).
^
For k (> 1)
outliers and wi = x i, the e s t i m a t o r B(i ) (w),
(3.1)
kwx k + (n-k) Xn-k
( i ) r e p r e s e n t s group of k outliers can also be given in
the form which shows weight r e d u c t i o n of outliers, by
: wkyk + (n-k) Yn-k.
kwy k + (n-k) Yn-k
R =
where
is e s t i m a t o r
population
(2.4)
of the population ratio R = ?/X,
means ? and X. The r a t i o
R
of
can
be
e x p r e s s e d as
wkxk + (n-k) Xn-k
P ?i + (l-P) 72
R
When wi = x#, ~ ( i ) ( w )
^
r =
_.
is given by
wkr k + ( n - k )
P 21 + (I-P) X2
where X1 and ?i
rn-k
wk + (n-k)
'
and when x i = i for i = I, 2 . . . . .
given by
(3.2)
9
(2.5)
N, ~ ( i ) ( w )
^
wky k+ (n-k) Yn-k
7 =
wk + (n-k)
'
are unknown population means of
outliers, X2 and 72
non-outliers, P
is
are unknown population means of
is unknown proportion of outliers in
the finite population of (x,y).
2
We also assume that
2
2
2
O2x, O2y are unit variances of non-outliers ~Ix' °ly
2
2
2
are unit variances of outliers, Olx > O2x and Oly >
(2.6)
and Xn-k are sample means of non-outliers, rk
and
2
A l s o , oi
°2y"
xy and °2xy are unit covariances of
outliers and non-outliers respectively. By substituting
rn-k are sample means of ratios ri = Yi/Xi
for
71 = 6172and 21 = 6222 we have
where Yk and Xk are sample means of outliers, Yn-k
outliers and non-outliers respectively. When w÷0,
in
limit these estimators are ~ = :Yn_k/Xn_k, r = rn-k
R=
and ? = Yn-k' which can also be obtained as optimal
estimators of ~3 under corresponding mean-shift model.
The estimator of the population total can be obtained
from (2.6)as NC7 and
72
[1 + P(cs2-1 )]
X2
.
(3.3)
Since in general, 61 # 62 , R ~ 72/22 .
We obtain the
conditional bias B(RIk) by Taylor series linearization
shows reduction of simple
N
random sampling weight n of k outliers by factor w
method as
and resulting adjustment of weights of all n units by
factor n/[n-k(1-w) ]. F r o m (2.5) and (2.6) it can be
B(P, l k ) "
it
seen that the results for r can be obtained from those
for ? by substituting r i for yi for all i.
kw? + (n-k) 72
[ I
]- R
kwX1 + (n-k) X2
(62 - 61)
P(n-k)
kw(l-P) } (__).
?2
= [1_--~(1_62)] {
n_k(l_w62)
X2
These estimators have been investigated in the
following Section 3 using conditional inference given
the number of outliers k assuming simple random
sampling from a finite population. Although it may
be possible to use prediction approach and to obtain
model dependent estimators incorporating design
weights, the results in this paper have been obtained
in the framework of finite population sampling. For
detection of outliers in the case of ratio estimation
and sample mean Cook's distance seems to have good
potential (Cook and Weisberg (1982)).
3.
[1 + P(61-1 )]
(3.4)
When w=O, R = Yn-k/Xn-k and
(62-61) P
B(RIR) = (i_-~i162))
?2
(X)"
2
When w=l, R = .Vn/Xn with no weight reduction and
(62-61)
B(Rlk) = ( l - P ( l _ ~ 2 ) )
Outlier Robust E s t i m a t i o n
We assume t h a t a finite population of size N
contains an unknown proportion P of outliers.
The
population mean X of an auxiliary variable x
is
assumed known. A simple random sample of size n
is
drawn without r e p l a c e m e n t from the population and
outliers are identified on the basis of values of some
test statistic.
Sample units may be identified as
outliers if the 5th unit values ( x i , Y 5 )
lie in some
region of the sample space d e t e r m i n e d by the t e s t .
The t e s t could be based on diagnostics such as Cook's
d i s t a n c e (see Ghangurde (1989)).
The
outlier
robust
ratio
estimator
of the
population mean of y's is given by ^
?R = R2, where
It can be seen that f(w)
=
nP-k } 72
{n_k~l_-~2 ) ~2"
P(n-k)-
kw(1-P)is
n_k(l_w62)
a
monotonic d e c r e a s i n g function of w in ( 0 , l ) and f(w)
kw
X 0 a c c o r d i n g as P ~ [ n - k ( 1 - w ) ] " Hence, if P > k,
^
conditional bias of R for w=0
is in absolute value
^
k
g r e a t e r than t h a t of R for w in (0 1) I f P < "
absolute conditional bias increases as w
[P(n-k)
in
1] and may not be in absolute value lesser
(l-P)k'
than that for w=O.
737
n
increases
This analysis of conditional bias
of R is only of t h e o r e t i c a l i n t e r e s t ; P is not
practice.
By
linearization
of
(3.4)
outliers, more e f f i c i e n t e s t i m a t i o n can be done by an
e x t e n s i o n of Minimum Norm Q u a d r a t i c Unbiased
E s t i m a t i o n (MINQUE) m e t h o d (see Rao (1970)) to
estimation
of
heteroscedastic
variances
and
c o v a r i a n c e s when x and y are random.
known in
and
taking
e x p e c t a t i o n over k, unconditional bias of R is given by
P((s2-(s1) ~'2
B(R) : i+P(~2_1) (~2)(1-P)
i
[(I+P) - w(l+P+P62) + w2p(s2] + 0(~).
The o p t i m u m w, although obtained
under the
v a r i a n c e - i n f l a t i o n model, can still be used in finite
population sampling and mean square e r r o r can be
e s t i m a t e d for values of w close to the o p t i m u m given
by (3.7) to decide on choice of a n o t h e r value of w.
H o w e v e r , due to instability of e s t i m a t e s of bias, it
may be p r e f e r a b l e to use w given in (3.7).
(3.5)
When w=O,
P(~2-~I ) (I-P2)
B(R) =
?2
In the case of s a m p l e mean ~ the optimum wo
can be o b t a i n e d by minimizing mean square e r r o r and
is given by
>
(l+P(~2-~) (X) `~ 0 a c c o r d i n g as ~2<~i .
2
Also when w=l, B(R) = 0 to 0 ( 1 ) .
outlier robust sample
mean ~
2 .....
o
given in (2.6) can be
obtained from above results by substitution
i=l,
n-k
w : ( i - N(l_p) ) o Y + (n-k)
The results for
xi=l
P(aI - I ) 2
722
(3.8)
kl
for
In the l i t e r a t u r e s e v e r a l e s t i m a t o r s obtained by
weight r e d u c t i o n and which can be considered as
linear combinations of Yk and Yn-k
have
been
i n v e s t i g a t e d (see e.g. Hidiroglou and Srinath (1981)).
N, and Xl = ~2 = 62 = I.
_
By Taylor series l i n e a r i z a t i o n method
V(RIk) " (Rkw)2 V(~klk ) + (n-k)R)2~ V(~n_klk )
X
X
Their
optimal r=r o
2Rk2w 2
~2
C°V(Xk'Yk I k)
- 2P'(~2k)2 Cov(Xn_k, Yn_klk),
whereX = [kwXl+(n-k ) X2], ~( = [kwYl+(n-k)
and R = ~(/X.
(3.6)
^
w =
o
?2]
w=
2
2
by
minimizing mean square
°2y
n
^2
( i - ~) Oly
+ k(l
-
+ k(1
k
n)
(Y
- Yn- )
k
2
k
k
- n ) ( y k - .Yn_k)
it is possible to obtain e s t i m a t e
(3.9)
2"
^2
Oly
from
sample outliers, the e s t i m a t o r is unstable for small
values of k.
A l t e r n a t i v e m e t h o d s for e s t i m a t i o n of
h e t e r o s c e d a s t i c v a r i a n c e s have been proposed in the
l i t e r a t u r e (see K l e f f e (1977)).
The MINQUE was
d e v e l o p e d for p>.1 and n unequal v a r i a n c e s (see Rao
(1970)).
Assuming t h a t the last k units
are
e s t i m a t o r s of unit v a r i a n c e s obtained by
method are given by
n
n z
(yi-Yn)2
°ly = i=n-k+l
k (n-2)
n-k
n z (yi-Yn)
"2
i=i
°2y = (n-k) (n-2)
(3.7)
2
n
e s t i m a t o r s are more
2
2
e s t i m a t o r s of Oly and O2y
efficient
obtained
estimators
by
of 6
weighted
S u b r a h m a n i a m (1971)).
738
2
z (yi-Yn)
i=1
- (n-l) (n-2) "
These
This expression for o p t i m u m w o b t a i n e d by assuming
infinite N is simple and needs e s t i m a t i o n of f e w e r
p a r a m e t e r s . The means Ux and uy can be e s t i m a t e d
by sample means Xn and Yn
respectively.
Although
2
2
Olx, Oly and Ol×y
can be e s t i m a t e d from sample
outliers,
MINQUE
n
z (yi-Yn)2
i=1
(n-l) (n-2) '
"2
2
°2x~y + °2y~x - 2°2xy~x~Y
2 2
2 2
Olx~y + OlyUx - 2OlxyUx~y
obtained
n ^2
(i - ~)
Although,
In order to obtain an o p t i m a l value of w
the
conditional mean square e r r o r of ~ given k
can be
minimized as a function of w. H o w e v e r , if conditional
bias ratio is small an optimum w
obtained
by
minimizing conditional v a r i a n c e o f ~ can be used.
By
equating derivative
of conditional v a r i a n c e
with
r e s p e c t to w to zero we have an equation of third
d e g r e e in w
involving
several
parameters.
An
analytical
solution
cannot
be
obtained
unless
e s t i m a t e s are s u b s t i t u t e d for s e v e r a l p a r a m e t e r s
involved. An a l t e r n a t i v e is to minimize limiting value
of V(~lk) for infinite N,
which is the s a m e as
minimizing superpopulation v a r i a n c e of R
under the
v a r i a n c e - i n f l a t i o n model with Ux and Uy, means of x
and y r e s p e c t i v e l y .
It may be noted t h a t under the
model, bias of R is zero. The optimum w is given by
2
with
~ = - ~ Yk + (1 - -~) Yn-k
e r r o r of 7 is the s a m e a s ? with optimalw=wo,
since
k w o / ( ( n - k ) + kwo) = rok/N.
The
important
a d v a n t a g e of e s t i m a t o r s s u g g e s t e d above is t h a t the
weight r e d u c t i o n can be e s t i m a t e d from survey d a t a
a n d extensions to o t h e r sample designs seem possible.
Thus wo can be e s t i m a t e d by
+ (k_.~w)2 V(~klk ) + (n_.~)2 V(~n_klk )
X
X
-
estimator
efficient
and
(3.10)
than
also
the
usual
give
more
as
compared
to
those
least
squares
(Rao
and
4.
References
ConcludingRemarks
Beckman, R.J. and Cook, R.D. (1983), "Outliers,"
Technometrics, Vol. 25, No. 2, pp. 119-149.
Chambers, R . L . (1986), "Outlier
robust
finite
population estimation," JASA, Vol. 81, No. 396,
pp. 1063-1069.
Cook, R.D. Holschuh, N. and Weisberg, S. (1982), "A
note on an alternative outlier model," J.R.S.S., B,
44, No. 3, pp. 370-376.
Cook, R.D. and Weisberg,S. (1982), Residuals and
Influence in Regression, Chapman and Hall.
Ernst, L.R. (1980), "Comparisons of estimators of the
mean w h i c h adjust for large observations,"
Sankhya, C, 42, pp. 1-16.
Fuller, W.A. (1970), "Simple estimators of the mean of
skewed
populations,"
Technical
Report,
Department of Statistics, Iowa State University.
Gather, U. and Kale B.K. (1988), "Maximum likelihood
estimation
in
the
presence
of
outliers,"
Communications
in
Statistics,
Theory
and
Methods, 17 (11), pp. 3767-3784.
Ghangurde P.D. (1989), "Outlier robust estimation in
finite population sampling," presented at the
Statistical Society of Canada meeting, Ottawa.
Hidiroglou, M.A. and Srinath, K.P. (1981), "Some
estimators of a population total f r o m simple
random samples containing large units," JASA,
Vol. 78, No. 375, pp. 690-695.
Kleffe, J. (1977), "Optimal estimation of variance
components," Sankhya, Vol. 39, B, pp. 211-244.
Pregibon, D. (1981)' "Logistic regression diagnostics,"
The Annals of Statistics, Vol. 9, No. 4, pp. 705724.
Rao, C . R . (1970), "Estimation of heteroscedastic
variances in linear models," JASA, Vol.65,
No. 329, pp. 161-172.
Rao, J.N.K. and Subrahmaniam, K. (1971), "Combining
independent estimators and estimation in linear
regression w i t h unequal variances," Biometrics,
27, pp. 971-993.
Thompson, R. (1985), "A note on restricted maximum
likelihood with an alternative outlier model,"
J.R.S.S. (B), 47, No. I, pp. 53-55.
The variance-inflation model seems to be an
appropriate model for outliers in sample surveys. In
the optimal estimator based on the model influence of
outliers with large residuals and high leverage is
reduced due to factor (l-w).
U n d e r appropriate
assumptions about d e p e n d e n c e of v a r i a n c e on x,
it
reduces to o u t l i e r robust e s t i m a t o r s R, r and Y,
in
which weights of outliers are reduced. In the case of
ratio
estimator,
although
the
problem
of
d e t e r m i n a t i o n of o p t i m a l weight was simplified by
assuming an infinite population, it could still be used
in the case of finite population sampling. Although it
may be possible to use p r e d i c t i o n a p p r o a c h and to
obtain
model d e p e n d e n t
estimators
incorporating
design
weights,
conditional
inference
in
finite
population sampling f r a m e w o r k shows t h a t t h e s e
o u t l i e r robust e s t i m a t o r s have desirable properties.
Extension of these e s t i m a t o r s to s t r a t i f i e d sampling
i n c o r p o r a t i n g design weights is being i n v e s t i g a t e d .
Implicit in the v a r i a n c e - i n f l a t i o n model for
outliers is the assumption t h a t superpopulation is a
mixture of distributions with the same mean but
d i f f e r e n t variances.
In the case of sampling from
mixture distributions belonging to the e x p o n e n t i a l
family, maximum likelihood e s t i m a t o r s of p a r a m e t e r s
have been obtained and sufficient conditions have
been established for outliers to occur in samples from
these distributions (see e.g. G a t h e r and Kale (1988)).
It would be of i n t e r e s t t o i n v e s t i g a t e into the
possibility of establishing similar conditions for
occurrence
of outliers
in sampling from
finite
populations which have mixture distributions.
Although unit v a r i a n c e s and c o v a r i a n c e s for
outliers can be e s t i m a t e d from outliers in the sample,
the e s t i m a t o r could be unstable for small values of k.
In the case of r a t i o e s t i m a t i o n an extension of
MINQUE method for e s t i m a t i o n of h e t e r o s c e d a s t i c
v a r i a n c e s and c o v a r i a n c e s of x and y is needed.
The
problem of v a r i a n c e e s t i m a t i o n was not discussed
since it does not involve any new methodology, once w
is t r e a t e d as a c o m p o n e n t of weights of outliers.
_
739