Study Design I. Sample Size Consideration Tuan V. Nguyen

Study Design I.
Sample Size Consideration
Tuan V. Nguyen
Garvan Institute of Medical Research
Sydney, Australia
The
The classical
classical hypothesis
hypothesis testing
testing
• Define a null hypothesis:
– Ho: Xt = Xc
• Define an alternative hypothesis:
– Ha: Xt > Xc , Xt < Xc , Xt not equal to Xc.
• Perform a test of significance on the null hypothesis.
– Assume that the null hypothesis is true.
– Determine the probability of obtaining the observations
found in the data.
• Accept or reject Ho
– If the Ho is rejected, the alternative hypothesis is accepted.
– But there are many alternative hypotheses!
Diagnosis
Diagnosis and
and statistical
statistical reasoning
reasoning
Disease status
Present
Absent
Significance Difference is
Present
Absent
(Ho not true)
Test result
+ve
True +ve
False +ve
Test result
Reject Ho No error
1-β
True -ve
Accept Ho
(sensitivity)
-ve
False -ve
(Specificity)
(Ho is true)
Type I err.
α
Type II err. No error
β
1−α
α : significance level
1-β : power
Study
Study Design
Design Issues
Issues
•
•
•
•
Setting
Participants: inclusion / exclusion criteria
Design: cross-sectional, longitudinal
Measurements: outcome, covariates / risk
factors
• Analysis
• Sample size / power issues
Sample
Sample size
size issues
issues
• How many observations / subjects?
– Practical and statistical issues
– Ethical issues
• Ethical issues in clinical studies
– Unnecessarily large number of patients may be
deemed unethical
– Too small a sample may also be unethical as
the study can’t show anything.
Large
Large difference
difference vs
vs Statistical
Statistical significance
significance
Status
Improved
Group A Group B
Status
9
18
Improved
No improved
21
12
Total
30
30
Chi-square: 5.4; P < 0.05
“Statistically significant”
Group A Group B
6
12
No improved
14
8
Total
20
20
Chi-square: 3.3; P > 0.05
“Statistically insignificant”
Effect of sample size: a simulation
True mean: 100
True SD: 15
True mean: 100
True SD: 35
Sample size
Est. M SD
Est. M SD
10
50
100
200
500
1000
2000
10000
100000
98.0
100.4
101.3
99.9
99.8
99.5
99.7
100.1
100.0
108.9
95.3
99.1
100.3
98.9
99.9
99.9
99.9
100.0
11.0
13.6
14.4
15.2
15.3
15.1
15.0
15.0
15.0
32.2
41.4
35.5
33.2
33.8
35.0
34.7
35.0
35.0
Specification
Specification for
for sample
sample size
size determination
determination
•
•
•
•
Parameter of major interest
Magnitude of difference in the parameter
Variability of the parameter
Bound of errors (type I and type II error
rates)
Parameter
Parameter of
of Interest
Interest
• Type of measurement of primary interest:
– Continuous or categorical outcome
• Examples:
– Mortality: proportion (or probability) of death/survival
– Blood pressure: difference in BP in mmHg
– Quality of life: change in QoL scores
Variability
Variability of
of the
the Parameter
Parameter of
of Interest
Interest
• If the parameter is a continuous variable:
– What is the standard deviation (SD) ?
• If the parameter is a categorical variable:
– SD can be estimated from the proportion/probability.
Magnitude
Magnitude of
of Difference
Difference of
of Interest
Interest
• Distinction between clinical and statistical relevance.
• Change from baseline or difference between groups.
• Examples:
– Probability of survival: 85% vs 80%
– Blood pressure: difference between groups by 1 SD.
– Quality of life: difference in the change in QoL
between groups by 5%.
0.95
0.95
Z2
-1.96
0
1.96
0.025
0
0.025
Prob.
0.80
0.90
0.95
0.99
Z1
0.84
1.28
1.64
2.33
Z2
1.28
1.64
1.96
2.81
Z1
1.64
0.05
Alpha
c
0.20
0.10
0.05
0.01
Zα
(One-sided)
0.84
1.28
1.64
2.33
Zα/2
(Two-sided)
1.28
1.64
1.96
2.81
Power
0.80
0.90
0.95
0.99
Z1−β
0.84
1.28
1.64
2.33
• The serum cholesterol levels of
Californian children have a mean of
175 mg/100ml and a standard
deviation of 30 mg/100ml. The
distribution of the cholesterol levels
is normal.
116
• 95% of the children should have
cholesterol levels ranged between
175 + (1.96x30) = 116 and 234
mg/100ml.
175
234
• If we let X be the chol. level for any
child, then X can be converted to a
variable with mean=0 and SD=1:
Z = (X – 175) / 30
mg/100l
Z
-1.96
Abnormal?
0
1.96
Abnormal?
Study
Study design
design and
and Outcome
Outcome
• Single population
• Two populations
• Continuous measurement
• Categorical outcome
• Correlation
Single
Single Population
Population
Sample
Sample size
size for
for estimating
estimating aa population
population mean
mean
• How close to the true mean
• Confidence around the sample mean
• Type I error.
• N = (Zα/2)2 σ2 / d2
σ: standard deviation
d: the accuracy of estimate (how
close to the true mean).
Zα/2: A Normal deviate reflects the
type I error.
• Example: we want to estimate the
average weight in a population,
and we want the error of
estimation to be less than 2 kg of
the true mean, with a probability
of 95% (e.g., error rate of 5%).
• N = (1.96)2 σ2 / 22
Sample size
96
138
188
246
311
384
400
350
300
Sample size
Std Dev (σ)
10
12
14
16
18
20
450
250
200
150
100
50
0
0
5
10
15
Standard deviation
20
25
Sample
Sample size
size for
for estimating
estimating aa population
population proportion
proportion
• How close to the true proportion
• Confidence around the sample
proportion.
• Type I error.
• N = (Zα/2)2 p(1-p) / d2
p: proportion to be estimated.
d: the accuracy of estimate (how
close to the true proportion).
Zα/2: A Normal deviate reflects the
type I error.
• Example: The prevalence of
osteoporosis in the general
population is around 30%. We
want to estimate the prevalence p
in a community within 2% with
95% confidence interval.
• N = (1.96)2 (0.3)(0.7) /
0.022 = 2017 subjects.
• N = (1.96)2 (0.3)(0.7) /
0.022 = 2017 subjects.
2500
2000
Sample size
• Example: The prevalence of
osteoporosis in the general
population is around 30%. We
want to estimate the prevalence p
in a community within 2% with
95% confidence interval.
1500
1000
500
0
0
0.02
0.04
0.06
Standard deviation
0.08
0.1
Sample
Sample size
size for
for estimating
estimating aa correlation
correlation coeffcient
coeffcient
• In observational studies which involve estimate a correlation (r) between two
variables of interest, say, X and Y, a typical hypothesis is of the form:
– Ho: r = 0
vs
H1: r not equal to 0.
• The test statistic is of the Fisher's z transformation, which can be written as:
1
é1 + r ù
t = log e ê
n−3
ú
2
ë1 − r û
• Where n is the sample size and r is the observed correlation coefficient.
• It can be shown that t is normally distributed with mean 0 and unit variance,
and the sample size to detect a statistical significance of t can be derived as:
(Z
α + Z1− β )
+3
N=
1é
1
+
r
æ
öù
log
ç
÷ú
e
4 êë
1
−
r
è
øû
2
Sample
Sample size
size for
for estimating
estimating r:
r: example
example
• Example: According to the literature, the correlation between salt intake and
systolic blood pressure is around 0.3. A study is conducted to test the
correlation in a population, with the significance level of 1% and power of
90%. The sample size for such a study can be estimated as follows:
2
(
2.33 + 1.28)
N=
+ 3 = 87
1é
æ 1 + 0.3 öù
log
÷
eç
4 êë
è 1 − 0.3 øúû
• A sample size of at least 87 subjects is required for the study.
Sample
Sample size
size for
for difference
difference between
between two
two means
means
• Hypotheses:
Ho: µ1 = µ2 vs. Ha: µ1 = µ2 + d
• Let n1 and n2 be the sample sizes
for group 1 and 2, respectively; N
= n1 + n2 ; r = n1 / n2 ; σ:
standard deviation of the variable
of interest.
• Then, the total sample size is
given by:
2
æ
ö 2
(r + 1)ç Z + Z
÷ σ
1− β ø
è α
N=
rd 2
Where Zα and Z1-β are Normal deviates
• If we let Z = d / σ be the “effect
size”, then:
ö
(r + 1)æç Z + Z
÷
1
α
−
β
è
ø
N=
rZ 2
2
• If n1 = n2 , power = 0.80, alpha
= 0.05, then (Zα + Z1-β)2 = (1.96
+ 1.28)2 = 10.5, then the
equation is reduced to:
N=
21
Z2
Two
Two Populations
Populations
Sample
Sample size
size for
for two
two means
means vs.“effect
vs.“effect size”
size”
Total sample size (N)
2400
2000
1600
1200
800
400
0
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Effect size (d / s)
For a power of 80%, significance level of 5%
2
Sample
Sample size
size for
for difference
difference between
between 22 proportions
proportions
• Hypotheses:
Ho: π1 = π2 vs. Ha: π1 = π2 + d .
• Let p1 and p2 be the sample proportions (e.g. estimates of π1 and π2) for
group 1 and group 2. Then, the sample size to test the hypothesis is:
(
Z
n=
α
2 p(1 − p ) + Z
p (1 − p ) p (1 − p
( p − p )2
1− β
1
1
2
1
2
2
))2
Where: n = sample size for each group ; p = (p1 + p2) / 2 ; Zα and Z1-β are
Normal deviates
A better (more conservative) suggestion for sample size is:
ù
4
né
n = ê1 + 1 +
ú
4ë
np −p û
a
1
2
2
Sample
Sample size
size for
for difference
difference between
between 22 prevalence
prevalence
• For most diseases, the prevalence in the general population is small (e.g. 1
per 1000 subjects). Therefore, a difference formulation is required.
• Let p1 and p2 be the prevalence for population 1 and population 2. Then, the
sample size to test the hypothesis is:
n=
(Z
[0.00061(arcsin
α
+ Z1− β )2
p1 − arcsin p2 )
2
]
Where: n = sample size for each group; Zα and Z1-β are
Normal deviates.
Sample
Sample size
size for
for two
two proportions:
proportions: example
example
• Example: In a condition, the remission rate is expected to be 70% for a new
treatment, and 60% for a conventional treatment. A trial is planned to show
the difference at the significance level of 1% and power of 90%.
• The sample size can be calculated as follows:
– p1 = 0.6; p2 = 0.7; p = (0.6 + 0.7)/2 = 0.65; Z0.01 = 2.81; Z1−0.9 = 1.28.
– The sample size required for each group should be:
(
2.81
n=
2
2 × 0.65 × 0.35 + 1.28 0.6 × 0.4 + 0.7 × 0.3 )
≈ 759
2
(0.6 − 0.7 )
• Adjusted / conservative sample size is:
2
ù
4
759 é
n =
ê1 + 1 +
ú = 836
4 ë
759 0.6 − 0.7 û
a
Sample
Sample size
size for
for two
two proportions
proportions vs.
vs. effect
effect size
size
Difference from p1 by:
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
424
625
759
825
825
759
625
424
131
173
198
206
198
173
131
73
67
82
89
89
82
67
45
.
41
47
50
47
41
31
.
.
28
30
30
28
22
.
.
.
19
20
19
17
.
.
.
.
14
14
13
.
.
.
.
.
10
9
8
.
.
.
.
.
P1
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Note: these values are “unadjusted” sample sizes
Sample
Sample size
size for
for estimating
estimating an
an odds
odds ratio
ratio
• In case-control study the data are usually summarized by an odds ratio (OR),
rather then difference between two proportions.
• If p1 and p2 are the proportions of cases and controls, respectively, exposed
to a risk factor, then:
p (1 − p2 )
OR = 1
p2 (1 − p1 )
• If we know the proportion of exposure in the general population (p), the total
sample size N for estimating an OR is:
(1 + r )2 (Zα + Z1− β )2
N=
2
r (ln OR ) p(1 − p )
• Where r = n1 / n2 is the ratio of sample sizes for group 1 and group2; p
is the prevalence of exposure in the controls; and OR is the hypothetical
odds ratio. If n1 = n2 (so that r = 1) then the fomula is reduced to:
4(Zα + Z1− β )
2
N=
(ln OR )2 p(1 − p )
Sample
Sample size
size for
for an
an odds
odds ratio:
ratio: example
example
• Example: The prevalence of vertebral fracture in a population is 25%. It is
interested to estimate the effect of smoking on the fracture, with an odds
ratio of 2, at the significance level of 5% (one-sided test) and power of 80%.
• The total sample size for the study can be estimated by:
4(1.64 + 0.85)
= 275
N=
2
(ln 2) × 0.25 × 0.75
2
Sample
Sample size
size for
for 22 correlation
correlation coefficients
coefficients
• In detecting a relevant difference between two correlation coefficients r1
and r2 obtained from two independent samples of sizes n1 and n2,
respectively, we need to firstly transform these coefficients into z value as
follows:
æ 1 + r1 ö
÷÷
z1 = 0.5 log e çç
è 1 − r1 ø
æ 1 + r2 ö
÷÷
z 2 = 0.5 log e çç
è 1 − r2 ø
• The total sample size N required to detect the difference between two
correlation coefficients r1 and r2, with a significance level of α and power 1β, can be estimated by:
2
4(Zα + Z1− β )
N=
( z1 − z2 )2
Where Zα and Z1-β are Normal deviates
Sample
Sample size
size for
for two
two r’s:
r’s: example
example
• The sample size required to detect the difference between r1 = 0.8 and r2 =
0.4 with the significance level of 5% (two-tailed) and power of 80% can be
solved as follows:
– z1 = 0.5 ln ((1+0.4) / (1-0.4)) = 0.424
– z1 = 0.5 ln ((1+0.8) / (1-0.8)) = 1.098
4(1.96 + 1.28)
N=
2 = 92
(0.424 − 1.098)
2
• 46 subjects is needed in each group.
Some
Some Comments
Comments
•
•
•
•
•
The formulae presented are theoretical.
They are all based on the assumption of Normal distribution.
The estimator [of sample size] has its own variability.
The calculated sample size is only an approximation.
Non-response must be allowed for in the calculation.
Computer
Computer Programs
Programs
• Software program for sample size and power evaluation
– PS (Power and Sample size), from Vanderbilt Medical Center. This can
be obtained from me by sending email to ([email protected] or
[email protected]). Free.
• On-line calculator:
– http://ebook.stat.ucla.edu/calculators/powercalc/
• References:
– Florey CD. Sample size for beginners. BMJ 1993 May 1;306(6886):1181-4
– Day SJ, Graham DF. Sample size and power for comparing two or more treatment groups in
clinical trials. BMJ 1989 Sep 9;299(6700):663-5.
– Miller DK, Homan SM. Graphical aid for determining power of clinical trials involving two groups.
BMJ 1988 Sep 10;297(6649):672-6
– Campbell MJ, Julious SA, Altman DG. Estimating sample sizes for binary, ordered
categorical, and continuous outcomes in two group comparisons. BMJ 1995 Oct
28;311(7013):1145-8.
– Sahai H, Khurshid A. Formulae and tables for the determination of sample sizes and power in clinical trials
for testing differences in proportions for the two-sample design: a review. Stat Med 1996 Jan 15;15(1):1-21.
– Kieser M, Hauschke D. Approximate sample sizes for testing hypotheses about the ratio and difference of
two means. J Biopharm Stat 1999 Nov;9(4):641-50.