Experiences with Interim Analyses and Data Monitoring Committees

Experiences with Interim Analyses
and Data Monitoring Committees
Workshop on Design and Analysis of Clinical trials
28th October 2011
Simon Day PhD
Experiences with Interim Analyses
and Data Monitoring Committees
Or: I’d never be so stupid as to do that, would I?
… an aside
• Simple (trivial?) teaching examples
• Option 1
– Take a large database (of anything, but hopefully
Butis what’s
the point?
something that
interesting)
This
just
makes statistics look like
– Sample
from
it (properly!)
something for its own sake
– Calculate summary statistics, point and interval
estimates, maybe even P-values
– Compare sample results to “true” results
3
… an aside
• Simple (trivial?) teaching examples
• Option 2
– Take a real problem (again anything, but hopefully
something Now
that isstatistics
interesting)is useful.
But(properly!)
how do we know if we
– Collect data
have the “right” answer?
– Calculate summary statistics, point and interval
estimates, maybe even P-values
– Declare this to be the correct result
4
Bias and Precision
5
Bias and Precision
6
So our purpose…
• Is to find the answers to questions to which we
don’t currently know the answers
– And we typically have little means to verify them
7
“Don’t ever do interim analyses”
Day, S (200?)
Unconfirmed report
8
“plans or decisions based on statistically imprecise
interim data may often be suboptimal”
Section 6.5
“Sponsor access to
interim data for
planning purposes”
9
One company’s approach…
“Except for studies with a multi-stage analysis
design (e.g. Simon two-stage design), interim
efficacy analyses are generally not recommended
for phase II studies -- even when done solely for
internal decision making. Such analyses can be
highly problematic due to the inherent
uncertainty from small sample sizes, and biases
that can result from incomplete follow-up or
unclean data.”
10
Example – tifacogin in severe sepsis
11
Design and objectives
• Double blind, placebo-controlled, multicentre,
phase 3 randomised controlled clinical trial
• Primary outcome – all cause 28 day mortality
• Target sample size 1550
(Increased by 400 at an interim analysis)
12
Interim analysis, N=722
• Mortality rates:
placebo 38.9%
tifacogin 29.1%
P=?
13
Interim analysis, N=722
• Mortality rates:
placebo 38.9%
tifacogin 29.1%
P = 0.
14
Interim analysis, N=722
• Mortality rates:
placebo 38.9%
tifacogin 29.1%
P = 0.006
15
Interim analysis, N=722
• Mortality rates:
placebo 38.9%
tifacogin 29.1%
P = 0.006
• “Tifacogin significantly attenuated prothrombin
fragment and thrombin:anti-thrombin complex
levels [secondary endpoints] in patients with high
and low INR [pre-specified subgroups]”
16
What should happen?
• Pre-specified interim analysis at ¼, ½ and ¾ of
patients completing 28 days
• Purpose: “safety, futility and efficacy”
• Safety and efficacy – kind of the same thing
• Tifacogin is effective – more patients are dying on
placebo
17
What should happen?
• If this trial continues, patients randomised
to the control arm will needlessly die
• Another 1100 patients due to be randomised
• We should order 160 coffins for patients who
will be given tifacogin
215 coffins for patients
who will be given placebo
• We should order
18
“Phew,
I got the real stuff!”
19
20
Example – MRC acute myeloid leukemia
The MRC AML12 trial
21
The trial
•
•
•
•
RCT 5 courses of therapy versus 4 courses
Primary endpoint – survival
1078 patients
Yearly interim analyses
22
Interim analysis (2)
• Hazard ratio 0.47 (in favour of 5 courses)
• 95% confidence interval 0.29 – 0.77, p=0.003
23
Interim analysis (3)
• Hazard ratio 0.47 (in favour of 5 courses)
• 95% confidence interval 0.29 – 0.77, p=0.003
• Hazard ratio 0.55 (in favour of 5 courses)
• 95% confidence interval 0.38 – 0.80, p=0.002
24
“That’ll be
five courses for me,
please, waiter”
25
26
Example – the CAPRICORN study
Lancet 2001; 357:1385–90.
27
Primary endpoint
28
New (changed) endpoint
• All-cause mortality
• At an interim analysis, changed to:
– All-cause mortality (p<0.005)
Or
– All-cause mortality or cardiovascular hospitalisation
(p<0.045)
29
Results
All-cause mortality
All-cause mortality or
Cardiovascular hospitalisation
Relative risk 0.77 (0.60 – 0.98)
p=0.031
Relative risk 0.92 (0.80 – 1.07)
p=0.296
30
Results
31
Some issues in CAPRICORN:
•
•
•
•
•
What did the DMC do?
Why did they do it?
Should they have done it?
Could someone else have done it?
Should someone else have done it?
32
Example – tifacogin in severe sepsis
33
Interim analysis, N=722
• Mortality rates:
placebo 38.9%
tifacogin 29.1%
P = 0.006
• “Tifacogin significantly attenuated prothrombin
fragment and thrombin:anti-thrombin complex
levels [secondary endpoints] in patients with high
and low INR [pre-specified subgroups]”
34
What did happen?
• They kept the trial going
(Yes, and people died!)
• How many died?
High INR
placebo 296 (33.9%)
tifacogin 301 (34.2%)
Low INR
118 (23%)
83 (12%)
P = 0.?
35
What did happen?
• They kept the trial going
(Yes, and people died!)
• How many died?
High INR
placebo 296 (33.9%)
tifacogin 301 (34.2%)
Low INR
118 (23%)
83 (12%)
P = 0.051
Pearson χ2
36
What did happen?
• They kept the trial going
(Yes, and people died!)
• How many died?
High INR
placebo 296 (33.9%)
tifacogin 301 (34.2%)
Low INR
118 (23%)
83 (12%)
P = 0.03
logistic regression
37
Conclusion
“plans or decisions based on statistically imprecise
interim data may often be suboptimal”
38
Example – MRC acute myeloid leukemia
The MRC AML12 trial
39
Interim analysis (3)
• Hazard ratio 0.47 (in favour of 5 courses)
• 95% confidence interval 0.29 – 0.77, p=0.003
• Hazard ratio 0.55 (in favour of 5 courses)
• 95% confidence interval 0.38 – 0.80, p=0.002
40
What should happen?
• Again, the DMEC (sic) kept the trial going, despite
endpoint being death
• From the paper:
“The main reason for not closing the randomisation
was that the treatment effects (53% and 45%
reduction in odds of death) were considered too
large to be clinically plausible.”
• They were Bayesians!
41
The sequence of events
“plans or decisions based
on statistically imprecise
interim data may often be
suboptimal”
42
The sequence of events
43
So the questions to ask yourselves
1. “That trial we stopped early… what would have
happened if we had kept going?”
2. “That fixed sample size trial that we did… what
would have happened if it recruited more
patients?”
44
Experiences with Interim Analyses
and Data Monitoring Committees