Trends in time: results from the NICE registry, PHJ van der Voort

Transcription

Netherlands Journal of Critical Care
Copyright © 2009, Nederlandse Vereniging voor Intensive Care.
All Rights Reserved. Received July 2008; accepted January 2009
orIGINAl
Trends in time: results from the NICE registry
PHJ van der Voort, F Bakhshi-raiez, DW de lange, rJ Bosman, E de Jonge, H Joore, r de Waal, rMJ Wesselink,
G van Berkel, r van Maanen, NF de Keizer
on behalf of the Dutch National Intensive Care Evaluation (NICE)Foundation.
Abstract - Introduction The NICE registry contains data from individual intensive care admissions concerning severity of illness and
outcome. This study was undertaken to analyze the trends in time related to crude mortality and severity of illness corrected mortality.
Methods Data on all patients admitted from January 1 1999 to January 1 2008 were used. Patients who did not fulfil the inclusion
criteria for the SAPS II prediction model were excluded. To analyze trends in time for individual ICUs, this study uses data from ICUs
that participated in the NICE registry for a minimum of 5 years between January 1, 2001 and January 1, 2008. Repeated cross-sectional
data analysis was performed for Standardized Mortality Ratio (SMR), and longitudinal data analysis using Variable Live Adjusted Displays
(VLAD both on individual and national level were used. Results Overall, 243 182 patients were included in the NICE registry. Of these,
144 696 met the SAPS II inclusion criteria and were included in this study. In the non-cardiac surgery population, the SAPS II model
was customized on 9227 admissions from 13 ICUs between 1999 and 2000 to obtain a baseline Dutch SMR of 1.0 for comparison of
subsequent years. In individual ICUs, the SAPS II model was customized on 15 863 admissions from 20 ICUs that participated in NICE
for a minimum of five years between 2001 and 2002 to obtain a baseline SMR of 1.0 for comparison of subsequent years. From 2001 to
2007 in the non-cardiac surgery population, a significant decrease in SMR was shown from 1.0 tot 0.83 (p=0.023). This decrease was
most significant in patients after elective surgery (p=0.01). Seven (25%) individual ICUs showed a significant decrease in their SMR.
The VLAD curve for individual ICUs showed a highly variable course. Conclusion Over time, the NICE registry showed a significant
improvement in SMR for all patient groups nationwide, and for 7 individual ICUs. Longitudinal analysis for individual ICUs by VLAD curve
shows a variable course between ICUs and within an individual ICU over time. This information can be used for quality improvement
initiatives both locally and nationwide.
Keywords
- NICE, mortality, quality of care, VLAD curve, intensive care
Introduction
Quality improvement in individual intensive care units (ICU)
benefits from a structured and continuous approach. Usually,
the plan-do-check-act (PDCA) cycle (Shewhart and Demming
cycle) is followed [1]. To guide this quality cycle, especially
in the check and act phases, it is necessary to collect data.
Continuous collection of data should show an improvement over
time on comparison with the previous situation. These changes
in results over time may represent an improvement or decline in
quality. Data obtained to measure potential changes in quality are
called quality indicators. Or, in other words: a quality indicator is
a screening tool to identify potentially sub-optimal clinical care
[2]. Quality indicators measure the structure, the process and the
outcome of care [3], and may serve as instruments to improve
health care [4]. Structure indicators are related to those resources
and means which enable treatment and care to be given. Process
Correspondence
PHJ van der Voort
Email: [email protected]
8
indicators refer to the activities related to treatment and care.
Outcome is defined as the state of health of a patient that can be
attributed to an intervention or to the absence of an intervention.
The Dutch National Intensive Care Evaluation (NICE) was
founded in 1996 according to the ideas of intensivists [5].
Over the 12 years that followed, an increasing number of ICUs
collected the Minimal Data Set (MDS) and exported these data to
NICE for purposes of benchmarking in a national database. This
MDS contains individual patient data on severity of disease on
admission and outcome data such as ICU and hospital mortality
and length of stay. As such, the NICE database primarily contains
outcome indicators. Well known limitations of outcome data are
that reasons for adverse outcomes are difficult to identify, and
that possibilities for direct improvement are limited. In addition,
they may stigmatize institutions and are subject to manipulation
of data [6]. To overcome some of these limitations, recently some
additional structure and process indicators have been defined by
the Dutch Society of Intensive Care, which began to be collected
in 2008 [7].
Currently available outcome indicators from the NICE
database can be used by intensivists and ICU managers in the
NETH J CRIT CARE - VOLUME 13 - NO 1 - FEBRUARY 2009
NJCC_01 v7 bwerk.indd 8
10-02-2009 13:11:13
following two ways. 1) The data from an ICU aggregated over a
period of time can be compared with aggregated data from other
ICUs and also with the national average over the same period of
time (cross-sectional analysis). 2) Data can be collected over time
to analyse change (longitudinal analysis). These changes can be
used for evaluation of local quality improvement programmes or
to identify unexpected changes in care. On an aggregated level,
longitudinal national improvement in care can be observed.
In general, NICE provides its participants with two different
mortality outcome indicators, the Standardized Mortality Ratio
(SMR) and the Variable Live Adjusted Displays (VLAD) curve.
The SMR gives an overall estimation of performance over a
certain period (cross-sectional) and can be used for benchmark
purposes. On the other hand, the VLAD curve enables cumulative
monitoring over a certain period of time (longitudinal), and can
be used to analyse changes over time in individual ICUs or
aggregated data. Care should apply to six quality domains as
defined by the Institute of Medicine and should therefore be safe,
effective, efficient, patient centered, timely, and equitable. This
report is a summary of analyses from the NICE database and
focuses on mortality as an indicator of effectiveness. It is focused
on trends in time for the general Dutch ICU population as well as
on trends in time in individual ICUs .
patients admitted to the participating ICUs. The data are for all
ICUs uniformly defined and collected [8]. The participating ICUs
are a mixture of tertiary, teaching and non-teaching settings in
urban and non-urban hospitals. The registry contains all variables
necessary to calculate, among other things, the APACHE II, SAPS
II and MPM24II prognostic models [9-12]. Since January 2007
the data for APACHE IV have also been collected.
As the aim of the study is to analyze trends in time, we wanted
to include data covering a long period of time and a large number
of admissions. Therefore, the analyses for the general non-cardiac
surgery population and for the individual ICUs were performed on
data over different time periods.
To analyze trends in time for the non-cardiac surgery
population, this study uses a dataset from the NICE registry with
data on all patients admitted from January 1 1999 to January 1
2008. To analyze trends in time for individual ICUs, the study uses
data from ICUs participating in the NICE registry for a minimum of
5 years between January 1, 2001 and January 1, 2008.
SAPS II prognostic model
The SAPS II model [10] is used for the calculation of case mixadjusted mortality risks. The selection of the SAPS II model for
this study is based on the fact that the SAPS II model has been
demonstrated to have the best fit to the Dutch ICU population [13].
For calculation of the mortality risks, the exclusion criteria defined
by the SAPS II model were applied to the data: all re-admissions
within the same hospital stay, admissions after cardiac surgery,
burns, missing information on admission type, and those younger
than 18 years, were excluded from the analysis.
Materials and Methods
Data
In 1996 the Dutch National Intensive Care Evaluation (NICE)
foundation started collecting data on patients admitted to
Dutch ICUs. The NICE database contains observational data on
Table 1. Baseline demographics in consecutive years
1999
2000
2001
2002
2003
2004
2005
2006
2007
Number of admissions
10843
12910
14387
20703
25224
31471
35925
44621
48302
Number of ICUs
9
21
22
32
33
36
48
58
59
Male (%)
65.8
64.9
64.7
61.8
62.4
61.7
61.6
60.8
59.8
Medical (%)
21.3
25.0
28.3
33.4
34.7
34.8
35.5
38.0
41.5
Urgent surgery (%)
11.2
11.0
12.1
16.2
14.0
13.7
13.7
15.4
15.8
Elective surgery (%)
67.6
64.1
59.6
50.4
51.3
51.5
50.8
46.6
42.7
Age (mean)
61.7
61.8
61.6
61.3
62.3
62.2
62.8
62.8
62.8
Age Survivors (mean)
61.3
61.0
60.6
60.2
61.3
61.2
61.7
61.6
61.5
Age Non-survivors (mean)
65.0
66.3
67.0
67.1
67.8
67.7
68.5
68.9
69.5
ICU Length of stay (median)
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
ICU mortality (%)
8.1
9.7
10.6
10.5
11.0
9.9
10.1
10.5
11.0
Hospital mortality (%)
12.4
14.1
15.6
15.6
16.1
14.8
15.6
16.1
16.7
Number of admissions
included for SAPS II
3909
5585
7383
11960
14762
18098
21851
28147
32101
SAPS II Score (mean)
34.8
35.1
34.1
33.5
34.7
34.5
34.3
34.6
34.78
SAPS II Probability (mean)
0.24
0.24
0.23
0.23
0.24
0.24
0.23
0.24
0.24
SMR based on original SAPS II model
1.03
1.02
1.04
0.91
0.90
0.87
0.88
0.82
0.83
Admission type:
9
10-02-2009 13:11:13
PHJ van der Voort, F Bakhshi-Raiez, DW de Lange, RJ Bosman, E de Jonge, H Joore, R de Waal, RMJ Wesselink,
G van Berkel, R van Maanen, NF de Keizer
Previous studies have shown that prognostic models may
produce unreliable predictions and consequently unreliable
impressions about health care performance if, in an external
population, prognostic reliability is poor [14]. Poor model
prognostic reliability can be avoided by customizing models to
the population concerned [15,16]. As a consequence, in order
to measure the real changes in ICU performance and not the
shortcomings of the performance of the model, the SAPS II
model was customized to the Dutch ICU population. In order to
do this, we used first level customization in which a new logistic
regression equation is fitted with observed in-hospital death as the
dependent variable and the logit-transformed original prediction
as the independent variable [17]. First level customization does
not change the influence of individual covariates included in
the model, but calibrates their joint influence to the observed
mortality in the external dataset.
To customize the SAPS II model to the non-cardiac surgery
population, we used data on all admissions between 1999 and
2001.
In 2002 and 2003, a large number of smaller ICUs with a slight
difference in case mix started participating in the NICE registry.
Applying the customized SAPS II model to analyze trends in time
for these individual ICUs was not suitable as the baseline SMR
was no longer 1.0. Therefore, to include the maximum number
of ICUs participating in the registry for a minimum of five years
for individual analysis, and to obtain a baseline SMR of 1.0, the
SAPS II was also customized for individual ICUs participating in
the NICE registry for a minimum of five years. To customize the
SAPS II model for this part of the analysis, we used data on all
admissions from these ICUs between 2001 and 2002.
SMR trend
The SMR with 95% confidence interval is calculated by dividing the
observed in-hospital mortality to the predicted case mix-adjusted
mortality. To assess the trend in SMR in the general population, a
weighted linear regression was fitted using “log(SMR)” as dependent
variable and the calendar-years as covariate. Considering yearly
changes in the standard error of the “log(SMR)”, “1/standard error”
was added to the model as weight.
To analyze the SMR trend for individual ICUs, again, a
weighted linear regression was fitted using “log(SMR)” as
dependent variable and the calendar-years as covariate and “1/
standard error” was added to the model as weight.
VLAD curve
For the calculation of the VLAD curve, the individual SAPS IIpredicted mortalities (pmortality) for each survivor were added, and
1 – pmortality was subtracted for each patient who died during
hospital stay. If a patient with a high probability of survival survived,
a low positive score was awarded, whereas if the same patient died,
Figure 1. Distribution of reasons for admission to ICU.
The SAPS II model was customized for the general non-cardiac surgery population on 9227 admissions from 13 ICUs between 1999 and 2000.
For individual ICUs, the SAPS II model was customized on 15 863 admissions from 20 ICUs between 2001 and 2002.
From January 1st 2001 to January 1st 2008, 22 1028 patients were admitted to 59 Dutch ICUs. The analysis of trends in time for the non-cardiac surgery
population was based on 135 469 of these admissions who met the inclusion criteria for SAPS II model.
Twenty-eight ICUs with a total of 129,770 admissions participated in the NICE registry for a minimum of five years between January 1 2003 and
January 1 2008. The analysis of trends in time for the individual ICUs was based on 77 394 admissions from those ICUs who met the inclusion criteria
for SAPS II model.
10
10-02-2009 13:11:15
a high negative score would be awarded. The cumulative survival
curve was obtained by summations of these scores over time.
Analysis
The analyses for the non-cardiac surgery population were further
stratified for the different admission types, i.e. medical, urgent
surgery and elective surgery. Continuous values were expressed
as means ± SD and categorical values were expressed in
absolute and relative frequencies. Crude mortality is presented
as a percentage of ICU admissions. In all cases, a two-sided p
value of 0.05 was defined as statistically significant. SPSS 14.0
for Windows was used to perform the statistical analyses.
results
The demographics of the overall population in each calendar
year are shown in Table 1. Figure 1 provides the distribution of
the reasons for ICU admission per year.
SMR
Figures 2 a-d show the annual SMR based on the customized
SAPS II model for the total non-cardiac surgery population and
for medical, emergency surgery and elective surgery patients
separately.
In seven ICUs the SMR decrease was significant. The number
of admissions in the database for some ICUs was too small to
demonstrate a statistical significant decrease in SMR. In hospital
Figures 2 a-d. Bars represent hospital mortality
Figure 2a: Crude mortality and SMR (+/-95%CI) per calendar year for non-
Figure 2c: Crude mortality and SMR (+/-95%CI) per calendar year for
cardiac surgery population
emergency surgical patients
Figure 2b: Crude mortality and SMR (+/-95%CI) per calendar year for
Figure 2d: Crude mortality and SMR (+/-95%CI) per calendar year for
medical patients
elective surgical patients
Linear regression analysis of these data showed a significant association between a decreasing SMR with increasing calendar years in all groups. The results are presented in Table 2. The decrease in SMR is highest for elective surgical patients.
Table 2. Results of weighted linear regression analysis with “log(SMR)” as dependent variable and the calendar-years as covariate.
COEFFICIENT OF CALENDAR YEAR
P-VALUE
Non-cardiac surgery population
-0.014
0.020
Medical patients
-0.011
0.020
Emergency surgery patients
-0.015
0.010
Elective surgical patients
-0.024
0.023
11
10-02-2009 13:11:16
10, the SMR decreased(!) from around 0.72 in the first three years
to 0.23 in 2007. This decrease was not significant (Coefficient of
calendar year = -0.065). A closer look at the data showed that
the decrease reflected a change in data gathering method rather
than a change in care performance, i.e. the hospital started using
a new PDMS at the end of 2006 and had difficulties with data
collection and extraction resulting in incorrectly low SMRs. None
of the ICUs showed a significant increase in SMR over time.
Table 3: Results of weighted linear regression analysis with
“log(SMR)” as dependent variable and the calendar years as
covariate. * indicates a P value of < 0.05.
Discussion
This study shows that on a national level the SMR is following a
gradual but significant decrease. Apparently, the quality of care,
measured by the outcome indicator mortality, is improving.
However, we do realize, that mortality is not the only or the ideal
way to evaluate quality of care. It is also shown that over the
years, an increasing number of ICUs have been participating in
the database and the observed decline in SMR may be caused
by the inclusion of ICUs with an SMR lower than that of the
ICUs that participated in the NICE registry earlier . This may be
caused by change in case mix due to the increasing number
Figures 3 a-d. give the VLAD curves for the general population and for individual admission types.
Figure 3 a: VLAD curve for non-cardiac surgery population
Figure 3 c: VLAD curve for emergency surgery patients
Figure 3 b: VLAD curve for medical patients
Figure 3 d: VLAD curve for elective surgical patients
As far as the individual ICUs are concerned, the extend of the SMR decrease seems to be ICU specific. Figure 4 shows per hospital per calendar year the
SMR based on the customized SAPS II model. The results of the weighted linear regression analysis for these hospitals are presented in Table 3.
Figure 4. Crude mortality and SMR (+/-95%CI) per hospital (numbers 1 -28) per calendar year
12
10-02-2009 13:11:18
of participants. For example, it was shown that the decrease
in SMR is most apparent in elective surgery patients. Other
explanations for the gradual decrease of SMR could be an
improvement in perioperative care and an improvement in postICU care.
Longitudinal analysis of the data of individual ICUs as
represented in a VLAD curve gives additional information as
it provides insight into the SMR trend over time. It not only
shows when the SMR increases or decreases, but also shows
when this change occurred. Also in ICUs with a non-significant
change in SMR, the VLAD curve can give information about
the performance. However, the slope of the VLAD curve
depends on the number of patients included in the analysis. As
a consequence, VLAD curves cannot be used for quantitative
comparison between individual ICUs (benchmarking). For this
reason, we have shown four representative VLAD curves. In our
analysis, the SAPS II model was used to correct for severity of
illness. Previously, we have shown that the SAPS II model is the
best fit to the Dutch population [13]. As the APACHE IV model
has been applied to the data provided by the NICE participants
since 2007, it is expected to replace the SAPS II model within a
few years.
The VLAD curves of the participating ICUs based on the
SAPS II data show variable slopes. Both increasing and
decreasing lines can be found indicating variable performance
between ICUs or indicating different case mixes as there are
case mix differences between ICUs which were not corrected
for by the SAPS II model. Lead-time bias for example, referring
to the treatments given just before ICU admission, may influence
predicted mortality and SMR, and was not corrected for in SAPS
II model [18].
A decreasing line might be an indication of decreasing
performance, although the SMR may still be below 1, but should
always lead to profound analysis. Structure and process data
can be used to obtain explanations. As shown in the VLAD
curves, a variable slope over time does occur as well. This
phenomenon should alert ICU managers to the necessity of
analyzing the cause. A change in case mix of the population
Table 3. Results of weighted linear regression analysis with “log(SMR)” as dependent variable and the calendar years as covariate.
* indicates a P value of < 0.05.
ICU NUMBER
ICU NUMBER
1
-0.010
15
0.005
2
-0.021 *
16
0.841
3
-0.059 *
17
-0.026 *
4
-0.022
18
0.002
5
0.006
19
-0.012
6
-0.005
20
-0.016 *
7
-0.005
21
-0.021
8
-0.013
22
-0.032
9
-0.027 *
23
0.004
10
-0.065
24
-0.024
11
-0.029
25
-0.002
12
-0.046
26
0.022
13
0.0208
27
-0.032 *
14
-0.015
28
-0.021 *
Not all VLAD curves for individual ICUs are presented here for reasons of space. The VLAD curves for four individual ICUs are shown in Figures 5a-d. Panel
A shows the VLAD curve for hospital 26 with an SMR of around one and a temporary slight increase of the VLAD curve over recent years. Panel B shows
the VLAD curve for hospital 15 with a stable period, then a decrease with an SMR of more than 1 and then again a stable period with an SMR of around 1
and then a decrease of the curve and an SMR of more than 1. Panel C shows the VLAD curve for hospital 2 with a stable increase of the curve corresponding with an SMR of less than 1 or a lower mortality than expected. Panel D shows the VLAD curve for hospital 21 initially with a stable curve corresponding
with a SMR of 1 but a decreasing curve in 2004 and 2006 corresponding with a SMR higher than 1 and an increasing curve in the last year corresponding
with a SMR below 1.
13
10-02-2009 13:11:18
is rarely the underlying cause but an intensive search for other
reasons should be made to identify a cause that might have
changed the slope in upward or downward (e.g. hospital 10
Results section). The quarterly feedback that is provided by
NICE may be helpful but usually more profound analysis of local
data is necessary. In addition, Statistical Process Control charts
that will be provided in the future may help in deciding whether
chance may play a role [19,20].
The extended NICE database with structure, process and
outcome indicators combined with TISS data and predefined
complications which are currently under construction, will
provide care givers and ICU managers with the information
to improve the quality of care. The local PDCA cycle can
continuously receive input from the NICE database. Local and
national quality projects as well as local and national research
can benefit from this database.
Conclusion
Over time, the aggregated SMR for all participating ICUs in the
NICE registry has declined. However, this may be due to a change
in case mix due to the increasing number of participating ICUs.
Individual analysis of outcome data over time (VLAD) shows
that some ICUs consistently perform better than expected but
that other ICUs perform consistently or occasionally worse than
expected. In some ICUs, there is a significant decrease in SMR
over time.
The NICE feedback for individual ICUs with their own data
compared with national data may indicate an inadequate level of
care, and should be used as input for local quality initiatives as
well as quality improvement on a national level.
Figure 5a-d
Figure 5a: SMR and VLAD curve for IC 26
Figure 5c: SMR and VLAD curve for IC 2
Figure 5b: SMR and VLAD curve for IC 15
Figure 5d: SMR and VLAD curve for IC 21
14
10-02-2009 13:15:47
references
1
http://en.wikipedia.org/wiki/PDCA
12 Zimmerman JE, Kramer AA, McNair DS, Malila FM: Acute Physiology and Chronic
2
AHQR. Guide to patient safety indicators. 2003. Agency for health care research
Health Evaluation (APACHE) IV: Hospital mortality assessment for today’s critically ill
and quality.
patients. Crit Care Med 2006 ;34(5):1297-310.
3
13 Peek N, Arts DG, Bosman RJ, van der Voort PH, de Keizer NF. External validation of
Skews G. Meehan T. Hunt G. Hoot S. Armitage P. Development and validation of
clinical indicators for mental health nursing practice. Aust N Z J Ment Health Nurs 2000;
prognostic models for critically ill patients required substantial sample size. J Clin Epide-
9(1):11-18.
miol 2007;60:491-501.
4
14 Iezzoni LI: The risks of risk adjustment. JAMA 1997 ;278(19):1600-7.
Donabedian A. An introduction to quality assurance in health care. 2003. New York.
Oxford university press.
15 Teres D. Lemeshow S: When to customize a severity model. Intensive Care Med
5
1999 ;25(2):140-2.
Dutch National Intensive Care Evaluation (NICE) foundation. http://www stichting-
nice nl/introductie jsp. Accessed 20th June 2008.
16 van Houwelingen HC: Validation. calibration. revision and combination of prognostic
6
survival models. Stat Med 2000;19(24):3401-15.
Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process
and outcome data in managing performance of acute medical care: avoiding institutional
17 Moreno R, Apolone G: Impact of different customization strategies in the perfor-
stigma. Lancet 2004;363:1147-54
mance of a general severity score. Crit Care Med 1997;25(12):2001-8.
7
18 Tunnell RD, Millar BW, Smith GB: The effect of lead time bias on severity of illness
Commissie kwaliteitsindicatoren NVIC. Interne indicatoren voor Intensive Care
afdelingen: een continue nationale registratie ten behoeve van kwaliteitsverbetering. Neth
scoring, mortality prediction and standardised mortality ratio in intensive care--a pilot
J Crit Care 2006;10:595-600
study. Anaesthesia 1998;53(11):1045-53.
8
19 Benneyan JC, Lloyd RC, Plsek PE: Statistical process control as a tool for research
Arts D, de Keizer N, Scheffer GJ, de Jonge E: Quality of data collected for severity
of illness scores in the Dutch National Intensive Care Evaluation (NICE) registry. Intensive
and healthcare improvement.Qual Saf Health Care 2003;12:458-464.
Care Med 2002;28(5):656-9.
20 Spiegelhalter D, Grigg O, Kinsman R, TreasureT: Risk-adjusted sequential probability
9
ratio tests: applications to Bristol, Shipman and adult cardiac surgery. International Jour-
Knaus WA, Draper EA, Wagner DP, Zimmerman JE: APACHE II: a severity of disease
classification system. Crit Care Med 1985;13(10):818-29.
nal for Quality in Health Care 2003;15:7-13
10 Le Gall JR, Lemeshow S, Saulnier F: A new Simplified Acute Physiology
Score (SAPS II) based on a European/North American multicenter study. JAMA
1993;270(24):2957-63.
11 Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J: Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients.
JAMA 1993;270(20):2478-86.
15
10-02-2009 13:11:19

Trends in time: results from the NICE registry, PHJ van der Voort

Transcription

Similar documents

Vlad Tepes

Nov 2015 – Minutes – Vlad Rep - Lancaster Herpetological Society

JUNIORS 2015/16 - Baden Powell Cricket Club

CHECKLIST-ICU Trial Checklist during multidisciplinary daily visits

Abdominal Sacrocolpopexy with possible removal of ovaries

Camper, Arctic Cat 650 Prowler, Cars, High Quality Furnishings

PE041 Tonsil and Adenoid Surgery

Heritage Surgery Center

May 12, 2013

saint john`s hospital - Hungarian Medical Care company for the