How to Evaluate new Instruments

How to Evaluate new
Instruments
Carol Briggs
Department of Haematology
University College London Hospitals
ICSH Session, Toronto, May 2013
Levels of evaluation
•  Complete operational evaluation
This should be done at a national evaluation centre or an independent
organisation
•  Local laboratory evaluation/validation
An abbreviated assessment appraising limited aspects of the equipment
in the its intended location and by staff who will be operating the
instrument
Complete operational evaluation
•  Type of evaluation
–  National (e.g. FDA,)
–  Beta site
Planning the Evaluation
•  Analyser availability
•  Time required
•  Staff time
•  Reagent quantities/Batches
•  Training
•  Contingency plan
Preliminary assessment
•  Requirements: Space, power, operating
environment…
•  Safety
–  Mechanical
–  Electrical – IEC 61010-1
–  Chemical
–  Microbiological – aerosol, splashing, waste disposal
National/Complete evaluation
• 
• 
• 
• 
• 
• 
• 
Precision/Reproducibility
Carryover
Linearity
Stability
Comparability (accuracy)
Efficiency
Storage of results and quality control
Precision testing
1. Within run precision.
The method usually consists of a single run of 20 measurements
on the same sample, all reported parameters should be
analysed. Normal, abnormal low and abnormal high samples for
WBC, Hb and platelets should be sought.
The results for mean, SD and CV% should be reported.
Precision testing
1
2
3
4
5
6
7
8
9
10
MEAN
SD
2SD
CV
MIN
MAX
DIFF
WBC
4.63
4.54
4.57
4.53
4.49
4.48
4.55
4.57
4.33
4.44
4.51
0.08
0.16
1.76
4.33
4.63
0.30
NEUT#
2.6
2.62
2.6
2.58
2.52
2.53
2.52
2.64
2.41
2.41
2.54
0.08
0.15
3.04
2.41
2.64
0.23
LYMPH#
1.45
1.4
1.46
1.42
1.4
1.39
1.41
1.29
1.36
1.4
1.40
0.05
0.09
3.23
1.29
1.46
0.17
MONO#
0.39
0.34
0.35
0.38
0.38
0.39
0.44
0.45
0.38
0.41
0.39
0.03
0.07
8.44
0.34
0.45
0.11
EOSI#
0.16
0.15
0.13
0.13
0.16
0.14
0.14
0.14
0.14
0.17
0.15
0.01
0.03
8.77
0.13
0.17
0.04
BASO#
0.03
0.03
0.03
0.02
0.03
0.03
0.04
0.05
0.04
0.05
0.04
0.01
0.02
26.34
0.02
0.05
0.03
RBC
4.22
4.3
4.27
4.3
4.31
4.26
4.28
4.33
4.28
4.28
4.28
0.03
0.06
0.67
4.22
4.33
0.11
HGB
12.9
13.1
13.1
12.9
13.1
12.9
12.9
12.9
12.9
13
12.97
0.09
0.18
0.69
12.90
13.10
0.20
HCT
41.1
41.9
41.6
41.8
41.9
41.4
41.6
42
41.6
41.5
41.64
0.26
0.52
0.62
41.10
42.00
0.90
MCV
97.4
97.4
97.4
97.2
97.2
97.2
97.2
97
97.2
97
97.22
0.14
0.28
0.14
97.00
97.40
0.40
MCH
30.6
30.5
30.7
30
30.4
30.3
30.1
29.8
30.1
30.4
30.29
0.27
0.54
0.89
29.80
30.70
0.90
MCHC
31.4
31.3
31.5
30.9
31.3
31.2
31
30.7
31
31.3
31.16
0.24
0.47
0.76
30.70
31.50
0.80
PLT
203
211
207
207
216
211
214
201
199
201
207.00
5.60
11.21
2.71
199.00
216.00
17.00
Precision testing
2. Between batch precision.
Between batch precision may be affected by calibration or drift. A single
measurement on a sample repeated each day for a period 20-30 days
is used to measure total between day (batch) precision for all
parameters
Carryover
•  Carry-over is defined as the contamination of a sample by
the sample analysed immediately preceding it.
•  Carry-over from a high sample to a low sample should be
assessed by running sample A (high sample) three times,
A1, A2, A3, followed by sample B (low sample) three times
B1, B2, B3.
Carry-over % is calculated by: B1 – B3
A3 – B3 x 100
Linearity
•  There should be a linear relationship between the parameter
measured and at various dilutions over a large a range as possible.
•  Dilutions should be chosen to include the entire pathological range,
from the highest counts to the lowest.
•  In laboratories where very low WBC and platelet counts are
encountered it is advisable to examine linearity in the low range
separately.
•  Replicate tests should be performed to give results at evenly spaced
concentrations.
•  Certain haematological parameters, e.g. red cell indices will not be
affected by dilution of a sample.
Linearity
WBC LINEARITY (100)
PLT-O LINEARITY (4000)
y = 23.766x - 0.0779
R2 = 0.9996
120
PLT-O X 10^9/L
WBC X 10^9/L
100
80
60
40
20
0
0.0
1.0
2.0
3.0
4.0
4500
4000
3500
3000
2500
2000
1500
1000
500
0
5.0
y = 358.52x - 124.48
R2 = 0.9955
0.0
2.0
HB CONCENTRATION g/dL
10.0
12.0 14.0
y = 5.1884x + 0.6034
R2 = 0.9964
60
50
PLT-O X 10^9/L
WBC X 10^9/L
8.0
LOW PLT-O LINEARITY.
y = 0.0956x - 0.0101
R2 = 0.9915
0.8
6.0
HB CONCENTRATION g/dL
LOW WBC LINEARITY
1
4.0
0.6
0.4
0.2
0
40
30
20
10
0
0
2
4
6
8
HB CONCENTRATION g/dL
10
12
0.0
2.0
4.0
6.0
8.0
HB CONCENTRATION g/dL
10.0
12.0
Linearity
NRBC LINEARITY
y = 0.8952x - 0.0532
R2 = 0.9971
NRBC x 10^9/L
80
60
40
20
0
0
100
90
80
70
60
50
40
30
20
10
0
10 20 30 40 50 60 70 80 90 100
y = 0.7603x + 0.6831
R2 = 0.7872
0
Hb CONCENTRATION
y = 0.9769x + 0.5791
R2 = 0.7557
11
10
9
8
7
6
5
4
3
2
1
0
0
1
10 20 30 40 50 60 70 80 90 100
Hb Concentration
WBC Linearity on NRBC Sampes.
WBC x 10^9/L
NRBC X 10^9/L
100
NRBC Linearity
2
3
4
5
6
7
Hb Concentration
8
9
10
Linearity
Flow v Analyser 2 NRBC/100 WBC
700
700
600
600
Analyser 2 NRBC/100 WBC
Analyser 1 NRB C/100 WBC
Flow v Analyser 1 NRBC/100 WBC
y = 0.8723x + 1.8398
R2 = 0.9862
500
400
300
200
100
0
y = 0.5419x + 20.971
R2 = 0.6299
500
400
300
200
100
0
0
100
200
300
400
500
Flow NRBC/100 WBC
600
700
0
100
200
300
400
Flow NRBC/100 WBC
500
600
700
Linearity
Sample Stability
•  Ability of a sample to retain the same value for a
measured quantity for a defined period.
•  5 normal and 5 abnormal samples should be
analysed at time zero and then analysed over time
for up to 72 hours.
•  Samples should be stored at room temperature
and 40C.
Sample Stability
100
2
90
3
80
4
70
5
60
MCV LONG TERM STABILITY 4 DEG C.
ABNORMALS
1
2
3
4
5
80
75
70
65
60
55
50
45
40
65
33
60
70
60
50
3
4
5
MCHC pg
80
2
32
31
30
29
28
1
2
3
4
5
RDW-SD
34
100
1
12
1
11
Series1
2
10
Series2
9
Series3
8
Series4
7
Series5
3
4
5
6
MPV LONGTERM STABILITY
4 DEG C. ABNORMALS
RDW-SD LONG TERM STABILITY
4 DEG C. ABNORMALS
MCHC LONG TERM STABILITY 4 DEG
C. ABNORMALS
110
90
MCV fl
35
34
33
32
31
30
29
28
27
26
25
MPV fL
1
RDW-SD fL
110
MCHC g/dL
MCV fl
120
MPV LONGTERM STABILITY
ROOM TEMP ABNORMALS
RDW-SD LONG TERM STABILITY ROOM
TEMP. ABNORMALS
MCHC LONG TERM STABILITY ROOM TEMP.
ABNORMALS
1
55
2
50
3
45
40
12
11
4
5
MPV fL
MCV LONG TERM STABILITY ROOM TEMP.
ABNORMALS
Series1
10
9
8
Series2
7
6
Series5
Series3
Series4
Comparability
Comparability rather than accuracy
• 
• 
Accuracy implies a ‘true’ value
Not applicable to many tests
• 
Reference values are available for
- Hb
- RBC
- PCV
- WBC
- Retics
- Leucocyte Differential
- Platelets
Reference platelet count
Reference platelet count
Platelets
Debris
Plt/RBC Coincidence
RBC
Reference Leucocyte Differential
•  This is the CLSI H20-A2 method
•  200 x 200 cell differential
•  Referee performs a 200 cell differential if there is
disagreement between the first differentials
Reference Leucocyte Differential
• 
Classification of cells subjective
Rümke Table – 95% CI
a
n=100
n=200
n=500
n=1,000
n=10,000
0
0-3.6
0-1.8
0-0.7
0-0.4
0-0.1
1
0.0-5.4
0.1-3.6
0.3-2.3
0.5-1.8
0.8-1.3
5
1.6-11.3
2.4-9.0
3.3-7.3
3.7-6.5
4.5-5.5
10
4.9-17.6
6.2-15.0
7.5-13.0
8.2-12.0
9.4-10.7
15
8.6-23.5
10.4-20.7
12.0-18.4
12.8-17.4
14.3-15.8
20
12.7-29.2
14.7-26.2
16.6-23.8
17.6-22.6
19.2-20.8
30
21.2-40.0
23.7-36.9
26.0-34.2
27.2-32.9
29,1-31,0
40
30.3-50.3
33.2-47.1
35.7-44.4
36.9-43.1
39.0-41.0
50
39.8-60.2
42.9-57.1
45.5-54.5
46.9-53.1
49.0-51.0
70
60.0-78.8
63.1-76.3
65.8-74.0
67.1-72.8
69.0-70.9
80
70.8-87.3
73.8-85.3
76.2-83.4
77.4-82.4
79.2-80.8
90
82.4-95.1
85.0-93.8
87.0-92.5
88.0-91.8
89.3-90.6
100
96.4-100
98.2-100
99.3-100
99.6-100
99.9-100
ICSH
Reference flow differential – Dr Brent Wood, USA and Mikael
Roussel, France
• 
Replace morphology (CLSI H20-A2) with flow cytometry as the
reference method for performing nucleated cell identification in blood
• 
Must be able to be used on common instruments in use in the
clinical laboratory
• 
5 colour or more the requirement
Candidate Protocols
• 
French
–  Faucher, et al. Cytometry A (2007) 71A; 934-944
–  5 color single tube
–  Lyse (Versalyse), no-wash
• 
Swedish
–  Bjornsson, et al. Cytometry B (2008) 74B: 91-103
–  5 color single tube
–  Lyse (formic acid), no-wash
• 
Seattle
–  Wood, et al. Unpublished
–  5 color dual tube
–  Lyse (NH4Cl+formaldehyde or Versalyse), no-wash
Core Populations
French
Swedish
French
Seattle
Swedish
Seattle
Lymphocytes
Lymphocytes
●
●
●
Monocytes
●
●
●
T cells
Neutrophils
●
●
●
NK cells
Immature grans
●
●
●
B cells
●
●
●
Plasma cells
●
●
pDCs
●
●
●
Blasts
●
●
●
CD16+
Monocytes
Mast cells
nRBCs
●
●
●
●
●
Basophils
●
Reactive
Cytotoxic
Eosinophils
●
●
●
●
●
●
1
2: A
3: A and C
4: A and C and F
SSCè
B
D
C
5: A and C and E
E
F
G
H
CD19 ECDè
SSCè
CD45 Cy7è
6: A and C and F and G
7: A and C and F and G
CD45 Cy7è
CD36 FITCè
CD16+
neutrophils
CD16 Cy5 è
WBC
CD2+CRTH2 PEè
A
8: A and C and F and H
CD45 Cy7è
CD45 Cy7è
SSCè
CD45 Cy7è
CD45 Cy7è
Imm Gran
CD16pos
Mono
CD36 FITCè
B-cells
11: A and D and I
I
SSCè
CD2 + CRTH2 PEè
B-blasts
10: A and D
9: A and B
Cytotox
My+
blasts
SSCè
Basophils
SSCè
Non cytotoxic
T-cells
CD36 FITCè
SSCè
CD16neg
Monocytes
CD45 Cy7è
Eosinophils
CD2 + CRTH2 PEè
Drs. Feuillard,
Koksch,
Simon-Lopez
SWE, US, FR
100
90
100
Neuts
80
80
70
70
60
60
50
y = 1.0026x - 4.3555
R² = 0.93665
40
30
y = 0.9552x - 4.9922
R² = 0.93808
20
0
0
20
40
60
80
50
y = 0.7353x + 1.4586
R² = 0.85695
40
30
y = 0.9804x + 1.5578
R² = 0.8758
y = 0.9026x + 2.5612
R² = 0.86061
20
y = 1.0095x - 7.4966
R² = 0.95214
10
10
0
100
0
60
20
40
60
80
100
70
50
FCM
%
Ly
90
Imm Gran
40
Blast
60
50
40
30
30
20
20
10
10
0
0
10
20
Manual Dif
30
40
50
60
0
0
10
20
30
40
50
60
70
Comparability
•  Comparison of the evaluation instrument’s results,
or assay, to those obtained by another designated
method should be examined for as many samples
as possible.
•  These should include normal samples, abnormal
samples, covering the entire pathological range,
and samples with interfering substances.
Abnormal WBC
Abnormal RBC
Abnormal
Platelets
Interfering
Substances
Extreme leucocytosis
Sickle cells
Giant platelets
Haemolysis
Extreme leucopenia
Target cells
Platelet clumps
Cryoglobulins
Blast cells
Fragmented cells
Paraproteins
Atypical lymphocytes
Microcytic cells
High bilirubin
Smear/Smudge cells
Macrocytic cells
Lipaemia
Immature Granulocytes
Spherocytes
Left shift/band neutrophils
Extreme polycythaemia
Extreme anaemia
Nucleated red blood cells
Reticulocytosis
Howell Jolly bodies
Heinz bodies
Pappenheimer bodies
Malarial parasites
Comparability
•  The paired t-test (when results are normally
distributed)
•  For non-Gaussian data the Wilcoxon Rank Sum
test or Mann-Whitney U test should be used
•  A p-value of less than 0.05 is usually considered
as statistically significant
Comparability
•  When discrepant results are found between the
instruments, where possible, the samples should
be measured using the reference methods
previously described
•  All samples must have the CLSI 2 x 200 cell
reference leucocyte differential
Comparability
•  The morphology of cells need to be assessed in
order to compare the efficiency of the suspect
abnormal cell flags generated by the instruments.
•  The sensitivity, specificity, positive predictive
value, negative predictive value and overall
efficiency should be calculated for each individual
flag
Comparability
Results from Gold Standard Reference Method
True i.e. abnormality
present
False i.e. abnormality
absent
Positive for abnormality
True Positive (TP)
False Positive (FP)
PPV =
TP / (TP + FP)
Negative for
abnormality
False Negative (FN)
True Negative (TN)
NPV =
TN / (TN + FN)
Sensitivity =
TP / (TP + FN)
Specificity =
TN / (FP + TN)
New Method Results
Comparability
RED CELL FRAGMENT FLAG: XE 26 XT 17 LH750 28
MANUAL MORPHOLOGY POSITIVE
RED CELL FRAGMENTS = 15
MANUAL MORPHOLOGY
NEGATIVE
RED CELL FRAGMENTS = 351
XE true positive = 8
XE false negative = 7
XE false positive = 18
XE true negative = 333
XT true positive = 7
XT false negative = 8
XT false positive = 10
XT true negative = 341
LH750 true positive = 9
LH750 false negative = 6
LH750 false positive = 19
LH750 true negative = 332
Sensitivity – XE 53% XT 47% LH750 40%
Specificity - XE 98% XT 98% LH750 98%
Predictive value of a positive result – XE 31% XT 41% LH750 32%
Predictive value of a negative result – XE 98% XT 98% LH750 98%
Overall efficiency – XE 93% XT 95% LH750 93%
Comparability
•  Some parameters are only available on a single or limited number of
instrument types
- % Hypochromic red cells
- Reticulocyte haemoglobin content
- Immature platelet fraction
•  Results should be assessed to determine if they are consistent and
appropriate with the diagnosis and clinical condition of the patient.
•  Where routine haematology analysers use monoclonal antibodies and flow
cytometry methods for the measurement of some cells, (e.g. platelets
labelled with anti-CD61 and lymphocyte subsets with anti-CD4 and antiCD8) the results should be compared to the results from a dedicated flow
cytometer
Local evaluation/validation
WHO defines validation as the action (or process)
of proving that a procedure, process, system,
equipment, or method used works as expected
and achieves the intended result
(WHO-BS/95.1793).
CPA definition
•  Validation - confirmation, through the provision of
objective evidence, that the requirements for a
specific intended use or application have been
fulfilled
CPA Standards
•  Examination procedures, including those for
sampling, shall meet the needs and requirements
of users.
•  Examination procedures shall be validated for
their intended use prior to introduction, and the
methods used and results obtained, recorded.
•  The laboratory shall determine the uncertainty of
results, where relevant and possible.
Why validate an analyser?
•  It is a CPA requirement
•  CE mark not sufficient
•  Manufacturer claims may be overestimated
Local evaluation/validation
•  Price: Instrument, reagents, consumables, leasing…
•  Volume of sample needed, automatic and manual mode,
types of sample tubes accepted
•  The units used for reporting results.
•  Service contract and response times
•  Interface
•  Installation and calibration
•  Training
Local evaluation/validation: Blood samples
• 
The new instrument will be compared to the current instrument in the lab and
the manual differential
• 
Samples should be carefully selected and follow the collection, processing,
transportation, and storage guidelines established
• 
The anticoagulant used should be recorded (K2EDTA or K3EDTA)
• 
The range of samples tested should cover the entire clinical range and include
the most severe abnormalities encountered by the laboratory.
• 
Samples should be included with possible interfering substances, such as
lipid, high bilirubin concentration, haemolysis or the presence of cryoglobulins.
• 
One third to half of samples should be normal
• 
A minimum of 50 samples are required to test for statistical significance
Local evaluation/validation: Analysis of
results
• 
• 
• 
• 
• 
Samples with extreme results
should not be included as this
influences statistical analysis.
Linear regression
Bland Altman
The paired t-test (when results are
normally distributed)
For non-Gaussian data the
Wilcoxon Rank Sum test or MannWhitney U test should be used
Sensitivity and specificity of
abnormal cell flags
550
500
450
400
350
300
250
200
150
100
50
0
y = 0.8502x + 2.4549
R2 = 0.9861
0
50
100
150 200 250 300 350 400 450 500 550
FLOW NRBC/I00 WBC
FLOW V MANUAL NRBC/100 WBC
175
MAN NRBC/100 WBC
• 
MAN NRBC/100 WBC
FLOW V MANUAL NRBC/100 WBC
150
125
100
75
50
25
0
0
25
50
75
100
125
FLOW NRBC/I00 WBC
150
175
Local evaluation/validation:
• 
Between batch precision.
Reference Intervals
•  Reference intervals, specific to the instrument, for all
components of the FBC should be calculated during the
instrument evaluation.
•  At least 30 apparently healthy individuals of each sex
should ideally be tested, if appropriate, ranges for children
of different ages should also be tested.
•  Samples should be tested within four hours of venesection.
Reference Intervals
Reference Intervals
Reference Intervals
•  Hb to reported as g/l
•  All reference ranges to be the same for all labs
and across all instrument types
•  MCHC to be a non-reportable parameter
Reference Intervals
Instrument
ABX Instruments
ABX Pentra series
Siemens Advia
Cell Dyn 1600 & 1700
Cell Dyn 3000 series
Cell Dyn 3200 & Ruby
Cell Dyn 4000 & Sapphire
Coulter S Plus series
Coulter T series
Sysmex K series
Sysmex X-Class
Sysmex SF3000 series
Sysmex pocH-100i
Sysmex XT series
0902FB1
88.90
84.90
88.96
95.26
98.41
85.66
90.40
94.82
94.01
93.47
102.82
94.37
97.06
103.16
0902FB2
85.61
80.23
86.94
91.52
94.98
84.07
88.13
90.51
90.05
90.95
99.78
92.02
93.65
99.14
Reference ranges
•  Where ever possible local reference ranges
should be established for any new method
•  Fresh or frozen plasmas from at least 30 healthy
individuals should be tested
•  Sex specific ranges may be required (e.g. protein
S assays)
Efficiency
Throughput
Real situation
Sick patients
Wide range of tests performed simultaneously
Reflex tests
Can several test methodologies run at once?
• 
Efficiency
• 
• 
• 
• 
Start up and shutdown
Sample identification
Presentation and storage of results
Reliability
Acceptability
•  Staff opinions and preferences should be taken into
consideration.
•  An assessment of the level of expertise required for the
operation of the instrument should be determined.
•  The impact of the instrument on the workflow and
organisation of the laboratory.
•  Any modifications to the laboratory design should be
considered.
Performance assessment
Good
precision and
accuracy
Good
precision,
poor
accuracy
Reasonable
accuracy,
reasonable
precision
Poor accuracy,
poor precision