Final report on data discrepancies in the CWT Tool study 1

Report on data discrepancies in the CWT
Tool – Study 1
Main authors: Dr Penny Newman, Amanda Dell, Ayo Adebamowo
Contributors: Ian Craddock, David Traynier, Orla Thunder, Michelle Fisher, Lorna Dewar, Hether
Buckle, Sandra Lyons, Dr Mary McStay, Dr Devy Basu, Dr Jennifer Collins, Denise Gale
Executive Sponsor: Dr Sean MacDonnell
This report was commissioned by Colchester Hospital University NHS Foundation Trust at
the request of the multiagency incident management team (IMT).
Dr Christine Macleod, on behalf of the Assurance Panel of the IMT, endorses the findings
within this report.
Signature:
Date: 16/12 /2014
Dr Christine Macleod, Medical Director
NHS England, Essex Area
Signature:
Date: 16/12 /2014
Dr Lucy Moore, Chief Executive
Colchester Hospital University Foundation Trust.
1
15th December 2014
Our gratitude goes to the following:
Colchester Hospital University NHS Foundation Trust





Main authors: Dr Penny Newman (Review Programme Director, Director of Service
Integration, GP and Consultant in Public Health), Amanda Dell (Project Management
Consultant), Ayo Adebamowo (Lead Cancer Information Analyst)
CHUFT Contributors and Review Team members: David Traynier (Audit support/Analysts),
Lorna Dewar (Oncology Clinical Trials Manager), Orla Thunder (Clinical Trials Data Nurse),
Michelle Fisher (Research Nurse), Ian Craddock (Site Matron, Senior Auditor), Heather
Buckle, Sandra Lyons (Clinical Auditors), Dr Mary McStay, Dr Devy Basu, Dr Jennifer Collins
(Lead Clinicians), Denise Gale (Cancer Programme Director), Valerie Northcroft-Brown
(Project Support)
Executive sponsor: Dr Sean MacDonnell (Medical Director)
Business Informatics team and in particular Michele Figg (Head of Business Informatics)
Health Records team and especially Barry Moult (Head of Information Governance & Health
Records),Teresa Frost (Health Records Manager) and Phil Frances (Health Records Clerk)
The Assurance Panel
Dr Christine Macleod (NHS England Essex Area Team Medical Director and Chair of Retrospective
Assurance Panel), Dr Shane Gordon (Clinical Chief Officer, NHS North East Essex Clinical
Commissioning Group), Karen Hindle (Senior Associate (communications) interim hub manager,
North, Midlands and East Communications), Dr Thomas Nutt (Chief Executive Officer, Healthwatch
Essex), Paul Pharoah, (Professor of Cancer Epidemiology, Department of Public Health and Primary
Care, University of Cambridge and Honorary Consultant in Public Health, Public Health England), Pól
Toner (NHS England Essex Area Team Director of Nursing and Quality). Observer: Representatives of
the Information Commissioner’s Office, Support: Christine Cooper (PA to NHS England Essex Area
Team Medical Director)
External reviewers


The East of England Strategic Clinical Network, particularly Dr Rory Harvey (SCN Cancer
Clinical Director) and Kate Patience (Rehabilitation & Quality Improvement Lead for East of
England SCN Cancer Team), all the external review Consultants and Karen Harland (Cancer
Services Manager, HOPE Clinical Unit, Hinchingbrooke Health care NHS Trust)
The Royal Marsden NHS Foundation Trust and particularly Nicky Browne (Director of
Performance & Strategy Implementation)
2
15th December 2014
Table of Contents
Table of Contents .................................................................................................................................... 3
Executive Summary............................................................................................................................... 5
Abbreviations and Acronyms Used in this Report .................................................................................. 8
1.
Introduction ................................................................................................................................... 9
1.1 The Retrospective Review Team and reporting arrangements .................................................. 10
1.2 The Assurance Panel ................................................................................................................... 10
2
National and local data on cancer waiting times ...................................................................... 11
2.1 Cancer Waiting Times Guidance ................................................................................................. 11
2.2 Comparative data........................................................................................................................ 12
2.3 Cancer mortality statistics and survival rates for Colchester Hospitals ...................................... 13
3
Aims and objectives..................................................................................................................... 14
4
Methods ........................................................................................................................................ 14
4.1 Audit process .............................................................................................................................. 14
4.2 Quantitative analysis................................................................................................................... 15
4.3 Analysis of cancer waiting time data .......................................................................................... 15
4.3.1 Analysis by cancer pathway (operational standard) ............................................................ 15
4.3.2 Analysis by data field ........................................................................................................... 15
4.3.3 Unverified data .................................................................................................................... 16
4.3.4 Discrepancies between the case notes and CWT tool ......................................................... 16
4.4 External review ........................................................................................................................... 16
4.5 Qualitative analysis ..................................................................................................................... 16
4.6 Queries and areas of concern ..................................................................................................... 17
4.7 Assurance panel review of potential data manipulation and/or areas of clinical concern ........ 17
5.
Results .......................................................................................................................................... 18
5.1 Sample and patient characteristics ............................................................................................. 18
5.1.1 Source of Referral ................................................................................................................ 19
5.1.2 Shared Pathways .................................................................................................................. 19
5.2 Data accuracy .............................................................................................................................. 20
5.2.1 Inaccuracy by cancer pathway ............................................................................................. 20
5.2.2 Data inaccuracy by tumour site ........................................................................................... 21
5.2.3 Spread of data errors by tumour site................................................................................... 22
5.3 Causes of data errors .................................................................................................................. 23
5.4 Impact of data inaccuracy ........................................................................................................... 24
5.4.1 Time between recorded and actual dates ........................................................................... 24
3
15th December 2014
5.4.2 Significance of variance between CWT tool and data in the notes ..................................... 27
5.4.3 Clinical pathways and care ................................................................................................... 29
5.5 Correlation with Patient surveillance log .................................................................................... 29
6.
Royal Marsden Hospital Review (Appendices 7 and 8) ........................................................... 31
7.
Areas identified for improvement from qualitative analysis ................................................... 32
7.1 Poor record-keeping ................................................................................................................... 32
7.2 Decentralised record-keeping ..................................................................................................... 32
7.3 Misunderstanding of Cancer Waits guidance ............................................................................. 32
7.4 Misunderstanding of “Decision to Treat” ................................................................................... 32
7.5 Poor information sharing with other providers .......................................................................... 32
8.
Discussion and conclusion .......................................................................................................... 33
9.
Recommendations ....................................................................................................................... 34
9.1 ..................................................................................................................................................... 34
9.2 ..................................................................................................................................................... 34
9.3 ..................................................................................................................................................... 34
9.4 ..................................................................................................................................................... 34
9.5 ..................................................................................................................................................... 35
9.6 ..................................................................................................................................................... 35
9.7 ..................................................................................................................................................... 35
Appendix 1: Study Protocol ................................................................................................................ 36
Appendix 2: Data on Cancer Survival, Mortality and Referrals ....................................................... 39
Appendix 3: Methodology for Obtaining Sample Data .................................................................... 41
Appendix 4: Audit Tool and Fields ..................................................................................................... 43
Appendix 5: Glossary .......................................................................................................................... 47
Level One Codes:............................................................................................................................... 47
Level Two Codes: .............................................................................................................................. 47
Level Three ........................................................................................................................................ 48
Decision to Treat Time-only codes.................................................................................................... 48
Final codes ........................................................................................................................................ 49
Appendix 6: Variance (in days) for Data Items with Incorrect Dates ............................................. 50
Appendix 7: The Royal Marsden NHS Foundation Trust independent review of Colchester
University Hospital NHS Foundation Trust Retrospective Review Team audit results ................ 51
Appendix 8: Marsden review of cases for Colchester Hospital NHS Foundation Trust as part of
retrospective review – cases where there was disagreement with the Trust................................ 52
4
15th December 2014
Executive Summary
Following concerns about cancer care at the Trust, in 2013 Colchester Hospital University NHS
Foundation Trust (CHUFT) was inspected by the Care Quality Commission (CQC), who reported
inconsistencies in data held on the Cancer Waiting Times system (CWT) and within 61 medical
records. In 22 of these cases, the delay in treatment may have had an impact on patients’ health
(see below).
“During our examination of 61 patient medical records we noted there to be inconsistencies
between information that was held on the cancer wait times system (CWT) and that which was
contained within people’s medical records. In total 22 of the files showed discrepancies in a
person’s pathways.
We found that entry dates on the CWT system did not always correlate with the patients' medical
records. We found that in 22 cases the treatment dates recorded on the system had been changed.
Details and examples of the altered records are reflected in the section for 'records' in this report.
The changes to the patient cancer pathway was identified through a review of the medical records
which demonstrated that treatment had been provided on a date different to the one recorded on
the CWT system. The changes in those 22 cases meant that people could have experienced a
negative impact in the form of their treatment being delayed. The risk of delayed treatment could
impact on their care and longer term health”.
CQC Report into Cancer Services at Colchester Pages 12 and 7, Published 5th November 2013.
The Retrospective Review was commissioned by NHS England and the Incident Management Team
(IMT) as part of a suite of measures arising from the inspection and review of cases. The review aims
to investigate, through a transparent audit process, the extent of data inaccuracies in Cancer Waiting
Times in the Trust and their impact on clinical care. This report describes findings from Study 1 into
the error rate of cancer pathway data and is one chapter of six in the Retrospective Review’s overall
report.
The Retrospective Review looked at two aspects of patient care at the Trust and any relationship
between them:
1. The Cancer Waiting Times data pathway to identify data inaccuracy
2. The clinical pathway to identify any potential clinical harm.
This study on data error rate looked more specifically at data issues. Its aim was to assess the
historical accuracy of the data held on the CWT through case note review of a stratified sample of
250 records between April 1, 2010 and October 31, 2013 so that accurate Trust cancer waiting time
performance can be assured.
The audit was a six stage process including data sampling, template development, case note review,
expert cancer wait pathway and clinical review, analysis and report writing and an external review of
a random 10% of records. This was undertaken by a cancer data expert from the Royal Marsden NHS
Foundation Trust to check auditors’ accuracy of assessment and validity of results and external
Consultants identified by the Anglia Cancer Network to assess quality of care. Cases of “potential
data manipulation” were reviewed against criteria by an Assurance Panel made up of all key
5
15th December 2014
stakeholders and chaired by NHS England Medical Director to identify any need for referral to the
police.
No data inaccuracy was found in 97 of the 250 cases (39%) reviewed and in 14 cases (6%)1 the
dataset could not be entirely verified as the activity was at other Trusts.
The audit indicates that, in 55% (139) of cases, across most specialties reviewed and for a number of
reasons, there were data inaccuracies in the CWT database. For all patients on a cancer pathway,
reviewers considered up to nine data fields, and there was an error rate of 10% in all completed data
entries (142 of 1,453 cases).
Although the actual error rate of 55% may not be representative of the whole CWT database, as the
sample was taken from a selected cohort of specialties with known issues, the finding that over half
of cases reviewed had at least one data inaccuracy indicates poor data recording and quality. As the
upkeep e.g. legibility, filing, and maintenance of data in the case notes was often poor, and as there
were multiple cancer databases, information was difficult to find.
There were multiple causes of data error, which was categorised by IMT and are defined below:
No errors
The record is found to be accurate and complete in all fields.
Data entry errors
When CWT and other sources of information, including 40% (100
patient notes, match although the data has not been cases)
updated e.g. on another pathway, appointment brought
forward, incorrect treatment entry
When an incorrect entry appears to have been made as a 11% (28 cases)
result of a misunderstanding of the national CWT guidelines.
Misinterpretation
of the national
guidance
Operating process
issue
Data with other
provider
39% (97cases)
A substantive error in a process or procedure, as opposed to 3% (7 cases)
the recording of that process or procedure.
Data available on the CWT but could not be checked against 6% (14 cases)
data in the notes which were held elsewhere.
Other causes included poor information sharing, document handling and recording i.e. documents
mislaid or incorrectly reported, and mis-recording of Decision to Treat (DTT) following discussion or
Multi-Disciplinary Team (MDT) meeting.
There were 105 instances or data points along the Cancer Waiting Times pathway (“episodes”),
relating to 83 cases, where auditors recorded a difference (or variance) between the dates in the
CWT and the notes. These relate to four fields: referral received date; date first seen, decision to
treat, and treatment start date.
1
While 8% of the patients were treated in other Hospitals, it was possible to verify some records as
information was available on RTT. Therefore only 6% of cases could not be verified due to shared care.
6
15th December 2014
Comparing the dates recorded in the notes and on the CWT tool:





In most instances where an error was identified the difference between the date recorded
on the CWT and the date in the notes was small. In 58% of episodes (61 of 105) the date
recorded in the notes was within five days before or after the date recorded CWT Tool
Dates recorded in the notes for all pathways range from between 77 days before to 52 days
after dates recorded in the CWT tool.
In the majority of episodes (65% or 68 of the 105 data points) the date recorded in the notes
was earlier than the date recorded on the CWT Tool.
For 10 patients, dates in the notes were between 10 and 52 days after dates recorded on the
CWT tool, indicating that, for some patients, and especially as the pathways may be
cumulative, there would have been unacceptable delays.
This study shows pockets of poor data recording, particularly in Upper and Lower GI and
Urology, especially around treatment dates. The size of the variance was larger in Lower GI
and Urology, although numbers are small.
On statistical analysis of the cases where there was a discrepancy in dates between the notes and
CWT there is some evidence of systematic bias making waits seem shorter than in reality. This could
have resulted from multiple causes as described above including erroneous interpretation of
complex CWT guidance or even subconscious bias. The methods of this study did not allow an
assessment whether this led to gain, for example, in recording of performance.
However, in the four individual cases assessed by the Assurance Panel in which auditors found data
irregularities, the Panel found no evidence of data manipulation. In these cases which were
assessed to determine if potential data manipulation might have led to delays harmful to patients,
there was no reason to refer any cases to the police.
In the sample of 250 cases, there were five cases referred for Serious Incident investigations (SI).
Two new referrals came from the retrospective review and three had already been referred
previously via the helpline, complaints, or the SI process. These latter three SIs related to aspects of
care other than those covered by the Error Rate audit. It should be noted that, during the period of
the helpline, the threshold for SI investigations was lowered. One case was considered by the
Assurance Panel a GP significant event (the equivalent to a Serious Incident but in primary care).
After reviewing these SIs, the Trust put new policies in place and reinforced aspects of the Cancer
Action Plan. As one significant event result from a GP referral on to a non-cancer pathway, and 5%
of GP cancer referrals were routine, the Trust needs to work closely with primary care to improve
early recognition of cancer and referral across the system. The Trust needs to work with other
hospitals to improve data sharing as, in 14 of 21 cases of shared care, insufficient data were available
to complete the audit.
The picture is of a problem with poor data quality in cancer. Data from other Trusts have not been
analysed to such a degree as a comparator. It is hoped that data recording will be addressed through
the introduction of the Somerset Cancer Registry information system.
In conclusion, although discrepancies in the recording of data were found in this study, there was no
evidence of this causing patient harm.
7
15th December 2014
Abbreviations and Acronyms Used in this Report
2WW Two Week Wait
IMAS Interim Management and Support
CCG Clinical Commissioning Group
IMT Incident Management Team
CHUFT Colchester Hospital University
Foundation Trust
IT Information Technology
MD Medical Director
CI Confidence Interval
MDT Multi-Disciplinary Team
CNS Clinical Nurse Specialist
MDTC Multi-Disciplinary Team Coordinator
COSD Cancer Outcome and Service Dataset
CQC Care Quality Commission
MEHT Mid-Essex Health Trust
NAO National Audit Office
CUP Cancer of Unknown Primary
NE North East
CWT Cancer Waiting Times
NEE North East Essex
DNA Did Not Attend
NHS National Health Service
DTT Decision to Treat
ONS Office for National Statistics
ECAD Earliest Clinically Appropriate Date
ECRIC Eastern Cancer Registration and
OPD Outpatients’ Department
PAS Patient Administration System
Information Centre
PH Public Health (England)
FDT First Definitive Treatment
RMH Royal Marsden Hospital
FT Full Time
RRT Retrospective Review Team
GDP General Dental Practitioner
RTT Referral to Treatment
GI Gastro-Intestinal
SCN Strategic Clinical Network
GMP General Medical Practitioner
SCR Somerset Cancer Registry
GP General Practitioner
TYA Teenagers and Young Adults
SI A definition of a serious incident is: ‘An
incident of such seriousness that it causes or
threatens to cause serious harm to patients,
staff, volunteers, members of the public,
contractors or the Trust itself. There are
specific criteria used to define SIs and this
includes the risk of reputational harm to the
NHS. A Never Event is a Serious Incident
which should never happen if known
preventive measures had been in place’(Series
Incident Procedure Trust Policy No 63a, 2014)
SI Cont. Please note where an SI is mentioned
throughout all reports in the retrospective
review this refers to an incident report form
(datix) being completed and referral for
serious incident investigation. The
investigation process can take up to 45 days
at which point a final report is submitted to
North East Essex CCG for review. Some
investigations are still on-going at time of
writing this report.
8
15th December 2014
1. Introduction
Following concerns about cancer care, in 2013 Colchester Hospital University NHS Foundation Trust
(CHUFT) was inspected by the Care Quality Commission, which identified 30 patients who may have
experienced harm resulting from potential data manipulation. As a result, the Trust reviewed all 30
cases and met with patients and their families. Where consent was given, cases were referred for
medico-legal review. In addition, the Trust undertook a look-back exercise to address specific areas
of concern identified in clinical pathways to ensure patients were safe.
The Retrospective Review was commissioned by NHS England and the Incident Management Team
(IMT) as part of a suite of measures including:



A review of cancer pathways by the East of England Cancer Network
An independent review of Trust Governance
The implementation of a comprehensive Cancer Action Plan by the Trust including a new
information system (Somerset Cancer Registry).
The review aims to investigate, through a transparent audit process, the extent of data inaccuracies
in cancer waiting times in the Trust and their impact on clinical care. It will recommend necessary
remedial action to ensure accurate reporting and improved systems of care, and provide assurance
to the public and stakeholders. More specifically the retrospective review will:
1. Using clinical and non-clinical audits, identify and report on the extent, at an individual and
system level, of the following:
- Evidence of data manipulation,
- Causation factors, and where possible, whether these were intentional,
- Whether patients have been exposed to any risk of harm,
- Remedial actions required, including the on-going management and monitoring of cancer
services.
2. To provide transparency and external validation through regular reporting to a Retrospective
Review Assurance Panel and by publishing results.
3. To refer to the police any cases of potential manipulation of data.
The Retrospective Review involves the study of six specific aspects of cancer care across all
specialties and a review over 1,700 records:
Study 1
To determine the error rate of the Cancer Waiting Tool (CWT)
Study 2
Long waits - to identify long waits (over 91 days) in cancer.
Study 3
To determine the prevalence of ‘delayed diagnoses’ in cancer pathways
(defined as ‘those patients restarting a cancer pathway within 90 days
from stopping an initial cancer pathway’).
Upper GI - to identify patients whose pathway was changed without
appropriate clinical input.
Urology - to identify patients who have been lost to superficial bladder
cancer surveillance.
Surveillance – review of all calls to the helpline, complaints and
significant events following the CQC investigation
Study 4
Study 5
Study 6
250 case
notes
290 case
notes
364 case
notes
120 case
notes
15 case
notes
684
patients
9
15th December 2014
All studies look, to varying degrees, at two aspects of patient care at the Trust and any relationship
between them:
1. The Cancer Waiting Times data pathway to identify data inaccuracy
2. The clinical pathway to identify any potential clinical harm.
This, Study 1 on error rate, looks in more detail at the data element. The protocol developed by IMT
is in Appendix 1.
1.1 The Retrospective Review Team and reporting arrangements
The Retrospective Review Team is composed of clinical auditors (two research nurses, a research
manager, three general nurses or 4.42 Whole Time Equivalent (WTE)), an analyst (interim, Full Time,
(FT), an audit facilitator (FT), a project manager (interim, FT) and a Programme Director (a GP and
Consultant in Public Health, whose role was previously Director of Service Integration, 0.6 WTE). The
team receive support on an ad hoc basis with CWT guidance from the Cancer Programme Director
and clinical advice from three Consultants in Upper GI and oncology. The Programme Director
reports to the Trust Medical Director (MD), Programme Steering Group and Turnaround Programme
Board, and indirectly to the Executive Team.
1.2 The Assurance Panel
Throughout the process, the Trust has reported to the Assurance Panel. This panel consists of all
stakeholders (NHS England, North East Essex Clinical Commissioning Group (CCG), and
representatives of the Information Commissioner, Healthwatch Essex, and Department of Public
Health and Primary Care, Cambridge University). Its role is to oversee and validate the process and
the results of the Retrospective Review, in order to provide assurance to the public, patients, staff
and relevant bodies on data quality and clinical care in cancer at CHUFT. More specifically, it acts to:




Review and advise on methodology to ensure it is transparent, robust and objective,
Validate the accuracy of results,
Gain assurance in the use of external consultants in the process for external validation,
Comment on, inform, and sign off, the final report.
10
15th December 2014
2 National and local data on cancer waiting times
2.1 Cancer Waiting Times Guidance
All work was undertaken are in reference to Cancer Waiting Times Guidance V8.0.2 This defines
important terms, of which several are pertinent to this report.
Decision to Treat (DTT):
This is defined in the CWTs Guidance as “the date the patient agrees a treatment plan”.
Two-Week Wait (2WW):
This is defined as “urgent GP (General Medical Practitioner (GMP) or General Dental Practitioner
(GDP) referral for suspected cancer to first outpatient attendance”. Patients referred on this
pathway are required to be seen in the Trust within fourteen calendar days from the receipt of their
referral. Patients on this pathway are concurrently on both the 62-Day pathway (from receipt of
referral) and the 31 day 1st definitive treatment pathway (from Decision to Treat).
31-Day First Definitive Treatment (DTT):
This is defined as “decision to treat to first definitive treatment”. Patients referred under this
standard are required to commence first definitive treatment for a new cancer diagnosis within 31
days of the date of decision to treat being made.
62-Day Standard:
This is defined as “urgent GP (GMP or GDP) referral for suspected cancer to first definitive
treatment”. Patients referred with suspected cancer under the two-week wait standard (2WW) are
required to commence first definitive treatment, if cancer is diagnosed, within 62 days from the date
of receipt of referral for the suspected cancer, and within 31 days of the Decision to Treat.
The 62-Day Screening:
This is defined as “urgent referral from NHS Cancer Screening Programmes (breast, cervical and
bowel for suspected cancer to first definitive treatment”. These patients are required to commence
first definitive treatment, if cancer is diagnosed, within 62 days from the date of receipt of referral
for the suspected cancer.
31-Day Subsequent Treatment:
This is defined as “decision to treat/earliest clinically appropriate date (ECAD) to start of second or
subsequent treatment(s) for all cancer patients including those diagnosed with a recurrence where
the subsequent treatment is surgery, anti-systemic cancer treatment (drugs), or radiotherapy.
These patients are required to commence treatment within 31 days of the date of decision to treat
being made or earliest clinically appropriate date.
Waiting Time Adjustment (First Seen):
This records the number of days that patients should have been removed from their calculated
waiting time for the two week wait period and potentially the 62 day period (if cancer is confirmed).
Waiting Time Adjustment (Treatment):
This data item is used to record the number of days that should be removed from the calculated
waiting time between the date of decision to treat and the treatment start date i.e. the number of
2
http://www.nwlcn.nhs.uk/Downloads/Cancer%20Intelligence/Going%20Forward%20on%20Cancer%20Waits
%20A%20Guide%20Version%208.0.pdf
11
15th December 2014
days that a clock can be paused for a 31 or 62 day period if a reasonable offer of treatment in
admitted care has been declined.
2.2 Comparative data
It is difficult to assess the significance of data error rate as comparative assessments of cancer
information systems have not been undertaken. This means there is a lack of benchmark data
against which to compare results.
Nationally, there is a published error rate of 57% (incorrectly recorded or no evidence) in waiting
times data (not cancer specific).3 The National Audit Office (NAO), NHS Waiting Times Elective Care
in England was published in January 2014. Findings from the report are reproduced below.
The NAO, NHS Waiting Times Elective Care in England
Executive Summary Page 7 and 8
There are errors in the trusts’ recording of patients’ waiting time. We reviewed 650
orthopaedic patient waiting times across seven trusts. More than half of these were not
supported by documented evidence or were incorrectly recorded. Although it was not a
representative sample for the country as a whole, we established clear data risks that
need to be managed. We found that:



in 281 (43%) cases waiting times had been correctly recorded and were supported by
documented evidence
in 202 cases (34%), waiting times were not supported by enough evidence to say
whether they had been correctly recorded; and
in 167 cases (10%), there was evidence of at least one error, leading to under- and overrecording of waiting time. There was an overall under-recording of three weeks (mean)
per patient, with a median of 11 days (paragraphs 2.14 to 2.18).
Mis-recording of data was identified at The North West London, Barnet and Chase Farm
and Colchester Trusts. The North West London Hospitals NHS Trust identified that it had
failed to record properly the waiting times of 2,700 (60 per cent) of its elective (prearranged) inpatients, including 12 who had waited more than 52 weeks for treatment.
Barnet and Chase Farm Hospitals NHS Trust identified that it had failed to monitor more
than 2,000 patients on the waiting list, 651 of which had waited between 18 and 51
weeks for treatment (paragraphs 2.20 to 2.22).
NAO Report Paras 2.0 to 2.22
The North West London Hospitals NHS Trust failed to record the waiting times of 2,700
(60 per cent) of its elective inpatients, including 12 patients who had waited more than
52 weeks for treatment. The errors were identified during an internal validation of
inpatient waiting lists and reported to the trust’s board.
In May 2013, Barnet and Chase Farm Hospitals NHS Trust identified a failure to monitor
more than 2,000 patients on the waiting list, of which 651 had waited between 18 and
51 weeks. The backlog developed as a result of a failure in an IT reporting system.
3
http://www.nao.org.uk/report/nhs-waiting-times-elective-care-england/
12
15th December 2014
At the time of the report to the board in May, there were no patients identified as
waiting more than 52 weeks for treatment. However, further validation of the waiting
list in July by trust staff identified 108 patients that had not been tracked and who had
waited more than 52 weeks for treatment; these patients were subsequently reported to
NHS England. In December 2013, Barnet and Chase Farm Hospitals NHS Trust had 181
patients who had waited more than 52 weeks for treatment.
Responding to whistle-blowers, the Care Quality Commission reported that Colchester
Hospital University NHS Foundation Trust had altered patient appointment and medical
records on its cancer waiting times system. The Care Quality Commission found that in
22 cases the treatment dates recorded on the system had been changed. The police are
now conducting an investigation at Colchester (paragraph 2.22).
The report recognises that some of the challenges facing trusts when managing waiting
lists are the perennial systemic issues of balancing financial and clinical capacity with
the demand for services.
2.3 Cancer mortality statistics and survival rates for Colchester Hospitals
The following statistics (Tables 1-5, appendix 2) are taken from the ONS, the National Cancer
Registry, the Public Health England Data and Knowledge Gateway, the National Cancer Intelligence
Network E Atlas:
•
•
•
•
•
•
NE Essex CCG one-year survival index (%) is within normal range for all cancers combined for
adults (aged 15-99 years)
In 2011 NE Essex CCG one-year survival index was 67.7% for all cancers combined for adults
(aged 15-99 years), slightly lower than all Essex CCGs combined (68.2%) and England as a
whole (68.2%) although higher than three other CCGs in Essex
NE Essex one and five-year survival rates for cancers within key specialties are all within the
normal range
NE Essex mortality 2008-2012 by cancer type are all within normal range except prostate,
lung and stomach which are significantly lower than UK average
Cancer incidence each year equates to around 0.5% of population for NE Essex higher than
national rate of 0.37%, and has increased over last 13 years
Total number of patients referred for 2WW is increasing although not translating into
numbers diagnosed and treated.
13
15th December 2014
3 Aims and objectives
The full protocol developed by IMT is in Appendix 1.
Study Title: To determine the error rate of the CWT Tool
Aim of the study: The aim of this study was to assess the historical accuracy of the data held on the
Cancer Waiting Tool so that accurate Trust cancer waiting time performance can be assured.
Primary Objective: To determine the historical accuracy of the data on the Cancer Waiting Tool.
Secondary Objective: To estimate the impact of the CWT data accuracy on cancer waiting time
performance reporting during the period of study
Outcome Measures/Endpoints


% data accuracy of CWT April 2010 – October 2013
Estimated impact on historic performance reporting
4 Methods
The audit reviewed a stratified sample of 250 cancer records from between April 1, 2010 and
October 31, 2013. The sampling process is detailed in Appendix 2 and includes a random sample
specifically selected from four specialties with known issues; Breast, Lower Gastro-Intestinal (Lower
GI), Upper Gastro-Intestinal (Upper GI), Urology and, to a lesser degree, other cancer pathways
Inclusion Criteria
1. Randomised sample of 250 case notes (comprising 50 case notes from Breast, Lower GastroIntestinal (Lower GI), Upper Gastro-Intestinal (Upper GI), and Urology, and 50 from other
cancer pathways),
2. Patients on either a 31-day or 62-day cancer pathway,
3. Patients on specified pathway between April 1, 2010 and October 31, 2013,
4. Patients who have had a first or second treatment.
Exclusion Criteria
1. Where a patient was on multiple pathways, only one pathway was selected,
2. Patients who were not treated,
3. Patients who were discharged with no cancer diagnosis,
4. Patients who were not on a cancer pathway during the specified period,
5. Patients selected for previous audits.
4.1 Audit process
The Retrospective Review process included six stages:
Stage 1:
Stage 2:
Stage 3:
Stage 4:
Stage 5:
Stage 6:
Identifying the sample.
Developing the audit tool and piloting
Undertaking the case note review
Clinical and cancer wait expert advice and/or review if necessary
Analysis and report writing
External review
14
15th December 2014
4.2 Quantitative analysis
As the auditors reviewed each case, they validated pre-populated information recorded in the CWT
Tool against data held in the Trust’s information systems in order to identify inaccuracies and
variances. The information systems included the Patient Administrative System (PAS), case notes
(general hospital notes and oncology notes), clinic letters, and pathology, radiology, and
radiotherapy databases, as well as the national Open Exeter system. Additional information was
sourced by contacting relevant clinical staff by phone or email. Where patients care was shared with
other Trusts, CWT data was mostly unavailable. The data fields in the audit tool are given in
Appendix 3.
In addition to using categories established by IMT, each case note was reviewed using a set of predefined options or codes developed by the review team that allowed critical findings to be
quantified and increasing levels of detail. A glossary of codes and definitions is given in Appendix 4.
4.3 Analysis of cancer waiting time data
Analysis of the data recorded on the audit template comprised a descriptive analysis of all 250 cases
and analysis of the congruence between the CWT database and notes by pathway type as a binary
(correct/incorrect) or quantitative answer (variance in days) in up to nine data fields.
4.3.1 Analysis by cancer pathway (operational standard)
The 250 cases audited included patients on different cancer pathways (Table 1). Patients can be on
one or more pathway e.g. patients referred on a 62 day standard cancer pathway (via 2WW) will
concurrently be on a 31 day 1st treatment pathway (once Decision to Treat has been confirmed).
2WW
31-Day 1st
Treatment
31-Day Subsequent
Treatment
62-Day Standard
62-Day
Screening
36% (91 cases)
70% (175 cases)
30% (75 cases)
36% (91 cases)
10% (26 cases)
Table 1: Distribution of sample of cases across CWT pathways (operational standards)
4.3.2 Analysis by data field
The accuracy of data recorded in the Trust’s CWT tool was established in up to nine fields in each
patient’s pathway. These were: Source of Referral, Date Referral Received, Date First Seen, Waiting
Time Adjustment (First Seen), Date of Diagnosis, Decision to Treat Date, Treatment Start Date,
Waiting Time Adjustment (Treatment), and Treatment Modality.
The data items available for each case review were dependent on those required for submission
according to CWT guidelines and pathway type i.e. whether a 2WW, 62-Day, Subsequent treatment:



Of the nine fields, only four are mandatory for all four pathways and could therefore be
analysed for all 250 patients. These fields include Date of Decision to Treat, Treatment Start
Date, Waiting Time Adjustment (Treatment) and Treatment Modality.
Data submission on Date of Diagnosis was not required for data returns until mid-2013 and
therefore data was incomplete.
The remaining four data items - Source of Referral, Referral Received Date, Date First Seen,
and Waiting Time Adjustment (1st Seen) are mandatory only for patients on 62-Day Standard
and 62-Day Screening pathways (117 patients).
15
15th December 2014
4.3.3 Unverified data
Finally data analysis was dependent on checking data in the CWT against data in the notes. In the
nine fields on average in 6% or 95 data points there was no other data to substantiate the CWT
(Table 2). Of these, 25% (24 of 95) were due to missing data at CHUFT and 75% due to data held at
other Trusts, equivalent to 1.5% and 4.5% of total data points respectively.
Data Item
No. of eligible
No. of verified No. of data items % unable to
data items
data items
unable to verify
verify
Source of Referral
117
109
8
7%
Date Referral Received
117
108
9
8%
First Seen Date
117
108
9
8%
Waiting Time Adjustment
117
117
0
0%
(First Seen)
Date of Diagnosis
80
75
5
6%
Decision to Treat Date
250
214
36
14%
Treatment Start Date
Waiting Time Adjustment
(Treatment)
Treatment Modality
Total data items
250
250
234
249
16
1
6%
0%
250
1548
239
1453
11
95
4%
6%
Table 2: Number of data items that could not be verified
4.3.4 Discrepancies between the case notes and CWT tool
Analysis was undertaken on each of the 1,548 data points to determine the number and percentage
of correct entries, incorrect entries, the size of discrepancy, and unverifiable entries (Table 2).
For incorrect entries, a further level of analysis was done. In all nine fields, data error was recorded
as a binary correct/incorrect answer while in five, auditors recorded the size of any discrepancy
between dates in the notes and CWT database (referral received date, date first seen, decision to
treat and treatment start date) and treatment modality. The data was analysed using Microsoft
Excel™.
4.4 External review
A random sample of 10% of audited records were assessed by a CWT expert from the Marsden
assessed to check accuracy of CWT assessment and hence validity of results (reported on page 28)
and by external Consultants from multiple specialties recruited by the Anglia Cancer Network to
assess quality of care and appropriateness of subsequent action taken by the Trust as part of the
review.
4.5 Qualitative analysis
A thematic analysis was undertaken of free text comments for cases coded as for “non-clinical
action” to identify areas that need to be addressed. Non-clinical action is defined as action taken to
change administrative, technical, or clerical activity over and above the care of an individual patient.
In addition, as they progressed, auditors listed broad themes and recommendations in an “Issues
Log”. These issues form the basis for some of the Recommendations.
16
15th December 2014
4.6 Queries and areas of concern
Where auditors had queries or concerns around data management and interpretation of the CWT
guidelines, the Trust’s Cancer Programme Director was consulted to provide clarification of CWT
guidance (v8.0).
Where there were issues or concerns around patient care, either the lead clinicians working with the
Retrospective Review Team were asked to assess the case to make recommendations for action or
the case referred as a Serious Incident.
4.7 Assurance panel review of potential data manipulation and/or areas of clinical concern
Following each case review, the auditors wrote a case summary. Together with the notes these were
discussed at the Assurance Panel on 20th May, 2014 to review any cases of possible clinical harm
and/or potential data manipulation. Cases were reviewed by the Assurance Panel against four
questions to identify if any necessitated referral to the police.
Potential data manipulation was defined as where there is no valid explanation why the CWT tool
had been altered to meet national reporting standards, data did not reflect actual patient
experience, there was variance between CWT and other data sources, there was no valid reason
and/or other sources gave rise for concern e.g. complaints.
17
15th December 2014
5. Results
The following summarises the key findings from the study, supplemented by qualitative analysis of
the auditors’ comments from the audit tool and Issues Log. In some places, the text is supplemented
by brief explanations of relevant Cancer Waiting Times guidance and codes from the Glossary
(Appendix 4).
The results are given in the following sections
1.
2.
3.
4.
Sample and patient characteristics
Data accuracy
Causes of errors
Impact of data inaccuracy
5.1 Sample and patient characteristics
The 250 records audited comprised 50 case notes drawn at random from each of the following five
tumour sites: Breast, Lower GI, Upper GI, Urology, and Others. The latter comprised the following
numbers of case notes: Gynaecology (5), Head & Neck (3), Haematology (16), Lung (16), Cancer of
Unknown Primary (CUP) (2), and Skin (8) (Figure 1).
In the sample, 51% (127) of the patients were male and 49% (123) were female (Figure 2). The
majority of patients were aged over 65 (67% or 168 patients), slightly higher than national cancer
statistics where 63% are over aged 65 years (Figure 2).4 While the majority of patients in the sample
were alive (72% or 180) at the time the audit was completed in April 2014, 28% (70 patients) were
deceased.
No. of Case Notes Audited Per Tumour Site
50
40
30
20
10
0
Breast
Lower GI Upper GI Urology
Others
(Haema)
Others
(Lung)
Others
(Skin)
Others
(Gynae)
Others
(H&N)
Others
(CUP)
Figure 1: Distribution of Case Notes Audited per Tumour Site
4
Figures from Cancer Research UK http://www.cancerresearchuk.org/cancerinfo/cancerstats/incidence/age/#Cancer
18
15th December 2014
Age Distribution
20%
18%
16%
14%
12%
10%
8%
6%
4%
2%
0%
25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90+
Figure 2: Age Distribution of Patients in Data Sample
5.1.1 Source of Referral
The patients were referred to the Trust through different routes. 36% were referred from their GP
via 2WW; 5% were routine referrals from GPs; 30% were referrals made by Consultants for
subsequent treatment; 15% were referrals made by Consultants for investigations for cancer; 10%
were referrals from a National Screening Programme, and 3% were referrals via emergency
admission (Figure 3).
Referral Source
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
104
75
37
26
7
GP
Subsequent
Treatment
Consultant
National
Screening
Programme
Emergency
Admission
1
Specialist Nurse
Figure 3: Referral Source
5.1.2 Shared Pathways
While the majority (92% or 229 patients) were treated at CHUFT, for 8% (21) of patients, the
pathways originated from CHUFT but included treatment at Ipswich Hospital NHS Trust (7 cases) and
Broomfield Hospital (14 cases).
Of all patients, 70% (175) received first definitive treatment at CHUFT, while 30% (75) received
second or subsequent treatment, perhaps because the Trust is a tertiary centre for radiotherapy.
19
15th December 2014
Patients Treated in Other Hospitals
35%
30%
7
6
25%
20%
15%
3
10%
3
2
5%
0%
Lower GI Breast
Others
(Gynae)
Others Upper GI Others
(H&N)
(Lung)
Urology
Others Others
(Haema) (CUP)
Others
(Skin)
Figure 4: Patients Treated at Other Hospitals by Speciality
5.2 Data accuracy
Nine key data items were audited critical to capturing, tracking, measuring timelines of patients on
cancer pathways, and assessing CWT performance (Appendix 3). Overall, data inaccuracies were
found in at least one of the nine key data items in 139 of 250 cases reviewed (55%). No data
inaccuracy was found in 97 of the 250 cases (39%) reviewed and in 6%5 data was missing as it was
held at other Trusts.
Overall, there were 1,548 data points mandatory for submission based on their CWT pathway type in
nine fields of which 95 could not be verified (6%). Of the 1,453 data points that could be verified,
141 had errors in them (10%) (Table 2).
5.2.1 Inaccuracy by cancer pathway
Inaccuracies are spread across the Cancer Waiting Times pathways. Treatment related data were the
most error-prone with their recording potentially the most misunderstood.
Overall, the highest error rate was found in ‘Date of Decision to Treat’ with 27% of cases including an
error in this field (57 of 214 cases), ‘Treatment Start date’ included 15% errors (35 of 234) and
‘Treatment Modality’ 5% ( 12 of 239)6 (Figure 5).
The proportion of records with data inaccuracies in the remaining six data items are: Date of
Diagnosis7 17% (13 of 75 cases); First Seen Date 7% (8 of 108 cases); Referral Received date 8% (9 of
108 cases); Source of Referral 5% (5 of 109 cases); Waiting Time Adjustment (First Seen) 2% (2 of 117
cases); Waiting Time Adjustment (Treatment) 0.4% (1 of 249 cases);
5
While 8% of the patients were treated in other Hospitals, in only 6% of cases was data missing.
There is an overlap in these fields and some of the cases with errors in one field e.g. Treatment date may also
have errors in another field.
7
Prior to mid-2013 there was no requirement to collect data on Date of Diagnosis, hence there were
noticeable inconsistencies this field.
6
20
15th December 2014
Distribution of Errors by Data Items
30%
25%
20%
15%
10%
5%
0%
Figure 5: Proportion of Errors by Data Items
5.2.2 Data inaccuracy by tumour site
There were data errors across all specialties (Figure 6). In the 139 cases where there was at least one
data error, the highest proportions of errors were in Lower GI (24% or 34 cases) and Upper GI (23%
or 32 cases), which together account for nearly half of all inaccuracies (47%).
Inaccuracies in other specialties included: Urology (17% or 24 cases), Breast (14% or 20 cases),
Haematology (8% or 11 cases), Lung (6% or 8 cases), Head & Neck (2% or 3 cases), Skin (2% or 3
cases), Gynaecology (2% or 3 cases), and Cancer of Unknown Primary (< 1% or 1 case).
Distribution of Errors by Tumour Site
30%
25%
20%
15%
10%
5%
0%
Lower GI Upper GI Urology
Breast
Others
(Haema)
Others
(Lung)
Others
(Gynae)
Others
(H&N)
Others
(Skin)
Others
(CUP)
Figure 6: Proportion of Errors by Tumour Site
21
15th December 2014
5.2.3 Spread of data errors by tumour site
The spread of errors across the pathways for each tumour site is listed in Table 3. It can be seen that
six specialties had errors in the majority (>5) of fields relating to the CWT standards.
Lower GI had the highest number of inaccurate key data items, with eight out the nine fields audited
having errors; Breast and Urology both had inaccuracies in six of the fields; Head & Neck,
Haematology, and Upper GI all had errors in five fields; Skin and Lung in three fields), Gynaecology
in two fields and Cancer of Unknown Primary in one field.
Data Items Where Inaccuracies Were Found by Tumour Site
Tumour Site
Date
of
DTT
Treat
ment
Start
Date
Date of
Diagnosis
Treatme
nt
Modality
Date
Referral
Received
Source of
Referral
Date
1st
Seen
Waiting
Time Adj.
(1st
Seen)
Total
Yes
Waitin
g Time
Adj.
(Treat
ment)
Yes
Lower GI
Yes
Yes
Yes
Yes
No
Yes
Yes
Breast
Yes
Yes
Yes
No
Yes
Yes
Urology
Yes
Yes
Yes
Yes
Yes
No
Yes
No
No
6
No
Yes
No
6
Head &Neck
Yes
Yes
Yes
No
Yes
Yes
No
No
No
5
Haematology
Yes
Yes
Yes
Yes
Yes
No
No
No
No
5
Upper GI
Yes
Yes
No
Yes
Yes
No
Yes
No
No
5
Lung
Yes
Yes
No
Yes
No
No
No
No
No
3
Dermatology
Yes
Yes
Yes
No
No
No
No
No
No
3
Gynaecology
Yes
No
No
No
No
Yes
No
No
No
2
CUP
No
No
No
Yes
No
No
No
No
No
1
Total
9
8
6
6
5
4
3
2
1
8
Table 3: Distribution of Inaccuracies in the 9 Key Data Items by Tumour Sites
Note. Number of cases reviewed - Breast, Lower GI, Upper GI, and Urology (all 50 cases each) and
Gynaecology (5), Head & Neck (3), Haematology (16), Lung (16), Cancer of Unknown Primary (CUP)
(2), and Skin (8).
22
15th December 2014
5.3 Causes of data errors
Data was coded at the end of the tool into categories set by IMT (Table 4). In 40% there is no data
error while in 39% data is available but not entered or updated on the CWT.
Code
Definition
No errors
The record is found to be accurate and complete in all fields.
Data entry errors
When CWT and other sources of information, including 40% (100
patient notes, match although the data has not been cases)
updated e.g. on another pathway, appointment brought
forward, incorrect treatment entry
When an incorrect entry appears to have been made as a 11% (28 cases)
result of a misunderstanding of the national CWT guidelines.
Misinterpretation
of the national
guidance
Operating process
issue
Potential data
manipulation
Data with other
provider
% found
(number of
cases)
39% (97cases)
A substantive error in a process or procedure, as opposed to 3% (7 cases)
the recording of that process or procedure.
Where there is no valid explanation why CWT has been 2% (4 cases)
altered to meet national reporting standards and data does
not reflect actual patient experience
• There is variance; and/or
• There is no valid reason
• Other sources give rise for concern e.g. complaints
Data available on the CWT but could not be checked against 6% (14 cases)
data in the notes which were held elsewhere.
Member of staff Clock stop on cancer pathway by any non-practicing clinician
working
outside
the scope of their
responsibility
0 (no case)
Table 4: Categorisation of data issues
To provide a more detailed analysis, where a data inaccuracy was recorded, cases could be ascribed
a further level of coding based on cause of the errors. The frequency of these errors is described
below and given in Figure 7.

39% (23 of 59 cases) were due to ‘Poor Information Sharing,’ where a discrepancy between
CWT and other records arose because information required for accurate record-keeping had
not been communicated effectively. This may include a failure to make information available
to anyone who might reasonably have use for it or failure to acquire necessary information
that is readily available.

39% (23 of 59 cases) were due to ‘Poor document handling,’ where a document had not
been processed or used correctly. Instances may include documents mislaid, misread,
incorrectly completed, misunderstood, or incorrectly reported.
23
15th December 2014

In 20% (12 of 59 cases) a treatment decision or discussion in the Multidisciplinary Team was
mis-recorded as the Decision to Treat Time. National guidance requires the Decision to Treat
Time to be the point at which treatment is agreed with the patient.

In 2% (1 of 59 cases) there were errors relating to use of Earliest Clinically-Appropriate Date
(ECAD) or mis-recording of the DTT, which resulted from confusion over the ECAD for
treatment.8
Distribution of Errors (Level Two)
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Poor information sharing Poor document handling
MDT date used in error
ECAD Issue
Figure 7: Basis of Errors (Level Two)
5.4 Impact of data inaccuracy
5.4.1 Time between recorded and actual dates
There were 105 data points on the cancer waiting time pathway (“episodes”) where auditors
calculated a difference or variance between the dates in the notes and the CWT tool. These occurred
in four fields: referral received date, date first seen, decision to treat, and treatment start date.
These data points are listed in Appendix 5.
Table 5 gives the size of difference in time between dates recorded in the notes and CWT tool for
each field, where a negative number indicates the date in the notes was before that in the CWT tool,
and positive number indicates the dates in the notes is after that recorded in the CWT tool. Figure 8
and 9 show the size and distribution of the difference.
8
An ECAD (Earliest Clinically Appropriate Date) can be inserted in the Decision To Treat field within
the CWT Tool in accordance with CWT Guidance when a patient is not clinically fit to be on a waiting
list i.e. the clock has not started because they are not yet fit to undergo the next activity on their
care pathway (para 3.10.15, page 90, CWTs guidance v8.0). The CWT database used the same field
to record DTT and ECAD.
24
15th December 2014
Comparing the dates recorded in the notes and on the CWT tool:





In most instances where an error was identified the difference between the date recorded
on the CWT and the date in the notes was small. In 58% of episodes (61 of 105) the date
recorded in the notes was within five days before or after the date recorded CWT Tool.
In the majority of episodes (65% or 68 of the 105 data points) the date recorded in the notes
was earlier than the date recorded on the CWT Tool.
Overall, only in treatment start date was the date recorded in the notes after that recorded
in the CWT tool; on average by six days. In the other three fields (referral received date, date
first seen, and decision to treat), dates recorded in the notes were, on average, before those
recorded in the CWT tool.
For 10 patients, dates recorded in the notes were between 10 and 52 days later than those
recorded on the CWT Tool, indicating that, for some patients, and especially as the pathways
may be cumulative, there would have been unacceptable delays.
For all pathways, dates recorded in the notes range from between 77 days before to 52 days
after dates recorded in the Cancer Wait Tool.
Referral
received date
8 episodes of
108
Date first seen
Decision to treat
Number
9 episodes of
55 episodes of 214
with
108
variance
Average -0.9 days
- 0.1 day
-7.7 days
Range
-3
+3
-13
+8
-77
+37
of
variance
(days)
Table 5: Summary of Date Variance Identified in 4 Data Fields
Treatment start
date
33 episodes of
234
6.5 days
-14
+52
Total
105
episodes
-77 to +52
Difference Between the Dates Recorded in the Notes and CWT Database in 5 Day
Intervals
Frequency of Occurence
40
35
30
25
20
15
10
5
0
-30+ -26-30 -21-25 -16-20 -11-15 -6-10
-1-5
1-5
6-10 11-15 16-20 21-25 26-30
30+
Number of Days Variance
Figure 8: Bar Chart to illustrate frequency of differences between notes and CWT dates in 5 day
intervals
25
15th December 2014
Difference in Recorded Date Between CWT and Notes
60
40
20
Days
0
0
20
40
60
80
100
120
-20
-40
-60
-80
-100
Data item or event in cancer pathway
Figure 9: Scatter plot of episodes of variance between dates recorded in notes and on CWT tool
Table 6 shows variance by specialty and Table 7 by specialty and field. Data is difficult to interpret
due to small numbers; however, there was a larger variance in DTT and start date, especially for
Lower GI, Lung, Haematology and Urology.
Total
No. of data items
Range of Variance
showing variance
(days)
Average Variance (days)
Tumour Site
Upper Gastro-Intestinal
24
-25 to 17
-2.8
Lower GI
25
-69 to 52
-1.5
Urology
17
-77 to 44
-4
Breast
17
-28 to 7
-3.9
Haematology
9
-20 to 37
3.2
Lung
7
-7 to 33
3.7
Skin
2
-2 to -2
-2
Head and Neck
3
-17 to 1
-6.6
Gynaecology
1
-18
-18
Cancer of unknown origin 0
TOTAL
105
-77 to 52
-2.1
Table 6: Average variance between the CWT and notes in days across 4 fields by tumour site
26
15th December 2014
Referral Received
Date
Tumour
Site (n =
cases of
variance)
Upper
GI
(24)
Lower
GI (24)
Urology
(17)
Breast
(17)
Haemat
(9)
Lung
(7)
H&N
(4)
Gynae
(1)
C.U.P
Skin(2)
Date First seen
No of
cases
Range
Aver
age
No. of
cases
2
-2
-2
2
1
-3
-3
5
3
-3 to -2
-2.3
2
-1 to 3
1
1
-1
-1
Range
-5 to
7
-13 to
8
Decision To Treat Date
Avera
ge
No. of
cases
1
15
-2.2
11
5
5
Averag
e
No. of
cases
-5.0
5
-8.7
8
-29.2
9
-6.2
1
3.8
4
-1.3
3
2
4
-25
to 17
-69
to 5
-77
to -6
-28
to 6
-20
to 37
-7 to
4
1
-17
-17
1
-18
-18
5
1
Range
13
4
Treatment Start Date
1
-2
-2
-13 to
-77
TOTAL
9
-3 to 3 -0.9 8
8
-0.1
55
to 37 -7.7
Table 7: Variance in days between the CWT and notes by field and speciality
1
33
Range
-4 to
17
-5 to
52
-4 to
44
Averag
e
2.2
8.9
9.4
7
-14
to 29
-7 to
33
-4 to
1
7
-2
-14
to 52
-2
3.8
10.3
-1.5
6.5
5.4.2 Significance of variance between CWT tool and data in the notes
The following statistical analysis considers the significance of any difference in dates between the
CWT tool and notes by pathway. If the errors were random it would be expected that the interval
to be longer in 50% and shorter in 50%.
a) Referral to First appointment (2WW)




91 patients were referred as 2WW. There are 14 patients in whom either the Referral
Received Date and/or the Date First Seen was incorrectly recorded in the
CWT. Consequently the time to first outpatient attendance would have been incorrectly
calculated from the CWT.
In four cases the actual time to first outpatient attendance (from notes) was shorter than
indicated by the CWT. For these actual performance is better than that indicated by CWT.
For 10 patients the actual time to first outpatient attendance (from notes) was longer than
indicated by the CWT. For these the actual performance is worse than that indicated by
CWT, but in only three patients was there a breach of the 14-day wait that was “missed” by
the CWT (in one patient a 16 day wait was recorded as a 14-day wait, in another a 15-day
wait was recorded as a seven- day wait and in the third a 17-day wait was recorded as an 11day wait).
This suggests a small tendency for errors to favor shorter waiting times on the CWT than
recorded in the notes
27
15th December 2014

The proportion of errors that make performance look better than it really is (10/14 (71%
(95% CI 42-92%) are not significantly different from 50% indicating this difference is likely to
be random.
b) Decision to treat to treatment (31-Day)
Using the same methodology as above on incorrect dates of Decision to Treat (DTT) and
Treatment



There were 214 patients with a verified Decision to Treat Date and Treatment Date
There were 71 patients (33% (27 – 40%)) where either the DTT or Treatment date was
incorrect so the time from DTT to Treatment may be incorrect in the CWT Tool.
- In three cases of these, the length of interval from DTT to Treatment is the same overall
based on dates recorded in both CWT tool and case notes (i.e. the data errors balance out
and do not affect the time from decision to treat and treatment)
- In 53 cases (75% (63-84%)), the interval is longer in case notes than recorded in CWT tool,
i.e. the patients waited longer in reality than that recorded in CWT tool.
- In the remaining 15 cases (21% (12 -32%)), the interval is shorter in case notes than in CWT
Tool (i.e. the patients did not wait for as long as wait recorded in the CWT tool)
If the errors were random we would expect the interval to be longer in 50% and shorter in
50%, but, as the Confidence Intervals excludes 50%, the data suggest a systematic tendency
for errors to favor the system.
c) Referral to first treatment (62-Day)
Using the same methodology for calculations on incorrect dates of Referral date and First
Treatment (1st Treatment) in the 62 day pathway



There were 108 patients where the referral and treatment date could be verified
There are 24 patients (22% (15-31%) where either the Referral or 1st Treatment date is
incorrect.
- Of these, in 14 cases 58% ((37 - 78%)), the interval is longer in case notes than recorded in
CWT tool (i.e. the patients waited longer in reality than that recorded in CWT tool).
- In the remaining 10 cases (42% (22-63%)), the interval is shorter in case notes than in CWT
Tool (i.e. the patients did not wait for as long as wait recorded in the CWT tool)
Again this suggests a systematic tendency for errors to favor the system, but as the CIs
include 50% the tendency is not statistically significant.
Overall, there is some evidence of systematic bias making waits seem shorter than in reality in DTT
dates recorded. This could have resulted from multiple causes including erroneous interpretation of
complex CWT guidance (see sections 5.3 and 7.4), such as a decision recorded as the date the MDT
made the recommendation rather than the date the patient made the decision, or even
subconscious bias. Data from other Trusts have not been analysed to such a degree as a comparator.
The methods of this study did not allow an assessment whether this led to gain, for example, in
recording of performance.
28
15th December 2014
5.4.3 Clinical pathways and care
Seven cases were referred to the Assurance Panel for potential data manipulation and/or clinical
harm. Following in depth discussion and review the following outcomes were agreed (Table 8).
Number of cases referred to Assurance Panel
4 cases of “potential data manipulation”
2 Serious Incidents raised by the Retrospective
Review Team discussed
1 Case potential clinical harm discussed due to
routine referral for red flag symptoms
Table 8: Outcome of Review by Assurance Panel



Outcome of review
No clinical harm and does not meet criteria for
referral to police. Agreed variance caused by
“potential data manipulation” or
“Misinterpretation of the national guidance”
Two cases raised for investigation as SIs by the
team and supported by Assurance Panel
Referred to GP as significant event
All four cases identified by the auditors as requiring consideration for “potential data
manipulation” were deemed by the Assurance Panel to have involved no clinical harm and
therefor did not meet the criteria for referral to police. It was agreed that the data variance
was caused by “potential data manipulation or misinterpretation of the national guidance”.
Three cases of the 250 patents in the sample were deemed by the Assurance Panel to have
potentially experienced clinical harm, of which two had already been referred through the SI
process9 by the team. One was a GP significant event10, given that a routine referral was
made despite red flag symptoms. No SI was attributed to potential data manipulation.
As a result of the review, the Trust is instituting a new policy for repeat Chest X-Rays, where
there is lung consolidation and pre-operative anaesthetic assessment, as well as
implementing related actions in the Cancer Action Plan.
Of the 21 cases treated at other hospitals, in 12 the potential impact on patients’ care could not be
established. Cases reviewed by auditors will be shared with other hospitals at the end of the review,
unless patient care is affected and more urgency is required.
5.5 Correlation with Patient surveillance log
During the CQC investigation, several processes were established to manage patient calls,
complaints and significant events. All patients who contacted the Trust in this way were recorded
on a “surveillance” log to ensure actions were followed and documented. There were 10 patients on
the log also reviewed by the review team (Table 8).
The log identified three additional SIs for unrelated issues that would be picked up in other audits
e.g. Long Waits and Delayed Diagnoses, bringing the total to five SIs in the study sample.
9
It is important to note that, during the inspection period, the threshold for referral for Serious Incident
investigation was lowered.
10
Significant Event Audit - also called Significant Event Review or Analysis - is an increasingly routine part of
general practice. It is a technique to reflect on and learn from individual cases to improve quality of care
overall i.e. the primary care equivalent to a Serious Incident. http://www.rcgp.org.uk/clinical-andresearch/clinical-resources/quality-improvement/significant-event-audit.aspx
29
15th December 2014
Origin in log
Number of cases on the log who called the
helpline
Number of cases on the log who had made
complaints
Number of cases on the log who were recorded
as SIs
Number of cases
6
1
3 cases for unrelated issues that would be
picked up in other audits e.g. long delay,
delayed diagnosis.
Table 9: Correlation between study sample and log of SIs, complaints and calls to the patient
helpline related to cancer
Of these 10 patients, three each were in Urology and Upper GI, and two each were in Lower GI and
Haematology.
Of the six patients who had phoned the Trust’s helpline, four required no further action, while the
remaining two were referred to the Trust’s Cancer Programme Director. The patient who made a
complaint was not deemed to have suffered any potential clinical or emotional harm and, as a result,
did not require any further action.
30
15th December 2014
6. Royal Marsden Hospital Review (Appendices 7 and 8)
The Royal Marsden Hospital (RMH) was asked to support the Retrospective Review of records for
cancer patients at CHUFT by undertaking an independent audit of the review teams’ application of
CWT guidance. RMH did this by reviewing a 10% (25) sample of audited patient notes.
Of the 25 cases reviewed, RMH agreed completely with Colchester audit results in 18 cases and was
unable to agree or disagree with the Trust results in a further two cases because treatment was
ultimately carried out at another hospital. Of the remaining five cases, the RMH reviewers disagreed
with the Trust’s assessment in at least one CWT data field out of nine per patient (Appendix 1).
The purpose of the RMH review was not to identify any potential data manipulation or patient harm
but to assess the accuracy of the assessments made by the Trust’s review team. On discussion of
these findings, the RMH agreed that:



CWT guidance is highly complex and therefore can be open to misinterpretation. Therefore
the importance of commissioner-agreed local guidance was emphasised.
The differences between the RMH and Colchester assessment were due to technical aspects
of CWT guidance and/or the complexity of the pathways reviewed.
To maintain high data quality standards providers need to regularly audit their CWT data to
identify and rectify any errors.
Under these circumstances, the RMH review does not invalidate the findings of the Colchester audit
but rather provides a helpful pointer towards the need for greater data accuracy and contextualises
the audit, illustrating the complexity of the CWT rules and the need for vigilance in interpreting them
appropriately.
31
15th December 2014
7. Areas identified for improvement from qualitative
analysis
Non-clinical action is defined as action taken to change administrative, technical, or clerical activity
over and above the care of an individual patient. Themes from comments relating to cases coded for
“nonclinical action” are listed below together with those from the Issues log:
7.1 Poor record-keeping
Record-keeping, both electronic and written, is patchy. Records are often incomplete, disordered,
illegible, and sometimes inaccurate. The physical state of case notes is often decrepit and binding
antiquated. The quality of electronic record-keeping is highly variable across the Trust, reflecting
varying IT literacy.
7.2 Decentralised record-keeping
Patient records are frequently stored across multiple locations within the Trust, both physically and
electronically (e.g. Oncology ‘blue’ notes, radiology notes, separate CNS contact notes). As a result,
staff may not know where to find important information, be unable to access it, or may not know
that it is held at all. This applies to paper records, but even more so to electronic systems. There are
numerous clinical systems in use, apparently with no facility for aggregating their data into a single
interface.
7.3 Misunderstanding of Cancer Waits guidance
There is evidence of a lack of understanding about Cancer Waits guidance.11 Examples include when
pathways start and stop, timing points within pathways, and patients being put on the incorrect
pathway including treatment.
7.4 Misunderstanding of “Decision to Treat”
In particular, there appears to be greater lack of understanding of what constitutes the Decision to
Treat (DTT) date leading to incorrectly recording it as the Outpatient appointment following the
correct DTT; the MDT discussion date; the date the patient signed the consent form (rather than the
discussion several days previously); the date seen by palliative team (treatment start date); the date
of the pre-op assessment; the date of an emergency admission; the date of investigations; and the
date of follow-discussion about surgery (rather than when surgery was decided upon).
7.5 Poor information sharing with other providers
There have been a significant number of cases in which the full patient record was not available to
auditors because information about parts of the pathway was held by another provider or providers.
11
Department of Health, “Cancer Wait Times (CWTs) A Guide” Ver. 8.0 (2011).
32
15th December 2014
8. Discussion and conclusion
This audit indicates considerable inaccuracy in the Cancer Waiting Times tool due to multiple
reasons across most specialties reviewed. Although the actual error rate of 54% may not be a true
picture, as the sample was taken from a selected cohort of specialties with known issues, over half
of cases having a data inaccuracy indicate poor data quality and recording. The condition of the
health care record is poor.
While most errors were relatively minor, and indicate that many patients would have been seen
before the date recorded on the CWT, the difference between the dates in the notes and that
recorded on the CWT pathway indicate a minority of patients would have experienced unacceptable
delays.
There is some evidence of systematic bias on statistical analysis of the data which make waits seem
shorter than in reality. This could have resulted from multiple causes including erroneous
interpretation of complex CWT guidance or even subconscious bias. In the review of individual cases
however by the assurance panel, evidence of data manipulation was not identified.
Data from other Trusts have not been analysed to such a degree as a comparator. The methods of
this study did not allow an assessment whether this led to gain, for example, in recording of
performance.
The picture is of a problem with data quality. There were pockets of particularly poor data recording
including Upper and Lower GI and Urology especially around treatment dates. The size of the
variation between CWT and data recorded in the notes is unacceptably large especially in Urology
and Lower GI, but also in other specialties.
In the sample of 250 cases, there were five that were referred for investigation as Serious Incidents.
Two referrals arose from the review and three had been referred previously via the helpline,
complaints, or the SI process. These latter three SIs related to aspects of care other than those
covered by the Error Rate audit. It should be noted that, during the period of the helpline, the
threshold for SI investigations was lowered.
As a result of review of these investigations, the Trust has put new policies in place and reinforced
aspects of the Cancer Action Plan. As one significant event was as a result of GP referral on to a noncancer pathway, and as 5% of GP cancer referrals were routine, the Trust needs to work closely with
primary care to improve early recognition and referral of cancer across the system. The Trust needs
to work with other hospitals to improve data sharing as, in 14 of 21 cases of shared care, insufficient
data were available to complete the audit.
33
15th December 2014
9. Recommendations
The following recommendations have been reviewed by the Cancer Programme Director and
associated actions already included in the Cancer Action Plan.
9.1 There should be improved staff training in Cancer Waiting Times (similar to Referral to
Treatment (RTT) and greater accountability for record keeping. The Trust needs to recognise the
importance of record-keeping and ensure that it is reflected in the job descriptions of those
responsible for it. There should be a set of core competencies and an on-going programme of
assessment. It is key that staff understand the importance of the data they are recording as part of
the Patient’s Clinical Record.
Action Taken:
As part of the Cancer Action Plan, the Trust is
1
2
3
Developing an e-learning module relating to CWTs guidance which will be used to test
understanding of CWT guidance, its application, and competences/skills required.
A set of MDT protocols is being developed for the MDT Co-ordinator team which will enable
robust cross –cover for MDTCs during periods of absence.
As part of the Cancer Action Plan, there is a standing agenda item on the MDT Co-ordinator
(MDTC) weekly team meeting for training on the CWTs guidance. The MDTC team determine
which aspect of CWTs guidance is covered each week.
9.2 There should be an ongoing process to assess data quality in cancer through a rolling programme
of audits reporting to the Board, which also reports on the quality of case notes as well as accuracy
of electronic records.
9.3 Any healthcare provider treating a patient should have ready access to that patient’s entire
record and history. There should be a robust and clear method for inter-trust communications. This
deficiency may be remedied to some extent by the introduction of digital record-keeping across the
NHS.
Action Taken:
As part of the Cancer Action Plan:
1
2
3
The implementation of the Somerset Cancer Registry (SCR) has enabled closer cross-working
with neighbouring Trusts. Mid Essex Hospitals NHS Trust has granted CHUFT read-only access to
its SCR system, reducing the need for patient data to be transmitted between organisations.
Once the roll-out of Phase 2 (MDT Live Data Collection) has been completed, CHUFT will grant
Mid Essex Hospitals read only access to its SCR system (estimated for end September ’14).
As a minimum, all inter-trust referrals should be accompanied by the Department of Health (DH)
defined Minimum Data Set for Cancer. An SCR module is being developed, which will generate
inter-trust referrals with the Minimum Data Set.
CHUFT is working with the SCN (Strategic Clinical Network)/ ECRIC (National Cancer Registration
Service, Eastern Office) to agree an Essex-wide inter-trust referral policy which sets out a clear
and robust methodology for transmission of data between Trusts. This policy is based on the
Anglia Clinical Network Intra Trust Referral Policy, which has been adopted by Ipswich Hospital.
9.4 Patient paper records should be stored together wherever possible. There needs to be a central
inventory of clinical systems with a view to their integration wherever possible. Departments and
34
15th December 2014
service areas need to have a clear map of where they store data and who has access rights. The
patient’s entire record should be accessible via a single interface. The Trust should consider
implementing an electronic patient record as a high priority.
9.5 This report should be fed back to clinical teams to consider the impact of data quality and
develop an action plan. Check reports have already been implemented between the MDT and
Business Informatics teams – to flag differences between PAS and the SCR.
9.6 Breaches should be continually monitored, especially around treatment dates, and daily
reporting put in place to escalate patients waiting, as well as feedback on data quality to the
Divisions and specialties.
9.7 The NAO recommends that Clinical commissioning groups and trusts should work together to
impress on patients their rights and responsibilities. Almost all patients in the small sample we
interviewed were unaware of the 18-week maximum waiting time and the implications if they failed
to attend their appointment – although patients failing to turn up for appointments is a longstanding challenge which should not be underestimated.12 The Trust and CCG should work with
patients to understand their rights and responsibilities around cancer waiting times.
12
Department of Health, “NHS waiting times for elective care in England,” HC964 Session 2013-14 23 January
2014, National Audit Office, p. 11.
35
15th December 2014
Appendix 1: Study Protocol
Study Title:
To determine the error rate of the Cancer Waiting Tool
Chief Investigator:
Dr. Christine Macleod Medical Director Essex Area Team NHS England
[email protected]
Sponsor:
Incident Management Team, Essex Area Team, NHS England
Confidentiality Statement
This document contains confidential information that must not be disclosed to anyone other than
the Sponsor, the Investigator Team, host organisation, unless authorised to do so.
Background to the project
The Care Quality Commission (CQC) report into Cancer standards at Colchester Hospital University
Foundation Trust (CHUFT), published on 5 November 2013, identified a number of failings in Cancer
Services in the Trust including “in 22 cases people were at risk of or did experience delays in their
care which could have a negative impact on the person using the service.”
An analysis by the NHS IMAS Intensive Support Team of cancer data indicated that 40,489 patients
had undertaken 66,997 cancer pathways at CHUFT from 1 April 2010 to date. The Cancer Waiting
Tool (CWT) data do not all fully reconcile with data in other systems such as PAS. There may be valid
reasons for this variance which can be clarified by this audit.
CHUFT cancer waiting time performance reporting is based on output from the CWT. If the CWT data
are inaccurate, the actual performance of CHUFT against the National Standards for Cancer Waiting
Times may also be inaccurate.
Aim of the study
The aim of this study is to assess the historical and current accuracy of the data held on the Cancer
Waiting Tool so that accurate Trust cancer waiting time performance can be assured.
Objectives
Objectives
Outcome Measures/Endpoints
Primary Objective
1
To determine the historical and current
accuracy of the data on the Cancer
Waiting Tool
% data accuracy of CWT April 2010 – November
2013
% data accuracy of CWT now in 2014
Secondary Objectives
2
To estimate the impact of the CWT
data accuracy on cancer waiting time
performance reporting
Estimated impact on historic / current
performance reporting
36
15th December 2014
Experimental design and methods (including statistical analysis)
1. Randomised sample of 250 case notes, for patients on the 31 or 62 day pathway between 1 April
2010 and 31 December 2013 (as advised by Dr. Paul Pharoah, Public Health & Primary Care,
University of Cambridge)
2. Sample stratified by the four cancer pathways identified by the CQC inspection, namely Urology,
Breast, Upper GI, Lower GI - 50 case notes from each pathway,
3. In addition, a random sample of 50 case notes across the other cancer pathways,
4. Scope for additional second stage sampling of specific cancer pathways if serious problems
identified in first stage,
5. Identification of key metrics from case notes:
•
•
•
•
•
•
•
•
•
referral source (GP, consultant, screening, other)
referral date
date referral received
(same as date referral received for 62-day pathway and date of decision to treat for nonurgent pathways) date first seen (where patient was a GP target referral or a screening
referral)
diagnosis date (if available retrospectively, as not ‘mandatory’ until 2013)
decision to treat date
date of First Definitive Treatment (FDT) as per national guidance
details of treatment / assessment of appropriateness of whether recorded treatment was
a recognised First Definitive Treatment (in line with national guidance – may require
independent specialist input to assess specialist clinical management)
date of subsequent treatment(s) or other appropriate actions e.g. pauses and recording of
DNA
6. Compare case note metrics:
•
•
•
•
with data on CWT
with data on the CNS database(s)
with data on Open Exeter
against National Standards for Cancer Waiting Times
7. Analyse error rate(s)
8. Further sampling of case notes as indicated
9. Repeat audit of a sample of notes to be undertaken in late February / March 2014 to assess any
difference in the error rate post the Trust interventions in late 2013 (precise methodology to be
confirmed following results of initial audits as above)
Ethical considerations
Ethical considerations and information governance issues that need attention include:

data confidentiality
37
15th December 2014

The essential elements of this review are technically audits and can be undertaken by those
who have a legitimate clinical relationship with the patient.
However given the amount of scrutiny that the Trust is under, the protocols for such reviews will be
agreed with NHS England, who will also provide external assurance, validity and scrutiny of the
process (including the checking of the accuracy of any reviews under the section 251 application).
Resources and costs
1. Decide on funders.
2. Scale of work - randomisation and pulling of case notes = 10 minutes per case note = 42
hours / 5 days
3. Scale of work - 40 minutes per case note = 170 hours / 21 days = £10k @ 500 per day, if at
clinical consultant rate.
4. Scale of work - analysis of data = estimate 3-4 days = £1-2k
5. Scale of work - report = estimate 1 day = £0.5k
6. Cost - approximately £15k but need clarification of costs for point 2 above.
7. Cost of 2014 re-audit sample to be determined
38
15th December 2014
Appendix 2: Data on Cancer Survival, Mortality and Referrals
Table 1: NE Essex (NEE) CCG one-year survival index (%) within normal range for all cancers
combined for adults (aged 15-99 years)
Source: ONS and National Cancer Registry
75.0
73.0
71.0
69.0
67.0
England
65.0
63.0
Essex
61.0
North East Essex
59.0
57.0
55.0
Source: ONS and National Cancer Registry
Table 2: NEE CCG one-year survival index (%) for all cancers combined compared to Essex CCGs
Source: ONS and National Cancer Registry
Table 3: NE Essex 1 and 5 year survival rates for cancers within key specialties are all within the
normal range
1 year survival
5 Year survival
Breast
Normal range
Normal range
Colorectal
Normal range
Normal range
Lung
Normal range
Normal range
Prostate
Normal range
Normal range
Urological
Normal range bladder, kidney,
Normal range bladder, kidney,
testicular
testicular
39
15th December 2014
Source: PH England data and knowledge gateway: National Cancer Intelligence Network e-atlas
Table 4: NE Essex mortality 2008-2012 by cancer type all within normal range except prostate, lung
and stomach which are significantly lower than UK average
Mortality for 2008-2011 and 2009-2012: National Cancer Intelligence
Network
Normal range
Normal range
Normal range
Persons significantly low, female low, male normal limits
Significantly low
Normal range bladder (male & female), kidney (male & female), testicular
Normal range cervical, uterine, vulval, ovarian cancer (male & female)
Normal range non-Hodgkin’s lymphoma (male & female)
Normal range oesophagus (male & female)
All cancers
Breast
Colorectal
Lung
Prostate
Urological
Gynaecological
Haematological
Upper GI
Stomach (all person low, male & female normal)
Skin
Normal range melanoma (male and female)
Brain
Normal range (male and female)
Melanoma
Normal range (male and female)
Pancreas
Normal range (male and female)
PH England data and knowledge gateway: National Cancer Intelligence Network e atlas NE Essex
Table 5: Total number of patients referred for 2WW is increasing although not translating into
increased numbers diagnosed and treated
Not Treated
Treated
1800
1600
1400
1200
1000
800
600
400
200
01/08/2013
01/06/2013
01/04/2013
01/02/2013
01/12/2012
01/10/2012
01/08/2012
01/06/2012
01/04/2012
01/02/2012
01/12/2011
01/10/2011
01/08/2011
01/06/2011
01/04/2011
01/02/2011
01/12/2010
01/10/2010
01/08/2010
01/06/2010
01/04/2010
01/02/2010
01/12/2009
01/10/2009
01/08/2009
01/06/2009
01/04/2009
0
40
15th December 2014
Appendix 3: Methodology for Obtaining Sample Data
1. Data Source: Historic record of Open Exeter quarterly submissions
2. Date Parameters: Data submitted to Open Exeter from Quarter One 2010/11 to Quarter
Three 3 2013/14 (i.e. April 2010 to December 2013)
3. Summary of procedures:
i.
All the records from the specified period (i.e. data submitted to Open Exeter) were
extracted and copied in to an Excel document.
ii.
The data were then filtered to the specific tumour sites required for the audit and
copied to a selection work sheet.
iii.
The selection worksheet contains formulas required to randomise the sample and to
identify any patients selected for previous audits.
iv.
A randomised number was generated against each record and records selected for
previous 2013 audits flagged by their NHS numbers or PAS numbers for exclusion.
v.
The randomised sample was then sorted and the first 50 for each of the 5 cohorts
were selected as the sample data.
vi.
An additional 16 patients were selected to bring the sample size to 266 records as a
contingency in case notes could not be found on time.
vii.
Following an initial assessment by the RRT, the 266 data sample was stratified down
in order to limit the sample size to 250 records in line with the audit’s requirement.
This was done by removing a total of 27 records using the following criteria: (a) 14
excess records yet to be reviewed in two cohorts (i.e. eight in Breast and six in Lower
GI); (b) seven duplicate records; (c) two records where the pathway type could not
be determined; (d) four records where pathways commenced outside the specified
date parameters. This resulted in a total of 239 unique records leaving a shortfall of
11 records.
viii.
To augment for observed shortfall, 11 additional records were selected at random
using the same methodology described in steps ii to vi above in the 3 cohorts where
the number of records was less than 50, viz Upper GI- 4, Urology- 3, other tumour
sites-4.
ix.
At some point, the RRT ran out of notes to audit and, to avoid any delay, an
additional 53 records were drawn at random, proportionate to the outstanding
number of records to be audited in each of the five cohorts.
x.
Once the threshold of 50 was reached in a cohort, case notes in that cohort were no
longer audited.
Notes on analysis
1. Patient age was determined by how old they were on the day they commenced treatment
rather than their age on the day their referral was received. This is because the date of
receipt of referral was not recorded for many of the patients in the sample.
2. Based on Cancer Outcome and Service Dataset (COSD) and CWT reporting guidelines, age
distribution of patients on cancer pathways is usually grouped into “Children” (aged under
16), “Teenagers and Young Adults” (TYA – aged 17-24) and “Adults” (aged 25 and above). To
enable a robust analysis however, the Office of National Statistics (ONS)’s 5-year age
distribution model was adopted.
3. None of the patients in the sample data was aged under 25 and so the cancer pathways in
young adults were not reviewed.
4. Certain elements of the shared pathway could not be verified for accuracy since the
information is held in other Trusts.
41
15th December 2014
5. In line with national statutory CWT reporting standards, certain data items on patients on
routine and subsequent treatment pathways are not mandatory for national submissions; as
a result many of these were not reviewed in the audit.
42
15th December 2014
Appendix 4: Audit Tool and Fields
Field
Pt Identifier
Pt in other Audits?
cwtMainId
Hospital Number
First Name
Second Name
NHS Number
Patient DoB
Pt Age Today
Pt Age @ Tx
Patient Sex
Death Date
Cancer Type
Description of source of referral
Priority Type Description
Cancer Referral to Treatment Period Start Date
Start Variance (Days)
Cancer Type 2 Week Wait
Date first seen
Date first seen variance (days)
Cancer treatment period start date
Decision To Treat Variance (Days)
1st or 2nd definitive treatment
Hospital of Treatment
Treatment start date
Treatment start date variance (days)
Treatment modality
Existing Complaint?
Existing Serious Incident?
Contacted Helpline?
Description of source of referral
Cancer Type (Re-grouped)
In SI list?
Pt Age Group
Cancer Type (Re-grouped)
Category
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Auto-populated
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Revised Pathway
based on treatment Event
Patient on Correct Pathway/ Cancer type?
Patient on Correct Pathway?- for analysis
Patient on Correct Pathway (Comments)
Referral Source Correct? (Code One)
Referral Source Correct? (Code One)- for analysis
Added later for analysis
Auditor-populated
Added later for analysis
Auditor-populated
Auditor-populated
Added later for analysis
43
15th December 2014
Field
If no, enter correct source
If no, comment (Referral Source)
Date Receipt of Referral Correct? (Code One)
Date Receipt of Referral Correct? (Code One)- for analysis
Date Receipt of Referral if not correct
Referral Receipt Date Variance (Code Two)
Referral Receipt Date Variance (Code Three)
Referral Receipt Date Variance (comment)
First seen date correct? (Code One)
First seen date correct? (Code One)- for analysis
If no, actual First Seen Date
Reason for date first seen variance (Code two)
Reason for date first seen variance (Code three)
Reason for Date First Seen Variance (comment)
Date First Seen Adjustment made?
Legitimate Date First Seen adjustment (Code one)
Legitimate Date First Seen adjustment (Code one)- for analysis
Legitimate Date First Seen adjustment (Code two)
Legitimate Date First Seen adjustment (Code three)
Number of days' Date First Seen adjustment
Legitimate Date First Seen adjustment ('No') - comment
Date of diagnosis (NB may not be available pre-2013)
Date of diagnosis correct? (Code One)
Date of Dx correct? (Code One) - for analysis
Reason for Diagnosis Date variance (Code two)
Reason for Diagnosis Date variance (Code three)
Reason for Diagnosis Date Variance - Comment
Decision to treat date correct? (Code One)
If no, correct Decision To Treat date
Decision to treat date correct (Code two)
Decision to treat date correct (Code three)
Decision to treat date correct (Comment)
Treatment start date correct? (Code One)
Actual treatment start date
Treatment start date correct? (Code two)
Treatment start date correct? (Code three)
Treatment start date correct? (Comments)
Treatment start date adjustment made?
Legitimate treatment start date adjustment (Code one)
Legitimate treatment start date adjustment (Code two)
Legitimate treatment start date adjustment (Code three)
Number of days treatment start date adjustment
Legitimate treatment start date adjustment ('No') - comment
Category
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Added later for analysis
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
44
15th December 2014
Field
Recorded treatment modality correct? (Code One)
If no, state recorded treatment
Planned 2nd/subsequent treatment? (Code One)
Planned 2nd/subsequent treatment? (Code One) -for analysis
2nd/subsequent treatment received?
Auditor comments on pathway
Who else was involved in audit?
Final code
Possible Impact (Code)
General comments
Comment Summary DT
Code 1
Code 2
Action(s) agreed (Code)
Assurance Panel
Name of Auditor
Record Locked?
Date completed
Actions taken
By Whom
Date
Supplementary Evidence Required
Category
Auditor-populated
Auditor-populated
Auditor-populated
Added later for analysis
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Added later for analysis
Added later for analysis
Added later for analysis
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Auditor-populated
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Revised Pathway
based on treatment Event
Priority of Ref
Added later for analysis
Added later for analysis
2 Week Wait LENGTH:
Ref-1st Seen
Added later for analysis
31Day LENGTH:
Decision To Treat-treatment
62Day LENGTH: Ref-1st treatment
Breach?
W/E Date Completed
?variance (all)
?variance (to include)
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Variance
in Date?
Date of Receipt of Referral Variance
Added later for analysis
Added later for analysis
Date of clinician review
Clinician Comments
Clinician Name
Previous Action Code
Previous Final Code
Previous Harm Code
45
15th December 2014
Field
Date 1st Seen Variance
DDT Variance
D treatment Variance
Category
Added later for analysis
Added later for analysis
Added later for analysis
Variance 2 Week Wait LENGTH:
Ref-1st Seen
31D
62D
Ayo's Comments
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Impact on accuracy
of CWT Performance
Audit Status
Refer McStay
Hospital of Treatment
Combined target breached
Pt Age
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
Added later for analysis
46
15th December 2014
Appendix 5: Glossary
Level One Codes:
0 Correct:
The CWT entry is verifiable from case notes or other internal (Trust) sources. This requires no further
specification
1 Incorrect:
The CWT entry conflicts with the information contained in case notes or other internal sources. This
can be further specified with seven Level Two codes:
 A Entry not in accordance with CWT Guidance
 B Entry not in accordance with Trust policy
 C Typographical error
 D Process error
 E Unexplained
 F Other
 G Suspected manipulation.
2 Not on CWT:
A given entry is missing from the CWT database. There is no further specification.
4 Unable to Verify CWT:
While it cannot be demonstrated to be incorrect, the CWT entry cannot be corroborated from case
notes or other internal sources. This can be further specified with two Level Two codes, ‘H Data
Missing’ and ‘I Data with other provider’ (See below).
Level Two Codes:
The following codes (A-G) are used to further specify the Level One code ‘1 Incorrect’.
A Entry not in accordance with CWT guidance:
An incorrect entry appears to have been made as a result of a misunderstanding of the national CWT
guidelines. For Decision to Treat only, this can be further specified with three Level Three codes:
 iii MDT date used in error
 iv ECAD issue
 vi Nullified by concurrent treatment
B Entry not in accordance with Trust policy:
An incorrect entry appears to have been made as a result of a misunderstanding of Trust policy.
There is no further specification.
C Typographical error:
An error that appears to result from carelessness, failure of keyboard skills, oversight, etc. It must
be apparently unintentional and not motivated by a misunderstanding of guidance or policy. There
is no further specification.
D Process Error:
A substantive error in a process or procedure, as opposed to the recording of that process or
procedure. This can be further specified with two Level Three Codes ( I Poor Information Sharing
and II Poor Document Handling).
47
15th December 2014
E Suspected manipulation:
A discrepancy appears to result from a deliberate attempt to misrepresent the patient pathway as
recorded in case notes or other non-CWT sources. Although it will most likely be an alteration to the
CWT that improves apparent performance, it does not necessarily have to be. There is no further
specification.
F Unexplained:
An error that cannot be otherwise explained. There is no further specification.
G Other:
This code is used for anything that does not fit under any of the Level Two codes. There is no further
specification.
The following two Level Two codes are used to further specify the Level One code ‘3 Unable to verify
CWT’.
H Data missing:
Corroborating information, which should be available in Trust-held patient records, cannot be
found. There is no further specification.
I Data with other provider:
Corroborating data can reasonably be expected to be held by another provider and so is unavailable
to the auditor. There is no further specification.
Level Three
Level Three codes further specify Level Two codes. The first two codes are available to all variance
fields. The remaining three apply to the Decision to Treat Time (DTT) variance field, only.
i) Poor information sharing:
A discrepancy between CWT and other records arises because information required for accurate
record-keeping has not been communicated effectively. This may include a failure to make
information available to anyone who might reasonably have use for it or failure to acquire necessary
information that is readily available.
ii) Poor document handling:
A document has not been processed or used correctly. Instances may include documents mislaid,
misread, incorrectly completed, misunderstood, or incorrectly reported.
Decision to Treat Time-only codes
iii) MDT date used in error:
A treatment decision or discussion in the Multidisciplinary Team is mis-recorded as the Decision to
Treat Time. National guidance requires the Decision to Treat Time to be the point at which
treatment is agreed with the patient.
iv) ECAD issue:
Any mis-recording of the DTT that results from confusion over the Earliest Clinically-Appropriate
Date for treatment.
vi) Nullified by concurrent treatment:
48
15th December 2014
The Date First Seen (DFS) coincides with the date of first treatment; e.g. the excision of a polyp or a
diathermy carried out at the same time as a cystoscopy.
Final codes
The RRT apply the following ‘Final Codes’ to each audited record as a whole. Where several failings
are present in a record (i.e. several fields with variance are found), the auditors will apply the code
corresponding to the worst failing found. This assumes that the codes ascend in degree of gravity
(i.e. a misinterpretation of guidance is more serious than a data entry error and potential data
manipulation is worse than a member of staff working outside their responsibility). So, for example,
if a record contains instances of typographical errors and operating process errors, code 3 would be
applied to the record as a whole due to process errors being more serious than typographical
mistakes.
0. No data inaccuracy:
The record is found to be accurate and complete in all fields.
1. Data entry error:
When CWT and other sources of information, including patient notes match although the data has
not been updated e.g. on another pathway, appointment brought forward, incorrect treatment
entry.
2. Misinterpretation of national guidance:
For example, pausing the pathway, when the pathways have been changed or recorded that is
consistent with national guidelines.
3. Operating process issue:
A substantive error in a process or procedure, as opposed to the recording of that process or
procedure.
4. Member of staff working outside the scope of their responsibility:
Clock stop on cancer pathway by any non-practicing clinician.
5. Potential data manipulation:
Where there is no valid explanation why CWT has been altered to meet national reporting standards
and data does not reflect actual patient experience
• There is variance; and/or
• There is no valid reason
• Other sources give rise for concern e.g. complaints
6. Data with other provider:
Data unavailable as a result of treatment occurring at other hospital.
49
15th December 2014
Appendix 6: Variance (in days) for Data Items with Incorrect
Dates
Variance
(No. of
Days)
-77
-69
-42
-28
-25
-23
-20
-18
-17
-14
-13
-12
-10
-8
-7
-6
-5
-4
-3
-2
-1
1
2
3
4
5
6
7
8
14
15
17
29
33
37
43
44
52
Grand
Total
Date
First
Seen
Decision
To Treat
Date
1
1
Referral
Received
Date
Treatment
Start Date
1
1
1
1
1
1
1
1
2
1
2
3
3
2
2
3
1
1
1
1
1
3
2
10
5
2
1
1
1
3
3
2
3
1
1
1
1
2
6
5
3
1
2
1
1
1
1
1
1
1
1
1
1
8
55
9
1
1
1
33
Grand
Total
Percentage
1
1
1
1
1
1
1
1
2
2
3
3
3
3
3
3
2
4
6
6
20
10
5
3
1
4
1
2
1
1
1
2
1
1
1
1
1
1
105
1%
1%
1%
1%
1%
1%
1%
1%
2%
2%
3%
3%
3%
3%
3%
3%
2%
4%
6%
6%
19%
10%
5%
3%
1%
4%
1%
2%
1%
1%
1%
2%
1%
1%
1%
1%
1%
1%
50
15th December 2014
Appendix 7: The Royal Marsden NHS Foundation Trust
independent review of Colchester University Hospital NHS
Foundation Trust Retrospective Review Team audit results
Introduction
The Royal Marsden (RMH) was requested by Colchester University Hospitals NHS Foundation Trust
(CHUFT) to support their Retrospective Review of records for cancer patients at CHUFT following the
CQC review by undertaking an independent audit of the application of the CWT guidance by the
Retrospective Review Team (RRT).
Methodology
 A sample of 25 patient notes were provided by the RRT for review by RMH team
 RMH used the same principle as external auditors (Deloitte) use in mandatory audits of
Cancer Waiting Time data in all NHS Trusts’ Quality Accounts.
 RMH undertook this audit of records at CHUFT on 24th April and 1st May 2014.
 RMH carried out a three phase process:
o First review was carried out ‘blind’ (i.e. with no reference to the Retrospective
Review Team (RRT) audit results). This process reviewed electronic and paper
records provided by CHUFT and included:
 Hospital case notes , including nursing notes, clinic letters, oncology
‘blues’, and radiotherapy data
 Patient Administration System
o Second review compared RMH audit results to the RRT audit results
o The RMH auditor presented her findings to an RMH validation team consisting of the
Director of Performance & Strategy Implementation, the Service Manager for
Performance and the Head of Information
Findings
 RMH has set out its findings case by case in the attached, which outlines whether RMH
agrees with the RRT audit results, the rationale for RMH’s findings and reference to the
national guidance.
 Of the 25 cases RMH agreed with the RRT audit results in 18 cases. RMH disagreed in five
cases and in two cases RMH was unable to audit due to the treatment having been carried
out at another hospital.
 In the five5 cases where RMH disagreed with the RRT audit:
o two cases should not have had a new CWT record – in one case this is due to specific
guidance for that other tumour type, in the other the patient was continuing on
active surveillance
o In the other three cases RMH disagreed with at one or more of the Trust’s CWT data
fields
Nicky Browne
Director of Performance and Strategy Implementation
The Royal Marsden NHS Foundation Trust
25 June 2014
51
15th December 2014
Appendix 8: Marsden review of cases for Colchester Hospital
NHS Foundation Trust as part of retrospective review – cases
where there was disagreement with the Trust
Patient
CWT data
identifier fields
1
Agree With
Retrospective
Review (Y/N)
Rationale for decision
Guidance reference
Agree
N
Referral
received date
Date first seen
Treatment
type
Referring tumour type ENT was the referring
condition, haem was the
final diagnosis but haem
was recorded as the
referring condition.
Biopsy effectively
removed the tumour so
counts as FDT as per
GFOCW v8.0 section
3.9.24
Disagree
Referring
tumour type
Clock stop.
Clock stop - TWR clock
start should have been
13/3/12 not 14/3/12.
Correct treatment
reported but marked as
an error by the RRT.
Patient had excision
biopsy, intention was to
remove tumour. Clearly
documented that tumour
was effectively removed
by the excision biopsy.
Local
guidance
needed
DTT
2
Agree
N
Referral
received date
Referring
tumour type
Date first seen
Clock stop
Treatment
type
Disagree
DTT
DTT - as diagnosis was
unknown at time of
procedure, local guidance
should be agreed with
commissioners regarding
where the DTT should be
taken as national
guidance does not
specify.
DTT - Both our review and
RRT review indicate the
OE DTT to be incorrect.
However we disagree on
the date (does not affect
breach status of patient).
RRT date 16.4.12 RMH
audit 13.4.12
"Partial
excision/debulking of a
tumour (but not a biopsy
for diagnostic
or staging purposes unless
it effectively removed the
tumour even if margins
are not clear".
Treatment as per GFOCW
v8.0 section 3.9.24
52
15th December 2014
3
4
5
Agree
treatment
type
Disagree
DTT
clock stop
Agree
DTT
treatment
type
Disagree
clock stop
N
Review team DTT 16/3/12
clock stop 18/3/12
RMH DTT 13/3/12 clock
stop 16/3/12.
Treatment as per GFOCW
v8.0 section 3.9.24
N
Treatment as per GFOCW
v8.0 section 3.9.24
Disagree
Entire
treatment
N
Additional information
required to clarify the
clock stop. Based on the
information available we
disagree with the RRT. RT
start date not recorded.
Treatment start date ie
clock stop ascertained
from RT end date and
working back by number
of fractions
Clock stop for Active
Surveillance should not
have been reported as
patient was already on
Active Surveillance.
Patient had been on AS
since 2011 no need to
record in 2013
Treatment as per GFOCW
v8.0 section 3.9.24
53
15th December 2014