Prioritisation of National Clinical Guidelines

National Clinical Effectiveness Committee
Preliminary Prioritisation Process*
National Clinical Guidelines
25th March 2015
*The NCEC has agreed this prioritisation process for National Clinical Guidelines. In order to balance the
prioritisation criteria scoring the NCEC is conducting an exercise to weight each of the prioritisation
criteria. The document title will be amended and ‘preliminary’ removed from the title once this exercise
is completed and the criteria weighting included in the document.
0|Page
Table of Contents
Table of Contents .......................................................................................................................................... 1
Glossary of Terms .......................................................................................................................................... 2
Purpose of this prioritisation document ....................................................................................................... 3
1.
National Clinical Effectiveness Committee........................................................................................ 4
NCEC Terms of Reference .................................................................................................................. 4
2.
Clinical Practice Guidance ................................................................................................................. 4
3.
Introduction to National Clinical Guidelines ..................................................................................... 5
4.
Prioritisation ...................................................................................................................................... 6
Stream 1 ............................................................................................................................................. 7
Stream 2 ............................................................................................................................................. 7
Stream 3 ............................................................................................................................................. 7
5.
Prioritisation methodology ............................................................................................................... 9
Prioritisation criteria ........................................................................................................................... 9
Process for National Clinical Guideline prioritisation ....................................................................... 11
Scoring system .................................................................................................................................. 12
1|Page
Glossary of Terms
National Clinical
Guideline
National Clinical guidelines are systematically developed statements,
based on a thorough evaluation of the evidence, to assist
practitioner and patient decisions about appropriate healthcare for
specific clinical circumstances, across the entire clinical spectrum.
Clinical Practice
Guidance
Clinical practice guidance is defined as systematically developed
statements or processes to assist clinician and patient decisions
about appropriate health care for specific clinical circumstances with
the choice of clinical practice guidance model determined by
evidence-based criteria and clinical requirements. Such guidance
may include but this is not an exhaustive list – local guidelines,
protocols, policies, procedures, checklists, standard operating
procedures, care pathways etc.
Clinician
A clinician is a health professional such as a physician or nurse
involved in clinical practice.
Guideline
commissioning
If a guideline is prioritised for commissioning the NCEC will identify a
guideline Chair in collaboration with the relevant national
programmes/clinical programmes. A timeframe and project plan will
be agreed. Guidance and support for elements of the commissioned
guideline will be provided by the Clinical Effectiveness Unit in the
Department of Health and HIQA e.g. systematic evidence review,
budget impact assessment.
GDG
Guideline Development Group
NCEC
National Clinical Effectiveness Committee
Notice of Intent
A notice of intent is notice provided by a GDG of a guideline in
development likely to be submitted to NCEC. The notice of intent
will be published on the NCEC website. It allows information to be
shared with stakeholders who have an interest in a clinical area, in
order for them to communicate with each other and to prevent the
possible duplication of effort where a number of stakeholders are
undertaking work in the same area. This notification neither
commits the GDG to submit their guideline to NCEC, nor does it
indicate that the guideline will be prioritised or endorsed by NCEC.
NCEC Schedule of
Guidelines
The NCEC schedule of guidelines will be a list of those guidelines
prioritised by the NCEC which are in development and are
progressing towards achieving National Clinical Guideline status.
This schedule will be published on the NCEC website.
2|Page
Purpose of this prioritisation document
What are guidelines?
Guidelines are statements which outline the most appropriate approaches for clinical practice. They are based on best
available research and are developed using an internationally recognised process. They are then used in conjunction
with clinical judgement and patient preference when making decisions about what is appropriate for each patient or
population of patients.
Do guidelines improve patient care?
Clinical guidelines are internationally recognised methods for defining healthcare interventions, improving the
effectiveness of care, and treatment, and reducing variation in care delivery. The implementation of clinical guidelines
can improve health outcomes for patients, increase standardisation and thereby reduce variation in practice and
improve the quality of clinical decisions. National Clinical Guidelines endorsed by the Minister for Health and published
by the Department of Health will inform patients about the care they should be receiving and empower them to make
more informed healthcare choices.
How do guidelines become National Clinical Guidelines?
In order to become National Clinical Guidelines Guideline Development Groups comprised of clinical and research
experts and other key stakeholders including patients develop the guideline using a robust methodology. This can take
up to two years. The National Clinical Effectiveness Committee (NCEC) prioritises and quality assures these guidelines to
assess if their methodology is of a high enough standard for them to be endorsed by the Minister for Health as National
Clinical Guidelines. See Section No. 1 for further detail on NCEC.
Why do we need National Clinical Guidelines?
National Clinical Guidelines, quality assured and recommended by NCEC for implementation in the Irish health system,
provide robust evidence-based approaches to underpin or define models of care as appropriate. There can be
confidence in how they were developed and the implementation plan that goes with them. Not all guidelines need to be
or should be National Clinical Guidelines (see Section No. 2 for further detail). We plan to publish in time a small suite of
25-30 National Clinical Guidelines.
How can we be assured that National Clinical Guidelines are implemented?
National Clinical Guidelines endorsed by the Minister for Health are mandated for implementation in the Irish health
system and their implementation will be monitored through audit, the HSE Performance Assurance Reports, compliance
with HIQA’s National Standards for Safer Better Healthcare (2012) and increased alignment with the clinical indemnity
scheme. Key performance indicators to measure implementation and impact are being developed.
Why do we need a prioritisation process for National Clinical Guidelines?
We need a process that supports and guides the health system and NCEC to plan for the development of the most
important National Clinical Guidelines for the Irish health system. The prioritisation process is a tool for guidance to help
the NCEC in terms of its deliberations in relation to identifying the potentially highest impact National Clinical
Guidelines to reduce variation in practice and improve safety and quality. The process provides a transparent process
for NCEC to consider potential guidelines. It supports NCEC and the health system to structure its guideline
development work for planned time periods. Prioritisation exercises by their nature must also incorporate the more
nuanced and subjective discussions that arise from applying a prioritisation process and these considerations will also
contribute to the final decision around which guidelines/topics are to progress to a National Clinical Guideline. This is an
evolving process and will be reviewed by the NCEC annually and updated as necessary.
Do we have any National Clinical Guidelines yet?
Yes, we have six National Clinical Guidelines published as follows: National Early Warning Score (Feb 2013), Prevention
and Control of MRSA (Dec 2013), Clostridium difficile Infection in Ireland (June 2014), Sepsis Management (Nov 2014),
Maternity Early Warning System (Nov 2014) and Clinical Handover in Maternity Services Guideline (Nov 2014). Further
details: http://health.gov.ie/patient-safety/ncec/national-clinical-guidelines-2/
3|Page
1. National Clinical Effectiveness Committee
The National Clinical Effectiveness Committee (NCEC) is a Ministerial committee established as part of the
Patient Safety First Initiative. The NCEC role is to prioritise and quality assure National Clinical Guidelines
and National Clinical Audit so as to recommend them to the Minister for Health to become part of a suite
of National Clinical Guidelines and National Clinical Audit.
NCEC Terms of Reference:
-
-
Apply criteria for the prioritisation of clinical guidelines and audit for the Irish health system
Apply criteria for quality assurance of clinical guidelines and audit for the Irish health system
Disseminate a template on how a clinical guideline and audit should be structured, how
audit will be linked to the clinical guideline and how and with what methodology it should
be pursued
Recommend clinical guidelines and national audit, which have been quality assured
against these criteria, for Ministerial endorsement within the Irish health system
Facilitate with other agencies the dissemination of endorsed clinical guidelines and audit
outcomes to front-line staff and to the public in an appropriate format
Report periodically on the implementation of endorsed clinical guidelines.
NCEC has published guidance for guideline developers:
-
Framework for Endorsement of National Clinical Guidelines
Guideline Developers Manual
Prioritisation of National Clinical Guidelines.
Information on the NCEC, NCEC documentation and endorsed National Clinical Guidelines is available at:
www.health.gov.ie/patient-safety/ncec
2. Clinical Practice Guidance
In clinical practice, there are different types of guidance that vary in complexity and scope. For example,
guidance can be a comprehensive overarching National Clinical Guideline or a more specific clinical
protocol; not all guidance requires the same pathway of development as a National Clinical Guideline.
However, regardless of the variation in scope and focus, it is important that the development of all clinical
guidance is underpinned by an appropriate evidence-based approach and quality assurance measures to
assist clinician and patient decisions about appropriate healthcare for specific clinical circumstances. Any
other clinical guidance in the Irish health system which is not a National Clinical Guideline can be
considered under the term ‘clinical practice guidance’.
Clinical practice guidance is defined as systematically developed statements or processes to assist
clinician1 and patient decisions about appropriate health care for specific clinical circumstances with the
choice of clinical practice guidance model determined by evidence-based criteria and clinical
requirements. Such models may include but this is not an exhaustive list – local guidelines, protocols,
policies, procedures, checklists, standard operating procedures, care pathways etc. The NCEC is in the
process of developing Standards for Clinical Practice Guidance. These will be published later this year.
1
A clinician is a health professional such as a physician or nurse involved in clinical practice.
4|Page
3. Introduction to National Clinical Guidelines
National Clinical guidelines are systematically developed statements, based on a thorough evaluation of
the evidence, to assist practitioner and patient decisions about appropriate health care for specific clinical
circumstances, across the entire clinical spectrum.
National Clinical Guidelines quality assured and recommended by NCEC for implementation in the Irish
health system provide robust evidence-based approaches to underpin or define clinical models, pathways
and programmes of care as appropriate. The implementation of clinical guidelines can improve health
outcomes for patients, reduce variation in practice and improve the quality of clinical decisions. National
Clinical Guidelines endorsed by the Minister for Health are mandated for implementation in the Irish
health system and their implementation will be monitored through the HSE Performance Assurance
Reports, compliance with HIQAs National Standards for Safer Better Healthcare and increased alignment
with the clinical indemnity scheme.
A rigorous methodological process of development is completed for each guideline:
-
A multi-disciplinary Guideline Development Group is established
Each guideline is developed in line with NCEC methodology2
A transparent systematic clinical literature review using recommended resources and/or
healthcare librarian support is completed to underpin guideline recommendations
Each recommendation is graded reflecting the strength of evidence associated with the
recommendation
Responsibility for implementation of each recommendation is outlined
Facilitators and barriers to guideline implementation are considered for each guideline
A budget impact assessment including a systematic economic literature review is completed for
each guideline
Each guideline identifies an implementation plan, audit criteria and key performance indicators to
measure guideline implementation and impact.
The NCEC manages the endorsement process for National Clinical Guidelines in line with the following four
steps – Figure 1.
Step 1
•Screening by
DoH Clinical
Effectiveness
Unit to confirm
application
meets National
Clinical
Guideline
definition
2
2
Step 2
•Prioritisation by
NCEC of
guidelines
•See Figure 1 for
prioritisation
process
Step 3
•Quality
assurance by
NCEC of
guideline as per
HIQA
guidance2
Step 4
•Endorsement
of National
Clinical
Guidelines by
Minister for
Health
HIQA (2011) National Quality Assurance Criteria for Clinical Guidelines. This document is
currently being updated and will be published as a 2015 HIQA/NCEC joint publication.
Further information: NCEC (2013) Guideline Developers Manual
5|Page
Guideline developers are encouraged to submit a notice of intent of guideline development planned for
submission to NCEC. This allows information to be shared with stakeholders who have an interest in a
clinical area, in order for them to communicate with each other and to prevent the possible duplication of
effort where a number of stakeholders are undertaking work in the same area. This notification neither
commits the Guideline Development Group to submit their guideline to NCEC, nor does it indicate that the
guideline will be prioritised or endorsed by NCEC. Notifications are published on the NCEC website:
http://health.gov.ie/patient-safety/ncec/national-clinical-guidelines-2/
The scope of this document applies to National Clinical Guidelines only and not National Clinical Audit.
Clinical guidelines submitted to the NCEC are screened to ensure that the guideline:


Has been recently developed or reviewed, and
Meets the NCEC definition of a clinical guideline.
4. Prioritisation
The NCEC in 2014 gave consideration to prioritisation processes that could assist the identification of key
areas for potential National Clinical Guidelines in the Irish healthcare system that best addresses the
healthcare needs of the Irish population. This consideration includes a weighting process for the
prioritisation criteria and the management of commissioned guidelines. The development of the NCEC
prioritisation process was informed by international literature, Irish requirements, NCEC members
considerations and a public consultation.
The prioritisation process provides NCEC with a transparent prioritisation tool. Prioritisation exercises by
their nature must also incorporate the more nuanced and subjective discussions that arise from applying a
prioritisation process and these considerations will also contribute to the final decision around which
guidelines/topics are to progress to a National Clinical Guideline. The process will be reviewed by NCEC on
an annual basis to examine its effectiveness and will be updated as necessary.
There are several prospective sources from where guidelines or guideline topics can originate, for
example, to address a significant patient safety or policy issue, from national and international reports
horizon scanning and from the HSE clinical programmes, national programmes or the wider healthcare
sector. Figure 2 summarises the National Clinical Guideline Prioritisation Process.
All guidelines regardless of prioritisation stream will, once prioritised, be listed on the NCEC schedule of
guidelines in development. The NCEC schedule of guidelines will be a list of those prioritised guidelines
which are in development and are progressing towards achieving National Clinical Guideline status.
All prioritised guidelines on completion will proceed to quality assurance in line with HIQA (2011) National
Quality Assurance Criteria for Clinical Guidelines.
6|Page
The prioritisation process involves three streams:
-
Stream 1 – Significant patient safety or health policy issue; horizon scanning (a commissioned
guideline process)
Stream 2 – Clinical and National programmes (developed through Clinical and National
Programmes and may or may not proceed through a commissioned guideline process)
Stream 3 – Wider health system submissions (developed through a Guideline Development
Group).
Stream 1
Stream 1 of the National Clinical Guideline Prioritisation Process is for guideline topics that are derived in
particular from a significant patient safety or policy issue. However, topics may arise from national and
international reports, horizon scanning or from the Clinical or National Programmes, should a significant
patient safety/policy issue be highlighted from this sector.
Stream 1 guideline topics proceed directly to the commissioned guideline process. This means that the
development of the National Clinical Guideline will receive executive support from the Clinical
Effectiveness Unit in the Department of Health to compile the guideline. This support may include:
- Guidance on aspects of guideline processes
- Support for establishment of Guideline Development Group and chair
- Systematic clinical literature review
- Project management
- Economic evaluation
- Write up and editorial assistance.
To ensure adequate capacity for both compilation and implementation of the National Clinical Guideline,
NCEC considers that no more than four National Clinical Guidelines per year be commissioned.
Stream 2
Stream 2 of the National Clinical Guideline Prioritisation Process is for guideline/topic submissions from
the HSE Clinical and National Programmes. These guidelines may progress in two ways:
a) Submission of a proposal/scoping of guideline topic to NCEC through the NCEC prioritisation
process outlining how that topic area fulfills the NCEC prioritisation criteria. This guideline topic
proposal may go on to be considered as a commissioned guideline (Stream 1) or the Programme
can progress the guideline.
b) Development of full National Clinical Guideline and submission directly to NCEC to undergo the
prioritisation and quality assurance process.
Where the Programme develops the guideline executive support in the form of guidance will be provided
from the Clinical Effectiveness Unit in the Department of Health to compile the guideline. A liaison person
from the Clinical Effectiveness Unit will be assigned to the Guideline Development Group.
Stream 3
The wider health system may also submit guideline proposal topics or guidelines for consideration to NCEC
as National Clinical Guidelines through the NCEC prioritisation process. These proposals from the wider
health system will be reviewed annually. Those submitting from the wider health setting are advised to
submit a topic proposal/scoping of guideline in preference to a full version guideline. Guideline developers
should give consideration as to whether the guideline is more appropriately developed through Stream 2
7|Page
i.e. consideration should be given to potential links to Clinical and National Programmes. In addition, the
topic proposals will need to be cognisant of the NCEC prioritisation criteria and the content of the proposal
should address these areas sufficiently. In addition this guideline topic proposal may go on to be
considered as a commissioned guideline (Stream 1) or the Guideline Development Group can progress the
guideline themselves. A liaison person from the Clinical Effectiveness Unit will be assigned to the Guideline
Development Group.
Where the wider health system develops the guideline executive support in the form of guidance will be
provided from the Clinical Effectiveness Unit in the Department of Health to compile the guideline. In
exceptional circumstances the NCEC may consider referring guidelines from this stream for consideration
in terms of the commissioning process if the clinical topic rates highly enough in the prioritisation process.
Figure 2: National Clinical Guideline Prioritisation Process
Source of proposals for National Clinical Guidelines
1. Significant
Patient Safety or
Policy Issue
From:
Health system,
National and
International reports.
Horizon scanning.
2. Clinical/National
Programmes
Guideline
Proposal
Clinical/National
Programmes*
Full
Guideline
Clinical/National
Programmes*
3. Wider Health
System
(Submissions
accepted annually)
Guideline
Proposal or Full
Guideline *
National Clinical Guideline
Prioritisation
National Clinical Guideline
Prioritisation
A guideline proposal may be
considered for the
Commissioning process and
referred to Stream 1.
Consideration should be
given to link to Clinical and
National Programmes.
Guideline listed on NCEC Schedule of
National Clinical Guidelines
National Clinical Guideline Quality Assurance
8|Page
Guideline topic proposals/full guideline must demonstrate how the clinical area
fulfils the NCEC prioritisation criteria
5. Prioritisation methodology
Prioritisation criteria
The NCEC has identified 7 criteria for prioritisation (Table 1). Descriptors for each of the criterion are
described.
Table 1 Criteria for NCEC Clinical Guideline Prioritisation
NCEC Criteria
1.
Patient Safety Issue
2.
Burden of Clinical Topic*
3.
Evidence Analysis
4.
Economic Impact
5.
Variability in Practice
6.
Potential for Addressing Health Issues
7.
Clinical Guideline Implementation
Total
*Includes disease/condition/circumstance etc.
Likert Scale
Major (5)
High (5)
Strong (5)
High (5)
Major (5)
High (5)
Strong (5)
35
Minor (1)
Low (1)
Weak (1)
Low (1)
Minor (1)
Low (1)
Weak (1)
7
Criteria 1 Patient Safety Issue
- What is the patient safety issue?
- Who is affected?
- How are they affected?
- Does the issue have national implications?
- What are the risks associated with this issue if not addressed?
- How can it be addressed?
- Is there potential for quality improvement in the area?
Criteria 2 Burden of Clinical Topic
- What is the incidence/prevalence of clinical topic (disease/condition/circumstance)? The
population (patients, public, etc.) to whom the guideline is meant to apply is specifically
described?
- What is the associated mortality and morbidity?
- What are the rates of relapse, re-admission and complications?
- Is there reduced quality of life?
- Is there patient dissatisfaction?
Criteria 3 Evidence Analysis
- Are clinical guideline recommendations based on an analysis of the evidence? This should
preferably be a systematic review of high-quality randomised controlled clinical trials or well
designed controlled studies that measure relevant outcomes demonstrating strong, clinically
important beneficial public health effects
- Is there detail of the search methods and evidence rating?
- Are recommendations graded based on the quality of evidence with an explicit link between the
recommendations and supporting evidence?
- Has the clinical guideline been externally reviewed prior to its submission to the NCEC? Ideally the
external review should provide commentary on the search strategy for the evidence review?
9|Page
Criteria 4 Economic Impact
While there is often limited Irish data available on the economic impact of healthcare interventions,
guideline developers should consider international evidence and make an effort to include some
estimation or approximation of the cost-effectiveness, and any possible budget increases or savings, if the
guideline is implemented.
- Would implementing this guideline have a substantial budget impact on the healthcare system?
o Have the resource implications of implementing the guideline been considered?
o Have the resources required for any initial set up or roll out phase been considered?
o Have the cost of these resources to the publicly-funded system been estimated?
- Are there potential cost savings to be realised if the guideline is implemented?
o Are there any potential cost savings due to changes in the use of resources?
o Have the benefits from improved outcomes been quantified and the associated costs or
savings been estimated?
- Is there national or international cost-effectiveness evidence to support implementing the
guideline?
o Is a summary of the cost-effectiveness evidence presented? Is this generalisable or
relevant to the Irish healthcare setting?
o Has this evidence been gathered using systematic searching methods and are these
methods documented?
Criteria 5 Variability in Practice
- Are there gaps between current clinical practice and evidence-based practice?
- Are significant variations in practice evident?
- What is the associated risk of the variance from best practice?
- Would reducing variation incur beneficial effects for patients?
- Would reducing variation reduce avoidable morbidity and/or mortality?
- To what extent is there a high risk impact for the health system?
- Are there high frequency risk factors (avoidable and inherent)?
Criteria 6 Potential for Improved Health
- The overall objective of the guideline is specifically described with the expected benefit or
outcome of the guideline clearly outlined
- Is there potential for improved health outcomes?
- What is the extent of potential improved quality of life?
- What is the extent of potential improved quality of care?
- Is there potential for health promotion at population health level?
- Is there potential for disease prevention at population health level?
- Will the clinical guideline reduce the extent of avoidable injury?
- Will the clinical guideline reduce inequalities in health?
- What are the potential short and long-term health outcomes taking into account the strength of
evidence associated with each?
- Is there a maximum likelihood of benefit and minimum harm?
- Will the clinical guideline reduce symptoms, avoid or delay need for other therapies or reduce
disease progression?
- Will the clinical guideline support the implementation of national health policy?
- Will the clinical guideline improve patient safety?
Criteria 7 Clinical Guideline Implementation
- What is the feasibility of implementation of the clinical guideline?
- What are the facilitators to the guideline application?
- Are there any significant barriers to implementation of the clinical guideline?
- What is the resource impact for implementation of the clinical guideline?
10 | P a g e
-
-
How acceptable will the clinical guideline be to relevant stakeholders (consumers and clinicians)?
Did the clinical Guideline Development Group include individuals from all the relevant professional
groups, methodological experts and intended users for example healthcare professionals, hospital
managers etc.?
Is there a degree of urgency for implementation of the clinical guideline?
What is likelihood of the clinical guideline implementation strategy being successful?
How accessible will the clinical guideline be?
Process for National Clinical Guideline prioritisation3







On receipt of a clinical guideline or guideline proposal from GDGs the Clinical Effectiveness Unit
identifies a chair and reviewers for the prioritisation team. The clinical guideline or guideline
proposal is made available to the team for assessment against the NCEC prioritisation criteria.
There will be a minimum of 5 reviewers. This may include members of the NCEC, NCEC working
group and external reviewers. Reviewers are not subject experts. The reviewer’s role is to evaluate
the clinical guideline in line with the prioritisation criteria. Reviewers will have collective expertise
in clinical practice guidelines, evidence-based healthcare, patient safety, audit and healthcare
policy. All reviewers will complete a conflict of interest declaration. At a minimum 3 of the
reviewers will have conducted at least 2 previous prioritisation exercises.
All reviewers will complete a conflict of interest declaration.
The prioritisation team will produce a report for consideration by the NCEC. This report will be
completed by the prioritisation team following a prioritisation meeting. Reviewers will complete a
review of the guideline in advance of the prioritisation meeting to minimise ‘group think’ however
reviewers may adjust their prioritisation scores in light of discussion at the meeting. Where there
is major divergence of opinion on scores which cannot be resolved this will be referred to the
NCEC chair for final decision.
The NCEC may seek clarifications and/or additional information from the GDG.
The NCEC reviews the report of the prioritisation exercise and makes decision with regard to
identification of prioritised clinical guidelines to proceed to appraisal. A copy of the clinical
guideline(s) will be available to committee members for information purposes at the NCEC
meeting.
The result of prioritisation exercise may be as follows:
Guideline Proposal
a. The clinical guideline proposal does not score highly enough against the prioritisation criteria to
progress to appraisal in line with the quality assurance criteria. The NCEC will advise which criteria
are not adequately addressed and offer a meeting with the GDG.
b. The clinical guideline proposal is successful and will be listed on the NCEC schedule of guidelines.
Full clinical guideline
a. The clinical guideline does not score highly enough against the prioritisation criteria to progress to
appraisal. The NCEC will advise which criteria are not adequately addressed and offer a meeting
with the GDG.
b. The clinical guideline is successful and will proceed to appraisal appraisal in line with the quality
assurance criteria.
3
Further information including detail on appeals is available in the NCEC Framework for Endorsement of National Clinical
Guidelines available at: http://health.gov.ie/patient-safety/ncec/resources-and-learning/ncec-processes-and-templates/
11 | P a g e
Scoring system
Reviewers’ scores will be collated as follows. The seven prioritisation criteria are expressed as statements
and rated on Likert scales. For example:
Criteria 2 Burden of Clinical Topic is rated high (5) to low (1)
High
5
4
3
2
1
Low
The Likert scale is a categorical, rather than a numerical scale. Therefore in order to create an overage
prioritisation score as a percentage for each clinical guideline, criteria scores are calculated by summing up
the scores of the individual criteria, and by scaling the total as a percentage of the maximum possible
score. The process utilised is similar to the scoring for the AGREE II tool. A worked example is provided in
Table 3.
Table 3 Clinical Guideline A: Ratings for 6 criteria by 4 reviewers
Criteria 1 Criteria Criteria Criteria
2
3
4
Reviewer 1
3
4
4
5
Reviewer 2
2
4
5
3
Reviewer 3
3
5
3
2
Reviewer 4
4
4
4
4
Reviewer 5
2
2
4
4
Total Likert Obtained 14
19
20
18
Score
Criterion weight*
Multiply
by
0.X
weight
Total Weighted Score
Criteria
5
5
3
4
4
3
19
Criteria
6
4
3
1
2
3
13
Criteria
7
3
3
2
4
3
15
Total
28
23
20
26
21
118
Total obtained score for 7 weighted criteria (X 5 reviewers) –
Minimum possible score for 7 weighted criteria (6 X 5 reviewers)
______________________________________________ X 100
Maximum possible score for 7 weighted criteria (30 X 5 reviewers) –
Minimum possible score for 7 weighted criteria (6 X 5 reviewers)
*The NCEC is conducting an exercise to weight each of the criteria. The weighting will be added to the
table above once finalised.
12 | P a g e