Abstracts of the 23rd Cochrane Colloquium

Transcription

Supplement 2015
Oral sessions
Session 01: Diagnostic test accuracy reviews
Session 02: Information retrieval
Session 03: Primary studies
Session 04: Qualitative evidence
Session 05: GRADE
Session 06: Disseminating evidence
Session 07: Communicating evidence
Session 08: Setting priorities to overcome inequality
Session 09: Systematic review and meta-analysis methods
Session 10: Filtering the information overload for better decisions
Session 11: Rapid reviews and overviews of reviews
Session 12: Prediction models
1
3
6
44
46
48
76
78
81
117
120
122
Rapid oral sessions
Session 01: Risk of bias assessment tools
8
Session 02: Editorial issues
11
Session 03: Advanced methods: Overviews of reviews and network
meta-analyses
16
Session 04: Quality of reporting
19
Session 05: Online databases for evidence dissemination
23
Session 06: Individual patient data
26
Session 07: Quality of reporting
50
Session 08: Evidence maps and priority setting
55
Session 09: Meta-analysis methods
59
Session 10: Filtering the information overload for better decisions 62
Session 11: Knowledge translation
67
Session 12: Specific study types / Avoiding research waste
70
Session 13: Translations
83
Session 14: Education and training
88
Session 15: Implementation and qualitative synthesis
91
Session 16: Rapid reviews
95
Session 17: Network meta-analysis
98
Session 18: Outcomes
101
Session 19: Global health: Helping with evidence
125
Session 20: Disseminating evidence
129
Session 21: Systematic review and meta-analysis methods
133
Session 22: Guidelines and GRADE
139
143
Session 24: Novel approaches of interpreting and implementing
healthcare evidence
147
Lunch rapid oral sessions
Session 01: Minimally important difference
Session 02: Priorisation
Session 04: Decision-making: End users and consumers
Session 05: Primary studies
Session 06: Health Economics
Session 07: Involvement of users and stakeholders
Session 08: Search filters
Session 09: Trial registration
Session 10: Missing data
Session 11: Diagnostic test accuracy reviews
Session 12: Decision-making: Policy and public health
30
35
37
41
106
109
112
114
152
155
159
161
Poster abstracts
Session 1
Session 2
165
224
Workshops
Workshops
291
Abstracts available online at www.cochrane.org
Oral session 1
Diagnostic test accuracy reviews
O 1.1
Authors need guidance on creating clinical
pathways in Cochrane diagnostic test
accuracy systematic reviews
Gopalakrishna G1 , Langendam M1 , Scholten R2 , Bossuyt P1 ,
Leeflang M1
1 University of Amsterdam, The Netherlands; 2 Dutch Cochrane
Centre, The Netherlands
Background: Developing the clinical pathway of a medical
test can help test accuracy review authors define more
precisely the target condition and study eligibility criteria
for their reviews. Such pathways also help in understanding
the impact of testing on clinical management decisions and
ultimately the impact on patient outcomes. Describing
the clinical pathway is a recent mandatory requirement in
Cochrane diagnostic test accuracy (DTA) reviews but there
is no explicit guidance on how to create such a pathway.
Objectives: To investigate how authors of Cochrane DTA
Review protocols described the clinical pathway in their
reviews. Methods: We searched the Cochrane Database
of Systematic Reviews for all DTA review protocols published
between 2014 and March 2015. We checked if the pathways
fulfilled a set of minimum criteria adapted from the analytical
pathway as published in the Agency for Healthcare Research
and Quality’s medical test manual. Results: We found a
total of 31 protocols. All addressed the clinical pathway
differently, both in content and format. Most (n = 26) provided
only a textual description. Four included both a visual and
textual description, and one only a visual description. A
detailed analysis of the 10 most recent protocols showed
tremendous variation in these pathways (Table 1). The
majority did not put the tests in the context of previous
or further testing, clinical management following the test(s)
or the impact on patient outcomes. In case of single test
evaluations, downstream consequences of introducing the
test were often not compared to the consequences of not
testing. Conclusions: In Cochrane DTA Review protocols there
is remarkable variation in interpretation, content and format
of the clinical pathway. Review authors could benefit from
guidance on how to create pathways to guide them in their
evidence selection and ultimately put that evidence in context
of downstream patient and/or management consequences.
Copyright c2015 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
O 1.2
What is the test’s accuracy in my practice
population Tailored meta-analysis provides
a plausible estimate
Willis BH1 , Hyde CJ2
1 University of Birmingham,
United Kingdom; 2 University of Exeter,
United Kingdom
Background: Diagnostic test accuracy studies and metaanalyses may, in some cases, provide estimates that are highly
improbable in practice. Tailored meta-analysis provides a
potential solution. Objective: To investigate the utility
of tailored meta-analysis in synthesizing estimates of a
test’s accuracy compared with conventional meta-analysis
for three case examples. Method: MEDLINE, EMBASE, and
CINAHL for relevant studies; routine data collected on the
test positive rate and disease prevalence from the case
settings to define an applicable region for each setting.
Three cases were evaluated: (1) Mammography in the
NHS Breast Screening Program; (2) PHQ-9 questionnaire to
screen for depression in general practice; (3) Centor’s criteria
used to diagnose group A beta-hemolytic streptococcus in
general practice. For conventional meta-analysis, studies
were selected using standard systematic review methods; for
tailored meta-analysis this selection was refined to those with
results compatible with the applicable region for the setting.
Baseline study characteristics, quality items and 2 × 2 tables
completed. Results: In each example, studies were excluded
as a result of incorporating an applicable region for the
setting. Comparing tailored with conventional meta-analysis,
1
Cochrane Database Syst Rev Suppl 1–327 (2015)
2
the positive likelihood ratios (with 95% confidence intervals
in brackets) were 36.5 (23.0 to 57.9) and 19.8 (12.8 to 30.9)
respectively for mammography; and 4.89 (2.02 to 11.8) and
2.35 (1.51 to 3.67) respectively for Centor’s criteria. This
had the effect of increasing the positive predictive value
from 17% to 27% for mammography and 23% to 38% for
Centor’s criteria. Conclusion: Tailored meta-analysis has the
potential to provide a plausible estimate for a test’s accuracy
which is specific to the practice setting. When compared
with conventional meta-analysis, the difference may, in some
cases, be sufficient to lead to different decisions on patient
management.
O 1.3
Including unpublished data in a systematic
review of diagnostic test accuracy: impact on
summary estimates
Korevaar D1 , Westerhof G2 , Wang J1 , Cohen J1 , Spijker R3 , Sterk P2 ,
Bel E2 , Bossuyt P1
1
Department of Clinical Epidemiology, Biostatistics and
Bioinformatics, Academic Medical Center, University of Amsterdam,
Netherlands; 2 Department of Respiratory Medicine, Academic
Medical Centre, University of Amsterdam, Netherlands; 3 Medical
Library, Academic Medical Center, University of Amsterdam,
Netherlands
Background: Studies with favorable results have higher
chances of being published than those with unfavorable
results. This could introduce bias in systematic reviews.
Objectives: We aimed to compare summary estimates
calculated from published and unpublished data in a
systematic review of diagnostic test accuracy. Methods:
We gathered published data by searching MEDLINE and
Embase for studies evaluating the diagnostic accuracy of
Fraction of exhaled Nitric Oxide (FeNO), blood eosinophils or
Immunoglobulin E (IgE) for detection of airway eosinophilia in
adults with asthma (inception to August 2014). We obtained
unpublished data by contacting authors of studies that did
not report on the diagnostic accuracy of these markers, but
seemed to have collected data from which estimates could
be calculated, and performed random-effects meta-analysis
of area under the receiver operating curve (AUC). Results:
Analysing 2919 search results, we included 12 published
studies reporting on the accuracy of FeNO, blood eosinophils
or IgE (published data). We contacted the authors of two of
3
O 1.4
providing similar point estimates and confidence intervals of
sensitivity and specificity, while prediction intervals were
narrower. In Lymphangiography review data, both BM
with nominal random effects and BM with latent classes
showed a better fit compared to standard BM, providing
us with a slightly higher estimate of specificity and, again,
with narrower prediction intervals. Conclusions: Although
simulation studies are needed, we can conclude that BM
with non-parametric random effects could represent a valid
alternative to the standard BM to obtain more accurate
prediction intervals of sensitivity and specificity.
References: Diaz, Mireya.
Performance measures of
the bivariate random effects model for meta-analyses of
diagnostic accuracy. Computational Statistics & Data Analysis
2015: 83: 82–90.
Glas, Afina S., et al. Tumor markers in the diagnosis of primary
bladder cancer. A systematic review. The Journal of urology
2003: 169.6: 1975–1982.
Scheidler, Juergen, et al.
Radiological evaluation of
lymph node metastases in patients with cervical cancer: a
meta-analysis. Jama 1997: 278.13: 1096–1101.
A generalized latent variable modelling
framework for the bivariate analysis of
sensitivity and specificity in diagnostic
reviews
Oral session 2
Information retrieval
these studies, and obtained additional diagnostic accuracy
data of markers that had also been assessed, but for which no
accuracy results were reported (unpublished data). Authors of
another 14 articles were contacted because they could have
data from which accuracy estimates could be calculated.
Seven shared the requested data, and we also included one
conference poster (unpublished data). Adding unpublished
data increased sample size for FeNO and blood eosinophils
by 28%, and for IgE by 103%. Summary AUC for FeNO was
0.75 (95%CI 0.72 to 0.78; 9 studies; 2263 patients) based on
published data, vs 0.73 (0.68 to 0.78; 7 studies; 627 patients)
for unpublished data. For blood eosinophils, these numbers
were 0.77 (0.71 to 0.83; 7 studies; 1592 patients) vs 0.77 (0.71
to 0.83; 5 studies; 446 patients), and for IgE 0.64 (0.57 to
0.70; 2 studies; 300 patients) vs 0.61 (0.54 to 0.67; 4 studies;
310 patients). Conclusions: Many authors seem to be willing
to share unpublished data. Adding unpublished data led
to a considerable increase in precision, but barely affected
summary estimates of accuracy.
Eusebi P1 , Orso M1 , Reitsma JB2 , Vermunt JK3
1 Regional Health Authority of Umbria, Italy; 2 University of Utrecht,
The Netherlands; 3 University of Tilburg, The Netherlands
Background: The current methodology for diagnostic
reviews relies on hierarchical models with normal random
effects for the true logit sensitivity and specificity. The
assumption of normally distributed random effects can
lead to overestimation of between-study variances in the
case of outliers, tails and mixtures, and this can results
in extremely wide prediction intervals. Furthermore, a
considerable number of studies is required to estimate the
random effects reliably (Diaz 2015). This results in optimistic
confidence intervals and large prediction intervals when
substantial heterogeneity is present. Thus it is vital to
expand the current methodology to address these issues.
Objectives: i) to develop new Bivariate Models (BM) for
diagnostic reviews incorporating non-parametric random
effects; ii) to provide a comprehensive review of all the
methodological extensions of the BMs for diagnostic reviews
within the framework of generalized latent variable modelling.
Methods: Several specifications of non-parametric random
effects will be formulated for the BM. These will include:
i) ordinal random effects; ii) nominal random effects; iii)
latent classes. Data from two published meta-analyses
were analysed for illustrating the properties of the models:
’Telomerase as a diagnostic marker for bladder cancer’ (Glas
2003), and ’Lymphangiography for the diagnosis of lymph
node metastasis in women with cervical cancer’ (Scheidler
1997). Results: In Telomerase review data, BM with ordinal
random effects showed a better fit compared to standard BM,
O 2.1
Improving access to reports of RCTs in
Embase: innovative methods enhance the
Cochrane Central Register of Controlled
Trials (CENTRAL)
Glanville J1 , Dooley G2 , Noel-Storr A3 , Foxlee R4
1 York Health Economics Consortium, United Kingdom; 2 Metaxis,
United Kingdom; 3 Cochrane Dementia and Cognitive Improvement
Group, United Kingdom; 4 Cochrane Editorial Unit, United Kingdom
Background: Cochrane Systematic Reviews rely on the
efficient identification of research evidence, specifically
evidence from randomised controlled trials (RCTs). The
largest single source of reports of RCTs is the Cochrane Central
Register of Controlled Trials (CENTRAL) in the Cochrane
Library. CENTRAL is mainly populated with records from
MEDLINE, but also contains a substantial and growing number
of records from Embase. The objective was to develop a new
bespoke search filter to identify reports of RCTs and novel
methods to assess the high volume of candidate reports
resulting from the filter. Methods: We developed, validated
and refined a sensitive search filter to identify reports of RCTs
in Embase. This filter was developed using textual analysis
of ten gold standard sets of RCT records (totalling 10,000
records over ten years). The filter performance was tested
on a second set of 10,000 RCT reports. Once implemented,
records retrieved by the filter were assessed for relevance by
a novel crowdsource approach. The search filter was refined
after one year of operation based on an assessment of the
4
records rejected by the crowd. Results: The development
of the search filter and the analysis of output from Embase
has resulted in a tiered assessment process, where the
most obvious RCT reports are fast-tracked for publication
in CENTRAL, leaving more capacity to assess the relevance
of less obvious candidate records. Over a 15-month period
the filter has identified 198,960 records and 55,042 reports
of RCTs have been added to CENTRAL (precision 28%).
Conclusions: The records identified by the filter and the
crowdsource process have made many thousands of reports
of RCTs that were unique to Embase, available in CENTRAL
at a high level of precision. These RCTs might be otherwise
inaccessible to Cochrane authors since many of them may
not have access to Embase.
O 2.2
Systematic reviews and meta-analyses of
traditional Chinese medicine must search
Chinese databases to reduce language bias
selection bias. The bias may be particularly severe in
systematic reviews of traditional Chinese medicine (TCM) as
most randomized controlled trials (RCT) in TCM are published
and accessible only in Chinese. Objectives: In this study we
investigated how often Chinese databases were not searched
in systematic reviews of TCM, how many trials were missed,
and whether a bias may occur if Chinese databases were not
searched. Methods & Results: We searched five databases
in English and three in Chinese for RCTs of Chinese herbal
medicine for coronary artery disease and found that 96.64%
(115/119) of eligible studies could be identified only from
Chinese databases. In a random sample of 80 Cochrane
Reviews on TCM, we found that Chinese databases were
only searched in 43 (53.75%), in which almost all the
included studies were identified from Chinese databases.
We also compared systematic reviews of the same topic and
found that they may draw a different conclusion if Chinese
databases were not searched. Conclusions: In conclusion,
an overwhelmingly high percentage of eligible trials on TCM
could only be identified in Chinese databases. We suggest
that reviewers of TCM search Chinese databases to reduce
potential selection bias.
Wu X1 , Tang J2 , Mao C2 , Yuan J1 , Qin Y1 , Chung V1
1
The Chinese University of Hong Kong, China; 2 The Hong Kong
Branch of the Chinese Cochrane Centre, China
Background: Systematic reviews that fail to search
non-English databases may miss relevant studies and cause
5
O 2.3
Web searching for systematic reviews: how to
develop the current guidance on methods
and reporting standards
Briscoe S1
1
University of Exeter, United Kingdom
Background: The development of the world wide web has
given researchers easy access to information. Although this
is a positive development, the web has also contributed
to the problem of ’information overload’. This problem
may be alleviated through guidance on the most effective
methods and reporting standards for web searching. To this
end, the author of this abstract has been asked to develop
the Cochrane Handbook section on web searching (section
6.2.2.6) for the next major revision. Objectives: This oral
session will review the current guidance on web searching
for systematic reviews and consider how to develop this
guidance. In particular, the session will consider methods
for web searching using search engines and websites, and
the reporting standards required to ensure transparency and
reproducibility of searches. Methods: Current guidance
on web searching for systematic reviews will be reviewed,
including guidance from Cochrane and the Centre for Reviews
and Dissemination (UK). In order to develop this guidance,
and searchers’ understanding of web searching, the session
will contrast some of the differences between web searching
and searching bibliographic databases. This will include the
lack of structured indexing on the web by comparison to
bibliographic records, and the difference between web-based
probabilistic search algorithms and the Boolean searches
used by bibliographic databases. Results: It will be
demonstrated how the search features of search engines and
websites impact on optimal search methods and reporting
standards. Recommendations for developing guidelines in
line with these findings will be presented. Conclusions: The
guidance on web searching to resource systematic reviews
currently lacks detail and should be updated. Better guidance
on web searching would help searchers and researchers
resource systematic reviews and would contribute to ’filtering
the information overload’.
6
O 2.4
The objectively-derived approach enables the
development of high-quality search
strategies for bibliographic databases
Hausner E1 , Guddat C1 , Hermanns T1 , Lampert U1 ,
Waffenschmidt S1
1
IQWiG, Germany
Background: The development of strategies for systematic
searches in bibliographic databases is a major challenge.
The search strategies are often complex and may comprise
several pages per database searched. The pertinent literature
currently recommends a conceptual approach for the
development of high-quality search strategies, which is based
on a comprehensive collection of synonyms. However, an
objectively-derived approach using terms identified in a text
analysis may offer advantages. We conducted two projects
to evaluate the objectively-derived approach. The first was a
retrospective validation of the objectively-derived approach
by means of published Cochrane Reviews. The results were
promising, so that the conceptual and objectively-derived
approaches were prospectively compared in a second project.
Objectives: To directly and prospectively compare the
objectively-derived approach with the conceptual approach
in the development of search strategies. Methods: Search
strategies were developed in parallel in MEDLINE for five
different research questions: external experts used the
conceptual approach to develop the related search strategies;
IQWiG used the objectively-derived approach for the same
research questions. Results: The research questions of
the five projects varied greatly and comprised searches for
studies of tyrosine kinase inhibitors, renal cancer therapy,
continuous blood-pressure measurement, effectiveness and
sustainability of inpatient psychiatric rehabilitation for adults,
and impeding and promoting factors in the implementation
of clinical guidelines. The objectively-derived approach
achieved a weighted mean sensitivity of 97% (95% confidence
interval (CI) 94% to 100%) with a weighted mean precision of
5% (95% CI 1% to 10%). The conceptual approach achieved
lower values, namely, a weighted mean sensitivity of 75%
(95% CI 47% to 100%) and a weighted mean precision of 4%
(95% CI 0% to 8%). Conclusions: The objectively-derived
approach for the development of search strategies achieves
higher sensitivity than the conceptual approach. It produces
robust search strategies and in particular offers advantages
for complex research questions.
Oral session 3
Primary studies
O 3.1
Olanzapine for schizophrenia: what do the
unpublished clinical trials reveal?
Lawrence KA1 , Beaumier J2 , Wright J3 , Perry, Jr. T1 , Puil L1 ,
Turner E4 , Mintzes B5
1
UBC Therapeutics Initiative, Canada; 2 Vancouver Coastal Health
Authority, Canada; 3 University of British Columbia, Canada;
4 Oregon Health and Science University, USA; 5 University of Sydney,
Australia
Background: Selective reporting of data is a major obstacle to
advancing and practicing evidence-informed medicine. Even
with access to high-quality Cochrane Reviews, the true net
benefit versus harm of an intervention cannot be established
without inclusion of the data from unpublished clinical trials.
Objectives: To compare data on key outcomes from published
randomized controlled trials (RCTs) and unpublished clinical
trials concerning the antipsychotic olanzapine versus placebo.
Methods: Using Cochrane search methods and a trial search
co-ordinator, all relevant, published RCTs were retrieved
comparing olanzapine with placebo in the treatment of
schizophrenia. Through the European Medicines Agency,
all relevant clinical study reports submitted by Eli Lilly for
olanzapine’s market authorisation for schizophrenia were
requested and obtained. We will compare data on the
following key outcomes in published and unpublished data,
and will examine the effects of any discrepancies on results of
meta-analyses: all-cause mortality, non-fatal serious adverse
events, and quality of life. Additionally, methods descriptions
in published and unpublished trial reports, and assessment
of risk of bias, will be compared. Conclusions: Accessing and
including data from unpublished clinical trials may prove to
create a more accurate assessment of benefit versus harm of
a medical intervention. Although the volume of clinical trial
data is large and the undertaking complex, this additional
level of scrutiny may be required in order to produce the most
accurate and unbiased Cochrane Reviews.
7
O 3.2
The association between baseline differences
and mortality in trials of atypical
antipsychotics in dementia: a
meta-regression analysis
Hulshof T1
1
University Medical Center Groningen, The Netherlands
Background: In 2005, a meta-analysis of 15 randomized
placebo-controlled trials showed that atypical antipsychotics
were associated with an increased risk of mortality in patients
with dementia. Consequently, health authorities warned
against their use. However, many trials had baseline
differences that might explain the meta-analytic findings.
Objective: To explore whether the association between
baseline differences and mortality could be estimated with a
meta-regression analysis. Methods: We abstracted from the
articles: (1) randomization features; (2) trial characteristics;
and (3) five baseline characteristics that predict death, that is
higher mean age, neuropsychiatric symptom (NPS) score, and
higher percentage of male, non-white and vascular/mixed
dementia patients per treatment group. Each characteristic
was scored as high risk if higher or unclear in active treatment
group, and otherwise as low. Results: Randomization
procedures were poorly described in all trials. Two reports did
not present a baseline table. We found not high percentage
of males in the active treatment group but studies with a high
percentage of females yielded a pooled increased risk of death.
The other variables predicted an increased risk of mortality as
expected. For each of the five baseline characteristics there
were more studies that scored high risk. We then calculated
a sum-variable ‘baseline differences’ per study that indicated
the number of variables that predicted death (0-5). Both
studies that had a haloperidol group in addition to a atypical
antipsychotic group scored 0, and nine of the other studies
had a score 3 , 4 of 5. In a meta-regression analysis of
12 trials, this sum-variable predicted the risk of mortality
(beta 0.32; 95% CI -0.03 to 0.7). Conclusion: Unfavorable
baseline characteristics in the atypical antipsychotic group
might have mistakenly suggested an increased mortality rate
in randomized placebo-controlled trials.
O 3.3
Meta-epidemiology of re-analyses
of randomized controlled trial data
Ebrahim S1 , Sohani Z2 , Montoya L3 , Agarwal A2 , Ebrahim KT2 ,
Mills E2 , Ioannidis J4
1
McMaster University; Stanford University, Canada; 2 McMaster
University, Canada; 3 University Health Network, Canada; 4 Stanford
University, USA
Background: Reanalyses of randomized controlled trial
(RCT) data may help the scientific community assess the
validity of reported trial results, but it may also cause
confusion. Objectives: To identify reanalyses of RCT data;
characterize methodological and other differences between
the original trial and re-analysis; evaluate the independence of
authors performing the re-analyses; and compare re-analysis
findings to those from the original trial to assess whether the
re-analysis modified interpretations from the original paper
about the kinds or number of patients who should be treated.
Methods: We completed an electronic search of MEDLINE
from inception to 9 March 2014 to identify all published studies
that completed a re-analysis of individual patient data from
previously published RCTs addressing the same hypothesis
as the original RCT. Four data extractors independently
screened papers and extracted data. We assessed for changes
in direction and magnitude of treatment effect, statistical
significance, and interpretation about who should be treated.
Results: We identified 37 eligible reanalyses in 36 published
papers, five of which were performed by entirely independent
authors, (two based on publicly available data, two on data
that were provided on request, the availability of data was
not clear for one). Reanalyses differed most commonly in
statistical or analytical approaches (n = 18) and in definitions
or measurements of the outcome of interest (n = 12). Four
reanalyses changed the direction and two changed the
magnitude of treatment effect, while four reanalyses led
to changes in statistical significance of findings. Thirteen
reanalyses (35%) led to interpretations that differed from
those of the original paper: three (8%) could be interpreted
as showing that different patients should be treated, one
(3%) that fewer patients should be treated, and nine (24%)
that more patients should be treated. Conclusions: A small
number of reanalyses of RCTs have been published to-date
and original study conclusions were altered in 35%.
O 3.4
Are we overestimating harm when high
numbers of placebo patients are switched to
active therapy as part of an adaptive trial
design? A meta-epidemiological study
Tarp S1 , Dossing A1 , Furst DE2 , Boers M3 , Luta G4 , Gluud C5 ,
Bliddal H1 , Christensen R1
1
Musculoskeletal Statistics Unit, The Parker Institute, Department
of Rheumatology, Copenhagen University Hospitals, Bispebjerg
and Frederiksberg, Denmark; 2 University of California, Los
Angeles, David Geffen School of Medicine, Los Angeles, USA;
3 Department of Epidemiology and Biostatistics, and Amsterdam
Rheumatology and Immunology Center at VU University Medical
Center, Amsterdam, The Netherlands; 4 Department of Biostatistics,
Bioinformatics, and Biomathematics, Georgetown University
Medical Center, Washington, DC, USA; 5 Copenhagen Trial
Unit, Centre for Clinical Intervention Research, Rigshospitalet,
Copenhagen University Hospital, Copenhagen, Denmark
Background: Placebo-controlled trials in rheumatoid arthritis
(RA) increasingly employ adaptive trial designs that offer
early rescue therapy for participants with a poor response at
an interim time point. This may compromise inferences
drawn from an ITT analysis. Objective: To examine if
adaptive trial design confounds inference from ITT analysis
8
and to explore the quantified impact of adjusting for skewed
dropout between trial groups in the evaluation of harm
outcomes. Methods: A systematic search identified RA trials
comparing a biologic or tofacitinib vs placebo in approved
dose. The main outcome was the ratio of odds ratios (ROR)
quantifying the degree of bias associated with adaptive
vs non-adaptive trials. An ROR > 1 implies that adaptive
trials exaggerate serious adverse event (SAE) estimates.
Subsequently, two meta-analyses of trials were performed
based on participant number with SAEs (numerator): i) based
on the ITT population (denominator) expressed as odds
ratio (OR); ii) based on the total person-years of exposure
(denominator) expressed as rate ratio (RR). If not reported,
exposure was estimated by assuming a linear dropout rate
from baseline to follow-up. To compare the pooled OR and
RR, the ratio of ratios were calculated as Exp(ln[OR]–ln[RR]),
assuming acceptable comparability due to low event rate.
A ratio of OR to RR > 1 implies that OR exaggerates harm
estimates. Results: All 10 drugs except infliximab and
anakinra were represented in both the 69 non-adaptive and
the 31 adaptive trials included in the analysis (32,101 patients,
approximately 18,902 person-years). Figure 1 illustrates
pooled OR for adaptive and non-adaptive trials. The estimate
was exaggerated in adaptive trials (ROR 1.29, 1.06 to 1.57). The
ratio between the OR and the RR found that OR exaggerates
harm estimates (ratio 1.11, 1.08 to 1.14). Conclusion: This
study suggests that evaluation of harm outcomes (e.g. SAEs) is
affected by adaptive trial designs per se. Applying ITT analysis
for odds or risk statistics might overestimate the true effect
when analysing harm data from trials with skewed dropout
rates. We suggest that an exposure-time adjusted model
should be applied.
Background: The ACROBAT-NRSi tool is designed to
promote rigour in the assessment of risk of bias (RoB) in
non-randomised studies (NRS). The development and text of
the ACROBAT-NRSi tool draws heavily on clinical examples
to illustrate the tool and its application, but the feasibility
of it being applied to public health evidence has not yet
been fully explored. Objectives: To assess the feasibility and
usefulness of applying the ACROBAT-NRSi tool to NRS included
in public health intervention reviews. Methods: We applied
the ACROBAT-NRSi tool to studies of housing improvement
included in a published Cochrane Review. A group of five
researchers, all with experience of critical appraisal, read the
guidance and met to discuss queries and agree protocol level
considerations before assessing the studies independently.
Overall assessment of RoB was compared with two tools
used to assess the studies in the review (Effective Public
Health Practice (EPHP) tool and the Cochrane RoB tool).
Results: Agreement in assessments improved over time,
but initially considerable discussion was required to agree
on the interpretation of criteria and their application to
complex social interventions. Specific issues included the
incorporation of quasi-experimental study designs and the
extent to which confounding may be uncontrollable. The
ACROBAT RoB for some studies was ’low’ for two domains
(intervention departure and measurement), but the overall
assessment was ’serious’ or ’critical’ for all studies. Using
the Cochrane RoB, all studies had two or more (range 2
to 8) domains assessed as ’high’, implying a ’high’ RoB
overall. The EPHP overall assessment was ’low’ for two
studies, and ’moderate’ for three studies. Conclusions:
ACROBAT-NRSi may benefit from further development to
improve applicability to quasi-experimental studies of social
interventions. The tool has limited sensitivity to identify
variations in study quality. Where the RoB is ’serious’ or
’critical’ for a body of evidence, greater sensitivity to variations
in study quality could allow differentiation of best available
evidence from poorer quality evidence.
RO 1.2
Assessing the usability of ACROBAT-NRSI
for studies of exposure and intervention
in environmental health research
Rapid oral session 1
Risk of bias assessment tools
RO 1.1
ACROBAT-NRSi for public health: reporting on
feasibility and utility of applying ACROBAT to
studies of housing improvement
Thomson H1 , Campbell M2 , Craig P2 , Hilton Boon M2 , Katikireddi
V2
1
Cochrane Public Health Group, University of Glasgow, United
Kingdom; 2 MRC/CSOSocial&PublicHealthSciencesUnit, University
of Glasgow, United Kingdom
Morgan RL1 , Thayer K2 , Guyatt G1 , Blain R3 , Eftim S3 , Ross P3 ,
Santesso N1 , Holloway AC4 , Schünemann HJ1
1
Department of Clinical Epidemiology and Biostatistics, McMaster
University, Canada; 2 National Toxicology Program Office of Health
Assessment and Translation, National Institutes for Environmental
Health Sciences, USA; 3 ICF International, USA; 4 Department of
Obstetrics and Gynecology, McMaster University, USA
Background: Recently, A Cochrane Risk Of Bias Assessment
Tool (ACROBAT) was released to examine internal validity
(risk of bias) in non-randomized studies of interventions
(NRSI). The applicability of ACROBA- NRSI to studies dealing
with exposures or interventions in environmental health
has not yet been explored.
Objectives: This study
9
evaluated the application of ACROBAT-NRSI in studies
of environmental health exposure.
Methods: Two
independent reviewers sequentially applied ACROBAT-NRSI
to two systematic reviews containing 14 and 17 individual
studies of environmental exposures. Material available
to reviewers included instructions for application of the
ACROBAT-NRSI, including a list of possible confounders
specific to the exposures considered in the two reviews.
After completing the first and second assessments with the
tool, reviewers provided feedback regarding clarity of the
instrument and ease of application, highlighting barriers
to use. Modifications to enhance usability for studies of
exposures (rather than interventions) were made to the
tool. Results: Based on feedback from reviewers, two
modifications were required: 1. replacement of ’intervention’
with ’exposure’ throughout the document; and 2. provision
of additional explanation and examples for application of the
tool to studies using a cross-sectional design. Modification
to the tool increased reviewers’ understanding during the
second application; however, disagreements occurred when
judgments on domain-level risk of bias were made, specifically
in domains assessing bias due to confounding and bias in the
selection of the reported result. Conclusion: Modifications to
the tool were limited and improved understanding, as well as
reduced disagreement, among reviewers. We identified areas
requiring more clarification to further improve consistency of
the judgments. We will perform additional work examining
studies of exposure with the goal of providing a definitive
instrument to assess risk of bias in studies of environmental
exposures.
RO 1.3
Bias in measurement of adverse outcomes
in non-randomised studies
Junqueira D1 , Lana LGC1 , Menezes de Pádua CA1 , Zorzela L2 ,
Vohra S2
1
Universidade Federal de Minas Gerais, Brazil; 2 University of
Alberta, Canada
Background: The assessment of risk of bias (RoB) in
systematic reviews has recently evolved with the development
of a Cochrane RoB tool for non-randomised studies of
interventions (ACROBAT-NRSI). The tool evaluates internal
validity of non-randomised studies (NRS) through signalling
questions addressing domains of risk of bias. One of
the domains assesses bias in measurement of outcomes,
with no distinction between efficacy or harmful outcomes.
Objectives: To investigate potential bias with regard to a set of
signalling questions proposed to investigate misclassification
of adverse outcomes. Methods: We used a sample of NRS from
a systematic review of a specific adverse effect (lipodystrophy
induced by antiretroviral therapy). The signalling questions
included: (1) use of precise definition of the adverse effect,
(2) active mode of harms data collection, (3) accurate
mode of harms detection, (4) comparability of outcome
assessment across intervention groups, (5) specification of
who ascertained the adverse effect, (6) training of the person
who ascertained the adverse effect, and (7) reporting of timing
and frequency of harms data collection. Results: Despite the
fact that all studies (n = 20) were designed to investigate the
adverse effect lipodystrophy, 25% failed to provide a clear
definition for the outcome. Studies did not specify either
who ascertained the adverse effect nor the training level of
the staff (20% and 25%, respectively), were not designed to
actively and accurately detect the adverse effect (15% and
20%), and failed in reporting timing and frequency of harms
data collection (15%). Comparability of outcome assessment
across intervention groups was unclear in 10% of the studies.
In summary, 45% of the NRS were considered at moderate,
serious or critical risk of bias. Conclusions: A significant flaw
in the assessment of adverse outcomes was found in a sample
of NRS designed to investigate a pre-specified adverse effect.
It is expected that a substantial error in the detection of harm
outcomes may also be found in intervention trials. Further
research on the impact of differential measurement error on
the detection of adverse outcomes is required.
RO 1.4
Inter-rater reliability of the Risk Of Bias
Assessment Tool: for Non-Randomized
Studies of Interventions (ACROBAT-NRSI)
Couto E1 , Pike E1 , Torkilseng EB1 , Klemp M1
1
Norwegian Knowledge Centre for the Health Services, Norway
Background: We conducted a systematic review to examine
the possible association between the use of benzodiazepine
derivatives and benzodiazepine related drugs and mortality.
Non-randomised studies were included in the systematic
review. We assessed risk of bias using the Cochrane Risk
Of Bias Assessment Tool: for Non-Randomized Studies of
Interventions (ACROBAT-NRSI). Objectives: To present the
agreement between reviewers who used ACROBAT-NRSI to
assess the risk of bias of studies included in this systematic
review. Methods: Three reviewers (with experience in
assessing risk of bias) independently examined the risk of bias
of 33 prospective cohort studies. Inter-rater reliability of the
bias assessment was estimated by calculating kappa statistics
(k) using Stata. This was performed for each domain of bias
separately and for the final overall assessment. Agreement
was categorized as poor (K < 0.01), slight (k = 0.01 to 0.20),
fair (K = 0.21 to 0.40), moderate (K = 0.41 to 0.60), substantial
(K = 0.61 to 0.80), or almost perfect (0.81 to 1.00). Results:
Overall, assessed studies were judged as being at serious
or critical risk of bias. There was slight agreement for
the overall assessment of risk of bias (k = 0.03). For the
seven domains of bias, agreement ranged from k = -0.19 to
K = 0.35. There was poor agreement for two domains (bias
in measurement of interventions and due to departures from
intended interventions), slight for two (bias in selection of
participants into the study and in selection of the reported
result), and fair for three (bias due to confounding, due to
missing data, in measurement of outcomes). Conclusions:
10
The inter-rater agreement varied across the domains of the
risk of bias tool, ranging from poor to fair. While we had
slight agreement for the overall assessment of risk of bias,
all reviewers independently assessed overall risk of bias of
examined studies to be either serious or critical.
included trials of adequate quality based on a ≥ 6 cut-off.
Nineteen meta-analyses did not include trials of adequate
quality according to the Cochrane approach. Agreement
between PeDro and Cochrane was poor for PeDro scores of
≥ 5 points (k = 0.12; 95% CI 0.07 to 0.16), slight agreement
for ≥ 6 points (k = 0.24; 95% CI 0.16 to 0.32), and ≥ 7 points
(k = 0.39; 95% CI 0.286 to 0.510), and fair agreement (k = 0.44;
95% CI 0.314 to 0.574) ≥ 8 points (Figure 1). Conclusions:
The PeDro and Cochrane approaches lead to different sets of
trials of adequate quality. A consistent approach to assessing
RoB in trials of PT should be adopted.
RO 1.6
Developing and validating a tool to quantify
risk of bias in systematic reviews of genetic
association studies
Sohani Z1
1
McMaster University, Canada
RO 1.5
PEDro or Cochrane to assess the quality of
physiotherapy trials? A meta-epidemiological
study
Armijo-Olivo S1 , da Costa BR2 , Cummings GG1 , Ha C1 , Fuentes J1 ,
Saltaji H1 , Egger M3
1
University of Alberta, Canada; 2 Institute of Primary Health Care,
University of Bern, Switzeralnd; 3 Institute of Social & Preventive
Medicine (ISPM), University of Bern, Switzerland
Background: The use of different tools for evaluating the
quality of randomized controlled trials (RCTs) in systematic
reviews (SRs) can lead to discrepancies and skewed
interpretations of SR results. The PeDro tool, which uses
summary scores to determine the quality of RCTs is commonly
used in physical therapy (PT). An alternative approach is
based on an assessment of individual components such
as the items used in the Cochrane ’Risk of bias’ (RoB)
tool. Objective: To determine the agreement between
the Cochrane and the PeDro approaches to identifying
physiotherapy trials of adequate quality. Methods: We
conducted a meta-epidemiological study. RCTs in PT were
identified by searching the Cochrane Database of Systematic
Reviews for meta-analysis of PT interventions. Final scores for
PeDro and Cochrane assessments were extracted. We defined
trials of adequate quality as having adequate sequence
generation, concealment of allocation, and blinding of
outcome assessors (based on the Cochrane RoB tool) or
as trials with a PEDro score of at least 5 or 6 points, the
cut-offs widely used in the literature. We calculated the
kappa statistics to assess the agreement between PeDro
scores and the Cochrane approach for classifying trial quality.
We compared effect sizes from trials of adequate quality for
each meta-analysis between the two approaches. Results:
Forty-one Cochrane reviews and 353 trials of PT interventions
were included. All meta-analyses included trials of adequate
quality based on a ≥ 5 PEDro points cut-off and 37 (87.8%)
Background: Completion of the human genome project
and advances in genotyping technology has resulted in
an increase in publication of genetic association studies.
Systematic reviews and meta-analyses are a common method
of providing synthesized effect of a genetic variant on a trait
of interest, but, summary estimates are subject to bias due
to the varying methodological quality of individual studies.
Objective: To quantify bias introduced from individual
studies and inform meta-analyses, we developed, validated,
and empirically evaluated a tool that assesses the quality
of genetic association studies. Methods: We used published
guidelines and recommendations to create a list of items
with potential impact on quality. We chose final items in
consultation with five experts. Evaluation of the tool was
performed in two parts. Firstly, four reviewers rated 30 studies
randomly selected from a published meta-analysis. Their
ratings were used to assess construct validity, reliability, and
item discrimination of the tool. We report G-coefficients as
measures of inter-rater reliability, internal consistency, and
overall reliability of the tool, as well as item-total correlations
and Cronbach’s alpha to assess the discriminative ability
of each item. Secondly, a systematic review of published
meta-analyses was conducted. The tool was applied to 50
randomly selected meta-analyses from the literature and
studies of ‘poor’ quality were excluded to assess whether
application of our tool improves precision of estimates and
reduces heterogeneity. Results: The tool demonstrates
excellent psychometric properties and generates a quality
score for each study with corresponding ratings of ‘low’,
‘moderate’, or ‘high’ quality. When applied to individual
meta-analyses to exclude studies of low quality, we found
a decrease in heterogeneity and an increase in precision of
summary estimates. Conclusion: Integration of our tool into
meta-analyses to inform selection of studies for inclusion,
conduct sensitivity analyses, and perform meta-regressions,
can help improve the state of evidence in the field of
genetic epidemiology, which is currently plagued with
irreproducible findings.
11
RO 1.7
RO 1.8
PROBAST: a risk of bias tool for prediction
modelling studies
AMSTAR: helping decision makers distinguish
high and low quality systematic reviews that
include non-randomized studies
Wolff R1 , Whiting P2 , Mallett S3 , Riley R4 , Westwood M1 ,
Kleijnen J5 , Moons K6
1
Kleijnen Systematic Reviews Ltd, United Kingdom; 2 University
of Bristol, United Kingdom; 3 University of Birmingham, United
Kingdom; 4 University of Keele, United Kingdom; 5 Kleijnen
Systematic Reviews Ltd/ Maastricht University, United Kingdom/
The Netherlands; 6 University of Utrecht, The Netherlands
Background: Quality assessment of included studies is
a crucial step in any systematic review (SR). Review and
synthesis of prediction modelling studies is a relatively new
and evolving area and a tool facilitating quality assessment
for prognostic and diagnostic prediction modelling studies
is needed. Objectives: To introduce PROBAST (prediction
study risk of bias assessment tool), a tool for assessing the
risk of bias (RoB) and applicability of prediction modelling
studies in a SR. Methods: A Delphi process, involving 42
experts in the field of prediction research, was used until
agreement on the content of the final tool was reached.
Existing initiatives in the field of prediction research such as
the REMARK (reporting recommendations for tumour marker
prognostic studies) and TRIPOD (transparent reporting of a
multivariable prediction model for individual prognosis or
diagnosis) reporting guidelines formed part of the evidence
base for the tool development. The scope of PROBAST
was determined with consideration of existing tools, such as
QUIPS (quality in prognostic studies) and QUADAS (Quality
assessment of diagnostic accuracy studies). Results: After 6
rounds of the Delphi procedure, a final tool was developed
that utilises a domain-based structure supported by signalling
questions similar to QUADAS-2. PROBAST assesses the RoB
and applicability of prediction modelling studies. RoB
refers to the likelihood that a prediction model leads
to distorted predictive performance for its intended use
and targeted individuals. The predictive performance is
typically evaluated using calibration, discrimination, and
(re)classification. Applicability refers to the extent to which
the prediction model from the primary study matches the SR
question, for example in terms of the population or outcomes
of interest. PROBAST comprises 5 domains (participant
selection, outcome, predictors, sample size and flow, and
analysis) and 22 signalling questions grouped within the
domains. Conclusions: PROBAST can be used to assess the
quality of prediction modelling studies included in a SR. The
presentation will give an overview of the development process
and introduce the final tool.
Shea B1 , The AMSTAR Development Group TADG2
1
OHRI, Ottawa Hospital, University of Ottawa, Canada; 2 Bruyere
Research Institute, Bruyere Continuing Care, Canada
Background: AMSTAR (A Measurement Tool to Assess
Systematic Reviews) is widely used in the critical appraisal
of systematic reviews (SRs). The instrument was validated
using SR of randomized controlled trials (RCTs). Surveys
have shown that over 60% of SRs of treatment effects include
non-randomized studies (NRS). Objectives and methods: To
develop a version of AMSTAR capable of appraising SRs that
include NRS. A literature review found no critical appraisal
instrument that incorporated suitable measures of risk of bias
(RoB) of the component studies. An expert group reviewed the
results and suggested a single instrument with separate paths
to incorporate RoB assessment of either RCTs or NRS. The
opportunity was taken to update the AMSTAR instrument
in a number of other domains. Results: The modified
AMSTAR instrument now has 15 items. The review process is
guided more clearly by the PICOT framework (i.e. population,
intervention, comparison, outcome, and timeframe), makes
a more detailed assessment of literature searching methods,
gives guidance on whether authors have made adequate
assessment of RoB of included studies and whether limitations
were acknowledged in making conclusions from the results
of the review. The domains of bias (modified from those
in the ACROBAT-NRSI tool) are: freedom from residual
or unmeasured confounding; freedom from selection bias;
freedom from bias in measuring exposures and outcomes;
freedom from selective outcome/analysis reporting. The
revised instrument is undergoing pilot testing. Conclusions:
assessing the quality of SRs that include NRS is very
challenging. The modified AMSTAR is a first step towards
a more accurate appraisal of this important type of study.
Editorial issues
RO 2.1
What evidence is available and what is
required, in humanitarian assistance?
Allen C1 , Clarke M1 , Puri J2 , Archer F3 , Wong D3 , Eriksson A4
1 Evidence Aid, United Kingdom; 2 3ie, India; 3 Monash University,
Australia; 4 Karolinska Institutet, Sweden
Background: Challenges of conducting impact evaluations of
interventions, actions and strategies in natural disasters and
complex political emergencies are well recognized. Without a
rigorous scoping study, there is no single resource informing
researchers what the priority areas for impact evaluations (IEs)
12
are, nor a mapping of the gaps that exist in current research.
Objectives: To investigate the current landscape of evidence,
with emphasis on evidence from IEs, in the humanitarian
sector to identify areas in which actionable evidence is
available and where evidence is needed to direct research.
Methods: The study incorporated multiple methods to assess
available evidence including an online survey of participants
knowledgeable in the humanitarian sector, semi-structured
interviews with experts in the sector, extensive literature
reviews of repositories of humanitarian studies and strategy
documents of major humanitarian organisations and a gap
map presenting results of a thorough search for completed,
ongoing and planned IEs of humanitarian interventions.
Results: With the exception of health and nutrition, most areas
in the humanitarian sector suffer from a paucity of evidence.
An evidence gap map provides an illustration of the landscape
of evidence. There is agreement amongst policymakers that
decisions should be based on research evidence and that IEs
can and should have a greater role to play in building the
evidence base. Conclusions: Key recommendations of this
study, beyond informing areas and questions for IEs are that: Humanitarians must agree upon ways of prioritising research
needs. This study suggests a framework for prioritizing further
research. - Efforts should be made to index and classify existing
evidence and a single unifying repository or portal would to
improve the ease of accessibility to existing evidence. - A single
set of templates and reporting guidelines should be agreed
upon to aid in the indexing and classification of evaluation
studies. It would also be beneficial to agree upon standards
for data collection in these studies.
RO 2.2
Evidence Aid Resources: improving access
to systematic reviews that are relevant
to disasters
Aburrow T1 , Allen C2 , Clarke M2
1
Wiley, United Kingdom; 2 Evidence Aid, United Kingdom
Background: Evidence Aid, with partners, assesses published
systematic reviews (SRs) to determine their relevance to
natural disasters, humanitarian crises or major healthcare
emergencies. The SRs will have health outcomes and
are featured on www.EvidenceAid.org/. Resources with a
summary of each review and link to the full text. The
main source of SRs for Evidence Aid continues to be
Cochrane Reviews, but several non-Cochrane SRs have
recently been added. Most of the Cochrane Reviews are
also included in the relevant Evidence Aid Special Collection
(www.thecochranelibrary.com). Objectives: To provide a
curated and unique collection of SRs to facilitate an informed
response by those preparing for and responding to natural
disasters and humanitarian crises. Methods: Evidence Aid
works closely with Cochrane and other external partners to
identify evidence with health outcomes that decision-makers
can assimilate quickly. Members of the Evidence Aid team
(largely volunteers), provide a brief summary of the SR, which
is featured alongside a link to the full article. Wiley’s editorial
team provides support to publish the resources, and relevant
information, on a dedicated website: www.EvidenceAid.org.
As SRs are added, they are communicated across social media
channels. Cochrane and Wiley have provided free one-click
access to the full version of all Cochrane Reviews featured
in the Evidence Aid resource, to everyone in the world and,
recently, Elsevier agreed to provide free click-through access
(from the Evidence Aid website) to a review published in
one of their journals. Results: Since November 2011, over
200 titles have been published in the resources collection
and these titles are split into 39 categories. The /resources
page is the most viewed on the site after the homepage, and
on average holds a 12% share of all traffic. Conclusions:
Evidence Aid provides a single source of health-related SRs
and specially prepared summaries that might be relevant to
those preparing for and responding to natural disasters and
humanitarian crises. Evidence Aid continues to identify SRs
and works with publishers to secure free, global access to the
reviews.
RO 2.3
Reviews in the humanitarian field:
adaptation and transparency
Krystalli R1 , Ott E2 , Stites E1
1
Feinstein International Center, Tufts University, USA; 2 Oxfam GB,
United Kingdom
Issue:
Humanitarian actors and researchers have
amassed evidence, but synthesizing this information and
communicating it to policymakers and practitioners remains
challenging. Although all reviews face methodological
challenges, the context for humanitarian evidence is
fundamentally different from the medical trial context from
which systematic reviews emerged. The settings –which
often involve active armed conflict, natural disasters, or
the immediate aftermath of conflict or disasters –render
data collection difficult or impossible. When data exist, the
reliability, representativeness, and generalisability may be
limited by access and numerous biases, including selection,
recall, and reporting bias. Even in humanitarian settings
that are not characterized by instability or insecurity, the
types of studies can be limited by the fragility of protracted
crises, the sensitivity of the information, and issues related to
identifying and accessing affected populations. Response: As
a response to this context and the low-level of understanding
around systematic reviews in the humanitarian field, the
Humanitarian Evidence Programme brought together and
adapted existing literature for a guidance note on evidence
synthesis in the humanitarian sector. Building on work from
Cochrane, EvidenceAid, Overseas Development Institute,
and 3ie, this note presents technical guidance in a manner
accessible to an audience with a strong humanitarian
identity. One challenge is to synthesize and communicate
’low-confidence’ qualitative evidence. All of our evidence
synthesis outputs strive to be transparent about which
13
evidence they synthesize, clear about gaps and limits in
the literature, and systematic in their mode of analysis of
existing research. The guidance note is currently being
used for a number of reviews, from acute malnutrition to
shelter and settlement strategies. As with other reviews,
the Humanitarian Evidence Programme reviews hope to
identify gaps in existing research and knowledge, form
conclusions with greater confidence than with individual
studies, showcase disagreement and diversity among the
literature, and highlight opportunities for further research.
RO 2.4
Evidence aid: improving the science and
evidence base of disaster response - a policy
Delphi engagement
Allen C1 , Jillson I2 , Clarke M1 , Waller S3 , Koehlmoos T4 , Mellon D1 ,
Kayabu B1
1
Evidence Aid, United Kingdom; 2 Georgetown University, USA;
3
Uniformed Services University of the Health Sciences, USA; 4 US
Marine Corps, USA
Background: The Policy Delphi Technique is a policy analysis
method used to engage groups of experts in dialogue
regarding a particular issue of focus. The Policy Delphi
approach entails a series of iterative, structured dialogues,
with each one referred to as a ‘round’ and building on
the one before. Respondents answer a series of questions
anonymously, the responses are summarized and used as a
basis for subsequent questions. Questions may be added as
the questionnaires are developed. Policy Delphi has been
used in disaster management since the 1970s; differing from
the traditional Delphi approach, it does not seek consensus,
but explores alternatives and their implications. Objectives:
The proposed policy Delphi would be conducted between
May and November 2015 as a means of preparing for the
proposed 2015 Evidence Aid meeting in the USA in November.
The purpose of the policy Delphi is to engage a wide range
of key stakeholders in dialogue regarding disaster responses,
including for example, evidence for best practices in disaster
response, approaches to improving investments in disaster
response with Cochrane-style (systematic review) analysis
of evidence, identification of gaps in the evidence base
for disaster response; and factors that impact on effective
disaster response. Methods: Three rounds: the first and
second online with around 100 selected panelists and the
third being conducted as part of the meeting. The broad
research questions are: To what extent is evidence for best
practices in disaster response available to a wide range of
stakeholders? To what extent is Cochrane-style (systematic
review) analysis used to assess evidence for best practices in
disaster response? What are the most effective approaches
to improving the cost-effectiveness of investments in disaster
response? How can the ethical, legal and social issues related
to disaster response decision-making be most effectively
addressed? What are the factors that impact on effective
disaster response decision-making?
Results & Conclusions: Each round has an interim report
with findings. The findings and conclusions from these data
will be shared at the Colloquium.
RO 2.5
Capacity-sharing for improved filtering
of information for decision-making
in the African context
Stewart R1 , Langer L1 , Zaranyika H1 , Choge I1 , Erasmus Y1 ,
Jordaan S1 , Mitchell J1 , Gerald L1
1
University of Johannesburg, South Africa
Introduction: There is limited capacity in systematic
reviewing and evidence-use in low- and middle-income
countries, particularly across Africa. This is reflected in
the Global Evidence Synthesis Initiative (GESI) in which
Cochrane plays a leading role. There are however significant
pockets of capacity. The Africa Evidence Network, which we
coordinate, seeks to build capacity by supporting existing
initiatives that aim to filter information for better decisions
in health care and other fields, amongst researchers,
governments and non-governmental organisations (NGOs).
This presentation will describe our activities, approaches
and our preliminary evaluation, and invite the audience to
share their own experiences. Background: The Network
was formed in 2012 in an attempt to bring together these
disparate initiatives and provide a forum for collaboration
and capacity-development. Since 2014 the Network has
been supported by the UK Department for International
Development via our University of Johannesburg’s ‘Building
Capacity to Use Research Evidence’ initiative. This BCURE
programme supports the Africa Evidence Network website,
regular newsletters and biannual Colloquia. We also run a
capacity-building programme to increase use of research by
the governments of Malawi and South Africa.
Key Approaches: 1. A southern-based initiative, consciously
avoiding ‘parachuting in’, instead working to ensure we build
lasting relationships with the communities in which we work.
2. Using multiple media to exchange knowledge, including
face-to-face interactions, social networks, podcasts etc.,
thus optimizing access to knowledge and networks beyond
sectoral expertise. 3. Taking great care to ensure we build
on existing initiatives, capacities and resources, to ensure
we meet real needs and truly contribute to current capacity
in research-use rather than merely replicating, duplicating
or contradicting current efforts. 4. Avoiding the practice of
training the trainers and instead adopting an apprenticeship
approach. 5. Playing the role of matchmakers, fostering
meaningful networks of research-users, research-producers
and intermediaries in the countries in which we work.
14
RO 2.6
RO 2.7
Policy BUDDIES: lessons learnt by
researchers engaging in ’buddying’ with
provincial policymakers in South Africa to
advance evidence-informed decision-making
Story of an epidemic - public health evidence
South Asia - creating a social epidemic by
experimenting with few
1
2
1
3
4
Young T , Naude CE , Wiysonge CS , Kredo T , Dudley L ,
Garner P5
1 Centre for Evidence-Based Health Care, Stellenbosch University;
South African Cochrane Centre, South African Medical Research
Council, South Africa; 2 Centre for Evidence-Based Health Care,
Stellenbosch University, South Africa; 3 South African Cochrane
Centre, South African Medical Research Council, South Africa;
4
Division of Community Health, Stellenbosch University, South
Africa; 5 Effective Health Care Research Consortium, Liverpool
School of Tropical Medicine, United Kingdom
Background: Successful dialogue between researchers
and policymakers may facilitate evidence-informed
decision-making by helping policymakers understand what
research can or cannot do, and researchers understand
how policy making works.
Objectives: To develop
and apply a formal one-to-one partnership (‘buddying’)
between provincial policymakers and evidence-based health
researchers (‘buddies’); and to evaluate the buddies’
experiences. Methods: Policymakers identified questions
relevant to them during interviews in the baseline situational
analysis. Each buddy initiated and drove the interaction
process with a policymaker for their question, drawing support
from the buddies group that included all researchers involved
in the project. A dedicated online website and monthly
contact sessions were used by buddies to share resources
and continuously document interactions, reflections and
experiences. Results: We applied buddying to five questions:
(1) adherence support for chronic diseases; (2) prevention
of mother-to-child transmission of HIV; (3) task shifting for
antiretroviral treatment; (4) health promotion for diet and
physical activity’ and (5) integration of care for chronic
diseases. Having policymakers who champion the use of
evidence appears to facilitate ‘buddying’. Engagement helped
researchers understand that systematic review evidence is
only a piece of the complex policymaking puzzle. Buddying
can be time-consuming and is affected by frequent staff
turnover of policymakers. Scenarios for evidence requests
were different: for some questions, evidence was sought
to endorse existing policies and for others, to inform new
policy development. Conclusions: Successful, sustainable
engagement between researchers and policymakers to
advance evidence-informed decision-making is not a quick,
simple process. Buddying, centred on partnerships and
relationships, may provide a knowledge translation platform
that could strengthen health interventions and services by
enhancing evidence into policy; however, many uncertainties
still remain.
Nair N S1 , Mujja A2 , Lewis M3 , Nagaraja R3 , T Venkatesh B4
1
Director, Public Health Evidence South Asia, Manipal University,
India; 2 Research Assistant, Public Health Evidence South Asia,
Manipal University, India; 3 Research Scholar, Public Health
Evidence South Asia, Manipal University, India; 4 Research Officer,
Public Health Evidence South Asia, Manipal University, India
Background: Public Health Evidence South Asia (PHESA) is
an initiative aiming to meet the public health evidence needs
of the South Asian region. The initiative, which includes the
South Asian satellite of the Cochrane Public Health Group
(CPHG), is based at Manipal University, India. Objectives:
This paper describes the formation and successful journey of
PHESA. This is a model that demonstrates that a few dedicated
and effective people can create a social epidemic like PHESA
- the theory of few. The paper talks on how infection began
and to what extent it has spread over a period of two years
from 2013. Methods: The first part of the paper concentrates
on genesis of PHESA; the struggles involved in convincing
Manipal University to support the activity, working out vision
and mission, acquiring funding, logistics and identifying
a dedicated workforce and so on. It also elucidates the
activities of networking with organizations and individuals,
sensitization workshops, conference presentations etc. The
second part of the paper shares the success story of PHESA.
Results: During the last two years PHESA has trained 150
researchers/students, collaborated with six national and six
international organizations like Public Health Foundation of
India (PHFI), the World Health Organization (WHO), Nossal
Institute of Global Health, University of Melbourne and
University of Groningen. Apart from this, PHESA has
three full time PhD scholars registered with funding, 15
completed student internship projects and has attracted
research funding of INR 8.5 million. The research output
includes nine published papers, nine communicated papers
and six papers in preparation. The first PHESA colloquium
was organized in February 2015 with 75 participants, four
plenary talks, 38 oral presentations, 25 posters and three
workshops. Conclusions: The message we are experimenting
is ’money is not all that matters but few contagious, sticky
people- connectors and experts (mavens) - can make a big
difference at the right context for public health in low- and
middle-income countries’. The journey is still on.
15
RO 2.8
Use of Cochrane Reviews in developing
Clinical Practice Guidelines (CPGs) in Malaysia
Abdullah Thani NSI1 , Muhamad NA1 , Lai NM2 , Tan ML3 , Ho J3
1 Institute for Medical Research, Malaysia; 2 Taylor’s University,
Malaysia; 3 Penang Medical College, Malaysia
Background: Clinical Practice Guidelines (CPGs) are
developed to improve the quality in healthcare practice based
on the best available evidence. The process of developing
the CPGs include systematically screening available evidence
from systematic reviews. To-date, there have been 70 CPGs
developed in Malaysia since 2004. Cochrane Reviews are
widely regarded as the chief source of best evidence in health
care that should be cited in the CPGs. Objectives: We
examined the citation of Cochrane Reviews in Malaysian
CPGs.
Methods: Seventy CPGs were retrieved from
the official Ministry of Health (MOH) Malaysia website:
http://www.moh.gov.my/. The reference list of each CPG
was screened manually to assess whether Cochrane Reviews
were included. Two authors determined whether there was
a key Cochrane Review that should be cited in each topic
of the CPG without knowledge of the CPG references, and
a third author assessed whether these key reviews were
included in the corresponding CPG. Key reviews are large
reviews with clear results and recommendations for practice.
Results: Seventy CPGs covered 17 topics. A total of 219 (2.94%)
Cochrane Reviews and 527 (5%) non-Cochrane reviews were
identified from 9893 references. A median of one Cochrane and
four non-Cochrane reviews were cited per CPG (Table 1). From
2004 to 2014 Cochrane Reviews have been increasingly cited
(Table 2). Out of 50 key Cochrane Reviews relevant to the CPG
topics, 16 key reviews (30.8%) were included while 34 were
not. Conclusions: Despite an encouraging trend of increasing
citation, which suggests rising awareness of the importance
of Cochrane Reviews, the inclusion of certain key Cochrane
Reviews in the Malaysian CPGs remains unsatisfactory.
RO 2.9
Measuring global health inequalities: the
Gini, Theil and Slope inequality index for 291
diseases and injuries and 76 risk factors
von Philipsborn P1 , Steinbeis F2 , Gotham D3 , Bender ME4 ,
Matthiessen C5 , Sauter C6 , Tinnemann P7
1
Technische Universät München and Universities Allied for Essential
Medicines (UAEM), Germany; 2 Charité Universitätsmedizin Berlin
and Universities Allied for Essential Medicines (UAEM), Germany;
3
Imperial College London, and Universities Allied for Essential
Medicines (UAEM), United Kingdom; 4 Charité Universitätsmedizin
Berlin, and Universities Allied for Essential Medicines (UAEM),
Germany; 5 University of Copenhagen, and Universities Allied for
Essential Medicines (UAEM), Denmark; 6 Universität Bonn, Germany;
7
Charité Universitätsmedizin Berlin, Germany
Background: {Numerous health care initiatives, including
Cochrane, aim to reduce global health inequalities.
Knowledge about the contribution of different diseases and
risk factors and existing disparities in health status can
contribute to evidence-based, equity-sensitive priority setting
in research and policy-making. Objectives: To calculate
concise, comprehensive inequality measures to synthesize the
16
vast amount of information on health inequalities provided by
the Global Burden of Disease Study 2010. Methods: We use the
population-weighted Gini and Theil index to measure relative
health inequalities, and the population-weighted Slope
Inequality Index (SII) to measure absolute inequality, applying
them to country-level disease burden data, measured in
disability-adjusted life years (DALYs). The Gini index and
the SSI are derived from the Lorenz Curve and the Pen’s
Parade, respectively (see Graphs 1 and 2), which can also be
used to visualize results. All three indices have been used
widely in analysis of inequality in the distribution of health
outcomes. Results: Overall relative and absolute global
health inequalities increased between 1990 and 2000, but
have decreased after 2000. Between 2000 and 2010, absolute
inequality fell below 1990 levels, while relative inequality did
not (see Graph 3). Cause groups that drove the rise in global
inequality between 1990 and 2000 were HIV, interpersonal
violence, and road traffic injuries. Overall, communicable,
maternal, neonatal and nutritional disorders (Gini = 0.583),
as well as injuries (Gin i= 0.302) contribute more to existing
inequalities than non-communicable diseases (Gini = 0.172)
(see Graph 4), with considerable variance within these groups.
In 2010, relative global health inequalities among women
(Gini = 0.318) were considerably larger than among men
(Gini = 0.285). Conclusions: While global health inequality
has decreased since 2000, overall levels remain high, with
a marked variance amongst different health conditions and
risk factors. Increased attention for those conditions and
risk factors that contribute most to existing global health
inequalities may be warranted.
Advanced methods: Overviews of
reviews and network meta-analyses
RO 3.1
What guidance is available for researchers
conducting overviews of reviews? a scoping
review and qualitative metasummary
Foisy M1 , Fernandes RM2 , Hartling L1
1
Alberta Research Centre for Health Evidence, University of
Alberta, Canada; 2 Cochrane Child Health Field; Cochrane Portugal;
Department of Pediatrics and Clinical Pharmacology Unit, Lisbon
Academic Medical Centre, Portugal
Background: Overviews of reviews (overviews) compile
data from multiple systematic reviews to provide a single
synthesis of relevant evidence for clinical and policy
decision-making. Despite their increasing popularity, there
is limited methodological guidance available for researchers
wishing to conduct overviews. Objectives: To identify and
collate all published and unpublished documents containing
methodological guidance for the conduct or reporting of
overviews. Our aim was to provide a map of existing guidance
documents and identify similarities, differences, and gaps
17
in guidance. Methods: We conducted a database search
supplemented by iterative snowball searching, handsearched
conference proceedings and relevant websites, and contacted
overview authors and Cochrane Review Groups. Guidance
statements across included documents were grouped by
stage of the overview process and analyzed within each
stage. Results: We identified 48 guidance documents
produced by 20 author groups: 29 contained explicit
methodological guidance; 10 were peer-review checklists
or reporting guidelines; and nine described the challenges an
author group faced when conducting one or more published
overviews. Overall, 79% of these documents were written
by authors affiliated with Cochrane. General guidance was
available for the following steps: deciding whether to conduct
an overview, specifying the scope, searching, screening, and
inclusion. This guidance was mostly consistent, though there
were some minor discrepancies across included documents.
There was limited or conflicting guidance on extraction and
analysis of results data, methodological quality assessment,
and grading the quality of evidence. Most documents
identified additional challenges in conducting overviews for
which they did not provide guidance. Conclusions: Compiling
existing methodological guidance for overviews provides a
valuable resource for overview authors. The results of this
project will inform our research on overview methods and
are being used to update the overviews chapter for the next
edition of the Cochrane Handbook.
RO 3.2
Scoping reviews versus systematic reviews:
results from a scoping review of scoping
reviews
Tricco A1 , Lillie E1 , Zarin W1 , O’Brien K2 , Colquhoun H2 , Kastner M1 ,
Levac D3 , Ng C1 , Pearson Sharpe J1 , Wilson K1 , Kenny M1 , Warren
R1 , Wilson C1 , Stelfox H4 , Moher D3 , McGowan J3
1
Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Canada;
2
University of Toronto, Canada; 3 University of Ottawa, Canada;
4 University of Calgary, Canada
Background: Scoping reviews are used to map concepts
underpinning a research area and the main sources and types
of evidence available. Currently, a lack of consistency exists
in the terminology, definition, methods, and reporting of
scoping reviews appearing in the literature. Objectives: We
aimed to synthesize scoping review approaches through a
scoping review. Methods: We conducted a scoping review of
scoping reviews. Nine multi-disciplinary electronic databases
were searched for scoping reviews or studies that discussed
scoping review methodology (e.g. the Cochrane Library,
PsycINFO, Philosopher’s Index). The citations were screened
independently by two reviewers for inclusion, and data
extraction was verified by a second reviewer. Descriptive
analysis was conducted. Results: We included 545 scoping
review articles. Preliminary results suggest that there has
been an exponential increase in the number of scoping
reviews conducted in the past five years. The majority
were conducted in North America (56%) and Europe (39%).
Most studies reported a source of funding (61%) that was
either from the government (25%) or a health authority
(16%). The most commonly reported approaches to the
conduct of scoping reviews included searching one or more
database (92%), scanning reference lists (59%), searching grey
literature (52%), limiting inclusion by date (68%) or language
(62%), and not conducting a formal quality (risk of bias)
appraisal (81%). In most cases, the results of the scoping
review were used to provide recommendations for future
research (78%), identify evidence gaps (76%), and/or to specify
policy or practice recommendations (54%). Conclusions: The
number of scoping reviews conducted per year is increasing
exponentially. Scoping reviews are used to set research
agendas and provide policy or practice recommendations,
yet there is variability in their reporting and the methods
that are being used. Further research on scoping reviews
is warranted, in particular, there is need for a guideline to
standardize reporting.
RO 3.3
Overview of screening recommendations
from evidence-based guidelines as decision
support for a reoriented parent-child
preventive care programme
Reinsperger I1 , Piso B1 , Winkler R1
1
Ludwig Boltzmann Institute for Health Technology Assessment,
Austria
Background: Against the backdrop of the observed
rapid increase in published clinical practice guidelines
(CPGs), systematic CPG analyses (‘overviews of CPGs’)
can be considered as information filters by summarising
recommendations of evidence-based CPGs. Objectives:
We aimed to provide the Ministry of Health with an
overview of antenatal, perinatal and postnatal screening
recommendations from international evidence-based CPGs
as decision support for a reoriented national parent-child
preventive care programme.
Methods: A systematic
search in two guideline databases was supplemented by
a comprehensive handsearch via websites of international
institutions developing CPGs. We included evidence-based
guidelines from western industrialised countries providing
screening recommendations for pregnancy and early
childhood that were published or updated within the past
five years. Guideline development had to be based on
a systematic literature search and recommendations had
to be explicitly linked to the evidence. We extracted the
screening recommendations, grades of recommendation,
the screening times and methods, treatment options
and potential harms.
Results: The systematic CPG
analysis provides a comprehensive, systematic overview
of recommendations from evidence-based guidelines of
international institutions for screening measures during
pregnancy and early childhood: We included a total of
138 guidelines, published by 14 institutions, and extracted
18
recommendations related to 92 health threats. Basically,
grades of recommendations were high for all pro-screening
recommendations, whereas certainty decreased across all
contra-screening recommendations. Conclusions: The policy
decision on the implementation of new screening measures
or the adaptation of existing screening approaches requires
the appraisal of the provided CPG analysis. Currently, an
interdisciplinary expert panel discusses the transferability
and applicability of the international CPGs and develops
recommendations for the national context.
development. Our Overview comprehensively, rigorously and
effectively signposts clinicians to reviews that are most likely
to support delivery of effective care.
RO 3.5
Constructing an overview of systematic
reviews of diagnostic test accuracy
Hunt H1 , Hyde C1
1 University of Exeter Medical School, United Kingdom
RO 3.4
Completing the first Cochrane overview
of stroke reviews: experiences of the
Cochrane Stroke Group
Pollock A1 , Farmer S1 , Brady M1 , Langhorne P2 , Mead G3 , Mehrholz
J4 , van Wijck F1
1
Glasgow Caledonian University, United Kingdom; 2 University
of Glasgow, United Kingdom; 3 University of Edinburgh, United
Kingdom; 4 Klinik Bavaria in Kreischa GmbH, Germany
Background: Cochrane Overviews aim to provide a succinct
overview of reviews relevant to a specific clinical question.
Objectives: To explore the key methods associated with
producing the first Cochrane Stroke Group Overview.
Methods: Six editors from the Cochrane Stroke Group
systematically explored methodological issues arising during
the overview process. Monthly teleconferences and one
full-day meeting were held to discuss and agree solutions
to these issues. Each methodological issue, and how it
was addressed, was recorded. Results: Our completed
overview included 40 reviews (19 Cochrane, 21 non-Cochrane).
Key methodological challenges encountered related to: 1.
inclusion of non-Cochrane reviews: this added substantial
complexity to review inclusion, primarily due to overlap
between reviews, requiring systematic exploration of the
inclusion of more than 700 studies; 37 reviews were
excluded as they were superseded by more up-to-date or
comprehensive reviews; 2. quality assessment of included
reviews: we found use of the AMSTAR quality assessment tool
challenging due to its use of many multifaceted questions.
We developed a modified version of the AMSTAR tool
(mAMSTAR), comprising 30 simple uni-variable questions. This
enabled transparent documentation of the process of quality
assessment. 3. quality of evidence within reviews: we planned
to use the GRADE approach to assign levels of evidence to
each of the 127 relevant comparisons included in the overview,
but found it difficult to achieve consistent application. We
therefore developed and used an objective algorithm based
on participant numbers, trial risk of bias, heterogeneity
and review quality. 4. strength of implications: only
1/127 comparisons was graded as high quality, and 76/127
comparisons were graded as low or very-low quality evidence,
limiting clinical implications.
Conclusions: Overviews
should benefit evidence-based stroke care, but tools used
to support Cochrane Overviews need further methodological
Objectives: The aim of this presentation is to present an early
career researcher’s experience of conducting an overview of
the existing systematic review evidence for the diagnostic
test accuracy of brief cognitive assessments used to identify
dementia in primary care. Description: systematic reviews
of existing health evidence such as those conducted by
Cochrane are recognised as the international ’gold standard’
for high quality trusted information. As more systematic
reviews are produced, there is a growing need to summarise
and appraise the systematic review evidence in order to
filter the information overload and give decision makers the
best information available. However, whilst guidance on
conducting an ‘Overviews of reviews’ is covered in Chapter
22 of the Cochrane Handbook for Systematic Reviews of
Interventions (version 5.1.0), there is no published Cochrane
guidance for authors conducting overviews of reviews of
diagnostic test accuracy. In this presentation, the author will
discuss: - What is the purpose of conducting an overview of
systematic reviews? - How do Cochrane and non-Cochrane
reviews differ, and should they be treated differently? What parallels and contrasts exist between accuracy and
non-accuracy reviews? - How should an author handle a
change in review perspective (e.g. primary care focus when
reviewing general setting reviews)? - What would a good
overview of systematic reviews of diagnostic test accuracy
look like? The author aims to situate the evidence for
overviews of systematic reviews of diagnostic test accuracy,
and prompt discussion of issues raised.
RO 3.6
Including observational evidence in a
network meta-analysis of RCTs: a case study
in schizophrenia
Efthimiou O1 , Mavridis D2 , Debray T3 , Leucht S4 , Samara M5 ,
Salanti G1
1
Department of Hygiene and Epidemiology, University of Ioannina
School of Medicine, Greece; 2 Department of Primary Education,
University of Ioannina, Greece; 3 The Dutch Cochrane Centre, Julius
Center for Health Sciences and Primary Care, University Medical
Center Utrecht, The Netherlands; 4 Department of Psychiatry and
Psychotherapy, Technische Universität München, Klinikum rechts
der Isar, Ismaningerstr, Germany; 5 Department of Psychiatry and
Psychotherapy, Technische Universität München, Klinikum rechts
der Isar, Ismaningerstr., Germany
19
Background:
Observational studies convey valuable
information about the effectiveness of interventions in real-life
clinical practice. Recent research has found no evidence
of systematic differences in the estimated effects between
observational and randomized controlled trials (RCTs).
However, there is little available guidance on how to combine
real-world data with evidence from RCTs, especially when
there are multiple competing available treatments for the
same condition. Objectives: To assess existing methodology
and develop new methods for jointly synthesizing evidence on
relative treatment effects provided by RCTs as well as evidence
from observational studies that report individual patient
data (IPD) in a network meta-analysis (NMA). Methods: We
adjusted the relative treatment effects from the observational
studies utilizing patient level covariates. We combined the
adjusted estimates with the evidence provided by RCTs in
a joint network meta-regression, accounting for trial-level
covariates, and exploring a variety of alternative methods.
We quantified the impact of different levels of confidence for
the adjusted estimates on the NMA relative effects. Results:
We applied our methods in a published network of 167 RCTs,
comparing 15 antipsychotics and placebo for schizophrenia.
We extended our evidence base by including IPD from a cohort
study involving five interventions and more than 10,000
patients. The adjusted evidence was found to be in agreement
with the evidence from RCTs. Statistical inconsistency in the
network was unaffected by the inclusion of observational
evidence. A range of sensitivity analyses shows our results
to be robust to the alternative methods used. Conclusions:
Including real-world evidence from observational studies can
corroborate findings of a NMA based on RCTs alone, increase
precision and enhance the decision-making process.
where we first used random-effects, found counter-intuitive
results, and then applied fixed-effects models. Results:
Applying the GRADE (Grading of Recommendations
Assessment, Development and Evaluation) approach, we used
random-effects models to generate direct, indirect, and
network estimates. We found instances where direct and
indirect estimates were similar, but CIs of NMA estimates
were far wider than the direct estimates. In these instances,
fixed-effects models generated narrower CIs consistent with
the direct estimates, e.g. in an NMA of alternative fluids for
resuscitation in septic shock, random-effects showed very
similar point estimates for direct and indirect comparisons;
the CI around the direct estimate was far narrower (Table).
Counter-intuitively, the NMA estimate was far wider than
the direct estimate. In the fixed-effect model, the sparse
data from the indirect estimate changed the NMA estimate
little in comparison to the direct estimate –as one would
intuitively expect (Table). We have encountered a number
of such situations. As in this example, in the face of similar
point estimates from direct and indirect comparisons, basing
clinical decision-making on an NMA estimate that is far less
precise than the direct estimate is inappropriate. Options
are to use the direct estimate only, or apply a fixed-effects
model that generates sensible results.
Conclusions:
Those conducting Bayesian NMA need to be aware of
potential problems with random-effect models and, when
counter-intuitive results arise, consider fixed-effects models.
RO 3.7
RO 4.1
Problems with Bayesian random effects
in network meta-analysis
The PRISMA-IPD Statement: preferred
reporting items for a systematic review and
meta-analysis of individual participant data
Guyatt G1 , Murad H2 , Heels-Ansdell D1 , Puhan M3
1
McMaster University, Canada; 2 Mayo Clinic, USA; 3 Institute for
Social and Preventative Medicine, Zurich, Switzerland
Background: In most network meta-analyses (NMAs),
substantial variability in results makes random-effects an
appealing model. A potential advantage of NMAs is the
possibility that consistent results from direct andindirect
comparisons will narrow confidence intervals (CIs) in
comparison to those from direct estimates alone. If direct
comparisons provide higher quality evidence than network
estimates, clinicians should use that higher certainty evidence
to guide their practice. Objective: To illustrate problems
in the use of random-effect models in Bayesian NMAs.
Method: We conducted a number of Bayesian NMAs
Quality of reporting
Stewart L1 , Clarke M1 , Rovers M2 , Riley R1 , Simmonds M1 , Stewart
G1 , Tierney J1
1
IPD Meta-analysis Methods Group, United Kingdom; 2 IPD
Meta-analysis Methods Group, The Netherlands
Background: Systematic reviews with meta-analysis of
individual participant data (IPD) aim to collect, check, and
re-analyze individual-level data from all studies addressing
a particular research question and are considered a gold
standard approach to synthesis.
Like most areas of
research, reporting could be improved, making it easier
for readers to understand, critique and implement findings.
Standard PRISMA guidelines are geared towards systematic
reviews based on aggregate data and so lack reference to
20
some important aspects of the IPD approach. Objectives:
The PRISMA-IPD extension was developed to provide a
framework for full and transparent reporting of IPD review
methods. Methods: Development followed the EQUATOR
Network framework guidance and used the standard PRISMA
statement as a starting point to draft additional relevant
material. A web-based survey informed discussion at an
international workshop. The statement was drafted and
refined iteratively and agreement on the PRISMA-IPD checklist
and flow diagram agreed by consensus. Results: PRISMA-IPD
contains 23 items in which the wording has been modified
to take reflect IPD approaches, and three new items on: (1)
methods of checking data integrity; and (2) reporting any
important issues identified; and (3) on methods of exploring
variation and one new item from re-arrangement. Although
developed primarily for reviews of randomized trials, many
items will apply in other contexts including for reviews of
diagnosis and prognosis. Conclusions: PRISMA IPD includes
a tailored checklist and flow diagram, which we hope will
improve reporting. If, as a result of current initiatives aiming
to make provision of clinical trial data for research purposes
a legal, regulatory or ethical requirement IPD become more
readily available, then it is likely that in the future more
systematic reviews will access and analyze IPD. This will likely
include synthesis of IPD released in controversial areas where
transparent, complete and high quality reporting is essential.
Authors and peer reviewers are encouraged to use to improve
reporting and journal editors to include it in their endorsement
of PRISMA.
RO 4.2
The Consensus on Exercise Reporting
Template (CERT): a Delphi study
investigating a standardised method for
reporting exercise programs
Slade S1 , Dionne C2 , Underwood M3 , Buchbinder R4
1 Monash University, Department of Epidemiology and Preventive
Medicine, Melbourne, Australia; 2 Monash Department of
Clinical Epidemiology, Cabrini Hospital, Melbourne, Australia;
3 Department of Rehabilitation, Faculty of Medicine, Laval
University, Québec, Canada; 4 Warwick Clinical Trials Unit, Division
of Health Sciences, Warwick Medical School, University of Warwick,
Coventry, United Kingdom
Background: Exercise is effective, integral to health
and important for people with chronic health conditions.
A systematic review of 73 reviews of exercise for
chronic conditions reported suboptimal descriptions of
interventions and concluded that this hinders replication and
implementation. Without explicit descriptions, clinicians,
patients and researchers remain unclear about effective
programs. Objectives: The aim was to develop a standardised
method for reporting key and essential exercise program
details evaluated in clinical trials. Methods: A modified Delphi
technique was used to gain consensus among an international
panel of experts identified by systematic review authorship,
international research and clinical profile and peer referral.
An online survey of 42 questions was developed from our
systematic review. Experts from 14 countries indicated
item importance on a scale of 0 to 10 and gave free text
comments. We used three sequential rounds of anonymous
online questionnaires. The final template will be piloted on a
random sample of systematic reviews and RCTs of exercise.
Modifications will be made and final approval sought from
the expert panel. Results: There were 57/137 respondents in
Round 1 (42%), 10 items were accepted in the original format
and 14 reformatted for Round 2 which had 53/57 respondents
(93%). Sixteen items were distributed for Round 3 which had
49/54 respondents (91%). The final core set was 16 items and
520 comments were thematically analysed. The final template
was modeled according to the CONSORT Statement and
TIDieR Checklist. An Explanation and Elaboration Statement
and implementation strategies such as journal endorsement
for manuscript submission and peer-review and CONSORT
Item 5 extension are planned. The CERT checklist is registered
on the Equator Network. Conclusions: The CERT checklist will
improve explicit reporting of exercise interventions in clinical
trials; increase clinical uptake of effective exercise programs;
enable research replication; improve patient outcomes; and
increase research impact on health.
RO 4.3
Essential items in reporting diagnostic
accuracy studies: 2015 update of the STARD
Statement
Bossuyt PM1 , Korevaar DA1 , Hooft L2 , Cohen JF3 , The STARD
Group &4
1 Department of Clinical Epidemiology,
Biostatistics and
Bioinformatics, AMC, University of Amsterdam, The Netherlands;
2 Dutch Cochrane Centre, Julius Center for Health Sciences and
Primary Care, University of Utrecht, The Netherlands; 3 INSERM UMR
1153, Obstetrical, Perinatal and Pediatric Epidemiology Research
Team, Center for Epidemiology and Statistics Sorbonne Paris Cité,
Paris Descartes University, France; 4 (85 members, International)
Background: Many studies that evaluated the diagnostic
accuracy of one or more medical tests, by comparing test
results against a clinical reference standard, fail to report
key elements of the design and the results. This makes
judgments about risk of bias and applicability of study findings
difficult, and complicates evidence synthesis. In 2003, a group
of researchers, editors, and other stakeholders published
the STARD statement (STAndards for Reporting Diagnostic
accuracy): a list of 25 essential items that should be included in
every report of a diagnostic accuracy study. Evaluations have
revealed modest, but significant, improvements in reporting
since then. Objectives: In 2013, the STARD steering committee
decided to prepare an update. Two considerations were key:
(1) the update should be based on the accumulated evidence
about sources of bias and variability and honest reporting;
(2) the update should make STARD easier to use. Methods:
Original STARD group members were invited to participate,
and to nominate new members. The STARD group now
21
consists of 85 researchers, editors, and other stakeholders.
In web-based surveys, they could respond to and suggest
changes in the list of items. A draft version of the new
list of items was put together during a live meeting of the
steering committee and then piloted. Results: The updated
list consists of 30 items. Compared to the 2003 version, a
few items were added, some were combined, others were
split into separate items, while the remaining items were
sometimes reworded, to make them easier to understand,
and to harmonize wording with other reporting guidelines.
Conclusion: The updated STARD list will be released later
in 2015. As establishing a list of essential items is not
sufficient to improve reporting, the release of the update will
be accompanied by additional initiatives, such as explanatory
documents, lectures, writing templates for authors and for
reviewers. Improving the completeness and transparency of
reporting will help us all to reduce avoidable waste in research,
and strengthen the evidence base for making decisions and
building recommendations about medical tests.
RO 4.4
Quality of reporting of diagnostic accuracy
studies in medical laboratory journals
Pecoraro V1 , Banzi R2 , Trenti T1
1 Ospedale Civile Sant’Agostino Estense,
Modena, Italy;
2
IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Milano,
Italy
Background: The Standards for Reporting of Diagnostic
Accuracy (STARD) checklist aims to improve the reporting of
diagnostic accuracy studies. So far, only a few journals have
adopted the STARD checklist. Objective: We investigated
the current quality of reporting of diagnostic accuracy studies
focusing on laboratory medicine. Methods: We searched
for journals formally adopting the STARD in the ’Medical
laboratory technology’ category of ISI Web of Knowledge
database. As a comparator, we included matching journals
that do not adopt the checklist. We selected studies reporting
an estimate of accuracy (sensitivity, specificity, area under
ROC curve, likelihood ratio, predictive values, diagnostic
odds ratio) published in 2014. For each article, we analyzed
the compliance to STARD items and classified the reporting
as poor (less than 10 items reported), acceptable or optimal
(more than 20 items reported). Results: Only three of 31
journals listed in the field of laboratory medicine mentioned
the STARD checklist in their instructions for authors. We
evaluated 24 studies published in these journals (Clinical
Chemistry; Clinical Chemistry and Laboratory Medicine, and
Annals of Clinical Biochemistry) and seven studies published
in Chimica Clinica Acta which does not adopt the STARD (data
collection is ongoing on other journals). Overall, the median
number of items reported was 11 (range: 5 to 18). None of
the studies complied with the entire checklist. Reporting was
evaluated as poor in 15 of 31 studies (48.4% Table 1). The
most frequently reported items were index test and reference
standards. Papers often lacked an accurate description of
the population enrolled.
Conclusions: The majority of laboratory medicine journals
do not adopt the STARD checklist. Some of the STARD
items may be irrelevant to the studies being evaluated,
however, preliminary data suggest that the use of the STARD
checklist improves the quality of reporting. Final data on
quality of reporting of journals adopting or not adopting the
STARD checklist will be presented. Inadequate reporting of
study participant characteristics may limit the relevance and
application of results in practice.
RO 4.5
Quality of reporting of clinical prediction
model studies: adherence to TRIPOD
Heus P1 , Damen J1 , Scholten R1 , Reitsma J1 , Collins G2 , Altman
D2 , Moons K1 , Hooft L1
1
Dutch Cochrane Centre, Julius Center for Health Sciences and
Primary Care, University Medical Center, Utrecht, The Netherlands;
2
Centre for Statistics in Medicine, NDORMS, Botnar Research Centre,
University of Oxford, United Kingdom
Background: There is a growing number of prediction
models, both diagnostic and prognostic, that are published
in the medical literature. Systematic reviews are required
to deal with this information overload. However, systematic
review authors are highly dependent on the quality of the
reporting of primary studies. To improve the reporting of
prediction models, a guideline for Transparent Reporting
22
of a multivariable prediction model for Individual Prognosis
Or Diagnosis (TRIPOD) was launched in January 2015. The
TRIPOD statement is a checklist of 22 main items considered
essential for good reporting of studies developing or validating
multivariable prediction models. Objectives: The objective
of our study is to assess the quality of reporting of prediction
model studies published before the launch of TRIPOD. In
addition, this study could possibly serve as a baseline
measurement for future studies evaluating the impact of the
introduction of TRIPOD. Methods: For 37 clinical domains we
selected 10 journals with the highest impact factors. A PubMed
search was performed to identify prediction models published
in May 2014. Publications that described the development
and/or validation of a prediction model, either diagnostic
or prognostic, were eligible. TRIPOD items were carefully
translated into a data extraction form, which was piloted
extensively. Consensus was reached on when to consider
an item ’adhered’. Results: Our search identified 4871
references, 347 of which were potentially eligible references
and were assessed in full text. Eventually 180 references
were included. We will present the adherence to TRIPOD
per publication, as well as per item, across studies, and
across clinical domains. Also, diagnostic and prognostic
models will be distinguished, and results will be separated
for studies addressing model development, model validation,
incremental value, or a combination of these. Conclusions:
Our study will provide insight into the current quality of
published reports about the development or validation of
prediction models across a wide variety of clinical domains.
Reporting issues will be identified that might require specific
guidance.
of bias tool, and the proliferation of new avenues to
disseminate SRs. We considered it timely to evaluate a more
recent cross-sectional sample. Objective: To investigate
the epidemiological and reporting characteristics of SRs.
Methods: We searched MEDLINE to identify potentially eligible
SRs indexed in February 2014. Citations were screened using
pre-specified eligibility criteria. We developed a standardised,
pilot-tested data extraction form comprising all items used
in [1], along with additional items to capture issues not
previously examined. Example items include focus of the SR
(e.g. therapeutic, diagnostic), number of included studies, and
reporting of eligibility criteria, search strategy, and statistical
analyses. Data in 300 SRs will be independently extracted
by one of several reviewers, with a 10% random sample
extracted in duplicate. We will compare characteristics of i)
Cochrane versus non-Cochrane reviews, and ii) the current
versus 2004 sample of SRs. Results: We identified 698
SRs indexed in February 2014. This suggests an annual
publication rate of approximately 8300 SRs (equivalent to 23
SRs per day), which is more than triple the estimated rate in
2004. Cochrane Reviews comprised only 14% of the sample.
Reporting characteristics will be presented at the Colloquium.
Conclusion: This study will provide up-to-date data on the
different types of SRs being published (e.g. therapeutic,
diagnostic, prognostic) and the reporting quality of SRs. The
findings will indicate where methodological and reporting
guidance for SRs is most needed, and will inform the SR
methodology research agenda.
Reference: Moher D, Tetzlaff J, Tricco A C, Sampson M, Altman
D G. Epidemiology and reporting characteristics of systematic
reviews. PLoS Medicine 2007: 4(3): e78.
RO 4.6
Epidemiology and reporting characteristics
of systematic reviews: 2014 update
Page MJ1 , Shamseer L2 , Altman DG3 , Tetzlaff J4 , Sampson M5 ,
Tricco AC6 , Catalá-López F7 , Li L8 , Reid E9 , Sarkis-Onofre R10 ,
Moher D2
1
School of Public Health and Preventive Medicine, Monash
University, Australia; 2 Ottawa Hospital Research Institute
and University of Ottawa, Canada; 3 University of Oxford,
United Kingdom; 4 Ottawa Hospital Research Institute, Canada;
5
Children’s Hospital of Eastern Ontario, Canada; 6 Li Ka Shing
Knowledge Institute, St. Michael’s Hospital, Canada; 7 Division
of Pharmacoepidemiology and Pharmacovigilance, Spanish
Medicines and Healthcare Products Agency (AEMPS), Spain; 8 The
First Clinical College of Lanzhou University, China; 9 Department
of Pharmacy, Vancouver General Hospital, Canada; 10 Federal
University of Pelotas, Brazil
Background: Moher et al. [1] examined a November
2004 MEDLINE-indexed sample of 300 systematic reviews
(SRs) and found that reporting quality was variable. The
SR landscape has changed considerably in the subsequent
decade, with the publication of a reporting guideline for SRs
of randomised trials of health care interventions (PRISMA),
methodological developments such as the Cochrane risk
RO 4.7
Challenges and considerations in assessing
the reporting quality of systematic reviews in
overviews using PRISMA
Jia P1 , Chen J2 , Zhang L1 , Zhao P1 , Zhang M1
1
Chinese Cochrane Center, China; 2 West China Hospital, China
Background: Overviews bring systematic reviews (SRs)
together and serve as a user-friendly ’digest’ that can be
used by clinicians and policy makers in making decisions.
Poor reporting of SRs diminishes value of the overview to
clinicians and policy makers. The reporting quality of SRs can
be evaluated using the PRISMA (Preferred Reporting Items for
Systematic reviews and Meta-Analysis) system. Objectives:
To evaluate the reporting quality of SRs involved in overviews;
to examine the difference exiting PRISMA scores make across
Cochrane and non-Cochrane SRs. Methods: We randomly
selected eight overviews and searched all the SRs included
in the overviews. Each PRISMA item is given a score of
1 if the criterion is met, a score of 0.5 if the criterion is
partially met, or a score of 0 if the criterion is not met, or
not applicable. Two reviewers independently evaluated the
quality of included SRs and discrepancies were solved by
23
consensus. Results: Ninety-eight SRs were included; 28 of
these were published in 2009. The 98 SRs were from four major
areas: cancer, hypertension, skin and stroke. Among the 98
SRs, 36 SRs were Cochrane SRs. The total PRISMA scores were
(22.5 ± 10.2). Average Cochrane scores (25.5 ± 11.7) were
much higher than the non-Cochrane reviews (20.5 ± 13.5). The
reporting quality varied in both Cochrane and non-Cochrane
reviews: several items were strong, including reporting of
data collection process (100%), selection of articles (100%)
and synthesis of results (100%). In some items, the reporting
quality of Cochrane SRs was better than the non-Cochrane
SRs, such as: protocol and registration and reporting of search
strategies. Conclusions: The reporting quality of Cochrane
SRs was higher than that of non-Cochrane SRs. The missing
information in the Cochrane and non-Cochrane SRs may
influence the use of overviews, so the reporting quality in the
Cochrane and non-Cochrane SRs needs to be improved.
Online databases for evidence
dissemination
RO 5.1
Alessandro Liberati online Library: a tool to
increase public health professionals’
knowledge and practices
Vecchi S1 , De Fiore L2 , Amato L3 , Davoli M3
1
Department of Epidemiology, Lazio Regional Health Service, Italy;
2
Associazione Alessandro Liberati/Network Italiano Cochrane,
Italy; 3 Department of Epidemiology, Lazio Regional Health Service;
Cochrane Drugs and Alcohol Group, Italy
Background: Public health professionals are increasingly
expected to use research evidence in decision making. Over
the last 10 years, there has been an increasing investment
to identify barriers to incorporate evidence in practice such
as the growing volume of literature, which forces health
professionals to adapt their knowledge and skills to the rapid
development of both medical and scientific research and
technical innovation. Some interventions aimed at addressing
this issue often do not take the real information needs of the
health personnel, nor the quality and reliability of information
sources accessed, into account. Objectives: To evaluate
whether the integration of an open access to evidence source
with a tailored education program improves adherence to
evidence-based clinical practices among health personnel in
the Lazio Region. Methods: A two-phase approach was used:
1. development of a regional online Medical Library (Biblioteca
Alessandro Liberati/BAL) in order to provide an organized
and well-structured bibliographic resource enabling health
professionals quick access to evidence-based information;
2. pilot testing of an educational programme aimed at
empowering health personnel to find the information of
interest, gaining knowledge of databases and search engines
and training them on the critical appraisal of published
studies. Participants have been engaged in interactive
workshops, formal lessons and common discussion on the
appropriate use of evidence through the BAL. An 11-item
questionnaire was developed and administered to explore
EBM attitudes and skills before the intervention. We planned to
administer the questionnaire 12 months after the intervention
to evaluate the improvement in skills for using evidence
sources, ability in searching and retrieval of high quality
information. Results: BAL has been available online since
May 2014. To date, four local health units in the Lazio
Region have been involved in the pilot test. Data collection
and analysis will be complete for presentation at the 2015
Colloquium.
RO 5.2
Patient portals for enhancing patient
empowerment
Lannig S1 , Ammenwerth E1 , Schnell-Inderst P1 , Hörbst A1
1
UMIT Private University for Health Sciences, Medical Informatics
and Technology, Austria
Background: Patient portals are applications that allow
patients to access patient-related health information that
is documented by a healthcare institution electronically.
As part of a patient portal, institutions may offer patients
(typically web-based) access to selected clinical data that
is governed by the institutions as part of the electronic
health record (EHR). The patient can access this clinical
data, read and print it, or integrate it into any (electronic
or paper-based) patient-owned record. Thus, patients are
supported to manage their own health care by using these
patient portals. Patient empowerment, patient satisfaction
and other health-related outcomes may be affected by using
patient portals. Already in 1986, the Ottawa Charter for Health
Promotions had made empowerment a key issue in the theory
of health promotion, but only a few controlled studies on the
impact of patient portals are available at the moment. There
seem to be large uncertainties regarding the impact of patient
portals on patient empowerment. Objectives: To conduct a
systematic review of the effect of patient portals on patient
empowerment as a patient-related outcome. Methods:
We will conduct a systematic literature search on patient
portals. We will search in health-related databases like
MEDLINE, CINAHL, and Academic Search Elite. In addition
we will handsearch for literature in journals of medical and
nursing informatics. Subsequently two independent scientists
will screen all located abstracts. Initially we will include
randomized controlled trials. After that we will assess the
quality of these studies based on the Cochrane ’Risk of bias’
tool. We will summarize the findings related to the impact of
patient portals especially for consumers. Expected results:
We expect to find a small number of patient-related outcomes
addressing the impact of patient portals. Probably, there will
be insufficient data to make a general recommendation for
implementing patient portals, but we may find an impact for
special sub-types such as portals for patients with diabetes.
24
RO 5.3
Cochrane Clinical Answers: filtering the
information overload for better clinical
decisions
Tort S1 , Pettersen K2 , Burch J1
1
Cochrane Editorial Unit, United Kingdom; 2 Wiley, United Kingdom
Background: Healthcare professionals need point-of-care
access to reliable and high-quality synthetized information
from up-to-date research.
Cochrane Reviews provide
a valuable source of information, but can take a long
time to read and have a large volume of data, much
of which may not be relevant to the busy healthcare
professional. Cochrane Clinical Answers (CCAs) provide an
accessible, clinically-focused summary of Cochrane Reviews
and therefore can support decision making at the bed-side.
Objectives: To describe the current and proposed coverage
of CCAs, and how their presentation aids information
dissemination. Methods: The CCA website was developed
to mimic the way clinicians approach information gathering.
To ensure appropriate coverage, the number of published
CCAs by discipline was reviewed and targets established
for 2015. Results: Each CCA addresses a question and
provides a concise answer. Full outcome data supporting
the answer are a click away. The population, intervention
and comparator (PICO) information, a narrative result, the
quality of evidence or risk of bias summary, a link to the
forest plot and absolute values are provided to allow quick
understanding and application of results. By February 2015,
we had published 645 CCAs across all disciplines. Coverage
ranged from 7% (in the areas of Neonatal, and Pregnancy and
Childbirth) to 89% (Metabolic and Endocrine Disorders), with
Oncology and Infectious Diseases being made priorities in
2015. Six-hundred CCAs were published by January 2015, and
our target for January 2016 is 1000. New developments involve
creating CCAs from overviews, network meta-analyses, and
narrative reviews, as clinicians would benefit most from
reliable and accurate clinician-friendly summaries of these
complex Cochrane Reviews. Conclusions: CCAs are a great
tool to filter the vast amount of evidence from Cochrane
Reviews and make it easier for healthcare professionals to
apply high-quality evidence when managing patients.
RO 5.4
Introducing KSR Evidence: developing a
database of critically appraised systematic
reviews in the field of pain relief
Misso K1 , Deshpande S1 , Stirk L1 , Westwood M1 , Kleijnen J1 ,
McLellan A1 , Kleijnen J1
1 Kleijnen Systematic Reviews Ltd, United Kingdom
Background: Publication of systematic reviews (SRs) has
increased dramatically in recent years, from 232 in 1990 to
11,146 in 2014 (PubMed estimate). Assessing quality, risk of
bias and applicability of review findings remains a challenge
for
clinicians and practitioners. KSR Evidence will include critical
appraisals of systematic reviews from 2010 onwards. KSR Pain
Evidence, as part of KSR Evidence, will enable quick and easy
access to current critical appraisals of published SRs in pain
management, and assist in interpretation and application
of findings into clinical practice. Objectives: To develop a
database that comprehensively covers the best systematic
review evidence in pain management, together with related
randomised controlled trials (RCTs) from PubMed. The
database includes critical appraisals for each SR, with a
summary of SR results and a concise clinically-relevant bottom
line. Methods: Sensitive searches were conducted on a
wide range of resources, including MEDLINE, Embase and
the Cochrane Database of Systematic Reviews (CDSR), to
retrieve systematic reviews on pain management from 2010
onwards (see Figure 1). For each included SR, a detailed
critical appraisal and risk of bias was undertaken by an expert
reviewer, culminating in a clinical bottom line. Appraisals
were conducted using an adaptation of the ROBIS (Risk of
Bias In Systematic reviews) tool, and are checked for quality
by a second independent reviewer. Results: To date, KSR
Pain Evidence comprises over 2456 critical appraisals, and
2183 related RCTs from PubMed. We will present KSR Pain
Evidence, describe the rationale behind its development and
the appraisal process. Search functionality and example
appraisals will be demonstrated. Conclusions: Further
work is ongoing to expand database coverage into other
disease areas, and it is anticipated that KSR Evidence will
extend to all published systematic reviews in the near future
(www.ksrevidence.com).
25
RO 5.5
CBRG QuickDecks: a new tool for sharing the
best evidence in back and neck pain care
Irvin E1 , Furlan A1 , Munhall C1 , Danak S1 , Kim J2 , Marin T3
1
Cochrane Back Review Group, Canada; 2 Cancer Care Ontario,
Canada; 3 York University, Canada
Background: It is important to create different summary
products in addition to scientific publications to support
claims about interventions based on evidence from Cochrane
Reviews. The Cochrane Back Review Group (CBRG) wanted
to create a product that would decrease the burden and
yet encourage the use of evidence from Cochrane Reviews
for busy researchers and clinicians. Objectives: To create
a product to share findings from reviews published within
the CBRG that provides a quick snapshot of the evidence in
an accessible format. Methods: QuickDecks are created for
each intervention review and classified into eight different
intervention themes. They consist of three summary slides
with a standard format presenting objectives, methods,
results and conclusions of the review. QuickDecks, prepared
by the CBRG, are sent to the review authors for approval. PDF
versions of the slides are available for anyone to download
and PowerPoint versions are available upon request. Results:
QuickDecks have been promoted through the CBRG Twitter,
Facebook, newsletters and email communications. Since the
QuickDecks have been available to download in August 2014,
there has been a lot of positive feedback and a large number
of downloads. Conclusions: QuickDecks are a new tool used
to share evidence and promote uptake of Cochrane review
findings. The results of this submission may be viewed at:
https://ccnc.cochrane.org/node/605/submission/5975
RO 5.6
IN SUM: mapping the evidence on effects of
child mental health and welfare interventions
Hammerstrøm K1 , Ludvigsen K1 , Axelsdottir B1 , Biedilæ S1 ,
Kornør H1
1
Centre for Child and Adolescent Mental Health, Eastern and
Southern Norway (RBUP), Norway
Background: Systematic reviews (SRs) on the effect
of interventions are important sources of research for
practitioners and other stakeholders in the fields of child
welfare and child mental health. However, finding, appraising
and using research literature may appear both difficult and
time-consuming to practitioners. Objectives: To support
practitioners’ use of research by identifying and quality
appraising all SRs on the effect of interventions in child
welfare and child mental health. Methods: To identify SRs
investigating the effect of any intervention concerning child
(< 18 years) welfare or child mental health we carried out
pilot searches in bibliographic databases (the Cochrane
and Campbell Libraries, PsycINFO, MEDLINE, EMBASE, DARE
etc.) focusing on specific topics of interest (e.g. child
depression and anxiety;). However, the overlaps between
the subject specific searches were considerable. For this
reason, we decided to carry out one broader, exhaustive
search per database, in order to identify all potentially
relevant SRs simultaneously. The searches will be re-run
semi-annually. Inclusion/exclusion: Two researchers will
independently screen identified titles/abstracts and quality
assess included SRs, using AMSTAR (A MeaSurement Tool to
Assess systematic Reviews). Following inclusion, references
will be coded and published in the IN SUM database, which
is under development - but will be accessible, and free, for
all. For selected SRs, we will produce Norwegian/English
plain language summaries. Where we identify synopses of
syntheses (e.g. DARE or Health Evidence summaries) for
included SRs, we will provide a link to the synopsis in the
relevant SR reference. Results: The exhaustive search will be
conducted in March 2015. We estimate that we will complete
the initial selection, appraisal and publication process by
1 July 2015. Conclusions: SRs are important in informing
decisions in child mental health and welfare. We believe that
through the provision of easy access to quality-appraised
relevant SRs and plain language summaries, the IN SUM
database has the potential of becoming a valuable source for
practitioners in our field.
RO 5.7
Building a database of respiratory guidelines:
why, how, and what for?
Welsh E1 , Stovold E1
1
Cochrane Airways Group, United Kingdom
Background: Cochrane Review Groups are encouraged to
demonstrate impact by tracking guidelines that incorporate
their reviews. To help with this we began to compile a
collection of guidelines that cite Cochrane Airways Group
reviews. This grew into the idea of establishing a general
collection of respiratory guidelines, including not only the
guidelines that cite our reviews, but other relevant guidelines
that may be helpful to our review authors and others with
an interest in respiratory medicine. Objectives: To create an
open database of respiratory guidelines. Methods: We began
with a known set of guidelines acquired through general
knowledge of the topic area. We set up a monthly search
alert in PubMed. The search was set up in January 2015,
and run back over 2014. Older guidelines were identified
by checking major respiratory society websites. We use
Zotero Groups to manage the database. It is a public group,
with closed membership. Anyone can view the records,
but membership is required to edit the database. The
database was promoted through social media, and is linked
on the Airways Group website. Results: We structured the
database around disease categories by creating individual
collections. The database of guidelines includes asthma,
wheeze, COPD (chronic obstructive pulmonary disease),
bronchiectasis, obstructive sleep apnoea, chronic cough,
interstitial lung diseases, and pulmonary hypertension. We
also have collections for emergency care and physiotherapy.
26
Conclusions: We established a reference collection of
respiratory guidelines, free for all to view. It has been
compiled through pragmatic methods using a simple search
strategy and is not an exhaustive collection. We would like
to develop this database further, get more people involved in
contributing guidelines, and seek feedback.
RO 5.8
Best evidence for optimal aging: filter the
information overload using the McMaster
Optimal Aging Portal
Dobbins M1 , Watson S2
1
McMaster University, Canada;
University, Canada
2
Health Evidence, McMaster
Background: There is so much health information freely
available on the internet relevant to ’optimal aging’ (staying
healthy, active and engaged as we grow older), that it is
difficult to know which messages are trust-worthy. The
McMaster Optimal Aging Portal does the work for you:
evaluating, summarizing and synthesizing aging research
and resources to provide you with a one-stop shop for
evidence-based information on healthy aging. Objectives:
The McMaster Optimal Aging Portal is a website dedicated
to sharing evidence-based information about optimal aging
and common health conditions with aging adults, their
caregivers and health professionals. Funded by the Labarge
Optimal Aging Initiative, the website was launched in October
2014. Methods: Content is drawn from three best-in-class
resources: MacPlus (a database of best evidence to support
clinical decisions); Health Evidence (an online registry of
quality-appraised systematic reviews evaluating public health
interventions); and Health Systems Evidence (an online
registry of evidence on how to strengthen or reform health
systems). Star-ratings allow users to filter through masses of
health information and research and easily identify the highest
quality resources they need to make evidence-based decisions
around optimal aging. Health professionals can access
abstracts and summaries of the best scientific evidence on
clinical, public health and health policy questions, including
links to full-text articles. Blog posts, web resource ratings and
evidence summaries provide quick and easy-to-read ‘bottom
line’ messages appropriate for all audiences, particularly
citizens. Tailored email alerts and @Mac AgingNews updates
deliver the latest research evidence on headline topics right
into your hands. Results and Conclusions: The McMaster
Optimal Aging Portal gives citizens, health professionals,
policy makers and researchers direct access to trusted,
evidence-based information about optimal aging to help us
remain healthy, active and engaged as long as possible.
RO 5.9
Coming full circle: returning to a bespoke
search approach for clinical evidence
Beaven O1
1 BMJ Group, United Kingdom
Background: Clinical Evidence (CE) is a long running product
from BMJ Group, which has been providing systematic
overviews of the best available research evidence on selected
interventions for numerous medical conditions for over 10
years. During this time CE has developed from print
only, to CD-ROM and on to its current online format and
has had a number of changes in process over this time
period - one such component being the approach taken to
searching the literature. Objectives: To reflect on how
the literature searching function has developed over time to
support the production of CE – specifically its progress from
bespoke searches, to a broad multi-product, condition-level
approach, and currently, to its ongoing return to specific
intervention-based search strategies. Methods: A personal
perspective, considering the advantages and disadvantages
of the searching approaches that have been applied, the
resources that have been utilised, the constraints and
limitations that have been negotiated and how we continue to
strive to get the balance right, in order to provide the quality
of search required to support an evidence-based, regularly
updated, commercial product. Results and Conclusions:
There has been a great deal of change over the lifetime
of CE and different methods have been applied to try to
maintain quality whilst expanding, enhancing and developing
a useful and relevant product for end users. Whilst the results
and conclusions are not yet formulated, I would hope that
the insight into the approaches that we have undertaken
will provide some interesting observations, different ideas,
and perhaps stimulate some discussion amongst other
information professionals about how to deliver pragmatic
searches for continually evolving products.
Individual patient data
RO 6.1
A framework for deciding if individual
participant data are likely to be worthwhile
Tudur Smith C1 , Clarke M1 , Marson T2 , Riley R1 , Stewart L1 ,
Tierney J1 , Vail A3 , Williamson P1
1
IPD Meta-analysis Methods Group, United Kingdom; 2 Cochrane
Epilepsy Group, United Kingdom; 3 Cochrane Menstrual Disorders &
Subfertility Review Group, United Kingdom
Background: Systematic reviews and meta-analysis that
are based on Individual Participant Data (IPD) are often
referred to as a ’gold standard’. However, they are usually
27
more resource-intensive than a traditional approach based
on aggregate data (AD) extracted from publications or made
available from authors. The advantages and disadvantages
of the IPD approach have been discussed extensively in the
literature. In many cases the decision about whether to
collect IPD can be straightforward. For example, if suitable
AD are not available in the trial publications, some data
will be needed from the original researchers if the trial is
to be included in any meta-analyses. However, in some
cases it can be difficult for a researcher, or research funder,
to decide whether the extra investment of time and money
will be worthwhile. Further guidance would be helpful to
assist with this decision making. Objectives: To provide
a framework for researchers, and research funders, to help
decide whether collecting IPD would be worthwhile. Methods:
Existing empirical evidence comparing IPD and AD, literature
describing the advantages and disadvantages of IPD, and
the expertise of the Cochrane IPD Meta-analysis Methods
Group will be used to develop an IPD decision framework.
Results: The framework will be described and made available
at the Colloquium. Conclusions: The framework will help
researchers and funders determine when IPD is likely to be
most beneficial. This could avoid resources being wasted
on unnecessary IPD reviews, help to achieve the maximum
potential of IPD, and help to prioritise research projects that
would benefit most from IPD.
RO 6.2
Imputation of systematically missing
predictors in an individual participant data
meta-analysis: a generalized approach using
MICE
Debray T1 , Jolani S2 , Koffijberg H1 , van Buuren S2 , Moons K1
1 Julius Center for Health Sciences and Primary Care, The
Netherlands; 2 Utrecht University, The Netherlands
Background: Individual participant data meta-analyses
(IPD-MA) are increasingly used for developing and validating
multivariable (diagnostic or prognostic) risk prediction
models. Unfortunately, some predictors or even outcomes
may not have been measured in each study and are
thus systematically missing in some individual studies of
the IPD-MA. As a consequence, it is no longer possible
to evaluate between-study heterogeneity and to estimate
study-specific predictor effects, or to include all individual
studies, which severely hampers the development and
validation of prediction models. Objectives: To describe
a novel approach for imputing systematically missing data,
which adopts a generalized linear mixed model to allow for
between-study heterogeneity. Methods: We illustrate our
approach using a case study with IPD-MA of 13 studies to
develop and validate a diagnostic prediction model for the
presence of deep venous thrombosis. We compare the results
after applying four methods for dealing with systematically
missing predictors in one or more individual studies: complete
case analysis (CCA) where studies with systematically missing
predictors are removed, traditional multiple imputation
ignoring heterogeneity across studies (TMI), stratified
multiple imputation accounting for heterogeneity in predictor
prevalence (SMI), and multilevel multiple imputation (MLMI)
fully accounting for between-study heterogeneity. Results:
Results from CCA were suboptimal and became completely
unreliable when the predictors were no longer missing
completely at random. TMI and SMI tended to mask the
actual degree of between-study heterogeneity and often lead
to overoptimistic standard errors of predictor effects. MLMI
was the optimal approach in terms of coverage and bias,
and the only approach that was able to ensure compatibility
of imputation and analysis models. Conclusions: MLMI
may substantially improve the estimation of between-study
heterogeneity parameters and allow for imputation of
systematically missing predictors in IPD-MA aimed at the
development and validation of prediction models.
RO 6.3
The advantages of individual participant data
(IPD) in smaller-scale systematic reviews
Scott N1 , Spiteri Cornish K2 , Lois N3
1 Medical Statistics Team, University of Aberdeen, United Kingdom;
2
Ophthalmology Department, NHS Grampian, Aberdeen, United
Kingdom; 3 Centre for Experimental Medicine (CEM), Queen’s
University, Belfast, United Kingdom
Background: Individual participant data (IPD) have been
used in many systematic reviews of randomised controlled
trials. Commonly-cited benefits include the ability to perform
meta-analyses of time-to-event outcomes, the ability to
undertake subgroup analyses and to ensure data quality.
Many published IPD reviews have involved large collaborative
groups and large numbers of studies. We report the unique
benefits of IPD in a small-scale review in ophthalmology.
Methods: Systematic review of studies comparing internal
limiting membrane (ILM) peeling during macular hole surgery
versus no peeling. Results: IPD were received from the three
identified fully-published studies. The authors of a fourth
study were not willing to provide data until full publication
but limited data were available from an abstract. The review
was able to answer key questions about the effectiveness of
ILM peeling. Benefits of IPD in this review included the ability
to standardise the unit of randomisation (eye or patient), to
include only randomised participants in one study with an
unusual format, to explore timing of assessments, to include
information from both IPD and published data, to conduct
subgroup analyses by stage of macular hole and to conduct
adjusted analyses controlling for study, size and duration
of the macular hole. Conclusions: This was a successful
collaboration and a review of published data alone would have
been very difficult. The benefits, however, were very different
to those in a previous large-scale IPD review conducted by
the primary author. In particular, the ability to standardise
the unit of randomisation was crucial to the success of this
review. This project demonstrates that there may be a variety
of reasons to seek IPD, even in smaller systematic reviews.
28
RO 6.4
Development of a unified database based on
CDISC for individual patient data (IPD)
analysis: practices in reproductive health
survey
Zhang Y1 , Fei Y1 , Li X1 , Liu J1
1
Center for Evidence Based Chinese Medicine, Beijing University of
Chinese Medicine, China
Background: IPD meta-analysis is suggested as the least
biased method, which might provide more detailed and
reliable conclusions than conventional meta-analysis. Since
the original data from individual studies can be re-categorized,
IPD meta-analysis can provide more consistent results. Before
initiating an IPD analysis, establishing comprehensive data
standards is essential to the development of a database
that enables the pooling of data from different sources.
The Clinical Data Interchange Standards Consortium (CDISC)
standards are preferred by regulators, industry, and other
research organizations as a means of facilitating regulatory
review, aggregation, and querying of data, sharing data
between entities, and streamlining the acquisition and
analysis of data. Objectives: We aimed to develop a unified
database to standardize and collect data from various sources.
Methods: Study Data Tabulation Model (SDTM) standard is
a foundational dataset for data management procedures.
SDTM is suited for collecting data of various types and
storing it in a relatively small number of observation classes.
We examined the data structures of three epidemiology
investigations, and then developed new data domains for
storing reproductive-specific data. With the standards
in place, patient-level data from the three studies were
remapped and used to construct the database. Logical
mapping and programmatic transformation were applied in
the whole process. Validating and reconciling were made
following related standard operating procedures. Results:
To date, the unified database contains six domains (DM,
SC, CM, MH, VS, QS) and has received data on a total of
23600 subjects from three remapped studies of reproductive
health submitted by two researchers in the National Research
Institute for Family Planning. Conclusions: Developing
data standards is essential for IPD meta-analysis. A unified
database would facilitate the pooling of data from legacy
studies. In addition, it is also a data reservoir for prospectively
collecting data in new trials without the need for remapping.
RO 6.5
Methods and characteristics of published
network meta-analyses using individual
patient data
Veroniki AA1 , Soobiah C1 , Tricco A1 , Elliott M1 , Straus SE1
1 Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto,
Canada
Background: Individual patient data (IPD) may increase
precision in network meta-analysis (NMA) compared to using
aggregated data (AD), as it utilizes both within-study and
across-study data. The use of IPD in NMA also prevents
misleading inferences due to bias, such as aggregation
bias, and allows for the identification of interactions due
to patient-level effect modifiers, that cannot be detected
using AD. While applications of IPD-NMA methods have been
published, a scoping review presenting all approaches to
applying an IPD-NMA has not been conducted. Objectives:
To synthesize existing IPD-NMA methods, summarize their
properties, and describe the characteristics of published
IPD-NMAs. Methods: We searched relevant electronic
databases from inception until September 2014 and for grey
literature through Google. Published and unpublished studies
were eligible, including studies suggesting and/or applying
IPD-NMA methods or studies addressing the methodological
quality of IPD-NMA. We included IPD-NMAs of all quantitative
study designs. Several approaches have been presented in
the literature for modeling IPD alone or in combination with
AD in NMA, which vary in popularity and complexity. We will
provide guidance for selecting an appropriate IPD-NMA model,
identify gaps where methodology is lacking, and highlight the
potential for novel statistical advances which are necessary
to evaluate the key assumptions in NMA. We will present
the characteristics of previously conducted IPD-NMAs, and
the methods that have been applied. Quantitative data (e.g.
number of patients) will be summarized by medians and
interquartile ranges, while categorical data (e.g. effect size)
will be summarized by numbers and percentages. Results:
The results will be ready by September 2015 and will be
presented at the Cochrane Colloquium. Discussion: Our
study will provide a comprehensive overview of the methods
for completing an IPD-NMA, as well as insight into the
characteristics of IPD-NMAs that have been conducted.
RO 6.6
Complex interventions and composite
outcomes in Individual Participant Data (IPD)
meta-analysis: double challenge in i-WIP IPD
Rogozinska E1 , Marlin N2 , Thangaratinam S1
1
Women’s Health Research Unit, Queen Mary University of London,
United Kingdom; 2 Pragmatic Clinical Trials Unit, Queen Mary
University of London, United Kingdom
Background: i-WIP (International Weight management In
Pregnancy) is an IPD meta-analysis of randomised controlled
trials looking at the differential effect of diet and physical
activity based interventions on maternal and fetal outcomes
in pregnancy. The interventions used in the studies eligible
for the IPD involve a range of weight management methods
aiming to improve diet, level of physical activity and lifestyle.
The interventions were delivered with various frequencies,
for different lengths of time and formats (group sessions
and individual consultations). Additionally the effect of
interventions was evaluated using a wide range of maternal,
29
fetal and neonatal outcomes. Objective: To discuss the
solutions adopted in i-WIP project to deal with a double
challenge of complexity of evaluated interventions with a
multitude of clinical outcomes. Complex interventions:
Building on the work of Hill et al. (2013), we developed
an 18-item questionnaire (Table 1) breaking down the
intervention into standard components. We asked the trials’
Principal Investigators (PIs) to fill out the questionnaire and
specify which of the items were present in their interventions.
Outcomes: We conducted a two-stage Delphi survey to
prioritize the outcomes for their importance to clinical care.
Secondly, using a validated method we developed maternal
composite, and fetal and neonatal composite outcomes
(Table 2). Discussion: The IPD meta-analysis gives the
researchers greater power than the aggregated one, however
it comes with certain challenges. If our abstract is accepted,
we will provide a detailed description of the solutions
applied in the i-WIP IPD; their strengths, limitations and
recommendation for future research.
Reference: Hill B., Skouteris H. and Fuller-Tyszkiewicz.
Interventions designed to limit gestational weight gain: a
systematic review of theory and meta-analysis of intervention
components. Obesity Reviews 2013; 435–450.
30
RO 6.7
Selective cutoff reporting in diagnostic test
accuracy studies of depression screening
tools: comparing traditional meta-analysis to
individual patient data meta-analysis
Levis B1 , Benedetti A1 , Thombs B1
1
McGill University, Canada
Background: Selective outcome reporting in clinical
trials is well understood, but has not been assessed
systematically in studies of diagnostic test accuracy, where
authors often report results for a small range of cutoffs
around data-driven ’optimal’ cutoffs maximizing sensitivity
and specificity.
Objectives: To compare traditional
meta-analysis of published results to individual patient
data (IPD) meta-analysis of results from all cutoffs, to:
(1) assess the degree to which selective cutoff reporting
exaggerates accuracy estimates; and (2) identify patterns of
selective cutoff reporting. Methods: Bivariate random-effects
models were used to compare results of traditional and
IPD meta-analysis, using studies included in a published
meta-analysis of the Patient Health Questionnaire-9 (PHQ-9)
depression-screening tool (Manea, CMAJ, 2012). Results:
Thirteen of 16 primary datasets were obtained. For the
’standard’ cutoff of 10, most studies (11 of 13) published
accuracy results. For all other cutoffs, only three to six of
the 13 studies published accuracy results. For all cutoffs,
specificity estimates in traditional and IPD meta-analyses
were within 2%. Sensitivity estimates were similar for cutoff
10, but differed by 5% to 15% for all other cutoffs. In
samples where the PHQ-9 was poorly sensitive, authors
reported results for cutoffs around the low optimal cutoff.
In samples where the PHQ-9 was highly sensitive, authors
reported results for cutoffs around the high optimal cutoff.
Consequently, in the traditional meta-analysis (but not in
the IPD meta-analysis), sensitivity increased as cutoff severity
increased for part of the range of possible cutoffs. Comparing
cutoff 10 across all studies, sensitivity was heterogeneous
(tau-squared = 1.95). Comparing optimal cutoffs, however,
sensitivity was more homogeneous (tau-squared = 0.68), but
cutoff values ranged from 5 to 15. Conclusion: Selectively
reporting well-performing cutoffs in small samples leads to
biased estimation of accuracy in traditional meta-analyses. To
reduce bias in meta-analyses, primary studies should report
accuracy results for all cutoffs.
Reference: Manea L, Gilbody S, McMillan D. Optimal cut-off
score for diagnosing depression with the Patient Health
Questionnaire (PHQ-9): a meta-analysis. CMAJ 2012; 184:
E191–6.
Lunch rapid oral session 1
Minimally important difference
LRO 1.1
Minimally important difference estimates for
patient reported outcomes in pediatrics: a
systematic survey
Ebrahim S1 , Nesrallah G2 , Vercammen K3 , Sivanand A3 ,
Carrasco-Labra A3 , Furukawa T4 , Guyatt G3 , Johnston B5
1
McMaster University; Hospital for Sick Children, Canada; 2 Li
Ka Shing Knowledge Institute, St. Michael’s Hospital, Canada;
3 McMaster University, Canada; 4 Kyoto University, Canada;
5
Hospital for Sick Children, Canada
Background: The minimally important difference (MID)
provides a measure of the smallest change in a patient
reported outcome (PRO) instrument that patients perceive as
important. An anchor-based approach is widely considered
the most appropriate method for MID determination. No study
or database currently exists that documents and appraises
all anchor-based MIDs associated with PRO instruments in
children. Objectives: To complete a systematic survey of
the literature to collect, characterize and appraise published
anchor-based MIDs associated with PRO instruments used in
evaluating the effects of interventions on chronic medical and
psychiatric conditions in children. Methods: We searched
MEDLINE, EMBASE, and PsycINFO (1989 to 11 February
2015) to identify studies reporting empirical ascertainment
of anchor-based MIDs in pediatric PROs. Teams of two
reviewers screened titles and abstracts, reviewed full texts of
citations identified as potentially eligible, extracted relevant
data, and assessed the credibility of all studies. Results:
Of 3910 citations, 706 were reviewed in full text; 32 proved
eligible. Anchor-based MID determinations were assessed in
23 (72%) disease-specific PROs and 10 (28%) generic PROs
(one study reported both). Fourteen PROs and anchors were
self-reported, 10 PROs and 13 anchors were proxy-reported
(parent or guardian), eight PROs and three anchors were both
self- and proxy-reported, and reporting of two anchors was
unclear. Anchor-based approaches consisted of a standard
approach (range of scores on an independent anchor that
was classified as ’minimally important change’) in 27 studies,
three used Receiver Operating Characteristic curves, and two
used an approach based on hypothetical scenarios. Detailed
results and the credibility assessments of the anchor-based
approaches will be presented at the Cochrane Colloquium.
Conclusions: We have compiled the first comprehensive
synthesis of anchor-based MIDs for PROs in children. This
compendium will help clinical trialists in child health improve
sample size calculations, and will enhance the interpretability
of treatment effects in systematic reviews and practice
guidelines.
31
LRO 1.2
Minimally important difference estimates
and assessment of its credibility for patient
reported outcomes in adults: a systematic
survey
Carrasco- Labra A1 , Johnston BC2 , Ebrahim S2 , Furukawa TA3 ,
Patrick DL4 , Hemmelgarn BR5 , Motaghi Pisheh S1 , Dennis B1 ,
Jacobs M6 , Vernooij RW7 , Fei Y8 , Younho Hong B1 , Tajika A3 ,
Takeshima N3 , Iwakami N3 , Hayasaka Y3 , Kaminski-Hartenthaler
A9 , Nussbaumer B9 , Colunga L10 , Gartlehner G9 , Schunemann
HJ1 , Guyatt G1 , Nesrallah G11
1
Department of Clinical Epidemiology & Biostatistics, McMaster
University, Canada; 2 Department of Anaesthesia and Pain
Medicine, The Hospital for Sick Children; Department of
Clinical Epidemiology & Biostatistics, McMaster University,
Canada; 3 Department of Health Promotion and Human Behavior,
School of Public Health, Kyoto University Graduate School of
Medicine, Japan; 4 Seattle Quality of Life Group, Department
of Health Services, University of Washington, USA; 5 Faculty of
Medicine, University of Calgary, Canada; 6 Department of Medical
Psychology J3-220-1, Academic Medical Center, The Netherlands;
7
Iberoamerican Cochrane Centre, Institute of Biomedical Research
(IIB Sant Pau), Spain; 8 Centre for Evidence-Based Chinese
Medicine, Beijing University of Chinese Medicine, Department
of Clinical Epidemiology & Biostatistics, McMaster University,
China; 9 Department for Evidence-Based Medicine and Clinical
Epidemiology, Danube University, Krems, Austria; 10 Hospital Civil
de Guadalajara, Fray Antonio Alcalde, Mexico; 11 Nephrology
Program, Humber River Regional Hospital; Division of Nephrology,
University of Western Ontario, Canada
Background: Patient-reported outcomes (PROs) have the
function of reflecting patients’ perspectives on treatment
benefits and harms. The ability to interpret and, consequently,
include PROs in the decision-making process along with other
patient-important outcomes relies on the understanding
of what improvement or reduction in the PRO represents
a change that is considered important to patients. The
most common reference point for interpretation of PROs is
the minimal important difference (MID), which provides a
measure of the smallest change in a PRO that patients would
perceive as an important benefit or harm. Objective: To
document published anchor-based MIDs associated with PRO
instruments used in evaluating the effects of interventions on
chronic medical and psychiatric conditions in both adult
and pediatric populations. Methods: We are currently
launching a study that summarizes and appraises available
methods to estimate anchor-based MIDs. We will also
document all anchor-based MIDs reported for adults and
pediatric populations and will develop a rating instrument
to assess their credibility. We will create an inventory of
available anchor-based MIDs, including the context in which
they were assessed (condition/disease), and the confidence
users can place in a particular MID. We will search MEDLINE,
EMBASE, PsycINFO, and CINAHL (1989 to present) to identify
studies addressing methods to estimate anchor-based MIDs
of PRO instruments or reporting empirical ascertainment
of anchor-based MIDs.
Teams of two reviewers will
independently screen citations identified, and extract relevant
data.
Discussion: No inventory of MIDs is currently
available, which means that clinicians and patients have
to navigate the vast literature in order to retrieve a specific
MID. Even if they find an apparently applicable MID, there
is no guidance regarding the extent to which users can
trust the reported MID. We will provide a summary of the
available MIDs in the literature and describe their overall
credibility.
LRO 1.3
Though not consistently superior, the
absolute approach to framing the minimally
important difference has advantages over the
relative approach
Zhang Y1 , Zhang S1 , Thabane L1 , Furukawa TA2 , Johnston BC3 ,
Guyatt GH4
1
University, Canada; 2 Department of Health Promotion and Human
Behavior, Kyoto University School of Public Health, Kyoto, Japan;
3
Department of Anaesthesia and Pain Medicine, The Hospital for
Sick Children, University of Toronto; Institute of Health Policy,
Canada; 4 Department of Clinical Epidemiology and Biostatistics,
and Medicine, McMaster University, Canada
Background: Whether the minimal important difference
(MID) is best framed using an absolute change or a
relative change in score when interpreting patient-reported
outcomes remains uncertain. Objectives: Using studies
that established MID using anchor-based methods, we set
out to address the relative merits of absolute and relative
changes in establishing an instrument’s MID. Methods: In
seven data sets, we calculated correlations between global
change ratings and absolute and relative score changes
and conducted meta-analyses. We considered that the
measure with the higher correlation represented the more
valid approach.
Results: The meta-analyses showed
no significant difference between pooled correlations of
absolute and relative difference on a health-related quality
of life instrument with global transition scores of symptoms,
emotional function, physical function, and cognitive function.
In four of five domains there was at least one study in which
the absolute was significantly superior to the relative; in
one of these four, one study showed statistically significant
superior performance of the relative. In an analysis restricted
to patients with low baseline scores for the domain of
cognitive function, the relative approach showed higher
correlation with global rating than did the absolute approach.
Conclusions: Although we found no consistent superiority
of either approach to establishing the MID, when differences
existed they usually favoured the absolute, which also has
advantages of simplicity and ease of pooling across studies.
Researchers may consider the absolute as a default, but also
compare both methods on an instrument by instrument basis.
32
LRO 1.4
A new tool to measure credibility of studies
determining minimally important difference
estimates for patient reported outcomes
Carrasco- Labra A1 , Johnston BC2 , Ebrahim S2 , Furukawa TA3 ,
Patrick DL4 , Hemmelgarn BR5 , Motaghi Pisheh S1 , Dennis B1 ,
Jacobs M6 , Vernooij RW7 , Fei Y8 , Younho Hong B1 , Tajika A9 ,
Takeshima N9 , Iwakami N9 , Hayasaka Y9 , Kaminski-Hartenthaler
A10 , Nussbaumer B10 , Colunga L11 , Gartlehner G10 , Schunemann
HJ1 , Nesrallah G12 , Guyatt GH1
1
Department of Clinical Epidemiology & Biostatistics, McMaster
University, Canada; 2 Department of Anaesthesia and Pain
Medicine, The Hospital for Sick Children; Department of
Clinical Epidemiology & Biostatistics, McMaster University,
Canada; 3 Department of Health Promotion and Human Behavior,
School of Public Health, Kyoto University Graduate School of
Medicine, Japan; 4 Seattle Quality of Life Group, Department
of Health Services, University of Washington, USA; 5 Faculty of
Medicine, University of Calgary, Canada; 6 Department of Medical
Psychology J3-220-1, Academic Medical Center, The Netherlands;
7
Iberoamerican Cochrane Centre, Institute of Biomedical Research
(IIB Sant Pau), Spain; 8 Centre for Evidence-Based Chinese
Medicine, Beijing University of Chinese Medicine, Department of
Clinical Epidemiology & Biostatistics, McMaster University, China;
9 Department of Health Promotion and Human Behavior, Kyoto
University Graduate School of Medicine/School of Public Health,
Japan; 10 Department for Evidence-based Medicine and Clinical
Epidemiology, Danube University Krems, Austria; 11 Hospital Civil de
Guadalajara, Fray Antonio Alcalde, Mexico; 12 Nephrology Program,
Humber River Regional Hospital; Division of Nephrology, University
of Western Ontario, Canada
Background: The ability to interpret and, consequently,
include patient-reported outcomes (PROs) in the
decision-making process along with other patient-important
outcomes relies on the understanding of what improvement
or reduction in the PRO represents a change that is important
to patients. With such understanding, treatment effects
can be understood in terms of their magnitude as large,
moderate, small but still important, or negligible. The most
common reference point for the interpretation of PROs is
the minimal important difference (MID), which provides a
measure of the smallest change in a PRO that patients would
perceive as an important benefit or harm. Although MIDs
are available for many PROs, users of these MIDs have no
guidance on how to determine how credible these estimates
are. Objective: To develop an instrument to measure the
credibility of MIDs estimates studies. Methods: We are
currently conducting a systematic survey that summarizes and
appraises available methods to estimate anchor-based MIDs.
We searched MEDLINE, EMBASE, PsycINFO, and CINAHL (1989
to present) to identify studies addressing methods to estimate
anchor-based MIDs of PRO instruments or reporting empirical
ascertainment of anchor-based MIDs. Teams of two reviewers
will independently screen the citations identified, and extract
relevant data. We will summarize the available methods
and develop a new instrument addressing the credibility of
empirically ascertained MIDs. We will user-test and pilot our
tool, and assess its inter-rater reliability. Discussion: Our new
instrument will allow users to determine the extent to which
the design and conduct of studies measuring MIDs is likely
to have protected against misleading estimates. In addition,
researchers interested in generating these estimates can learn
about key aspects that need to be considered in MID study
designs to minimize bias. We will present the methods and
final draft of the instrument.
LRO 1.5
Meta-analysis of mean differences in minimal
important difference units: application with
appropriate variance calculations
Shrier I1 , Christensen R2 , Juhl C3 , Beyene J4
1 Centre for Clinical Epidemiology, Lady Davis Institute, McGill
University, Canada; 2 Musculoskeletal Statistics Unit, The Parker
Institute, Bispebjerg and Frederiksberg Hospital, Denmark;
3 Research Unit for Musculoskeletal Function and Physiotherapy,
University of Southern Denmark, Denmark; 4 Department of
Mathematics and Statistics, McMaster University, Canada
Background: Practicing evidence-based medicine requires
succinctly summarized data, preferably in a meta-analysis if
the data are appropriate. For continuous outcomes using
different measurement instruments, systematic reviewers
may calculate the standardized mean difference (SMD), ratio
of means (RoM), or a more recent method based on mean
difference between groups expressed in minimal important
differences (MD-mid) units, where the MID is considered a
constant. Although standardization in MID units is easily
interpretable clinically, considering MID as a constant imposes
important limitations. Objectives: Our objective is to
illustrate how considering the MID as a random variable,
with a distribution, provides solutions to these limitations.
Methods: We calculated the variance of MD-mid using the
delta method. Using sensitivity analyses, we compare results
using the two methods of calculating the MD-mid variance.
Results: Considering the MD-mid as a random variable instead
of a constant, (1) enables investigators to obtain estimates for
questionnaires with no previous MID, and (2) makes underlying
assumptions more transparent. Furthermore., considering
MD-mid as a random variable allows investigators to avoid
making the unrealistic assumptions that (1) the coefficient of
variation for MID is independent of the measure, and (2) there
is no correlation between the MID and mean difference. Using
sensitivity analyses for different assumptions, we illustrate
that the variance of MD-mid calculated when MID is considered
a constant instead of a random variable can be under or
over-estimated to significant degrees. We explore the effects
on two different datasets, (1) data originally used to present
the MD-mid, and (2) data from osteoarthritis studies using
different pain scales and disability scales. Conclusions:
Considering the MID as a random variable instead of a
constant makes underlying assumptions more transparent
and accounts more appropriately for the true variance.
33
LRO 1.6
Communicating the magnitude of treatment
effects: a survey of clinicians in eight
countries
Johnston B1 , Alonso-Coello P2 , Friedrich J3 , Mustafa R4 , Tikkinen
K5 , Neumann I6 , Vandvik P7 , Akl E8 , da Costa B9 , Adhikari N3 , Mas
Dalmau G2 , Kosunen E10 , Mustonen J10 , Crawford M11 , Thabane
L12 , Guyatt G12
1 The Hospital for Sick Children Research Institute, University
of Toronto, Canada; 2 Iberoamerican Cochrane Center, Spain;
3
University of Toronto, Canada; 4 University of Missouri-Kansas
City, USA; 5 University of Helsinki, Finland; 6 Universidad Catolica
de Chile, Chile; 7 University of Oslo, Norway; 8 American University
of Beirut, Lebanon; 9 University of Bern, Switzerland; 10 University
of Tampere, Finland; 11 The Hospital for Sick Children Research
Institute, Canada; 12 McMaster University, Canada
Background:
Meta-analyses of continuous outcomes
typically provide enough information for decision makers
to evaluate the extent to which chance can explain apparent
differences between interventions. The interpretation of
the magnitude of these differences - from trivial to large can, however, be challenging. Objectives: To investigate
clinicians’ understanding and perceptions of usefulness of
six statistical formats for presenting continuous outcomes
from meta-analyses: standardized mean difference (SMD),
minimal important difference units (MID), mean difference
(MD), ratio of means (RoM), relative risk (RR), and risk
difference (RD). Methods: We invited 610 staff and trainees in
family medicine and internal medicine programs in eight
countries (Europe, North and South America, and the
Middle East) to participate. Paper-based, self-administered
questionnaires presented summary estimates of hypothetical
interventions versus placebo for chronic pain. The estimates
were either for a small or large effect for each of the six
presentation approaches. Questions addressed participants’
understanding of the magnitude of treatment effects and
their perception of usefulness of the format. We randomized
participants to size of effect and order of presentation of
the different formats. Results: 531 (87%) of clinicians
responded (Table 1). Clinicians best understood RD, followed
by the RR and RoM (Figure 1). Similarly, clinicians perceived
34
35
dichotomous presentation of continuous outcomes (RD; RR)
most useful (Figure 2). Compared to family medicine,
internal medicine had better understanding and reported
higher perceived usefulness of the various presentation
formats (P value < 0.005; Figure 3). Conclusions: No
format was very well understood or perceived as extremely
useful. Clinicians best understood continuous outcomes
when presented as dichotomies (relative and absolute risk
differences) and perceived these presentations to be the most
useful. Presenting results as SMD, the longest standing and
most widely used approach, was poorly understood and
perceived as least useful. Further efforts in medical education
regarding understanding research results are necessary.
Priorisation
LRO 2.1
The UK top 10 clinical research priorities in
neuro-oncology
Bulbeck H1 , Grant R2 , Oliver K3 , Day J4 , Macdonald L5 , Morley R6
1
Brainstrust, United Kingdom; 2 Department of Clinical Sciences,
Western General Hospital, Edinburgh, United Kingdom;
3
International Brain Tumour Alliance, United Kingdom; 4 NHS
Lothian, United Kingdom; 5 Cochrane Collaboration, United
Kingdom; 6 James Lind Alliance, United Kingdom
Background: The James Lind Alliance (JLA) supports the
National Institute of Health Research (NIHR) to develop
research evidence. Many clinical questions about brain
and spinal cord tumours remain unanswered –therefore
patients, carers and clinicians should work collaboratively
to establish the most important research priorities. A
Neuro-Oncology Priority Setting Partnership was established
following JLA guidelines to identify the top 10 uncertainties
in diagnosis, treatment and management of primary
central nervous system (CNS) tumours. Objectives: The
primary objective of the Neuro-Oncology PSP was to
identify the top 10 treatment uncertainties in diagnosis
and management of primary CNS tumours. A secondary
objective was to highlight the necessity of a brain tumour
community collaborative approach to priority setting in
clinical research. Methods: Stakeholder representatives
from a cross-section of the UK brain tumour community
were recruited. The scope was clinical interventions for
primary brain or spinal cord tumours. The first public
survey was widely publicised to patient forums, professional
organisations and charities. Questions were categorised
and standardised into PICO (participants, interventions,
comparisons, outcomes) format. A systematic literature
search was performed to ensure the questions were genuine
’uncertainties’. Initial prioritisation by stakeholders was
followed by a second public survey. A final workshop
determined the top 10 questions using a modified Delphi
and Nominal Group technique. Results: Over 600 individual
questions were generated from the initial survey, a patient
forum and the UK Database of Uncertainties about the Effects
of Treatment (DUETs). After analysis and selection, these
reduced to 44 PICO questions. The second public survey
reduced these further. The last 25 questions were discussed
at a full-day stakeholders’ workshop. The final top 10 priorities
were agreed. Conclusions: The brain tumour community has
worked collaboratively to identify the top 10 priorities for
clinical research. These will be promoted to governmental,
neuroscience and charity funders throughout 2015/16.
LRO 2.2
Setting priorities for Cochrane Reviews in
health communication and participation
Synnot A1 , O’Sullivan M2 , Hill S2
1 Consumers and Communication and National Trauma Research
Institute, Australia; 2 Consumers and Communication Review
Group, Australia
Background:
Increasingly, the research community,
including Cochrane, is looking to include consumers, carers,
health professionals and policy makers to work in partnership
with researchers to set research priorities. The Cochrane
Consumers and Communication Review Group (CC&CRG) is
undertaking a research priority-setting project to identify new
topics for Cochrane Reviews (CRs) within our scope. To
our knowledge, this will be the first project of this nature
to have such a broad scope encompassing many areas of
communication and participation in health. Objectives:
We have two objectives: (1) to identify Australian and
international priority topics for systematic reviews in health
communication and participation; and (2), to identify the
top five Australian priorities for new CRs within the scope of
the CC&CRG. Methods: Drawing upon existing priority-setting
frameworks, we will undertake a multi-stage project involving
an online survey, a face-to-face workshop and evidence
mapping. We will involve key stakeholders (consumers, carers,
health professionals, policy makers and researchers) from
Australia and internationally, in every stage and the results
will be widely disseminated. Results: A multidisciplinary
steering group has been convened to oversee the project and
will be responsible for defining the project scope, establishing
criteria for identifying priorities and providing guidance over
the course of the project. Strategic partnerships with local
stakeholders are being sought to enhance dissemination and
translation opportunities for new CR topics and related
outcomes.
Subsequent project stages (online survey,
face-to-face workshop) will be completed by October 2015.
Conclusions: Upon completion, we will have up to five new
topics for CRs. In 2016, we will seek author teams to undertake
these high-priority reviews. Novel consumer co-production
models may be trialled as part of the review commissioning
process.
36
LRO 2.3
LRO 2.4
Prevalence of insurance medicine related
outcomes in update reviews of Cochrane’s
2015 priority list
Use of framework synthesis to prioritise
systematic review topics among multiple
stakeholders: a descriptive case study
Kunz R1 , Vogel N1 , deBoer W1 , von Allmen D1
1 Swiss Academy of Insurance Medicine, University Hospital Basel,
Switzerland
Brunton G1 , Llewellyn A2 , Sutcliffe K1 , Rees R1 , Oliver S1 , Caird J1 ,
Stokes G1 , Thomas J1
1
EPPI-Centre, UCL IOE London, United Kingdom; 2 Centre for
Reviews & Dissemination, University of York, United Kingdom
Background: Cochrane Insurance Medicine has recently
registered as a Field with Cochrane, and went public in
March 2015. Insurance medicine deals with health-related
issues in the context of insurance coverage, in particular
sick leave, short- and long-term disability, and interventions
to facilitate return to work. While almost all medical
disciplines benefit from outcome research and their
synthesis in the form of systematic reviews, insurance
medicine has missed out this development.
Equally,
many healthcare areas ignore the importance of outcomes
related to sick leave and short-and long-term disability,
and, thereby, neglected important consequences of health
care to patients and society. Objectives: To explore
the prevalence of insurance medicine related outcomes
in updated reviews on Cochrane’s 2015 priority list
(editorial-unit.cochrane.org/cochrane-priority-review-list2015-16). Methods: We used a cohort of systemic reviews
(SR) from the Cochrane priority list for updates: Population:
SRs on adults of working age with health conditions that
commonly result in temporary or permanent inability to work
(sick leave, short- and long-term disability). Interventions:
Healthcare interventions that may impact on the course
and/or duration of the disease.
Outcomes: Duration
of sick leave, short- and long-term disability and their
prevention; return-to-work and other work-related disability
outcomes as defined by Cochrane Occupational Safety and
Health Group (osh.cochrane.org). We will report type and
frequency of primary and secondary outcomes. Results:
The priority list identified 144 of 297 review topics from 49
Cochrane Review Groups with an urgent need for update,
100 of which met our inclusion criteria. The analysis
is currently underway. Results will be presented at the
Colloquium. Conclusions: We anticipate that despite their
importance for individuals and society, very few reviews
(< 5%) will consider outcomes related to reduction of sick
leave, short- or long-term disability, and successful return
to work.
Background: Systematic reviewers are challenged to
mediate between an increasing amount of research and
stakeholder priorities about the most important synthesis
topics. Framework synthesis offers an innovative method
to meet both needs. Objectives: To describe and reflect
on framework synthesis as both a method of research
synthesis and to scope and set research priorities for policy
decisions, using a recent systematic review of extrahepatic
conditions in hepatitis C. Methods: To familiarise researchers
with the review topic, scoping searches were undertaken
and an a priori framework of extrahepatic conditions
selected. An information scientist searched for quantitative
studies examining associations between hepatitis C and any
extrahepatic condition. Included studies were indexed/coded
into the framework (i.e. coded) by their main extrahepatic
condition and any newly identified conditions added. Two
reviewers screened and coded reports, establishing inter-rater
reliability, then undertook lone assessments. To chart the
data, results were ordered by publication frequency and
presented to public advocacy groups and researchers, whose
prioritised extrahepatic conditions were compared and added
to the framework. To interpret the findings, topics most
frequently reported and also identified by all advocacy
groups were prioritised for further synthesis. Results:
Searches identified 194 extrahepatic conditions. Framework
synthesis structured stakeholder discussions to prioritise
extrahepatic conditions and identified disparities between
most frequently researched conditions and those identified as
important by people experiencing the conditions. Differences
in presentation of the framework during consultations
may have influenced which topics each group prioritised.
Conclusions: Framework synthesis integrated different
perspectives, facilitating transparent research decisions. Use
of an a priori framework enabled coding within short timelines.
The differing roles and priorities of consulted stakeholders
suggest that equal involvement is more likely, and response
bias reduced, if all stakeholder groups are given consistent
information.
37
LRO 2.5
Using cumulative meta-analysis methods for
prioritizing Cochrane Review updates
Spence G1 , Fanshawe T1
1 University of Oxford, United Kingdom
Background: The regular updating of Cochrane Reviews is
a demanding, ongoing process. To focus resources, reviews
in which the pooled results are inconclusive or highly likely
to be altered by new studies should be prioritized over those
whose results are more stable, or even definitive. As such,
quantitative methods to assess the strength of evidence within
meta-analyses are desirable. To monitor the accumulation
of evidence over time, cumulative meta-analysis can be
undertaken in which the effect size estimate is updated
sequentially as new results become available. Objectives: The
aim of this study is to compare alternative statistical methods
for assessing whether sufficient data have been accumulated
at different points within cumulative meta-analyses, to
determine whether a review update should be prioritized.
Methods: Several methods to quantify the strength of
evidence within a meta-analysis were applied to a database of
the systematic reviews published or updated in the Cochrane
Library between 2008 and 2012. Such methods–including the
calculation of monitoring boundaries based on an ‘Optimal
Information Size’ –have mostly been adapted from the
interim analysis of clinical trials, in which the same issues of
multiplicity from repeated testing and random fluctuations in
effect size are important. Results: The methods investigated
in this study were appraised by comparing their application
to a number of cumulative meta-analyses. Differences
in the results obtained from the various methods will be
demonstrated with examples, and the suitability of the
methods in different situations compared. Conclusions:
We draw conclusions about the attributes and suitability
of different statistical methods of assessing the strength
of evidence within a meta-analysis. Furthermore, such
methods can provide an objective, evidence-based approach
to recommendations for systematic review updates.
LRO 2.6
Focus on registered studies, adequately
powered analyses and true data: filter
unreliable evidence from Cochrane Reviews!
Sydenham E1
1 Cochrane Injuries Group, United Kingdom
Background: Information on the effectiveness and safety of
health care interventions should be valid, precise, up-to-date,
comprehensible and freely available. Currently none of these
criteria is adequately satisfied. Many systematic reviews
contain a biased sample of under-powered trials (some
include fabricated trials) and reviews are often long and
unreadable. Methods: In response to these concerns, the
Cochrane Injuries Group developed a new policy (Roberts
2015). In addition to following standard Cochrane methods,
authors of new and updated reviews should: 1. Only include
a trial if it was prospectively registered (unless the trial report
was published before 2010). 2. Provide a sample size estimate
showing how many participants need to be included in a
meta-analysis for reliable results. Trial sequential methods
should be used to explore all ’significant’ treatment effects
obtained before reaching the required sample size. 3. The
editorial team will conduct statistical checks on data included
in a review. Authors of selected trials will be asked to provide
the original trial data for checking. If authors decline to provide
it, the trial will be removed from the review. Conclusions: We
invite critical discussion of our policy and urge other review
groups to consider adopting it. Reference: Roberts I, Ker K,
Edwards P, Beecher D, Manno D, Sydenham E. The knowledge
system underpinning health care is not fit for purpose and
must change. BMJ 2015: 350: h2463.
LRO 3.1
An evidence-based approach to scoping
reviews
Khalil H1 , Peters M2 , Godfrey CM3 , McInerney P4 , Baldini Soares
C5 , Parker D6
1
Monash University, School of Rural Health, Australia; 2 The
University of Adelaide, The Joanna Briggs Institute, Australia;
3
Queen’s University, School of Nursing, canada; 4 University of Wits,
South Africa; 5 University of São Paulo, Brazil; 6 The University of
Queensland, Australia
Background: The Joanna Briggs Institute (JBI) has as its
central focus not only effectiveness, but also appropriateness,
meaningfulness and feasibility of health practices and delivery
methods. These questions are often answered by considering
other forms of research evidence. Scoping reviews are used
to assess the extent of a body of literature on a particular
topic, and often to ensure that further research in that area
is a beneficial addition to world knowledge. Objectives: To
develop a methodology for scoping reviews based on the JBI
focus on literature synthesis. Methods: A working group
consisting of members of the Joanna Briggs collaborating
organisations met to discuss the proposed framework for
the methodology and develop a draft for the scoping review
methodology based on the Arksey and O’Malley framework
(2005) and that of Levac, Colquhoun and O’Brien (2010). This
was followed by a workshop attended by other members of
the organisations consisting of 30 international researchers
to discuss the proposed methodology. Further refinement of
the methodology was undertaken as a result of the feedback
received from the workshop. Results: The development of
the methodology focussed on five stages of the protocol and
review development. These were: identifying the research
question by clarifying and linking the purpose and research
question; identifying the relevant studies using a three-step
38
literature search in order to balance feasibility with breadth
and comprehensiveness; careful selection of the studies
using a team approach; charting the data in a tabular
and narrative format; and collating the results to identify
the implications of the study findings for policy, practice,
or research. Conclusions: Following the JBI trend of valuing
different types of scientific literature in order to achieve better
health care for all, a proposed methodology was developed
based on the Arksey and O’Malley and the Levac, Colquhoun
and O’Brien frameworks and was successfully trialled.
LRO 3.2
Development of an objectively derived and
validated search strategy to retrieve
overviews of systematic reviews in MEDLINE
Lunny C1 , McKenzie J1 , McDonald S1 , Lunny C2
1
Australasian Cochrane Group, Australia; 2 Australasian Cochrane
Group, Monash University, Australia
Background: There has been a steady increase in the number
of overviews of systematic reviews published in the last 15
years. The ability to dentify published overviews efficiently
is of importance to users of evidence syntheses, as well
as to researchers interested in investigating their methods.
Locating overviews is difficult due to a lack of a validated
search strategy and inconsistent terminology used to describe
overviews. Objectives: To develop a validated search strategy
to retrieve overviews of systematic reviews in MEDLINE.
Methods: We derived a ’gold standard’ test set of overviews
published between 1998 and 2011 from the references of
two methods papers. Two population sets were used to
identify discriminating terms; that is, terms that appear
frequently in the test set, but infrequently in two population
sets of references found in MEDLINE. We used a text mining
package in the statistical program R to conduct a frequency
analysis of terms appearing in the titles and abstracts. Terms
that had a sensitivity of 10% or more in the test set and
a corresponding low sensitivity (less than 2%) in the two
population sets were selected for further testing. Candidate
terms were combined in search strategies and tested in
MEDLINE. Sensitivity and precision were used to evaluate filter
performance. We evaluated the performance of the strategy
against an externally derived set of overviews published
between 2012 and 2014. Results: Two search strategies were
39
developed. The sensitivity-maximising strategy offers 93%
sensitivity in retrieving overviews of reviews, with a low level
of precision (4%). The sensitivity-and-precision-maximising
strategy offers a less sensitive strategy (66%), but a higher
precision (21%). Conclusions: We have developed two
validated search
strategies for locating overviews of systematic reviews using
a text mining approach. Consistent language used to describe
overviews of reviews would aid in their identification, as would
a specific MEDLINE publication type.
LRO 3.3
More than expert searchers? A case study for
librarians exploring roles beyond databases
Ayala AP1 , Lenton E1
1
Gerstein Science Information Centre, University of Toronto,
Canada
Background: At the University of Toronto (UofT), librarian
involvement in a variety of reviews and grant proposals
happens largely and primarily on an ad-hoc basis. The role
of health sciences and medical librarians as expert searchers
is well established and recognized, yet librarians can, and
have, also been called upon to navigate other components
of the research process (Dudden 2011). Involvement includes
but is not limited to: project manager, research coordinator,
and process/protocol expert. Librarian involvement is not only
recommended, but has become a requirement for grants from
key funding agencies, such as the Canadian Institutes of Health
Research (CIHR), for a proposal to be considered complete and
admissible (CIHR 2010). This is wonderful news, but are we
prepared? Objectives: The objective of this study is to develop
a sustainable set of service standards in order to clarify the
roles and tasks to be performed by librarians in research
initiatives such as systematic reviews, but perhaps more
importantly, scoping and realist reviews, as support requests
for these newer, less familiar, methodologies have become
increasingly common. Methods: We will gather and organize
pre-established protocols for each type of research study
and disseminate them widely. An environmental scan will
be performed to learn more about how librarians at UofT are
currently supporting research teams’ reviews. This knowledge
can then be shared among our colleagues and incorporated
into the content of our instruction portfolio. Results: The
results of this initiative and anecdotal gathering are ongoing
and will be presented at the Colloquium. Conclusions:
Librarians offer significant contribution and insight beyond
their searching skills; they can successfully act as project
managers, process experts, and disseminators of protocols
(communicators). Their role continues to evolve. Above all
our goal remains to have an open and honest conversation on
what librarians and information specialists’ experiences have
been so far in these type of initiatives, what our roles are and
what they should be.
LRO 3.4
The impact of selective searching on the
results of systematic reviews
Hartling L1 , Featherstone R2 , Nuspl M2 , Shave K2 , Vandermeer B2
1 Child Health Field; University of Alberta, Canada; 2 Alberta
Research Center for Health Evidence; University of Alberta, Canada
Background: One of the hallmarks of a well-conducted
systematic review (SR) is a comprehensive search. However,
40
empiric evidence is lacking on the impact of database
selection on the results of SRs. Objectives: To examine
the impact on the results of SRs based on selective searching.
Methods: The sample was all child-relevant SRs with at
least one meta-analysis from three Cochrane Review Groups
(Acute Respiratory Infections (ARI), Infectious Diseases,
Developmental Psychosocial and Learning Disorders;
n = 129). We searched 13 databases to determine the
proportion of relevant studies that were indexed in each
source. We re-ran the primary meta-analysis with the studies
identified in MEDLINE only and in MEDLINE plus each of
the other databases. We calculated a ratio of the new
point estimate to the original point estimate and a ratio of
confidence interval widths. We determined how often results
changed in statistical significance. Results: These results are
based on the 57 SRs from ARI; complete results for all SRs will
be available in October 2015. The average number of relevant
studies was highest for MEDLINE (85%), EMBASE (80%), and
BIOSIS (65%). From a total of 398 studies, 65 were not found in
MEDLINE; 20 of the 65 were found in EMBASE and 13 in BIOSIS;
other databases identified three or less. The mean ratio of
point estimates for MEDLINE to the reference standard for
dichotomous outcomes (n = 37) was 1.06. The smallest ratio
was 1.03 for MEDLINE+BIOSIS. The mean ratio of confidence
interval widths of MEDLINE to the reference standard was
1.12; the smallest ratio was 1.09 for MEDLINE+EMBASE. The
mean ratio of standardized mean differences (n = 19) was
0.01 for MEDLINE alone, with the mean ratio of confidence
interval 1.17 (smallest for MEDLINE+EMBASE, 1.15). For
MEDLINE alone, five meta-analyses changed in significance.
The fewest number of meta-analyses changing in significance
(n = 1) was for MEDLINE+EMBASE. Conclusions: This study
provides quantitative data regarding the impact on SR results
of restricting searches to select databases. This information
may be useful to increase efficiencies in the conduct of SRs.
LRO 3.5
Will web search engines replace bibliographic
databases? Review
Taylor B1 , Best P2 , McQuilkin J1
1 Ulster University, United Kingdom; 2 Queen’s University Belfast,
United Kingdom
Background: The task of identifying relevant research
is becoming more demanding, particularly for retrieving
research on psychosocial as well as pharmacological
interventions and for studies using diverse research designs
suited to various stages of developing health and social
care interventions (Medical Research Council 2008). Search
methods must address precision (rejecting unwanted papers)
as well as sensitivity (retrieving relevant papers). The advent
of web search engines adds a new dimension to the usual
searching of bibliographic databases. Objectives: To review
papers reporting empirical studies that compare searching
the world wide web with searching bibliographic databases
to retrieve health and social care research. Methods: An ‘ideal’
formula involving 73 search terms (plus truncation variants)
in three concept groups on the review topic retrieved over
20,000 hits on just one database. This search was optimised in
terms of precision to make it manageable whilst retaining
maximum possible sensitivity by eliminating less precise
terms in sequence. The search was run on six bibliographic
databases and four web search engines to identify papers
about empirical studies on searching databases and the world
wide web for research on health and social care. Papers
were excluded if they did not study the relative merits of
databases and web search engines in terms of sensitivity or
precision. Results: Results regarding sensitivity and precision
of searching bibliographic databases and the world wide web
will be summarised in tabular format. Conclusions regarding
constructing search formulae will be summarised in narrative
form with examples from papers and drawing on our own
search process. Preliminary results suggest that: (1) Number
Needed to Read is a readily-understood metric for precision;
and (2) web search engines are not sufficiently sensitive or
precise to replace bibliographic databases. Conclusions:
Papers will be discussed in the broader theoretical context
of ‘satisficing’ in relation to information-seeking behaviour
(Prabha 2007; Simon 1956). Conclusions will be drawn for
optimising precision of search strategies.
References: 1. Medical Research Council (2008) Developing
and Evaluating Complex Interventions, London: MRC Prabha
C, Connaway LS, Olszewski L & Jenkins LR (2007) What
is enough? Satisficing information needs. Journal of
Documentation 63(1), 74–89.
2. Simon HA (1956) Rational choice and the structure of the
environment. Psychological Review 63, 129–138.
LRO 3.6
Googling for grey: using Google and
Duckduckgo to find grey literature
Hagstrom C1 , Kendall S2 , Cunningham H1
1
University of Toronto, Canada; 2 Mt. Sinai Hospital Library, Canada
Background: Using search engines such as Google or
Duckduckgo for literature searches can result in literally
millions of citations. Results depend on various elements,
such as the search engine’s page ranking, or the user’s habitual
search patterns. Objectives: Google and Duckduckgo can be
effectively used to retrieve government or other institutional
reports and other hard-to-find grey literature. Methods: We
will demonstrate how Google and Duckduckgo function. We
will compare the two search engines and will give examples of
how they can be searched to eliminate many irrelevant results.
Results: Different types of grey literature can be found using
Google and Duckduckgo. Conclusions: Using search engines
such as Google and Duckduckgo need not be as onerous as
one might think when looking for grey literature.
41
Decision-making: End users
and consumers
LRO 4.1
Are online learning resources effective to
build capacity for evidence-informed
decision making?
Dobbins M1 , Yost J1
1 McMaster University, Canada
Background: Public health professionals are expected
to consider the best available research evidence when
making practice decisions.
However, they encounter
challenges such as time constraints, limited resources
and skills to find, appraise, interpret and apply evidence.
Objectives: To evaluate the usage and change in learning
outcomes associated with use of online learning resources
for evidence-informed decision making (EIDM). Methods: The
National Collaborating Centre for Methods and Tools worked
with Web developers and online learning specialists to create
the Learning Centre. As a management system, the Learning
Centre offers multiple online learning resources for which
the content, practical examples and assessments have been
developed in collaboration with public health professionals.
Quantitative and qualitative evaluation data are continuously
collected before and after use of the online learning resources.
Results: Evaluation data from June 2012, when the Learning
Centre was launched, to May and September 2015, will be
presented. As of March 2015, the resources in the Learning
Centre have been used by more than 3000 public health
professionals. Quantitative data demonstrate significant
increases in knowledge and self-efficacy for EIDM following
use of the resources. Quantitative and qualitative data
indicate learners are satisfied with the online format of
the Learning Centre; users return to the resources multiple
times, suggesting they value the resources as references to
support EIDM capacity. Conclusions: Evaluation of this
project validates that the online learning resources offered
by the National Collaborating Centre for Methods and Tools
are innovative, accessible and effective in supporting EIDM
capacity development among public health professionals.
42
LRO 4.2
LRO 4.3
Outcomes of the use of targeted online
consumer health information in primary care:
a participatory systematic mixed-studies
review
Identifying patient-important practical issues
for shared decision making
Pluye P1 , El Sherif R1 , Granikov V1 , Tang DL1 , Bujold M1 , Hong
QN1 , Galvao C2 , Repchinsky CA3 , Hutsul J3 , Dunikowski L4 , Frati
F1 , Shoet L5 , Vineberg L6 , Bartlett G1 , Grad R1 , Shulha M1 , Kloda
L1 , Burnand B7 , Desroches S8 , Legare F8 , Millerand F9 , Rihoux B10 ,
Vedel I1
1 McGill University, Canada; 2 Universidade de São Paulo, Brazil;
3
Canadian Pharmacists Association, Canada; 4 College of Family
Physicians of Canada, Canada; 5 Center for Literacy, Canada;
6 Patient Representative, Canada; 7 Institut Universitaire de
Médecine Sociale et Préventive, Switzerland; 8 Universite Laval,
Canada; 9 Universite du Quebec a Montreal, Canada; 10 Universite
Catholique de Louvain, Belgium
Background: Consumers regularly seek online health
information.
Systematic reviews in public health and
oncology, and on decision aids, suggest the use of
targeted information improves knowledge, participation
in health care and health.
However, little is known
about the outcomes associated with information-use in
primary care (outside shared decision-making aids). Thus,
for primary healthcare patients, what are the outcomes
associated with the use of targeted (vs non-targeted)
online consumer health information? Objective: Identify
types of patient health outcomes and related conditions
(context, information-seeking, use). Methods: - Participatory
systematic mixed studies review (integrating qualitative and
quantitative evidence). - Eligibility criteria: qualitative,
quantitative, or mixed-methods study; English or French;
1990 to 2014; community-based primary health care;
online consumer health information-use. - Information
sources: MEDLINE, Embase, PsycINFO, CINAHL, LISA, and
grey literature. - Search strategy built by four specialized
librarians. - Selection (DistillerSR) and critical appraisal
(Mixed Methods Appraisal Tool) of included studies by two
independent researchers. - Data extraction and synthesis:
Deductive-inductive qualitative thematic data analysis by four
researchers, and disambiguation of themes. Results: Flow
diagram (Fig. 1): 50 included studies with diverse designs,
four concern targeted information (one non-randomized, two
quantitative descriptive, and one qualitative). Inter-rater
reliability scores (kappa) indicate substantial agreement
between researchers for selection (0.62) and appraisal
(0.67). The qualitative synthesis suggests key conditions
and negative/positive outcomes (e.g. decreased/increased
worries). Conclusions: Results will allow us to explore
‘condition-outcome’ configurations (content and Boolean
analysis). Using a mixed-methods research design, this is
the first systematic review to explore outcomes of targeted
online consumer health information-use in primary care. This
will lead to design a tool to help patients for finding and using
(e.g. with partners or clinicians) information such as decision
aids.
Fog Heen A1 , Montori VM2 , Lloyd A3 , Vandvik PO1 , Agoritsas T4
1 Department of Medicine, Innlandet Hospital Trust, Gjøvik, Norway
and Institute for Health and Society, Faculty of Medicine, University
of Oslo, Norway; 2 Knowledge and Evaluation Research Unit, Mayo
Clinic, Rochester, Minnesota, USA; 3 Institute of Primary Care
and Public Health, Cardiff University School of Medicine, United
Kingdom; 4 Department of Clinical Epidemiology and Biostatistics,
McMaster University, Canada
Background: To help patients and clinicians engage in
collaborative deliberation, tools for shared decision making
(SDM) usually include evidence on the nature and likelihood
of patient-important outcomes associated with management
alternatives. However, patients face practical issues when
implementing the chosen option in their daily life. Evidence
summaries typically include the former but overlook the
latter. Objectives: To map and categorize patient-important
practical issues related to treatment options, and develop a
generic framework for their incorporation into the creation
of SDM tools. Methods: We systematically reviewed and
mapped two main sources (HealthTalk.org registry and
Option Grids) that included a large sample of conditions
and applied a rigorous methodology to identifying patient
experience and questions. We systematically screened every
topic, abstracting all practical issues relevant to management
options. Two independent reviewers grouped the issues
in themes, compared results, resolved discrepancies and
arrived at final a set of themes. We tested the applicability
of this framework in identifying specific practical issues when
creating 14 SDM tools using the SHARE-IT method. Results:
Thematic analysis identified 15 categories from 967 issues
across 297 topics in HealthTalk.org and 29 Option Grids:
five were care-related (medication routine, tests and visits,
procedure and device, recovery and adaptation, co-ordination
of care); five related to daily life (food and drinks, exercise and
activities, social life and relationships, work and education,
travel and driving); and five related to miscellaneous
issues (adverse effects, interactions and antidote, physical
well-being, emotional well-being, pregnancy and nursing,
costs and access).
In creation of 14 SDM tools on
antithrombotic therapy and chemotherapy, all categories
were applicable and used. Conclusions: We identified 15
overarching themes of patient-important practical issues. This
generic framework can complement trustworthy evidence
summaries from systematic reviews and help thus create
tools that are more helpful in discussing with patients issues
that matter to them.
43
LRO 4.4
LRO 4.5
Combining network meta-analysis results
with patient preferences to facilitate shared
decision making
Sources of information and attitudes towards
evidence-based healthcare: a survey of Swiss
physicians
Naci H1 , van Valkenhoef G2 , Higgins J3 , Fleurence R4 , Ades T3
1
London School of Economics and Political Science, United
Kingdom; 2 University of Groningen, The Netherlands; 3 University
of Bristol, United Kingdom; 4 Patient Centered Outcomes Reserach
Institute, USA
Burnand B1 , Bengough T2 , Barry A3 , Pidoux V3 , Amiguet M3 , Bovet
E4
1
Cochrane Switzerland and Institute of Social and Preventive
Medicine, Lausanne University Hospital, Switzerland; 2 Austrian
Federal Institute of Health Care, Austria; 3 Institute of Social and
Preventive Medicine, Lausanne University Hospital, Switzerland;
4 Haute Ecole de Santé Vaud and Institute of Social and Preventive
Medicine, Lausanne University Hospital, Switzerland
Background: Even in cases when comparative evidence
about drugs exists, prescribers and patients often struggle
to weigh the relative benefits and harms of multiple
options. One complexity of drug therapy is the difficulty
in making trade-offs between the benefits and harms of
two or more alternatives. Objectives: To formalize the
incorporation of patient preferences into treatment selection
decisions using statins as a case study. Methods: We
combined network meta-analysis and multi-criteria decision
analysis. Specifically, using a systematic review and network
meta-analysis of randomized trials of statins, we calculated
absolute risks of all-cause mortality, coronary events,
cerebrovascular events, discontinuations due to adverse
events, myalgia, transaminase elevations and creatine kinase
elevations associated with each statin. We then applied a
structured benefit risk model that allows evidence on multiple
outcomes to be combined using qualitative preference
statements. When combining the evidence on multiple
outcomes, we adopted simple preference statements about
the relative importance of different outcomes and considered
the effect of statins on preventing mortality to be more
important than either coronary or cerebrovascular events,
which were in turn more important than any one of the harm
outcomes. Results: Fluvastatin has a considerable probability
of both being the best (41%) and worst (12%) statin (based
on the combination of benefits and harms), highlighting
the uncertainty in its evidence base. Both simvastatin and
atorvastatin have a high probability of better rank, with
a negligible probability of ranking worst: atorvastatin and
simvastatin have the most favorable benefit and harm profiles.
Conclusions: In the future, summaries of clinical evidence
obtained from network meta-analyses can be combined with
patient preferences, and considered alongside the knowledge
and clinical expertise of prescribers when making treatment
selection decisions.
Background: Barriers to knowledge translation generate
delays to adopt new effective interventions, persistence of
obsolete treatments, overuse of ineffective interventions
and underuse of effective care. Objectives: To examine
current knowledge translation practices of Swiss physicians.
Methods: We surveyed French- and German-speaking private
practice physicians. Semi-structured open-ended interviews
and focus groups targeting family physicians, psychiatrists,
cardiologists, endocrinologists, and orthopaedic surgeons
in Switzerland, and the existing literature, allowed us to
draft an interview questionnaire for the computer-assisted
web interview (CAWI) of a large sample of these groups
of physicians (summer 2014). We targeted sources of and
barriers to information, and adherence to evidence-based
medicine (EBM) principles.
We computed a score of
adherence to EBM based on five items and used uni- and
multivariate analyses. Results: Twenty-nine physicians (21
men) participated in the qualitative study and 985 in CAWI
(15% participation, similar among the three Swiss language
regions and physicians’ specialty). CAWI indicated that the
main sources of information of family physicians included
congresses (41%), practice guidelines (38%), colleagues (37%)
and specialists (33%), systematic reviews (28%), knowledge
syntheses (24%) and quality circles (21%).
Perceived
usefulness of EBM in daily practice was considered very
high by 20%, high by 56%, low by 20% and very low by
4% of physicians. Adherence to EBM decreased slowly
with age (-2.1 percentage-points/10 years), and was lower
in endocrinologists, family physicians, orthopaedic surgeons
and psychiatrists ( -7.0, -8.4, -12.8, -13.8 percentage-points,
respectively) compared to cardiologists. Conclusion: We
observed that physicians working in Switzerland use various
sources of information to update their knowledge, but that no
single source dominates. Evidence-based sources are often
used and three-quarters of physicians adhere to its principles.
The optimal source, type and format of information for busy
physicians are uncertain.
44
LRO 4.6
Evidence synthesis activities of a hospital
evidence-based practice center and impact
on hospital decision making
Jayakumar K1 , Lavenberg J1 , Mitchell M1 , Doshi J1 , Leas B1 ,
Goldmann D1 , Williams K1 , Brennan P1 , Umscheid C1
1
University of Pennsylvania, USA
Background: Hospital evidence-based practice centers
(EPCs) seek to implement evidence into local policy and
practice, but their impact on institutional decision making
is unclear. Objectives: To assess the evidence synthesis
activities and impact of the University of Pennsylvania
Health System’s Center for Evidence-based Practice (CEP),
a hospital EPC. Methods: Descriptive analysis of CEP’s
internal database of rapid systematic reviews since CEP’s
inception (July 2006 to June 2014), and survey of review
requestors from CEP’s last four fiscal years. Descriptive
analysis examined requestor and report characteristics; a
questionnaire examined report usability and impact, and
requestor satisfaction (higher scores on five-point Likert
scales reflected greater agreement). Results: CEP has
completed 249 reviews since inception. The most common
requestors were clinical departments (29%, n = 72), chief
medical officers (19%, n = 47), and purchasing committees
(14%, n = 35). The most common technologies reviewed
were drugs (24%, n = 60), devices (19%, n = 48), tests (12%,
n = 31) and care processes (12%, n = 31). The mean report
completion time was 70 days. Thirty reports (12%) informed
computerized decision support interventions. More than half
of reports (56%, n = 139) were completed in the last four fiscal
years for 65 requestors. Forty-six of the 64 eligible survey
participants responded (72%). Requestors were satisfied
with the report (mean = 4.4), and agreed it was delivered
within the expected timeframe (mean = 4.4), informed their
final decision (mean = 4.1), and that their final decision was
consistent with report findings (mean = 4.0). Conclusions:
Our findings suggest hospital EPCs can efficiently synthesize
and disseminate evidence addressing a range of clinical topics
for diverse stakeholders, and can influence decision making.
Hospital EPCs may be an effective infrastructure paradigm for
promoting evidence-based decision making.
Oral session 4
Qualitative evidence
O 4.1
Using purposive sampling within qualitative
evidence synthesis: experiences from a
Cochrane Review of parents’ views and
experiences of vaccination communication
Ames H1 , Glenton C2 , Lewin S3 , COMMVAC Project 21
1
The Norwegian Knowledge Centre for the Health Services, Norway;
2 The Norwegian Knowledge Centre for the Health Services and
Cochrane Norway, Norway; 3 The Norwegian Knowledge Centre for
the Health Services and South African Medical Research Council,
Norway
Background: Reviews of qualitative research, or qualitative
evidence synthesis (QES), are increasingly common and are
now being done by Cochrane. Identifying and including
all possible studies is a goal for reviews of intervention
effectiveness, but is not necessarily the best approach for
QES. Too much data due to a large number of studies may
threaten the quality of the analysis. Using purposive sampling
of included studies is one way of reducing the number of
articles to be analysed. Objectives: To describe an approach
to purposively sampling included studies in a QES. Methods:
We searched electronic databases to identify studies that
met the following inclusion criteria: (1) qualitative methods
of data collection and analysis; (2) parents or carers as
participants; and (3) a focus on views and experiences of
vaccination information. For studies that met the inclusion
criteria, we extracted information regarding study country
and focus and assessed data ’richness’ on a 1-5 scale. Studies
meeting the inclusion criteria were sampled using a three-step
purposive process: (1) We sampled any study from a low- or
middle-income country (LMIC) to help ensure geographic
spread, as the majority of studies on the topic were from
high-income settings; (2) We sampled all studies that scored
3 or more on the data richness scale; (3) We reviewed the
abstracts of the remaining studies and sampled those that
answered the review question most closely and in the most
detail. Results: Sixty studies met our inclusion criteria.
We sampled seven studies from LMIC settings; 14 studies that
scored more than 3 on the data richness scale; and five studies
that answered the review question best, leaving 26 studies
for data analysis. Conclusions: This systematic, three-step
approach may prove useful to other researchers attempting
to synthesise qualitative literature from a large number of
studies. Strengths included that it allowed us to achieve
a geographic spread of articles together with rich data that
closely answered the review question. The weakness is that we
may have overlooked articles that did not meet our sampling
criteria, but would have contributed to the synthesis.
45
O 4.2
Sensitivity analysis in systematic reviews of
qualitative evidence: an example from a
mixed-methods systematic review
Langer L1 , Stewart R1 , Winters N2
1
University of Johannesburg, South Africa; 2 University of Oxford,
United Kingdom
Background: The systematic synthesis of qualitative evidence
is a recent development of systematic review methodology.
Systematic reviews of qualitative evidence need to subscribe
to the same methodological rigour and transparency that
characterises reviews of quantitative evidence. As a result,
reviewers should assess the sensitivity of the findings of any
qualitative evidence synthesis to a priori defined variables.
Objectives: To report on the methodological insights gained
from a sensitivity analysis conducted as part of a qualitative
evidence synthesis in a mixed-methods systematic review.
Methods: We applied thematic synthesis to synthesise
the findings from qualitative research in a mixed-methods
systematic review. Prior to extracting themes from the primary
research evidence, each study was subjected to a critical
appraisal assessing the reliability of the study’s findings. After
configuring findings in the thematic synthesis, the results
from the synthesis were subjected to a sensitivity analysis.
We assessed whether the synthesis results were sensitive
critical appraisal tool; (2) the applied qualitative research
methodologies; and (3) the inclusion of individual studies
yielding a larger than average number of themes. Results:
We found that the results of the synthesis were sensitive
to the inclusion of individual studies that yielded a larger
than average number of themes. We also established that
the exclusion of studies after the application of the critical
appraisal tool would have generated a number of themes not
identified in the included evidence. Conclusions: To ensure
the rigour and transparency of qualitative evidence syntheses,
both a critical appraisal of the qualitative primary evidence,
as well as a sensitivity analysis of the synthesis results, using
a priori defined variables are required.
O 4.3
Unpackaging context dependence in
systematic reviews of complex interventions:
a methodological approach for managing
double-layered complexity
Parrott J1 , Handu D2 , Benson-Davies S2
1 Rutgers University, USA; 2 Academy of Nutrition and Dietetics, USA
Background: Measures of effect size in systematic reviews
of complex interventions, like pediatric weight management,
are notoriously difficult to estimate meaningfully - in part due
46
to to: (1) the exclusion of studies after the application of the
the highly contextually dependent nature of multicomponent
interventions. In short, complex treatment
configurations are compounded by variations in method
of delivery, length of treatment, setting, etc. This is
complexity on top of complexity. Objectives: To devise
an analytic approach for managing the layered complexity
of complex interventions and understanding how different
configurations of study/treatment context characteristics
could affect outcomes. Methods: Data from 73 controlled
clinical trials of pediatric weight management interventions
published since 2005 were extracted into an online platform
using a detailed extraction form designed to capture 30
different treatment and intervention context characteristics.
Multiple components analysis was used to construct a
reduced dimension adjacency matrix, which was then used
to derive theoretically-informed groups of similar treatment
’mixes’. Characteristic by group Chi-square analyses provided
description of the identified groups.
Random-effects
meta-analysis was then used to estimate treatment effect
differences between groups. As heterogeneity was still high,
and based on the theoretical position that configurations
of context characteristics (rather than single characteristics)
affect treatment results, crisp set qualitative comparative
analysis (QCA) was used to identify discrete ’paths’ to
positive or negative weight status outcomes at six and 12
months post-treatment. Results: The analysis identified
a limited number of treatment/context configurations that
were consistently associated with both positive and negative
weight status outcomes (Figure 1).
Conclusions: A
combination of analytic procedures may be used to manage
and gain a deeper understanding of heterogeneity in context
dependent multicomponent interventions.
This mixed
approach has great potential for use in program design
by practitioners to identify conditions where particular
treatment mixes are likely to lead to positive or negative
outcomes.
O 4.4
Linking Cochrane qualitative evidence
syntheses with effectiveness reviews:
experiences from the development of an
EPOC qualitative evidence synthesis
Munabi - Babigumira S1 , Glenton C1 , Lewin S2 , Fretheim A3 ,
Nabudere H4
1
Global Health Unit, Norwegian Knowledge Centre for the Health
Services, Norway; 2 Global Health Unit, Norwegian Knowledge
Centre for the Health Services; Health Systems Research Unit,
Medical Research Council of South Africa, Tygerberg, Norway;
3
Global Health Unit, Norwegian Knowledge Centre for the Health
Services; Institute of Health and Society, University of Oslo, Norway;
4
College of Health Sciences, Makerere University, Uganda
Background: Qualitative evidence syntheses published in
the Cochrane Library provide additional evidence to inform
or supplement effectiveness reviews. However, the methods
used to link the findings of qualitative evidence syntheses and
effectiveness reviews are not yet established, and tend to vary
across review teams. Objectives: To discuss the methods
used, and our experiences when supplementing evidence
from the review ‘The effectiveness of policies promoting
facility-based deliveries in reducing maternal and infant
morbidity and mortality in low- and middle-income countries’
with the qualitative evidence synthesis: ‘Factors that influence
the provision of intrapartum and postnatal care by skilled birth
attendants in low-and middle-income countries’. Methods:
When synthesizing qualitative evidence, we utilise a logic
model that represents the theories and assumptions about
the links between various inputs and activities through which
skilled attendance at birth influences outcomes. Reviewers
then map the review findings to the logical model; adapt
or elaborate the model as needed; and subsequently assess
how the findings relate to the results of the effectiveness
review. In particular, we seek to provide explanations of
how the context or other relevant findings influence the
outcomes reported in the effectiveness review. In addition to
presenting the methods we used, we will share our experiences
from the process of linking the two reviews. Results: Close
collaboration between the two review teams, facilitated by
face-to-face contact, has contributed to the development of
both reviews. This collaboration has enabled us to identify
areas where the qualitative evidence synthesis can explore
and explain the findings from the effectiveness review. Both
reviews are in progress and additional results will be available
at the time of the Colloquium. Conclusions: Methods for
linking qualitative evidence synthesis reviews to effectiveness
reviews are yet to be established. Close collaboration of review
teams facilitates this process. Data from qualitative evidence
synthesis provides useful additional information about how
and why interventions work.
Oral session 5
GRADE
O 5.1
Defining and evaluating quality of evidence in
’Summary of findings’ tables
Hultcranz M1 , Schunemann H2 , Guyatt G2
1 Swedish Council on Health Technology, Sweden;
University, Canada
2 McMaster
Background: Cochrane has adopted the GRADE (Grading of
Recommendations
Assessment,
Development
and
Evaluation) approach to categorizing quality of evidence
- most prominently seen in ’Summary of ’indings’ tables
–defined as reflecting the extent of confidence that the
estimates of effects are correct. This definition leaves
ambiguity: does it refer to confidence in point estimates, or
confidence in the range of possible estimates? Objective:
To clarify the definition and approaches to judging
47
evidence quality. Method: Discussions, workshops, iterative
refinement of ideas, and discussion at a GRADE Working Group
meeting. Results: Based on this process, quality of evidence
may be better considered as the confidence or certainty that
a true effect lies on one side of a specified threshold, or within
a chosen range. This interpretation raises the challenge of
defining the threshold or range for making quality ratings.
We developed five possible approaches for making this
judgment. For guidelines, what we call a fully contextualized
approach requires simultaneously considering all critical
outcomes and their relative value. We defined four less
contextualized approaches more appropriate for systematic
reviews. We applied all five approaches to the outcome of
myocardial infarction in a review of dual antiplatelet therapy
(DAPT) versus single antiplatelet therapy (SAPT) in patients
who have undergone stenting of their coronary arteries
(Table). The four approaches most appropriate for systematic
reviews involve judging certainty that the true effect lies
within the 95% confidence interval; that the effect lies within
some range other than the 95% confidence interval; that the
effect is something other than no effect at all; and that the
effect lies within ranges of what we might consider small,
moderate, or large. Conclusions: The proposed approach
is now under consideration by the GRADE Working Group. If
adopted by GRADE, and subsequently by Cochrane, it will
provide a useful clarification of how Cochrane authors can
make quality of evidence judgments.
piloting of the tool on qualitative evidence syntheses.
Results: CERQual bases assessments of confidence on
four components: 1. the methodological limitations of
the individual studies contributing to a review finding,
assessed using a quality-assessment tool for qualitative
studies; 2. the coherence of each review finding, assessed
by looking at the extent to which a review finding is based
on data that are similar within and across multiple individual
studies and/or incorporates convincing explanations for any
variations across individual studies; 3. the relevance of
a review finding, determined by the extent to which the
primary studies supporting a review finding are applicable to
the context specified in the review question; and 4. the
adequacy of data supporting a review finding, assessed
by an overall determination of the degree of richness or
scope of the evidence and quantity of data supporting a
review finding. After assessing each component, an overall
judgement of the confidence in each review finding is made.
Confidence is assessed as high, moderate, low, or very
low. Conclusions: GRADE-CERQual provides a transparent
method for assessing the confidence of evidence from reviews
of qualitative research. Like other GRADE tools, it may
facilitate the use of these findings alongside reviews of
effects, in Health Technology Assessments and in guideline
development processes.
O 5.3
O 5.2
Assessing how much confidence to place in
findings from qualitative evidence syntheses:
a new version of the GRADE-CERQual tool
Glenton C1 , Lewin S2 , Booth A3 , Noyes J4 , Garside R5 , Tuncalp O6 ,
Carlsen B7 , Bohren M6 , Wainwright M8 , Rashidian A9 , Colvin C8 ,
Munthe-Kaas H1
1 Norwegian Knowledge Centre for the Health Services, Norway;
2
Norwegian Knowledge Centre for the Health Services; Medical
Research Council of South Africa, Norway; 3 University of Sheffield,
United Kingdom; 4 Bangor University, United Kingdom; 5 University
of Essex Medical School, United Kingdom; 6 World Health
Organization, Switzerland; 7 Uni Research Rokkan Centre, Norway;
8 University of Cape Town, South Africa; 9 Tehran University of
Medical Sciences, Iran
Background: Systematic reviews of qualitative studies
(qualitative evidence syntheses) are increasingly used to
bring together findings from qualitative studies. In order
to use the synthesised findings to inform decisions we need
methods to assess how much confidence to place in these
findings. Objectives: To describe the latest version of a
Grading of Recommendations Assessment, Development and
Evaluation (GRADE) tool for assessing how much confidence
to place in findings from qualitative evidence syntheses.
Methods: GRADE’s Confidence of the Evidence from Reviews
of Qualitative research (CERQual) tool was developed through
review of existing tools; working group discussions; and
Using the GRADE approach to network
meta-analysis
Guyatt G1 , Murad H2 , Puhan M3
1 McMaster University, Canada; 2 Mayo Clinic, USA; 3 Institute of
Social and Preventive Medicine, Zurich, Switzerland
Background: Until recently, network meta-analyses (NMA)
failed to address the certainty associated with different
paired comparisons in a network. The GRADE working group
has developed an approach to NMA that involves rating the
certainty in estimates for each paired comparison. Objective:
To illustrate GRADE’s approach to NMA. Method: The GRADE
approach was developed through discussion and iterative
refinement; we have applied the approach in a number of
NMAs. The approach involves four steps: (1) present direct
and indirect treatment estimates for each paired comparison;
(2) rate the certainty of each direct and indirect estimate; (3)
present the NMA estimate for each comparison; (4) rate the
certainty of each NMA estimate. Results: Rating the certainty
of each direct estimate is straightforward using established
GRADE methodology. Rating the certainty for indirect
estimates involves identifying, for each paired comparison,
the 1st order loop (those loops that involve only a single
additional intervention linked to the two treatments/nodes of
interest) that contributes most to the indirect estimate; rating
the certainty for the two direct comparisons contributing to
the indirect comparison; and choosing the lower of the two
ratings. If direct and indirect estimates are consistent, the
higher rating of the two represents the overall certainty of the
48
network estimate. If they are inconsistent, GRADE suggests
using the higher of the two, rather than the network estimate,
as the best estimate of effect. Examples illustrate the utility of
the approach. For instance, in a NMA addressing optimal fluid
administration in septic patients, high certainty evidence was
available for the comparison of starch vs crystalloid, but only
very low certainty for gelatin vs crystalloid. A challenge to full
application of the GRADE approach is difficulty in generating
separate indirect effect estimates in NMAs with large numbers
of comparisons. Conclusions: Since clinicians need to take
into account not only best estimates, but also certainty, in
making their management decisions, the GRADE approach
enhances the utility and interpretation of NMA.
O 5.4
Research synthesis to help going from
evidence to decision: a practical application
of the GRADE/DECIDE Evidence to Decision
Framework
Parmelli E1 , Amato L1 , Vecchi S1 , Minozzi S1 , Davoli M1
1
Department of Epidemiology, Lazio Regional Health Service, Italy
Background: Healthcare systems are offered with a wide
range of technologies and services, but they have to
cope with decreasing resources and the uncertainty about
what is effective and more appropriate. Making decisions
about healthcare interventions is complex.
Decisions
should be informed by the best available evidence,
which, in order to be comprehensive needs to take into
account all the relevant aspects (e.g. efficacy, safety,
equity, costs), and gathered within a limited time period.
DECIDE (www.decide-collaboration.eu) is a project funded
by the European Community that, using the GRADE
(Grading of Recommendations Assessment, Development and
Evaluation) methodology (www.gradeworkinggroup.org),
aims at implementing strategies to enhance dissemination
and communication of scientific evidence to support on-time
evidence-based decision making. Objectives: To prepare
an informative and exhaustive evidence synthesis to help a
Regional Commission taking a coverage decision using the
GRADE/DECIDE Evidence to Decision (EtD) Framework (see
attached file). Methods: We performed a systematic review
of the available evidence about the efficacy and safety of
trans-aortic valve implantation (TAVI) for aortic stenosis in
high risk patients. We also searched for information about
economic evaluations, patients’ preferences, feasibility and
equity. We used the EtD Framework to present the information
to the members of the Regional Commission. Results: We
identified 4072 records: four Health Technology Assessment
(HTA) reports, 13 systematic reviews (SR), three randomised
controlled trials and 15 observational studies were eligible for
our topic (the single studies were not included in the SRs). We
prepared two different EtD Frameworks for the two questions
of interest: (1) TAVI versus surgery and (2) TAVI versus medical
therapy. Conclusions: The relevance of this experience
lies in the opportunity to pilot test the GRADE/DECIDE EtD
Framework for real world coverage decisions. Feedbacks
about its usefulness and usability will be collected from the
members of the Regional Commission.
Oral session 6
Disseminating evidence
O 6.1
Lessons learnt from a planned, targeted
dissemination of a systematic review on a
controversial topic in the field of nutrition
Naude CE1 , Schoonees A1 , Young T2 , Senekal M3 , Garner P4 ,
Volmink J2
1 Centre for Evidence-Based Health Care, Stellenbosch University,
South Africa; 2 Centre for Evidence-Based Health Care, Stellenbosch
University; South African Cochrane Centre, South African Medical
Research Council, South Africa; 3 Division of Human Nutrition,
University of Cape Town, South Africa; 4 Effective Health Care
Research Consortium, Liverpool School of Tropical Medicine, United
Kingdom
Background: Low carbohydrate diets for weight loss and
cardiovascular benefits have generated controversy between
advocates, media and health professionals. At the request
of the Heart and Stroke Foundation South Africa, we
conducted a systematic review and actively disseminated
its findings to inform ongoing debates. Objectives: To
describe the stakeholder-specific dissemination strategy for a
systematic review and assess its ‘reach’ and reflect on lessons
learnt. Methods: We planned, developed and implemented
a stakeholder-specific dissemination strategy, including
identifying priority stakeholders, communication objectives
and channels; tailoring approaches for stakeholders;
developing appropriate products; and interacting with key
role players. ‘Reach’, defined here as exposure to information,
was measured by capturing all dissemination, including
media monitoring and non-traditional metrics. Results:
Stakeholders included scientists, health professionals, media,
public and policymakers. Implementation resulted in wide
‘reach’, including 14 scientific presentations, two national
television and five radio interviews, broad social media
coverage, Altmetric score in the 99th percentile and more
than 75 print and digital media contributions. The ‘reach’
was likely positively influenced by the controversy and
public interest, as well as by interaction between the
researchers and key stakeholders. The ‘reach’ may have
increased stakeholders’ motivation and ability to use the
evidence to inform decision-making. There is value in
responding to stakeholders, producing a timely, rigorous
review, disseminating it via a planned, targeted strategy,
and being prepared and responsive to media requests.
Conclusions: Planning and implementation of a targeted
dissemination strategy for a systematic review is likely to
increase the ‘reach’ and possibly use of rigorous evidence. Part
49
of the success related to the timeliness of completion. Review
teams should be mindful that detailed, opportune planning
and preparation of a dissemination strategy is essential,
especially for controversial, media-hot topics.
O 6.2
Cochrane 2.0: Tweeting and blogging
to disseminate child health evidence
Hartling L1 , Hamm M2 , Newton A3 , Fernandes R4 ,
Featherstone R5 , Thomson D1
1
Child Health Field, University of Alberta, Canada; 2 Alberta
Research Center for Health Evidence, University of Alberta, Canada;
3
Department of Pediatrics, University of Alberta, Canada; 4 Child
Health Field; Cochrane Portugal; Department of Pediatrics and
Clinical Pharmacology Unit, Lisbon Academic Medical Centre,
Canada; 5 Alberta Research Center for Health Evidence; University
of Alberta, Canada
Background: Healthcare providers desire ready access to
reliable synthesized information to support point-of-care
decision-making. Virtual communities, facilitated by the
adoption of social media tools such as Facebook, Twitter, and
YouTube, are increasingly used for knowledge mobilization,
bridging the gap between knowledge generation/synthesis
and knowledge implementation. Objectives: To implement
and evaluate a structured social media strategy to
disseminate high quality, child health evidence from Cochrane
systematic reviews and overviews to healthcare providers
caring for children. Methods: The Child Health Field’s
social media strategy has three components: daily tweets,
weekly blog posts, and a monthly Twitter journal club. Each
tweet, blog, and journal club shared Cochrane evidence on
a child health topic (e.g. treatments for asthma, prevention
of childhood obesity). The social media strategy will be
evaluated in the following ways: (1) Twitter and blog site
analytics –measuring engagement with tweets, and blog site
visits; (2) bit.ly statistics –measuring interaction with URL
links; (3) Altmetrics –data on the change in scores of social
media engagement with source evidence after our promotion;
and (4) participant feedback on the journal clubs. We are
also tracking the time and skill-sets required to develop
the dissemination materials and manage the social media
strategy. Results: A new blog post was uploaded on Monday
of each week from November 2014 through March 2015 (22
weeks). We posted tweets three times per day; the tweets
linked to the blog or to additional resources on the weekly
blog topic. We ran journal clubs via Twitter each month
from January to March. Results of the evaluation will be
presented at the Colloquium. Conclusions: This work will
provide empiric evidence for the utility of specific social media
strategies for the dissemination of Cochrane evidence to
professionals providing health services to children and youth.
The results will provide useful information to others interested
in developing a social media strategy or understanding the
uptake of their social media activities.
O 6.3
Partnerships to co-ordinate and evaluate
dissemination efforts
Ruotsalainen J1 , Morata T2 , Lum M2 , Stewart G3 , Verbeek J1
1 Cochrane Work, Finland; 2 National Institute of Occupational
Safety and Health, USA; 3 Wiley, United Kingdom
Background: Dissemination of review results is one of the
core objectives of Cochrane. With limited resources, how
do we ensure maximum reach and impact for our reviews?
Cochrane Work developed a new structured outreach strategy
that is based on aligning dissemination efforts with domestic
and international partner organizations. We aim to reduce
repetition, expand reach and ultimately improve the uptake
of review results. Objectives: We report on the initial phase of
our co-ordinated dissemination efforts in partnership with the
National Institute of Occupational Safety and Health (NIOSH)
in the USA. Methods: We evaluate our concerted social media
delivery with bibliometric and Altmetric data. To prepare
for the release of the Cochrane Work review ‘Preventing
occupational stress in healthcare workers’ we co-authored a
NIOSH Science Blog and tweeted actively. We examined the
effects of our dissemination efforts on blog views, number of
tweets and review downloads. Results: The NIOSH Science
Blog reached 1149 views from 960 unique visitors between
10 December 2014 (release date) till 24 March 2015. The
Altmetrics score for this review was 36 and included 32 tweets
from 28 accounts with a total reach of 28,480 combined
followers. Plotting the tweets along the same timeline as
the number of downloads did not reveal a clear pattern.
While the number of downloads only slightly increased after
publishing the blog, the Altmetrics score was among the
top 5% of the scores in the Cochrane Library. Conclusions:
While it is very early to evaluate our strategy beyond extended
reach, partnering with institutions outside Cochrane improves
dissemination efforts. It is feasible to use review downloads
and Altmetrics scores as performance measures for review
dissemination.
O 6.4
RITES: a new systematic review tool to assess
the characteristics of included trials along
the efficacy/effectiveness continuum
Wieland LS1 , Berman BM1 , Altman DG2 , Barth J3 , Bouter LM4 ,
D’Adamo C1 , Grimshaw J5 , Linde K6 , Moher D7 , Mullins D1 ,
Treweek S8 , van der Windt D9 , Merrick Z5 , Witt C6
1
Cochrane Complementary Medicine Field, USA; 2 Cochrane Bias
Methods Group, United Kingdom; 3 Cochrane Complementary
Medicine Field, Switzerland; 4 Cochrane Complementary Medicine
Field, The Netherlands; 5 Cochrane Effective Practice and
Organisation of Care Group, Canada; 6 Cochrane Complementary
Medicine Field, Germany; 7 Cochrane Bias Methods Group,
Canada; 8 Cochrane Methodology Review Group, United Kingdom;
9 Cochrane Complementary Medicine Field, United Kingdom
50
Background: Trials may address efficacy, i.e. intervention
effects under ideal conditions, or effectiveness, i.e.
intervention effects in real world circumstances. When
systematic reviews are used to inform clinical or policy
decisions, it is important to understand how applicable the
evidence from the included trials is to the decision-maker’s
context. Authors of systematic reviews can help by explaining
where included trials are on the efficacy/effectiveness
continuum. Objectives: To develop a valid, comprehensive
tool that informs systematic review readers about the
placement of individual trials on the efficacy/effectiveness
continuum. Methods: We identified all relevant existing
tools, then extracted the domains and the advantages and
limitations of each tool as described in the literature. We
conducted two online surveys with 72 stakeholders and
organized two teleconferences and written consultations
with 10 methodological experts to condense the domains
and develop a draft of the tool. The feasibility and
inter-rater reliability of the tool is being tested by piloting
the tool with 10 trials included in three recent Cochrane
complementary medicine reviews. The tool will be refined
after piloting. Results: The RITES (Rating Included Trials
along the Efficacy-effectiveness Spectrum) tool consists of
four domains: (1) participant characteristics; (2) trial setting;
(3) flexibility of interventions; and (4) clinical relevance of
interventions. Evidence from the trial is rated in each domain
on a 5-point Likert scale along a continuum from maximum
emphasis on efficacy to maximum emphasis on effectiveness.
Piloting is ongoing and results will be presented. Conclusions:
RITES will help users of systematic reviews assess the ways
in which the evidence from included trials provides pertinent
information about effectiveness or efficacy. This will aid in
the appropriate application of systematic review evidence in
clinical practice or policy decisions.
RO 7.1
How often is a Cochrane review the best
available evidence? Analysis of the more
relevant clinical questions
Rada G1
1
Epistemonikos Foundation, Chile
Background: Relying on Cochrane Reviews (CRs) is
recognized as an efficient way of making evidence-based
health decisions. Objectives: To explore how often CRs
are available for clinical questions; if they outperform other
systematic reviews (SRs); and if they include all the relevant
evidence. Methods: In the context of a project aiming to
create friendly summaries of evidence, we established groups
composed of at least one content expert in 15 departments of
the faculty of Medicine of the Pontificia Universidad Católica
de Chile. Each team prioritized at least five questions about
interventions that were considered hot topics in the area.
We prepared matrices of evidence in Epistemonikos (a table
comparing all of the reviews and all of the primary studies
for a question). We measured the number of available SRs
for each question; the number of questions answered by a
CR/non-Cochrane review (NCR); and the number of times a
CR was the newer/more complete. Results: Fifty-two clinical
questions have been summarized at the moment. The average
number of SRs per question was five (range 0 to 26). For 43/52
(83%) we identified at least one SR, but only 24/52 (46%) were
answered by a CR. For the 24 questions for which a CR was
available, in 16/24 (67%) the CR was the more complete, and
in 12/24 (50%) it was the more recent. A total of 138/172
(80%) of the studies included in the CRs are included in other
reviews. These results will be updated one month before
the Colloquium. Conclusion: A substantial proportion of
questions can be answered with SRs. However, non-Cochrane
reviews (NCRs) had better coverage than CRs. When there
were both CRs and NCRs, CRs were more complete, but there
is ample room for improvement. A large proportion of the
included studies in a CR can be obtained from existing reviews.
Our results emphasize the need for producing reviews covering
the more relevant topics and the challenges for Cochrane to
be the more reliable source of evidence. Using existing SRs
might be an efficient way to filter the information overload
and to position CRs as the best available evidence.
RO 7.2
Discrepancies in outcome reporting exist
between protocols and published oral health
Cochrane Reviews
Pandis N1 , Fleming PS2 , Worthington H3 , Dwan K4 , Salanti G5
1
University of Ioannina & University of Bern, Switzerland; 2 Barts
and The London School of Medicine and Dentistry, Queen Mary
University of London, UK; 3 Cochrane Oral Health Group, School
of Dentistry, The University of Manchester, UK; 4 Department of
Biostatistics, University of Liverpool, UK; 5 Department of Hygiene
and Epidemiology, University of Ioannina School of Medicine,
Greece
Background: In systematic reviews, selective reporting may
develop for a variety of reasons, for example, due to the
use of multiple measurement scales, outcomes or time
points and selective inclusion of specific outcomes. There
is paucity of information on selective reporting based on
comparisons between review protocols and final systematic
review reports. Objectives: To assess discrepancies between
protocols and published reviews in oral health systematic
reviews (COHG) on the Cochrane Database of Systematic
Reviews (CDSR). Methods: Information on the reported
outcomes from all COHG systematic reviews in CDSR and
the corresponding protocols was recorded by two reviewers
independently. Results: One-hundred and fifty-two reviews
were included; the median number of labelled primary and
secondary outcomes was two (range: 0 to 11) and four
(range: 0 to 36) respectively both in the protocols and reviews.
For primary outcomes, 11.2% (17/152) were downgraded
51
to secondary outcomes, 9.9% (15/152) were omitted and
18.4% (28/152) were introduced in the final publications.
For secondary outcomes, 2.0% (3/152) were upgraded to
primary, 12.5% (19/152) were omitted and 30.9% (47/152)
were newly introduced in the publication. Overall, 45.4%
(69/152) of reviews had at least one discrepancy that was
justified in 14.5% (10/69) reviews. Sixty-three reviews included
meta-analyses. For primary outcomes the risk of reporting
significant results was lower for both downgraded (RR 0.52,
95% CI 0.17 to 1.58; P value 0.24) and upgraded or newly
introduced outcomes (RR 0.77, 95% CI 0.36 to 1.64; P value
0.50) compared to outcomes with no discrepancies. For
primary outcomes the risk for reporting significant results
was higher for upgraded or newly introduced outcomes
compared to downgraded outcomes (RR 1.19, 95% CI 0.65
to 2.16; P value 0.57). None of the comparisons reached
statistical significance. Conclusion: There is evidence that
discrepancies between outcomes of pre-published protocols
and final reviews continue to be common, on the basis of
this analysis of SRs published within the COHG. Alternative
solutions to reduce the prevalence of this issue may need to
be explored.
RO 7.3
Shedding light on the maze of study labels
and features: analysis of CBA and ITS studies
in Cochrane Systematic Reviews
Polus S1 , Pieper D2 , Fretheim A3 , Burns J1 , Ramsay C4 ,
Pfadenhauer L1 , Rehfuess E1
1
Institute for Medical Informatics, Biometry and Epidemiology,
University of Munich, Germany; 2 Institute for Research in
Operative Medicine, Department for Evidence-Based Health
Services Research, Witten/Herdecke University, Germany; 3 Global
Health Unit, Norwegian Knowledge Centre for the Health Services,
Oslo, Norway; 4 Health Services Research Unit, University of
Aberdeen, Foresterhill, Aberdeen, United Kingdom
Background: Increasingly, Cochrane Systematic Reviews
draw on nonrandomized study designs when assessing the
effectiveness especially of complex interventions. In this
context, controlled before-after (CBA) and interrupted time
series (ITS) studies receive particular attention due to their
recognition by the EPOC Group. There is, however, much
confusion as a result of inconsistencies in study design labels
and features and in relation to risk of bias. Objectives:
To analyse CBA and ITS studies included in Cochrane
Systematic Reviews descriptively in terms of their publication,
application and methodological characteristics, focusing on
strengths and weaknesses of the two designs. Methods:
We searched the Cochrane Library for systematic reviews
including nonrandomized studies from June 2012 to March
2015, updating a methodological study by Ijaz 2014. First, we
classified the reviews including CBA or ITS studies according
to ten pre-specified types of intervention. Subsequently, and
separately for ITS and CBA studies, we purposively selected
two reviews from each type of intervention. From each of
these reviews, we randomly sampled two CBA or two ITS
studies, respectively. Data extraction and analysis were
conducted separately by two authors using distinct tools
for the two designs. Results: Our searches identified 126
reviews. From the sample of 40 CBA and 40 ITS studies, we
described publication details, field of application (i.e. type
and level of intervention), methodological characteristics
(i.e. data collection, analysis), and quality assessment. The
descriptive analysis yielded notable differences in relation to
field of application of the two designs across health policy,
health systems, behavioural, environmental, occupational,
clinical, pharmaceutical, nutrition, screening and vaccination
interventions and uncovered differences in quality of study
conduct and reporting. Conclusions: This study provides
further proof of heterogeneity in definitions and study features
associated with the ’CBA’ and ’ITS’ labels. The strengths and
weaknesses of the two designs have important consequences
for reporting guidelines and quality appraisal tools.
RO 7.4
Systematic review of adverse effects of
antidepressants in healthy volunteer studies
Bielefeldt AØ1 , Danborg PB1 , Gøtzsche PC1
1
The Nordic Cochrane Centre, Denmark
Background: Clinical trials are often not published and
adverse effects are often under-reported.
Access to
unredacted clinical trials is vital for independent researchers.
In 2011 the Nordic Cochrane Centre got access to unredacted
antidepressant trials from the European Medicines Agency
(EMA), including 29 trials conducted in healthy volunteers.
We wished to examine adverse effects (AEs) of SSRIs in
healthy volunteers. Objectives: 1. A systematic review
and meta-analysis of AEs of selective serotonin reuptake
inhibitors in healthy volunteer studies, particularly activating
effects and other AEs that can predispose to suicide. Specific
for EMA trials: 2. A comparison of unredacted CSRs with
published articles to investigate the selectivity of reporting. 3.
A comparison of protocols with CSRs to investigate if protocol
planned assessments were carried out. Methods: Searching
for supplemental studies on PubMed and Embase for the
systematic review. Eligibility criteria were set up. Studies were
screened and relevant data extracted. Descriptive statistics
were used to describe the included studies, and Peto OR for
meta-analysis. Results: A systematic review of ∼150 healthy
volunteer studies on antidepressants showed under-reporting
of AEs. About one-third of studies did not mention AEs or
their absence; 14 studies were eligible for meta-analysis, this
showed that the incidence of AEs doubled when compared
to placebo (Peto OR 2.20, 95% CI 1.35 to 3.59; P < 0.002).
Only half the EMA trials (N = 29) were published (N = 15), and
a mere few of these detailed AEs data fully (N = 2). Nine
published articles had investigated AEs in their CSRs, but less
than half (N = 4) reported these AEs, though these four studies
all reported AEs partially. Result for objective 3 is pending.
Conclusions: We investigated the AEs of antidepressants in
52
healthy volunteer studies. A systematic review of about 150
studies shows that about a third of all trials omit AE data and
the meta-analysis shows that the incidence of the AEs possibly
predisposing to suicide is doubled. Analysis of the EMA trials
shows selective publication and inadequate reporting of AEs.
RO 7.5
Agreement in results data between
conference abstracts and full reports of
randomized controlled trials: should we
depend on conference abstracts?
Saldanha I1 , Scherer R1 , Dickersin K1
1
Cochrane Eyes and Vision Group, USA
Background: Including conference abstracts in systematic
reviews promotes comprehensiveness, but conference
abstracts are not usually peer-reviewed and often contain
preliminary data. Conclusions about intervention efficacy in
randomized controlled trials (RCTs) are generally based on
primary outcomes (POs). Objective: To quantify agreement
between PO results of RCTs presented as conference abstracts
and their corresponding full reports. Methods: We included
all abstracts of RCTs presented at the 2001-2004 Association
for Research in Vision and Ophthalmology conferences. We
identified corresponding full reports through to 2013 by
electronic searching and emailing authors. We extracted data
about the PO from each abstract and full report. For each
abstract-full report pair where the PO was the same and
direction of results (positive or neutral) could be determined
in both publications, we examined whether the results agreed.
We classified any discordance as quantitative (any change in
magnitude but not direction of effect), qualitative (change
in direction of effect), or both. Results: Two-hundred and
thirty (44.8%) of the 513 eligible abstracts had been published
in full. Within these 230 abstracts, direction of results for
the PO was positive for 64 (27.8%), neutral for 55 (23.9%),
and could not be determined for 119 (51.7%). POs differed
in 82/230 pairs (36.9%). POs were more likely to differ
if the abstract’s results were neutral compared to positive
(RR = 1.28, 95% CI 1.15 to 1.40). Among the 103 abstract-full
report pairs in which the PO was the same and direction of
results could be determined, results agreed in 20 (19.4%),
there was quantitative discordance in 74 (71.9%), and both
quantitative and qualitative discordance in 9 (8.7%; Figure 1).
Conclusions: POs in more than one-third of RCTs presented
as abstracts differed from POs in the full reports; a difference
was more likely if the direction of PO results in the abstract
was neutral. When the POs were the same in both, only
one-fifth of pairs agreed, and almost one-tenth of pairs had
qualitatively different results. Systematic reviewers should be
aware of, and cautious about, these differences.
53
RO 7.6
Methods for data extraction from figures in
protocols of Cochrane Systematic Reviews
Vucic K1 , Jelicic Kadic A2 , Puljak L2
1 Agency for Medicinal Products and Medical Devices, Croatia;
2
Cochrane Croatia, Croatia
Background: When conducting a systematic review, some
data that need to be extracted from primary studies are
presented only as figures. One can contact study authors to
request original data from figures, but those requests for data
seldom yield a response. In those circumstances data need to
be extracted from figures. However, the Cochrane Handbook
does not cover issues relating to data extraction from figures.
Objectives: To analyze whether protocols of Cochrane
Systematic Reviews mention whether and how the authors
will extract data from figures in included trials. Methods:
Cochrane Protocols published between May 2013 and May
2014 were screened by two authors independently and the
following data were collected: date of protocol publication,
country of authors’ origin, number of authors, number of
affiliated institutions, Cochrane Review Group, whether the
protocol contains description of data extraction from figures,
the method of data extraction from figures, and literature
references for a method of data extraction from figures.
Results: From a total of 589 protocols, 33 (5.6%) mentioned
data extraction from figures in the Methods section. Most of
those studies mentioning data extraction from figures (18/33)
indicated that data from figures would be included only if
two reviewers independently extracted the same result. Only
one protocol specified that computer software would be used
for data extraction from figures, and few stated estimation or
approximation, while others did not provide any description of
methodology for data extraction from figures. One protocol
specifically indicated that data from figures would not be
used, ‘‘because of [the] possibility of making measurement
errors when estimating from graphs’’. Conclusion: Very
few Cochrane Protocols mention data extraction from figures
and, even when they do mention data that are presented
only in figures, it is not necessarily to explain how those data
will be extracted. Some even plan to disregard those data
completely. The Cochrane Handbook and methodological
standards for Cochrane Systematic Reviews should address
methodology for extracting data from figures.
RO 7.7
Reliability of data extraction using manual
measurements of published graphs or figures
Pérez-Gaxiola G1 , Cuello-Garcia C2
1 Cochrane México, Hospital Pediátrico de Sinaloa, Mexico;
2
University, Canada
Background: Meta-analyses combine results of specific
outcomes from individual primary studies, in which trial
authors occasionally publish results with figures or graphs
without the exact numerical data needed for further analysis.
Although the most appropriate way to deal with missing
numbers is to contact authors, sometimes this is not feasible.
One alternative is to calculate manually by extracting numbers
using measurements in figures as a proxy. Objectives:
To analyse the reliability/accuracy of data extraction using
manual measurements from figures published in individual
studies. Methods: Four figures illustrating a clinical outcome
without an exact numerical value embedded in them were
selected. The corresponding published study did contain
an exact value that we used as gold standard. We asked
participants to extract from the figures the value of the
clinical endpoint of interest using ruler measurements within
a PowerPoint slide as a proxy. Raters were blinded to the gold
standard. We measured inter-rater reliability of participants
using Type A intra-class correlation (ICC) coefficients with
absolute agreement definition; we also compared the mean
from the raters for each figure to their correspondent gold
standard. Results: Twenty-nine raters agreed to participate.
In the four figures, less than 1% of the measurement
corresponded to random variation (ICC coefficient 0.99, 95%
CI 0.96 to 1.0). Only in one figure there was a statistically
significant difference between the participant-estimated
mean and the gold standard. Conclusions: Measurement
of an estimate of interest that is not displayed on figures
from individual studies could be extracted from them by
hand using any presentation program that a methodologist
is familiar with. As with any extraction process in a review,
extraction by duplicate is warranted to ensure exactitude of
the final estimate.
RO 7.8
A graphical display of reporting factors
associated with systematic reviews: an
example from diabetic retinopathy
Hui X1 , Clearfield E1 , Lindsley K1 , Virgili G2 , Scherer R1
1
Cochrane Eyes and Vision Group, Johns Hopkins University, USA;
2 Cochrane Eyes and Vision Group, University of Florence, Italy
Background: Systematic reviews (SRs) are considered
to be the highest level of evidence, but may lead to
biased conclusions if not properly conducted. We aim
to evaluate SRs of interventions for diabetic retinopathy
(DR), the leading cause of blindness in the working-age
population in developed countries. Objectives: To assess
the descriptive, methodological, quantitative, qualitative,
and other characteristics of SRs of interventions for DR.
Methods: We searched PubMed/MEDLINE, EMBASE and the
Cochrane Central Register for Controlled Trials (CENTRAL)
up to May 2014 for all SRs related to eyes and vision with
a focused question and explicit scientific approach. We
identified all SRs that examined an intervention for DR. Two
authors independently evaluated 15 characteristics adapted
from the Critical Appraisal Skills Programme (CASP) and A
Measurement Tool to Assess Systemic Reviews (AMSTAR;
54
Figure 1
55
Table 1). We noted whether each item was reported or not
reported/can’t tell and calculated the proportion of SRs
reporting that characteristic. We plotted the calculated
proportions using a radar plot in MS Excel (Figure 1). Results:
We identified 25 SRs published from 2002 to 2014; four were
Cochrane SRs. The SRs included a median of six studies
(range: 0-11). Common interventions included anti-vascular
endothelial growth factor (VEGF) treatments (14 trials) and
corticosteroids (10 trials). The majority of SRs presented
eligibility criteria (96%), however only 36% reported that
the criteria were pre-specified. Only 56% of searches were
comprehensive. A risk of bias assessment was performed
in 52% of SRs, and 61% discussed limitations. Seventy-nine
per cent used correct methods when results were combined
quantitatively. Non-industry or no funding was reported
by 56%. Conclusions: The radar graph visually displays
substantial gaps in the reporting of descriptive factors
and appropriate SR methodology for SRs in DR. Reporting
of quantitative factors was slightly better compared with
qualitative methods, such as discussing limitations.
types. Outcomes related to clinical or nutritional status
assessment were most frequent (n = 385; 96%), followed
by disease incidence or prevalence (n = 199; 49%). Nonrandomized studies were included in only three reviews
(< 1%) and GRADE was utilized in 85 (21%). The quality
assessment is in progress and will be reported at the
Colloquium. Conclusions: A relatively small proportion of
Cochrane Reviews address nutrition-related interventions.
These are mainly concentrated in two CRG’s, with remaining
reviews thinly spread across other groups. Current reviews
focus on nutritional supplements and clinical or nutritional
status outcomes. These findings can inform Cochrane
actions to improve the quantity, quality and relevance of
nutrition-related reviews for better nutrition policies and
practice.
Evidence maps and priority setting
RO 8.1
RO 7.9
Scope and quality of Cochrane Reviews of
nutrition-related interventions
Harper A1 , Naude CE2 , Durao S3 , Volmink J4
1
Division of Human Nutrition, Stellenbosch University, South Africa;
2 Centre for Evidence-based Health Care, Stellenbosch University,
South Africa; 3 South African Cochrane Centre, South African Medical
Research Council, South Africa; 4 South African Cochrane Centre,
South African Medical Research Council; Centre for Evidence-based
Health Care, Stellenbosch University, South Africa
Background: Despite the importance of nutrition for health
and development, a limited number of nutrition-related
Cochrane Reviews exist. These reviews are of uncertain
quality and relevance. Objectives: To assess the scope
and methodological quality of nutrition-related reviews
in the Cochrane Database of Systematic Reviews (CDSR).
Methods: We screened the CDSR (February 2014) to
identify nutrition-related reviews and protocols according to
pre-specified eligibility criteria. Two authors independently
performed data extraction (PICOS) and quality assessment
(AMSTAR) of eligible reviews. We report our findings using
statistical and narrative description. Results: We screened
8217 records, which yielded 536 (6.5%) relevant reviews
(n = 400) or protocols (n = 136) distributed across 45 Cochrane
Review Groups (CRGs). Eight CRGs had no nutrition-related
reviews. The median number of nutrition-related reviews
across all CRGs was four (interquartile range: 1 to
9).
CRGs with the most reviews were Pregnancy &
Childbirth (n = 64/527), Neonatal (n = 60/303), Metabolic
& Endocrine Disorders (n = 26/100), and Developmental,
Psychosocial & Learning Disorders (n = 23/121). The most
common type of intervention evaluated was nutritional
supplements (n = 232), followed by food/dietary patterns
(n = 139), with some reviews including several intervention
EBRNetwork –a call to action for more
(efficient) systematic reviews
Nasser M1 , Westmore M1 , Lund H2 , Brunnhuber K1 , Dobbins M3 ,
Robinson K4 , Leenars M5 , Nykvist H6 , Christensen R2 , MacLeod
M1 , Dorch B2 , Ford P1
1 Evidence-Based
Research Network,
United Kingdom;
2
Evidence-Based Research Network, Denmark; 3 Evidence-Based
Research Network, Canada; 4 Evidence-Based Research Network,
USA; 5 Evidence-Based Research Network, The Netherlands;
6
Evidence-Based Research Network, Norway
Background: Initiating new research projects without
reviewing the existing evidence systematically is wasteful,
unscientific, and potentially unethical. Scientists are expected
to refer to earlier research results when they argue for the need
for a new study. However, research-on-research shows that
the papers cited are often insufficient and biased towards
the interest of the researchers; that researchers are not
supported (through funding, time, training) in the production
and updating of systematic reviews; and that there is a
need for new ways of conducting mandatory systematic
reviews that are more efficient yet rigorous. Objectives:
To address these problems, a group of researchers have
initiated an international network, the ‘Evidence-Based
Research Network’ (EBRNetwork - www.ebrnetwork.org),
established in Bergen, Norway in December 2014. Methods:
We are presenting the findings of several relevant studies,
including the use of previous research by scientists; the
problem of ongoing research after a benefit or harm of an
intervention has been unequivocally established; and an
international comparison of research funders, regulators and
publishers regarding policies mandating systematic reviews
prior to new research. We issue an invitation to join the
EBRNetwork and will work with the members of the audience
toward identifying and prioritising key initial workstreams.
Results: Researchers embarking on research after reviewing
56
the existing evidence systematically will be able to clarify
whether the study is truly needed (adding value to health
care),or whether it is asking a research question for which
we have already have high quality evidence –and thus could
be considered wasteful. Conclusions: The new EBRNetwork
is an international collaboration that aims to ensure that no
new studies are conducted without prior systematic review,
and that works towards more efficient production, updating
and dissemination of systematic reviews. The Network issues
a call to participate in developing a consensus statement to
accomplish these aims.
RO 8.2
Evidence gap maps –a tool for promoting
evidence informed policy and prioritising
future research
Snilstveit B1 , Vojtkova M1 , Stevenson J1 , Bhavsar A1 , Gaarder M2 ,
Phillips D1 , Gallagher E1
1 International Initiative for Impact Evaluation, United Kingdom;
2
World Bank, United Kingdom
Background: A range of organisations are engaged in the
production of evidence on the effects of social and economic
development interventions. The growth in the production of
evidence also presents challenges. How can decision makers
avoid information overload and access the best available
evidence in a specific field? How can we present research
in a format that is useful and accessible for a non-technical
audience? And how can we ensure limited resources are
spent efficiently and important evidence gaps are prioritised?
Objectives: Evidence gap maps (EGMs) aim to respond to
these challenges and present a new addition to the tools
available to support evidence-informed policy making. The
objective of this project was to develop a method for evidence
mapping and visualisation to aid knowledge translation and
strategic research prioritisation. Methods: We reviewed the
methodological literature on evidence mapping and synthesis
to identify existing methods to identify, appraise and present
evidence from primary studies and systematic reviews. We
designed a visual framework for presenting the evidence and
worked with web developers to create an interactive and
visual platform. Results/Conclusions: EGMs provide a visual
overview of what we know and don’t know about the effects
of interventions in a sector or sub-sector. EGMs are based on
systematic methods and are structured around a framework
which schematically represents the types of interventions
and outcomes of relevance to the thematic area. They enable
policy-makers and practitioners to explore the findings and
quality of the existing evidence to facilitate evidence informed
decision-making. EGM also identify important ’gaps’ where
little or no evidence from impact evaluations and systematic
reviews is available, and can inform a strategic approach to
building the evidence base in a particular sector. The paper
provides an introduction to the concept of evidence gap maps
and their methodology, and a demonstration of the EGM tool
using existing examples.
RO 8.3
Evidence gap maps for eye health: tool for
promoting evidence and gaps in research
Virendrakumar B1 , Jolley E1 , Gordon I2 , Bascaran C3 , Pellegrino
C1 , Schmidt E1
1 Sightsavers, United Kingdom; 2 Cochrane Eyes and Vision Group,
London School of Hygiene & Tropical Medicine, United Kingdom;
3
London School of Hygiene & Tropical Medicine, United Kingdom
Introduction:
Sightsavers
is
an
international
non-governmental organisation (NGO) working to eliminate
avoidable blindness in low- and middle-income settings
(LMIS). High quality evidence is essential for planning
and delivering our programmes, however, evidence on
what works to improve eye health in LMIS is relatively
scarce and there is no single repository where all relevant
research may be found. To address this need, Sightsavers is
developing eye health Evidence Gap Maps (EGMs) using the
methodology developed by the International Initiative for
Impact Evaluation (3ie). EGMs summarize, critically appraise
and present evidence from systematic and literature reviews
on a particular topic in a user-friendly format. The quality
appraisal is integrated into the visual presentation of the EGM
and shows where the strengths and weaknesses of evidence
lie. We will present our experiences of developing the cataract
EGM including the challenges and benefits along with the
initial evaluation of its potential to inform the development
of EGMs for other eye diseases. Methods: Following a
comprehensive search of relevant databases, websites and
reviews of references, we sifted, and extracted data from
all relevant reviews on cataract. Critical appraisal was
conducted by two reviewers independently using Supported
Use of Research Evidence (SURE) checklist and a summary of
quality assessment was shared with the authors for comment.
Results: Out of 1197 unique studies identified, 52 reviews met
the inclusion criteria. Most reviews (18) addressed surgical
methods for cataract, followed by quality of clinical care (17).
Of the remainder, seven looked at risk factors/prevention, five
studies were impact/economic evaluations, and three each
addressed the burden of the disease and the accessibility of
cataract services. Twenty-five studies were graded as being
of low methodological quality, and 13 as being high quality.
Discussion: EGMs will enable decision makers to make more
informed choices by presenting the effectiveness of eye
health interventions in a simple visual format. They will also
enable researchers to prioritise areas where little evidence
exists.
57
RO 8.4
Accountability and health information
systems: from health systems’ priority
setting to a pragmatic systematic review
Leon N1 , Brady L1 , Kwamie A2 , Lewin S3 , Odendaal W1 , Daniels K1
1
South African Medical Research Council, South Africa; 2 School
of Public Health, University of Ghana, Ghana; 3 Norwegian
Knowledge Translation Centre for Health Services and Health
Ssytems Research, Norway
Background: The Alliance for Health Policy and Systems
Research recently funded a two-year initiative in South
Africa to conduct four systematic reviews on health systems
questions. Central to the initiative was engagement with
stakeholders to identify and select priority review questions.
While the priority setting process was rich and very broad,
these broad questions were not always easily translated into
questions that could be addressed by a review with limited
budget and timeframes. Objective: To share lessons learned
in turning a broad health systems policy priority question
about effective accountability interventions, into a pragmatic
systematic review question. Results: Through the process
of stakeholder engagement 39 priority review questions were
shortlisted. The four highest-ranking questions for which
no published or ongoing review existed were selected for
further investigation. One of these questions focused on the
effectiveness of interventions to improve the accountability
of public sector health personnel and management. While
enhanced accountability is important to policy makers and
managers, we soon realised the broad and abstract nature
of this concept raised challenges in relation to definition,
operationalisation and measurement. Through a process of
literature review, consultation and discussion we explored
the components of accountability, and how it might be
operationalised within the health system. This process
led us to shift our focus to health systems performance
monitoring, as a proxy measure of accountability. The
result was a revised review question on the effectiveness
of interventions to improve health information systems.
Improved health information systems are an essential tool
for measuring progress and accountability and for filtering
information for better decision-making. Conclusion: Policy
priorities as articulated by stakeholders may not be easily
translated into systematic review questions. Thus flexibility
and ongoing dialogue are required between researchers
and policy stakeholders when identifying health systems
systematic review topics collaboratively.
RO 8.5
Evidence gaps in health policy and systems
research in 15 countries of the Eastern
Mediterranean Region: a mapping exercise
El-Jardali F1 , Bou-Karroum L1 , Akl E1
1 American University of Beirut, Lebanon
Objective: This study aims to map the gaps in the production
of Health Policy and Systems Research (HPSR) evidence in
15 countries of the Eastern Mediterranean Region (EMR). The
study also assesses the alignment between the existing HPSR
and high-level regional and global priorities. Methods: We
searched MEDLINE for articles on HPSR published between
2000 and 2013 in the 15 EMR countries (Bahrain, Egypt, Jordan,
Kingdom of Saudi Arabia, Kuwait, Lebanon, Libya, Morocco,
Oman, Palestine, Qatar, Sudan, Syria, Tunisia, and Yemen).
Teams of two reviewers assessed eligible articles using a
coding form that included HPSR themes on governance,
financial and delivery arrangements and implementation
strategies. We then matched articles identified as HPSR
to regional and global priorities pertaining to 11 themes
including health human resources, health financing, role of
non-state sector, access-to-medicine, primary healthcare,
non-communicable diseases, universal health coverage,
emergency preparedness and response. Results: Out of the
29,126 articles published in the 15 EMR countries, 9% fitted
the criteria for HPSR. There was an increase in production
after 2005. HPSR articles focused on themes of delivery
arrangements (68%) and implementation strategies (19%).
The evidence gap was noted in the themes of financial (3%)
and governance arrangements (9%). We found misalignment
between HPSR produced in the region and the regional and
global priorities. Conclusions: The mapping of evidence
gaps persistently showed low production of HPSR in the
region. The mapping should inform the research agenda (for
researchers and funders) for the field, including identifying
review questions for systematic reviews. It also demonstrates
the need for aligning the production of HPSR with policy needs
and priorities, and for building the capacity in conducting
HPSR at the individual, team, institutional and system level.
RO 8.6
Establishing and prioritising a local health
research agenda with end users
Moore D1 , Abbott R1 , Rogers M1 , Bethel A1 , Stein K1 ,
Thompson-Coon J1
1
Background: The National Institute for Health Research
Collaboration for Leadership in Applied Health Research
and Care in the South West of England (PenCLAHRC) is
developing a portfolio of clinically relevant, locally tractable
and patient-informed research projects through a process
involving a wide group of end users at all stages, from
the inception of research ideas through to delivery of
outputs. Objectives: This paper describes and evaluates
the research prioritisation process being used by PenCLAHRC;
a partnership of local National Health Service (NHS) and
public health organisations across Somerset, Devon and
Cornwall, with the Universities of Exeter and Plymouth.
Methods: PenCLAHRC identifies research questions from
its partners, including members of the public. Questions
are identified during engagement with service users, clinical
58
teams and organisations around the use of evidence and
via the PenCLAHRC website.
Questions received are
prioritised in a process involving all PenCLAHRC stakeholders
(including our Peninsula Patient and Public Involvement
group). Prioritisation is based on a set of explicit criteria
including importance, local relevance and feasibility. In
2014 we piloted a novel approach consisting of two rounds of
electronic voting, before a face-to-face meeting to discuss and
rank the prioritised questions. Results: Seventy-two initial
questions were reduced to 50 and then to nine in two electronic
rounds of comments and voting. Priority briefings for the nine
questions were prepared and were discussed and voted on
at a face-to-face meeting in December 2014. The process
has been evaluated by stakeholders. Feedback indicates that
stakeholders found voting electronically, sharing comments
on questions and more than one round of voting valuable.
Issues regarding the time available for prioritisation activities
and the quality of some identified questions were raised.
Conclusions: Engagement of end users in establishing a
research agenda increases its relevance and may promote use
of research. With minor changes, future rounds of question
prioritisation will build upon the process outlined here.
RO 8.7
A priority setting tool for prioritizing review
questions for systematic reviews in health
policy and systems research: development
and validation
Akl E1 , El-Jardali F1 , Fadlallah R1
1
American University of Beirut, Lebanon
Background: Groups conducting systematic reviews in health
policy and systems research need to prioritize topics according
to the needs of policymakers and stakeholders. Such
prioritization can help ensure systematic review production
is aligned with policy priorities, which, in turn, can promote
their use in policy making and practice. A systematic search
did not identify a tool to support such prioritization process.
Objectives: The aim of our study is to develop and validate a
tool for prioritizing questions for systematic reviews in health
policy and systems research. Methods: The development and
validation methodology includes the following three stages:
1. definition of the purpose and scope of the too; 2. tool
development/item generation; and 3. validation of tool. The
latter stage involves the following steps: 1. establishing
content and face validity; 2. pilot testing of tool in real life
priority setting exercises with policymakers and stakeholders
from different contexts; 3. exploratory factor analysis; 4.
reliability testing; 5. external construct validity; 6. refinement
and finalization of tool; and 7. translation of tool into different
languages. Results: We have completed the first two stages,
and are currently working on the second step of the final
stage. The tool includes two modules: one (14 items) to be
completed by policymakers and stakeholders; and the second
(nine items) to be completed by systematic reviewers. In the
presentation, we will provide a detailed description of the
’finalized’ tool, the process and outcome of its development,
and how it could be used by systematic review teams and
centers. Discussion will also delve into lessons learned and
implications for other groups that conduct priority setting
exercises for systematic reviews. Conclusions: The tool
might be helpful for organizations in both prioritizing their
systematic review work, and ensuring that they are used to
inform policy decisions.
RO 8.8
Priority setting for research and practice of
child health in low- and middle-income
countries (LMICs) based on evidence from
Cochrane Reviews
Sinha A1 , Ovelman C2 , Pradhan A1
1
Cochrane Neonatal Group, India; 2 Cochrane Neonatal Group,
Canada
Background: Systematic Reviews (SRs) identify, appraise and
synthesize research-based evidence and present it in an
accessible format for clinicians and healthcare practitioners.
The knowledge emerging from SRs may be of help to
health planners when setting priorities for health research,
implementing proven interventions, and using limited
resources judiciously. Objectives: To identify interventions
with high-level evidence and research leads/primary research
questions in child health emanating from Cochrane SRs
relevant to low- and middle-income countries (LMICs).
Methods: We searched six Cochrane Review Groups
(Acute Respiratory Infections (ARI); Infectious Diseases;
Neonatal; Cystic Fibrosis and Genetic Disorders; Airways;
Developmental, Psychosocial and Learning Problems (DPLP))
that produce reviews relevant to child health in LMICs for
published and updated Cochrane SRs from 1 March 2009 to 18
March 2015 in the Cochrane Library. We extracted data from
the objectives, main results and authors’ conclusions sections
of the Cochrane SRs to identify primary research questions.
Results: We found a total of 679 Cochrane SRs (105 from ARI,
178 from Airways, 100 from Cystic Fibrosis, 78 from DPLP, 63
from Infectious Diseases, and 155 from Neonatal) addressing
research questions of importance to LMICs. About 50% of
these Cochrane SRs concluded that there was insufficient
evidence for making valid conclusions (not enough RCTs,
small sample sizes in included studies). Data on interventions
with high-level evidence applicable to public health system in
LMIC, and also data on lead research questions (from reviews
with insufficient evidence) will be presented in the paper.
Conclusions: Cochrane Reviews with children as subjects
may be used to implement proven interventions, identify
knowledge gaps to guide future research as per national child
health priorities in LMICs facing resource constraints.
59
Meta-analysis methods
RO 9.1
The selection of fixed- or random-effect
models in recent published meta-analyses
Kuan Y1 , Tam K1
1
Shaung-Ho Hospital, Taipei Medical University, Taiwan
Background: Most meta-analyses are based on one of two
statistical models. A fixed-effect meta-analysis assumes all
studies are estimating the same (fixed) treatment effect,
whereas a random-effects meta-analysis allows for differences
in the treatment effect from study to study. The selection of
a model must be based on the question of which model
fits the distribution of effect sizes, and takes account of the
relevant sources of error. Objectives: The study aims to
evaluate the preference and selection of statistical model in
recent published meta-analyses. Methods: The published
meta-analyses were extracted from PubMed searches before
18 March 2015. We retrieved 60 studies and investigated
their selection of statistical models. Results: Six of these 60
studies did not report whether fixed- or random-effects was
used for meta-analysis. Random-effects pooling model were
conducted in 27 meta-analyses. Both fixed- and random-effect
models were used simultaneously in five studies. In another
22 studies, a fixed- or random-effect model was chosen
according to the heterogeneity.
For example, studies
with an I2 statistic of > 50% were considered to have
substantial heterogeneity, and therefore, a random-effects
model analysis was used. Otherwise, a fixed-effect model
was initially employed in the analysis. Interestingly, 21 of
the 60 meta-analyses were reported from China, and 15
of them selected fixed- or random-effect models according
to the heterogeneity. Conclusions: The fixed-effect model
starts with the assumption that the true effect size is the
same in all studies. However, in many systematic reviews
this assumption is implausible. Therefore, when studies are
gathered from the published literature, the random-effects
model is generally a more plausible match. Unfortunately,
36.7 % of our included studies starting their analysis with a
fixed-effect model and then moved to a random-effects model
if the test for heterogeneity was significant. The above strategy
is thought to be a mistake, and should be discouraged.
RO 9.2
Empirical comparison of two methods for
assessing interaction in aggregate-data
meta-analysis of randomized controlled
trials: a literature-based survey and analysis
Yang Z1 , Mao C1 , Tang J1
1 The Hong Kong Branch of the Chinese Cochrane Centre, China
Many treatments are more effective in some patients than
others, which means that the effectiveness of treatment
varies with some factors, usually called ’interaction’. There
are several methods available for investigating interaction
in aggregate-data meta-analyses. Suppose we want to
examine the potential interaction between sex and treatment,
using relative risk (RR) as the outcome measure. The best
method (’interaction term approach’) would be to calculate
an interaction term based on RR-male and RR-female within
each trial, and then combine the interaction terms across
trials to obtain a summary interaction term. However,
the majority of existing meta-analyses do not use this
method. Instead, they combined the treatment effects
in male subgroups across trials first (RR-maletotal), and
then combined the treatment effects in female subgroups
(RR-femaletotal), and then test for difference between
the two subgroup-specific summary estimates (’subgroup
approach’). This method can be easily implemented with
commonly available software for meta-analysis such as
RevMan. However, the subgroup approach is problematic.
First, it calculates the subgroup-specific summary estimates
first, thus ignoring the within-trial correlations between male
and female in the trials with data on both subgroups. This
would lead to reduced efficiency and inappropriate estimation
of standard errors. In addition, to compare the two summary
estimates of male and female subgroups, the heterogeneity in
the meta-analyses to obtain the two summary estimates has
to be ignored. For these reasons and based on the conceptual
comparisons, researchers have recommended avoiding using
the subgroup approach. In this study, we searched PubMed
and randomly selected 100 empirical meta-analyses to
re-evaluate the interactions, which were originally assessed
by the subgroup approach, by using the interaction term
approach. The results on interactions by the two methods
were compared both qualitatively and quantitatively. Study
characteristics associated with differences in the results
were described. Recommendations were made for future
meta-analyses. Details will be presented at the meeting.
RO 9.3
An investigation of the type I error rate when
testing for subgroup differences in the
context of random-effects meta-analyses
Guddat C1
1
IQWiG, Germany
Background: There are different approaches to test for
differences between two or more subgroups of studies in
the context of a meta-analysis. The Cochrane Handbook for
Systematic Reviews of Interventions refers to two methods.
One is a standard test for heterogeneity across subgroup
results rather than across individual study results. The
second is to use meta-regression analyses. Objectives:
Our aim was to compare the performance of these two
approaches with respect to the type I error rate when 10
or fewer studies are available. Methods: Assuming the
60
random-effects model, we have conducted a simulation study
for the planed comparison. Two versions of the test for
heterogeneity have been considered: one with separate
estimates of the between-study variance in each subgroup,
and one with a pooled estimate of this variance among
all subgroups. The meta-regression was conducted using
two modifications for the variance estimator of the regression
coefficients by Knapp and Hartung (2003). Besides the number
of studies, we varied, amongst other parameters, the extent of
heterogeneity between the studies, the number of subgroups
and the distribution of the studies to the subgroups. Results:
Both versions of the test for heterogeneity give extremely
high error rates when the heterogeneity between the studies
is large and the distribution of the studies to the subgroups
is unequal. In contrast, the error rates of the F-test using
either of the two considered variance estimators in the
context of a meta-regression are acceptable, irrespective
of the chosen parameters. Conclusions: Overall, the F-test
using the refined variance estimator by Knapp and Hartung
(2003; 1. modification) is the most appropriate choice out
of the evaluated tests with respect to the type I error rate
when the number of studies is 10 or fewer. References:
Knapp G, Hartung J. Improved tests for a random effects
meta-regression with a single covariate. Statistics in Medicine
2003: 22(17): 2693–2710.
RO 9.4
Meta-analyzing correlation coefficients
derived from cohort studies
Thumburu KK1 , Singh M2 , Kaur J1 , Singh S1 , Jaiswal N1 , Chauhan
A1 , Agarwal A1 , Paul N3 , Sagwal S3
1 ICMR Advanced Centre for Evidenced-Based Child Health,
Advanced Pediatric Centre, PGIMER,Chandigarh, India;
2
Department of Pediatrics and ICMR Advanced Centre for
Evidenced-Based Child Health, Advanced Pediatric Centre,
PGIMER,Chandigarh, India; 3 Department of Pediatrics, Advanced
Pediatric Centre, PGIMER,Chandigarh, India
Background: Correlation coefficient (r) is a measure of
strength and direction of linear association between two
variables or used as quantitative surrogate of individual
variables. There have been no or few studies in which
pooling of r values has been done and hence, we are
presenting a meta-analysis of non interventional cohort
studies by combining the r values.
Objectives: To
meta-analyze correlation coefficients derived from cohort
studies. Methods: We conducted a systematic review on
association of anthropometric measures during childhood
and risk of becoming overweight or obese in later life. The
studies included were prospective cohort or longitudinal
studies. Tracking estimates such as the correlation coefficient
are preferred compared to difference estimates when
assessing body-mass index (BMI) tracking. Basic information
recorded consisted of cohort size n, (mean) age at baseline,
and at follow-up measurement. The effect size calculated
was correlation coefficient (either Pearson’s or Spearman’s).
Standard error was calculated using the information from
cohort size and r values. Meta-analysis was done using inverse
variance random-effects model. Stata/MP 12.2 software) was
used to perform the meta-analysis. Results: Twenty-eight
studies provided correlation coefficient data on BMI tracking
from 40,219 individuals for meta-analysis. Follow-up time
ranged from two to 65 years. We derived standard errors from
the available sample size and r value. Then r values were
pooled using the inverse variance random-effects model. The
estimate size with confidence intervals provided the net effect.
Conclusions: Pooling of correlation coefficients is a useful
and feasible means of meta-analyzing quantitative data from
cohort studies. Acknowledgment: The study was supported
by World Health Organization, Geneva, Switzerland and ICMR,
New Delhi, India
RO 9.5
Joint models to account for informative
losses to follow-up in longitudinal studies
Brignardello-Petersen R1 , Singer L1 , Herridge M2 , Tomlinson G1
1
Institute of Health Policy, Management and Evaluation, University
of Toronto, Canada; 2 Toronto General Research Institute, University
Health Network, Canada
Background: A major challenge in the analysis of longitudinal
studies is data that are missing because participants’
follow-up ends before the planned follow-up period ends.
Many times, these data are missing not at random, and use
of conventional longitudinal models (e.g. linear mixed-effects
models) can lead to biased results for the coefficients relating
participants’ characteristics to outcomes. Joint models (JM)
that simultaneously estimate time-to-event and longitudinal
data models are a new alternative that account for informative
losses to follow-up in longitudinal studies. Objectives: To
explore whether the results of longitudinal studies with a
high proportion of informative losses to follow-up change
when the results are analyzed using JMs compared to linear
mixed-effects models. Methods: We reanalyzed the data from
two longitudinal studies with a high proportion of informative
losses to follow-up due to patient death, using JM. In the
first study the aim was to determine the trajectories of
quality of life of patients after receiving a lung transplant.
In the second study the aim was to assess the trends
in function of patients discharged alive from the ICU, and
determine whether these differed among risk groups. There
was approximately 35% and 25% of informative losses to
follow-up, respectively. The data was analyzed using the
package JM in R. Results: The change in the primary outcome
over time was statistically significant when analyzing the
datasets using linear mixed-effects models. In the second
example, the change in the primary outcome was also different
among groups. Although in both datasets, the change in the
slope over time was different when using JM to account for
the informative losses to follow-up, there were no important
changes in the statistical significance of the results for the
main variables of interest. However, in both cases, the
differences in the point estimates of the slopes were too small
61
to change the clinical conclusions drawn from the analysis.
Conclusions: In these examples, the clinical conclusions from
the longitudinal studies did not change when accounting for
the informative losses to follow-up using JM.
RO 9.6
The influence of mortality time-points on
pooled effect estimates in critical care
meta-analyses
Roth D1 , Herkner H1
1 Medical University of Vienna, Department of Emergency Medicine,
Austria
Background:
There is an on-going debate among
meta-analysis methodologists and statisticians whether it
is appropriate to pool mortality estimates from clinical
trials that use mortality outcomes ascertained at different
time-points. If the relative effects vary over time, which
might especially be the case in critical care, standard
pooling of studies with different follow-up times within one
meta-analysis would not be justifiable. Current Cochrane
guidelines propose pooling of short-, middle-, and long-term
effects as a potential solution, but include no specific
guidelines. Objectives: Describe the current practice of
dealing with different mortality time-points and analyze the
influence of different time points on pooled effect estimates
in actual Cochrane critical care meta-analyses. Methods:
The CDSR was searched for critical care-reviews. Review
characteristics including strategy for dealing with different
follow-up times and study characteristics were extracted.
Meta-analyses were recalculated using all described strategies
and influence of such strategies on deviation of pooled effect
estimates compared to a ”use last time-point available”
approach was analyzed using meta-regression and multilevel
mixed-effects linear regression. Results: We evaluated 835
reviews, and included 80 meta-analyses of 298 studies,
representing 107,605 patients. 49 (61%) reviews did not
state any strategy, 9 (11%) used separate analyses for each
time-point, 9 (11%) used the last available, 6 (8%) used a
closest to defined time-point, 3 (4%) performed separate
analyses for last and predefined, 2 (3%) mixed some, 1 (1%)
computed predefined time-points from study-data, and 1
(1%) pooled all but performed a sensitivity analysis. Among
388 recalculated meta-analyses no influence of the strategies
”pool short-, middle-, long-term”, ”use closest to defined”
and ”separate” on effect estimates was found compared to
”use last available”. Conclusions: Reviews use a large variety
of strategies to deal with different mortality time-points,
however more than 50% do not report any strategy. We
found no influence of different strategies on effect estimates
in critical care reviews.
RO 9.7
Multiple-arm trial data: using a corrected
standard error for GIV analyses
Cates C1
1 Airways Group, United Kingdom
Background: Sometimes multiple-arm trials only report the
pairwise difference between each arm and the control (with a
95% confidence interval (CI)). These paired differences can be
individually entered into Review Manager 5 using the Generic
Inverse Variance (GIV) method, but they cannot be combined
in the same forest plot, as the control arm would be counted
more than once. Objectives: To find a simple adjustment that
allows the paired comparisons to be combined in the same
forest plot by adjusting the standard error (SE). Methods:
From first principles it is possible to derive a formula for
adjusting the SE in each paired comparison in order to allow
pooling in a forest plot. The adjustment is based on the
assumption of equal variance in each of the trial arms. The
method should not be used if the size of each trial arm
is clearly unequal. Results: - If there are N active arms
in the study, the SE for each paired comparison should be
multiplied by the square root (SQRT) of (N+1)/2)). So for a
three-arm trial with two active arms and one control, the SE
for each of the two paired comparisons should be multiplied
by SQRT(3/2) = 1.225. - Where it is possible to calculate
the average of the mean differences (for example) across the
paired comparisons, and each paired comparison has a similar
95% CI, then the appropriate SE for the combined comparison
groups compared with control can be obtained (under the
same assumptions of equal variance across the arms) by
dividing the paired SE by SQRT ((N+1)/2N). Conclusions: A
simple method is available to allow pooling of data from
multi-arm trials in the same forest plot using GIV analysis.
However, if a random-effects meta-analysis is being used,
combining the active arms will give more weight to the study
data than separate paired comparisons.
RO 9.8
Meta-analysis of adverse events data using a
hierarchical Bayesian model: a case study
Mesgarpour B1 , Schmitz S2 , Walsh C3 , Herkner H4
1 National Institute for Medical Research Development (NIMAD),
Iran; 2 Department of Pharmacology and Therapeutics, Trinity
College Dublin, Ireland; 3 Department of Statistics, Trinity College
Dublin, Ireland; 4 Department of Emergency Medicine, Medical
University of Vienna, Austria
Background: Reporting adverse events of pharmacological
treatments is generally inadequate and incomplete in
randomized clinical trials (RCTs). The safety profile of
medications can be assessed more robustly in the long-term
follow-up of observational studies such as cohorts. Standard
methods of incorporating both study designs put much weight
on the usually large observational studies, despite their,
62
presumably, lower internal validity. Objectives: A case
study to combine safety data from RCTs and observational
studies into a hierarchical Bayesian model adjusting for
design-related weights. Methods: We set up a systematic
review on adverse events of erythropoiesis stimulating agents
(ESAs) in off-label indications in critically ill patients. Eleven
databases were searched up to April 2012. We considered
RCTs and controlled observational studies in any language
that compared off-label ESAs treatment with other effective
interventions, placebo or no treatment in critically ill patients.
We used frequentist and Bayesian models to combine studies,
and performed sensitivity analyses by using different weights
for the observational trials. Results: We included 48 studies
from 12,888 citations (34 RCTs; 14 observational). In order to
combine data from RCTs and observational studies, we fitted
a three-level hierarchical Bayesian model, which accounts
for between-trial design heterogeneity. ESAs increased
the risk for venous thromboembolism (VTE) in the RCTs
using frequentist analyses, but had no effect on VTE in
the observational studies, or when Bayesian methods were
applied. We found no statistically significant difference in
the mortality risk from treatment with ESAs compared to
non-ESAs in a Bayesian estimate of combining data from
RCTs and observational studies. Sensitivity analysis using
the Bayesian approach is consistent with the main analysis.
Conclusions: A Bayesian approach can be used in a systematic
review of adverse events to combine the information from all
available controlled study designs by putting less weight on
the observational studies.
RO 9.9
Prediction intervals should be routinely
reported in meta-analyses
IntHout J1 , Ioannidis JP2 , Rovers MM1 , Goeman JJ1
1
Radboud University Medical Center, The Netherlands; 2 Stanford
University, USA
Background: Evaluating the variation in the strength of the
effect across studies is a key feature of meta-analyses. This
variability is reflected by measures like τ 2 or I2 but their
clinical interpretation is not straightforward, especially not if
the meta-analysis is in odds ratios or risk ratios. Objectives: A
prediction interval presents the expected range of true study
effects in similar studies. We aim to show how it can help
to understand the uncertainty about whether a treatment
works or not. Methods: Conclusions based on confidence
intervals (CI) may not hold in all settings. We evaluated
the differences in conclusions based on 95% CIs and 95%
prediction intervals in statistically significant meta-analyses
published in the Cochrane Database of Systematic Reviews
between 2009 and 2013. If the estimated I2 was > 0 we used
this I2 for the prediction interval. When the estimated I2 was
0, we imputed low levels of heterogeneity (I2 = 20%), because
heterogeneity estimates are often imprecise. Results: A
statistically significant meta-analysis does not guarantee
that the treatment will be effective in all settings: in 347
(72%) of the 479 statistically significant meta-analyses with
I2 > 0, the prediction interval showed that the treatment
could be ineffective. Imputation of an I2 of 20% in the
other 441 statistically significant meta-analyses gave similar
results: 329 (75%) of the treatments could be ineffective.
Conclusions: The CI is inadequate for clinical decision making
because it only summarizes the average treatment effect. The
prediction interval is more informative as it shows the range
of possible effects in relation to the no-effect and clinical
benefit thresholds. A narrow prediction interval completely
on the beneficial side of a clinically relevant threshold
increases confidence in an intervention. A broad prediction
interval may indicate the existence of settings where the
treatment has a suboptimal and possibly even harmful effect.
Prediction intervals should be routinely reported to allow
more informative inferences in meta-analyses.
Filtering the information overload
for better decisions
RO 10.1
Integrating evidence: existing systematic
reviews and primary studies
Robinson K1 , Chou R2 , Berkman N3 , Newberry S4 , Fu R5 , Harling
L6 , Dryden D6 , Butler M7 , Foisy M8 , Anderson J9 , Motu’apuaka M9 ,
Revelo R9 , Guise J9 , Chang S10
1
Johns Hopkins University EPC, USA; 2 Pacific Northwest EPC, USA;
3
RTI International, USA; 4 Southern California EPC, RAND, USA;
5 Portland VA Research Foundation, USA; 6 Department of Pediatrics,
University of Alberta, Canada; 7 Minnesota EPC, USA; 8 Department
of Pediatrics, University of Alberta, USA; 9 Scientific Resource Center
for the AHRQ Effective Health Care Program, USA; 10 Center for
EvidenceandPracticeImprovment, AgencyforHealthcareResearch
and Quality, USA
Background: The exponential growth in the number of
systematic reviews (SRs) being published, the need to
regularly update existing reviews, along with the time and
resources required to undertake a review, motivates the desire
to integrate existing reviews into a new review. However,
there is a trade-off between accepting the results of a
prior review and redoing selected elements, or the review
in its entirety. Objectives: The goal of this systematic
review methods development project was to enhance
guidance on approaches and considerations for integrating
existing SRs into new reviews. Methods: Over a two-year
period, a workgroup of SR methodologists from across the
US Agency for Healthcare Research and Quality (AHRQ)
Evidence-based Practice Centers (EPCs) clarified guidance
needs and developed new recommendations. Deliberations
were supported by a review of guidance developed by the
EPCs and eight other organizations, including Cochrane
and the National Institute for Health and Care Excellence
(NICE ); interviews with key informants from organizations
63
that conduct or use SRs; and a search of the literature.
Recommendations were developed in an iterative manner
and based on consensus of workgroup members. Results:
In Year 1, the workgroup found general consensus across
organizations for assessing the relevance of prior reviews
and scanning their references to identify studies for a new
review. In Year 2, the workgroup identified eight areas where
additional guidance was needed. No literature relevant to
informing discussions was identified. Recommendations
were developed concerning: minimum eligibility criteria
for including an existing review based on its approach
and methodological rigor; criteria for using risk of bias
assessments from an earlier review; presenting findings
from the earlier review distinctly from new studies and
synthesizing the two; and an approach for strength of
evidence. Conclusions: Preliminary guidance was developed
promoting an efficient and unbiased approach for integrating
old and new evidence.
RO 10.2
Incorporating non-trial data into systematic
reviews: opportunities and risks
Tsafnat G1 , Gallego-Luxan B1 , Coiera E1
1
Centre for Health Informatics, Australian Institute of Health
Innovation, Macquarie University, Australia
Background: Clinical data and measures that do not come
from clinical trials (e.g. electronic health records, social media)
bring additional information that may enhance traditional
systematic reviews of the evidence. Current methods for
critical appraisal cannot be applied to such data. Emerging
methods for interpreting evidence, (e.g. crowdsourcing,
automatic systematic reviews [1]), may provide opportunities
to incorporate non-trial data whilst ensuring integrity and
reliability. Objective: To examine the opportunities and risks
of integrating non-trial data into systematic reviews using
novel methods and technologies. Results: Non-trial data offer
the following advantages: - More data and longer follow-up
[2] - Better representation of healthcare services including
comorbidities and settings - Include non-drug therapies Data for circumstances where randomized controlled trials
are unethical or infeasible - Broader sets of patient outcomes
- Capture rare events The following risks also exist: - Lack
of randomisation, control and unmeasured confounders
introduce biases - Self-reported data (e.g. from social
media) may have selection bias - Limited to retrospective
analysis and probably unavailable for new therapies These
risks may be mitigated by: - Transparent, objective and
repeatable protocols in automatic systematic reviews [1] Crowdsourcing editorial services for robust fact-checking Control of bias using multiple datasets in lieu of multiple
arms, and statistical methods that ensure significance of
results - Monitor conventional and social media for reports
on harms or harmful sentiment (e.g. anti-vaccine) that may
affect practice [3]
to bias but provide crucial information unavailable from
clinical trials. More research is needed on when and how
non-trial data can be used in systematic reviews.
Reference: Tsafnat et al. The automation of systematic
reviews. BMJ 2013: 346: f139. 2. Gallego et al. Role of EHR
in comparative effectiveness research. JCE 2013: 2: 529–532.
3. Zhou et al. Using social connection information to improve
opinion mining. MedInfo 2015.
RO 10.3
Application of text mining and machine
learning for problem formulation in
systematic reviews
Thayer K1 , Howard B2 , Holmgren S1 , Pelch K1 , Walker V1 , Lunn R1 ,
Shah R2
1
National Institute of Environmental Health Sciences
(NIEHS)/National Institutes of Health (NIH), USA; 2 SciOme LLC,
USA
Background:
Identifying addressable questions for
systematic reviews can be a challenge, especially in
environmental health where evidence from human, animal,
and in vitro studies is often integrated in assessments.
Text-mining and machine learning tools hold promise to
help with problem formulation. Objectives: To explore
the utility of using the Sciome Workbench for Interactive,
Computer-Facilitated Text-mining (SWIFT) software to
visualize literature search results for three complex topics:
research trends for ∼500 endocrine-disrupting chemicals;
environmental influences on the epigenome; and health
effects associated with night shift work, light at night, or
circadian disruption. Methods: Literature search results
from PubMed were uploaded into SWIFT for each project.
Customized search strategies were developed for evidence
stream (i.e. human, animal, in vitro), exposure, and health
outcome. The unsupervised topic clustering functionality of
SWIFT was used to group articles by subject matter. Users
created intersections of various tags to focus on specific topics,
e.g. night shift work and metabolic disorders. Together these
functions were used to create interactive reports. Results: The
interactive, visual reports produced by SWIFT allowed users
to identify and formulate focused research questions more
efficiently. The reports helped identify topics that have been
extensively studied, as well as emerging areas of research
(Figure 1). Viewing results by evidence stream helped users
determine how much evidence integration might be required
(Figure 2). The topic clustering results are also used to identify
’seed studies’ for the purpose of training a machine-learning
model that priority ranks relevant studies in focused areas.
Conclusions: Text-mining and machine learning programs
such as SWIFT are valuable tools for problem formulation.
These types of analyses could be considered for a type of
scoping review that can be used for various purposes, ranging
from showing trends in research, to identifying targeted
questions that could be addressed in systematic reviews.
Conclusion: Non trial sources of clinical data are more prone
64
65
RO 10.4
Mining the information overload: a scoping
review of patients’ responses to different
forms of health risk information
Pennington A1 , Noble C1 , Garner J1 , Harris R1
1
University of Liverpool, United Kingdom
Background: The explosion of new information technologies
and the global growth in the academic sector present
challenges to systematic reviewers that have grown in parallel
to the burgeoning of electronic databases. Reviewers
are increasingly at risk of either being overwhelmed
by unmanageable search results, or missing evidence.
The challenges are greatest for reviews of ’broad’ social
determinants of health and interventions, particularly when
they include phrases such as ’health information’. New
information technologies/methods also provide potential
solutions. Objectives: Our scoping review aimed to describe
the extent and nature of evidence on patients’ responses to
different forms of information on their health status/risk of
disease, with the objectives of mapping key concepts, sources
and types of evidence, commonalities, themes and gaps in the
research. Methods: To strike the balance between sensitivity
(finding all articles in an area) and specificity (finding only
relevant articles), the review used standard systematic review
approaches to develop search strategies in conjunction with
text mining approaches utilising Automatic Term Recognition
(ATR) software. Sample papers were identified in initial
searches and screened for inclusion. ATR software identified
search terms/phrases within the sample papers, producing
107 precise phrases (as opposed to Boolean searches of words,
e.g. ’health NEAR/3 information’) after manual selection by
reviewers. Phrases were combined with simple operators and
run across nine databases. Results: The searches identified
6662 unique articles, compared to standard approaches which
identified 100,000+ articles. Title and abstract screening
identified 358 papers for full text screening - high numbers
here being a measure of the sensitivity of the search of
this broad area. Twenty-six articles were included in the
review. Conclusions: Our systematic approach combining
traditional search term development methods and new text
mining methods produced results that were sensitive, specific
and, of growing importance to the increasingly overwhelmed
community of reviewers, manageable.
RO 10.5
Using innovative methods in trial
identification for 12 Cochrane Intervention
Reviews
Noel-Storr A1 , Ware J1 , Gouda P2 , Hull M3 , Jackson D1 , Murrieta
Álvarez I4 , Yaman H5 , Forbes S6 , Rutjes A7
1
Cochrane Dementia and Cognitive Improvement Group, Oxford
University, UnitedKingdom; 2 NationalUniversityofIreland, Ireland;
3
George Mason University, USA; 4 Universidad Popular Autónoma
del Estado de Puebla, Mexico; 5 Akdeniz University, Turkey;
6 Okanagan College, Canada; 7 University of Bern, Switzerland
Background: In June 2014 the Cochrane Dementia and
Cognitive Improvement Group began a programme of work
to produce 12 intervention reviews looking at modifiable risk
factors (MRFs) for dementia. To produce this large body of
evidence the Cochrane Dementia Group have implemented
a number of innovative methods in review production. This
study focuses on the two innovative methods used in trial
identification. Objectives: Our objective was to identify all
relevant trials for potential inclusion in 12 intervention reviews
related to MRFs in dementia using innovative methods,
including crowdsourcing, and to evaluate those methods
against appropriate reference standards. Methods: Two main
innovations were utilised in the search and screen process.
The first was to take a ‘suite’ approach and develop one search
strategy per suite of related reviews rather than one search
per review; and the second was to recruit a crowd through
Students for Best Evidence to screen the search results using
an online screening tool. Both methods were to be validated
or compared against traditional methods. Results: Taking
a suite approach to the search meant that far fewer unique
sets of search results were produced (four instead of 12).
This meant that fewer overall results were identified with no
compromise on the sensitivity of the searches. A total of
48 participants signed up to screen citations, with 23 (48%)
screening 500 or more. A deadline of four weeks was given and
met. Crowd performance, measured against random samples
screened by members of the core teams (under evaluation),
will be presented in terms of the crowds’ sensitivity (the
collective ability to identify the RCTs correctly) and specificity
(identify the rejects correctly). Conclusions: We made gains in
efficiency through having fewer citations to screen overall and
by harnessing a crowd to screen those citations for trial design.
This work contributes to the growing body of evidence on the
beneficial role of crowdsourcing in the review production
process. As we are confronted with ever increasing amounts
of data to process, we need to find new methods to deal with
the information overload.
RO 10.6
Enhancing search strategy development
using word clouds
Robalino S1
1 Newcastle University, United Kingdom
Background: Systematic reviews increasingly seek to address
issues beyond the relative effectiveness of interventions. This
can be challenging when the concepts to be searched for
are poorly indexed or difficult to describe. Using two case
studies, one focusing on the ethics and acceptability of an
intervention and the other focusing on economic evaluations,
an approach to enhancing search strategy development is
outlined. Objectives: To use text mining principles to examine
metadata and full-text papers to discover terms and concepts
not discovered in standard search strategy development.
Methods: Using known relevant papers, metadata from
bibliographic databases and full-text papers alongside freely
available word cloud generators, both MeSH and free-text
terms were examined to highlight undiscovered descriptors
66
for use in a systematic review search strategy. Initially text
was taken from the bibliographic database metadata (MeSH
and keywords), title and abstract, and full-text papers. These
sets of text were then examined separately to determine
the most useful texts to explore through word clouds which
highlight frequency of terms in their display. Results: The
use of word clouds to discover ‘hidden’ MeSH and keywords
for search strategies and filters works well. In both case
studies its use added to the final search strategies in both
projects. The inherent simplicity of word clouds makes this
an accessible method of discovery, however some types of
text yielded better results than others, due to the limitations
of word cloud software. Conclusions: This method is easily
accessible to anyone wishing to develop search strategies for
hard-to-define or unfamiliar topics as it does not require large
volumes of texts nor expertise in programming. This method
of discovery can be utilised alongside standard approaches
to provide a richer set of terms with minimal additional effort.
RO 10.7
The Embase project 3: the 48-hour citation
screening challenge
Noel-Storr A1 , Dooley G2 , Glanville J3 , Foxlee R4
1
University, United Kingdom; 2 Metaxis Ltd, United Kingdom;
3
York Health Economics Consortium, United Kingdom; 4 Cochrane
Editorial Unit, United Kingdom
Background: The Embase project aims to identify all reports
of randomised trials and feed those into Cochrane’s Central
Register of Controlled Trials (CENTRAL). During 2011, 2012 and
2013 a backlog of records to screen had built up. To tackle the
backlog in a time efficient way two methods were employed:
(1) a small team of screeners were assigned to focus on these
records; and (2) a citation screening challenge was organised.
Objectives: The main objective of the citation screening
challenge was to screen as many citations as possible within a
48-hour period. Methods: A challenge crowd was recruited in
the weeks before the event. Backlog records were loaded into
a copy of the online citation screening tool. The 48-hour period
began on 31 October and finished on 2 November. The event
was a fundraiser, with a micro amount set for each citation
collectively screened. The proceeds were to go to Médecins
Sans Frontières for the Ebola relief effort. Results: A total of
75 people from 20 countries took part and 20,709 citations
were screened in a 48-hour period; 1713 RCTs/q-RCTs were
identified and GBP 5177.25 was raised for the Ebola relief
effort. The challenge crowd screened over twice as many
records in the final 24 hours of the challenge compared to
the first 24 hours. An evaluation of challenge crowd accuracy
is underway and will be presented. In an after-challenge
questionnaire completed by 84% of participants, 95% said
they would take part in another challenge. Conclusions: The
48-hour citation screening challenge was the first of its kind
and proved an incredibly effective and fun method to help to
screen a large number of citations in a very short space of
time.
RO 10.8
Transforming from information overload to
information excitement using the Pearl
Harvesting Information Retrieval Framework
Sandieson R1
1
Western University, Canada
Background: Information overload derives from a number
of sources including an expanding body of research that is
more dispersed across journals and databases (Hall 2004).
Traditional information retrieval strategies are cumbersome
in the digital environment (Arendt 2007; Sandieson 2010). The
Pearl Harvesting Information Retrieval Framework solution
(PHIRF; Sandieson 2006; 2010; 2013) is a design science
approach that uses rich-text searches based on synonym
clusters. Synonym clusters include ALL the terms used by
researchers and indexers to identify a topic. Term harvesting is
accomplished from a broad range of sources to avoid the bias
of using a restricted set of search terms, which is a potential
problem with existing search term development. Present
Study: There has been a recent dramatic production and
evolution of systematic reviews and research databases have
not kept pace with indexing these. Common search strategies
to search for systematic reviews use ’systematic review’ OR
’meta analysis’ OR ’literature review’. The term ’review’ is
recommended, but has very low precision. In the present study
the PHIRF was used to produce a comprehensive synonym
cluster for systematic reviews. Results: The Pearl Harvesting
systematic review synonym cluster contained 25 terms and
produced 57% more citations than the combined standard
search terms (excluding ’review’; using .af) in MEDLINE. When
paired with a synonym cluster for autism, the PH search
produced 1146 citations versus 685 using the standard search.
In PsycINFO, the PHIRF search produced 90% more systematic
review citations than the standard search (using ALL). The
PHIRF search produced 4104 systematic review citations for
autism versus 3716 using the standard method. Conclusions:
Pearl Harvesting improved the number of citations retrieved
for systematic reviews in two sample databases. Synonym
clusters, once established, are placed in a public wiki for
anyone to use. Our experience of teaching people how to do
PHIRF searches is that when they experience comprehensive,
relevant results they transform from a state of information
overload to information excitement.
References: Arendt, J. (2007).
How Do Psychology
Researchers Find Studies to Include in Meta-Analyses?
Behavioral & Social Sciences Librarian, 21, 1–23.
Hall, A. & Walton, G. (2004). Information overload within the
health care system: a literature review. Health Information
and Libraries Journal, 21, 102–108.
Sandieson, R. (2006). Pathfinding in the research forest: The
Pearl Harvesting method for effective information retrieval.,
Education and Training in Developmental Disabilities, 41(4),
401–409.
Sandieson, R.W., Kirkpatrick, L.C., Sandieson, R.M.,
Zimmerman, W. (2010). Harnessing the Power of Research
Databases: The Pearl Harvesting Methodological Framework
67
for Information Retrieval. The Journal of Special Education.
44, 161–175.
Sandieson, R.W., & McIsaac, S. M. (2013). Navigating the
information maze of giftedness using the Pearl Harvesting
Information Retrieval Methodological Framework. Talent
Development and Excellence 5(2), 101–112.
RO 10.9
Knowledge translation
RO 11.1
Does the medium matter when getting
the message?
Systematic review on social media as a data
source for information on adverse effects
Hanratty J1
1
Queens University Belfast, United Kingdom
Golder S1 , Norman G2 , Loke Y1
1
Cochrane Adverse Effects Methods Group, United Kingdom;
2 Cochrane Wounds Group, United Kingdom
Background: Cochrane Reviews provide high quality
summaries of evidence with the healthcare professional in
mind, but time constraints often mean that professionals
only read summaries or abstracts. Objectives: The aim is
to test understanding of the key messages in a review when
participants are randomly assigned to receive a summary
of a review in one of four mediums (conditions). Methods:
Participants were recruited online via mailing lists of health
professionals and university staff. A recent Cochrane Review
of Parent Infant Psychotherapy (Barlow 2015) was chosen.
Participants were randomly assigned to one of four conditions;
reading the review abstract, the plain language summary, a
blog post aimed at health professionals or listening to a
podcast interview with the review author. Participants were
then asked about the key findings of the review and how,
if at all, it would impact on their practice. Results: This
study is ongoing and results will be presented in terms of the
proportion of respondents in each condition who correctly
identified the core messages of the review.
Background: Social media (such as patient forums, Twitter,
and Facebook) are increasingly popular and contain a vast
array of unpublished up-to-date information. There are two
avenues through which social media can provide adverse
effects data: first by providing references to published and
unpublished literature and second by providing patients’
experience of adverse effects. Social media have not yet been
fully explored as a potential source of adverse effects data
for systematic reviewers. Objectives: To review the literature
that has evaluated systematically or analyse the use of social
media to collect information on adverse effects. Methods:
Eighteen databases (including MEDLINE and Embase) were
searched for relevant studies, in addition to handsearching
key journals, conferences, newsletters and blogs, contacting
experts and reference checking. Any type of evaluation was
considered eligible for inclusion if it assessed the use of social
media to collect information on adverse effects and presented
the outputs from searching social media. Data extraction and
quality assessment was undertaken independently by two
reviewers. Due to the nature of the included studies the
quality criteria were customized to evaluate aspects such
as the selection of data, duplicate data and misinformation
and validity and representativeness. Results: A total of
3045 records were retrieved (4457 before duplication). The
included studies were heterogeneous in nature and thus a
narrative synthesis with descriptive analysis was undertaken.
We summarise the existing research on the potential value of
social media for information on adverse effects and discuss
the implications for systematic reviewers. Conclusions: Case
reports of adverse effects identified in social media could
be a useful source of evidence in systematic reviews. In
addition, posts on social media could help us identify those
adverse effects most important to patients and thus help
us formulate and prioritise questions on adverse effects for
future systematic reviews.
RO 11.2
Communication of systematic review
findings: exploring format preference
Thompson Coon J1 , Bethel A1 , Abbott R1 , Rogers M1 , Moore D1 ,
Stein K1
1 University of Exeter Medical School, United Kingdom
Background: Ensuring that research findings reach their
intended audience so that they may make a difference to
health care is a challenge that is gaining greater emphasis.
Traditional methods of dissemination may result in limited
communication.
Method: Following the completion of a systematic review
about the use of gardens by people with dementia resident in
care homes, we produced a range of dissemination products
containing the findings of the review: a peer-reviewed
academic paper, an A4 hand-out of a conference poster,
a tri-fold leaflet and a video clip. At a multi-disciplinary
dementia conference we asked individuals to choose which
format they preferred and to give their reasons for this
choice. Results: The activity attracted a lot of attention
from conference attendees who were keen to participate and
help to identify approaches to make research findings more
readily accessible to them. A total of 40 people took part;
16 health care professionals (including nurses, occupational
68
therapists and social workers), four carers/members of the
public, four lecturer/researchers, two policy makers, four
students (social work, nursing), three trainers of healthcare
professionals, three members of other support organisations
(housing, memory café) and three who did not specify. The
leaflet was the most popular option (18 votes), followed by
the video (9 votes), the poster (8 votes) and the peer-reviewed
paper (5 votes). The most common reasons for preferring
the leaflet were that it was easy to read and easy to access.
Conclusions: These pilot results suggest that consumers of
systematic review findings are seeking to engage with them
in a variety of different formats. Ease of access and the use
of lay language are important considerations. Further work
is planned to explore preferred level of content, the layout of
dissemination products and methods of distribution.
RO 11.3
Adapting fishbone diagrams: a new approach
for displaying findings of systematic reviews
Morgan L1 , West S1 , Gartlehner G2 , Jordan H1 , Kampov-Polevoy
A3 , Garbutt J3 , Bobashev G1
1
RTI International, USA; 2 DUK, Austria; 3 University of North
Carolina, Chapel Hill, USA
Background: The traditional approach to presenting the
results of systematic reviews is to use multiple summary
tables. Although highly informative, these tables are often
dense and do not allow a reader to evaluate the evidence
on various and outcomes ‘‘at a glance’’ and instead require
large parts of the publication to be read. Since they can easily
and clearly illustrate cause and effect, fishbone diagrams are
widely used in industrial studies and are beginning to be used
in health care. The goal of this paper is to show how these
diagrams can be adapted for use in graphically capturing
the results of systematic reviews in health care research,
and to stimulate further consideration of pictorial summary
approaches. Methods: We applied the fishbone diagram to
summarize the findings from a sample of systematic reviews
visually. The head of the fish represents the balance of
benefits and harms of an intervention or comparison, and the
bones of the fish represent the individual health outcomes
or moderators. To illustrate a range of applications, we
abstracted and compiled information from the summary
tables of three recent systematic reviews to populate each
fishbone diagram. Results: Based on real world systematic
reviews, we provide three examples that show how fishbone
diagrams might be used to display complex information in a
clear and succinct manner. We present fishbone diagrams
depicting: 1. the body of evidence used for a clinical practice
guideline; 2. the body of evidence for development of
population-based screening recommendations; and 3. the
relationship between an outcome and various positive and
negative moderators. Conclusions: Fishbone diagrams are
compact visualizations that may prove useful for summarizing
the findings of systematic reviews, especially for healthcare
providers and guideline developers who do not have the time
to review the full evidence in its entirety.
RO 11.4
Interactive ’Summary of findings’ table (iSoF)
Rosenbaum S1 , Morelli A1 , Moberg J1 , Oxman A1 , Rada G2 ,
Collaboration D3
1 Norwegian Knowledge Centre for the Health Services, Norway;
2 Evidence-Based Health Care Program, Pontificia Universidad
Católica de Chile, Chile; 3 GRADE Working Group, EU
Background: When making informed healthcare decisions,
people need to understand the likely risks and benefits of
the treatments they are considering. ’Summary of findings’
(SoF) tables help people understand key messages from
Cochrane Reviews in a concise format. However, many
people still have difficulty understanding the numbers and
some of the concepts in SoF tables. Static SoF tables also
lack formatting flexibility and limit the amount of detail
that can be presented to more expert users. Objective: To
create an interactive SoF table (iSoF) that helps all users
find and understand the main messages from a systematic
review by allowing them to choose the amount of detail
and mode of presentation that is appropriate for them
to inform their decision-making. Method: The iSoF was
developed iteratively by the DECIDE project (funded by
the EU) informed by feedback from stakeholders, user
testing and an advisory board of international experts.
Results: iSoF tables present the key messages from an
evidence summary: the most important outcomes (benefits
and harms), the size of the effects, and the certainty of
the evidence. They offer a layered presentation, enabling
producers to tailor tables for different audiences or types
of evidence or users to ‘drill-down’ for more information by
scrolling over terms, concepts or interactive footnotes to
find explanations. Each outcome can be viewed in several
formats: plain language that describes effect sizes and the
certainty of the evidence in a way most people can readily
understand; absolute effects presented as numbers, text,
or interactive graphic visualizations; and as relative effects.
iSoF tables provide an optimal starting point for other
evidence summaries of a review, as they contain all the
building blocks necessary to present key findings in concise,
understandable format. iSoF tables can be prepared in
different languages and exported for use in other documents.
Conclusion: iSoF tables give producers and end users with
varying degrees of expertise flexibility, control and support
for understanding the main findings of an evidence summary.
http://isof.epistemonikos.org/#finding/5263ba01f30d0c11a5
3f7950
69
RO 11.5
RO 11.6
Can an evidence-informed taxonomy be used
to map vaccination communication
interventions in ‘real world’ settings?
Findings from Nigeria
Glossary of Evaluation Terms for Informed
Treatment choices in plain language (GET-IT)
Oku A1 , Oyo-ita A1 , Lewin S2 , Glenton C2 , Fretheim A2 , Ames H2 ,
Kaufmann J3 , Hill S3 , Cartier Y4 , Cliff J5 , Muloliwa A6 , Rada G7 ,
Bosch-Capblanch X8
1
University of Calabar, Nigeria; 2 The Norwegian Knowledge
Centre for the Health Services, Norway; 3 La Trobe University,
Australia., Australia; 4 International Union for Health Promotion
and Education, France; 5 Universidade Eduardo Mondlane,
Mozambique; 6 Provincial Directorate of Health, Mozambique;
7 Catholic University of Chile, Chile; 8 Swiss Tropical and Public
Health Institute and University of Basel, Switzerland
Background: Effective communication is key to improving
childhood vaccination coverage and is the focus of
the Communicate to Vaccinate (COMMVAC) project. In
earlier work, we developed a taxonomy of communication
interventions to help decision makers understand and
consider options for communication. As interventions in
systematic reviews become more complex and multifaceted,
such taxonomies are also assuming greater relevance for
systematic review methods.
Objectives: To describe
the application in the Nigerian context of a taxonomy of
vaccination communication interventions and to explore the
usefulness of the taxonomy for grouping and describing
interventions and identifying important gaps. Methods: The
study was conducted in Bauchi and Cross River States in
North and Southeast Nigeria respectively. Interviews were
carried out amongst purposively selected stakeholders in
the health services and in agencies involved in vaccination
communication planning.
Document review was also
conducted.
The COMMVAC taxonomy of vaccination
communication interventions was used to organize the
identified strategies based on their intended communication
purpose and their target group. Results: Most of the
communication strategies identified fell into the COMMVAC
taxonomy categories of ‘inform and educate’, targeting
caregivers and community members, and ‘enhancing
community ownership’. Religious and traditional leaders,
community mobilizers and women’s groups were common
targets for immunization messages and for efforts to sustain
community demand in both Bauchi and Cross River, though
more visibly in Bauchi. Most communication interventions
were used in the context of vaccination campaigns rather
than routine immunization programmes, and interventions
targeting health workers were also limited. Conclusions:
The evidence-informed taxonomy was helpful in mapping
communication strategies for childhood vaccination. It could
assist programme managers to identify gaps and consider the
communication purposes of the interventions they are using.
The study suggests that evidence-informed taxonomies may
be a useful tool for mapping interventions in ’real world’
settings.
Moberg J1 , Austvoll-Dahlgren A1 , Treweek S2 , Badenoch D3 ,
Harbour R4 , Rosenbaum S1 , Oxman A1 , Chalmers I5
1
Norwegian Knowledge Centre for the Health Services, Norway;
2 Health Services Research Unit, University of Aberdeen, United
Kingdom; 3 Minervation, United Kingdom; 4 Glasgow, United
Kingdom; 5 James Lind Initiative, United Kingdom
Background: Well-informed healthcare decisions depend on
the ability to understand information, particularly research
evidence. The use of jargon can be a barrier to people’s
understanding and use of evidence to inform their choices.
Objective: To create an online glossary that facilitates
informed choices about treatments by promoting consistent
use of plain language, providing plain language definitions
and explanations of terms that people might need to
understand if they wish to assess claims about treatments.
Methods: GET-IT (Glossary of Evaluation Terms for Informed
Treatment choices) was developed collaboratively by the
Informed Healthcare Choices project (funded by GLOBVac),
Testing Treatments interactive, and the DECIDE project
(funded by the EU). We screened 15 sources to identify terms,
including a sample of relevant glossaries. Evaluation of
the glossary includes feedback from collaborating partners,
a survey of organisations that have expressed an interest
in using GET-IT, user testing, and an evaluation of the
impact of the glossary on users’ understanding of terms.
Results: The glossary includes over 200 terms. For each
term, there is a suggested plain language term, synonyms,
a short definition, a full explanation, examples, and links to
additional resources. The glossary can be translated, tailored
to different audiences, and embedded in other websites.
Other websites can also link to definitions. Conclusions:
The GET-IT glossary is open-access, and the use of plain
language facilitates understanding by a wide range of people.
It is useful for people who communicate research evidence
(including systematic reviewers), and who teach others about
how to assess claims about treatments, in addition to those
who want to make informed choices about treatments.
RO 11.7
A review of online evidence-based practice
point-of-care information summary
providers: 2014 update
Kwag KH1 , Gonzàlez-Lorenzo M2 , Banzi R3 , Bonovas S4 , Moja L1
1 Clinical Epidemiology Unit, IRCCS Galeazzi Orthopaedic Institute,
Milan, Italy; 2 Clinical Epidemiology Unit, IRCCS Galeazzi
OrthopaedicInstitute, Milan; DiparmentofBiomedichalSciencesfor
Health, UniversityofMilan, Italy; 3 LaboratoryofRegulatoryPolicies,
IRCCS-Institute for Pharmacological Research Mario Negri, Milan,
Italy; 4 Department of Biomedical Sciences for Health, University of
Milan, Italy
70
Background: The complexity of modern practice requires
health professionals to be active information-seekers.
Evidence-based point-of-care (EBP) information summaries
can connect clinicians with best practice strategies at the
bedside to optimize patient care. This study is the 2014
update of a previous review that assessed the strength of
these products. Objective: To review EBP summaries against
their claims of being ‘‘evidence-based,’’ and determine
their progress. Methods: We searched MEDLINE, Google,
librarian association websites, and information conference
proceedings from August 2012 to December 2014. We included
online EBP summaries delivering literature syntheses that
claim to be evidence-based. We extracted data on general
characteristics and content presentation. We assessed
products quantitatively according to breadth, editorial
quality, and evidence-based methodology. We explored
improvements in these dimensions and their relevant
associations. Results: We screened 58 online products: 26
met our inclusion criteria, 10 of which were newly identified
in 2014. We were able to access 23 products. Most summaries
were produced by major publishers in the USA, with a minority
in Europe and none in developing- or low-income countries.
No significant association between the pairs of variables
was found. There were improvements in evidence-based
methodology and breadth since 2009. The main target
audience remained physicians, although several products
targeted nurses, physiotherapists, emergency specialists, and
pediatricians. Conclusions: New EBP summaries are targeting
specialized groups of health professionals, reflecting the
strength of the existing market of point-of-care summaries
for physicians. UptoDate, Best Practice, and Dynamed
scored the highest across all dimensions, although other
products excelled in one or two dimensions. Consumers
should weigh their needs carefully when selecting a product
for implementation. New generation EBP summaries aim
to fulfill patient-specific needs by integration into Electronic
Health Records, as well as providing an option for Continuing
Medical Education to promote lifelong, self-directed learning.
RO 11.8
Abstracts’ concept selection to reduce
information overload
Alper B1 , Malone-Moses M1
1
Evidence-Based Medicine Research and Development, Quality and
Standards for EBSCO Health, USA
Many users will access abstracts of Cochrane reviews to
determine key points, either to represent the complete
concept or to determine if full-text reading of the review
is warranted. Abstracts need to be concise yet include the
most important concepts. We will analyze 100 Cochrane
review abstracts compared to key concepts selected from the
Cochrane reviews for point-of-care summaries.
RO 11.9
Development and validation of a
questionnaire to measure people’s ability to
assess claims about treatment effects (CLAIM)
Austvoll-Dahlgren A1 , Oxman AD1 , Chalmers I2 , Semakula D3 ,
Nsangi A3 , Guttersrud Ø4
1
Norwegian Knowledge Center for the Health Services, Norway;
2
James Lind Initiative, United Kingdom; 3 Makerere University
College of Health Sciences, Uganda; 4 University of Oslo, Norway
Objective: People are confronted with claims about the
effects of treatments and health policies daily. There is little
current evidence available that describes people’s ability
to assess claims about treatment effects and there is no
instrument available to measure this. We are developing such
an instrument as part of an international Project - ’Informed
Healthcare Choices’ - with the aim of improving the ability
of people in low-income countries to assess claims about
treatment effects. Methods: We developed multiple-choice
items to address 31 key concepts that people need to
understand to be able to assess claims about treatment
effects. We obtained feedback on the applicability and
relevance of the items from an advisory group and conducted
cognitive interviews and a field test with children and adults in
Uganda. Results: We have prepared a pool of approximately
240 items (six to eight per concept) intended to measure
basic and applied understanding using plain language and
universally relevant scenarios. Members of the advisory group
judged the items to be relevant to the concepts. Terminology,
instructions and formats were rewritten to address the
feedback from children and adults. The items’ psychometric
properties will be assessed using Rasch analysis. The final
instrument will include approximately 31 items (one for each
of the 31 concepts). Conclusion: The ’CLAIM’ Instrument is,
to our knowledge, the first instrument to be developed to
measure people’s ability to assess claims about treatment
effects. We will use the instrument in randomized trials to
measure the effects of the ’Informed Healthcare Choices’
teaching resources and news service. The items will also
be posted on the Informed Healthcare Choices website as a
teaching resource for people to test themselves
Specific study types / Avoiding
research waste
RO 12.1
Should case series be included in systematic
reviews when controlled trials are scarce?
Sjögren P1 , Kindblom J1 , Liljegren A1 , Strandell A1 , Wikberg
Adania U1 , Jivegård L1
1 HTA-centrum, Region Västra Götaland, Sweden
71
Background:
Randomized (RCT) or non-randomized
controlled trials are regularly used for assessment of certainty
of evidence (GRADE) in systematic reviews (SRs) while case
series are mostly ignored. For some interventions RCTs
are difficult to design, but well-designed case series are
available. Objectives: Studying the influence on conclusion
and certainty of evidence (GRADE) by using case series in an
SR with only one small RCT. Method: A systematic literature
search in an SR of hypoglossal nerve stimulation (HGNS) in
adults with obstructive sleep apnoea (OSA) not tolerating
Continuous Positive Airway Pressure treatment resulted in one
small RCT (therapy withdrawal design) and seven case series.
The case series (n=232) were assessed by a modified checklist
(Guo 2013). For the outcome ‘resolution of OSA’, measured
by apnoea-hypopnea index (AHI), the conclusion and the
certainty of evidence (GRADE) were assessed separately for
the RCT and the case-series and then combined. Results: The
RCT evaluated the effect of HGNS switched ‘on’ or ‘off’ during
one week, 12 months after implantation, in responders. The
intergroup difference in AHI was 11-fold in favour of HGNS
‘on’. The RCT (starting at + + ++) was downgraded two
steps for study limitations regarding blinding and uncertain
directness and precision, resulting in ++OO. The case series
(starting at +OOO), several well-designed, were downgraded
two steps for some limitations in blinding, inconsistency, and
serious indirectness. Then the case series were upgraded one
step for > 2-fold AHI reduction at one year, clearly different
from the published natural course (AHI increase over time) of
OSA, and an additional step due to large upper airway volume
increase with immediate resolution of OSA with HGNS,
resulting in +OOO. Combined, the certainty of evidence was
low (++OO). Conclusion: Inclusion of case series reduced
the estimate of the treatment effect on AHI from 11-fold in the
RCT (short term) to 2-fold in long-term, while the certainty
of evidence remained at ++OO. Case series may provide
additional information in an SR when controlled studies are
scarce and the natural course of the condition is known.
Reference: Guo B, Moga C, Schopflocher D, Harstall C.
Validation of a quality assessment checklist for case series
studies. In: Better Knowledge for Better Health. Abstracts of
the 21st Cochrane Colloquium; 2013 19–23 Sep; Québec City,
Canada. John Wiley & Sons; 2013
RO 12.2
Integration of multiple study designs
in systematic reviews
Peinemann F1 , Tushabe D2 , Kleijnen J1
1
Maastricht University, The Netherlands;
Birmingham, United Kingdom
2
University
of
Background: A systematic review may evaluate different
aspects of a healthcare intervention. To accommodate the
evaluation of various research questions, the inclusion of more
than one study design may be necessary. Objectives: To find
and describe articles on methodological issues concerning the
incorporation of multiple types of study designs in systematic
reviews on healthcare interventions. Methods: We searched
PubMed, the Cochrane Database of Systematic Reviews, and
the Cochrane Methodology Register on 31 March 2012 and
identified 42 articles that reported on the integration of
single or multiple study designs in systematic reviews. We
summarized the contents of the articles qualitatively and
assessed theoretical and empirical evidence. Results: Many
examples of reviews incorporating multiple types of studies
exist and every study design can serve a specific purpose. 85%
(36 of 42) of reviews reported that nonrandomized studies
should be integrated in systematic reviews to complement
available RCTs or replace lacking RCTs. The clinical questions
of a systematic review determine the types of designs that are
necessary or sufficient to provide the best possible answers.
Conclusions: The integration of multiple study designs in
systematic reviews is required if patients are to be informed
about the many facets of patient-relevant issues of healthcare
interventions.
RO 12.3
There is no such thing as a one-dimensional
hierarchy of evidence: a critique and a
perspective
Hannes K1 , Bath-Hextall F2 , Behrens J3
1 Cochrane Qualitative and Implementation Methods Group,
2
University of Nottingham, United Kingdom;
Belgium;
3
Martin-Luther-Universität, Germany
There are many reasons why people favor particular study
designs in the context of producing systematic reviews.
In a Cochrane context, this choice has historically been
guided by the type of design that was most likely to
produce robust research findings: the randomized controlled
trial, recently extended with a theoretical option of high
quality observational research without confounders (GRADE:
Grading of Recommendations Assessment, Development and
Evaluation). It was argued that decisions toward patients
and clients should primarily be based on such evidence.
In this paper we argue that this is a destructive choice
for two main reasons: (1) it would prevent us from acting
in a situation or event for which no robust evidence is
available; and, (2) it would lead to an undesirable hierarchy of
research questions, with more complex questions not being
answered (or funded). We propose two models to respond
to these issues: (1) a model integrating external (research
driven) and internal evidence (client driven) in supporting
evidence-based decision making; and, (2) a question-driven
model for a mega-synthesis approach to support review
authors in tackling a broad variety of questions. If we are
serious about improving health care practice at the point of
care, then we should investigate the conditions for realizing
a coherent decision making process, based on the evidence
that is available (not unavailable) to us.
72
RO 12.4
Meta-analyses neglect previous systematic
reviews and meta-analyses about the same
topic
Helfer B1 , Prosser A2 , Samara M1 , Geddes J3 , Cipriani A3 , Davis J4 ,
Mavridis D5 , Salanti G5 , Leucht S1
1 Cochrane Schizophrenia Group, Germany; 2 Centre for Addiction
and Mental Health, Toronto, Canada; 3 University of Oxford,
United Kingdom; 4 Cochrane Schizophrenia Group, USA; 5 Cochrane
Statistical Method Group, Greece
The following abstract is a part of an article accepted for
publication in BMC Medicine. Background: As the number
of systematic reviews is growing rapidly, we investigate
systematically whether meta-analyses published in leading
medical journals present an outline of available evidence by
referring to previous meta-analyses and systematic reviews.
Methods: We searched PubMed for recent meta-analyses of
pharmacological treatments published in high impact factor
journals. Previous systematic reviews and meta-analyses
were identified with electronic searches of keywords and
by searching reference sections. We analyzed the number
of meta-analyses and systematic reviews that were cited,
described and discussed in each recent meta-analysis.
Moreover, we investigated publication characteristics that
potentially influenced the referencing practices. Results:
We identified 52 recent meta-analyses and 242 previous
meta-analyses on the same topics. Of these, 66% of identified
previous meta-analyses were cited, 36% described, and only
20% discussed by recent meta-analyses. The probability
of citing a previous meta-analysis was positively associated
with its publication in a journal with a higher impact factor
(odds ratio (OR) 1.49; 95% confidence interval (CI) 1.06 to
2.10) and more recent publication year (OR 1.19; 95% CI
1.034 to 1.37). Additionally, the probability of a previous
study being described by the recent meta-analysis was
inversely associated with the concordance of results (OR
0.38; 95% CI 0.17 to 0.88), and the probability of being
discussed was increased for previous studies that employed
meta-analytic methods (OR 32.36; 95% CI 2.00 to 522.85).
Conclusions: Meta-analyses on pharmacological treatments
do not consistently refer to or discuss findings of previous
meta-analyses on the same topic. Such neglect can lead to
research waste and be confusing for readers. Journals should
make the discussion of related meta-analyses mandatory.
RO 12.5
Unnecessary repetitions of pediatric clinical
trials: cumulative meta-analyses
Janiaud P1 , Cornu C2 , Kassai B2
1
University of Lyon, France; 2 EPICIME-Clinical Investigation Center,
INSERM, France
Background: As children are a vulnerable population, it is
essential to avoid unnecessary trials when enough evidence
is available. Objectives: Our objective was to identify
unnecessary repetitions of randomized clinical trials (RCTs) in
the pediatric population, using cumulative meta-analyses,
when either the beneficial or deleterious effect of the
treatment has been confirmed. Methods: We searched
three electronic databases for meta-analyses. Double-blind,
placebo–controlled RCTs including adults and children were
eligible. The random-effects model was used to pool effect
over time by including studies according to their year of
publication. We performed cumulative meta-analysis to
identify significant benefit or risk when multiple RCTs were
available for children. Results: Initially 89 meta-analyses
were identified, including 992 RCTs amongst which only
294 were in children and evaluating 124 drugs. For 53
drugs, only one pediatric trial was included. Ten drugs
for the same indication and with the same outcome were
evaluated by more than one meta-analysis. Ondansetron
for postoperative nausea and vomiting was assessed by 27
pediatric RCTs. The treatment benefit was confirmed in 1995
(Figure 1). Cumulative meta-analysis showed that 19 trials,
which included 840 children, were unnecessary. Conclusions:
Our preliminary results suggest the presence in the literature
of reports of unnecessary RCTs in children. Clinical research
in children should focus on unmet needs and avoid exposing
children to unnecessary trials.
73
Figure 1
RO 12.6
Does access to clinical study reports from the
European Medicines Agency reduce reporting
bias?
Tonia T1 , Rohner E1 , Petavy F2 , Pignatti F2 , Bohlius J1
1
Institute of Social and Preventive Medicine, University of Bern,
Switzerland; 2 European Medicines Agency, United Kingdom
Background: In a previous Cochrane Review on the effects
of erythropoiesis stimulating agents (ESAs) in cancer patients
(1) we identified potential publication and outcome reporting
biases for several outcomes. Objectives: We collaborated
with European Medicines Agency (EMA) to retrieve clinical
study reports (CSRs) and investigated whether inclusion of
data reported in these CSRs may help to reduce reporting
bias. Methods: We identified all randomised controlled
trials (RCTs) on the effect of ESAs on cancer patients from
our previous Cochrane Review (1) and updated literature
searches. We set up a collaborative agreement with EMA and
requested the CSRs for all identified RCTs. We conducted
random-effects meta-analyses and compared the pooled
estimates based on data as reported in the public domain
(i.e. articles, abstracts, other public sources) with the pooled
estimates based on data from the public domain and the
CSRs. We present preliminary results for the outcomes:
number of patients receiving red blood cell transfusions
(RBCTs); anaemia symptoms measured with the Functional
Assessment of Cancer Therapy-Anaemia (FACT-An); and
hypertension; analyses for additional outcomes are on-going.
Results: From the review (1) and updated literature searches
we identified 92 RCTs, the EMA identified two additional,
unpublished studies. For these 94 RCTs the EMA had the
CSRs for 16 RCTS. The number of studies and participants
analysed and the pooled effect estimates per outcome are
shown in Table 1. For RBCTs and hypertension the inclusion of
two and nine additional studies, respectively, based on data
reported in CSRs did not alter the overall effect estimates.
For FACT-An the inclusion of six additional studies based on
data reported in CSRs reduced the pooled mean difference
of changes from 5.93 (95% CI 4.37, 7.49) to 2.79 (95% CI
0.88, 4.70), which is below the threshold (defined as ≥ 4) of
a clinically important difference for FACT-An (1). This result
will change the conclusion of the review (1). Conclusions:
Unpublished clinical study reports held by EMA may be a
useful source to reduce outcome reporting bias. Reference:
(1) Tonia et al CD003407.
74
RO 12.7
Agreements on publication rights: an
investigation of protocols and publications of
randomized clinical trials
Kasenda B1 , Amstutz A2 , Von Elm E3 , You J4 , Blümle A5 , Tomonaga
Y6 , Saccilotto R2 , Bengough T7 , Meerpohl J5 , Stegert M2 , Olu K2 ,
Tikkinen K4 , Neumann I4 , Carrasco-Labra A4 , Faulhaber M4 , Mulla
S4 , Mertz D4 , Akl E4 , Bassler D8 , Busse J4 , Ferreira-González
I9 , Lamontagne F10 , Nordmann A2 , Gloy V2 , Raatz H2 , Moja L11 ,
Ebrahim S4 , Schandelmaier S2 , Sun X12 , Vandvik P13 , Johnston B4 ,
Walter M14 , Burnand B3 , Schwenkglenks M6 , Kasenda L2 , Bucher
H2 , Guyatt G4 , Briel M2
1
Royal Marsden Hospital, United Kingdom; 2 Basel Institute for
ClinicalEpidemiologyandBiostatistics, UniversityHospitalofBasel,
Basel, Switzerland; 3 Cochrane Switzerland, Institute of Social
and Preventive Medicine (IUMSP), Lausanne University Hospital,
Lausanne, Switzerland; 4 Department of Clinical Epidemiology and
Biostatistics, McMaster University, Hamilton, Ontario, Canada;
5
German Cochrane Centre, Medical Center –University of Freiburg,
Freiburg, Germany; 6 Epidemiology, Biostatistics and Prevention
Institute (EBPI), University of Zurich, Zurich, Switzerland; 7 Austrian
Federal Institute for Health Care, Department of Health and Society,
Vienna, Austria; 8 Department of Neonatology, University Hospital
Zurich, Zurich, Switzerland; 9 Epidemiology Unit, Department of
Cardiology, Vall d’Hebron Hospital and CIBER de Epidemiologı́ a
y Salud Publica (CIBERESP), Barcelona, Spain; 10 Centre de
Recherche Clinique Étienne-Le Bel and Department of Medicine,
Université de Sherbrooke, Sherbrooke, Canada; 11 CS Orthopedic
Institute Galeazzi, Milano, Italy; 12 Chinese Evidence-based Medicine
Center, West China Hospital, Sichuan University, Chengdu, China;
13
Department of Medicine, Innlandet Hospital Trust-Division Gjøvik,
Oppland, Norway; 14 Institute of Nuclear Medicine, University
Hospital Bern, Bern, Switzerland
Background: For-profit companies conducting randomized
clinical trials (RCTs) often make contracts with academic
investigators. Little is known about the content of publication
agreements documented in RCT protocols, and the accuracy
of corresponding statements in published RCTs. Objectives:
To investigate: 1. existence and type of publication
agreements in RCT protocols; 2. the completeness and
accuracy of the corresponding reporting in subsequent
publications. Methods: We identified a retrospective cohort
of RCTs based on archived protocols approved by six research
ethics committees in Switzerland, Germany, and Canada
between 2000 and 2003. Last follow-up of RCTs was 27 April
2013. Only RCTs with industry involvement were eligible.
Outcomes: Documentation of publication agreements in RCT
protocols and their concordance with reported statements in
journal publications. Results: Six-hundred and forty-seven
RCT protocols were eligible (Figure 1), of these, 456 (70.5%)
mentioned an agreement regarding publication of results.
Of these, 156 (34.2%) documented an industry-sponsor’s
right to disapprove proposed manuscripts; 237 (52.0%) an
industry-sponsor’s right to review any manuscript before
publication; and 39 (8.6%) of agreements had no constraints
regarding publication. The remaining protocols referred
to separate agreements that were not accessible to us.
We identified 388 publications corresponding to the 647
protocols (60%); in 290 (74.7%) authors did not report
any agreement regarding publication although 197 of the
corresponding 290 protocols (67.9%) explicitly mentioned
such agreements. Only 30 (30.6%) out of 98 (25.3%)
publications reporting on agreements about publication were
concordant with statements in the corresponding protocols
(Table 1). Conclusions: Agreements on publication rights
between industry sponsors and academic investigators are
common in RCT protocols. In a third, the industry sponsors
retain the right to disapprove manuscripts for publication.
Journal articles seldom report on publication agreements
and, if they do, statements are often discrepant with the trial
protocol.
75
Figure 1
76
RO 12.8
Are Health Technology Assessment (HTA)
funded trials using systematic reviews to
inform their design? A retrospective cohort
Bhurke S1 , Cook A2 , Tallant A1 , Young A1 , Williams E1 , Raftery J3
1 National Institute for Health Research Evaluation, Trials
and Studies Co-ordinating Centre (NETSCC), United Kingdom;
2
University of Southampton and University Hospital Southampton
NHS Foundation Trusts, United Kingdom; 3 Wessex Institute,
University of Southampton, United Kingdom
Background: Limited evidence exists on how systematic
reviews are used in the design of new trials. Jones
and colleagues (2013) investigated this using a cohort of
randomised controlled trials (RCTs) funded by the National
Institute for Health Research (NIHR) Health Technology
Assessment (HTA) programme during the period 2006 to
2008. Their study found that 11 (23%) of 48 applications made
no reference to a systematic review. Twenty (54%) of the
37 applications referenced a systematic review and reported
their use in designing of the proposed trial. Objectives: To
replicate and validate Jones’ study, to explore the reasons
for applications not referencing a systematic review. The
study also investigated a cohort of NIHR HTA trials funded
during 2013 to identify if there were improvements over
time. Methods: Two cohorts of NIHR HTA-funded RCTs were
included. Cohort I included the same trials as Jones et al
(except for one trial that has been discontinued) and cohort
II included all trials funded during 2013. Two reviewers
undertook data extraction independently.
Descriptive
statistics were used and no formal statistical comparisons
were conducted. Results: Nine trials were identified in cohort
I (19%) and three in cohort II (9%) that did not reference a
systematic review, but each had a justifiable reason for this.
Systematic reviews were referenced in 85% of NIHR HTA trials
and nearly 62% referenced more than one systematic review.
In total 108 systematic reviews were referenced, 43 of which
were Cochrane systematic reviews. Conclusions: Systematic
reviews were referenced in 85% of NIHR HTA trials. Fifteen
per cent of trials that did not reference a systematic review
had justifiable reasons. NIHR requires that proposals for
primary research are justified by existing evidence and our
study confirms that this was true for all NIHR HTA trials that
could do so.
RO 12.9
Global evidence, local research: a country
map of included studies from Cochrane
Systematic Reviews
Yao L1 , Cui R1 , Wang X1 , Wang Q1 , Wei D1
1
Evidence-Based Medicine Center, School of Basic Medical Sciences,
Lanzhou University, China
Background: Cochrane Systematic Reviews (SRs) are one
source of high quality evidence, which are widely used
in many countries in the development of clinical practice
guidelines and health policies.
However, worldwide,
different ethnic groups have different tolerance levels to
the same intervention. So it is important to analyze
the geographical distribution of included studies from
Cochrane SRs. Objectives: To analyze the geographical
distribution of included studies from Cochrane SRs. Methods:
Three-hundred Cochrane SRs with full text were randomly
selected by computer from the Cochrane Database of
Systematic Reviews (CDSR), and the countries or regions
of their included studies were analyzed. Results: The 300
Cochrane SRs included 2456 original studies. Among these
2456 studies, 1033 (42%) were conducted in Europe, 779
(32%) were conducted in North America, 442 (18%) were
conducted in Asia, 107 (4%) were conducted in Oceania, 60
(3%) were conducted in Africa and 35 (1%) were conducted
in South America. Conclusions: Seventy-five per cent of
studies were conducted in Europe and North America. Few
original studies come from Asia, South America, Africa and
Oceania. The unbalanced geographical distribution of original
studies might cause indirectness when the Cochrane SRs are
disseminated throughout the world.
Oral session 7
Communicating evidence
O 7.1
Presenting evidence of effectiveness of
interventions and more: the evidence of
effects page
Nunan D1 , Heneghan C1 , Mahtani KR1 , Howick J1 , Thompson M1
1
University of Oxford, United Kingdom
Background: Patients are increasingly expected to be
actively involved in their care.
Moreover, making
evidence-based decisions with patients to aid healthcare and
treatment decisions is well recognised. Few evidence-based
communication tools exist that successfully engage and aid
consumer understanding in a format that also facilitates
evidence-based clinical decisions. Objectives: To develop a
tool that presents the best available evidence for treatment
effects that enables both patients and clinicians to make better
informed treatment and healthcare decisions. Methods:
Using treatment of hypertension with angiotensin converting
enzyme inhibitors (ACEi) as an example, we searched the
output of the Cochrane Hypertension Review Group for
relevant systematic reviews and found one suitable review. We
extracted data into Excel on the mean effect (95% confidence
limits (Cl)) of 14 ACEi on systolic blood pressure (SBP) only.
These data were used to create a modified bar chart. Each
ACEi was displayed in descending order according to the
certainty of effects based on 95% Cl. The modified chart was
inserted into a table column with the heading ’Blood pressure
effect performance’. Other columns included ‘Dose’, ‘Cost’,
‘Sample size’ and ‘Duration’. These columns provide data on
77
the dosage of drug for the observed blood pressure effect,
the cost on a daily basis based on the dose, the number
of studies and participants for the observed effect and the
duration of treatment/follow-up of these studies. Individual
sections with the headings ‘Technical Information’, ‘Cost
information’, ‘Dose information’, ‘Quality information’ and
‘Usage information’ are placed under the table and provide
clarification and further details of the information contained
therein. Results: We present the methods used to derive the
first Evidence of Effects Page (EEP) as a new and effective way
to present the evidence for treatment effects. Conclusions:
EEPs for treatments of most health conditions can now be
developed, and their efficacy in improving informed and
shared-decisions can be assessed in suitable trials.
O 7.2
Increasing patient comprehension of benefits
and risks of medicines using graphics
Beyer A1 , Hoekstra T1 , Kingma B1 , Fasolo B2 , Hillege H1
1
University of Groningen, The Netherlands; 2 London School of
Economics and Political Science, United Kingdom
Background:
Graphics are known to be useful in
communicating complex information, however they are rarely
used in regulatory communication with patients. Objective:
To measure patients’ comprehension of benefit and risk of
medicines using five presentation formats (text, table, bar
graphs, pictograms and survival curves). Methods: Data
were collected via a web-questionnaire from patients in the
United Kingdom, France and the Netherlands diagnosed with
atrial fibrillation (AF), breast cancer (BC) and type II diabetes
(DB). Patients were shown presentation formats for benefits
and risks of medicines specific to their disease then asked
comprehension questions after each presentation. Patients
were also asked to indicate their preferred format. Results:
We report data for 770 patients (419 DB, 161 AF and 190 BC):
age range 46 to 75 years; predominantly male for AF and DB
and all female for BC; only 36% to 43% had more than 12 years
of education. The table format was most understandable
for AF patients with 71% responding correctly to all three
questions for both benefits and risks. Among the BC patients
the table format was also easier to understand with 82%
responding correctly for the risk questions and 70% on the
benefit questions. The majority (69%) of the BC patients also
responded correctly for the benefit questions when presented
in a survival curve. For the DB patients, 80% responded
correctly for the risk questionsm but only 54% for the benefit
questions when presented with the table format. The results
show that across all formats the DB patients did not easily
comprehend the benefits. Patients across all disease areas
preferred the table format. Among BC patients 49% ranked
the table as the most preferred, while for DB and AF it was
43% and 39% respectively. Conclusion: The textual format
currently used by regulators to communicate benefits and
risks to the public does not result in high comprehension
among patients. Regulators should consider adding tables to
better support patient comprehension.
O 7.3
User-involvement in a Cochrane systematic
review update: using structured methods to
enhance clinical relevance, usefulness and
usability
Pollock A1 , Campbell P1 , Baer G2 , Choo PL1 , Morris J3 , Forster A4
1
Glasgow Caledonian University, United Kingdom; 2 Queen
MargaretUniversity, UnitedKingdom; 3 UniversityofDundee, United
Kingdom; 4 University of Leeds, United Kingdom
Background: Active involvement of people with a health
condition may enhance the usefulness of systematic review
evidence, addressing barriers to uptake of review findings.
Challenges to the development of effective methods of
involvement within reviews are compounded by poor
description and limited evaluation of involvement within
many reviews. Objectives: To describe the structured
methods used to involve stroke survivors, carers and health
professionals in an update of a Cochrane Systematic Review
relating to physiotherapy after stroke, and explore the
perceived impact of involvement. Methods: After securing
relevant funding and ethical approval, we recruited stroke
survivors, carers, physiotherapists and educators, and held
three pre-planned meetings during a Cochrane Review
update. We used formal group consensus methods, based on
nominal group techniques, to reach consensus decisions on
key issues relating to the structure and methods of the review.
Meetings were audio-recorded, and transcribed data used to
supplement data on voting decisions. User-group members
completed a questionnaire exploring perceived impact of
involvement. Results: The user-group comprised 13 people,
including stroke survivors, carers and physiotherapists with
a range of different professional experience. At Meeting 1
consensus was reached that the methods of categorising
interventions used in the original Cochrane Review were no
longer appropriate or clinically relevant (11/13 participants
disagreed with previous categories). At Meeting 2 group
members discussed and reached consensus over a new
method of intervention categorisation (12/12 agreed) and at
Meeting 3 key messages emerging from the completed review
were agreed. All participants strongly agreed that the views of
the group impacted beneficially on the review update. They
further believed that other Cochrane Reviews would benefit
from involvement of a similar user-group. Conclusions: We
involved stroke survivors and carers in reaching consensus
decisions relating to our review by adopting clearly described,
structured methods. We believe that this approach has
important implications for future Cochrane Reviews.
78
Figure: The conceptual framework of the Ecosystem
O 7.4
Trustworthy digital evidence ecosystem to
improve diagnosis and treatment of patients
Vandvik PO1 , Agoritsas T2 , Kunnamo I3 , Guyatt G2 , Brandt L4
1
Institute for Health and Society, Faculty of Medicine, University
of Oslo, Norway; 2 Department of Clinical Epidemiology and
Biostatistics, McMaster University, Canada; 3 Department of
General Practice and Primary Health, University of Helsinki, Finland;
4 Department of Medicine, Innlandet Hospital Trust-division Gjøvik,
Norway
Background: Major advances in standards, systems
and technological platforms for evidence production and
dissemination may together reduce waste and increase
value in medical research, reduce information overload and
result in better decisions at the point of care. Innovative
technological platforms can connect people doing primary
research, systematic reviews, guidelines, those creating
computerized decision support systems and those involved
in quality improvement. Such platforms –which we call an
evidence ecosystem –can interact to create, disseminate
and implement trustworthy research evidence in clinical
practice. Objectives: In this first part of our project to
create an evidence ecosystem we developed a conceptual
framework and demonstrated its feasibility and relevance
through a real life example. Methods: The framework for the
Ecosystem uses a PICO (patient, intervention, comparator,
outcome)-based shared health data model developed in
collaboration with several partners - including Cochrane –that
adheres to updated and internationally accepted standards
and systems (e.g. GRADE) for trustworthiness. The data-model
is implemented in a web-based authoring and publication
platform (MAGICapp) used to create, disseminate and
dynamically update evidence summaries, recommendations
and decision aids. This is integrated with other innovative
electronic platforms, e.g. Covidence for key steps in
systematic reviews production, and EBMeDS for decision
support systems. We applied the ecosystem to address
the issue of overtreatment with surgery for meniscal tears.
Results: The figure illustrates the ecosystem conceptual
framework. We will, at the Colloquium, demonstrate
how such an ecosystem can facilitate the processing of
high quality evidence from primary research (nationwide
observational study and randomized trials of meniscectomies)
into systematic reviews, guidelines and decision support tools,
followed by quality performance measures and observational
studies to document change in practice and outcomes, using
overtreatment of meniscal tears as an example. Conclusions:
A living evidence ecosystem could improve diagnosis and
treatment of patients.
Oral session 8
Setting priorities to overcome
inequality
O 8.1
The global burden of infectious disease as
reflected in the Cochrane Database of
Systematic Reviews
Marzec NS1 , Dmitruk S1 , Boyers L2 , Karimkhani C3 , Czaja CA1 ,
Trikha R4 , Janoff EN5 , Dellavalle RP6
1 University of Colorado School of Medicine, USA; 2 Georgetown
University School of Medicine, USA; 3 Columbia University College
of Physicians and Surgeons, USA; 4 Rosalind Franklin University of
Medicine and Science, USA; 5 Department of Veterans Affairs Eastern
Colorado Healthcare System, USA; 6 Cochrane Skin Group, USA
79
Background: As research funding continues to be limited,
an approach for setting research priorities is needed. The
Cochrane Database of Systematic Reviews (CDSR) publishes
rigorous and timely systematic reviews as well as protocols,
which together represent the breadth of current and planned
prioritization in the Cochrane database. This marker of
research priority can be compared to the global burden of
disease based on the estimated disease burden measured in
disability-adjusted life years (DALYs) reported by the Global
Burden of Disease 2010 Project (GBD). This comparison sheds
light on current research priorities as well as a possible
method of determining future priorities. Objectives: To
assess the representation of infectious diseases within the
systematic reviews and protocols in the CDSR and compare
this to worldwide infectious disease burden as reported by the
GBD. Methods: Two investigators independently searched
the CDSR database of systematic reviews and protocols
for titles, abstracts and keywords related to the infectious
diseases reported in the GBD 2010. Reviews and protocols
were included or excluded based on whether they added to
knowledge regarding the infectious disease that had been
searched for. If needed, consensus was reached by a third
investigator. The number of reviews and protocols found in
the CDSR was used as a surrogate to estimate the existing
body of literature regarding each disease and compared
to the global burden of the same disease (measured in
DALYs). Results: The CDSR was searched for literature
relating to 45 infectious diseases based on GBD categorization.
These were then divided into three groups: those that were
appropriately represented based on DALY (Table 1), those
that were under-represented (Table 2), and those that were
over-represented (Table 3). Conclusions: Fourteen of the
45 infectious diseases investigated were under-represented
in the current literature; 12 were over-represented and 15
were appropriately represented. Infectious diseases research
priorities could be informed better using this information.
80
O 8.2
Strategies for improving health systems in
low-income countries: lessons learnt from
four overviews of systematic reviews of
health systems interventions
Pantoja T1 , Opiyo N2 , Ciaponni A3 , Herrera C4 , Lewin S5 , Oxman
A5 , Paulsen E5 , Rada G6 , Wiysonge C7
1 Pontificia Universidad Catolica de Chile, Cochrane EPOC Group,
Chile; 2 Child and Newborn Health Group, Kenya Medical Research
Institute/Wellcome Trust Research Programme, Kenya; 3 Argentine
Cochrane Centre IECS, Institute for Clinical Effectiveness and
Health Policy, Argentina; 4 Department of Public Health, Evidence
Based Health Care Program, Pontificia Universidad Católica
de Chile, Chile; 5 Global Health Unit, Norwegian Knowledge
Centre for the Health Services, Norway; 6 Department of Internal
Medicine, Evidence-Based Health Care Program, Faculty of
Medicine, Pontificia Universidad Católica de Chile, Chile; 7 Centre
for Evidence-Based Health Care, Stellenbosch University, South
Africa
Background: We have conducted four Effective Practice
& Organisation of Care (EPOC) overviews of systematic
reviews (SRs) of interventions for improving health systems in
low-income countries (LICs), including delivery, financial and
governance arrangements and implementation strategies.
Our objectives were to: (1) provide a broad overview
of what is known about the effects of health systems
interventions in LICs based on the findings of up-to-date
SRs; and (2) identify priorities for new SRs and primary
research of such interventions. Objectives: To reflect on
the overall findings and lessons learnt from conducting four
overviews of systematic reviews of strategies for improving
health systems in LICs. Methods: We used structured
discussions to identify the key challenges and lessons from
conducting the overviews. These discussions took place
during face-to-face and online meetings over a 24-month
period (March 2013 to March 2015). Results: The four
overviews will be published in the Cochrane Library in 2015,
and structured summaries of the included reviews will be
available at www.supportsummaries.org. The overviews
summarize a very substantial body of evidence from Cochrane
and non-Cochrane reviews on the effects of interventions for
improving health systems. However, there are important
differences in the extent of the available evidence across
the four overviews, with substantially less evidence available
for governance and financial arrangements. Synthesizing
the findings of overlapping reviews of the same intervention
was challenging. Also, for many comparisons the evidence
is of low or very low certainty (mainly due to risk of bias
and inconsistency), and for many interventions there are
few data on equity outcomes, resource use or unintended
consequences. Conclusions: The overviews are an important
resource for evidence-informed decision making on health
systems in LICs. They are also a resource for identifying
important gaps in the body of evidence for this field and have
informed priorities for new reviews and research.
O 8.3
Using priority setting to enhance the policy
relevance of systematic reviews: a case study
from the South African Initiative for
Systematic Reviews on Health Policies and
Systems
Odendaal W1 , Daniels K1 , Lewin S2 , Kredo T3 , Young T4
1
South African Medical Research Council, South Africa; 2 South
African Medical Research Council; Norwegian Knowledge Centre for
the Health Services, South Africa; Norway; 3 South African Cochrane
Centre, South African Medical Research Council, South Africa;
4 Centre for Evidence-Based Health Care, University of Stellenbosch;
Council, South Africa
Background: Identifying and prioritising systematic review
topics that address stakeholders’ interests is not easy,
particularly when time and resources are constrained. The
Alliance for Health Policy and Systems Research recently
81
funded a two-year initiative in South Africa to conduct four
systematic reviews on health systems questions. Central to
the initiative was engagement with stakeholders to identify
and select priority review questions. Objectives: To share
lessons learned in engaging stakeholders and identifying
their priorities for new systematic reviews. Results: We used
a multi-step process to identify priority topics for reviews. We
invited 154 public health policymakers, managers, providers
and researchers across sub-Saharan Africa to submit priority
topics for health system reviews. Ninety topics were
received. A core team then grouped these into broad themes;
transformed these themes into systematic review questions;
and searched for existing reviews on these questions. This
resulted in a shortlist of 39 review questions. A small advisory
panel then used an iterative process to select four review
topics. This included ranking each question as high or
low priority and discussions within the panel to resolve
differences in ratings. Conclusions: The strengths of this
prioritisation approach included: (1) rapid engagement of a
wide range of health stakeholders; and (2) the identification
of review questions that address important national and
regional health systems questions. A key challenge was
transforming the themes into review questions as, (1) many
of the suggested topics were not questions that could best
be addressed using systematic review methods; (2) some
topics encompassed multiple questions that had to be split
into reviewable questions; and (3) several topics were partly
covered by existing reviews. In addition, the final shortlist of
review questions was not verified with those who proposed
these topics, and the questions may therefore have diverged
from the original topic.
O 8.4
Building a systematic review movement
in the humanitarian field
Ott EM1
Oxfam GB, United Kingdom
1
Background and objectives: Humanitarian actors and
researchers have amassed evidence about the state of
knowledge in the humanitarian sector, including grey
literature assessments of what works and what does
not. Synthesizing this information and gaining research
uptake by policymakers and humanitarian practitioners
remains challenging. The Humanitarian Evidence Programme
commissions a series of systematic reviews to distil evidence
in the humanitarian sector and focuses on uptake and
communication of findings to key stakeholders, with the
ultimate goal of improving humanitarian policy and practice.
The programme is a partnership between Oxfam GB and
Feinstein International Center (FIC) at Tufts University, funded
by UK Aid. This partnership is unusual in its emphasis on
practitioners and the symbiotic involvement of individuals
occupying multiple roles in the humanitarian field. Methods:
The programme collected data for deciding the review
questions and research uptake strategy through a survey
with 273 responses from individuals based in 55 different
countries with an average of 15 years of experience as well as
45 key informant interviews. Results: The data show that key
humanitarian practitioners access and use research through
conversations with key colleagues as well as through briefings,
articles, and the media. Importantly, some individuals act
as ‘nodes’ through which a number of individuals access
trusted research. The resulting research uptake strategy for
the programme uses three core approaches: 1. meaningful
consultation throughout the life process of the programme;
2. targeting of key thought leaders; and 3. leveraging existing
platforms and using a multi-channel communications strategy
to increase reach. Conclusions: For the Humanitarian
Evidence Programme, this research contributed to an iterative
process to develop a list of systematic review questions in the
humanitarian field and a research uptake plan. The first call
for proposals solicited a strong response, showing interest in
this programme. For the broader systematic review field, this
model may be of interest and adaptable for other programmes
and reviews.
Oral session 9
Systematic review and
meta-analysis methods
O 9.1
Extent and handling of missing dichotomous
outcome data in 100 Cochrane and
non-Cochrane systematic reviews
Kahale LA1 , Diab B1 , Brignardello-Petersen R2 , Mustafa R3 , Busse
JW2 , Agarwal A2 , Kwong J4 , Li L4 , Neumann I5 , Lopes LC6 , Olav
Vandvik P7 , Briel M8 , Iorio A2 , Guyatt GH2 , Akl EA1
1 American University of Beirut, Lebanon; 2 McMaster University,
Canada; 3 University of Missouri-Kansas City, USA; 4 Sichuan
University, China; 5 Pontificia Universidad Católica de Chile, Chile;
6 University of Sorocaba, Brazil; 7 Norwegian Knowledge Centre
for the Health Services, Norway; 8 University Hospital of Basel,
Switzerland
Background: Reports of randomized clinical trials (RCTs) do
not typically report whether certain categories of participants
(e.g. non-compliers, those who withdraw consent) were
followed-up for the outcome(s) of interest.
It is not
clear how systematic reviews (SRs) address this lack of
information, i.e. assume they have missing participant
data (MPD) or not. Objectives: To describe how SRs
report and handle categories of participants that could be
potentially counted as having MPD. Methods: We included SRs
reporting a group-level meta-analysis of a patient-important
dichotomous efficacy outcome, with a statistically significant
effect estimate.
Thirteen reviewers, working in pairs,
independently extracted data from eligible SRs. We focused
on 10 categories that could be counted as potentially having
MPD: ‘ineligible participants’, ‘did not receive any treatment’,
‘withdrew consent’, ‘outcome not assessable’, ‘explained and
unexplained lost to follow-up’, ‘dead’, ‘experienced adverse
events’, ‘non-compliant’, ‘discontinued prematurely’, and
82
‘cross-over’. We assessed whether the systematic reviewers:
(1) planned in the methods section to collect information
about those categories; (2) reported in the results section,
the number of participants in those categories; and (3)
reported on methods for handling MPD in their primary
meta-analysis. Results: We included a random sample of 100
Cochrane and non-Cochrane SRs published in 2012. Tables
1–3 report our main findings. Twenty-five per cent of the SRs
reported a plan to collect information about the categories
of interest in their methods section, whereas 45% reported
in their results section the number of participants in those
categories. Only 13% reported a method to handle MPD in
their primary meta-analyses, which were primarily complete
case analysis. Conclusions: Most SRs do not explicitly report
sufficient information on categories of participants that could
be potentially counted as having MPD, or handle MPD in their
primary meta-analysis.
O 9.2
The changing world of data sharing and data
transparency: what does this mean for
individual participant data reviews?
Nolan S1 , Tudur Smith C1 , Marson A1
1
Setting: There are many advantages to undertaking an
individual participant data (IPD) meta-analysis compared
to a summary data approach. IPD meta-analysis allows
more flexible, complex statistical exploration of data, often
increasing the number of clinical questions that can be
addressed. In many contexts where summary data are
insufficient or unavailable, an IPD analysis is the only
feasible approach. However, procedures for obtaining IPD are
time-consuming, resource intensive and often unsuccessful
due to unavailability of data or concerns regarding patient
privacy, collaboration and data sharing. Many initiatives
promoting data sharing and data transparency within the
pharmaceutical industry have developed quickly in recent
years; such changes could have a large impact on IPD
meta-analysis. Background: From 1997 to present, the
Cochrane Epilepsy Group has undertaken IPD meta-analyses
and an IPD network meta-analysis is currently being updated.
For all studies identified as eligible for these reviews, an IPD
request was made to the lead or corresponding author or
the sponsoring organisation if a study was industry-funded.
Methods of contacting relevant data providers, requirements
of an IPD request, content of data provided and success rates
of IPD requests have changed dramatically over time. The
launch of ClinicalStudyDataRequest.com (CSDR) platform and
the SAS multi-sponsor data access environment in the last two
years to allow researchers to request and remotely analyse
de-identified IPD has been a significant change. Discussion:
IPD requests from 40 studies eligible for the IPD network
meta-analysis from 2011 to the present will be discussed
including requests pre-, during, and post the launch of CSDR.
Advantages and practical issues of CSDR and the SAS data
access system will be presented and the potential impact of
these new initiatives on future IPD analyses will be discussed.
83
O 9.3
Multivariate meta-analysis of multiple
correlated outcomes with individual
participant data: how much do we gain?
Frosi G1 , Riley R2 , Williamson P1 , Kirkham J1
1
Biostatistic Department of University of Liverpool, United
Kingdom; 2 Research Institute of Primary Care & Health Sciences,
Keele University, United Kingdom
Background: Missing treatment effect estimates for particular
outcomes in a study have the potential to affect the
conclusions in a meta-analysis, especially if missingness is
a result of outcome reporting bias (ORB). As well as missing
treatment effect estimates at the study level, outcome data
may also be missing within studies at the individual participant
level. Multivariate meta-analysis of individual participant data
(IPD) has the potential to overcome the impact of both these
problems, by utilising the correlation between outcomes.
Objectives: To investigate, in a range of ORB and missing data
scenarios, the magnitude of bias in pooled treatment effect
estimates for multiple outcomes using standard (univariate)
meta-analysis, and to quantify how much the ‘borrowing
of strength’ (BoS) from multivariate meta-analysis reduces
such bias and increases precision. Methods: A simulation
study was conducted where IPD was generated from an
assumed multivariate fixed-effect model, and missing data
were created at either the patient-level or the study-level, or
both. In each simulation, the bias, precision and coverage
of univariate and multivariate methods was compared, and
the BoS quantified. Results: Results show that the BoS in a
multivariate model can substantially reduce the magnitude
of bias and increase precision in the pooled estimates,
especially when ORB is present and when correlation is
modelled at both the patient-level and the study-level. In
a missing at random scenario (0.8 w/s correlation) the BoS
was 34.9% (outcome 2), meaning a reduction of 35% in the
variance of the pooled estimate. Moreover BoS increases
as the correlation and amount of missing data increases.
Conclusions: Meta-analysis results may be unreliable if there
are missing outcome data. A multivariate meta-analysis
approach is a potential statistical solution for reducing the
impact of missing data, and is especially appealing when IPD
are available to deal with missing data at both the patient and
study levels.
O 9.4
Evaluation of the consistency between
network meta-analysis and standard
pairwise meta-analysis: a
meta-epidemiological study
Yuan J1 , Fu X1 , Mao C1 , Yang Z1 , Tang J1 , Yuan J1 , Zhang Y1
1 Division of Epidemiology, School of Public Health and Primary
Care, The Chinese University of Hong Kong, Hong Kong, China
Background: Network meta-analysis is a relatively new
type of data synthesis statistics, there is still considerable
concern about its validity. Objectives: To evaluate the
consistency between standard pairwise meta-analysis and
network meta-analysis and explore the potential factors that
are associated with the inconsistency. Methods: PubMed,
the Cochrane Library, EMBASE were searched and reference
lists of relevant methodological reviews were checked to
identify network meta-analyses that reported the estimate
effects of both standard pairwise meta-analysis and network
meta-analysis, or provided raw data to allow us to calculate
direct estimates. We assessed both quantitative consistency
(whether the difference between the matched estimates
is statistically significant) and and qualitative consistency
(whether the direction or statistical significance of the
paired estimates is different). The association between
inconsistency and potential influential factor was tested by
the χ 2 test. Results: A total of 1901 matched estimates of
standard pairwise meta-analyses and network meta-analyses
were obtained from 90 studies. Of which 20 (1.1%, 95%
confidence interval (CI) 0.6% to 1.5%) and 367 (19.3%, 95%
CI 17.55% to 21.1%) matched estimates showed quantitative
inconsistency and qualitative inconsistency, respectively. The
qualitative inconsistency was significantly associated with
continuous data (P value 0.002), random-effects network
meta-analysis model (P value 0.045), closed loop in the
network (P value 0.001), invalid coherence assumption (P
value 0.036), and high risk of publication bias (P value
0.019). The quantitative and qualitative inconsistency rates
for matched estimates of indirect comparisons and standard
pairwise meta-analysis were 9.1% (5/55) and 32.7% (18/55),
respectively. Conclusions: Network meta-analysis is a reliable
statistical procedure if it is done correctly. It hardly shows
any quantitative inconsistency when compared to standard
pairwise meta-analysis. However, qualitative inconsistencies
exist in about one-fifth of the estimates, which should be
interpreted with caution.
Translations
RO 13.1
One year in: facts and figures about the
impact of Cochrane’s translation strategy
Ried J1 , Hassan H2
1
Cochrane Central Executive, Translations Co-ordinator, Germany;
2
Cochrane Central Executive, Translations Support Officer, United
Kingdom
Background: Cochrane approved a translation strategy in
2014 with the aim of increasing the capacity and providing
central coordination for sustainable Cochrane translation
activities in a broad range of languages. The translation
strategy contributes to strategic goals 2 and 3: ‘Making
84
our evidence accessible’; and ‘Advocating for evidence’.
Objectives: To describe the main components of Cochrane’s
translation strategy, and to show what impact it has made
to date on translation and related activities. Methods: The
analysis will consist of the collection of metrics as well as a
brief quantitative and qualitative survey with our language
project managers, and will investigate the following aspects:
- the number of languages we are translating into, and their
importance on the global linguistic landscape; - how much
and what content has been translated; - how many people
are involved in translations, and in what capacity; - how
much and what type of resources the projects have; - how
productive the projects are; - how sustainable the projects
are; - how translations may have led to local outreach and
dissemination activities; - how translations may have led to
a local increase of interest in Cochrane (e.g. access to our
content, media coverage); - whether translations may have
led to new funding or partnerships. Retrospective comparison
data will be provided where possible. Results: The results
from the analysis will be available for the Colloquium. As of
March 2015, we have published more than 12,000 translations
of Cochrane summaries across 12 different languages. The
majority of our translation teams rely on low resources and
volunteers. Most new projects require several months to
prepare and then reach a certain continuity, and a number of
teams have successfully reached this point even without any
dedicated funding. Conclusions: While the implementation
of the translation strategy is still at its early stages, increase
in production, languages and access already suggests that
our approach has a lot of potential to increase our capacities
even in low resource settings, and with relatively low central
investment.
members of Cochrane Croatia. Crowdsourcing was identified
as a potential low-resource approach to increase translation
capacity. Objectives: To test the involvement of volunteer
medical students and health professionals in translating PLS
into Croatian and to measure the impact on production
of translations. Methods: In July 2014, we started inviting
students and university colleagues to participate as volunteers
in our translation project. A public call was also published
through the Cochrane Croatia Facebook page, Cochrane
Croatia website and in the University newspapers. Medical
school teachers incorporated translation of one PLS as a
mandatory part of the exam in research methodology courses,
and some of the students volunteered to translate more after
the exam. We developed supporting material describing
the translation process and guidelines. All translations
submitted by the volunteers are checked and edited by
Cochrane Croatia’s volunteer staff to ensure accuracy and
consistency. Volunteers are acknowledged along with the
published translations. Results: Fifty-six volunteers signed
up to contribute translations (as of February 2015). In the
first 18 months of the translation project, prior to adopting a
crowdsourcing approach, we translated about 200 PLS. Within
the first six months of crowdsourcing we were able to complete
350 additional PLS translations–about five times more than
before. Volunteer time of the two project administrators
devoted to the translation project approximately doubled
after introduction of crowdsourcing. Conclusion: Involving
volunteers using a crowdsourcing approach and assigning
translations to students as a part of their medical curriculum
substantially increased our capacity to translate Cochrane
evidence, but it required significantly more volunteer time
from the translation managers.
RO 13.2
RO 13.3
Involving volunteers in translating Cochrane
Plain Language Summaries to increase
capacity
The importance of critical revision
of translated abstracts
Puljak L1 , Ried J2
1
Cochrane Croatia, Croatia; 2 Translations Co-ordinator, Cochrane
Central Executive, Germany
Background: Cochrane Croatia started translating the
Plain Language Summaries (PLS) of Cochrane Reviews into
Croatian in 2013 to overcome the language barrier. Initially,
translations were mainly completed by two volunteer staff
Logullo P1 , Albuquerque JV1 , Porfirio G1 , Martimbianco ALC1 ,
Riera R1 , Atallah AN1 , Torloni MR1
1
Brazilian Cochrane Center, Brazil
Reviewing the translations of abstracts done by volunteers: is
it worth the effort? Experience from the Brazilian Cochrane
Center: Background: Since 2013, volunteers from the
Brazilian Cochrane Center (BCC) have translated English
Cochrane Abstracts into Portuguese; these are uploaded to
Table 1
85
the Cochrane Library only after being reviewed by a language
and a content expert. English belongs to the Germanic
family of languages, while Portuguese is part of the Italic
language family, therefore, the construction of sentences
is different, as well as the use of double negatives, the
choice for verbal tenses and the order of words. These
differences make translating Cochrane Abstracts a challenging
task. To help translators, we created and distributed a
short guide on the most common mistakes in translating
these abstracts to Portuguese. Objectives: To assess the
most frequent Portuguese language errors found in abstracts
translated by BCC volunteers. Methods: We analyzed
the last 12 abstracts translated by volunteer physicians,
psychologists, physical therapists and a dietitian.
A
Portuguese language expert experienced in reviewing medical
texts and a physician reviewed all abstracts for language and
content accuracy. Language errors and corrections were
classified by type and frequency. Results: Most corrections
(mean 15.75/translation) concerned style and were made to
improve text readability. However, all 12 texts also had
grammatical (12.37/translation) as well as misspelling errors
or ’typos’ (11.85/translation; Table 1). Conclusions: Although
the volunteer translators are highly motivated and qualified
health professionals, their translations can be considerably
improved by a professional language review. Since translated
Cochrane Abstracts are read by thousands of professionals
and consumers worldwide, it is important to use a review
process to ensure not only that their content is accurate, but
also that the text is grammatically correct, and is easy to
read and understand for all users. This language review is
important and should be encouraged in other languages to
ensure the quality of translations produced by Cochrane.
for this project. Objectives: To create usable, reliable,
fully automatic translation of public health information,
initially testing with translation from English into Czech,
Polish, Romanian and German. Methods: We will use recent
advances in machine translation (MT) to create a system for
the automatic translation of public health information, with
a focus on preservation of meaning. We will include recent
work on domain adaptation, translation into morphologically
rich languages, terminology management, and semantically
enhanced MT to build a reliable system for the health domain.
We will iterate cycles of incorporating improvements into
the MT systems annually, with careful evaluation and user
acceptance testing. We will develop metrics to evaluate
the quality and measure the impact on post-editing of the
obtained results in each cycle. Cochrane and NHS24 content
will also be translated in each cycle and published on their
websites, and users invited to participate in a survey to give
feedback on the quality and usability of the translations. Web
usage statistics will be analysed. Results and Conclusions:
The first version of the MT system is to be deployed in
Sept 2015. Preliminary results and conclusions will be
presented at the Colloquium as available. The focus will be on
process and the Cochrane use case, rather than technological
details.
RO 13.5
Lessons from translating Cochrane Reviews
in nursing into Korean
Park M1 , Lee M1 , Jang I2
1
College of Nursing Chungnam National University, South Korea;
2
Chungnam National University Hospital, South Korea
RO 13.4
Health in my Language: health domain
adapted machine translation as a means to
tackle the resource issues for translation
Ried J1 , For the HimL consortium N2
1
Cochrane Central Executive, Translations Co-ordinator, Germany;
2
University of Edinburgh, Charles University Prague, Ludwig
Maximilian University of Munich, NHS 24, Cochrane, and Lingea, NA
Background: To an ever-increasing extent, web-based
services are providing a frontline for healthcare information
in Europe. They help citizens find answers to their questions,
and they help them understand and find the available local
services. However, due to the number of languages spoken
in Europe, and the mobility of its population, there is a
high demand for these services to be available in many
languages. In order to satisfy this demand, we need to rely
on automatic translation, as it is infeasible and unaffordable
to translate manually into all languages requested. Health
in my Language (HimL) is an EU-funded, three-year project,
aiming at addressing this need. Health information produced
by Cochrane and NHS24 (health information and self care
advice for the people of Scotland) will serve as the test case
Background: There are current projects to translate Cochrane
content into various languages to improve the availability
of its content. In Korea, we started collaborated efforts
(medicine, oriental medicine, nursing, and public health area)
to translate Cochrane Reviews using professional translation
and editing. Objectives: This paper describes the experiences
of translating Cochrane Reviews in the nursing area into
Korean and discusses lessons arising from the experience.
Translation process: We set the high priority topics in the
nursing arena which needed to be translated first. In
nursing, the priority was given to recently published reviews in
gerontology and evidence-based nursing. These translations
were complemented by the Smartling system, which makes
working together easier for the project manager and the
translator. Lessons learned: It is important that the objective
of the translation (literal translation, conceptual translation,
or culturally equivalent translation) be explicitly identified
before the translators begin their work. While the translated
reviews are grammatically correct and follow the wording of
the original, it is hard to translate the professional language
into plain Korean for lay people. There is a need for an
agreement of standardized language in each specialty area.
An interdisciplinary standardized language system needs to
be used. The translation work is time- and effort-consuming,
86
it is highly recommended that the priority list of reviews is
set up with the members’ agreement. It would be helpful
to make a system that facilitates review of the translation
by peers and lay people. A high percentage of the reviews
in nursing concluded that the effects of certain interventions
were not clear. It would be a good strategy to prioritise
those reviews with a clear conclusion. Acknowledgment:
This work was supported by Basic Science Research Program
through the National Research Foundation of Korea (NRF)
funded by the Ministry of Education, Science and Technology
(No. 2010-0024922).
RO 13.6
Russian translations of Cochrane Plain
Language Summaries: experience with and
feedback from an online survey on quality
and importance
Ziganshina LE1 , Yudina EV1
1
Kazan Federal University, Affiliated Centre in Tatarstan of the
Nordic Cochrane Centre, Russian Federation
Background: In 2014 Cochrane initiated its translation
strategy. The Russian translation project was started in
May 2014 by a team of volunteers from the Kazan Federal
University, an affiliated center in Tatarstan of the Nordic
Cochrane Centre. Objectives: To assess the quality of
Russian translations of Cochrane Plain Language Summaries
(PLS) and their potential impact on the Russian speaking
community through user feedback with the aim of furthering
the translation project.
Methods: We conducted an
online survey (3-24 March 2015) via Google Docs. We
invited respondents through the electronic Russian language
discussion forum on Essential Medicines (E-lek), links to a
survey on the Russian cochrane.org website, and invitations
to Cochrane contributors registered in Archie from potential
Russian-speaking countries. We set up the survey in Russian
and English. Results: By 24 March 65 people had taken
part in our survey (56 in Russian, 9 in English), mostly
representing health professions (n = 48; 74%). The Russian
text of translations and basic meaning were clear to the
majority of respondents (n = 63; 97%) to varying degrees. The
respondents rated the quality of translations as ’excellent’
(n = 20; 31%), ’good’ (n = 33; 51%) and ’satisfactory’ (n = 9;
14%). All respondents noted good compliance of the
Russian translation with the original English text. Nearly
all of them recognized the need for Cochrane evidence
for Russia and Russian-speaking countries (n = 63, 97%),
for their work/school/life (n = 61; 94%) and the potential
impact on their practice/attitude to drugs or diagnostic
procedures (n = 60; 92%). Many respondents (n = 42; 65%)
preferred translation texts worded in Russian without precise
compliance to the original text; 24 respondents volunteered
to become members of the translation project.
We
received valuable suggestions for improvement and further
development of the Russian translation project. Conclusions:
The survey provided positive feedback so far. We will continue
the survey to confirm or adjust our findings as we translate
more PLS and to reach out to a wider audience and to attract
new volunteers to work in the project.
RO 13.7
Strategies for eliciting mutual benefit from
translating Cochrane Review content: the
example of Cochrane Kompakt in German
Toews I1 , Voigt-Radloff S1 , Flatz A2 , Nussbaumer B3 , Meerpohl JJ1 ,
von Elm E2
1
2
Cochrane Germany, Germany;
Cochrane Switzerland,
Switzerland; 3 Cochrane Austria, Austria
Background: Cochrane Reviews provide health information
in an era of information overload. With its ’Strategy to
2020’, Cochrane has approved a translation strategy and
established an infrastructure to support translation projects.
The German-language translation project Cochrane Kompakt
(www.cochrane.org/de/kompakt) is a collaboration between
Cochrane entities in Switzerland, Austria and Germany and
actively reaches out to the health professions and their
organisations to maximise benefit from the limited resources
available. Methods: We work with health professionals,
who are interested in disseminating Cochrane Reviews, to
translate Cochrane Plain language summaries into German on
an ongoing basis. Through their networks, they have access
to domain-specific audiences and dissemination channels.
The cooperation we target consists of them lending us their
expertise for translation; in return translated summaries
are co-published in German-language discipline-specific
journals. This increases the dissemination and impact
of translated Cochrane summaries. Via workshops and
seminars, opportunities to get involved in Cochrane Kompakt
are promoted. A standard approach to instruct translators
was developed to ensure quality and consistency. The
coordinators introduced new translators to the web-based
translation platform, Smartling, by using tailored instructions,
and provided them with ongoing assistance. All texts are
redacted by at least two project collaborators. Results:
As of March 2015, we published 174 Cochrane Review
summaries in German on Cochrane Kompakt. Of those,
25 were co-published in scientific journals (e.g. pt Zeitschrift
für Physiotherapeuten), and another 12 are scheduled for
publication in 2015. Mutual benefit for both Cochrane and
the involved health professions increases the number of
translation resources and translations, keeps up motivation
and has a snowball effect with current translators promoting
their own work and recruiting additional volunteers. It
is worthwhile for other translation teams to consider this
strategy. Active involvement of health professions will ensure
that translated content is disseminated to target readerships.
87
RO 13.8
Enhancing dissemination of Cochrane
evidence via German translations of
physiotherapy-related Cochrane Plain
Language Summaries - a collaborative project
Braun C1 , Bossmann T2 , Ehrenbrusthoff K3 , Lohkamp M4 , Jahnke
N5 , Handoll H6
1 Hochschule 21, Department of Health (Physiotherapy), Buxtehude,
Germany; 2 Technische Universität München, Department of Sport
and Health Sciences, Germany; 3 Hochschule für Gesundheit,
Department of Applied Health Sciences, Bochum, Germany; 4 SRH
Hochschule, School of Therapeutic Sciences, Heidelberg, Germany;
5
Cochrane Cystic Fibrosis and Genetic Disorders Group, Liverpool,
United Kingdom; 6 Cochrane Bone Joint and Muscle Trauma Group,
Manchester, United Kingdom
Background: The exponential increase in health information
constitutes a universal challenge, more so for non-English
speakers as new scientific knowledge is almost exclusively
published in English. As part of its Strategy to 2020, Cochrane
is working to ‘‘make Cochrane evidence accessible and useful
to everybody, everywhere in the world’’. A key focus has
been to translate Cochrane abstracts and plain language
summaries (PLS) into different languages, which poses
various challenges as a resource-intensive task requiring both
linguistic and context expertise. Objectives:- To provide
German translations of the physiotherapy-related PLS of 10
Cochrane Reviews by the Bone, Joint and Muscle Trauma
Group (BJMTG). - To examine and document the translation
process and provide feedback to BJMTG on aspects related
to the translation and content of the PLSs. Methods: We
are building on our ongoing initiative to provide German
translations of PLS of physiotherapy-related Cochrane
Reviews on Cochrane websites (www.cochranelibrary.com
and www.cochrane.org/de) and in the German physiotherapy
journal Zeitschrift für Physiotherapeuten. A team of German
physiotherapists, all experienced in translating PLS, with
input from a native English advisor, translate the PLS of the
selected BJMTG Reviews. Translation follows a standardised
process involving at least two translators per PLS and the
use of the translation software Smartling R. Feedback is
documented on a pre-developed form and includes aspects
such as difficulties experienced when translating the PLS into
German plain language and the completeness of the PLS. The
translation work is funded by BJMTG. Preliminary results and
prospects: The project is ongoing; it started in February 2015.
We will report our findings at the Colloquium. So far (March
2015), two translations have been completed. A third one
is underway. We hope that insights gained from our project
will stimulate and facilitate the involvement of professional
groups in other Cochrane translation projects, particularly for
languages outside larger-scale Cochrane translation projects.
RO 13.9
Using social media to disseminate translated
Cochrane evidence and to promote
high-quality health information in
non-English speaking regions
Puljak L1
1 Cochrane Croatia, Croatia
Background: Social media have emerged as a potentially
useful tool for knowledge transfer (KT) in recent years. To
date, however, there is little information about the impact
of dissemination of translated Cochrane Plain language
summaries (PLS) via social media tailored to regional
audiences. Objectives: To test and evaluate the use of a
Croatian Facebook page and Twitter account as dissemination
tools for Croatian PLS translations. Methods: In early 2013,
Cochrane Croatia started translating PLS into Croatian to
overcome the language barrier. In March 2013, a ‘Cochrane
Croatia’ Facebook page was set up. It was mainly used to
post new PLS translations and other relevant information
in Croatian. A Croatian ‘Cochrane Health’ Twitter account
was set up as well in April 2014 for the same purpose.
Results: To date, the Cochrane Croatia Facebook page has
attracted 1858 followers (February 2015), mostly from Croatia
and neighboring countries with similar languages. Most of
the followers are in the 25-44 years age group. The most
popular PLS were related to pregnancy, childbirth, dental
care, occupational health, depression and prostate cancer.
The Facebook page has enabled direct interaction with our
audience, which allowed us to determine that our followers
include lay persons, health professionals and journalists.
As a result of the Facebook activities, we experienced a
substantial increase in media uptake of Cochrane stories from
an average of one per month before Facebook activities to
about 20 per month now (February 2015). The popularity
of Twitter in Croatia is not comparable to that of Facebook.
As a result, the Twitter account attracted only 34 followers.
Conclusion: Facebook proved successful in creating multiple
opportunities to engage with our Croatian target audience,
to disseminate evidence-based health information, and to
increase the uptake of Cochrane evidence by regional media.
Cochrane groups should however focus on the most popular
social media tools in their setting for KT purposes.
88
Education and training
RO 14.2
Train the trainers: Medical Library Literacy
for Health Professionals in Ethiopia
RO 14.1
Building capacity for systematic reviews in
low-income countries beyond the classroom:
the Africa centre for systematic reviews and
knowledge translation
Obuku E1 , Kinengyere A1 , Ssenono R1 , Sewankambo N1
1
Africa Centre for Systematic Reviews and Knowledge Translation
(Africa Centre), Makerere University College of Health Sciences,
Uganda
Background: Researchers in sub-Saharan Africa have limited
knowledge and skills to conduct systematic evidence
syntheses. The ’Africa Centre’ at Makerere University, College
of Health Sciences Uganda was set up to address this gap in the
East African region. Objectives: To describe our experience
in building teams of systematic reviewers in Uganda and the
East African Community. Methods: Design: We administered a
sequential three-phase course to two groups of scientists from
13 research and teaching institutions in Africa (started Aug
2013 and Dec 2015). These modules consisted of a self-driven
introductory learning about systematic reviews for 10 and 21
days respectively: phase 1: face-to-face didactic lectures and
hands-on group work for five days at Makerere University,
Kampala, Uganda; phase 2: technical support (Skype calls,
online metorship) in conducting systematic reviews until
publication of the review report (>18 months). Outcome
measures: Systematic reviewing activities including: question
identification, protocol development, protocol registration,
review execution, publication, and grant application. Results:
Participants (n = 51) came from eight countries including
Rwanda (4), Ethiopia (1), Botswana (3), Southern Sudan (1),
Tanzania (4), Kenya (4), Cameroon (1) and Uganda (33).
Twenty-one (41%) were female scientists. Forty-eight (94%)
and 51 (100%) completed the self-driven introductory learning
and face-to-face sessions respectively. During this period,
participants identified 16 new potential review questions.
Three new protocols have been registered in PROSPERO and
one with Cochrane HIV group; three new protocols are under
development whilst three older protocols were revived and
under conduct. Only one review is published and only one
of three new systematic review grant application attempts
have been successful. Conclusions: These preliminary
results depict some progress in the quest to build systematic
review teams in the East African Community. In our
next steps we aim to collaborate with the South African
Cochrane Centre to conduct a data synthesis (GRADE: Grading
of Recommendations Assessment) and manuscript writing
workshop.
Kendall S1 , Hagstrom C2
1
Mount Sinai Hospital, Canada; 2 University of Toronto, Canada
Background: In response to the Ethiopian Government’s
aim to train 5000 specialist MDs and PhDs and 10,000
Masters graduates by 2018, the Toronto Addis Ababa
Academic Collaboration (TAAAC) was established to co-build
capacity and sustainability in graduate programming at Addis
Ababa University (AAU).Enhancing the discipline of medical
library sciences is essential to the successful support of
this expansion. Objectives/methods: With international
collaboration, a program for clinical medical librarians will
build capacity in library literacy skills, including the accessing
of up-to-date information, with foundational knowledge of
appraising the evolving literature in evidence-based medicine
and critical thinking skills to support medical faculty and
trainees. TAAAC-Library Sciences program has embarked on
a plan for a prolonged partnership between the University
of Toronto (UofT) and AAU by supporting library services
with annual train-the-trainer programs, and the provision of
access by affiliated researchers to the UofT libraries (Ptolemy)
to access online resources including clinical decision support
tools. Results: Responding to the needs outlined by AAU,
librarians and library assistants participate in workshops to
upgrade clinical library skills and then pass this knowledge
to others. We have begun to assess the impact of our
library training sessions and the outcomes of this transfer
of knowledge. We do know that librarians from UofT play
a valuable role in supporting and assisting the learning
of Ethiopian librarians and in return benefit from a wider
understanding and experience of knowledge translation skills.
Conclusions: We have now taught over 700 learners. These
learners include library staff, physicians, medical and nursing
students. Our next steps include the beginnings of our
distance or remote library search support, a review of the
use of the UofT Libraries’ e-resources, assignment based
scenarios for clinicians, and web-based instructional modules
to complement in country training sessions.
RO 14.3
Spread of evidence-based practice into
directors of nursing by an outreach campaign
in Taiwan
Weng Y1 , Kuo K2 , Chiu Y2 , Chen K2 , Chen C2
1
Chang Gung Memorial Hospital, Taiwan;
University, Taiwan
2
Taipei Medical
Background: Directors of nursing (DONs) have important
influence in the dissemination of evidence-based practice
(EBP) in the hospital settings. Nevertheless, diffusion of EBP
into DONs was not well surveyed. Objectives: A nationwide
program provided information resources and promotional
89
activities of EBP in regional hospitals of Taiwan since 2007.
The current study aimed to evaluate the impacts of this
campaign on DONs. Methods: A cross-sectional questionnaire
survey for DONs was conducted in 2007, 2009 and 2011 to
examine views related to EBP, including changes in beliefs,
attitudes, knowledge, skills, and barriers. Chi-square test
was used for statistical analyses (SPSS 19.0 for Windows,
SPSS, Chicago, IL, USA). Results: This study enrolled 267
DONs in 2007, 257 DONs in 2009, and 287 DONs in 2011.
During the four-year study period, their knowledge and skills
of EBP increased (P value < 0.001). However, their beliefs and
attitudes were not significantly changed. A majority of barriers
significantly declined, including no capable designated
personnel (P value < 0.001), deficient convenient application
kits (such as personal digital assistants and brochures; P
value < 0.001), insufficient time (P value < 0.001), inadequate
basic knowledge of EBP (P value < 0.001), lack of skill in
literature searching (P value = 0.043), and insufficient library
resources (P value < 0.001). Conclusions: A multifaceted
campaign can increase the knowledge and skills of DONs
and reduce their barriers toward EBP. Spreading EBP
simultaneously with a mass media campaign may serve as
a key element to accelerate its implementation. The data
suggest an outreach intervention is useful to disseminate the
implementation of EBP for DONs.
RO 14.4
Students 4 Best Evidence: ‘What’s
the evidence for this?’ campaign
Millward H1
1 Cochrane UK, United Kingdom
Introduction: Students 4 Best Evidence (S4BE) is an online
community for students interested in evidence-based health
care; it aims to help students learn more about evidence-based
practice and the methodological concepts underpinning it.
S4BE involves students from school age to university through
relevant, useful resources as well as being a space for them
to communicate their knowledge and interact with fellow
students. Aims: In April 2015, S4BE is launching a year-long
campaign to ask, ‘What’s the evidence for this?’ The aim of
the campaign is to ignite student champions globally, to fight
for evidence, to spread understanding on the use of evidence
and show how it can improve global health. We will do this
through a series of mini events and projects throughout the
year. We would like to invite Colloquium attendees to join
the campaign and share the ‘S4BE What’s the evidence for
this?’ campaign with their student networks. Results: We
want the campaign to result in improved understanding of
evidence and the concepts used in evidence-based practice.
We seek to strengthen the S4BE community with a new group
of student champions, willing to ask ‘What’s the evidence for
this?’ Conclusions: We would like to build on the success
of blogs like, ‘A beginner’s guide to interpreting odds ratios,
confidence intervals and P values’, that has been viewed over
100,000 times since publication in August 2013 and make
sure students know how to campaign successfully for best
evidence.
RO 14.5
‘Cochrane Lehrmodule’: development and
implementation of teaching modules in
university-affiliated courses
Mahlknecht P1
1
Cochrane Austria, Austria
Background: The Danube University Krems, where Cochrane
Austria is hosted, is offering a variety of courses in the field
of medicine and health sciences. Within these courses,
modules on evidence-based research skills and methods
have gained popularity. Cochrane Austria was requested
to develop teaching modules on evidence-based methods
and contents to meet their increasing demand. This is in
line with the Cochrane Strategy to 2020 and the Cochrane
training and professional development strategy, including the
establishment of university-affiliated courses, and providing
appropriate training opportunities. Objectives: To develop
teaching modules and implement them in university-affiliated
courses, transferring research skills and basic knowledge in
the area of evidence-based healthcare. All modules follow
a standard structure that can be tailored to the audience
of the various courses. The modules are also intended for
people without prior knowledge of evidence-based research
methods. Development and structure: In consultation with
the heads of departments and course directors, needs,
requirements, and previous knowledge of the students are
assessed in order to turn expectations into realistic and
effective learning goals. A challenge is to define the
level of knowledge and skills that is essential (compulsory
modules) or optional (elective modules) for the general
audience to achieve.
Depending on learning goals,
compulsory modules are delivered in blocks of five days
and include evidence-based medicine and healthcare, facts
about Cochrane, epidemiology, statistics, study designs,
critical appraisal of studies, including qualitative research
and guidelines, and literature searches. Elective modules
are implemented based on the background, interests and
needs of the respective audience. Prospects: The first block
of teaching modules will be launched end of April. The
purpose of this presentation is to highlight issues around the
implementation of the training, share teaching experiences
so far, and discuss potential modifications to the modules. It
will be of interest to a wide audience involved in training and
teaching evidence-based research methods.
90
RO 14.6
Online systematic review methods training
resources: environmental scan and
identification of key characteristics
Parker RM1 , Visintini S2 , Ritchie KC3 , Hayden JA1
1
Dalhousie University, Canada; 2 Maritimes SPOR Support Unit,
Canada; 3 IWK Health Centre, Canada
Background: There are at least 26 systematic reviews
published every day, answering the increased demand for
synthesis of health research produced around the world. To
produce quality research, the authors of these reviews needed
to learn the appropriate methods for conducting and reporting
their work. In addition to mentoring and in-person training,
reviewers can learn the required skills using instructional
tools and resources that are available online, including
e-books, video tutorials, web-based methods guides, and
webinars. Many of these resources have the advantage of
being asynchronously available; they can be accessed at the
point of need and can be selected for the specific experience
and requirements of the learner. Objectives: To compile a
database of online training resources for systematic review
methods, from which we will identify common and review
author-preferred elements to determine best practices in
format, content, and delivery. Methods: We will conduct an
environmental scan to locate online instructional supports
for systematic review authors by searching the grey literature
and online presences of English-language health research
organizations and academic centres. We will extract and
analyze data regarding the elements present in the resources
to identify common practices. For example, information
will be collected about the resource creator, purpose,
audience, learning objectives, pedagogical approach, format,
means of access, and content. These elements will be
presented to a focus group of prospective and current
systematic review authors for input on preferences regarding
delivery methods and pedagogical style. Results: The
results of the environmental scan will be presented and
the key characteristics of the training resources highlighted.
Conclusions: Publishers of systematic reviews, including
Cochrane, have a vested interest in prospective authors
acquiring the knowledge and skills to produce high-quality
systematic reviews. This research will compile the existing
resources available to learners and will identify elements that
support the learning objectives of systematic review authors.
RO 14.7
Building capacity to support
evidence-informed public health: an
innovative knowledge broker mentoring
program
Dobbins M1 , Ciliksa D2 , Yost J1
1 McMaster University, Canada; 2 The National Collaborating Centre
for Methods and Tools, Canada
Background: To achieve evidence-informed decision making
(EIDM) in public health requires both organizational support
and commitment alongside the development of knowledge,
skill and capacity among individual practitioners. One
promising strategy for EIDM is knowledge brokering.
Objectives: 1. Assist health departments in identifying
and developing organizational capacity for EIDM. 2. Build
individual capacity for EIDM among four to six public
health professionals at five health departments. Methods:
The senior management team of each participating health
department participated in a three-hour focus group to assess
organizational capacity for EIDM. During the assessment,
priorities for supporting EIDM at the organizational level
were identified and strategies were developed to address
these priorities. The senior management teams identified
six individuals to participate in a 16-month knowledge
broker mentoring program where participants receive intense
training in all steps in the EIDM process. The program consists
of face-to-face interactive workshops at McMaster University
(10 days in total), monthly webinars, and biweekly follow-up
teleconferences with an expert in EIDM. Individual knowledge
and skill is being evaluated at baseline and will be evaluated
again at completion of the program along with another
organizational assessment. Results: Initial feedback from
the first face-to-face workshop was overwhelmingly positive.
All organizations are developing strategies to integrate EIDM
into routine decision making. Each health department has
allocated resources to cover the costs of their staff to attend
the workshops, as well as the cost to deliver the workshops,
and has committed two hours per week per staff member
to continue developing their EIDM knowledge and skills
throughout the program. Conclusions: This program will
address an priority area in public health by assisting health
departments to develop and support evidence-informed
decision making throughout the organization.
RO 14.8
A program to evaluate Cochrane training for
authors of systematic reviews
Cumpston M1 , Page M2
1
Cochrane Central Executive Team, Australia; 2 Cochrane Australia,
United Kingdom
Background: Although Cochrane has conducted training
in systematic review methods since its inception, these
activities have undergone only superficial evaluation based
on attendance rates or participant satisfaction. The Cochrane
Central Executive is initiating a new program to test our
success in conveying the skills and knowledge required
to complete a Cochrane review more rigorously, and to
contribute to the body of research in the teaching of
evidence-based medicine and research methods. Objectives:
To identify a tool to assess skills and knowledge in
conducting systematic reviews, and evaluate the effectiveness
of Cochrane training in systematic review methods. Methods:
An assessment tool to assess the skills and knowledge
required to conduct systematic reviews will be identified,
91
and any necessary adaptation of the tool completed. As
it is likely that the tool will have been developed for a
different purpose (such as appraisal or reporting of systematic
reviews), the tool will be validated with a diverse sample
of authors to ensure construct validity and discrimination
between different levels of expertise. A standardised program
of evaluation will then be implemented across the Cochrane
Trainers’ Network. This will enable the tool to be tested across
a much larger sample, including participants in face-to-face
and online training, while providing access for the first time
to a data set assessing the skills and knowledge of trainee
Cochrane authors. Additional components of the evaluation
program will include a review of Cochrane Training materials
to ensure alignment with core skills and knowledge; an
assessment of first submitted draft manuscripts to assess
implementation of skills and knowledge in practice; and an
audit of equity of participation in training by geographically
and linguistically diverse authors. Results: Preliminary results
and detailed project plans for this evaluation program will be
presented. Conclusions: We anticipate that progress towards
a meaningful evaluation of the effectiveness of Cochrane
training activities will benefit both authors and trainers,
and ultimately improve the quality of Cochrane systematic
reviews.
Implementation and qualitative
synthesis
RO 15.1
How are realist reviews conducted? A
systematic mapping review of realist reviews
Berg R1 , Nanavati J2
1
Norwegian Knowledge Center for the Health Services, Norway;
2
Lakeside Upper School, USA
Background: A realist review is an interpretive theory-driven
narrative summary of findings from primary studies of
complex social interventions. Such a review provides an
explanatory analysis aimed at discerning what works for
whom, in what circumstances, in what respects, and how.
Objectives: To describe, with regard to realist review, how
much has been done, in what areas, and what are the
methodological characteristics of published realist reviews.
Methods: We used a systematic mapping review approach,
which captures and describes the literature in one specific
field of study. We conducted systematic searches in seven
international databases from January 2004 to January 2015,
examined relevant websites and listserves, and contacted
experts. Study selection was performed by two reviewers
independently and data extraction was performed by one
reviewer and checked by a second. For each included review,
details of the review characteristics and methodological
process were extracted (53 variables), and descriptive
analyses were performed. Results: We included 54 realist
reviews (73 publications), conducted primarily by British
researchers. Three-quarters of the titles identified the study
as a realist review or realist synthesis. About half (46%) of the
reviews explained the selection and appraisal of documents,
and a quarter (24%) explained the data extraction process.
Twenty-nine of the reviews (54%) used a data extraction form.
The processes of selecting documents, appraisal of evidence,
and data extraction were rarely done in duplicate (13% to
22%). About half (54%) provided a document flow diagram
and 39% provided information on the characteristics of the
documents included in the review. The range of documents
included was six to 276 (it was unclear in five). Conclusions: In
ten years, there has been an exponential growth of published
realist reviews from a broad spectrum of health and related
research areas. The reviews vary greatly in terms of purpose,
methodological rigor, and quality of reporting. There may
be a need for methodological standardization to ensure the
utility and strength of its evidence.
RO 15.2
Development of a checklist to assess
implementation (Ch-IMP) in systematic
reviews: the case of provider-based programs
targeting children and youth
Cargo M1 , Stankov I1 , Thomas J2 , Saini M3 , Rogers P4 , Mayo-Wilson
E5 , Hannes K6
1
University of South Australia, Australia; 2 UCL Institute of
Education, University College London, United Kingdom; 3 University
of Toronto, Canada; 4 RMIT University (Royal Melbourne Institute
of Technology), Australia; 5 Johns Hopkins Bloomberg School of
Public Health, USA; 6 KU Leuven, Belgium
Background: Information on implementation is required in
systematic effectiveness reviews to facilitate the translation
and uptake of evidence. To capture whether and how
implementation is assessed in reviews a checklist for
implementation (Ch-IMP) was developed and piloted in a
cohort of systematic reviews on provider-based prevention
and treatment interventions targeting children and youth.
Objectives: This presentation reports on the inter-rater
reliability and feasibility of the Ch-IMP and outlines reasons
for discrepant ratings. Methods: Checklist domains were
informed by a framework for program theory. Items in
four domains (i.e. environment, process evaluation, action
model, change model) were generated from a literature
review.
The checklist was pilot-tested on 27 reviews
targeting children and youths. Two raters independently
extracted information on 47 items, which included fidelity,
dose and reach. Inter-rater reliability was evaluated using
percentage agreement and kappa coefficients. Reasons for
discrepant ratings were content analysed. Results: Kappa
coefficients ranged from 0.37 to 1.00 and were not influenced
by one-sided bias. Most kappa values were classified as
excellent (n = 20) or good (n = 17) with a few items categorised
as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa
coefficients indicate good or excellent agreement for all but
92
Figure 1
one item. Four areas contributed to scoring discrepancies:
(1) clarity or sufficiency of information provided in the
review; (2) information missed in the review; (3) issues
encountered with the tool; and (4) issues encountered at
the review-level. Use of the tool demands a time investment.
As such, adjustment is required to improve its feasibility
for wider use. Conclusions: Results suggest that the
Ch-IMP is a promising checklist for assessing whether reviews
of provider-based programs targeting children and youth
consider the impact of implementation variables. Used by
authors and editors, the checklist may improve the quality of
systematic reviews. Furthermore, the checklist shows promise
as a pedagogical tool to facilitate the extraction and reporting
of implementation characteristics.
RO 15.3
Mixed studies reviews: types of synthesis
designs used to combine qualitative and
quantitative evidence
Hong QN1 , Pluye P1 , Wassef M2 , Bujold M1
1 McGill University, Canada; 2 Institut national de santé publique du
Québec (INSPQ), Canada
Background:
Reviewing qualitative and quantitative
evidence has been advocated to provide in-depth answers
to review questions involving complex phenomena. This
type of review is emerging, and several synthesis designs
have been proposed. Objective: Using a mixed methods
research framework, we aim to identify and describe the
different types of synthesis designs applied in systematic
mixed studies reviews (SMSR). Methods: SMSR were searched
for in six databases (MEDLINE, PsycINFO, Embase, CINAHL,
AMED, and Web of Science) from inception through December
2014. Academic journal papers were selected for inclusion
if they were systematic reviews combining qualitative,
quantitative and/or mixed methods studies. Results: A
total of 7006 records were found, 470 of which met our
eligibility criteria. The number of published systematic
reviews combining qualitative and quantitative evidence has
increased considerably over the past decade; passing from
less than 15 in 2006 to more than 100 in 2014 (Figure 1).
Three main types of synthesis designs were identified. A
first type is convergent synthesis designs, in which results of
included studies are transformed into either qualitative (e.g.
themes) or quantitative (e.g. statistics) results. A second
type is sequential synthesis designs, in which two synthesis
methods are used and the results of both syntheses are
compared and integrated: a quantitative synthesis informs
the qualitative synthesis (sequential exploratory synthesis
design) or a qualitative synthesis informs the quantitative
synthesis (sequential explanatory synthesis design). A third
design, named parallel, consists in two independent syntheses
of qualitative and quantitative data without integration of the
results of both syntheses. Conclusion: Using the mixed
methods research framework as a starting point, this review
describes the main types of synthesis designs usually applied
in SMSR for combining qualitative and quantitative evidence.
Results show that these reviews are becoming popular and
warrant the need for additional methodological development.
RO 15.4
Adults with kidney disease and dietary
recommendations: integration of
quantitative and qualitative evidence to
involve patient perspectives
Palmer S1 , Soroka K1 , Hanson C2 , Craig J1 , Ruospo M3 , Campbell
K4 , Strippoli G1 , Tong A2
1
Cochrane Renal Group, Australia; 2 Centre for Kidney Research,
Australia; 3 University of Eastern Piedmont, Italy; 4 University of
Queensland, Australia
Background: Patients rank understanding the effectiveness
of lifestyle programs, including diet, as their top treatment
uncertainty in the treatment of kidney disease. However,
93
existing research about the effects of specific dietary
recommendations in kidney disease has not been summarized
and does not account for patient experiences. Objectives:
To summarize effects of dietary advice among adults
with kidney disease together with patient experiences to
generate recommendations for research. Methods: We
conducted a systematic review of randomized controlled
trials (RCTs) of any dietary intervention in chronic kidney
disease and meta-analysis of treatment effects. Concurrently
we did a thematic synthesis of qualitative studies of patient
perceptions and experiences of dietary management. The
two reviews were combined in a cross-study synthesis
to identify potential dietary interventions informed by
patient perspectives. Results: Seventy-eight RCTs (5945
patients) with kidney disease evaluated dietary interventions
including protein intake and type, olive oil supplements, and
counselling. Outcome data were sparse and effects of dietary
approaches on survival and quality of life were very uncertain.
From 46 qualitative studies on the experiences of dietary
restrictions involving 816 patients, we identified five themes.
Patients reported that dietary interventions interfered with
relationships, required navigation, led to constant fighting
against temptation, allowed them the opportunity to
optimize their health, and increased patient empowerment
in healthcare. Cross-study synthesis showed that dietary
recommendations are experienced intensely by patients,
but issues such as social acceptance, self-efficacy, and
navigating change are not considered within existing trials.
Conclusions: Our review identified a sparse evidence-base for
dietary recommendations among people with kidney disease,
which is grossly mismatched to the intense burden and
importance of dietary modifications experienced by patients.
Interventions that address a patient-centered research
agenda including dietary implementation, social disruption
and treatment effectiveness are needed. effectiveness are
needed.
RO 15.5
Applying framework synthesis to understand
complexity in systematic reviews: a
methodological systematic review
Brunton G1 , Oliver S1 , Thomas J1
1
EPPI-Centre, UCL IOE London, United Kingdom
Background: The amount of data extracted from primary
studies during a complex systematic review can create
challenges for synthesis, interpretation, and subsequent
knowledge exchange. ‘Black box’ reviews that minimise
data do not cope easily with, nor address adequately,
heterogeneous interventions and populations. Systematic
reviews increasingly employ framework synthesis to address
these issues. However, some differences exist in how
methods, outputs and underlying ways of knowing are
described. It is thus timely to examine the evolving use of
framework synthesis in systematic reviewing. Objectives:
This systematic review aimed to address the following
research questions: How do methods of framework synthesis
compare across reviews? Where is framework synthesis
located within a range of research synthesis methods (e.g.
meta-ethnography, meta-analysis)? What problems are
addressed specifically by framework synthesis?
Methods: Reference searches were undertaken via ASSIA,
PsycInfo, PubMed, and Web of Science, key contacts and
reference lists. Included reports described and/or reflected
on the use of framework synthesis in systematic reviews. Data
addressing the research questions were extracted. Report
characteristics were compared and contrasted using thematic
analysis. Results: We included 15 out of 144 citations located.
Seven reports reflected on the applied use of framework
synthesis; four described specific details to apply the method;
and four placed it within a range of other synthesis methods.
The methods of theory selection and development varied, and
a deductive but iterative approach was used most often across
studies. In terms of epistemological position, most reports
placed the method between critical and scientific realism.
Conclusions: Use of an a priori framework, iterative coding
and thematic development indicate that framework synthesis
is deductive; but its practical use in engaging stakeholders
places the method between critical and scientific realism.
Methods of selecting the a priori framework and use as a
stakeholder engagement decision tool are developing; these
could fill gaps not yet addressed by other methods.
RO 15.6
Complex interventions and evidence
translation to specific contexts: how
additional analytic approaches can meet the
needs of decision-makers
Sutcliffe K1 , Richardson M1 , Rees R1 , Thomas J1
1
EPPI Centre, Institute of Education, University College London,
United Kingdom
Background: Identifying the ‘active ingredients’ of complex
interventions is challenging but essential for practical
implementation of review findings; some intervention
approaches may be appropriate and effective in some
contexts, but fail in others. For example, Weight Management
Programmes (WMPs) are complex multicomponent
interventions, and whilst we know that, overall, WMPs
incorporating both diet and exercise components are more
effective than those addressing diet or exercise alone,
attempts to explain the substantial residual variance in
outcomes have been unsuccessful. This equivocal picture
therefore cannot help decision-makers determine the detail of
which interventions to favour in which situations. Objectives:
In a review for policy makers and WMP commissioners in
the UK we sought to reveal sufficiently fine-grained evidence
about the characteristics of successful WMPs to enable this
evidence to be used in decision-making. Methods: Firstly, in
addition to identifying intervention characteristics from trial
reports, we conducted a synthesis of qualitative research to
identify WMP characteristics based on users’ and providers’
94
experiences. We then coded descriptions of evaluated WMPs
according to how they matched the views of WMP users and
providers. Secondly, to quantify the extent to which these
WMP characteristics were associated with trial outcomes, we
undertook a qualitative comparative analysis (QCA). QCA can
handle situations where there are relatively small numbers
of studies and multiple pathways to success better than
meta-regression. Thirdly, to enhance the utility of the review,
we consulted with a small number of local authorities and
service commissioners on the nature and range of WMPs
they currently provide and the appropriate presentation of
findings. Conclusion: The translation of research knowledge
into policy and practice is essential if systematic reviews are
to be more than an academic exercise. New approaches for
translation work may be necessary for examining complex
social interventions.
of the study in both sample size calculation and statistical
analysis. Information on general quality criteria such as cluster
consent, trial registration or publication of study protocol was
also reported more rigorously in those studies. Conclusions:
We found that two-thirds of c-RCTs dealing with complex
interventions in a primary care setting were unable to provide
evidence of improvements in patient care. This has to be kept
in mind when designing future studies aimed at improving the
current health care system.
RO 15.8
Challenges conducting a Cochrane Review of
public health ‘implementation strategies’
Williams C1 , Wolfenden L2
Hunter Medical Research Institute, The University of Newcastle,
Australia; 2 The University of Newcastle, Australia
1
RO 15.7
Is effectiveness of complex interventions in
primary care in cluster-randomized trials
linked to methodological rigor? A systematic
review
Pregartner G1 , Siebenhofer A2 , Erckenbrecht S3 , Berghold A1
1 Institute for Medical Informatics, Statistics and Documentation,
Medical University of Graz, Austria; 2 Institute of General Practice
and Health Services Research, Medical University of Graz, Austria &
Institute of General Practice, Goethe University Frankfurt, Germany;
3
AQUA-Institute for Applied Quality Improvement and Research in
Health Care, Goettingen, Germany
Background: In primary care, most interventions aimed
at improving patient care involve those providing it and
are complex in nature. Cluster-randomized controlled trials
(c-RCTs) are common because they take into account the
special structure of these studies. However, they also require
a lot of effort to design and conduct. Objectives: The
aim of this study was thus to assess how often c-RCTs
dealing with complex interventions were able to show
significant improvements over routine care. Furthermore,
we investigated potential quality differences between studies
showing an effect and those that did not. Methods:
We searched MEDLINE and the Cochrane Database of
Systematic Reviews for c-RCTs published in eight journals
(BMJ, British Journal of General Practice, Family Practice,
Preventive Medicine, Annals of Internal Medicine, Journal
of General Internal Medicine, Paediatrics, Canadian Medical
Journal) from 1946 to April 2014. We considered c-RCTs of
complex interventions in general practices that had at least
one-year follow-up and included a patient-relevant primary
outcome. For each study, we assessed the effectiveness
of its intervention and the reporting of 18 quality criteria.
Results: We found 21 papers that fulfilled our inclusion
criteria, seven of which (33%) showed an intervention effect
for at least one primary outcome; four showed an effect
for some but not all primary outcomes; and three for all.
The latter studies all accounted for the clustered structure
Background: Targeting health risk behaviours in children
is an important aspect of chronic disease prevention, as
heath behaviours established in childhood are likely to
track into adulthood. Schools are an attractive setting
for the implementation of child-focused chronic disease
prevention interventions. However, many recommended
prevention interventions are not adopted or implemented
by schools, despite government mandates to do so.
Studying the effectiveness of strategies designed to improve
implementation, and why these strategies succeed or fail,
provides critical information for future improved adoption
and use of recommended practices. We are conducting
two systematic reviews to assess the effectiveness of
implementation strategies for school-based chronic disease
prevention policies, programs or practices (interventions).
Objectives: To describe and discuss conceptual and practical
challenges encountered in conducting an implementation
review for public health. Methods: Data concerning the
conceptual and practical difficulties encountered in the
conduct of the reviews has been systematically captured
and will be presented as a case study. Results: The reviews
are currently underway. The key challenge encountered
has been distinguishing ‘intervention strategies to support
implementation’ from the ‘intervention’ to be implemented.
The issue is exemplified by whether health behaviour
interventions, which do not primarily aim to influence
implementation constitute implementation. Undoubtedly,
the implementation strategy and the ‘intervention’ should
be considered when interpreting our findings as they both
have an effect on outcome. However, the naming function for
the ’intervention’ in the Cochrane framework potentially adds
further complexity to this situation. Conclusions: An extensive
account of these conceptual and practical challenges will be
presented at the 2015 Cochrane Colloquium.
95
RO 15.9
Making sense of complex interventions:
evidence synthesis through harvest plots and
post-review expert consultations
Burns J1 , Polus S1 , Brereton L2 , Chilcott J2 , Ward S2 , Rehfuess E1
1
University of Munich, Germany; 2 ScHARR, University of Sheffield,
United Kingdom
Background: Systematic reviews of complex interventions
frequently collect and synthesize clinically and
methodologically heterogeneous evidence. In such cases,
authors often deem meta-analysis inappropriate and instead
perform a narrative synthesis. Other forms of synthesis in
such instances are currently understudied. Objectives: To
employ harvest plots to assess the effectiveness of a complex
palliative care intervention, and to use gap analysis and
post-review expert stakeholder consultations to make sense
of the heterogeneous evidence base. Methods: We updated
a review of home-based palliative care services, originally by
Gomes et al 2013. Due to expected heterogeneity, we decided
a priori to forego meta-analysis. We synthesized evidence
using harvest plots, allowing us to visualize trends in overall
effectiveness, and among subgroups. Interventions focusing
on informal caregivers (which we refer to as reinforced
home-based services), were of particular interest. In a
subsequent gap analysis, we identified issues either not
addressed through, or arising as a result of the review.
With the goal of exploring and explaining the results from
the effectiveness assessment further, we discussed these
gaps with experts in designing, implementing and evaluating
palliative care services (n = 8). Results: We included 10
new studies (total n = 30). Harvest plots showed a mix
of mostly positive and neutral effects across patient and
caregiver outcomes, including pain, symptom control, quality
of life, psychological health, death at home, hospitalization,
coping, and satisfaction with care. Reinforced home-based
models, however, showed little positive effect for caregivers.
Research gaps discussed with experts include potential active
components, timing of palliative care, and possible reasons
for the ineffectiveness of reinforced services. Conclusions:
The use of harvest plots facilitated the synthesis and clear
presentation of a very heterogeneous evidence base. The
subsequent gap analysis and stakeholder consultations with
palliative care experts allowed further exploration of the
evidence, leading to more meaningful and usable conclusions
for decision-makers.
Rapid reviews
RO 16.1
‘Rapid review’ methodology to inform a
policy decision: when is it fit for purpose? A
case study
Llewellyn A1 , Simmonds M1 , Brunton G2 , Thomas J2 , Sowden A1
1
Centre for Reviews and Dissemination, University of York, United
Kingdom; 2 EPPI-Centre, Social Science Research Unit, UCL Institute
of Education, United Kingdom
Background: There appears to be an increasing use of
‘rapid reviews’ to inform policy decisions in a timely manner.
However, there is relatively little consensus on rapid review
methodology. Objectives: To present as a case study our
experience of conducting a ‘rapid review’ to inform a policy
decision, and to discuss the strengths and limitations of our
approach. Methods: A systematic review of the impact
of antiretroviral therapy (ART) on liver disease progression
to inform policy decisions relating to patients co-infected
with HIV and chronic hepatitis C was commissioned by the
Department of Health in England. The short timeline (10
weeks) for completion of the review meant that a priori
decisions were made to restrict: (1) the number of sources
searched; (2) duplication in study selection; (3) patient and
public consultation. Results: We searched MEDLINE and
EMBASE and identified 1748 potentially relevant studies.
Thirteen observational studies were included in the review,
of which six were combined in a meta-analysis. We found
that lack of randomised controlled trial evidence is more
likely to have a greater impact on the conclusions drawn
than weaknesses in review methods. We will discuss specific
issues encountered during the review, notably involvement
of different stakeholders, ‘trust’ in rapid review findings, and
willingness of decision makers to potentially compromise
methodological rigour. Conclusions: Our review provided a
concise and transparent picture of the evidence available to
inform future policy decisions related to patients co-infected
with HIV and chronic hepatitis C. It was produced within a
short time frame and omitted elements of the systematic
review process such as comprehensive searching and double
screening, and was therefore potentially at higher risk of bias
and error than reviews adopting ‘gold standard’ methods.
Implications of the review methodology used, the nature
of the evidence synthesised and the policy context will be
discussed.
96
RO 16.2
Streamlining systematic reviews to provide
healthcare decision-makers with more timely
evidence: the DERP experience
Peterson K1 , Selph S1 , Holmes R1 , Feltner C2 , Jonas D2 ,
Holzhammer B1 , McDonagh M1
1 Pacific Northwest Evidence-Based Practice Center,
USA;
2
Department of Medicine, University of North Carolina-Chapel Hill,
USA
Background: High-quality systematic reviews are the gold
standard of evidence synthesis, but can be time-prohibitive
for healthcare decision-makers faced with shorter-term
information needs. In an effort to better provide Drug
Effectiveness Review Project (DERP) decision-makers with
the right information at the right time, in 2012 the Pacific
Northwest Evidence-based Practice Center (PNW EPC) started
streamlining their standard systematic review processes,
reducing timelines for new reviews by four months and by
three months for updates. The PNW EPC also created two
new abbreviated products, single drug reviews and reviews
of reviews. Objective: Describe the DERP streamlining
process and its impact on quality of reports produced
between July 2012 and October 2015. Methods: For
each streamlined DERP report, two authors completed
a questionnaire that assessed: (1) types of scope and
methodological modifications; (2) impact on report quality
indicators and conclusions; (3) workload manageability; and
(4) suggestions for improvement. We used a mixture of
question formats, including yes/no, multiple choice, Likert
scale ratings and open-ended responses. We will use
descriptive statistics and narrative methods to summarize
author responses. Results: We assessed six new and
four updated DERP reports. Streamlining included scope
restrictions, abbreviated data abstraction, less dual review
of evidence strength ratings, and less meta-analysis. For
report quality, authors indicated no change to key finding
clarity, organization, applicability, or risk of bias, but reported
slightly worse limitations and workload manageability. Due
to removal of indirect and noncomparative data, streamlined
updates resulted in more evidence gaps. Our findings are
potentially limited by recall bias due to retrospective data
collection, and by being specific to drug-drug comparisons.
Conclusions: In order to provide health care decision-makers
with more timely evidence, it is possible to streamline the
standard systematic review process with limited impact on
author-assessed report quality. Further assessment of report
quality is needed from external reviewers and users.
RO 16.3
Rapid reviews to inform state health policy
decisions
King V1 , Gerrity M1
1
OHSU, USA
Background: Medicaid, a US federal-state partnership,
provides care to poor and disabled children and adults.
Seventeen state Medicaid programs collaborate, as the
Medicaid Evidence-based Decisions (MED) project, with
the Center for Evidence-based Policy (CEBP) to produce
rapid reviews (RR) to inform health policy decisions.
Objectives: To describe the methods used to produce
RRs for US state policymakers. Methods: Review of
CEBP RR methods, their evolution, and current outputs.
Results: Between 2006 and 2015 the CEBP produced
over 300 RRs. Initial MED evidence reports (2006-2008)
were brief evidence summaries of SRs from trusted core
sources such as the Cochrane Library. Reports sometimes
included searches to update SRs, but critical appraisal
(CA) of studies was not routine. In 2008, participating
policymakers requested more complete evidence reviews,
and wanted the option to add clinical practice guidelines,
with CA of the guidelines, and payer coverage policies.
Since 2009, CEBP has routinely incorporated the following
methodology elements into RR production: refinement of
topic scope and protocol with nominating state; multiple
database search for SRs and update searches using
PubMed; single reviewer study selection/abstraction/CA, with
second reviewer validation; narrative synthesis; internal
review; and proprietary publication. Optional elements,
incorporated based on the topic and needs of the nominator,
include: peer-reviewed search strategy; search of additional
databases; industry dossiers; guidelines and payer policies;
dual review of selection/abstraction/CA; external review and
public dissemination. Production timeline is generally three
to four months from protocol agreement. Conclusions:
Medicaid programs have successfully collaborated with the
CEBP over the last decade to develop RRs to inform policy
decisions. Evidence reviews alone seldom provide sufficient
information for good policy formation. However, critical
analyses of evidence and appraisal of the quality and
concordance of guidelines and policies with that evidence can
assist policymakers in making evidence-informed decisions
within relevant timeframes and available resources.
RO 16.4
If a rapid review is the solution, what is the
problem?
Mann M1 , Naughton A2 , Maguire S1 , Kemp A1 , Cowley L1
1
Cardiff University, United Kingdom; 2 Public Health Wales, United
Kingdom
Background: Rapid reviews have emerged as an efficient
approach to synthesizing evidence. There is a growing
number of rapid review ’methods’ nevertheless; there is
not an accepted standardised methodology. Consequently,
we used systematic review methodology to conduct a rapid
review to identify evidence behind the self-reported features
in adolescents who are experiencing neglect and/or emotional
maltreatment. Objectives: To identify the methodological
challenges related to the conducting a rapid review using
streamlined systematic review methods. Methods: We used
a focused search for all types of studies published in the UK
97
from 1990 to 2014. Due to the topic area, we searched a wide
range of databases relating to health, medicine, education
and psychology. Two reviewers independently screened the
papers to determine relevance according to eligibility criteria,
followed by full data extraction and critical appraisal by one
reviewer, with independent verification by a second reviewer.
Results: The search retrieved 4388 studies and 2568 records
were screened after manual de-duplication and removing
clearly irrelevant studies. Two reviewers independently
evaluated 279 studies for quality and eligibility; subsequently
19 papers were included in the review. Conclusions: Drawing
upon lessons learned from our review, we identify challenges
encountered at each stage of the review process and discuss
the solutions identified.
RO 16.5
Rapid review on the effectiveness of personal
protective equipment for healthcare workers
caring for patients with filovirus disease
Hersi M1 , Stevens A2 , Quach P1 , Hamel C1 , Thavorn K3 , Garritty
C2 , Skidmore B4 , Vallenas C5 , Norris SL5 , Egger M6 , Eremin S5 ,
Ferri M7 , Shindo N5 , Moher D3
1
Ottawa Hospital Research Institute, Canada; 2 Ottawa Hospital
Research Institute, Canada; 3 Ottawa Hospital Research Institute,
University of Ottawa, Canada; 4 Independent Consultant, Canada;
5
World Health Organization, Switzerland; 6 University of Bern,
University of Cape Town, Switzerland/South Africa; 7 University
of Calgary, Canada
Background: The West African outbreak of the Ebola virus
disease in 2014 evolved rapidly, yielding the highest number
of cases and deaths of outbreaks to date. We performed a
rapid review of the evidence to inform recommendations
issued by the World Health Organization on the use of
personal protective equipment (PPE) by healthcare workers
managing patients with known or suspected filovirus (Ebola
and Marburg) disease. Objectives: To determine the
comparative benefits and harms of various PPE (e.g. double
gloves, full face protection) components. Methods: A
rapid review (accelerated and/or modified systematic review
methods) guided by a protocol was conducted over seven
weeks. Bibliographic databases, grey literature sources,
and supplemental sources were searched. Eligibility criteria
initially included only comparative studies on Ebola and
Marburg diseases reported in English or French, but criteria
were expanded to studies on other, viral hemorrhagic fevers
and to non-comparative designs because comparative studies
were lacking. Titles and abstracts were reviewed by one
person, and a second person verified potentially excluded
records. Full-text articles were reviewed by two independent
people. Meta-analysis of data was not done, but plots
summarizing data were produced where appropriate. The
domains of the Grading of Recommendations Assessment
Development and Evaluation framework (GRADE) were used
to inform judgments on the quality of the evidence. Results:
No comparative studies were located. Thirty non-comparative
(eight related to Ebola) studies were included; 27 studies
provided data on viral transmission, while nine studies
reported on other adverse events, such as needle-stick
injuries. The quality of the body of evidence for all outcomes
was low. In general, studies reported information on PPE
components and infection prevention and control protocols
poorly. Conclusions: Insufficient evidence exists to draw
conclusions on the comparative effectiveness of various
types of PPE. Additional research is urgently needed, and
considerations exist for future research.
RO 16.6
Rapid reviews to inform Ebola preparedness
efforts at hospitals
Mitchell M1 , Price D1 , Umscheid C1 , Mull N1
1
University of Pennsylvania Health System, USA
Background: US hospitals are preparing to evaluate and
treat returning travelers at risk for Ebola Virus Disease (EVD).
Our hospital is one of 35 designated by the Centers for
Disease Control and Prevention (CDC) as an Ebola treatment
center. Objectives: Acquire, appraise, and apply best
available evidence to develop protocols rapidly for the
treatment of patients with possible EVD. Methods: The focus
and scope of our reports were determined in an iterative
process between the research analyst, the clinicians, and
the center director. Rapid review products from the center
include evidence inventory reports that describe the type and
quantity of evidence available in a specified area, evidence
advisories based on secondary sources, and evidence reviews
analyzing primary studies. Results: The initial clinical
query sought evidence on the effectiveness of rehydration,
anti-emetic, and anti-diarrheal treatments in patients with
EVD. An advisory was completed in 14 days, finding one
guideline for caregivers in rural Gabon, but no studies in EVD
patients. Given the limited direct evidence, the clinicians
then requested a rapid review of oral rehydration for the
broader population of patients with any acute infectious
gastrointestinal diseases. That review identified a strong
consensus of guidelines favoring reduced-osmolarity oral
rehydration solutions. Based on those guidelines, the scope
of review of primary literature was narrowed to studies of
adults, which helped us complete the evidence tables and
provide preliminary findings 14 days after the report request,
and finalize the review and incorporate into a clinical protocol
six business days later. The quality of the evidence base
was low. Conclusions: Close consultation between local
evidence-based practice centers and clinical decision-makers
can expedite evidence reviews to respond to urgent needs.
RO 16.7
Possibilities of standardisation for rapid
reviews –proposal for the German Federal
Ministry for Education and Research
Buchberger B1 , Kossmann B1 , Krabbe L1 , Mattivi JT1
1 University of Duisburg-Essen,
Institute for Health Care
Management and Research, Germany
98
Background: Through comparison of systematic reviews
and rapid reviews for surgical interventions using AMSTAR,
an instrument for quality assessment, we identified
ambiguity concerning the term ‘rapid’ as well as a need
for standardisation of methods for rapid reviews. Due
to the urgency of some questions in health care, an
acceleration of the review process is a high priority, and
therefore, standardisation will be very helpful. There
are many possibilities for methodological restrictions, but
weak empirical evidence regarding the consequences such
as increased potential of bias. Objectives: Our aims
are to develop and discuss a standardised rapid review
methodology, to conduct a rapid review based on that
methodology and a systematic review at the same time and
concerning an identical research question, and to compare
and discuss the results further. Methods: The research
project will be divided into three parts: (A) Comprehensive
systematic literature searches and handsearches for existing
standards for rapid reviews will be conducted on websites of
Health Technology Assessment agencies, and governmental
and non-governmental organisations. Based on a synopsis
of the results, we will interview methodology experts and
organise focus groups for discussion. The results will lead to
the development of a standardised format for rapid reviews
that will be revised by experts. (B) The developed format
will be tested by conducting a rapid review and a systematic
review at the same time on an identical question. (C) The
results, assessments of included evidence, conclusions, and
recommendations of both reviews will be compared and
discussed in focus groups consisting of decision makers
in health care. Conclusions: Comparing the results of a
rapid review conducted according to a newly developed
methodology with a systematic review on the same topic
and in parallel design will lead to conclusions about possible
methodological restrictions, their potential of bias and the
time, as well as resources, saved by accelerating the review
process. A standardised methodology for rapid reviews will
help policy-makers with time pressures to decide swiftly with
greater certainty.
Network meta-analysis
RO 17.1
Characteristics of network meta-analysis
reviews from Cochrane
Porfirio G1 , Nogueira G1 , Cabrera A1 , Parra M1 , Freitas C1 , Torloni
MR1 , Riera R1 , Atallah A1
1
Brazilian Cochrane Centre, Brazil
Background: Network meta-analysis is a study design that
allows indirect comparisons between interventions when
there are no studies with direct comparisons. Although
there is no standard method, crucial points, as the level of
similarity within studies (called transitivity or similarity) and,
the analysis and results (called consistency or coherence)
became well known. Considering the recent growth and the
lack of well-established standards for network meta-analysis,
a question arises: What are the characteristics of network
meta-analysis reviews from Cochrane? Objectives: To
describe the characteristics of network meta-analysis reviews
published by Cochrane. Methods: We performed a descriptive
study at the Brazilian Cochrane Centre (BCC). We included
Cochrane Systematic Reviews using the expression ’network
meta-analysis’ in any part of the text. We selected the
studies after full readings. The extracted outcomes were:
review characteristics, related review group, year and stage
of publication. Results: The search was conducted in March
2015. We identified 104 studies and excluded 71 studies after
full text reading. From the 33 remaining studies, 16 were
reviews, 12 protocols and 5 overviews distributed in 21 review
groups; 48% of studies were published in 2014 (16/23); 36%
mentioned ’network meta-analysis’ in the title. Considering
only finished studies (reviews and overviews), 57% (12/21)
mentioned having performed transitivity assessment and 57%
mentioned that that assessed inconsistency, although they
were not exclusively the same studies. Graphs and figures
representing the network meta-analysis were found in only
three reviews. Eight reviews mentioned the use of STATA
and seven mentioned WinBUGS as the software for analysis.
Conclusions: We believe that there is a need to define
description standards for networks meta-analysis within
Cochrane, with emphasis on transitivity and inconsistency.
Standardizing graphs and figures would also be beneficial.
Those were present in a minority of studies.
RO 17.2
Comparing various approaches for network
meta-analysis regarding effect estimation
and evaluation of inconsistency
Bender R1 , Sturtz S1 , Sieben W1 , Kiefer C1
1
IQWiG, Germany
Background: Network meta-analysis is an important
extension of pairwise meta-analysis to compare more than
two interventions and to analyze indirect and direct evidence
simultaneously. It is becoming more and more popular
in systematic reviews and health technology assessment.
Although for pairwise meta-analysis the properties of the
different approaches are well examined, little is known for
the approaches in network meta-analysis. Objectives: To
compare methods that are currently available for estimating
and assessing inconsistency in network meta-analysis.
Methods: A simulation study was conducted to evaluate
complex networks with up to five interventions and different
patterns. We analyzed the impact of different network-sizes,
different amounts of inconsistency and heterogeneity on
MSE and coverage of established and new approaches.
This exceeds previous simulations studies, which have been
conducted by others. Methods: We found that, with a high
degree of inconsistency in the network, none of the evaluated
effect estimators produced reliable results. For a network
with no or just moderate inconsistency the Bayesian and the
frequentist estimator showed acceptable properties, whereas
99
the latter one showed slightly better results. We also found a
dependency on the amount of heterogeneity in the network.
Conclusions: Our results underline the need to assess
inconsistency in network meta-analyses reliably as available
measures for inconsistency may be misleading in many
situations. We therefore conclude that it is also important
to assess similarity (regarding population, intervention, etc.)
and heterogeneity to reduce inconsistency in the network
in advance. Nevertheless, effect estimators should be used
which are suitable in the case of moderate inconsistency.
RO 17.3
Statistical models for network meta-analysis:
an empirical comparison
Debray T1 , Schuit E2 , Efthimiou O3 , Jansen J4 , Ioannidis J5 , Moons
K1
Netherlands; 2 Stanfor University, USA; 3 University of Ioannina,
Greece; 4 Redwood Outcomes, USA; 5 Stanford University, USA
Background: Network meta-analysis (NMA) can be conducted
to compare the relative efficacy of multiple treatments. It
is currently unclear whether access to individual participant
data (IPD) is beneficial in NMA. Methods: We discuss four
key issues in a NMA and present several statistical methods
to address these issues. In particular, we describe methods
to overcome baseline imbalance (e.g. due to unlucky
randomization), to investigate between-study heterogeneity
in treatment effect, to investigate within-study heterogeneity
in treatment effect and to minimize network inconsistency.
For all methods, we highlight the role of IPD, and illustate
their implementation in an empirical example dataset.
Results: Results indicate that when NMA do not incorporate
patient- or study-level covariates, one-stage and two-stage
models lead to similar estimates of relative treatment effect.
However, two-stage models yielded slightly larger standard
errors and substantially larger estimates of between-study
heterogeneity. For all models, estimates of treatment effect
gained precision and network consistency improved when
adjusting for covariates. Most importantly, NMA that were
based on IPD showed greater power in investigating the
presence of effect modification, and helped to improve overall
network consistency. Conclusions: When investigating
the relative efficacy of multiple treatments, NMA that are
(partially) based on IPD may help to overcome several
key issues hampering the validity and interpretability of
traditional NMA analyses that are solely based on published
aggregate data.
RO 17.4
Exploring and accounting for the impact
of interventions with scarce evidence in
network meta-analysis
Chaimani A1 , Salanti G1
of Hygiene & Epidemiology, School of Medicine,
University of Ioannina, Greece
1 Department
Background: Previous empirical evidence has revealed that
interventions evaluated in very few trials are often
placed among the most effective interventions in a
network meta-analysis (NMA). Although this phenomenon
can be expected for recently marketed treatments,
older treatments with scarce evidence could show
exaggerated results due to publication bias or because they
have been compared only to suboptimal alternatives. If the
network is poorly connected, such treatments are informed
mainly by their potentially biased direct comparisons and
may rank spuriously high in the overall ranking. Objectives:
To explore the impact of interventions with scarce evidence
in a NMA and to estimate the relative ranking of treatments
as a function of the amount of available information in the
network. Methods: We developed a network meta-regression
model using the contribution of each direct comparison to
the network as a predictor of the relative effects. We assumed
that interventions poorly connected to the rest of the network
might be favored in a direct comparison. We applied the
model in a network that compares eight different stents for
myocardial infraction, in which results are highly affected
by the direct comparison between bare-mental stents (BMS)
and control. Results: Based on the cumulative ranking
probabilities, the NMA model indicated BMS as the best
intervention, although newer treatments are well-known to
be safer. After applying our meta-regression model BMS
was placed at a lower rank and results were closer to those
expected from clinical practice. Conclusions: In poorly
connected networks, the contribution of indirect information
might be low and some interventions may appear high in
ranking because they are informed primarily by their direct
comparisons. Modeling the amount of available evidence
for each intervention might be a useful tool to evaluate
the robustness of results from NMA. How results change if
suspicious comparisons are given a lower weight might also
be explored via a sensitivity analysis.
RO 17.5
The impact of publication bias and small
study effects on the efficacy and ranking of
antipsychotics
Mavridis D1 , Efthimou O2 , Mavridis S3 , Salanti G2
1
Department of Primary Education, University of Ioannina, Greece;
2
Department of Hygiene and Epidemiology, University of Ioannina,
Greece; 3 Department of Psychiatry and Psychotherapy, Technische
Universität München, Germany
Background: Evidence from placebo-control trials comparing
the least effective antipsychotics is distorted by small study
effects and publication bias (PB). This raises concerns about
the validity of the results from network meta-analysis (NMA).
Biased evidence in placebo-control comparisons may affect
the treatment effects and the ranking of the antispychotics
as placebo-controlled trials account for one-third of the
information in the entire network. Objectives: To assess
the potential impact of small study effects (SSE) and PB on
100
the recently estimated relative effectiveness and ranking of
pharmacological treatments for schizophrenia. To suggest
statistical methods for adjusting results in the presence of
SSE or PB. Methods: We used a recently published network
of 167 studies involving 36,871 patients and comparing the
effectiveness of 15 antipsychotics and placebo. We used novel
visual and statistical methods to explore whether smaller
studies are associated with larger treatment effects and if the
probability of publication is associated with the magnitude
of effect. We conducted a NMA of the published evidence
as our primary analysis and employed a sensitivity analysis
considering low, moderate and severe selection bias (that
correspond to the number of unpublished trials) with an
aim to evaluate robustness of point estimates and ranking.
We explored whether placebo-controlled and head-to-head
trials are associated with different levels of PB. Results:
We found that small placebo-controlled trials exaggerated
slightly the efficacy of antipsychotics and PB was not unlikely
in the evidence based on placebo-controlled trials; however,
ranking of antipsychotics remained robust. Conclusions: The
total evidence comprises many head-to-head trials that do
not appear to be prone to SSE or PB and indirect evidence
‘washes-out’ some of the bias in the placebo-controlled trials.
RO 17.6
Incorporating external information on
between-study heterogeneity in network
meta-analysis
Turner R1 , Rhodes K1 , Jackson D1 , White I1
1 Medical Research Council Biostatistics Unit, United Kingdom
Background: In a network meta-analysis comparing multiple
treatments, between-study heterogeneity variances are often
very imprecisely estimated because data are sparse, and
so standard errors can be highly unstable.
External
evidence obtained from modelling data from the Cochrane
Database of Systematic Reviews can provide informative
prior distributions for heterogeneity, tailored to particular
settings. Objectives: To explore and compare approaches
for specifying informative priors for multiple heterogeneity
variances in a network meta-analysis.
Methods: If
heterogeneity variances can be assumed to be equal across all
pairwise comparisons of treatments, it is straightforward to
construct an informative prior for the common between-study
variance. Models allowing heterogeneity variances to be
unequal are more realistic, however, care must be taken
to ensure that the implied variance-covariance matrices
remain valid. We consider two strategies for specifying
informative priors for multiple heterogeneity variances:
proportional relationships among the variances; or unequal
heterogeneity variances with a common informative prior.
Results: Appropriate prior distributions are obtained through
modelling empirical data from the Cochrane Database
of Systematic Reviews.
The models are applied to
a network meta-analysis comparing four treatments for
smoking cessation. Incorporating external information on
heterogeneity in the equal and unequal variance models leads
to smaller heterogeneity estimates, with narrower intervals.
This causes changes to the odds ratios, and their 95% intervals
narrow substantially. For example, the odds ratio comparing
group counselling against standard care is estimated as
3.39 (95% CI 0.98 to 16.8) when using vague priors, in the
unequal variances model. This changes to 3.10 (95% CI
1.60 to 6.17) when using informative priors for heterogeneity.
Conclusions: Relevant prior information on heterogeneity
can be incorporated into network meta-analyses, without
making unrealistic assumptions. This may improve precision
for estimating treatment differences.
RO 17.7
Prospective use of network meta-analysis
using formal statistical monitoring
Nikolakopoulou A1 , Mavridis D2 , Salanti G1
1 Department of Hygiene and Epidemiology, University of Ioannina,
School of Medicine, Ioannina, Greece; 2 Department of Primary
Education, University of Ioannina, Ioannina, Greece
Background: Pairwise and network meta-analyses (NMA) are
traditionally used retrospectively to assess existing evidence.
However, previous knowledge of trial results can introduce
bias due to potential selective inclusion of the key components
of the review question (PICO criteria).
Prospective
meta-analysis overcomes this limitation by requiring the
identification of eligible trials before the disclosure of their
results. It has been suggested that NMA can also be undertaken
prospectively; this practice though has not been adopted yet.
Objectives: Designing a NMA prospectively creates a multiple
testing scenario. Our objective is to develop a method for
the prospective design of NMA by using formal monitoring.
Methods: We extend ideas of sequential monitoring of trials
to random-effects NMA. We construct efficacy and futility
stopping boundaries for NMA estimates and we present
the sequential NMA procedure using repeated confidence
intervals (RCI). When a RCI excludes the point of no effect
the NMA effect is considered conclusive. Our method leads
to recommendations of whether further research is required
to inform NMA and for which comparisons. We illustrate the
method using a network that evaluates the effectiveness of
bare mental stent (BMS), coronary artery bypass (CAB) and
drug eluting stent (DES) for all-cause mortality in diabetic
patients. Studies are added in a chronological order to the
NMA model. Results: After the inclusion of eight studies
none of the NMA summary effects was conclusive. The
incorporation of the 9th and the 10th trial lead to a conclusive
NMA effect estimate for the comparisons ‘BMS versus DES’
and ‘CAB vs BMS’ respectively showing the relative inferiority
of BMS after accounting for multiple testing. If NMA had
been planned prospectively, it would have stopped with the
inclusion of the 13th study when NMA effects are conclusive
for all comparisons. Conclusions: Use of sequential methods
in NMA can be adopted so that trials do not address already
answered questions. Such a procedure would help to save
resources and prevent the allocation of participants of RCTs
101
to treatments proved to be inferior.
RO 17.8
Hierarchical network meta-analysis models
accounting for variability in nodes by
treatment, dosage-category and single
dosage
Veroniki AA1 , Tricco A1 , Del Giovane C2 , Blondal E1 , Thavorn K3 ,
Hutton B3 , Straus S1
1
Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto,
Canada; 2 Department of Diagnostic, Clinical and Public Health
Medicine, University of Modena and Reggio Emilia, Modena, Italy;
3 Department of Epidemiology and Community Medicine Faculty of
Medicine, University of Ottawa, Canada
Background: Healthcare providers, policy-makers and
patients are often interested in the effect of specific
treatment dosages or categories of dosage (e.g. low,
moderate, high), and the treatment-effects overall. Decisions
regarding network geometry (e.g. lump related dosages
together into one treatment node or maintain their
separation) may impact the network meta-analysis (NMA)
results, and influence decision-making. Objectives: To
present hierarchical NMA models accounting for effects
of treatments, dosage-categories, and single dosages,
providing additional insight on different heterogeneity
levels.
Methods: We developed three approaches
accounting for the relationship between treatment and
dosage, and including up to four heterogeneity levels. The
first approach assumes the dosage-effects equal to their
parent treatment-effects, and involves within-study and
between-study heterogeneity levels within dosage. The
second approach assumes exchangeable dosage-effects from
the same distribution with a common mean, and incorporates
an additional heterogeneity level, the between-dosage
within-treatment. The third approach assumes exchangeable
dosage-effects within a specific dosage-category with either
fixed or random mean dosage-category-effects across
treatments. The third model still considers the first two
heterogeneity levels, yet the between-dosage is evaluated
within dosage-category, and an extra heterogeneity level
is added, the between-dosage-category within treatment.
Consistency constraints on dosage-effects can also be placed
in each model. Results: Results from the application
of the different approaches to empirical examples, will
be ready by September 2015, and will be presented at
the Colloquium along with strengths and limitations of
the models. Conclusions: Different approaches regarding
the classification of treatments in a network may result
in important variations in interpretations drawn from
NMA. We suggest that researchers account for different
treatment dosages in NMA, providing additional insight on
heterogeneity, inconsistency, intervention ranking, and hence
decision-making.
Outcomes
RO 18.1
Planning and reporting of quality of life
outcomes in cancer trials: a retrospective
cohort study
Schandelmaier S1 , Amstutz A1 , Von Elm E2 , Conen K1 , Hemkens
LG1 , Bucher HC1 , Guyatt GH3 , Briel M1 , Kasenda B4
1
University Hospital, Basel, Switzerland; 2 University Hospital,
Lausanne, Switzerland; 3 McMaster University, Hamilton, Canada;
4
Royal Marsden Hospital London, United Kingdom
Background: Information about the impact of cancer
treatments on patients’ quality of life (QoL) is of paramount
importance to patients, treating physicians, and policy
makers.
Although cancer specific and validated QoL
instruments exist, reports of randomized clinical trials (RCTs)
involving cancer patients often do not include QoL outcomes;
instead, they typically focus on survival or tumor size as
their primary outcome. Possible reasons for lack of QoL
outcomes include failure to collect QoL outcomes and failure
to publish collected QoL outcome data. Objectives: To
investigate whether QoL outcomes were specified in protocols
of industry- or investigator-initiated RCTs enrolling cancer
patients, and whether they were subsequently reported.
Methods: We conducted a retrospective cohort study of
RCT protocols approved by six research ethics committees
in Switzerland, Germany, and Canada between 2000 and
2003 and compared 173 protocols of cancer trials to
108 corresponding publications that we identified through
literature searches and investigator surveys. Results: Of the
173 RCTs, 90 (52%) specified QoL outcomes in their protocol,
two (1%) as primary and 88 (51%) as secondary outcomes; 35
(20%) reported QoL outcomes in a corresponding publication
(four modified from the protocol), 37 (21%) remained
unpublished, and 18 (10%) were published but failed to report
QoL outcomes. Failure to report QoL outcomes was not
significantly associated with sponsorship, sample size, single
center status, or trial discontinuation. None of the 83 (48%)
RCTs that did not specify a QoL outcome in their protocol
subsequently reported a QoL outcome. Conclusions: About
half of cancer RCTs specified QoL outcomes in their protocols.
However, only 20% reported any QoL data in associated
publications. Highly relevant information for decision-making
is not available to patients, physicians, systematic reviewers,
and policy makers. If published, QoL outcomes in cancer trials
are prone to selective reporting.
102
RO 18.2
Missing the wood for the trees: clinically
relevant outcome measures in traumatic
brain injury rehabilitation, a review of
reviews
Viswanathan A1 , Samuelkamaleshkumar S2 , Senthilvelkumar T2
1
South Asian Cochrane Network and Centre, India; 2 Christian
Medical College, India
Background: The Cochrane Handbook suggests that ‘‘indirect
or surrogate outcome measures are potentially misleading
and should be avoided or interpreted with caution because
. . . many interventions reduce the risk for a surrogate outcome
but have no effect or have harmful effects on clinically
relevant outcomes, and some interventions have no effect
on surrogate measures but improve clinical outcomes’’. This
is very relevant to the field of neurorehabilitation, where
tangible clinical end points that matter to the patients, such
as functional independence, return to work and community
integration should be preferred over surrogate measurements
such as cognition, spasticity, and muscle power. Objectives:
1. To analyze proportions of clinically relevant outcome
measures versus surrogate outcomes reported in Cochrane
Systematic Reviews (CSRs) on rehabilitation interventions
following traumatic brain injury (TBI). 2. To analyze the
proportion of individual trials in these CSRs that report clinical
outcomes. Methods: The Cochrane Library was searched
for CSRs on rehabilitation interventions following traumatic
brain injury. Exclusion criteria were non-traumatic etiology,
and pharmacological interventions. Three authors analyzed
the search results. Outcomes were extracted and categorized
as clinically relevant or surrogate. Results: Thirty-seven CSRs
related to TBI were screened, five met the inclusion criteria: 18
out of 49 outcomes (37%) enlisted in five CSRs were clinically
relevant outcomes; the rest were surrogate outcomes. A total
of 17 trials were included in these reviews. Eight trials from
two CSRs (47%) had reported at least one clinically relevant
outcome. In the remaining three CSRs, no study reported any
clinically relevant outcome. Meta-analysis was not done in any
of the five CSRs. Conclusions: Less than half of the outcomes
reported in CSRs on TBI rehabilitation were clinically relevant.
More than half of the trials included in these reviews did not
report any clinically relevant outcome measure.
RO 18.3
Outcomes in Child Health: comparing
outcomes evaluated in Cochrane Reviews and
patient-important outcomes identified by
parents using social media
Hamm M1 , Fernandes R2 , Scott SD1 , Hartling L1
1
University of Alberta, Canada; 2 Cochrane Child Health Field;
Cochrane Portugal; Department of Pediatrics and Clinical
Pharmacology Unit, Lisbon Academic Medical Centre, Portugal
Background: Selection of appropriate outcomes for clinical
research is increasingly focused on patient-centeredness,
but outcomes reported in the literature do not necessarily
align with outcomes important to patients. Objectives: To
compare outcomes reported in Cochrane Reviews on pediatric
acute respiratory infections (ARI) with those identified by
parents as important to them and their children. Methods:
Parents of children who had an ARI were recruited via social
media to provide input on the outcomes that are most
important to them when their child is sick. We used the
registry of child-relevant Cochrane Reviews maintained by
the Cochrane Child Health Field to identify reviews applicable
to ARI and extracted data on their reported outcomes and
the frequency with which they appeared across reviews. This
set of outcomes was compiled into an online survey in which
parents were asked to rate each on a scale of 1–100 based
on how important they considered it to be. We also asked
parents to identify other important outcomes that were not
included in the list. We analyzed data descriptively using a
predefined outcome domain framework and compared the
prominence of outcomes evaluated in Cochrane Reviews with
those identified as being important by parents. Results:
We identified 35 reviews relevant to pediatric ARI. In the
reviews and the survey, safety was commonly endorsed, with
adverse events reported 26 times across reviews, and parents
ranking major side effects (mean score 86.7/100) among
their most important concerns. Parents identified severe
complications as their most important concern (94.5/100).
Resource utilization was more prominent in reviews, with
admission rates and length of stay/time to discharge ranking
in the highest quartile of outcomes measured, while parents
scored similar items in the 75th percentile. Social and family
outcomes (e.g. quality of life) were not often reported
in reviews, but made up the most frequently mentioned
outcomes when parents were asked about other concerns
that were not listed. Conclusions: Safety is a primary concern
to parents. Integrating patient priorities will ensure the
relevance of research results.
RO 18.4
A map of outcomes in trials of communication
interventions for childhood vaccination:
measuring too few concepts in too many ways
Kaufman J1 , Ryan R1 , Bosch-Capblanch X2 , Cartier Y3 , Cliff J4 ,
Glenton C5 , Lewin S6 , Rada G7 , Ames H5 , Muloliwa AM8 , Oku A9 ,
Oyo-Ita A9 , Hill S1
1 Centre for Health Communication and Participation, La
Trobe University, Australia; 2 Swiss Tropical and Public Health
Institute, University of Basel, Switzerland; 3 International Union
for Health Promotion and Education, France; 4 Universidade
Eduardo Mondlane, Mozambique; 5 Global Health Unit, Norwegian
Knowledge Centre for the Health Services, Norway; 6 Global
Health Unit, Norwegian Knowledge Centre for the Health Services,
Norway/Health Systems Research Unit, South African Medical
Research Council, South Africa; 7 Catholic University of Chile,
Chile; 8 Provincial Directorate of Health, Mozambique; 9 University
of Calabar, Nigeria
103
Background: Core outcome sets (COSs) bring consistency
to outcome measurement and strengthen systematic
reviews.
Most COSs focus on clinical interventions
for health conditions; few address preventive health
interventions.Communication for childhood vaccination
is key to improving uptake and can take many forms.
Measurement requires appropriate outcomes, but it is
not clear what these outcomes should be or how COS
methods apply to this context. Here we describe novel
methods for developing a Trial Outcomes Map (TOM),
a helpful tool for the early stages of COS development.
Objectives: To catalogue which outcomes have been
measured in childhood vaccination communication trials
and describe (in)consistencies and patterns in these with a
TOM. Methods: We searched the Cochrane Central Register of
Controlled Trials (CENTRAL) for relevant trials and extracted
outcome details. Outcomes were inductively coded into
thematic groups, forming the map. Consistencies, gaps
and trends were analysed. Results: We extracted data on
209 outcomes from 112 trials. We found variation and gaps
in reported outcomes.Thematic grouping identified three
overarching categories: vaccination-, consumer- and health
system-related outcomes. Vaccination-related outcomes
were reported most frequently: 80% of trials measured at least
one such outcome, although these outcomes were defined
and measured in a wide range of ways. Consumer-related
outcomes, tied mainly to knowledge, attitudes or beliefs,
were measured by 28% of trials. Health system-related
outcomes were measured in 20% of trials. Trials published
after 2000 appeared to include more consumer-related
outcomes, possibly reflecting increased interest in shared
decision making. Conclusions: This map highlights variation
in the measurement of key vaccination outcomes, leading to
difficulties in interpretation and synthesis. A trend towards
more consumer-related outcomes emerges, but intermediate
consumer- and health system-related outcomes that clarify
vaccination communication causal pathways are not
measured sufficiently frequently in trials. This map is a tool
for trialists and reviewers and will inform the next stages of
COS development.
RO 18.5
Development of a core outcome set for infant
colic
Steutel NF1 , Korterink JJ2 , Benninga MA2 , Langendam MW3 ,
Tabbers MM2
1
Dept Clinical Epidemiology, Biostatistics and Bioinformatics and
Dept Pediatric Gastroenterology and Nutrition, Academic Medical
Centre, The Netherlands; 2 Dept Pediatric Gastroenterology and
Nutrition, AcademicMedicalCentre, TheNetherlands; 3 DeptClinical
Epidemiology, Biostatistics and Bioinformatics, Academic Medical
Centre, The Netherlands
Background and
problem with a
This self-limiting
consequences.
objectives: Infant colic (IC) is a common
worldwide prevalence of 5% to 25%.
disorder can have negative long-term
So far, its etiology remains unknown,
resulting in a wide variety in interventions. A previous
study showed heterogeneous outcome measures are used
across therapeutic trials of IC. To facilitate and improve
evidence synthesis, development of a core outcome set
(COS) is necessary. Methods: In 2014, 133 health care
professionals (HCPs) participated in a survey that investigated
which outcome measures they used to guide their decision
making when treating IC. Every outcome that was mentioned
by 10% or more of the respondents was carried forward
to Phase 2. Of the respondents to Phase 1, 68 (51%)
agreed to complete a second questionnaire. HCPs were
then asked to rate the outcomes and to select the five most
relevant outcomes, for both an inpatient and outpatient
setting. Results (preliminary): To date 36 (53%) HCPs
have completed the second questionnaire. Duration of
crying, reduced family stress, sleeping time of the infant
and discomfort of the infant were considered to be most
important in both outpatient and inpatient setting. In the
outpatient setting quality of life was rated as important, and
in the inpatient setting reduced hospital admission (and/or
duration). Conclusions: These preliminary results show that
there is not much difference in outcome measures that guide
treatment decisions in an outpatient and inpatient setting for
IC. Furthermore, the results appear to be quite homogeneous
across respondents. In a later phase of the project parental
views will be taken into account to complete this COS. This
COS may enable researchers to standardize the outcomes they
measure when setting up a new clinical trial. This will enhance
homogeneity and may encourage consensus in the field of IC.
RO 18.6
Randomized clinical trials of the impact
of alternative diagnostic strategies on
patient-important outcomes: a systematic
survey
El Dib R1 , Tikinnen K2 , Akl E3 , Mustafa RA4 , Agarwal A5 , Gomaa
HA6 , Carpenter CR7 , Zhang Y8 , Nascimento Jr P9 , Jorge EC9 ,
Almeida RA9 , Doles JVP9 , Mustafa AA10 , Sadeghirad B8 , Lopes
LC11 , Bergamaschi CC11 , Suzumura EA12 , Cardoso MM9 , Stone
SB13 , Schünemann HJ14 , Guyatt GH14
1
Botucatu Medical School, Unesp, Universidade Estadual Paulista,
Brazil; 2 Helsinki University Central Hospital and University of
Helsinki, Finland; 3 American University of Beirut, Lebanon;
4
University of Missouri-Kansas City, Missouri, USA; 5 University
of Toronto, Canada; 6 Tanta University, Egypt; 7 Washington
University in St Louis, USA; 8 Department of Clinical Epidemiology
and Biostatistics, McMaster University, Hamilton, ON, Canada;
9 Botucatu Medical School - Unesp - Universidade Estadual Paulista,
Brazil; 10 Jordan University of Science and Technology, Erbid,
Jordan; 11 University of Sorocaba, Sao Paulo, Brazil; 12 Hospital
do Coração - HCor, São Paulo, Brazil; 13 Northern Ontario School
of Medicine, Canada; 14 Department of Clinical Epidemiology and
Biostatistics, McMaster
University, Canada
Background: Diagnostic tests represent a pivotal part of
patient management. Often clinicians adopt tests for clinical
104
105
use on the basis of findings of diagnostic accuracy. Although in
some instances simply knowing accuracy can allow inferences
classified as low risk of bias were: generation of allocation
(n = 66; 50.7%); allocation concealment (n = 44; 33.8%);
blinding (n = 28; 21.5%); missing outcome data (n = 74;
56.9%); selective reporting (n = 44; 33.8%) and; free of other
problems (n = 37; 28.4%). Of the 130 RCTs, 44 evaluated
mortality; two reported statistically significant results, but
neither provided an estimate of relative effect; 28 did not
report whether the results were statistically significant and
14 of them reported a non-statistically significant results.
Investigators reported the impact of morbidity in 75 RCTs: 16
reported statistically significant results, two of which reported
a risk ratio (RR) less than 0.8 and two reported a RR greater
than 1.0 –the remaining 14 did not report RRs; 32 did not
report whether the results were statistically significant and 27
reported a non-statistically significant results. Conclusions:
Randomized trials of diagnostic tests are not rare, seldom
show clear benefits on patient-important outcomes, and
often suffer from limitations in reporting and conduct.
RO 18.7
Social network analysis for identifying
central outcomes for clinical research: a case
study using Cochrane reviews of HIV/AIDS
Saldanha I1 , Li T1 , Yang C2 , Ugarte-Gil C2 , Rutherford G3 , Dickersin
K1
1 Cochrane Eyes and Vision Group, USA; 2 Johns Hopkins Bloomberg
School of Public Health, USA; 3 University of California, San
Francisco, USA
Background: The underlying affinity between outcomes that
leads to their co-occurrence in a research study identifies
central outcomes, i.e. outcomes most important to the
connectedness of the network of outcomes. Core outcome
sets (COS) are the minimum outcomes that should be
measured in research in a field. Current methods to develop
COS are inconsistent. Objectives: We conducted a social
network analysis (SNA) of outcomes in systematic reviews
(SRs) of HIV/AIDS to understand outcome co-occurrence
and compare the most central and the most frequent
outcomes. Methods: We examined all Cochrane SRs of
HIV/AIDS as of June 2013 and grouped individual outcomes
106
into 1/14 categories. Our SNA only considered outcomes
that co-occurred in ≥ 2 SRs. To identify central outcomes,
we used normalized node betweenness centrality (nNBC),
i.e.
the extent to which connections between other
outcomes in a network rely on a given outcome as an
intermediary. The higher the nNBC, the more central
is the outcome to a network. We identified the 7 most
central and most frequent outcomes because the Cochrane
Handbook recommends including ≤ 7 main outcomes in
SRs. We also examined centrality and frequency of outcomes
for the 4 pre-defined HIV/AIDS intervention subgroups:
clinical management, biomedical prevention, behavioral
prevention, and health services. Results: 140 SRs, measuring
294 unique outcomes, were eligible. The most central
outcomes in the overall network were all-cause mortality
(nNBC = 23.9) and cost/cost-effectiveness (nNBC = 16.4;
Fig 1). The most central and most frequent outcomes
differed overall and for each sub-network. For example,
for biomedical prevention, adverse events (specified), a
patient-important outcome, was among the most central
but not among the most frequent outcomes. Only 4/7
outcomes overlapped between the network and frequency
analyses for biomedical prevention (Fig 2). Conclusions: SNA
is a novel application for systematically identifying central
outcomes in SRs. Co-occurrence and frequency are both
important considerations for developing COS, particularly
because the two contribute different information. SNA should
be used when developing COS.
RO 18.8
The COMET Initiative database of core
outcome sets: increasing its value for users
Gargon E1 , Gorst S1 , Williamson P1
1
Background: The COMET Initiative developed an online
resource to collate the knowledge base for core outcome
set (COS) development. It will help authors present their
findings clearly and succinctly in Cochrane Reviews, e.g.
within the ’Summary of findings’ table. Initial searches (2013)
identified 198 COS in different areas of health. Ensuring that
the database is as comprehensive as possible and keeping
it up to date are key to its value for users. Objectives: To:
1. identify studies published since the original search with
the aim of determining which outcomes/domains to measure
in all clinical trials in a specific condition; 2. assess website
usage in 2014; 3. find out why people are using the database
Methods: 1. Searches in MEDLINE, SCOPUS; 2. Analyse
website statistics (Google Analytics); 3. A pop-up survey to
ask people their reason for searching the database. Results:
In January 2015 we searched databases and identified 4980
potentially relevant abstracts. Screening is ongoing to identify
the final set of included studies (expected completion April
2015). These results will be compared with the original review
to identify any changes in COS development and reporting.
Use of the website continues to increase, with more than
16500 visits in 2014 (36% increase over 2013), 12257 unique
visitors (47% increase) and 9780 new visitors (43% increase).
There has been a rise in the proportion of visits from outside
the UK: 8565 visits in 2014 (51% of all visits). By December
2014, a total of 6588 searches had been completed, with 2383
in 2014 alone (11% increase). Pop-up survey is on-going and
includes ‘I am planning a systematic review of clinical trials’
as a response option. Conclusions: These updates will bring
the database of COS up to date, improving its value for users.
It will also highlight clinical areas where gaps exist, providing
opportunities for future COS development. We will continue
to include studies in an ad-hoc way, until the next update,
ensuring that the database is as current as possible. The
pop-up survey will provide valuable insight into why people
search the database, which will also enable us to improve its
value for users.
Primary studies
LRO 5.1
Quality of clinical research: a systematic
review of definitions and development of a
comprehensive framework of indicators
von Niederhaeusern B1 , Guyatt G2 , Schandelmaier S3 , Hemkens
L3 , Bonde MM1 , Brunner N1 , Rutquist M1 , Bhatnagar N2 , Briel M3 ,
Pauli-Magnus C1
1 Clinical Trial Unit, Department of Clinical Research, University
Hospital, Basel, Switzerland; 2 Department of Clinical Epidemiology
and Biostatistics, McMaster University, Canada; 3 Basel Institute for
Clinical Epidemiology and Biostatistics, Department of Clinical
Research, University Hospital, Basel, Switzerland
Background: Clinical research is a prerequisite for advancing
our understanding and treatment of disease. While there
are factors that some understand as indicators of clinical
research quality (e.g. impact factors or whether good
clinical practice guidelines were followed), a comprehensive
definition considering various stakeholders’ perspectives
remains elusive.
Objectives: To: (1) systematically
review definitions of quality in clinical research; and (2) to
create a conceptual framework of indicators allowing for
a comprehensive assessment of clinical research quality.
Methods: Systematically we searched for definitions of
quality in clinical research by screening websites (and any
linked information) of various stakeholders involved in clinical
research.
Stakeholders included governmental bodies,
regulatory agencies, pharmaceutical industry, academic and
commercial contract research organizations, research ethics
committees, patient organizations and funding agencies from
12 countries. In addition, we systematically searched MEDLINE
with support from an information specialist. Data synthesis
will involve a structured summary of quality definitions. The
summary will inform a conceptual framework that we will
107
circulate among methodological experts and experienced
clinical researchers until no further ideas emerge using the
Delphi method. Results: At the time of the Colloquium, we will
present how different stakeholders from different countries
conceptualize clinical research quality. We will further provide
a framework of indicators of clinical research quality that
will span all phases of clinical research (i.e. planning to
dissemination) and include at least the following dimensions:
(1) absence of bias; (2) precision; (3) external validity; (4)
innovation/relevance; (5) transparency/public access to data;
and (6) well-trained personnel/number of trainees involved.
Conclusions: A holistic definition of clinical research quality
is still missing. Based on a systematic review we propose a
comprehensive framework of quality indicators that may be
used as the basis to develop a measurement tool of clinical
research quality.
LRO 5.2
The time-series studies for health policy
research in China
1
1
1
1
Xiuxia L , Jinhui T , Jingchun F , Kehu Y
1 Evidence-Based Medicine Center, School of Basic Medical Sciences,
Background: Time series analysis is an effective statistical
means for analyzing and forecasting the trends of things,
which has been effectively tried and applied in the field
of public health. Interrupted time-series studies (ITSs) are
usually considered as a valuable research tool for systematic
reviews of health systems research. Objectives: Based on
the principles of evidence-based medicine to explore and
show the evidence of time-series studies in Chinese health
policy research. Methods: Four Chinese databases (CBM,
CNKI, VIP, WANFANG), four international databases (PubMed,
the Cochrane Library, the Campbell Library, Web of Science)
and relevant websites were searched in November 2014.
The search terms were time-series, health, policy, strategy
and China/Chinese. EndNote X4 and Excel were used for
data description and analysis. Results: A total of 1232
articles were retrieved; this included 45 (3.7%) time-series
studies on national health policies, the first of which was
published in 2002, the most recent in 2013 (11; 24.4%), with
35 (77.8%) articles published from 2010 to 2014. Eighteen
(40.0%) were concerned with medical insurance, 10 (22.2%)
with pharmaceutical policy, seven (15.6%) with hospital
management, five (11.5%) wth theoretical exploration, and
five (11.1%) with public health. One (2.2%) study was
published in the International Journal of Medical Informatics
which is included in the Science Citation Index (impact factor
2.716, in 2013), eight (17.8%) were published in the periodicals
of the Chinese Science Citation Database, others (80.0%) were
published in general periodicals. Conclusions: Our research
shows the evidence of interrupted time-series studies in health
policy research has an increasing trend year on year in China,
but there are fewer concerned with rural health, health reform,
health workforce and so on. Very few studies were published
in high level periodicals. Further time-series analyses for
health policy research should promote and improve this.
LRO 5.3
Premature discontinuation of studies
approved by research ethics committees: a
comparison of randomized and
non-randomized clinical studies
Blümle A1 , Schandelmaier S2 , Oeller P1 , Kasenda B3 , Briel M2 , von
Elm E4
1
German Cochrane Centre, Medical Center, University of Freiburg,
Germany; 2 Institute for Clinical Epidemiology and Biostatistics,
University Hospital, Basel, Switzerland; 3 Institute for Clinical
Epidemiology and Biostatistics, University Hospital, Basel, United
Kingdom; 4 Cochrane Switzerland, Institute of Social and Preventive
Medicine (IUMSP), University Hospital, Lausanne, Switzerland
Background: More than 25% of planned randomized
controlled trials (RCTs) are prematurely discontinued.
Most discontinuations are not reported to research ethics
committees (RECs) and most discontinued RCTs remain
unpublished. This raises serious ethical concerns, including
waste of scarce resources, loss of collected patient data,
and missed opportunities to learn from failure. Similar
concerns apply to other prospective studies (non-RCTs)
such as non-randomized controlled trials, uncontrolled
trials, or cohort studies. However, the prevalence of
discontinuation of non-RCTs and the reasons thereof are
unknown. Objectives: To assess the prevalence of and the
reasons for discontinuation in non-RCTs, and to compare
the results to those in RCTs. Methods: Systematically
we surveyed studies that were approved by six RECs
(Germany, Switzerland, Canada) from 2000 to 2003, enrolled
patients or healthy volunteers, and collected outcome data
prospectively after study initiation. We had access to study
protocols of non-RCTs at one REC (Freiburg, Germany) and
to RCT protocols at all six RECs. We will collect study
characteristics such as medical area, type of participants,
sample size, collaboration, and funding source. We will
identify subsequent publications and survey investigators to
elucidate whether a study was discontinued and, if so, why.
Since some reasons such as benefit or harm only apply to
experimental studies, we will stratify the results by study
design. Results: An initial screen of study protocols suggests
that about 1000 studies were RCTs, 150 uncontrolled trials,
60 non-randomized controlled trials, 25 cohort studies, and
217 ineligible retrospective or cross-sectional studies. We
will present data on non-RCTs compared to RCTs regarding
the prevalence of discontinuation and the reasons thereof.
Conclusions: Compared to RCTs, typical features of non-RCTs
likely increase the risk for discontinuation (e.g. first in human
study) while others may decrease this risk (e.g. explorative
character). Non-RCTs are increasingly included in systematic
reviews, so that a a better understanding of their premature
discontinuation is warranted.
108
LRO 5.4
Are the estimates of blood pressure (BP)
lowering effect the same in parallel and
cross-over trials?
Wong GW1 , Wright JM1
1
Cochrane Hypertension Group, Canada
Background: Parallel and cross-over design are two possible
designs for randomized control trials assessing the blood
pressure (BP) lowering effect of antihypertensive drugs. Each
design has strengths and limitations. Both study designs
are valid scientific tools to measure the real effect of these
interventions. One would expect the effect estimate from
parallel studies to be similar to that from cross-over studies.
We conducted four systematic reviews on the BP lowering
efficacy of beta-blockers. Both parallel and cross-over
studies were included in our review so it was possible for
us to conduct an indirect comparison of the two types of
designs. Method: We compared the BP lowering efficacy
from parallel and cross-over studies within the same subclass
of beta-blockers. The data were synthesized using RevMan
5.2 and the mean estimates from the two designs were
compared using Student’s t tests. Results: The analysis
included 97 trials that examined the BP lowering efficacy
of 20 different beta-blockers. To be included the studies
had to be randomized, double-blind, placebo-controlled
trials in primary hypertensive patients taking fixed dose
beta-blockers for 3 to 12 weeks. The BP lowering estimates
(systolic/diastolic) from cross-over trials exceeded those from
parallel trials for all subclasses: nonselective beta-blockers
-12/-9 mmHg vs -5/-4 mmHg, alpha and beta receptor blockers
-11/-7 mmHg vs -4/-3 mmHg, partial agonists -9/-5 mmHg vs
-7/-3 mmHg and beta-1 selective blockers, -13/-11 mmHg
vs -7/-6 mmHg. The overall weighted mean was -13/-9
mmHg and -7/-5 mmHg respectively. All the comparisons
were statistically significant except for the partial agonists.
Conclusion: The findings suggest that the BP lowering effect
of beta-blockers is greater by 6/4 mmHg in cross-over trials
than parallel trials. This cannot be a real effect and creates
a serious dilemma for reviewers who are pooling results from
cross-over and parallel trials. In this case we suspect that the
estimate from the parallel trials is closer to the real effect.
These findings need to be replicated in other reviews. It is
possible that results from cross-over trials are not reliable.
LRO 5.5
Are routinely collected data studies reliable
surrogates when there are no randomized
trials? A meta-epidemiological study
Hemkens LG1 , Contopoulos-Ioannidis DG2 , Ioannidis JPA1
1 Stanford Prevention Research Center, Department of Medicine,
Stanford University School of Medicine, Stanford, California, USA;
2 Department of Pediatrics, Division of Infectious Diseases, Stanford
University School of Medicine, Stanford, California, USA
Background: Routinely collected data (RCD) studies are
proposed to inform health care decisions when randomized
controlled trials (RCTs) are not timely available, but have
an inherent risk of selection bias due to confounding by
indication. Propensity-score-analyses are frequently used to
address selection bias issues, but biases due to remaining
unaccounted confounders cannot be excluded. Objectives:
To assess the agreement between results of RCD-studies
and subsequent RCTs on the same topic in areas without
RCT evidence to guide health care decisions. Methods: We
searched PubMed (up to November 2014) for RCD-studies
published up to 2010 that used propensity scores and
reported comparative effects of medical interventions for
mortality endpoints. We included RCD-studies that were
conducted before any RCT was published for the same
topic and same compared interventions. We searched
systematically for subsequently published pertinent RCTs,
extracted mortality data, and calculated the mortality odds
ratio (OR). When more than one subsequent RCT was
identified, we combined them with random-effects models (or
Peto’s approach for rare events) and calculated the summary
OR. We analyzed the agreement between RCD-studies and
RCTs by calculating the relative OR (ROR; i.e. summary OR
of RCT(s) divided by RCD-study estimate; ROR > 1 means
RCD overestimate survival benefits). We then synthesized
the individual ROR data across all RCD-RCT pairs to calculate
the summary ROR (sROR) as overarching measure of the
agreement of early RCD-effects and subsequent RCT-results.
Results: We identified 929 records, evaluated 420 in
full-text, and screened 124 RCD-studies further. We analyzed
16 RCD-studies without preceding RCTs and for which
subsequent pertinent RCTs were identified. Preliminary
results indicate limited agreement between RCD-effects and
subsequent RCT evidence. RCD-studies showed significantly
inflated results (sROR 1.31; 95% CI 1.04 to 1.65). Conclusions:
RCD-studies that have been conducted prior to RCTs seem
to systematically and substantially overestimate the benefits
of medical treatments. Final results will be available at the
Cochrane Colloquium 2015.
LRO 5.6
Quasi-randomisation in emergency setting
trials: a recipe for selection bias, or an
efficient approach?
Corbett M1 , Moe-Byrne T1 , Oddie S1 , McGuire B1
1
Centre for Reviews & Dissemination, University of York, United
Kingdom
Background:
Recruitment into randomised trials in
emergency settings may be difficult because of the inherent
time restrictions encountered when enrolling, randomising
and treating participants. Although quasi-randomisation
might mitigate some of these problems, systematic reviewers
are likely to judge such trials as having a high risk of
bias, due to the possibility of selection bias. However,
selection bias may not actually occur since the very nature
109
of emergency settings might preclude opportunities for
bias. Objectives: To compare quasi-randomisation with
random sequence generation when used in emergency setting
trials, on the prevalence of possible selection bias, and on
trial recruitment outcomes. Methods: Systematic reviews
of participants with an acute injury or illness, requiring
intervention as quickly as was clinically practicable were
identified using the ‘emergency medicine’ topic in the
Cochrane Library. Eligible reviews incorporated at least
one trial using truly random sequence generation and at
least one trial using quasi-randomisation. Evidence of
possible selection bias was identified by assessment of
group baseline characteristics. Clinical advice and published
studies were used to identify important prognostic indicators,
and important magnitudes of group difference. Results:
The seven eligible reviews included 27 eligible trials: 11
used quasi-randomisation and 16 used random sequence
generation. Important group imbalance was identified in
two of the 11 quasi-randomised trials (18%) and four of
the 16 trials using true sequence generation (25%); of the
latter, three trials described appropriate methods to conceal
treatment allocation, though all three also had small sample
sizes. Clinical heterogeneity and poor reporting limited the
assessment of trial recruitment outcomes. Conclusions: In
emergency care settings quasi-randomised trials may be no
more likely to result in biased recruitment of participants
than trials using true randomisation. The likelihood of chance
imbalances affecting results is arguably more important to
consider, for both systematic reviewers and trial investigators.
Health Economics
LRO 6.1
Landscape review of costing methods for
reproductive, maternal and child health
interventions
Das J1 , Bhutta Z1
1
Aga Khan University, Pakistan
Background: Cost information is essential to improve the
economic efficiency of healthcare systems and costing
studies play a major role in this regard. However the methods
used across different economic evaluation studies are
diverse, and it is it is imperative to understand these methods
and their applicability in different settings. Objectives:
We undertook a review to evaluate various cost scale-up
evaluations from regional, national and multinational
scale-up studies for Reproductive, Maternal, Newborn and
Child Health (RMNCH) interventions to identify the major
methods used for estimating the costs. We also aimed
to compare the features of each costing method using an
evaluation matrix based on peer review and publication
quality. Methods: Peer-reviewed literature search was
conducted for studies on scale-up costing for RMNCH. A
data abstraction table was created to extract details on the
objective and components of the method, as well as the
areas of application for each method. Scale-up methods
were then categorized and tabulated. Results: We included
24 studies focusing on regional, national or multi-country
scale-up of RMNCH interventions. We prepared a summary of
the components of the evaluation matrix from each of these
studies. Cost scale-up projections for RMNCH interventions
are difficult, however various methods have been used to
estimate the scale-up costs for evidence-based decision
making. Different cost concepts and costing methodology are
being used depending upon the purpose for which cost data
are being used, the perspective of the study, the type and
complexity of the health service, the precision required, the
requirements of generalizability and representativeness and
the availability of reliable and valid data. Conclusions: Large
scale regional and global cost scale-up analysis mostly relies
on scenarios of how results vary under certain situations or
on averaging the results from a set of models. There is a need
for a standardized costing methodology, if all stakeholders,
including providers, purchasers and policy makers, are to
make informed decisions. Accurate costing can contribute to
the efficient allocation of resources.
LRO 6.2
HPV vaccines in low- and middle-income
countries: a synthesis study of economic
evaluations
Canelo-Aybar C1 , Mezones-Holguin E1 , Balbin G2 , Perez-Gomez
A3 , Florez ID4
1 School of Medicine, Universidad Peruana de Ciencias Aplicadas,
Peru; 2 Casimiro Ulloa, Peru; 3 Health Technology Assessment
Institute, Colombia; 4 McMaster University, Canada
Background: HPV vaccines are an important intervention
for preventing cervical cancer. In public health, associated
costs should be included for making decisions. Thus,
economic evaluations (EE) play a relevant role, especially
in the context of low- and middle-income countries (LMICs).
Objectives: To describe and appraise critically the available
published EE evidence on HPV vaccines in LMICs. Methods: A
systematic literature review using a validated search strategy
was carried out to identify a complete EE (cost-effectiveness,
cost-utility and cost-benefit analysis) for HPV vaccines in
LMICs published up to March 2015 in PubMed/MEDLINE,
EMBASE and the Cochrane Library. Language restrictions
were not used. Congress abstracts were excluded. Quality
assessment was performed using the Consensus on Health
Economic Criteria instrument. Also, age at which vaccinated,
analysis perspective, clinical and economical outcomes, type
of vaccine, model performed, Incremental cost-effectiveness
ratio (ICER), and founding were analyzed, as well as other
variables. Results: A total of 1827 citations were obtained, of
which 415 were duplicates. After title and abstract screening,
64 unique studies were reviewed (Fig 1) and 43 EEs were
included for full qualitative synthesis -18 additional studies
110
from a previous systematic review. Only seven studies were
done specifically in low-income countries. Vaccination age
ranged between from 10 to 15 years old. Most investigations
incorporated only cervical cancer (32), but others additionally
included harms (11) in their analysis. Most studies (34) were
funded by public resources and indirect costs were assessed in
23 studies. The static models are the most frequent technique
used and 10 studies had a multi-country evaluation. In
most studies, findings suggested that vaccination is likely to
be cost-effective; however these results are dependent on
context and can be affected by GDP per head and existing
cervical cancer screening program. Conclusions: Beyond
the significant heterogeneity and limited quality of current
evidence, there is a tendency to support the cost-effectiveness
of HPV vaccines in these countries.
LRO 6.3
Pharmaceutical policies: effects of reference
pricing, other pricing, and purchasing
policies (Updated Review)
Acosta A1 , Ciapponi A2 , Aaserud M3 , Vietto V4 , Austvoll-Dahlgren
A5 , Kösters JP6 , Vaca C7 , Machado M8 , Diaz Ayala DH8 , Oxman A5
1
School of Pharmacy, Universidad Nacional de Colombia,
Colombia; 2 Argentine Cochrane Centre IECS, Institute for Clinical
Effectiveness and Health Policy, Hospital Italiano de Buenos
Aires, Southern American Branch of the Iberoamerican Cochrane
Centre, Buenos Aires, Argentina; 3 Statens Legemiddelverk,
Norwegian Medicines Agency, Oslo, Norway; 4 Division of Family
and Community Medicine, Hospital Italiano de Buenos Aires,
Argentina; 5 Norwegian Knowledge Centre for the Health Services,
Oslo, Norway; 6 The Nordic Cochrane Centre, Rigshospitalet,
Copenhagen, Denmark; 7 Grupo RAM, Universidad Nacional de
Colombia, Bogota, Colombia; 8 Departamento de Farmacia,
Universidad Nacional de Colombia, Bogota, Colombia
Background: Pharmaceuticals are important interventions
that could improve people’s health. Pharmaceutical pricing
and purchasing policies are used as cost-containment
measures to determine or affect the prices that are
paid for drugs. Internal reference pricing establishes a
benchmark or reference price within a country which is
the maximum level of reimbursement for a group of drugs.
Other policies include price controls, maximum prices,
index pricing, price negotiations and volume-based pricing.
Objectives: To determine the effects of pharmaceutical
pricing and purchasing policies on health outcomes,
healthcare utilisation, drug expenditures and drug use.
Methods: Policies in this review were defined as laws; rules;
financial and administrative orders made by governments,
non-government organisations or private insurers. To be
included a study had to include an objective measure of at
least one of the following outcomes: drug use, healthcare
utilisation and health outcomes or costs (expenditures); the
study had to be a randomised trial, non-randomised trial,
interrupted time series (ITS), repeated measures (RM) study
or a controlled before-after study of a pharmaceutical pricing
or purchasing policy for a large jurisdiction or system of
care. Results: We included 18 studies (seven identified
in the update): 17 of reference pricing, one of which also
assessed maximum prices, and one of index pricing. None of
the studies were trials. All included studies used ITS or RM
analyses. The quality of the evidence was low or very low for
all outcomes. Four studies reported effects on mortality and
healthcare utilisation, however they were excluded because
of study design limitations. Conclusions: The majority of
the studies of pricing and purchasing policies that met our
inclusion criteria evaluated reference pricing. We found that
internal reference pricing may reduce expenditures in the
short term by shifting drug use from cost share drugs to
reference drugs. Index pricing may reduce the use of brand
drugs, increase the use of generic drugs, and may also slightly
reduce the price of the generic drug when compared with no
intervention.
Reference
Acosta A, Ciapponi A, Aaserud M, Vietto V, Austvoll-Dahlgren
A, Kösters JP, Vacca C, Machado M, Diaz Ayala DH, Oxman AD.
(2014) Pharmaceutical policies: effects of reference pricing,
other pricing, and purchasing policies (Review). The Cochrane
Library, Issue 10.
LRO 6.4
Calculating costs: facing challenges for a cost
comparison of day surgical and inpatient
varicose vein surgery as an example
Fischer S1
1
Ludwig Boltzmann Institute for Health Technology Assessment,
Austria
Background: Evidence-based medicine (EbM) and Health
Technology Assessments (HTA) aim to improve health care
and allocation of resources, so besides clinical aspects,
111
cost calculations are becoming more important, with
selection of data being crucial for an appropriate calculation.
Objectives: Comparison of the costs of day surgical and
inpatient varicose vein surgery as an example to show the
results of using different data sources. Methods: For
the first approach, we used meta-data from the Austrian
‘‘Dokumentations- und Informationssystem für Analysen im
Gesundheitswesen’’. We considered generated lump sums
from hospital reimbursement, total costs that occurred in the
departments and number of patients in an equation. For the
second approach, we used data from individual hospitals to
calculate the costs for the surgery itself and nursing. The third
approach contained the adaption of international cost data,
by an adjustment for inflation and prices for Austria. Results:
The calculated costs of varicose vein surgery differed between
EUR 859 and EUR 4664 for a day case and EUR 1720 to 2330
for an inpatient treatment –depending on the used approach.
The main strength of the first approach is that it can be done
relatively quickly. However, the validity of the calculated
costs is low. Thus, a difference between real and calculated
costs is likely. The use of hospital data takes more time,
though, the quality of the data is much better. A weakness
of this approach is that individual costs are not available for
a specific intervention. Furthermore, these costs are hospital
specific and generalising for other hospitals is difficult. The
fast acquisition of the international reference costs is a main
strength, though, the costs are from a different healthcare
system and therefore, the transferability of the costs is limited.
Conclusions: The results have shown that an examination of
administrative data is indispensable for proper cost evidence.
Calculations of costs for EbM and HTA that are adjusted to
national circumstances are required. However, many issues
of an efficient healthcare system and appropriate allocation
of resources are unsolved.
LRO 6.5
Impact of pay-for-performance on behavior
of primary care physicians and patient
outcomes: a systematic review
Liao G1 , Lin Y2 , Huang J3 , Lin Y2
1
West China Hospital of Stomatology, Sichuan University, China;
2 Department of Urology, West China Hospital, Sichuan Univerisity,
China; 3 West China Hospital, Sichuan Univerisity, China
Background: Pilot countries such as the UK have suggested
that pay-for-performance (P4P) is an effective payment
method, which links physicians’ income to the quality of their
services. With its wide application in primary health care, more
countries are exploring its role among general practitioners.
Interestingly, research on P4P has yet to reach an agreement,
so a systematic review was performed for a more affirmative
result. Objectives: To provide a comprehensive and objective
evaluation of P4P for decision-makers. Methods: Studies were
identified by searching PubMed, EMBASE and the Cochrane
Library. Electronic searching was conducted in the fourth
week of January 2013. As the included studies had significant
clinical heterogeneity, a descriptive analysis was conducted,
and a Quality Index was used to assess the quality of the
evidence. Results: Among the 44 included articles, 36 were
about the effect of P4P on quality of chronic diseases; 20
were about equity, five about satisfaction of patients, four
about cost-effectiveness, 24 about a factor of implementation
of P4P(the baseline of P4P and the practice size). P4P
had an overall positive effect, which varied in accordance
with the baseline quality of medical care and the practice
size. The implementation of P4P could bring about new
problems regarding the inequity of medical care, patients’
dissatisfaction and the rise of medical cost. Conclusions:
Decision makers should consider the baseline conditions of
medical quality and the practice size before new medical
policies are enacted. Furthermore, most studies investigated
here were retrospective and observational with high levels
of heterogeneity, though the descriptive analysis is still of
significance.
LRO 6.6
Network meta-analysis of multiple outcomes
to inform a cost-effectiveness analysis:
interventions for the induction of labour
Keeney E1 , Alfirevic Z2 , Dowswell T2 , Welton NJ1 , Medley N2 , Dias
S1 , Jones LV2 , Gyte G2 , Caldwell DM1
1
University of Bristol, United Kingdom; 2 University of Liverpool and
Liverpool Women’s Hospital, United Kingdom
Background: Network meta-analysis (NMA) allows estimation
of multiple intervention effects across a network of
interventions. NMA is particularly useful for economic
evaluations, which compare the cost-effectiveness of multiple
interventions.
However, economic evaluations usually
depend on several outcomes, whereas a systematic review
typically reports one outcome at a time. Objectives: Our
aim is to illustrate the joint modelling of multiple outcomes,
and assessment of inconsistency in NMA, for providing inputs
to an economic model. Methods: We use as an example a
systematic review and NMA to identify the most cost-effective
intervention(s) for induction of labour. Trials in this area
report multiple, but related, outcomes for the mother and
baby. The economic model requires estimates of the
probabilities of vaginal delivery (VD) within 24 hours, VD after
24 hours, and caesarean section, as well as the proportion
of babies admitted to intensive care. These outcomes
are not independent, and require estimating jointly in the
NMA. Careful definition of denominators allows estimates of
conditional probabilities for inputs for the economic model
to be obtained. We use various approaches to explore and
deal with inconsistency in the NMA, including node-splitting,
continuity correction and removing trials at high risk of bias.
Results: Titrated (low-dose) oral misoprostol solution is the
treatment that not only has the lowest cost but also results
in the highest utility for mothers and babies. Conclusions:
Related outcomes should be jointly modelled in order to
provide the required estimates for economic models. We
112
found that trials did not always provide enough information
to be able to identify the outcomes needed in the economic
model. Exploring and dealing with inconsistency is essential
for the validity of conclusions drawn from an economic
evaluation based on a NMA.
LRO 7.2
Policymakers’ familiarity with and
preference for overall quality of evidence
definitions
Kriz HM1 , Harrod CS1 , Gerrity MS1 , King VJ1
1
Oregon Health & Science University, USA
Involvement of users and
stakeholders
LRO 7.1
Patient Involvement in systematic reviews
–why it is so important?
Rhodes C1 , Jordan JL1 , Belcher J1 , Yardley S1 , Hyde C1 ,
Higginbottom A1 , Worrall A1 , Taylor R1
1 Research Institute for Primary Care and Health Sciences Keele
University, United Kingdom
Background: The Primary Care and Health Sciences Research
Institute at Keele University in the UK has an established
patient and public involvement (PPI) group informing research
studies. Objectives: To describe how patients developed
skills to understand research methods patients were involved
in two systematic reviews: 1. narrative synthesis: shared
decision-making in primary care consultations; and 2. realist
synthesis: learning and healthcare delivery in primary
care. Methods: Support for patients included workshops
on systematic reviews, evidence-based medicine and
understanding statistics. A PPI coordinator assessed needs
throughout, ensuring adequate time and support within
project timescales.
Narrative synthesis: five patients
participated in three sequential workshops ensuring the
research question was relevant and that factors important to
patients in sharing decisions were identified. Realist synthesis:
two patients, a PPI co-ordinator and researcher produced a
data extraction sheet useable by all team members and
papers were split between them to review. Results: Narrative
synthesis: patients identified additional factors, poorly
represented in the literature and planned dissemination of
results with different audiences. Realist synthesis: patients
ensured data extraction forms were developed in an easy
to read format and formed a seminar panel to disseminate
methods used, informing future primary care delivery on
multimorbidity. Conclusions: Patients working within a
research team can inform the review question and undertake
extraction, interpretation and analysis of data. Patients
involved in research need support to enable them to develop
skills to make sense of scientific evidence. Patients bring
a unique perspective to many areas of the research review
process, identifying areas of importance. Researchers need
support to translate research terms into accessible formats to
ensure the patient voice is truly integrated into their review.
Background: Oregon Health & Science University’s Center for
Evidence-based Policy (Center) conducts evidence summaries
for policy makers using the Grading of Recommendations
Assessment, Development and Evaluation (GRADE) approach
to assess overall quality of evidence (QoE). Many policymakers
find it difficult to interpret and apply GRADE QoE definitions.
Objectives: Assess policymakers’ familiarity with GRADE
and preference for five different QoE definitions. Methods:
Policymakers participating in Center projects were invited to
complete a 24-item web-based survey assessing familiarity
and preference of five QoE definitions including: GRADE,
Agency for Healthcare Research and Quality (AHRQ),
Evidence-based Practice Center (EPC), and three alternative
definitions developed by the Center. Familiarity with GRADE
was based on a 2-item composite variable. Policymaker
degree of certainty that findings will remain stable over
time was rated from 0% to 100% for each QoE definition.
Odds ratios were used to assess the association between
familiarity and definition preference. Results: Out of 92
policymakers, 42 (46%) completed the survey with 25 (60%)
indicating that they were familiar with GRADE. Nearly half
of respondents preferred the EPC definitions. Fifteen (38%)
preferred alternative definitions, but none selected a brief lay
terminology definition. Only five (13%) preferred the standard
GRADE definition. For each GRADE QoE level, the median per
cent certainty that findings from evidence would remain stable
was 90% for high QoE, 75% for moderate QoE, and 45% for
low QoE. Although 29 (69%) policymakers had experience with
GRADE, none were extremely familiar with it. Familiarity with
GRADE was not associated with a preference for standard
(GRADE or EPC) QoE definitions compared to alternatives
(OR = 1.3, 95% CI 0.4 to 5.1; P value 0.7). Conclusions: The EPC
definitions were preferred by policymakers. Despite having
experience with GRADE, no policymaker felt extremely familiar
with it. Policymakers’ familiarity was not associated with
definition preference. Strategies to enhance policymakers’
understanding and application of GRADE ratings are needed.
LRO 7.3
An approach for systematic review teams to
engage policymakers and stakeholders: from
question selection to knowledge translation
El-Jardali F1 , Fadlallah R1 , Akl E1
1 American University of Beirut, Lebanon
Background: Engaging policymakers and stakeholders
in evidence synthesis activities is an essential step to
113
promote the use of evidence in policy making and practice.
This creates a need for a well-developed and structured
approach that spans from selecting relevant review questions
and synthesizing evidence, to using it in informing policy
decisions. Objectives: To describe an innovative approach
to engage policy makers and stakeholders in evidence
synthesis activities and knowledge translation. Methods: The
development methodology followed a multifaceted process
that encompassed the following steps: 1. comprehensive
literature review; 2. development of approach; 3. expert
opinions; 4. refinement of approach; and 5. real life
experience (implementation).
Results: The proposed
approach is shown in Figure 1. It is broadly divided into
three phases: 1. preparatory work for selection of potential
review questions; 2. priority setting and evidence synthesis;
and 3. knowledge translation. The approach is characterized
by its comprehensiveness, the ongoing engagement of
Figure 1
policymakers and stakeholders, and the segregation of review
questions into those for systematic reviews and those for
knowledge translation. In the presentation, we will provide
a detailed description of the approach, including the process
and outcome of its development. In addition, we will draw on
case studies to demonstrate the application of the approach.
We will also discuss lessons learned and implications for
other groups and systematic review centers planning to
engage policymakers and stakeholders in evidence synthesis
activities. Conclusions: The proposed approach will help
align the production of evidence syntheses with policy
priorities and promote their use in decision-making. It can
also help leverage and harness existing systematic reviews
on priority questions and devise appropriate knowledge
translation products.
LRO 7.4
The challenges of using evidence-based
methods for decisions at a regional level
involving stakeholders without research
training
Grillich L1 , Sommer I2 , Mahlknecht P1 , Gartlehner G1
1
Danube University, Krems, Austria; 2 Danube University, Krems,
Austria
Background: Childhood obesity is a main public health
challenge. Children who are overweight or obese are
at greater risk of poor health in adolescence and also
in adulthood. Thus, the regional government of Lower
Austria wanted to implement an evidence-based decision
process involving stakeholders and experts whether or not
to implement a life-style intervention program for obese
children/adolescents. Objectives: To describe the challenges
of implementing an evidence-based decision process
using GRADE (Grading of Recommendations Assessment,
Development and Evaluation) and DECIDE (Developing and
Evaluating Communication Strategies to Support Informed
Decisions and Practice Based on Evidence) frameworks at
a regional level involving local stakeholders without prior
knowledge of evidence-based or other research methods.
Methods: We used GRADE and DECIDE to appraise and
communicate the quality of evidence and the strength of
recommendations. To assess the relative importance of
the desirable and undesirable outcomes we conducted two
focus groups (one with overweight children/adolescents
and one with parents of these). Scientific experts and
relevant stakeholders selected and prioritized outcomes.
We carried out an umbrella review to critically appraise,
synthesise and grade the current evidence and guided a shared
evidence-based recommendation making process with all
stakeholders. Results: Using GRADE for a complex public
health question at a regional level was challenging: we had to
convey to experts and stakeholders that prioritizing outcomes
is necessary for decision making and we had to disentangle
the large number of prioritized outcomes and map them to
potentially quantifiable outcomes. Last but not least, the
heterogeneity of outcome measures reported in the literature
required the adaptation of the GRADE system to rating
the quality of evidence. Conclusions: Although using the
GRADE and DECIDE frameworks at a regional level involving
stakeholders without research training was challenging, it
provided a structured and well received method to reach a
conclusion about implementation of a life-style intervention
program for obese children/adolescents.
114
LRO 7.5
Extensive consumer/stakeholder
engagement leads to practical key messages
Van Eerd D1 , Irvin E2 , Munhall C2 , King T1
1 Institute for Work & Health, Canada; 2 Cochrane Back Review
Group, Canada
Background: The goal of systematic reviews is to provide
an up-to-date synthesis of the evidence that can be used in
practice to improve the health of people. Individuals who
have a stake in the evidence include researchers, providers/
clinicians and consumers/clients. Changing practice involves
consumers/clients, therefore providing information in a way
that is useful to these stakeholders is paramount. Objectives:
Describe the engagement of consumers/stakeholders in a
review process and the resulting changes in key messages.
Methods: This is a case study using a review of the
effectiveness of workplace prevention of upper extremity
musculoskeletal disorders employing a Cochrane approach.
In this review, we adapted a stakeholder engagement process
(Keown 2008). The engagement process was iterative and
engaged consumers/stakeholders in the co-development of
key messages from the review. Results: An iterative approach
was used to engage multiple consumers/stakeholders at
multiple time-points through the review process. In particular,
consumers/stakeholders were heavily engaged in the
generation of relevant and practical messages that they
could implement in their day-to-day practice. They were also
instrumental in determining and accepting the threshold
for sufficient evidence and accompanying messages.
Conclusions: An iterative approach resulted in practical
messages and recommendations. Through this iterative
process messages became more practical and ‘useful’ to
consumers/stakeholders. While engaging with multiple
consumers/stakeholders is time-consuming we experienced
greater uptake of the review results and key messages with
this process.
ensure they are appropriate and accessible to different
population groups, but the involvement of people with
the condition is not always considered when designing the
intervention. Appropriateness, relevance and acceptability of
interventions are not routinely considered as criteria when
conducting reviews, but may contribute to variations in
effectiveness. Objectives: Our realist review aimed to develop
a preliminary theory of the contexts that promote/enable
patient involvement in diabetes research, and whether
involvement could produce more relevant interventions.
Our review questions explored how people have been
involved in diabetes research, and whether involvement
influenced the development and implementation of diabetes
interventions. Methods: We assembled a review team
of diabetes researchers who are seeking information on
involvement to inform the design and conduct of their
studies. A preliminary theory for successful involvement
was developed from involvement literature. We searched
MEDLINE, CINAHL, EMBASE in the first instance to identify
articles discussing involvement with people at risk for/with
diabetes. Cluster searching identified multiple articles on
the same study. Several typologies for patient involvement
were tested to determine their utility as data extraction tools.
Results so far: To date we have included 79 articles. Cluster
searching identified eight projects that generated multiple
articles (37). Two different approaches to involvement are
used: a community-based participatory approach where
people are actively involved in all stages of development and
implementation, and a more ‘traditional’ approach where
researchers selectively involve people at specific points.
Outcomes chaining is being used to identify relationships
between the involvement process, uptake and attainment of
diabetes-related goals.
Search filters
LRO 8.1
LRO 7.6
Search filters: mapping the literature using a
novel form of content analysis
Does involvement in knowledge creation
contribute to development of complex
interventions? A realist review of
involvement in diabetes research
Wilson J1 , Anderson M1
1 CDPLPG, United Kingdom
Harris J1 , Graue M2 , Kirkevold M3 , Haltbakk J2 , Rokne B4 , Dunning
T5
1 Qualitative&ImplementationResearchMethods, UnitedKingdom;
2
Bergen University College, Norway; 3 University of Oslo, Norway;
4
University of Bergen, Norway; 5 Deakin University, Australia
Background: Systematic reviews are considered as the
foundation of knowledge translation, but their quality can
only be as good as the primary studies that are included
in the review. Recently there have been several reviews
looking at the importance of tailoring interventions to
Background: Search filters are combinations of words
and phrases designed to retrieve an optimal set of
records on a particular topic (subject filters) or study
design (methodological filters). Information specialists are
increasingly turning to reusable filters to focus their searches.
However, the extent of the academic literature on search filters
is unknown. We provide a broad overview to the academic
literature on search filters. Objectives: To map the academic
literature on search filters from 2004 to 2015 using a novel form
of content analysis. Methods: We conducted a comprehensive
search for literature between 2004 and 2015 across eight
databases using a subjectively derived search strategy. We
115
identified key words from titles, grouped them into categories,
and examined their frequency and co-occurrences. Results:
The majority of records were housed in Embase (n = 178) and
MEDLINE (n = 154). Over the last decade, both databases
appeared to exhibit a bimodal distribution with the number of
publications on search filters rising until 2006, before dipping
in 2007, and steadily increasing until 2012. Few articles
appeared in social science databases over the same time
frame (e.g. Social Services Abstracts, n = 3). Unsurprisingly,
the term ‘search’ appeared in most titles, and quite often, was
used as a noun adjunct for the word ‘filter’ and ‘strategy’.
Across the papers, the purpose of searches as a means
of ‘identifying’ information and gathering ‘evidence’ from
‘databases’ emerged quite strongly. Other terms relating
to the methodological assessment of search filters, such as
precision and validation, also appeared albeit less frequently.
Conclusions: Our findings show surprising commonality
across the papers with regard to the literature on search
filters. Much of the literature seems to be focused on
developing search filters to identify and retrieve information,
as opposed to testing or validating such filters. Furthermore,
the literature is mostly housed in health-related databases,
namely MEDLINE, CINAHL, and Embase, implying that it is
medically driven. Relatively few papers focus on the use of
search filters in the social sciences.
LRO 8.2
UK filter project: methodological approach to
reduce information overload by developing
validated UK geographic search filters for
MEDLINE and Embase
Ayiku L1 , Craven J1 , Levay P1 , Finnegan A1 , Barrett E1
1 National Institute for Health and Care Excellence (NICE), United
Kingdom
Background: Studies from United Kingdom (UK) health and
social care settings can be required to inform evidence-based
guidance developed by the UK National Institute for Health
and Care Excellence (NICE). However, robust methods for
retrieving UK studies in systematic literature searches do
not exist. The pragmatic use of non-validated UK search
strategies risks an unknown proportion of UK studies being
missed. Conversely, sifting search results that are not
restricted by geographic location creates a costly and
time-consuming information overload when UK studies are
required. Information specialists at NICE are developing
validated UK geographic search filters to resolve this issue.
Objectives: To develop validated UK geographic search filters
for MEDLINE and Embase (Ovid) with the best balance of recall
and precision for systematic literature searches. Methods:
1. Gold standard: This will be generated using the relative
recall method. UK studies that have informed NICE guidance
will be identified and the studies that are indexed in MEDLINE
or Embase will form the gold standard for each database. A
power calculation will determine the number of gold standard
studies required to develop each filter. 2. Filter development:
The gold standard for each database will be randomly divided
into a ‘development set’ and a ‘test set’. The UK-related search
terms for the MEDLINE and Embase filters will be derived
objectively using a text analysis system on the ‘development
set’ of each database. 3. Validation: The effectiveness of
the filters will be validated against the ‘test set’ of each
database by calculating their recall and precision. Expected
conclusions: The validated UK geographic search filters will
enable the robust retrieval of UK studies from MEDLINE and
Embase in systematic literature searches. The filters and
accompanying methods will be shared via online search filter
resources and peer-reviewed journal articles. Professional
peers will be able to use the filters to retrieve UK studies or to
transfer the development methods to create search filters for
other geographic regions.
LRO 8.3
Feasibility of developing and validating a
geographic filter for a small country like
Austria
Jeitler K1 , Semlitsch T2 , Horvath K3 , Posch N2 , Berghold A4 ,
Siebenhofer A2
1
Institute of General Practice and Evidence-Based Health Services
Research and Institute for Medical Informatics, Statistics and
Documentation, Medical University of Graz, Austria; 2 Institute
of General Practice and Evidence-Based Health Services Research,
Medical University of Graz, Austria; 3 Institute of General Practice
and Evidence-Based Health Services Research and Department
of Internal Medicine, Division of Endocrinology and Metabolism,
Medical University of Graz, Austria; 4 Institute for Medical
Informatics, Statistics and Documentation, Medical University of
Graz, Austria
Background: In search strategies focusing on studies
conducted within local health care systems highly sensitive
geographic filters (GFs) may be a reasonable means to reduce
research resources. At present, only few validated GFs are
available and to our knowledge no GF for Austrian studies
has been published. Objectives: To develop and validate a
GF to retrieve studies with Austria as a reference point for use
in the PubMed database, if feasible. Methods: At first, we
will estimate roughly the prevalence of Austrian publications
among all PubMed entries in the last five years searching
for the term ’Austria’ in the affiliation [ad], title and abstract
[tiab] fields as well as in the geographic locations category of
the medical subject headings [MeSH]. We will consider further
steps to be appropriate, if at least 10 relevant publications in
a validation reference set of 2000 abstracts can be expected.
We will limit the reference set to a workable number of 2000
abstracts as the abstracts have to be categorized manually
by relevance on the basis of the information available from
the database. The GF to identify studies performed in Austria
will be created using English and German terms derived
from the names of the country, its provincial states, and
districts. These terms will be searched for in additional fields
like transliterated title [tt] or other term [ot] besides the
aforementioned fields. A random sample of 2000 references
will be drawn from articles published in PubMed within the
last five years. Two reviewers will independently identify
116
relevant articles for the reference set. We then will assess
the performance of the Austrian GF in terms of its specificity
and sensitivity. Results: We will present results on whether
developing a geographic filter for a small country like Austria
is feasible with an acceptable amount of effort and discuss
possible caveats. Furthermore, filter details and performance
data will be published.
LRO 8.4
Finding qualitative literature on dementia in
MEDLINE, EMBASE, PsycINFO and CINAHL:
assessment of three qualitative search filters
Rogers M1 , Bethel A1
University of Exeter Medical School, United Kingdom
1
Background: Systematic reviews of qualitative data and
mixed methods reviews play an important role in dementia
care and care planning. Finding qualitative data in medical
databases to inform reviews can be difficult. Objectives: 1.
To investigate whether qualitative search filters help with
finding qualitative research in dementia. 2. To find out
which filter works best in terms of effectiveness and ease
of use. 3. To examine the effectiveness of three qualitative
filters on four major databases (MEDLINE, EMBASE, PsycINFO
and CINAHL). Methods: We identified qualitative or mixed
methods systematic reviews in dementia care by searching
the Cochrane Database of Systematic Reviews (CDSR), the
Database of Abstracts of Reviews of Effects (DARE), the
Health Evidence database (http://www.healthevidence.org/),
MEDLINE, EMBASE, PsycINFO and CINAHL. Qualitative studies
identified in these reviews were used to form a test set
of literature. Three qualitative methodology filters were
selected (one with few, broad-based terms, one with many
specific free-text terms and thesaurus terms for qualitative
studies), adapted for the four databases, and were tested
for sensitivity, precision and specificity against the test
literature. Results: The results for the performance of each
filter for each database will be presented. Conclusions: This
work will help to inform whether it is useful to utilise a search
filter to identify qualitative literature in dementia, and which
is the most effective filter for locating this literature in four
major databases.
LRO 8.5
Use of filters in electronic searching
for observational studies
Jaiswal N1 , Singh M1 , Thumburu K2 , Kaur J2 , Chadha N3 , Kumar
M2 , Agarwal A2
1
Department of Pediatrics & ICMR Advanced Centre for
Evidence-Based Child Health, Post Graduate Institute of Medical
Education and Research, India; 2 ICMR Advanced Centre for
Evidence-Based Child Health, Post Graduate Institute of Medical
Education and Research, India; 3 Dr.Tulsidas Library, Post Graduate
Institute of Medical Education and Research, India
Background: Searching various databases is an exhausting
job for researchers. Different databases require different
terminologies and different filters and multiple databases
need to be searched for ensuring comprehension and avoiding
bias. The search strategies formulated are therefore long
and involve great expertise. Using search filters provided
by the databases is therefore an aid to decrease the load.
There are available search terms for searching for randomized
controlled trials, but no such standardized strategy exists for
searching for observational studies. Objectives: Comparison
of search outputs with or without using the search filters
in searches involving observational studies. Methods: We
illustrate the use of search filters and searching without
the use of specific filters through an example of a systematic
review of observational studies. The databases searched were
Pubmed, Embase and MEDLINE via Ovid SP. Two reviewers
searched the databases independently, one used the search
filters provided by the database and the second reviewer did
not use the specific filters, but instead used the specific search
terms for the observational studies for electronic searching of
the databases. The other blinded reviewers then compared
and screened the results. Results: Use of specific search
filters significantly decreased the total number of hits from
16323 records to 6423 records and did away with most of
the irrelevant results, but missed out on 5% of the included
studies. Conclusions: The specific search filters reduce the
workload, but need to be used with caution as some of the
important records may be missing. Acknowledgements: This
study was supported by WHO, Geneva & ICMR, New Delhi
LRO 8.6
Refinement of search filters to reduce
information overload when retrieving
evidence on variations in practice for NICE
Quality Standards
Craven J1 , Ayiku L1 , Barrett E1 , Walton L1
1
National Institute for Health and Care Excellence, United Kingdom
Background: The National Institute for Health and Care
Excellence (NICE) Quality Standards are concise sets of
prioritised statements designed to drive measurable quality
improvements within a particular area of health or care
in the United Kingdom (UK). A central aim of Quality
Standards is to reduce variations in practice in care
that arise from non-adherence to guidance, and which
result in poor quality care.
In 2012, to support the
development of Quality Standards, information specialists
within the guidance Information Services (gIS) team at NICE
developed a search filter to identify literature on variations
in practice from bibliographic databases including MEDLINE,
Embase, PsycINFO, and the Health Management Information
Consortium (HMIC). Having used the filter over a period of
time, systematic reviewers from the Quality Standards team
reported that a large number of irrelevant references were
being retrieved by the filter. In response, the information
specialists undertook a project to increase the precision
of the filter. The revised filter was then piloted for a
117
period of eight months until March 2015. Objectives: To
improve the precision of the ‘variations in practice’ search
filter without excluding key papers. Methods: 1. Analysis
undertaken of the irrelevant results from previous searches
to identify which aspects of the filter could be modified. 2.
Modification of the filter, and assessment of its performance
in retrieving key papers that had informed previous Quality
Standards topics. 3. Piloting the revised filter on new Quality
Standards topics to assess its ability to retrieve key papers
and reduce the volume of irrelevant references. Results:
Precision has been improved: the number of irrelevant
references retrieved has been reduced to an acceptable
level. Conclusions: Information overload through retrieval
of irrelevant references has been reduced, saving the sifting
time of systematic reviewers. The modified filter has proved
to be robust in identifying key papers and will be used for all
forthcoming Quality Standards variations in practice searches.
Oral session 10
Filtering the information overload
for better decisions
O 10.1
The Embase project 2: crowdsourcing citation
screening
1
3 York Health Economics Consortium, United Kingdom; 4 Cochrane
Background: The Embase project has been managed since
April 2013 by a consortium made up of Metaxis Ltd, the
Cochrane Dementia and Cognitive Improvement Group and
the York Health Economics Consortium. It uses a novel
crowdsourcing method to screen citations. Objectives:
To evaluate the effectiveness of using crowdsourcing to
identify unique reports of randomised trials in Embase and
to submit those reports to Cochrane’s Central Register of
Controlled Trials (CENTRAL). Methods: We recruited a crowd
to screen the search results identified from the monthly
sensitive searches run in Embase (via Ovid SP). Using a
bespoke online citation screening tool the crowd classify
citations as ‘RCT/CCT’, ‘Reject’, or ‘Unsure’. Main outcome
measures are performance of the crowd in terms of collective
classification accuracy, quantity screened, timeliness, and
screener recruitment, retention and engagement. Results:
To date (March 2015) over 950 people have signed-up to
take part in the project and over 120,000 citations have
been collectively screened by the crowd. The results for
four independent validation studies (two completed; two
ongoing) to assess crowd accuracy will be presented. The
two completed validations show crowd sensitivity of 99.8 and
99.9, and specificity of 99.8 and 99.7. The reference standard
used in both cases was determined by expert screeners.
Conclusions: This new approach to screening has brought
significant efficiencies to trial identification. Crowdsourcing
has proved to be both feasible in terms of recruitment and
robust methodologically. It has meant that more trials have
been identified more quickly, making CENTRAL a more valued
repository of trial reports.
O 10.2
A machine learning approach to classifying
articles for literature reviews based on titles
and abstracts
Steeves S1 , Buchanan-Hughes A1 , Leonard S1
1
Costello Medical Consulting Ltd, United Kingdom
Background: Machine learning could improve the efficiency of
conducting literature reviews by automating review of titles
and abstracts for relevant articles. Objectives: This study
investigated if it is feasible to use the Fisher classification
method to distinguish between relevant and irrelevant articles
for literature reviews based on their titles and abstracts,
and to determine if success varies depending on the type
of classification being performed. Methods: Datasets
were created from abstract lists from three systematic
reviews. To explore if the algorithm performed better
on particular types of classification (by study design, by
outcomes measured, by interventions used or by disease
area), decisions at various points of the eligibility flowcharts
Table 1
118
were tested separately. Articles were labelled as ‘relevant’
or ‘irrelevant’ at each of these stages. The datasets were
processed to remove duplicates and to adjust for imbalances
in ‘relevant’:‘irrelevant’ abstracts as a possible confounder.
Articles with only a title or only an abstract were retained.
Each dataset was divided into training (60%), cross-validation
(20%) and test sets (20%). Accuracy was measured using
classification accuracy and the F2 score which favours correct
classification of ‘relevant’ items. After training the classifier
algorithm, we optimised the F2 score on the cross-validation
set by adjusting the thresholds at which a ‘relevant’ or
‘irrelevant’ label was assigned. Items falling below these
thresholds were marked as ‘uncertain’ by the algorithm and
excluded from the F2 score calculation. The final F2 score
was calculated on the test set. Results: Classifiers were
trained on six datasets with final F2 scores varying from
0.663 to 0.879 (Table 1). Classifications by study type and
disease area outperformed those based on outcome measures
or interventions. Conclusions: The Fisher classification
method was successful at classifying relevant and irrelevant
articles based on their titles and abstracts, particularly for
classifications based on study design and disease area. We
intend to investigate the performance of other machine
learning algorithms on this task.
to complete, with fewer resources, are needed. Objectives:
To evaluate the performance of a new natural language
processing (NLP) algorithm. Methods: We developed a new
NLP algorithm based on diverse relevance ranking models
for MEDLINE citations. A linear combination of two ranking
scores from semantic relevance ranking and latent Dirichlet
allocations was used to predict an overall relevance ranking
score for each citation. To evaluate the performance of
this new method, we selected a convenience sample of five
SRs published by Cochrane. We estimated area under curve
(AUC), sensitivity, false-positive rate, total screening burden,
and percentage of reduction in screening. We compared
the new pooled effect size to the published one using the
Altman and Bland method. Results: The new NLP algorithm
achieved an average AUC of 0.82 (range: 0.49 to 0.95). With
70% reduction of the number of citations to be screened,
we observed over 80% sensitivity in four out of five SRs.
We did not find significant difference between the published
effect size and the new pooled effect size even after 90%
reduction of citations. Conclusions: NLP algorithms showed
promising results on accelerating the SR process and reducing
workloads. Future work is needed to expand the search
beyond MEDLINE and validate this pilot study.
O 10.4
O 10.3
Natural language processing to increase
efficiency of systematic reviews: a pilot study
Murad MH1 , Wang Z1 , Li D1 , Liu H1
1
Mayo Clinic, USA
Background: Systematic reviews (SRs) require significant
resources and time to complete. A typical SR takes between
six and 18 months. Interventions that can make SRs swifter
Evaluation of the priority ranking capabilities
of SWIFT (Sciome Workbench for Interactive,
Computer-Facilitated Text-mining) software
Walker V1 , Holmgren S1 , Thayer K1 , Rooney A1 , Macleod M2 , Currie
G2 , Sena E2 , Sherratt N2 , Rice A3 , Howard B4 , Shah R4 , Pelch K1
1 National
Institute of Environmental Health Sciences
(NIEHS)/National Institutes of Health (NIH), USA; 2 Centre for Clinical
Brain Sciences, University of Edinburgh, Scotland; 3 Department of
Surgery and Cancer, Imperial College, England; 4 SciOme LLC, USA
Table 1
119
120
Background: There is growing interest in assessing
the ability of machine learning approaches to priority
rank studies as a way to reduce the human burden in
screening literature when conducting a systematic review.
Objectives: To assess the performance of Sciome Workbench
for Interactive, Computer-Facilitated Text-mining (SWIFT)
priority ranking algorithm to identify studies considered
relevant based on manual screening. Methods: Four
case studies representing a range of complexity and
size were used to assess the performance of SWIFT: (1)
transgenerational inheritance of disease, (2) bisphenol A (BPA)
and obesity (3) perfluorooctane sulfonate/perfluorooctanoic
acid (PFOS/PFOA) and immunotoxicity, and (4) neuropathic
pain. For each case study two independent reviewers
manually screened results to determine study relevance
and identify test sets of 30 to 400 included and excluded
references. The test sets were used to priority rank the
literature search results in SWIFT for relevance using an
algorithm that considers term frequency (title, abstract, MeSH
headings and SuppChem annotations) and Latent Dirichlet
Allocation (LDA) topic modeling. This ranking was evaluated
with respect to (1) number of studies that needed to be
screened in order to identify 90% and 95% of known relevant
based on manual screening, and (2) the ’Work Saved over
Sampling’ (WSS) performance metric, which defines, for a
specific level of recall, the percentage reduction in effort
achieved by a ranking method compared to a random ordering
of the documents. Results: For all four datasets, using 100
training examples and LDA topic modeling, the prioritization
procedure reduces the number of citations that must be
screened to achieve a recall rate of 90% (Table 1) by 50% or
more. For the more stringent recall rate of 95%, the range in
the number of citations screened was reduced by 44%, for the
neuropathic pain dataset, to 80%, for the PFOS/PFOA dataset.
The greatest increases in screening efficiency were observed
in the more targeted topics. Conclusions: Text-mining and
machine learning programs such as SWIFT can be valuable
tools to reduce the human screening burden.
Oral session 11
Rapid reviews and overviews
of reviews
O 11.1
An evidence map of studies evaluating
methods for conducting, interpreting, and
reporting overviews of systematic reviews
Lunny C1 , McKenzie J1 , McDonald S1 , Lunny C1 , Brennan S1 ,
Lunny C2
1
Australasian Cochrane Group, Australia; 2 Australasian Cochrane
Group, Monash University, Australia
Background: Methods for conducting, interpreting, and
reporting overviews of systematic reviews are in their infancy.
Where possible, guidance for overviews should be based on
methods that have been evaluated and shown to have better
performance. It is timely to map the methods literature
on overviews, so as to determine what methods research
should be undertaken as a priority. Objectives: 1. To
populate a framework with methods that have, or may be
used, in conducting, interpreting, and reporting overviews
of systematic reviews. 2. To create an evidence map
of studies that have evaluated these methods. Methods:
The research will be undertaken in two stages to meet
objectives 1. and 2. We plan to search methods-specific
databases (Cochrane Methodology Register, Meth4ReSyn
library, AHRQ Effective Health Care Program), in addition
to a general search for overviews in MEDLINE. Stage I:
Methods that have been used in overviews will be identified
from descriptive cross-sectional studies on methods used
in overviews; guidelines; and editorials. We will extract
data on: the type of paper (descriptive cross-sectional
paper; guideline; editorial); description of the method;
noted advantages/disadvantages; and where the method
is located in the review process. Stage II: Evidence of the
performance of methods will be identified from systematic
reviews of methods studies, and methods studies. Evaluations
will be described narratively, and mapped to the framework
of methods identified in stage I. Results: The results will be
presented at the Colloquium. Conclusions: The results of
this mapping process will be useful for cataloguing overview
of systematic review methods, informing guidance, and
identifying and prioritising methods research in this field.
O 11.2
Rapid review programs to support healthcare
and policy decision making: a descriptive
analysis of processes and methods
Polisena J1 , Garritty C2 , King V3 , Stevens A2
1 Canadian Agency for Drugs and Technologies in Health, Canada;
2
Ottawa Hospital Research Institute, Canada; 3 Oregon Health
Sciences University, USA
Background: Healthcare decision makers often make
decisions under limited timeframes that preclude the
completion of more comprehensive evidence synthesis. Rapid
reviews (RRs), using streamlined systematic review methods,
are used frequently for evidence synthesis to support such
decisions. Objectives: Our primary objective was to
describe the processes and methods used across international
programs to produce RRs. We also sought to understand
the underlying themes associated with these programs and
to identify research opportunities in rapid review methods
and impact. Methods: We contacted representatives of
healthcare RR programs to inquire about their methods and
processes. The characteristics of each program’s processes
and methods were summarized and compared to highlight
potential themes and trends related to the practice of RRs.
In addition, knowledge gaps to inform research priorities
in RRs were identified. Results: Twenty-nine RR programs
representing academia, government, research institutions,
121
and not-for-profit organizations participated in our survey.
Responses revealed that the main objectives for RRs were to
inform decision making on funding healthcare technologies,
services and policy, and program development. Central
themes influencing the process and methods used were
timelines, available resources, the complexity and sensitivity
of the research topics, and proprietary nature of the product.
Conclusions: Observed differences in processes and methods
across programs may result from the continuous development
of RR methods, customization of RRs for decision makers, and
definition of ‘rapid’ by organizations. The primary research
priority is to develop a typology of RRs to understand the
strengths and limitations of various RR forms better and
the level of synthesis performed. Future research also should
investigate the impact of RR methods and reporting to support
informed healthcare decisions, the effects of potential biases
that may be introduced with streamlined methods, and the
effectiveness of RR reporting guidelines on transparency.
We selected 32 pairs of documents. The preliminary analysis
showed no serious mismatching. The final results will be
presented at the Colloquium. Conclusions: The timeframe
to produce evidence is becoming shorter. It is critical to
determine if ultra-rapid evidence summaries produced by
HTA trained teams are reliable.
O 11.3
Background: Treatments that are matched to patient risk
(stratified care) have the potential to improve effectiveness
of primary care for patients with musculoskeletal pain.
However musculoskeletal pain conditions are extensive and
the knowledge base is large. To inform development of
matched treatments a rapid yet detailed evidence summary
on effectiveness of available treatment options was needed.
Objectives: To develop an approach to synthesizing large
evidence summaries; rapidly to synthesise and appraise
current best evidence on treatment options for five most
common musculoskeletal pain presentations in primary care
and to summarise the available evidence on treatments for
patient risk subgroups using stakeholder groups. Methods:
Evidence synthesis followed a pyramidal approach using
national clinical guidelines, policy documents, clinical
evidence pathways and summaries as starting point.
Recommendations on available treatment options for
shoulder, neck, knee, back and multisite pain were extracted
consecutively.
Systematic searches of bibliographic
databases were conducted to identify and retrieve additional
published trials that had not yet been summarised or where
evidence gaps existed. Quality of evidence was assessed
based on modified GRADE quality ratings and strength of
evidence. Evidence summaries were subsequently presented
to stakeholders (including health service managers,
clinicians and researchers) for interpretation and to identify
appropriate treatment options that might be matched to
patient risk subgroups. Results: Via a rapid yet systematic
and comprehensive approach, pragmatic summaries of the
evidence base on treatment options for five musculoskeletal
pain presentations were completed. Based on current
best evidence, identification of matched treatment options
according to patient risk subgroups appears feasible across
musculoskeletal pain presentations. Conclusions: Effective
healthcare delivery and clinical practice depend on high
quality evidence which provides depth as well as breadth of
coverage. A novel and pragmatic approach to rapid synthesis
of large evidence is explored and found applicable within
musculoskeletal pain field.
Are 3 days enough to capture the key
evidence for HTA documents?
Ciapponi A1 , Bardach A1 , Glujovsky D1 , Rey Ares L1 ,
Garcı́ a Martı́ S1
1
Instituto de Efectividad Clı́ nica y Sanitaria - IECS, Argentina
Background: Rapid reviews have emerged as an efficient
approach to synthesizing evidence, within a four to five
week timeframe, for informing decision makers in health
care settings. It is uncertain whether even very shorter
timeframes that are needed sometimes in specific healthcare
decisions, e.g. two to three days elaboration-time, would
still be adequate for capturing the key evidence that forms
part of more elaborate Health Technology Assessment (HTA)
documents. Objectives: To compare the conclusions and
analyze the amount and direction of the evidence included
in HTA documents produced in an ultra-rapid way compared
to the more normal four to eight weeks. Methods: IECS
is an Argentinean HTA agency that produces both of the
aforementioned types of documents according to the urgency
and needs of decision-makers. The documents are based
on focused search strategies in meta-search engines and
online biomedical literature databases, to identify systematic
reviews, clinical practice guidelines, HTAs, coverage policies,
and selected primary research.
The ultra-rapid HTAs
are prepared by highly-trained staff who select the most
important evidence according to their own judgement. The
’slower’ HTAs allow a more exhaustive assessment of the
evidence. We selected pair of documents, one done in two
days and the other in five weeks, oriented to the same research
question. The longer document needed to be published
within a year of the shorter one. The additional evidence
identified by the newer document, which was compiled
at a later date than the ultra-rapid HTA, was excluded
and the conclusions modified wherever necessary. Pairs
of independent researchers extracted the outcomes, and
disagreements were solved by a third researcher. Results:
O 11.4
Effective treatment options for
musculoskeletal pain conditions: a rapid
meta-synthesis of current best evidence
within primary care
Babatunde OO1 , Jordan JL1 , Van der Windt DA1 , Foster NE1 ,
Protheroe J1 , Working Group S1
1 Research Institute for Primary Care and Health Sciences, Keele
University, Keele, Staffordshire, ST5 5BG, United Kingdom
122
Oral session 12
Prediction models
O 12.1
CHecklist for critical Appraisal and data
extraction in systematic Reviews of clinical
prediction Modelling Studies (CHARMS)
Moons K1 , de Groot J1 , Bouwmeester W1 , Vergouwe Y1 , Mallett
S2 , Altman D2 , Reitsma J1 , Collins G2
Netherlands; 2 University of Oxford, United Kingdom
Background: Publications on multivariable clinical prediction
models have become abundant for both prognostic and
diagnostic purposes. Systematic reviews of these studies
are increasingly required to identify and critically appraise
the existing evidence. There is currently no checklist or
tool providing guidance for systematic reviews of studies
developing or validating prediction models that can assist
reviewers to define the review objectives and appraise
study methodology. Objectives: To develop a checklist
to help reviewers framing a well-defined review question,
and to determine which details to extract and critically
appraise from primary studies on the development or
validation of multivariable diagnostic or prognostic prediction
models, with a view to assessing the risk of bias and
sources of heterogeneity. Methods: We critically examined
existing reporting guidelines and quality assessment tools,
key methodological publications on clinical prediction
modelling, and tools used in published systematic reviews
of multivariable prediction models, to identify the relevant
characteristics and domains. The checklist was tested in
various systematic reviews. Results: We identified seven
items important for framing the review question (diagnostic
versus prognostic model, intended scope of the review, type
of prediction modelling studies, target population, outcome
to be predicted, time span of prediction, intended moment
of using the model), and 11 domains to critically appraise
the primary included studies (source of data, participants,
outcome, predictors, sample size, missing data, model
development, model performance, model evaluation, results,
interpretation). Both were combined into the CHecklist for
critical Appraisal and data extraction for systematic Reviews
of prediction Modelling Studies (CHARMS). Conclusions:
CHARMS is designed to assist reviewers to help systematic
reviewers when framing their review objectives, and to
determine which data to extract and critically appraise from
primary studies on the development and/or validation of
(diagnostic and prognostic) prediction models.
O 12.2
Individual Participant Data (IPD)
meta-analysis of diagnostic and prognostic
modeling studies: a practical introduction to
their rationale and conduct
Debray T1 , Riley R2 , Rovers M3 , Reitsma J1 , Moons K1
1
Julius Center for Health Sciences and Primary Care, The
Netherlands; 2 Keele University, United Kingdom; 3 Radbound
University Nijmegen Medical Center, The Netherlands
Background: The development and (external) validation
of diagnostic and prognostic prediction models is an
important area in contemporary medical research. During
the past few years, evidence synthesis and meta-analysis of
individual participant data (IPD) has become increasingly
popular, not only for intervention research but also for
improving the development, validation and generalizability
of diagnostic and prognostic prediction models. IPD metaanalyses (IPD-MA) provide unique opportunities to improve
development and enhance the applicability of prediction
models across (sub)populations and settings. There is,
however, little guidance on how to conduct an IPD-MA to
develop and validate diagnostic and prognostic prediction
models, or how to interpret their findings. Objectives: To
describe how IPD-MA of diagnostic and prognostic modeling
studies differ from IPD-MA for assessing treatment effects.
Methods: We identify key advantages and challenges in
IPD-MA of prediction models. Subsequently, we provide
recommendations for the design of such IPD-MA including the
selection of relevant studies, Finally, we discuss statistical
methods for handling between-study heterogeneity, missing
data, and other issues regarding prediction model
development and validation. We illustrate all these concepts
using various empirical examples across medical disciplines.
Conclusions: The guidance provided in this work may help
meta-analysts in prediction modeling research to decide upon
appropriate strategies when conducting an IPD-MA, and assist
readers, reviewers and practitioners when evaluating the
quality of resulting evidence.
123
124
O 12.3
Imputation of missing adjusted data for
non-significant factors in meta-analysis
of prognostic studies
Wang L1 , Kennedy S2 , Romerosa B3 , Kwon H4 , Kaushal A2 , Craigie
S1 , Chang Y2 , Almeida C5 , Izhar Z2 , Couban R1 , Parascandalo S6 ,
Guyatt G2 , Reid S2 , Khan J2 , McGillion M2 , Busse J1
1 Michael G DeGroote Institute for Pain Research and Care, McMaster
University, Canada; 2 McMaster University, Canada; 3 Complejo
Hospitalario de Toledo, Spain; 4 Wayne State University, USA;
5 Federal University of Rio Grande do Sul -UFRGS, Brazil; 6 University
College Cork, Republic of Ireland
Background: Many prognostic studies use data-driven
models: each independent factor is tested in a bi-variable
analysis, and only those that show evidence of association
(e.g. P value ≤ 0.05) are entered into an adjusted model.
Others report data only for those independent factors in
their adjusted model that show a significant association
with the dependent variable. Systematic exclusion of
these data presents a risk of overestimation by only
pooling estimates of association from predictors that appear
in adjusted regression models, and for whom data is
provided. Objectives: To investigate if imputation of
missing non-significant data in final regression model will
avoid overestimation of the predictive power of the risk
factors, using predictors for persistent pain after breast
cancer surgery as an example. Methods: We pooled nine
predictors to explore their association with the development
of persistent pain following breast cancer surgery using
random-effects models. We imputed an odds ratio (OR) of
‘1’ for predictors that were excluded from adjusted analyses
due to non-significant bi-variable analyses, or that were
reported, but with no data due to lack of significance in the
final regression model. We acquired the associated variance
for all such imputations using the hot deck approach. We
performed sensitivity analysis to examine the impact of
imputing data for non-significant predictors excluded from
adjusted analyses by re-running our analyses and excluding
the imputed data. Results: Fifty-nine (51.3%) out of 115
study-sets for nine predictors failed to reported the adjusted
data for non-significant predictors. Our sensitivity analyses
found no significant differences in results whether or not
we incorporated missing data for non-significant predictors
125
(Table 1). However, the associations of the nice predictors
with persistent pain were consistently larger in meta-analyses
based on the adjusted data only than in the full analyses
including imputed missing data. Conclusions: Imputation
of missing data for non-significant predictors did not cause
any significant associations to lose significance, but the
magnitude of association was reduced.
O 12.4
Are predictions from test accuracy
meta-analyses valid in practice?
invalid a tailored estimate may be appropriate (Table 1).
Furthermore the differences in estimates between the models
could potentially affect patient management. Conclusion:
Statistical validation of a test’s PPVs and NPVs could be part
of the synthesis process of a test evaluation meta-analysis.
Both the likely validity of the standard and tailored estimate
in a new population could be ascertained at this stage. In the
examples analysed, standard meta-analysis seldom yielded
a valid estimate for the test’s performance. Importantly, this
was often remedied by taking a tailored approach to the
meta-analysis, which was more likely to yield a valid estimate.
Willis BH1 , Riley RD2
University of Birmingham, United Kingdom; 2 University of Keele,
United Kingdom
Global health: Helping with evidence
Background: Although meta-analysis may synthesize
estimates for a test’s accuracy, heterogeneity often blights
these and it is not always clear whether clinicians should trust
these estimates when applying them to their own practice.
Objectives: Apply a novel statistical method to determine
whether the summary estimates of a test’s positive and
negative predictive values (PPV and NPV) are likely to be
valid in practice. Methods: Using four test accuracy reviews
as examples, a univariate meta-analysis model was used to
derive estimates for the likelihood ratios. Based on previously
reported methods this model was extended to produce a
tailored estimate specific to the setting of interest. The PPV
and NPV for the tests were the main outcomes of interest.
A new validation statistic, Vn is introduced that assesses
the validity of the standard and tailored estimate for the
PPV and NPV in a new population. This is derived using
a cross-validation procedure that compares the observed
and expected post-test predictions in an omitted study on
multiple occasions. A significant Vn (P value < 0.05) suggests
the estimate would not be valid in a new population. Results:
For the four examples, each meta-analysis model derived
four PPV estimates and four NPV estimates. Using the
standard meta-analysis model, Vn was significant in seven
of the eight estimates. In contrast, for tailored meta-analysis,
Vn was significant in only three of the eight estimates
suggesting that when a standard estimate is likely to be
RO 19.1
1
From Dublin to Vienna - the Cochrane Waltz
Larun L1 , Gundro Brurberg K1 , Odgaard-Jensen J1
1
Norwegian Knowledge Centre for the Health Services, Norway
Background: The time from protocol to review was around
two years (median) in 2014. Cochrane Review authors have
access to online resources, workshops and courses in addition
to the Cochrane Handbook and the group’s specific resources
and a Cochrane Author Support Tool to be published. This
should enable the review authors to waltz through the review
process (Waltz definition 1: to succeed easily or breeze
through). Why is it that some authors feel they are stuck
in a true Viennese waltz (Waltz definition 2: a complex dance
that consists only of multiple turns and changes in step)?
Objectives: How to keep review authors in the loop?
Methods: Case study including three professional systematic
review authors experienced in writing Cochrane Reviews.
Data collected from Archie, emails and personal memory
analysed narratively. Results: Review authors are a vital
part in the production of Cochrane Reviews and we should all
take care that they feel valued. Review authors are asked to
submit revised versions within three months and this should
apply to editorial processes as well. We should consider
126
specific editorial support for non-native English speakers to
facilitate clear writing. Review authors should be made
aware of likely timelines in both absolute months and elapsed
time from title to protocol and protocol to review to set
clear expectations. Conclusions: Review authors, like users,
consumers or customers might have valuable insight into
pitfalls and enhancers for how to keep reviewers and how
to get people on board. Review authors should always be
informed about the viability of producing a Cochrane Review
in relation to existing workload so they can make informed
decisions.
RO 19.2
Cochrane protocols: strategies for improving
production
Marcus S1 , Noel-Storr A1
1
Dementia and Cognitive Improvement Group, United Kingdom
Background:
Protocols describe the rationale and
methodological framework for a subsequent systematic
review and must be published before the review. Whilst
they usefully describe the need to answer a valid research
question, they are time consuming for authors and can take
several months or even years to publish. The peer referee
and editorial process delay publication further. Objectives:
1. To assess protocol generation times within the Cochrane
Dementia Group. 2. Trial new method/s in protocol production
within the context of a large NIHR funded programme grant.
3. Propose other possible models of protocol production.
Methods: 1. We performed a cross-sectional analysis based
on data in Archie of our ten most recent intervention protocols
to see how long they took from registration to publication. 2.
For a suite of 12 reviews on Modifiable Risk Factors we used a
generic protocol written in-house with more input than usual
from the editorial base –(i) consultant methodologist; (ii)
designated systematic reviewer (iii) greater use of our group’s
specialist/contact editors; (iv) one peer referee. Results:
1. Times taken from registration to publication varied from
32 to 327 weeks. Mean time: 109 weeks. 2. We will
report on time taken from registration to first draft, and
to publication. We will report what worked well and what
remained a challenge. Conclusions: Both methodologies
are time consuming, but option 2 has potential for reducing
registration to publication time. However a third option
might be to publish the protocol as an appendix to the full
review. This could incorporate method 2 above, obviating the
need for sequential publication. It could considerably reduce
time from registration to review publication by reducing the
protocol editorial process. Additionally a generic protocol
could increase consistency across reviews.
RO 19.3
A new approach to ‘empty’ reviews
Kew KM1 , Welsh EJ1
1
Airways, United Kingdom
Background: Cochrane review groups are encouraged to
prioritise reviews that are likely to include studies over
those that might not, but questions that identify no trials
can be important to people making healthcare decisions.
‘Empty’ reviews based on comprehensive literature searching,
especially those that have been identified in prioritisation
exercises, can confirm and highlight important research gaps
and inform the design of future trials. Within Cochrane
‘empty’ reviews are subject to the same authoring and
editorial requirements as reviews with included studies. This
conflicts with the founding principle of Cochrane and the
strategy to 2020 which encourages prudent use of resources.
Objectives: To develop and evaluate new approaches to
‘empty’ reviews Methods: We will collect data about the
time taken for protocol and review development of recent
‘empty’ Airways Group reviews. We will also look at our review
proposals’ log to assess the number of times titles were
deemed clinically important but rejected because no trials
were anticipated. Results: We will put forward alternative
approaches to ‘empty’ reviews. Our criteria will be: 1.
maintains methodological integrity up to the point that a
review can be confirmed empty; 2. minimises the workload
for authors, editorial staff, peer reviewers and copy-editors; 3.
publicises an important research gap quickly and succinctly
to funders and researchers. The approaches we describe
aim to prevent research waste and are not intended for
non-priority questions. Conclusions: ’Empty’ reviews for
important research questions can be a useful tool to confirm
and publicise research gaps.
RO 19.4
Stop press! Experience of managing serious
complaints about a Cochrane Review
Armstrong S1 , Jordan V2 , Farquhar C2
1 Cochrane Menstrual Disorders and Subfertility Group, United
Kingdom; 2 Cochrane Menstrual Disorders and Subfertility Group,
University of Auckland, New Zealand
Background: Serious complaints about Cochrane Reviews
are rare, but can pose a challenge to authors as to how
best to respond. Cochrane has a webpage dedicated to
submitting a complaint, however it doesn’t publish guidance
on how to address complaints. Should authors seek advice
beyond the scope of the review? Should complaints and
responses be published? How much credence should authors
give critics who have an obvious but undeclared conflict of
interest? Objectives: To offer a personal perspective on
managing serious criticism of a Cochrane Review. Methods:
An account of how a group managed a series of complaints
about a Cochrane Review. Results: The initial complaint
127
letter contained diverse criticisms including the title, the
primary outcome and the use of the term ‘cell-tracking
algorithms’ amongst others. There was agreement that we
had robust evidence to defend our protocol and to not make
any changes. We sought advice from an expert researcher
and clinician in embryology, who supported our conclusions.
In the meantime, the full review had been published, and we
decided to republish the review with both letters appended. A
second letter was received one month later. It condemned the
use of the intention-to-treat principle, the use of miscarriage
per randomized woman and the heterogeneity of definitions
of miscarriage between studies. This letter concluded with
the demand for the review to be withdrawn and republished
under different authorship. Again, we sought expert advice
and referenced statistical advice from the Cochrane Menstrual
Disorders & Subfertility Group and CONSORT. A reply was sent
that defended our methodological decisions. An internet
search of the critics revealed that they both have undisclosed
links with the time-lapse imaging industry. Conclusions:
Seeking advice from methodological, statistical and content
experts is useful to help clarify the complaints raised and
determine how to respond. Commercial conflicts of interest
may not be disclosed by critics, but it is important to respond
equitably and to publish all correspondence with the review.
RO 19.5
Discernment of academic conflicts of interest
for Cochrane Systematic Reviews
Miyazaki C1 , Ota E1 , Mori R1 , Sasaki H1 , Kirkham J2 , Dwan K3
1 Department of Health Policy, National Center for Child Health and
Development, Japan; 2 The University of Liverpool, UK; 3 Cochrane
Editorial Unit, UK
Background: Cochrane has set a ground policy to ensure
that its systematic reviews are free of conflict of interest
(COI) and to prevent potential influence from commercial
sponsors or sources. Although Cochrane’s declaration of
interest focuses largely on avoiding potential financial COI,
guidance on academic COI remains ambiguous. Health policy
promotion occurs through gathering evidence of healthcare
interventions from systematic reviews, and involves clinical
experts and consensus from academic conferences. The
extent to which the evidence is influenced by academic
and political interest, and the career driven incentive of
the authors, is not known. Apart from disclosures of a
financial nature, academic interest may be the driving force
for drawing a conclusion. There are many research groups
and institutes that have different standards in their code of
conduct policy, therefore, there is an implication of a lack of
consistency in practising the elimination of non-financial COI.
Objectives: To conduct a survey of the existing guidance on
COI management within the 54 Cochrane Review Groups and
to characterize academic COI. Methods: An internet-based
survey will be used to acquire any statement of academic
COI policy from the 54 Cochrane Review Groups. The survey
will identify whether a comprehensive definition of academic
COI exists. Review groups without a disclosed academic COI
will be selected for assessment, by relating review authors
to authors of included clinical trials in published Cochrane
Systematic Reviews. Results: The results will show how
many Cochrane Systematic Review authors have conducted
their own clinical studies and the Cochrane Review Groups in
which affiliations are common. Conclusions: This research
will present the underlying concern about the need to include
independent people in review teams to exercise avoidance of
academic COI, retain impartiality in the outcome judgments
of the reviews and to promote well-balanced review teams.
RO 19.6
Selective reporting of outcome data: are we
following the Cochrane Handbook guidance?
Rutjes AW1 , Maione A1 , Silletta MG1 , Di Nisio M1
1 Centre for Systematic Reviews, Fondazione Universita’ ‘‘Gabriele
d’Annunzio”, University of Chieti-Pescara, Italy
Background: The Cochrane Handbook advises to judge
the risk of bias (RoB) potentially introduced by selective
reporting of outcome data (SORB) on the within-trial level,
while judging if the outcomes expected to be reported in
a given field are addressed. Objectives: To summarize
and discuss applied methods to judge and handle RoB due
to selective reporting in Cochrane Reviews. Methods: We
obtained unique reviews from the Cochrane Musculoskeletal
Group (CMSG) and the Cochrane Peripheral Vascular Diseases
Group (PVD) published in 2011 to 2015, including at least
1 RCT. Extraction items included definitions used to judge
the RoB by selective reporting, the frequency of scoring
low, high or unclear RoB due to selective reporting, and
the handling of this item in the ‘Summary of findings’
tables (SoF). Results: Complete results will be presented
at the Colloquium. Up to 2013, 162 reviews were identified,
of which 42 PVD and 24 CMSG reviews were considered.
Figure 1 shows that 83% of the reviews considered SORB.
Of these, the majority of reviews referred to the Cochrane
Handbook for the operationalization of how to assess SORB
Figure 1
128
Table 1
(Table 1). Most reviews assess SORB at the within-trial level,
matching outcomes reported in protocols, trial registrations
and methods sections to outcomes addressed in the results
section. Only a minority (9%) followed the Handbook guidance
to assess SORB at the within-trial and topic level; 17% focused
on completeness of reporting of review-defined outcomes
(review outcome level). SoF tables were provided in 44% of
the included reviews (Figure). SORB was infrequently (15%)
mentioned as a reason to downgrade the confidence in the
estimate of the effect. Conclusions: Cochrane guidance on
SORB is insufficiently followed by CMSG and PVD reviewers.
SORB is typically assessed on the within-trial level, without
considering whether at least the main outcomes relevant to
the topic or review question are addressed across trials. SORB
seems to be given insufficient attention while GRADE-ing
our confidence in the estimates of the effects. We suggest a
stakeholder meeting to evaluate and discuss hurdles to assess
SORB and to improve methods.
RO 19.7
Developing core competencies for scientific
editors of biomedical journals
Galipeau J1 , Moher D2 , Shamseer L2 , Barbour V3 , Bell-Syer S4 ,
Cumpston M5 , Deeks J6 , Garner P7 , MacLehose H8 , Straus S9 ,
Tugwell P10 , Wager E11 , Winker M12
1
Ottawa Hospital Research Institute, Canada; 2 Ottawa Hospital
Research Institute; University of Ottawa, Canada; 3 Committee On
Publication Ethics (COPE), Australia; 4 Cochrane Wounds Group,
UK; 5 Cochrane Central Executive, Australia; 6 Cochrane DTA
Working Group, UK; 7 Cochrane Infectious Diseases Group, UK;
8
Cochrane Editorial Unit, UK; 9 University of Toronto, Canada;
10
Cochrane Musculoskeletal Group, Canada; 11 Sideview, UK;
12 World Association of Medical Editors (WAME), USA
Background: Scientific editors play a key role in the
assessment, acceptance, and dissemination of high quality
reports of medical research. However, as a group they are
generally not well educated in their position. This may be,
in part, due to a lack of evidence of what makes a good
scientific editor or how to train them properly. The result
is that there are no standard criteria for determining one’s
suitability for the position or training needs; in short, we don’t
know what competencies are essential for effective scientific
editing. Objective: The objective of this project is to develop
a globally accepted minimum set of core competencies for
scientific editors (i.e. those responsible and accountable for
selecting the scientific content of a peer-reviewed biomedical
journal), including Cochrane editors. Methods: Phase 1
involves searching published and unpublished literature for
information relating to competencies of scientific editors;
the output will be a scoping review. Phase 2 consists of a
training needs assessment to ascertain the needs of scientific
editors from diverse regions and varying journal types and
sizes. Phase 3 is a Delphi process open to all scientific editors
globally; the output will be a list of competencies deemed
important for performing the role of scientific editor. Phase
4 features a consensus meeting to achieve agreement on a
minimum set of competencies for scientific editors. Phase 5
involves the development of a globally accessible curriculum,
training modules, and certification program. Discussion: We
will provide an update on our progress and will share available
results from the first three project phases. The goal of the
project is to provide enhanced credibility and consistency to
the role of scientific editor across the spectrum of biomedical
journals. Funded in part by the Cochrane Central Executive
as a high priority issue, the project is international in scope
and includes several key stakeholder groups, such as the
World Association of Medical Editors (WAME), the Committee
on Publication Ethics (COPE), and Cochrane.
RO 19.8
Updating patient information: a literature
review
Wegmann M1 , Will R1 , Ehrlich M1 , Janßen I1
1 IQWiG, Germany
Background: The IQWiG produces evidence-based health
information (HI) that is updated every three years. Maintaining
a fixed update cycle requires a major investment of resources
and does not always appear to provide the most suitable
timeframes for some of the information. Objectives: (1)
Which solutions does the current literature offer for updating
HI? (2) How do other international organisations deal with this
issue? (3) Are there recommendations or suggestions that can
be meaningfully applied for our purposes?
Methods: A systematic search of the literature accompanied
by supplementary handsearch was performed.
Any
129
publications offering data relevant to our objectives were
included. The search strategy design was created by two
reviewers and the subsequent search was carried out by
the same two reviewers. Two people independently viewed
the titles and abstracts. Relevant titles were ordered as
full articles. One reviewer extracted the data from these
articles with a second checking the work. Organisations also
involved in the production of HI were contacted via email.
Results: The search produced 2324 hits. Following title
and abstract screening 52 references remained. Following
full-text assessments, 25 references were included. Six of
the organisations we contacted responded to our request.
Conclusions: (1) The current literature does not provide any
suggestions concerning optimal update cycles or strategies
for HI. (2) Interviews with the organisations showed that
they have not implemented a standardised procedure for
updating, nor do they have a fixed cycle for updates. Some
of the organisations use ’horizon scanning’. (3) The literature
identified confirms that regular updates are necessary and
that marking information with a recent date sends an
important message to the reader. The use of statistical
methods to calculate the influence of new study results on
meta-analysis effect estimators is highly time-consuming.
Nevertheless, this would be a way of identifying HI with stable
effects that would remain up-to-date longer. New techniques
such as semi-automated title and abstract screening or
abbreviated searches may be viable for our updating process.
249 recommendations and supporting evidence summaries
for antithrombotic therapy, we performed user-testing with
12 physicians in Norwegian hospitals. Our multidisciplinary
team of designers, programmers, clinicians and researchers
applied qualitative methods with both direct observation
and semi-structured interviews and performed iterative
improvements in functionality of MAGICapp. Results: We
implemented improvements in MAGICapp related to the
following main issues in accessibility: delay from login
to accessing content (e.g. more than three minutes on
old browsers on hospital desktop stations); need for a
comprehensive disclaimer for use in practice, suboptimal
display of recommendations and PICO questions (e.g.
physicians did not intuitively find where to click to access
deeper layers of information); poor scrolling functionality
on mobile devices; too much screen space used for less
relevant information (e.g. table of contents); need for better
offline access, and search functionality for recommendations.
Problems with understanding and utility of content related to:
conceptual understanding of GRADE (e.g. quality of evidence,
risk and uncertainty), wording being sometimes imprecise
or technical, incomplete access to references and abstracts.
Conclusions: The barriers identified illustrate challenges and
solutions for getting evidence into practice through innovative
technological platforms.
Disseminating evidence
RO 19.9
Getting best current evidence into practice
through a web-based authoring and
publication platform: qualitative study of
technological barriers and solutions
Brandt L1 , Kristiansen A1 , Agoritsas T2 , Fog Heen A1 , Guyatt G3 ,
Vandvik PO4
1
Department of Medicine, Innlandet Hospital Trust-division Gjøvik,
Norway; 2 Department of Clinical Epidemiology and Biostatistics,
McMaster University, Canada, Canada; 3 Department of Clinical
Epidemiology and Biostatistics, McMaster University, Canada;
4 Institute for Health and Society, Faculty of Medicine, University of
Oslo, Norway
Background:
Trustworthy evidence summaries and
guidelines are key information resources for clinicians.
Digital structuring of content and optimization of
presentation formats for a variety of digital platforms
can facilitate successful dissemination and use at the
point of care.
Objectives: To test and improve
the accessibility, understandability and utility in meeting
clinicians’ information needs of evidence summaries and
recommendations published in multilayered formats on the
web, smartphones and tablets. Methods: We have created
a platform (www.MAGICapp.org) to author, publish and
update dynamically trustworthy evidence summaries and
guidelines developed with the GRADE system. Following
publication of the first guideline through MAGICapp with
RO 20.1
The development of an evidence based
chronic diseases management resource for
health practitioners to use at the point of care
Khalil H1 , Chambers H1 , Munn Z2 , Porritt K2
1
Monash University, School of Rural Health, Australia;
University of Adelaide, Australia
2
The
Background: There is a large gap between evidence and
practice within health care, particularly within the field
of chronic disease. To reduce this gap, and improve the
management of chronic disease, a collaborative partnership
between two schools within a large university and two
industry partners (a large regional rural hospital and a rural
community health centre) was developed to research the area.
Objectives: The aim of the collaboration was to promote the
development of translation science and the implementation
of evidence-based health care in chronic disease with a
specific focus on developing evidence-based resources that
are easily accessed by clinicians. Methods: A working group
consisting of members of the collaborating organisations
and an internationally renowned expert reference group
was formed. The group acted as a steering committee
and were tasked to develop a taxonomy for the resources.
In addition, a peer review process of all resources was
established. A corresponding reference group, consisting of
researchers and clinicians who are clinical experts in various
130
fields, was involved in the review process. The resources
developed by the group include evidence summaries and
recommended practices made available on a web-based
database and accessed via subscription by clinicians and
researchers worldwide. Results: As of early 2015, there
were 147 new evidence summaries and 30 recommended
practices detailing the best available evidence on topics
related to chronic disease management including asthma,
diabetes, heart failure, dementia and others. Training
sessions and a newsletter have also been developed for
clinicians within the node to enable them to use the content
effectively. Conclusions: The successful development of the
collaborative partnership and its evolution into producing a
valuable resource for the translation of evidence into practice
in the areas of chronic disease management will enable
clinicians to access updated, pre-appraised summaries at the
point of care.
RO 20.2
Disseminating good evidence for Cochrane
Systematic Reviews: practice and experience
from China
Li X1 , Yang G1 , Liang N1 , Wieland S1 , Liu J1
1
Centre for Evidence-Based Chinese Medicine, Beijing University of
Chinese Medicine, China
Background: In 2007, the Centre for Evidence-Based Chinese
Medicine from Beijing began training volunteers to identify
and translate citations and abstracts of (non)randomized
controlled trials (RCTs/CCTs) published in Chinese journals
to the CAM Field for submission to CENTRAL. In 2014, we
started exploring translation of Cochrane Review abstracts
and plain language summaries (PLS) into simplified Chinese
to contribute to Cochrane’s translation strategy. We introduce
our work to share our experiences in translation, recruiting,
training, and managing translators in the hope of generating
discussion about the skills for these projects. Translation of
Chinese RCTs/CCTs: In the past eight years, we have enrolled
and trained over 80 volunteers from Beijing University of
Chinese Medicine to translate Chinese RCTs/CCTs, and have
submitted 11,405 translated records to CAM Field. There
are currently 15 active volunteers, and we plan to submit
another 1500 records in 2015. The RCTs/CCTs are mostly
from electronic searching for specific topics, and many are
from the included/excluded studies in systematic reviews
conducted by our centre. The volunteers are mainly 3rd,
4th and 5th year undergraduates recruited annually by email
and orally. We train volunteers in software (ProCite) use,
translation principles, and methodology. Translation of
Cochrane Review abstracts/PLS: We have trained 30 centre
members and submitted abstracts/PLS for three Cochrane
Reviews as an initial practice in coordination with Cochrane’s
Translations Coordinator, while the Chinese Cochrane Centre
decides overall strategy. A list of reviews are being translated
using Smartling, and we are recruiting and training more
volunteers. Conclusions: Since 2007 we have successfully
recruited and trained volunteers to assist in translating
projects with Cochrane. The type of volunteer and training
needed varies across projects. Information about these
projects may benefit others involved in translation.
RO 20.3
Assessing the impact of bibliographical
support on the quality of medical care in
patients admitted to an internal medicine
service
Sarti M1 , Pastori MM2 , Pons M3 , Barazzoni F1
1
Ente Ospedaliero Cantonale (EOC) Head Office, Bellinzona,
Switzerland; 2 Università degli Studi di Genova (D.S.Sal.),
Dipartimento di Scienze della Salute, Genova, Italy; 3 EOC
Department of Internal Medicine, Regional Hospital of Lugano,
Switzerland
Background: Application of the best current knowledge to
decisions in individual patients is the key to evidence-based
medicine practice. Research of relevant scientific information
through direct consultation of textbooks or journals or online
resources is a different way to answer the questions that occur
during the visit of the patients. Different studies highlight that
the ideal information source must be directly relevant, contain
valid information and can be accessed with a minimum
amount of work for physicians, but only a few studies
address, as objective, the impact of literature assistance
on clinical outcome. Objectives: To assess and quantify the
impact of literature in diagnostic decisions and treatment
of patients admitted to an internal medicine service, using
evidence-based medicine methodology. Methods: From
November 2012 to February 2013, patients who were
hospitalised in the internal medicine service and generated
questions about medical care, were randomly assigned to
two groups: an intervention group (supported by literature
research) and a control group. The information obtained from
literature was submitted by email to all the medical team
within 12 hours of asking the question. Eight-hundred and
sixty-six patients were hospitalised in the analysed period;
201 of these generated questions, and were divided into
intervention (n = 101) and control (n = 100) groups. In the
intervention group, bibliographical research was possible
for 98 subjects. The medical team accepted the results
and implemented the research for 90.8% of these subjects
(89/98). Statistical analyses were carried out on the intention
to treat (ITT) and on the per-protocol populations. Results:
Bibliographical research had a significant protective effect on
transfer to an intensive care unit (RR = 0.30; 95% CI 0.10 to
0.90; X2 = 5.3, P = 0.02) and hospital readmissions were also
influenced by bibliographical research (RR = 0.42; 95% CI, 0.17
to 1.0; X2 = 3.36, P = 0.05) in the ITT population. Conclusions:
Our results point out the importance of bibliographical
support on the quality of medical care. In particular, they
show its possible impact on clinical outcome.
131
RO 20.4
The Ptolemy Project: knowledge translation
and equity in East Africa
Hagstrom C1 , Kendall S2
1 University of Toronto, Canada; 2 Mt. Sinai Hospital, Canada
Background: Many practitioners in East Africa rely on
Google for literature searches, resulting in untold amounts
of irrelevant or non-scholarly material in addition to lack of
access to full text. The Ptolemy Project was established in
2001 to allow surgeons in East Africa access to the University
of Toronto’s online resources. Participation has increased
to include physicians in all medical fields. Objectives: The
Project offers a means of obtaining up-to-date information for
clinical decision support via the online resources (databases,
e-journals, e-books) available from the University of Toronto
Library as well as additional resources directly from individual
publishers. The University of Toronto library system is
ranked third in North America, after Harvard and Yale, with
over 1,000,000 electronic resources. Methods: Registration
is required; the registration form is on the Project’s
website. Registrants are vetted by staff at the Mount
Sinai Hospital library in Toronto. Participants log in with
a computer-generated username and password and can then
link to available resources. Usage is periodically checked;
unused accounts are cancelled to allow others to register, as
there is a finite number of participants. Results: Registration
is selective and participation now stands at about 450, up
from 90 in 2001. A brief survey was distributed to all users in
July 2014, with positive results. Respondents reported that
access to material through Ptolemy improved their practice
and patient outcomes. Conclusions: The Ptolemy Project is
an invaluable means for physicians and clinical residents in
East Africa to obtain both up-to-date and historical health
information that would not otherwise be readily available.
RO 20.5
Use of research evidence in newborn and
child health policies and practices in Kenya
Karumbi J1 , Mulaku M2 , Opiyo N3 , English M3
1
Ministry of Health, Kenya, Kenya; 2 School of Pharmacy, University
of Nairobi, Kenya; 3 Kemri-Welcome Trust Research Programme,
Kenya
Background: The process of evidence synthesis began in
a very informal process; three topics for which people felt
there were gaps were chosen, namely: sickle cell disease,
umbilical cord care and fluid therapy in children. Systematic
reviews in these areas were conducted and published.
Objectives: To develop recommendations for newborn care.
Methods: A multi-disciplinary panel was selected jointly
by Ministry of Health and Kenya Paediatric Association.
The full systematic reviews, short summaries and GRADE
(Grading of Recommendations Assessment, Development
and Evaluation) ’Summary of findings’ tables were sent to the
panel four weeks before the meeting. At the meeting, the panel
members were exposed to one day of training on the GRADE
process. Evidence was presented for each of the three topics
(one topic per day), followed by discussions facilitated by
the panelists. The discussions mainly focused on, balance of
benefits/harms, feasibility, costs, acceptability (local values),
and equity of the proposed interventions. Consensus was
developed through voting where necessary. Results: The
panels were able to come up with three recommendations
that have been incorporated as policies by the Ministry of
Health. 1. For sickle cell disease: Hydroxyurea should be
considered for use in children under five years of age with a
severe form of sickle cell disease where minimum monitoring
conditions and appropriate formulation are available; this
recommendation was based on low quality evidence. 2. For
umbilical cord care: Chlorhexidine 4% should be applied to
the umbilical cord immediately after birth, and thereafter
daily until the cord separates in babies of > 1000 g (or
> 28 weeks) born in facilities; recommendation based on
moderate quality evidence. 3. For fluid therapy: In children
with severe febrile illness and impaired circulation without
signs of severely impaired circulation maintain hydration
with appropriate maintenance fluids and do not give a rapid
fluid bolus; recommendation based on high quality evidence.
Conclusions: The involvement of key stakeholders at the
development of recommendations is key and possible even
in low-resource settings.
RO 20.6
Impact of Cochrane Corners in Portuguese
journals
Sousa R1 , M Fernandes R2 , Caldeira D3 , Costa J3 , Vaz Carneiro A3
1
Department of Pediatrics, Hospital Santa Maria, Lisbon Academic
Medical Centre, Portugal; 2 Department of Pediatrics, Hospital
Santa Maria, Lisbon Academic Medical Centre; Portuguese Branch
of the Iberoamerican Cochrane Centre (Cochrane Portugal), Lisbon,
Portugal, Portugal; 3 Portuguese Branch of the Iberoamerican
Cochrane Centre (Cochrane Portugal), Lisbon, Portugal
Background: Evidence summaries such as Cochrane Corners
are key knowledge translation tools to improve dissemination
of high-quality evidence to different end-users. Objectives:
To describe and evaluate the impact of Cochrane Corners
published in three major Portuguese medical journals by
the Portuguese Branch of the Iberoamerican Cochrane
Center (Cochrane Portugal). Methods: We developed an
editorial partnership with three medical journals published
in Portugal: Acta Médica Portuguesa (general medical
journal), Acta Pediátrica Portuguesa (pediatrics journal) and
Revista Portuguesa de Cardiologia (cardiology). Topics for
Cochrane Corners were prioritized based on relevance and
novelty. Corners were developed based on previous models,
consisting of concise summaries of individual Cochrane
Reviews or Overviews, complemented by a pragmatic clinical
commentary that puts the evidence in the context of
132
national guidelines and practices. We sought to engage an
inclusive and diverse set of clinicians authors for each Corner,
based on clinical background, training, level of care, and
geographical representability; the Branch provided editorial
and methodological guidance at all stages. Available metrics
of access by Corner and journal include paper downloads
and user data (country, platform, link). Results: We will
analyze recently published Corners and present metrics of
impact for these papers at the Cochrane Colloquium. As
measures of relative impact, we will compare these data with
access statistics for other published articles in each journal,
as well as for individual systematic reviews or overviews that
Corners were based on, with a focus on Portuguese-speaking
countries. We will also provide a perspective on the challenges
and benefits of this approach, particularly regarding the
interest of an editorial partnership, the required workload,
and the dissemination of Cochrane content for different
stakeholders. Conclusions: Cochrane Corners are successful
formats to disseminate evidence from Cochrane Reviews.
Tailored content that puts the evidence in a local clinical
perspective is key to improve its impact.
RO 20.7
Best evidence for better practice: using social
media to help nurses engage with evidence
Chapman S1
1 Cochrane UK, United Kingdom
Background: In February 2014, a Cochrane Review was
published that found no evidence that the use of structured
pressure ulcer risk assessment tools, which is a standard
practice in National Health Service (NHS) hospitals in the UK,
reduces pressure ulcer incidence. In April, the review was
cited in new National Institute for Health and Care Excellence
(NICE) guidance making the use of a tool an optional addition
to clinical judgement. Objectives: We wanted to share this
with the nursing community and encourage them to reflect
on their practice in the light of the evidence. Methods:
We engaged with nurses though Twitter and our weekly
blog, Evidently Cochrane. Other opportunities arose from
this initial activity: co-hosting two tweetchats (scheduled,
live discussions on Twitter) with the WeNurses community
and involvement in Stop The Pressure Month, led by NHS
England with the support of The Royal College of Nursing (The
RCN). Results: Sharing the evidence on Twitter generated
responses from nurses, who were surprised by the review’s
findings, and some of these were incorporated into a blog. In
March, over 100 participants joined a tweetchat to discuss the
evidence. In November, Stop The Pressure Month, a follow-up
tweetchat was held, to see if anyone had initiated changes
in practice. This, and other recent Cochrane evidence on
pressure ulcer prevention and management strategies, was
shared in blogs on our own site and on The RCN’s blog site ‘This
Is Nursing’. Nurses gave reasons why they would continue to
use the tools, for purposes other than that for which they are
intended, such as demonstrating activity and strengthening
requests for equipment. Conclusions: Social media was
used very effectively to disseminate the evidence and engage
nurses in debate about research and practice. It enabled us
to widen our audience and raise awareness of Cochrane. New
relationships with influential individuals and organizations
were forged online which resulted in offline meetings and
opportunities for future collaboration and communication.
RO 20.8
Dissemination of Cochrane Reviews in social
media channels and the influence of
‘positive’ versus ‘negative’ results
Cuello-Garcia C1 , Perez-Gaxiola G2 , Colunga-Lozano LE3 ,
Delgado-Figueroa N4 , Agarwal A5
1 Department of Clinical Epidemiology and Biostatistics, McMaster
University, Canada; 2 Cochrane Mexico, Hospital Pediatrico de
Sinaloa, Mexico; 3 Department of Internal Medicine, Hospital Civil
de Guadalajara ‘‘Dr. Juan I. Menchaca’’, Mexico; 4 Department
of Evidence-Based Medicine, University of Guadalajara, Mexico;
5
University of Toronto, Canada
Background: Currently there is a tendency to measure
the research impact of scholarly articles in social media
sites (e.g. Twitter, Facebook, Scientific Blogs, mass media,
etc.). The Cochrane Library uses Altmetrics, a new tool
to encompass social activity in the form of mentions
on social media channels, online scholarly activity and
commentaries. Objectives: To assess how Cochrane Reviews
are disseminated in social media channels and if those with
‘positive’ results (i.e. with a statistically significant result)
are more disseminated than those with ‘negative’ results.
Methods: We analysed all systematic reviews published in
the Cochrane Library from January to December 2013. Metrics
obtained from each review included the number of Tweets,
Facebook posts, Blog posts, News media, F1000 evaluations,
and the total Altmetric score. Reviews were classified as
‘positive’ if the main outcome effect presented a statistically
significant result (P value < 0.05, or if confidence intervals
reject the null hypothesis) or ‘negative’ if there was not a
statistically significant result, or evidence from individual
studies was insufficient or null. Results: A total of 1007
reviews were evaluated; 37 had been withdrawn from the
Cochrane Library and were excluded from the analyses. The
970 reviews included had a median and interquartile range
(IQR) of seven included studies (2 to 15), four Tweets (1 to
11), and an Altmetric score of 4 (1 to 11). Reviews with
‘positive’ results had more tweets than the ‘negative’ reviews
(11 [5 to 23] vs 4 [1 to 10]; P value < 0.001) and a higher
Altmetric score (6 [2 to 18] vs 3 [1 to 8]; P value < 0.001).
After adjusting for number of included studies, country of first
author, and status of the review, the differences remained
statistically significant. Conclusions: Twitter is the main
channel used for disseminating Cochrane Reviews followed
by Facebook. Reviews with positive results are tweeted and
disseminated in social media channels more often than their
negative counterparts.
133
RO 20.9
New evidence, new ways to share it, new
audiences. Pushing the boundaries with
social media
Chapman S1
1
Cochrane UK, United Kingdom
Background: Many Cochrane reviews relate to common
health conditions and could be used by many people making
decisions about their health. An obstacle is that they may
never have heard of Cochrane, perhaps know nothing about
evidence-informed decision-making and aren’t looking for
evidence. Social media offers unprecedented opportunities
to reach these new audiences. Objectives: Cochrane UK
shares evidence primarily though a weekly blog, ’Evidently
Cochrane’, and Twitter. We had established a format for
the blogs, mostly aimed at patients, which set the evidence
in context, gave the main findings and a key message and
talked about the strength of the evidence. We sought to
expand our audience by experimenting with new ways to
present Cochrane evidence, which would make it engaging,
relatable and easy to understand. Methods: We introduced
quarterly campaign weeks on popular topics (men’s health,
palliative care, the menopause), linking in with national
events. We invited blogs and commentary from ‘experts’, both
health professionals and patients (our youngest blogger being
just nine), exploring evidence in the context of experience.
Through Advent we published daily posts in varied formats
from Lego animations to a film of a pantomime cast talking
about Cochrane evidence. We have also blogged responsively
to feature evidence relating to health topics that came
under the media spotlight. Results: Cochrane UK has a
growing following and we are enjoying frequent interactions
on Twitter. The blog has been selected by public vote as one
of the top ten health organization blogs in the forthcoming UK
Blog Awards 2015. Our activities have helped us establish new
relationships and created opportunities for collaboration.
Conclusions: Innovative use of social media is enabling us
to increase our reach, bring evidence to new audiences and
create new partnerships. We’re looking forward to seeing
what we can do next!
Systematic review
and meta-analysis methods
RO 21.1
How often does an individual trial agree with
its corresponding meta-analysis? A
meta-epidemiologic study
Threapleton D1 , Tam W2 , Di M1 , Tsoi K3 , Tang JL1
1 The Hong Kong Branch of the Chinese Cochrane Centre, The
Chinese University of Hong Kong, Hong Kong; 2 Alice Lee Centre
for Nursing Studies, Yong Loo Lin School of Medicine, National
University of Singapore, Singapore; 3 Jockey Club School of Public
Health and Primary Care, The Chinese University of Hong Kong,
Hong Kong
Background: Evidence from multiple randomized controlled
trials (RCTs) accumulates over many years and waits for
meta-analysis to provide ‘conclusive’ evidence of treatment
effect.
This practice may delay the implementation
of potentially beneficial treatments. Additionally, even
after sufficient evidence has built up, RCTs are still
conducted, potentially wasting resources or introducing
harm. Objectives: To examine how often the result of
the first trial, last trial or any trial selected at random agree
with the corresponding meta-analysis. Methods: Published
meta-analyses were identified from five key medical journals
and also from the Cochrane Database of Systematic Reviews.
Meta-analyses were included where an intervention was
compared with placebo, no intervention or usual care. The
effect size of the earliest, most recent or any trial in the
meta-analysis was compared with the meta-analysis effect
size. Results: A total of 647 meta-analyses were included; the
median number of included studies was 5.2. In 36% (n = 233)
of meta-analyses, the earliest published study reported a
significant result and the corresponding meta-analysis was
significant and in the same direction 84.1% (95% confidence
interval (CI) 79.4 to 88.8) of the time. When the earliest
study did not have a significant result, the meta-analysis
was non-significant in 57.9% (95% CI 53.2 to 62.8) of cases.
Similar results were observed when the last or any trial, were
substituted in analyses. No instances were identified where
the earliest trial and meta-analysis were both significant
but in the opposite direction. In meta-analyses with five
or more studies, the median time from first to fifth study
was 6.5 years. Conclusions: The first trial (or any trial
to hand) demonstrating clinical effectiveness is likely to
reflect the true treatment effect. This finding has important
implications in urgent or critical circumstances where no other
effective treatment is available. Cautious recommendations,
depending on circumstances or potential harms, may be
made if the first trial demonstrates effectiveness. Effective
treatment could be implemented many years in advance of a
meta-analysis.
134
RO 21.2
Confounding, effect modification and the
odds ratio: common misinterpretations
Shrier I1 , Pang M1
1 Centre for Clinical Epidemiology, Lady Davis Institute, McGill
University, Canada
Background: When an outcome is dichotomous and
investigators are concerned about potential confounding
or effect modification across subgroups (e.g. diabetics
versus non-diabetics), they often report both the crude
(unadjusted) odds ratio and the stratum-specific odds ratios
(or adjusted odds ratios). When the stratum-specific odds
ratios are different from each other in the absence of bias in
either observational studies or randomized controlled trials,
authors often interpret this as causal effect modification
(biological interaction). Objectives: To illustrate that
between stratum-specific odds ratios are actually expected
to be different if the variable of interest affects the prevalence
of the outcome, even when both causal effect modification
and bias are absent. Methods: We demonstrate how and
why this phenomenon occurs using hypothetical data from a
randomized trial where the one-year untreated mortality was
52%, and the proportion of diabetics (a cause of the outcome)
was 30% in each group. Results: In our example, the relative
risk was 50% in all patients, in non-diabetic patients, and in
diabetic patients. However, the odds ratio was 0.32 in all
patients, 0.38 in non-diabetic patients and 0.17 in diabetic
patients, with the interaction term from the statistical model
being statistically significant. Remembering that the odds
ratio becomes more extreme compared to the relative risk
as the prevalence increases, our results are a truism since
diabetes affects the outcome in our example. The difference
between the stratum-specific odds ratios is dependent on
the combined effect of the variable that affects the outcome
prevalence (proposed effect modifier), and the baseline risk.
Conclusions: Although logistic regression is an important
tool and reporting adjusted odds ratios (or Cox regression
and rate ratios) is appropriate in many contexts, investigators
and readers should be wary of claims of effect modification
or biological interaction when the covariate is known to be
an independent cause of the outcome, and the disease is
common.
RO 21.3
Cross-over trials –are we doing it right? Are
they doing it right?
Nolan S1 , Dwan K2
1
University of Liverpool, United Kingdom; 2 Cochrane Cystic Fibrosis
and Genetic Disorders Group, United Kingdom
Background: Previous work by Elbourne (2002) indicated
that the description in the methods section of systematic
reviews regarding how cross-over trial data were to be
included was insufficient and recommendations were made
for improvement. Recent work found that reports of cross-over
trials often omit important methodological information
relating to design, analysis and presentation of results.
Objectives: - To assess review methodology for including
cross-over trials in reviews published by the Cochrane
Cystic Fibrosis and Genetic Disorder (CFGD) Group and to
investigate whether there has been an improvement since
the Elbourne results published in 2002. - To assess the
quality of reporting of cross-over trials within the trial reports
themselves, and subsequently, within Cochrane Reviews.
- To establish recommendations for how cross-over data
should be included in Cochrane Reviews, in line with
the Methodological Expectations of Cochrane Intervention
Reviews (MECIR) standards. Methods: We accessed 135
CFGD reviews (published to January 2015) with 655 included
studies; 104 reviews indicated cross-over studies were eligible
for inclusion and 52 included at least one cross-over study.
We identified 212 unique cross-over studies. We checked the
methods sections of reviews to identify how review authors
planned to manage cross-over data. We accessed cross-over
trial reports, the quality of reporting of the results, and
compared them to the data included in the review for the
primary outcomes. Results: Methods sections of reviews
varied in detail. Half of the reviews included specific details
regarding paired analyses, Elbourne ‘3 stage’ method or first
period analysis, the other half made no statement regarding
cross-over trials, but referred to the work by Elbourne without
giving further details, intended to consult the Cochrane
Handbook or a statistician or intended to analyse cross-over
data as parallel data. Full results regarding the quality of
reporting within the trial reports and how results were actually
included in the reviews will be presented. Conclusions and
recommendations at study and review level will also be
presented.
RO 21.4
How do Cochrane review authors interpret
subgroup analyses?
Richardson M1 , Garner P1 , Donegan S1
1 Cochrane Infectious Diseases Group, United Kingdom
Background: Treatment by covariate interactions can be
explored in reviews using subgroup analyses. Such analyses
can identify how treatment effect varies by subgroup and
are an important methodological approach for stratified
medicine. A review of 52 recently published Cochrane Reviews
found that 63% applied subgroup analyses [1]. However,
interpretation of subgroup analyses in Cochrane Reviews is
poor; the review found that only 3% of reviews reported
whether or not there was an interaction (i.e. a difference
between subgroup results) [1]. Reasons for the lack of
interpretation have not been explored. Objective: To identify
how review authors interpret subgroup analyses and why
analyses are not always interpreted. Methods: We will
develop and pre-pilot survey/interview questions based on
subgroup analysis scenarios (i.e. no interaction; qualitative
and/or statistically significant interaction). We will ask review
135
authors: (1) to interpret each analysis; (2) whether there is
a statistically significant interaction; (3) how they decided if
the interaction was statistically significant. We will also ask
why subgroup analyses were not interpreted in their review
(if applicable). We will survey correspondence authors for
the 52 Cochrane reviews that were included in the recent
review [1]. The review included the most recently published
review for each Review Group published in the Cochrane
Database of Systematic Reviews (searched 8 August 2013).
Review authors based locally will be interviewed. Results:
We will summarise qualitative results (e.g. interpretations
of analyses) and quantitative results (e.g. number of review
authors who correctly identified a significant interaction).
Review authors will be anonymised. Conclusions: This
research will identify how reviews authors interpret subgroup
analyses, why subgroup analyses are often not interpreted,
and improve how analyses are interpreted in future reviews.
Reference
1. Donegan S, Williams L, Dias S, Tudur-Smith C, Welton N.
Exploring treatment by covariate interactions using subgroup
analysis and meta-regression in Cochrane Reviews: a review
of recent practice. Submitted to PLoS one.
RO 21.5
Is the risk difference really a more
heterogeneous measure?
Shrier I1 , Poole C2 , Vanderweele TJ3
1
Centre for Clinical Epidemiology, Lady Davis Institute, McGill
University, Canada; 2 Department of Epidemiology, University of
North Carolina at Chapel Hill, USA; 3 Department of Epidemiology,
Harvard School of Public Health, Harvard University, USA
Background: There are claims in the literature that the risk
difference is a more heterogeneous measure than the odds
ratio or risk ratio. These claims are based on surveys of
meta-analyses showing that tests reject the null hypothesis of
homogeneity more often for the risk difference than for ratio
measures. Objectives: To illustrate that differences in results
of homogeneity tests across different scales may be related
to statistical power rather than differences in homogeneity
itself. Methods: We use hypothetical examples where lack
of homogeneity is arguably the same across different scales,
but the power (and therefore the expected P value) for the
different scales are remarkably different. Results: In the first
example, we simulated 75 participants per treatment group
where the probability of outcome is 0.206 in Population A
regardless of treatment. In Population B, treatment increased
the probability of remission from 0.270 to 0.460. Simulating
a meta-analysis based on these effects would yield a power
of 47% for risk difference, but only 35% for the odds ratio.
These numbers exactly match the results used by others to
suggest the risk difference is more heterogeneous than the
odds ratio. We also simulated 150 participants per group,
where baseline remission probability was 0.2 in Population
A and B. In this example, treatment increases probability of
remission to 0.304 in Population A, and to 0.350 in Population
B. but has an effect in Population B, leading to different risk
differences and odds ratios in the two populations. These
numbers also recreate a power of 47% for risk differences and
only 35% for the odds ratios. The effect can also be reversed,
with the test of homogeneity having less power for the risk
difference compared to the odds ratio. Conclusions: Because
current methods cannot be used to conclude that one scale
is more heterogeneous than another within meta-analyses
of the same data, claims that the risk difference is more
heterogeneous should be considered, at best, tentative. The
meta-meta-analytic results, therefore, do not support a policy
of routinely shunning any of the three measures, including
the risk difference.
RO 21.6
Small studies are more heterogeneous than
large ones: a meta-meta-analysis
IntHout J1 , Ioannidis JP2 , Borm GF1 , Goeman JJ1
1 Radboud University Medical Center, The Netherlands; 2 Stanford
University, USA
Background:
Between-study heterogeneity plays an
important role in random-effects models for meta-analysis.
Most clinical trials are small, and small trials are often
associated with larger effect sizes. Possibly there is also
a difference in the extent of between-study heterogeneity (τ )
between small and large trials. Objectives: To evaluate
empirically whether there is a relationship between trial
size and heterogeneity (τ ), using meta-analyses from the
Cochrane Database of Systematic Reviews from 2009 to
2013.
Methods: We selected the first meta-analysis
per intervention review with a dichotomous (n = 2009) or
continuous (n = 1254) outcome. The association between
estimated τ and trial size was evaluated within meta-analyses
using a Bayesian hierarchical model and across meta-analyses
using regression. Small trials were predefined as those having
standard errors over 0.2 standardized effects. Results: Most
meta-analyses were based on few studies (median 4, Q1 to
Q3: 2 to 6), and most (74%) primary studies were small. Within
the same meta-analysis, the small-study τ S2 was larger
than the large-study τ L2: mean ratio 2.11 (95% Credible
Interval 1.05 to 3.87) for dichotomous and 3.11 (95% Credible
Interval 2.00 to 4.78) for continuous meta-analyses. The
imprecision of τ S was larger than of τ L: median SE 0.39
versus 0.20 for small-study and large-study meta-analyses
with a dichotomous outcome and 0.22 versus 0.13 for those
with a continuous outcome. Similar results were found in the
across-meta-analyses approach. Conclusions: Heterogeneity
between small studies is larger than between larger studies.
The large imprecision with which τ is estimated in a typical
small-studies’ meta-analysis is another reason for concern
and sensitivity analyses are recommended.
136
RO 21.7
RO 21.8
Evaluation of statistical heterogeneity
measures and their precision in systematic
reviews with meta-analysis and the impact in
their interpretation
Systematic reviews should explore a priori
hypotheses to explain heterogeneity even
when I2 is low
Ruiz Gaviria RE1 , Rodriguez Malagon MN1
1
Clinical Epidemiology and Biostatistics Department, Faculty of
Medicine, Pontificia Universidad Javeriana Medical School, Bogota,
Colombia
Background: Systematic review, followed by meta-analysis,
is one of the most powerful and widespread tools for
understanding efficacy and safety evaluation of medical
interventions. However, statistical heterogeneity is one
of the most serious issues when conducting this type of
analysis. Cochran Q coefficient and I2 (I-squared) are the
main statistics used to evaluate it. Confidence intervals (CI)
can be calculated for both of them in order to determine
their precision. Objectives: The aim of this study was to
review the application of the method to determine statistical
heterogeneity, and its precision, in a group of selected
systematic reviews that also considered meta-analysis and
are published by Cochrane. Methods: We searched in
PubMed and Wiley for systematic reviews with meta-analysis
that investigated chronic obstructive pulmonary disease and
were produced by the Airways Group and published on the
Cochrane Library. We included systematic reviews published
between January 2013 and February 2015. We extracted the
number of studies used in subgroup and total analysis, the
Q coefficient, its P value, I2 statistics and their CI. If the CI
was not present, we calculated it using the method proposed
by Higgins et al. Results: We included five reviews in the
analysis. None reported the CI of the I2. The calculated
CIs were consistently large, showing heterogeneity that could
be interpreted as mild to severe. We found inconsistencies
between the interpretation of the heterogeneity of the studies
and the calculated CI in all of them, especially when the
authors stated that there was no heterogeneity. Conclusions:
Heterogeneity is a serious concern in meta-analysis. We found
inconsistencies between the interpretation of the reported
values and the CIs of the I2. We propose that authors and also
reviewers consider the existence of heterogeneity as an issue
that really affects meta-analysis results because the selection
of whether to apply a random-effects or fixed-effect model
depends on its presence.
Siemieniuk R1 , Meade M1 , Alonso P2 , Briel M1 , Vandvik P3 ,
Guyatt G1
1
McMaster University, Canada; 2 Iberoamerican Cochrane Center,
Biomedical Research Institute (IIB-Sant Pau-CIBERESP), Barcelona,
Spain; 3 Norwegian Knowledge Centre for the Health Services, Oslo,
Norway
Background: There is general agreement that systematic
review authors should generate a priori hypotheses to
explain heterogeneity and test these hypotheses when
heterogeneity proves substantial.
However, when the
meta-analysis suggests low heterogeneity, as represented
by a low I2, controversy exists. In these circumstances, some
advocate, and practice, omitting statistical exploration of
heterogeneity. Others disagree. Objective: To illustrate
the advisability of exploring possible subgroup effects even
when I2 is low. Method: We conducted a systematic
review and meta-analysis addressing the desirability of
adjunctive administration of corticosteroids in patients with
community-acquired pneumonia. We generated four a priori
hypotheses to explain heterogeneity, including the severity
of pneumonia (expectation of larger effect on mortality when
over 70% of patients had severe pneumonia). Results:
Random-effects meta-analysis showed a relative risk (RR)
of 0.67, 95% confidence interval (CI) 0.47 to 0.97, I2 =7%,
for overall mortality (Figure 1). Despite the low I2 we
undertook Chi-square tests for effect modification for our
a priori hypotheses. We found an apparent mortality benefit
in trials that met our ’more severe’ criteria (6 studies; n = 388;
RR = 0.39, 95% CI 0.22 to 0.67; I2 = 0%) but not in those
that did not (6 studies; n = 1586; RR = 1.00, 95% CI 0.64 to
1.56; I2 = 0%; interaction P = 0.009; Figure 2). The subgroup
finding gains credibility from the large magnitude of effect,
its biological plausibility (a greater inflammatory response
in more severe pneumonia), the small number of a priori
hypotheses with specified direction, and a small interaction
P value. It is based, however, on differences between studies
rather than within studies, and was driven to a considerable
extent by a small study that was stopped early for benefit
and almost certainly represents a large overestimate of
effect. Overall, the credibility of the subgroup effect is
moderate. Conclusions: A low I2 should not deter systematic
review authors from exploring a priori hypotheses to explain
heterogeneity.
137
138
RO 21.9
Adding non-randomized and non-controlled
studies to an existing review on the
effectiveness of chronic disease management
programs for asthmatic patients
Arditi C1 , Burnand B1 , Peytremann-Bridevaux I1
1
Cochrane Switzerland and Institute of Social and Preventive
Medicine, Lausanne University Hospital, Switzerland
Background: Systematic reviews (SRs) of interventions
usually include randomized controlled trials (RCTs) only.
However, complex interventions such as chronic disease
management (CDM) programs are often evaluated using study
designs other than RCTs, mainly for reasons of feasibility.
Considering non-randomized and non-controlled (NRNC)
studies in an SR on CDM programs could be informative
both in terms of intervention context and effectiveness.
Objectives: To assess if and how setting, population,
intervention characteristics, and results varied when adding
NRNC studies to an ongoing Cochrane SR with the Effectivea
Practice and Organization of Care Review Group. Methods:
All NRNC studies identified as evaluating CDM programs in
asthmatic adults, but excluded from the Cochrane SR because
of the design, were considered in this review, in addition to the
already included studies. Two review authors independently
extracted data and assessed the methodological quality using
the Cochrane risk of bias tool, with additional questions for
non-controlled studies. We compared study and intervention
characteristics, as well as effectiveness, by study design.
Results: Thirty-seven studies were added to the 20 studies
already included in the original SR. The added studies were of
poor methodological quality (e.g. no statistical adjustments
for confounding in controlled before-after studies). Study
and intervention characteristics of the NRNC studies differed
from those in the original SR (Table 1). For instance, CDM
programs were implemented in a larger variety of countries,
139
included larger numbers of patients, and reported process and
healthcare use outcomes more often. Effectiveness also varied
according to study design (Table 2). Overall, the proportion
of outcomes that improved in response to CDM was higher
in the NRNC studies compared with the studies from the
original SR. Conclusions: While NRNC studies provide results
from interventions in real-life settings that may be valuable
to policymakers and other stakeholders implementing CDM
programs, the methodological quality of the studies is
poor, limiting the validity and the interpretation of their
results.
Guidelines and GRADE
RO 22.1
Strength of the evidence and treatment
recommendations: a survey of the Cochrane
Database of Systematic Reviews
for the first listed primary outcome in SoFs in each review
and reasons for upgrade or downgrade were recorded. For
reviews with high quality of evidence, those with low quality of
evidence, and a random sample of 50 reviews with no GRADE
assessment, we also extracted the results for the evaluated
outcomes and recommendations for the use of interventions.
Results: A total of 1394 SRs were identified; 608 (43.6%)
of these incorporated GRADE. Within these reviews 13.5%,
30.8%, 31.7% and 24.0% reported high, moderate, low and
very low levels of evidence, respectively, for the evaluated
primary outcome. High level quality was more common in
updated than new reviews and for pharmacological than other
types of interventions. Even when all outcomes listed in the
SoFs were considered, only 116/608 (19.1%) SRs had at least
one outcome with a high quality of evidence. Twenty-five
(30.5%) of 82 SRs with high quality evidence had both
significant results and interventions recommended by the
authors. Conclusions: A minority of systematic reviews in
the CDSR have high level evidence and even in this small set,
an intervention was recommended in only one-third of the
reviews.
Fleming P1 , Koletsi D2 , Ioannidis J3 , Pandis N4
1 Barts and the London School of Medicine and Dentistry, United
Kingdom; 2 University of Athens, Greece; 3 Stanford University School
of Medicine, USA; 4 University of Bern, Switzerland
RO 22.2
Background: The GRADE (Grading of Recommendations
Assessment, Development, and Evaluation) tool allows
systematic appraisal of the quality of a body of evidence
based on specific domains leading to an overall score
assessment presented in ’Summary of findings’ tables (SoFs).
Objectives: To survey the level of evidence contributing to
recommendations derived from Cochrane Reviews and how
this level of evidence correlates with whether an intervention
is recommended or not. Methods: All systematic reviews on
the Cochrane Database of Systematic Reviews (CDSR) from 1
January 2013 to 30 June 2014 were identified and selected by
two authors from the Cochrane Library. Data were extracted
independently by two authors and the quality of the evidence
Selph S1 , Holmes R1 , McDonagh M1
1 Pacific Northwest Evidence-Based Practice Center, USA
Predictors of change in updated quality
of evidence ratings in Cochrane Reviews
Background: Using GRADE (Grading of Recommendations
Assessment, Development and Evaluation tool) to rate the
quality of the body of evidence is complex and subjective,
and how consistently reviewers apply the rating system is
unclear. Objectives: The aim of this study is to evaluate
how changes in evidence contribute to changes in quality
of evidence ratings by comparing Cochrane Reviews having
quality of evidence tables with earlier and later versions.
Methods: We conducted a MEDLINE search of the Cochrane
Library for systematic reviews with quality of evidence tables
140
in sequential updates. For each outcome with a change
in number of studies, sample size, or quality rating, we
abstracted publication years, review authors, effect estimates,
numbers of included studies, and sample sizes and conducted
regression analyses to determine predictors of a change in
quality rating. Results: A total of 1814 systematic reviews
were marked in the Cochrane Library as recently updated.
Preliminary analysis of 350 reviews found nine that provided
quality ratings tables for sequential updates, representing 36
different outcomes. Two factors associated with a change in
quality rating in univariate analysis were: change in numbers
of participants (P value 0.041) and change in magnitude of
the effect estimates (P value 0.008). Changes in precision or
statistical significance of effect estimates were not associated
with a change in quality rating. Quality of the evidence was
downgraded (n = 13) more often than upgraded (n = 4) and
was sometimes downgraded with no change in the evidence.
Conclusions: Changes in the magnitude of effect estimates
and numbers of participants were associated with changes
in quality of the body of evidence ratings. Quality ratings
were downgraded more often than upgraded which may
indicate a reviewer ’experience’ factor as the reasons for
downgrading were not always obvious and it would be helpful
if reviewers explained the rationale for these changes. Further
analysis including the remainder of the reviews and the use
of multivariable regression to explore relationships between
variables is in progress.
RO 22.3
Reported use and perceptions of value of
Cochrane evidence by South African guideline
developers
1
1
2
3
1
4
Abrams A , Kredo T , Young T , Louw Q , Grimmer K , Daniels K
1 South African Cochrane Centre, South African Medical Research
Council, South Africa; 2 South African Cochrane Centre, South
African Medical Research Council and Stellenbosch Univeristy,
South Africa; 3 Stellenbosch University, South Africa; 4 South African
Medical Research Council, South Africa
Background: Systematic reviews are a key source of
transparent evidence assessments for guideline development.
This research, as part of the South African Guidelines
Excellence (SAGE) project, maps the use and perceived
benefits of Cochrane Systematic Reviews amongst primary
care guideline developers in South Africa. Methods: A
modified snow-ball technique identified key stakeholders.
Individual interviews were recorded and transcribed. Content
and descriptive analysis was undertaken. Interview scripts
were analysed using an inductive approach. Results:
Twenty-four interviews have been conducted with 23 key
stakeholders actively involved with guidelines in government,
public and private health care. Early exploratory analysis
indicates that guideline development is a complex web of
interactions between stakeholders; a process that is informed
by individual values, politics and power at both the individual
and group level. Initially Cochrane-related work and research
was not a focus of this study, but a trend emerged within
the interviews. Without prompting from interviewers, 8/22
respondents named Cochrane as a key resource for guideline
evidence assessments, or as a key player in reviews of
evidence. ‘‘Obviously I’m going to look at Cochrane . . .I’m
going to try look at the highest level first. So the issue
is what you want, you want integrity . . .’’ (INT01/398). Of
the 14 interviewees who did not name Cochrane, four rely
on World Health Organization (WHO) guidelines to inform
the evidence they use in local guideline development. The
remainder describe the need for an evidence grading process,
but highlight a variety of barriers to establishing this process
in the local context including limited human capacity and
time. Conclusion: Within the complex guideline development
process in South Africa, it is clear that Cochrane evidence
plays a role. Understanding this role can provide Cochrane
reviewers with input on key areas of interest for policy and
guideline writers, identifying where Cochrane Reviews can
cater specifically to the needs of guideline writers, especially
in LMICs.
RO 22.4
An algorithm to assign GRADE levels of
evidence to comparisons within systematic
reviews
Pollock A1 , Farmer S1 , Brady M1 , Langhorne P2 , Mead G3 , Mehrholz
J4 , van Wijck F1 , Wiffen P5
1
Glasgow Caledonian University, United Kingdom; 2 University
of Glasgow, United Kingdom; 3 University of Edinburgh, United
Kingdom; 4 Klinik Bavaria in Kreischa GmbH, Germany; 5 University
of Oxford, United Kingdom
Background: An essential part of a Cochrane overview is the
assessment of the quality of evidence within included reviews.
We planned to use the GRADE (Grading of Recommendations,
Assessment, Development and Evaluation) approach in our
overview (as recommended in the Cochrane Handbook), but
its subjectivity led to inconsistency of application. Objectives:
To develop and use an algorithm to assign GRADE levels of
evidence objectively to comparisons within reviews included
in a Cochrane Overview. Methods: After initial exploration
of applying GRADE levels of evidence, authors agreed criteria
perceived to be most relevant for judging quality of the
particular evidence synthesised within the overview. Key
criteria judged to be of most importance were: number
of participants; risk of bias of trials, heterogeneity; and
methodological quality of the review. An initial algorithm was
drafted, applied to a convenience sample of 43 comparisons,
and compared to previous, independently-applied, subjective
judgements of GRADE level. An iterative process explored
impact of criteria ’weighting’ within the algorithm and
the number of consequent downgrades of quality level.
We created and applied four different formulae to assign
downgrades to each of the 43 sample comparisons and
explored the resulting levels of evidence. Results: An
algorithm, judged to assign the most appropriate levels of
evidence (see Table 1), and a formula, for assigning GRADE
141
level of evidence based on number of downgrades determined
using the algorithm (see Table 2), were developed. This
algorithm was applied to all 127 comparisons included within
the overview. Conclusions: This algorithm enabled us to
justify and report the GRADE level of evidence clearly for
each comparison, and also to be consistent, transparent
and efficient in decisions around levels of evidence for each
comparison. Whilst mechanistic in application, this algorithm
aims to capture what was subjectively judged to be of greatest
importance to the quality of this specific evidence-base. We
propose that this methodological approach has implications
for assessment of quality of evidence within future evidence
syntheses.
RO 22.5
Development of a useful system for grading
of recommendations and its application on
the Korean Guideline
Jang J1 , Shin E1 , Yeon J1 , Chang S2 , Kim D3 , Lee Y4
1
Korean Academy of Medical Sciences, South Korea; 2 Department
of Urology, Kyung Hee University School of Medicine, South Korea;
3
Department of Diagnostic Radiology, Yon Sei University School
of Medicine, South Korea; 4 Department of Legal Medicine, Seoul
National University College of Medicine, South Korea
Background: To alleviate discrepancies between evidence
and practice, the Korean Modified Grading of Recommendations Assessment, Development and Evaluation (GRADE)
system has been developed and applied to clinical practice
guideline on diabetes mellitus. Objectives: To review
142
the determinants for grading system used in Korean
clinical practice guidelines (CPGs) and identify the strong
recommendations with low quality evidence, and apply the
Korean Modified GRADE system. Methods: To review the
grading factors of recommendations in Korean CPGs, 43
guidelines from the Korean Medical Guideline Information
Center were classified according to a variety of categories.
Based on the classification, the Korean Modified GRADE
system is developed by reflecting the opinion of end-users
like clinical physicians. The Korean Modified GRADE system
is then applied to 2014 Evidence-based recommendations for
type 2 diabetes in primary care developed with the Korean
Academy of Medical Science and the Korean Centers for
Disease Control & Prevention, and an online survey was
conducted by 30 doctors through the Delphi process. Results:
The determinants to be considered in 43 Korean guidelines
were ‘balance of benefits and harms’, ‘patient’s value and
preference’, ‘quality of evidence’, ‘cost’, ‘expert opinion’ and
‘health setting’. Strong recommendations with low evidence
constituted 47 of 215 recommendations, accounting for
about 21.9%. After applying the Korean Modified GRADE
system, about 12.6% of the total recommendations of ‘2014
Evidence-based recommendations for type 2 diabetes in
primary care’ have been adjusted to up-grading. Conclusions:
The Korean Modified GRADE system can be utilized as an
effective tool to better reflect the opinion of the physicians at
medical field.
RO 22.6
Do Cochrane Systematic Reviews inform
clinical practice guidelines in urology?
Gudeloglu A1 , Ozdemir B1 , Dahm P2
1
Prostatic Diseases & Urological Cancers Group, Turkey; 2 Prostatic
Diseases & Urological Cancers Group, USA
Background: Clinical practice guidelines (CPGs) aim to
provide evidence-based guidance to inform individual patient
and health policy decision-making. They should ideally be
based on high quality systematic reviews, for example as
provided by Cochrane. Uptake of Cochrane Reviews by
guideline developers also represents an important measure
of relevance and impact for Cochrane. Objectives: We
performed this study to assess the uptake of Cochrane Reviews
in urological guidelines. Methods: We retrieved the most
up-to-date version of CPGs by five major guideline developers
relevant to urology, namely those of the European Association
of Urology (EAU), the American Urological Association (AUA),
the National Institute for Health and Care Excellence (NICE),
the National Comprehensive Cancer Network (NCCN), and
United States Preventive Services Task Force (USPSTF). Two
independent reviewers abstracted the number of citations
of Cochrane Reviews and the review group from which they
came. Results: We identified a total of 66 CPGs; the largest
number was published by the EAU (n = 26) and AUA (n = 20),
followed by the USPSTF (n = 8), NCCN (n = 6) and NICE (n = 6).
Less than half (30/66; 45.5%) of all CPGs cited at least one
Cochrane Review. Overall, Cochrane Reviews were cited 199
times. The proportion of CPGs by different organizations that
cited at least one Cochrane Review was 83.3%, 61.5%, 50.0%,
20.0% and 16.6% for NICE, EAU, USPSTF, AUA and NCCN,
respectively. The largest contributions of cited Cochrane
Reviews originated from the Incontinence (29.1%), Prostatic
Diseases and Urological Cancers (23.6%), Pain Palliative and
Supportive (17.6%), and Renal (12.1%) Groups. Conclusions:
The uptake of Cochrane Reviews by CPGs developers for
the field of urology is suboptimal, especially in the USA.
Increased efforts to promote high quality systematic reviews
by the relevant Cochrane groups are therefore critically
important.
RO 22.7
The predictive validity of GRADE for the
stability of effect estimates was low: a
Evans T1 , Gartlehner G2 , Dobrescu A3 , Bann C1 , Lohr K1
1
RTI International, USA; 2 RTI International, Cochrane Austria,
Danube University, USA, Austria; 3 Victor Babes University of
Medicine and Pharmacy, Timisoara, Romania
Background: Many organizations have adopted the GRADE
(Grading of Recommendations Assessment, Development and
Evaluation) approach to rate researcher confidence in an
available body of evidence. GRADE’s definition of quality
of evidence (QoE) links individual grades to the degree of
confidence that estimates are close to the true effect (and
thus will remain stable as new evidence accrues). Objectives:
To determine the predictive validity of the GRADE approach
(i.e. whether GRADE discriminates reliably between evidence
that remains stable and evidence that changes as new studies
emerge). Research Design/Methods: To determine the
predictive validity, we randomly assigned 13 researchers
who are producers and users of systematic reviews using the
GRADE approach from six US Evidence-based Practice Centers
and Cochrane Austria 160 bodies of evidence to grade. Using
likelihoods from the survey as reference points, we calculated
c-statistics to determine the predictive validity. Results:
GRADE did not discriminate well between bodies of evidence
that remained stable and those that changed (c-scores 0.56
to 0.58). Conclusions: GRADE is a suitable method for
systematic review producers to convey uncertainties to users.
The predictive validity of GRADE was compromised by grades
of QoE that seemed, in general, too low.
143
RO 22.8
Reproducibility of Grading of
Recommendations Assessment, Development
and Evaluation (GRADE) factors on the
strength of recommendations: an empirical
assessment
Kumar A1 , Miladinovic B1 , Guyatt GH2 , Schunemann HJ2 ,
Djulbegovic B1
1
Morsani College of Medicine, USF Health, Program for CER, Tampa,
USA; 2 McMasters University, Hamilton, Canada
Background: GRADE is a widely used methodology for the
development of clinical practice guideline but its
reproducibility has not been tested in context of development
of clinical practice guidelines. Objective: Assess the
reproducibility of all GRADE factors among guidelines panel
members with limited exposure to GRADE methodology.
Methods: The study was conducted as part of the
clinical practice guideline development process of American
Association of Blood Banking (AABB) for the use of
prophylactic versus therapeutic platelet transfusion in
patients with thrombocytopenia. The results from systematic
review and meta-analysis for each question were summarized
as a GRADE evidence profile. Inter-rater agreement for
all GRADE factors and strength of recommendations was
summarized using a weighted kappa statistic with 95%
confidence intervals (CI). Results: Eighteen panel members
participated in the deliberation of making recommendations
and completed the online questionnaire. They were given two
one-hour lectures about GRADE. The inter-rater agreement
for the domain of quality of evidence was good (kappa
value: 0.68; 95% CI 0.541 to 0.837), and fair for balance
of benefit and harms (kappa value: 0.4; 95% CI 0.253 to
0.574) and use of resources (kappa value: 0.275: 95% CI
0.116 to 0.421). The inter-rater agreement was moderate for
the GRADE domain of patients’ values and preferences (kappa
value: 0.441; 95% CI 0.307 to 0.555). The inter-rater agreement
for making a for/against recommendation was good (kappa
value: 0.738; 95% CI 0.331 to 0.914) and fair for strong/weak
recommendation (kappa value: 0.391; 95% CI 0.175 to 0.681).
Conclusions: While not all elements of GRADE system had
good agreement, the inter-rater agreement for assessing the
quality of evidence and issuing a recommendation of for,
versus against, was substantial. This is probably because
GRADE has operationalized these two areas in more details
than other domains. Further operationalization of all GRADE
domains would likely improve its reproducibility across the
entire GRADE system.
RO 23.1
Searching for trials in systematic reviews
versus a specialist trials register: a case study
comparison of source, time and costs
Brunton G1 , Stokes G1 , Shemilt I2 , Stansfield C1 , Sutcliffe K1 ,
Thomas J1
1 EPPI-Centre, UCL IOE London, United Kingdom; 2 Behaviour and
Health Research Unit, Dept. of Public Health & Primary Care,
University of Cambridge, United Kingdom
Background: Some public health systematic reviews examine
interventions that reflect transdisciplinary health concepts,
creating challenges in identifying either too many or too
few eligible studies. Two sources may be of use: existing
reviews’ evidence tables and reference lists, and a specialist
public health trials register (TRoPHI). The costs and benefits of
searching these sources to identify eligible trials are unknown.
Objective: To assess the incremental costs and benefits of
searching for eligible trials in evidence tables and reference
lists of existing systematic reviews, compared with a TRoPHI
search, in a case study systematic review of community
engagement interventions. Methods: We searched six
multidisciplinary databases for existing reviews, and TRoPHI,
to capture records of eligible trials. Two researchers
independently screened systematic reviews and TRoPHI
records, followed by full-texts, to identify eligible trials. The
researchers prospectively tracked the flow of records and
studies through screening and time taken to complete the
process (searches, screening and duplicate removal). Results:
The TRoPHI search retrieved 865 unique records, of which
226 corresponding full-texts were screened to identify 153
reports of eligible trials: the search, screening and duplicate
removal process took 31.0 hours. The systematic reviews
search retrieved 342 reviews containing 703 unique records
of trials, of which 153 were reports of eligible trials: the
search, screening and duplicate removal process took 128.5
hours. No additional eligible trial reports were identified
by searching existing systematic reviews. Analysis of costs
is in process. Conclusions: Searching existing systematic
reviews increased the cost of study identification with no
incremental benefit to the review in this case. Researchers
undertaking complex reviews are encouraged to consider the
transdisciplinary nature of their review topic to help determine
144
the relative trade-offs for each source in terms of time, effort
and yield of unique citations. Further case studies could
help establish appropriate methods of searching systematic
reviews to source primary studies.
RO 23.2
Searching trials registers to identify relevant
studies for Cochrane Reviews: verifying the
optimal search approaches
Metzendorf MI1 , Richter B1
1
Metabolic and Endocrine Disorders Group, Germany
Background: The Methodological Expectations of Cochrane
Intervention Reviews (MECIR) require Cochrane authors
to search ClinicalTrials.gov (CT.gov) and the International
Clinical Trials Registry Platform (ICTRP) in addition to
bibliographic databases. The evidence on how to search trials
registers soundly is evolving. According to previous research,
high retrieval performance (defined as best sensitivity) is
achieved only when using sensitive search strategies in
specific interfaces of the trials registers. Objectives: To
determine which search approach in which interface of CT.gov
and ICTRP yields the best retrieval performance to identify
relevant trials on a review’s topic and to verify previously
conducted research. Methods: We tested searches from
CMED reviews using the methodology by Glanville 2014.
We tested four search strategies per review topic: sensitive
(various terms for intervention/condition) and highly precise
(single term for intervention/condition) each run in the basic
and advanced interface of the trials registers respectively.
We did not evaluate the highly sensitive (various terms
for intervention) nor the precise search strategy (single
term for intervention), as searches developed for most
Cochrane Reviews will include intervention combined with
condition. Results: The reviews comprised pharmacological,
complementary, surgical and nutritional interventions. The
conditions included type 2 diabetes mellitus, obesity,
idiopathic short stature and cardiovascular disease. In most
cases the sensitive approach performed better in both trials
registers. In CT.gov both search interfaces performed well, but
the advanced search offered an improved precision without
compromising sensitivity. In ICTRP the standard search was
clearly superior to the advanced search. Conclusions: Our
results are consistent with previous findings. When searching
trials registers researchers should apply sensitive search
strategies including synonyms. In CT.gov both basic and
advanced search interfaces can be used for optimal retrieval.
In the ICTRP the advance search performes generally poor,
therefore researchers should only use the standard search
interface.
RO 23.3
Efficiently searching for exhaustiveness: how
to perform high quality searches faster
Bramer WM1 , de Jonge G1 , Mast F1
1 Erasmus MC, The Netherlands
Background: Searching exhaustively, as is necessary for
systematic reviews, can be very time consuming for both
librarians and researchers. Selecting the search terms
(balancing sensitivity and specificity) and translating queries
between databases is challenging. Objectives: A method was
developed for translating a research question into optimal
queries in several databases, but are the results good enough,
and what are the success factors?
Methods: In order to guarantee term completeness an
optimization technique was developed. This method easily
identifies missed relevant terms. Macros in MS Word have
been developed to convert syntax automatically between
databases and interfaces. Information specialists have used
these methods to create searches for systematic reviews.
Many parameters of the search are recorded in the process,
and after completion of the review, the included references
are checked for database coverage and retrieval. Results:
Over 425 searches have been tracked. For more than 30
of these the development of the optimization process, and
for more than 100 searches, the relevant articles have been
chosen by the reviewers. The parameters and outcomes of
these searches is compared to data from published reviews
from other Dutch University Medical Libraries. Ninety per cent
of the searches performed with this method by one of the
authors were completed within two hours. The median time
needed was 70 minutes. Still, the searches are longer than the
comparative data, use more databases, retrieve more relevant
references, and have a higher precision. Conclusions: The
information retrieval process can be shortened, without loss
of quality. This allows us to assist many more review projects
than was possible with conventional methods, increasing the
overall quality of our institutes’ scientific research.
RO 23.4
Is it possible to focus EMTREE without loss of
sensitivity when searching Embase for
systematic reviews? Evidence from practice
Duffy S1 , Ross J1 , Misso K1 , Noake C1 , Stirk L1
1
Kleijnen Systematic Reviews Ltd, United Kingdom
Background: Systematic reviews require a comprehensive
search of numerous databases in order to minimise
bias.
MEDLINE and Embase are the most commonly
searched bibliographic databases when undertaking reviews
of healthcare interventions. As the overall search results for
systematic reviews appear to be getting increasingly larger, it
would help reduce workload if search results could be made
smaller. Focusing EMTREE subject heading terms in Embase
could significantly reduce the number of records retrieved.
145
Objectives: To investigate whether restricting EMTREE
indexing terms in order to focus when searching Embase
will reduce the number of records retrieved without loss of
relevant studies. Methods: Embase searches conducted in
recent systematic reviews undertaken by Kleijnen Systematic
Reviews will be retrospectively compared with search
strategies in which the EMTREE terms have been focused.
The records retrieved by the focused EMTREE search will be
investigated to see if included studies identified by the original
unrestricted Embase search strategy have been retrieved.
The review searches under investigation will cover various
subjects, such as anaphylaxis, bile acid malabsorption, and
KRAS for colorectal cancer. A prospective investigation will be
undertaken on a current review: medicinal cannabis. Results:
The data collected will be analysed to identify: total with and
without restriction to focus, yield of included records, and
Number Needed to Read (NNR) to detect relevant references.
Conclusions: The project will explore overall yield and recall
of relevant included records by each approach, and whether
focussing EMTREE terms can reduce screening burden without
significantly impairing recall of relevant records. Reducing the
number of records retrieved from systematic review searches
without a loss of sensitivity will improve efficiency, save time
and minimise costs.
Only a few, however, were included in the review, because
most of authors did not search the Ichushi-Web database
of the Japan Medical Abstracts Society (JMAS). Objectives:
We examine whether results from the existing reviews may
be influenced by those of the Ichushi-Web, when articles did
not appear in PubMed. Methods: Using the literature search
strategy in the review by Chartrand 2012, we searched for
Japanese and English studies on the Ichushi-web between
2000 and March 2015. We extracted outcomes regarding
RIDTs accuracy and assessed risk of bias. Results: From the
Ichushi-Web, four studies written in Japanese matched the
inclusion criteria for RIDTs accuracy for adults and children.
These are shown in Table 1. Conclusions: JMAS was
established in 1903, and more recently launched Ichushi-Web,
which was an exhaustive collection of Japanese biomedical
literature for the previous 110 years. The number of citations
in the database increases by more than 300,000 per year
and now exceeds nine million citations. Results of the RIDTs
accuracy review might be influenced by Japanese studies,
as four studies seemed to be eligible for the RIDTs accuracy
review. Adding a search of a Japanese literature retrieval
database might be recommended, especially for certain topics
considered in Japanese research studies.
RO 23.6
RO 23.5
How do the search results of the Ichushi-Web
influence systematic review? The case of
rapid influenza diagnostic tests (RIDTs)
Kojimahara N1 , Kawai F2 , Morizane T3
1
Tokyo Women’s Medical University, Japan; 2 St.
Luke’s
International University Library, Japan; 3 Medical Information
Network Distribution Service (MINDS), MINDS Guideline Center,
Japan
Background: Some clinical studies regarding diagnosis for
influenza were carried out by primary care physicians and
published only in Japanese. Medical settings in Japan tend to
have higher rates of adopting rapid influenza diagnostic tests
(RIDTs) for suspicious influenza-like illness in outpatients.
Searching Chinese biomedical databases:
current practice among Cochrane reviewers
Cohen JF1 , Korevaar DA1 , Wang J1 , Spijker R2 , Bossuyt PM1
1
The Netherlands; 2 Dutch Cochrane Centre, University Medical
Center, Utrecht, The Netherlands
Background: Cochrane recommends searching the literature
without language restrictions to avoid language bias. China is
currently ranked second in the world, after the USA, in terms of
the total number of published scientific papers. Objectives:
To describe the use that Cochrane reviewers make of Chinese
biomedical databases. Methods: In a systematic overview of
the Cochrane Library from inception to 2014, we assessed
Table 1
146
the extent to which Cochrane reviewers had searched
five major Chinese databases (China National Knowledge
Infrastructure (CNKI), Chinese Biomedical Literature (CBM),
Chinese Medical Current Content (CMCC), VIP, and WANFANG
(China Online Journals)). These Chinese databases index
about 2500 journals, of which less than 6% are also indexed in
MEDLINE. Results: Only 243 of 8680 published Cochrane
Systematic Reviews (3%) searched one or more of the
five major Chinese databases. All 243 systematic reviews
evaluated an intervention; 179 (74%) had at least one
co-author with a Chinese affiliation; 118 (49%) addressed
a topic in complementary and alternative medicine, such
as acupuncture. Conclusions: Cochrane reviewers rarely
include Chinese databases in their literature searches. Further
research should evaluate the consequences of this failure
to search Chinese databases when performing systematic
reviews.
RO 23.7
Supplementary searches of PubMed to
improve currency of MEDLINE and MEDLINE
In-Process searches via Ovid: evidence from
practice
Duffy S1 , Misso K1 , Noake C1 , Ross J1 , Stirk L1
1
Kleijnen Systematic Reviews Ltd, United Kingdom
Background: When conducting literature searches for
systematic reviews one bibliographic database, MEDLINE,
is almost always included. MEDLINE content can be searched
via numerous different search interfaces, the majority of
which are provided by fee-based subscription services (e.g.
OvidSP), although it can be accessed for free using PubMed.
Many information specialists would choose to search using
sophisticated interfaces (usually fee-based) that enable the
design of complex search strategies. PubMed has more
limited search capabilities, for example proximity searching is
not possible. However, 2% of PubMed records are not found
in MEDLINE, including newly published and ahead-of-print
articles. Objectives: To investigate whether conducting a
supplementary search of PubMed in addition to the main
MEDLINE (OvidSP) search is worthwhile. We will explore
whether an extra PubMed search can be conducted quickly
and retrieve unique studies that are included in the review.
Methods: Searches of PubMed will be conducted after
MEDLINE and MEDLINE In-Process (OvidSP) searches have
been completed. Due to different levels of functionality in
search interface, exact translation of the MEDLINE strategy
may not be possible; in this instance the objective is
not to duplicate searches, rather to supplement them.
Search strategies will be simplified, using phrase searching
predominantly. The searches will be limited to records not in
MEDLINE, using the following limit: (pubstatusaheadofprint
OR publisher[sb] OR pubmednotmedline[sb])
that PubMed searches are as current as possible can determine
the timeliness of the final review. It may also be worth
establishing search alerts in PubMed until final publication.
RO 23.8
The Cochrane EMBASE project for populating
the Cochrane Central Register of Controlled
Trials (CENTRAL): can we stop searching
EMBASE separately?
Faulkner S1 , Noel-Storr A2 , Eisinga A3
1 Cochrane ENT, United Kingdom; 2 Cochrane Dementia and
Cognitive Improvement, United Kingdom; 3 Cochrane UK, United
Kingdom
Background: The Cochrane EMBASE project aims to identify
randomized and quasi-randomized trials (RCTs/q-RCTs)
using highly sensitive, validated search strategies developed
by information specialists that are run in EMBASE.
The results are screened for eligibility and inclusion in
CENTRAL by volunteers using a crowdsourcing model
(http://www.cochranelibrary.com/help/central-creationdetails.html). One aim of this centralized search effort
is to improve Cochrane Review Group’s Trials Search
Co-ordinators’ (TSCs) efficiency by reducing duplication of
effort in identifying studies in EMBASE for their registers
and for CENTRAL. Objectives: To determine whether
the Cochrane ENT Group searches in EMBASE, designed
to populate their register and CENTRAL, identify any
unique RCTs/q-RCTs not found by the centralised EMBASE
search. Methods: The search strategies used are published
in the Cochrane ENT module of the Cochrane Library
(http://ow.ly/KtDMN). We will run these searches from May
2014 to May 2015 in Ovid EMBASE. We expect to retrieve
approximately 3500 references. We will also run the ENT
register searches in PubMed and CENTRAL, and upload
results to the ENT segment of the Cochrane Register of
Studies (CRS) first. We will then upload the records from
EMBASE into CRS. We will use the CRS deduplication tool
to exclude records already identified as relevant, or already
excluded as ineligible. We will then screen the remaining
references to identify any RCTs/q-RCTs in ENT that have
not been identified through the centralised EMBASE project,
the centralised PubMed search or the ENT Group’s search
of PubMed. Results: We will present the results of the
searches, and the deduplication and screening of the records
identified. If any additional trials are identified, we will inform
the EMBASE project team. Conclusions: We will seek to draw
conclusions on whether or not it is efficient for the Cochrane
ENT TSC to continue to search EMBASE separately, how these
data might inform the wider TSC community and future
centralised searching projects and how any anticipated
efficiency savings might impact on making skilled resources
available for other Cochrane projects.
Results: We will consider whether additional searching time
and screening burden from supplementary PubMed searches
generates unique included references. Conclusions: Ensuring
147
RO 23.9
Search summary table for systematic reviews
(SRs)
Bethel A1 , Rogers M1
1
Background: It is good practice in systematic reviews (SRs)
to publish the search strategy used for each database, as
recommended by the Cochrane Handbook for Systematic
Reviews of Interventions. These can be used to inform future
related reviews and update searches. However, it is currently
not considered necessary to publish information about which
databases held the relevant records and whether these were
found by the search. Tracking and publishing this information
as an additional search summary will aid future reviewers
in choosing and using resources more effectively and will
improve the efficiency of update searches. Objectives: To
develop a search summary table that can easily be used at the
end of all SRs to discover: 1. which databases the included
references came from; 2. which other databases searched
found the reference but it was excluded as a duplicate; 3.
which other databases searched contained the reference
but the search did not pick it up. Methods: The search
summary was designed and developed by the authors over
six systematic reviews including one Cochrane Systematic
Review. These were then used to establish whether a future
search on the same topic (or an updated search for the same
review) could be carried out more efficiently with targeted
database searching. Results: The results for each systematic
review show, via a summary table, in which databases the
included studies were found. An overall picture of the six
summaries has also been produced to discover whether any
pattern has emerged from the individual results. Conclusions:
The search summary will inform future systematic reviews
including updates to SRs or the update searches as part of the
SR. This will provide evidence to the review team about which
databases to search, particularly if there are time constraints.
It will also be another step in sharing and refining search data.
Background: Context and implementation of health
interventions have received increasing attention over the
past decade.
Of particular interest is their influence
on the effectiveness and reach of complex interventions.
Yet, much confusion exists around these terms and their
underlying concepts. Objectives: To analyze both concepts
in a state-of-the-art assessment for the health sciences in
order to create a common understanding of the use of
’context’ and ’implementation’ within systematic reviews
and Health Technology Assessments (HTA). Methods: We
performed two separate systematic searches for context
(EMBASE,MEDLINE) and implementation (Google Scholar)
to identify relevant theories, models and frameworks; 17
publications on context and 35 on implementation met our
inclusion criteria. Pragmatic utility (PU) concept analysis was
employed to analyze both concepts. PU comprises three
guiding principles: selection of literature, organization and
structuring of literature, and asking analytic questions of
literature. Both concepts were analyzed according to four
features of conceptual maturity, i.e. consensual definitions,
clear characteristics, fully described preconditions and
outcomes, and delineated boundaries. Results: Context
and implementation were highly intertwined, influencing
and interacting with each other. Context is defined as a
set of characteristics and circumstances that surround the
implementation effort. Implementation is conceptualized
as a planned and deliberately initiated effort with the
intention to put an intervention into practice.
The
concept of implementation presents largely consensual
definitions and relatively well-defined boundaries, while
distinguishing features, preconditions and outcomes are not
fully articulated. In contrast, definitions of context vary widely,
and boundaries with neighboring concepts, such as setting
and environment, are blurred; characteristics, preconditions
and outcomes are ill-defined. Conclusions: The maturity
of both concepts should be improved further to facilitate
operationalization in systematic reviews and HTA, e.g. by
developing a framework that will allow an assessment of both
concepts.
RO 24.2
Novel approaches of interpreting and
implementing healthcare evidence
RO 24.1
Implementation and context: a concept
analysis
Pfadenhauer LM1 , Booth A2 , Burns J1 , Gerhardus A3 , Hofmann
B4 , Lysdahl KB4 , Mozygemba K3 , Tummers M5 , Rehfuess EA1
1
Institute for Medical Informatics, Biometry and Epidemiology,
University of Munich, Germany; 2 University of Sheffield, United
Kingdom; 3 Institute of Public Health and Nursing Research,
University of Bremen, Germany; 4 Centre for Medical Ethics,
University of Oslo, Norway; 5 Radboud University Medical Center,
The Netherlands
Personalized music for residents with
dementia in a long term care setting. Putting
wheels on the evidence from the ALOIS
database
Hofstetter C1
1
Dementia and Cognitive Improvement Group, Canada
Background: At Extendicare Guildwood, a community
long-term care home in Toronto, the program director
decided to expand the music program and employ the
evidence that Dan Cohen utilized in his Music and Memory
Project. Another dimension was the addition of the 4Life
Foundation DJ Program youth to provide music expertise.
Objectives: To provide personalized music for the residents
on a regular basis and observe their reactions. To engage
148
the broader community, in particular ’at-risk’ youth, to create
an intergenerational communication. Methods: A search
of the ALOIS database for studies on music and dementia
produced 56 studies offering evidence on the benefit of
music therapy. A resident’s family donated iPods and CDs
to Guildwood, which allowed a wide variety of music to be
downloaded for the project. Personalized music folders were
created for each resident by the 4Life Foundation volunteers.
Results: Once weekly in group settings, residents experienced
personalized music through headphones. The young men
from the 4Life Foundation became part of the volunteer
group at Guildwood and formed relationships with the music
program participants. The residents who experienced the
music intervention showed changes in affect/mood and
changes in their demeanor and behaviour were observed
by staff. Agitation in dementia patients was reduced during
the intervention. Within 60 days of implementation of this
project at the Guildwood site, Extendicare Canada committed
to training and certifying the program directors in all 70 of
their long-term care homes in Ontario to deliver the Music
and Memory Project to their 10,000 residents. Conclusions:
The studies in the ALOIS database are a valuable resource for
improving care and quality of life in a long-term care setting.
RO 24.3
The use of statistics as a drunken man uses
lamp-posts: for support rather than
illumination
Corradini A1
1
Designskolen Kolding, Denmark
Statistics is the science that studies the collection, analysis,
interpretation and presentation of data. It makes it possible
to learn, to extract patterns from data as well as to measure,
and communicate uncertainty. The application of statistical
methods to scientific, industrial, or societal problems allows
trends to be determined in a specific context while also
predicting what may happen in the future. It is therefore
not surprising that in our fact-minded western cultures, the
language of statistics is something very powerful to wield.
When used correctly, statistical methods are very useful
tools for analyzing what is happening in the world around
us. On the other hand, statistics can easily be used to
oversimplify a phenomenon or situation, thus ultimately
resulting in confusing information, data misinterpretation,
and deceived readers. Even worse, statistics can be purposely
abused and misused in the attempt to sensationalize or inflate
data reports and opinion polls related to social, economic,
political, medical trends and conditions. Nonsense is what
results when the data analysis is conveyed with dishonesty,
with the wrong wordings or with the application of an incorrect
statistical approach. Nonsense is also what results when the
data analysis is presented to an audience that does not know
what the statistics mean. Statistics plays an important role in
the medical field and notably in the area of shared decision
making. Many patients wish to know their individual risk of
being diagnosed with a certain condition or what are their
chances of survival when they undergo a certain treatments.
Explaining data to patients is however not an easy task. In
this presentation, we review some use cases, which show
how statistical data can easily be misinterpreted and, as such,
should be avoided when using statistics for medical purposes.
RO 24.4
Change score or follow-up score? An
empirical evaluation of the impact of choice
of mean difference estimates
Fu R1 , Holmer H2
1 Pacific Northwest Evidence Based Practice Center, Department
of Public Health and Preventive Medicine, Oregon Health, Science
Univeristy, USA; 2 Department of Public Health and Preventive
Medicine, Oregon Health, Science Univeristy, USA
Background: In randomized controlled clinical trials,
continuous outcomes are typically measured at both baseline
and follow-up, and mean difference could be estimated by
using the change score from baseline, using the follow-up
scores, or using the analysis of covariance (ANCOVA) model.
When there is baseline imbalance, the ANCOVA estimate is
least biased, but often not reported. The impact of using the
change versus the follow-up score has not been well studied.
Objectives: Funded by the Agency for Healthcare Research
and Quality, this study was to empirically assess the impact
of using the change score versus the follow-up score on the
conclusion of meta-analysis (MA). Methods: We included a
total of 63 MAs (156 trials) from six comparative effectiveness
reviews. We evaluated differences in baseline scores on the
MA level and compared combined mean differences using
the change score or the follow-up score. Discrepancy in
conclusion occurs when one estimate (e.g. change score)
shows significant difference and the other (e.g. follow-up
score) does not. We also evaluated whether the impact
varied qualitatively by alternative random effect estimates.
Results: Based on the Dersimonian-Laird (DL) method, using
the change score versus follow-up score led to five out of the
63 MAs (7.9%) showing discrepancy in conclusions; and based
on the profile likelihood (PL) method, nine (14.3%) showed
discrepancy. Using the change score was more likely to show
a significant difference in effects between interventions (four
out of five using the DL method, and seven out of nine using
the PL method). The impact of the choice of the scores when
using the restricted maximum likelihood method was similar
to using the PL method. Using the Knapp-Hartung method led
to most (10) MAs showing discrepancy. A significant difference
in baseline scores did not necessarily lead to discrepancy in
conclusion. Conclusions: Using the change score versus
the follow-up score could lead to important discrepancies in
conclusions. Sensitivity analyses using both change scores
and final values should be conducted to check the robustness
of results to the choice of mean difference estimates.
149
RO 24.5
Is Kudos a valuable resource for Cochrane
authors?
Stewart G1 , Hilton J2 , Woodward G1
1 Wiley, United Kingdom; 2 Cochrane Editorial Unit, United Kingdom
Background: Kudos is an online platform that aims to help
researchers explain, enrich and share their publications for
greater research impact. Registered users of Kudos are able
to ‘claim’ and create profiles for their published articles and
increase readership and citations by adding links to resources,
creating lay summaries and sharing information via social
networks. In 2014, the publisher of the Cochrane Library,
Wiley, took part in a trial of the Kudos platform. Wiley sent the
details of over 25,000 published journal articles to Kudos. The
details of all new Cochrane Reviews published in 2014 were
also supplied. The contact author of every article was invited
by Kudos to register and ‘claim’ their published work on the
Kudos platform. The trial was extended to include all new and
updated Cochrane Reviews published in 2015. Objectives:
To establish how many Cochrane authors responded to the
invitation from Kudos and what level of interaction there was
with the Kudos platform. Why did some authors choose to use
the Kudos platform and why did others ignore the invitation
to use the platform? Did using the Kudos platform prove to be
a positive experience for Cochrane authors and did they see
increased levels of engagement in their work?
Methods: Using data provided by Kudos we will establish
the number of author registrations, the number of ‘claimed’
articles and the number of articles that have been enriched by
Cochrane authors. This dataset will be compared to data from
Wiley journal articles. A small survey of all Cochrane authors
contacted by Kudos will be conducted. Two case studies
will be used to provide an author’s perspective of using
the Kudos platform. Results: In 2014, 1986 Wiley authors
registered and 2233 articles were claimed by the registered
authors. Initial results show that interaction among Cochrane
authors was higher than among authors of other Wiley articles.
Conclusions: Data to inform the conclusions will be gathered
up until August 2015. This session will aim to establish whether
the Kudos platform should be made more widely available
to Cochrane authors as a platform to share the outcomes of
Cochrane Reviews.
RO 24.6
The use of content knowledge in the
assessment of bias in trials
Luijendijk H1
University Medical Center Groningen, The Netherlands
1
methods. Objectives: The aim of our paper is to show how
reviewers can use their content knowledge to identify bias
in trials. Methods: We introduce causal diagrams (DAGs)
to illustrate how structural bias occurs as result of baseline
differences, incorrect outcome measurement, and attrition.
Selective reporting is, by exclusion, the fourth source of
bias. Results: According to source of bias, we illustrate with
examples how content knowledge allows the identification
of (potential) bias, even if important trial methods such as
randomization have been applied correctly. We also show
how causal diagrams can help to understand the goals and
proper use of trials methods. Conclusions: Reviewers can use
their medical or other content knowledge to assess sources
of bias in trials. Causal diagrams complement the definitions
for correct application of trial methods in current assessment
tools.
RO 24.7
Cochrane Reviews in dentistry: making sense
of evidence at its highest level –an analysis
Mulimani P1
1 Melaka Manipal Medical College, Malaysia
Cochrane reviews are considered to provide scientific
evidence at its highest level by processing all available data
regarding a research question through rigorous tests and
standardized exacting specifications. Often the methodology,
to those not involved or to those viewing the information
from outside, appears complex or technical. Cochrane itself
has made provisions including plain language summaries,
Cochrane Clinical Answers, participation of consumers in
Review Groups or forums, to make the information access
easier and more meaningful to stakeholders including
lay-people, patients, care-givers and clinicians.
This
paper provides an analysis of the various channels that
are, and could be, employed to convey and disseminate
the importance of high-quality evidence and to interpret
and translate the findings of Cochrane research into
understandable, clinically relevant and practically applicable
information of utility to patients and doctors in the field of
dentistry. Additionally this paper will also discuss how the
importance of evidence, awareness of levels and quality of
evidence and significance of Cochrane Reviews and methods
can be made more pervasive and wide-spread among the
healthcare professionals through continued professional
development programs. We explore exposure to these during
the formative years of their professional lives, as students,
through various teaching-learning activities, thus creating
a newer generation of health professionals who are more
research and evidence-savvy.
Background: Current assessment tools focus on the methods
that are commonly used in trials to reduce bias. However,
these methods minimize but do not guarantee lack of bias. It
is important that sources of bias are assessed too. Moreover,
many reviewers do not have in-depth knowledge of trial
150
RO 24.8
A systematic review and external validation
of prediction models for gestational diabetes:
the RESPECT study
de Ruiter M1 , Kwee A1 , Naaktgeboren CA2 , Franx A1 , Moons KGM2 ,
Koster MPH1
1 Department Women and Baby, UMC Utrecht, The Netherlands;
2
Julius Center for Health Sciences and Primary Care, UMC Utrecht,
The Netherlands
Background: The rising incidence of gestational diabetes
(GDM) contributes to an increasing number of adverse
pregnancy outcomes (for example macrosomia and neonatal
hypoglycemia).
Currently, risk stratification in early
pregnancy is based on single medical and/or obstetrical
history risk factors. GDM, commonly diagnosed in late
pregnancy, is particularly suited for early prediction in
pregnancy. Timely recognition and treatment of GDM will offer
opportunities to improve pregnancy outcome. Objectives:
To identify and validate existing prediction models for GDM
in the first trimester of pregnancy. Methods: MEDLINE and
EMBASE were searched up to December 2014. First trimester
non-invasive prediction models for GDM in current pregnancy
were included as well as external validation studies thereof.
Eligibility was appraised by two reviewers. For each prediction
model, data were extracted according to the CHecklist for
critical Appraisal and data extraction for systematic Reviews
of prediction Modelling Studies (CHARMS). Models were
validated using data from a large prospective study, the
RESPECT cohort (3641 pregnancies of which 171 developed
GDM). Results: Our systematic review identified 20 articles,
16 of which were model development studies and four were
external validations. Of the 16 models, 12 could be validated
(Figure 1). In developmental studies C-statistics ranged from
0.63 to 0.82. Four models have been externally validated
with C-statistics from 0.60 to 0.76. In our validation study,
11 recalibrated models yielded C-statistics of 0.67 to 0.78.
At a fixed false positive rate of 10%, sensitivity ranged from
27% to 43%. Most of the models showed acceptable to good
calibration (Figure 2). Common predictor variables included
151
152
in the models were age, body mass index, ethnicity, history
of GDM and family history of diabetes mellitus. Conclusions:
Prediction models for GDM can be used to identify high
risk pregnancies. This allows early risk stratification and
enables customized targeted interventions. Screening will
contribute to the development of personalized obstetric care
and improve the utilization of health care resources.
The results of sensitivity analyses showed only three (0.1%)
reviews’ pooled effect sizes were affected by the data of those
studies. Conclusions: Most Cochrane Reviews have searched
the trial register, and small number of reviews include the
studies from registers. However, few reviews’ results are
affected by the studies from registers.
LRO 9.2
Trial registration
LRO 9.1
The impact of studies from a trial register for
the results of systematic review: a survey of
Cochrane reviews
Wei D1 , Qu Q2 , Chang X3 , Song X4 , Gao Y4 , Zhang Y4 , Ling N4 , Chen
Y1 , Yang K1
1
School of Basic Medical Sciences, Lanzhou University;
Key Laboratory of Evidence-Based Medicine and Knowledge
Translation of Gansu Province, Lanzhou, China; 2 The Second
Clinical Medical College of Lanzhou University, Lanzhou, China;
3
The First Clinical Medical College of Lanzhou University, Lanzhou,
China; 4 School of Basic Medical Sciences, Lanzhou University,
Lanzhou, China
Background: There are rigorous requirements for each
step of a systematic review in the Cochrane Handbook for
Systematic Reviews of Interventions. For instance, there must
be two independent reviewers when screening eligible studies
and extracting the data, which supports robust evidence.
Furthermore, it suggests that the trial register should be
searched for each Cochrane Review. To date, there are
more 20 international trial registers, such as the World Health
Organization International Clinical Trials Registry Platform
(WHO ICTRP), ClinicalTrials.gov, etc. However, is it really
significant to search the trial register for systematic review?
Objectives: To investigate the search of trial register in
Cochrane Reviews and analyze whether it would change the
results of the reviews. Methods: We sampled the Cochrane
Reviews published in 2013. Two reviewers extracted the data
independently. The extracted data included whether it had
searched trial register, the number and name of trial register,
the trials included in reviews, and whether the researchers
could achieve the data, etc. We assessed the impact of
studies from trial register for the pooled effect size of the
reviews through sensitivity analyses. Results: There were 992
Cochrane reviews published in 2013. Of those, 974 (98.2%) had
searched the registers (mean = 2, range: 1 to 20). The top five
frequently searched registers were Cochrane Group Registers
(91%, 890/974), ClinicalTrials.gov (43%, 423/974), WHO ICTRP
(34%, 331/974), CCT (19%, 181/974), and the metaRegister
of Controlled Trials (mRCT; 12%, 120/974). Thirty-two (3%)
reviews included the studies from registers, and nine (1%)
reviews synthesized the data from studies from registers.
Comparison of the attention paid to trial
registration in the Asia-Pacific region in the
last three years
Tam KW1 , Huang TW2 , Kuan YC1
1 Taipei Medical University, Shuang Ho Hospital, Taiwan;
2
HungKuang University, Taiwan
Background: Trial registration is known to improve research
transparency and will ultimately strengthen the validity
and value of the scientific evidence base. More and more
biomedical journals support clinical trial registration policies
after the requirement for registration was introduced by
several medical journals, led by the International Committee
of Medical Journal Editors (ICMJE). However, the attention
paid to trial registration varies among countries, particularly
in the Asia-Pacific Region. Objectives: To evaluate the
attention paid to trial registration in selected countries in
the Asia-Pacific Region in the last three years. Methods:
The published randomized controlled trials (RCTs) from each
country were extracted by restricting the ’affiliation’ field in
PubMed searches to individual country names. The selected
countries or regions consisted of Japan, South Korea,
Singapore, Hong Kong and Taiwan. Past (October 2011 to
March 2012) and present (June 2014 to January 2015) periods
were searched. We then compared the performance of each
country with regard to RCTs in terms of trial registration.
Results: We identified that from the five countries and
regions included in the study, 29.4% (past) and 85.2 %
(present) of the RCTs produced by Hong Kong had trials
that were registered, followed by Singapore (18.2% in the
past; 77.8% at present), Taiwan (13.9% in the past; 58.6%
at present), Japan (26.9% in the past; 53.6% at present),
and South Korea (22.8% in the past; 51.1% at present).
Conclusions: In the Asia-Pacific Region, trial registration as
part of the current research paradigm is of greater concern
in the present period. However, the proportion of trials that
are registered is still not satisfactory. Our results suggest that
researchers should pay more attention to registering their
studies on the registry platform.
153
LRO 9.3
LRO 9.4
Tuberculosis clinical trial activity on the
African continent: an analysis of tuberculosis
trials registered on the Pan African Clinical
Trials Registry and WHO-ICTRP
How complete are prostate cancer studies
conducted in Germany as displayed in the
national trial register of the Deutsche
Gesellschaft für Urologie (German
Association of Urology)?
Pienaar E1 , Kredo T1 , Abrams A1 , Lutje V2
1
SACC, South Africa; 2 Cochrane Infectious Diseases Review Group,
South Africa
Background: Thirty-four per cent of tuberculosis (TB)-related
deaths occur in Africa. Among people living with HIV, at
least one in four deaths can be attributed to TB, often in
resource-limited settings. The Pan African Clinical Trials
Registry (PACTR), the only African member of the World
Health Organization’s (WHO) Primary Registry Network,
contributes to WHO’s International Clinical Trial Registry
Platform’s (ICTRP) central database of clinical trials as well as
initiatives to strengthen regulatory and research frameworks
by providing a venue and support for trial registration. This
study describes the status of TB trials registered on PACTR and
the WHO-ICTRP. Objectives: To analyse TB interventions trials
conducted in Africa and registered on PACTR and WHO-ICTRP.
Methods: We searched PACTR and WHO-ICTRP for trials of TB
interventions. Data extraction included the number of trials,
their location, intervention studied, principal investigator’s
(PI) location, participant age range and funder. Descriptive
analysis was run in MS Excel.
Results: There are 430 studies registered on PACTR, with 44
trials reporting planned or ongoing TB research. On ICTRP
173 intervention trials were found as well as 25 TB-related
observational studies. Twenty-four PACTR trials explored
treatment, seven diagnosis, five prevention trials included two
vaccines, four education and counselling, and two supportive
care interventions. Sample sizes range from 12 to 27 000.
Twenty-one trials were single-centred in 10 countries; 23
multi-centre trials in 10 African countries; 38 PIs are African,
one trial had multiple PIs - nine from Europe and USA. Nine
trials received funding from the European and Developing
Countries Clinical Trials Partnership (EDCTP); none were
industry funded. Further analysis to include and compare
with trials registered on WHO-ICTRP is ongoing. Conclusions:
As PACTR registrations increase, the registry will become a
tool for regulatory and funding bodies to monitor the changing
TB trial landscape in Africa. Both PACTR and WHO-ICTRP are
valuable resources for systematic review authors to identify
ongoing TB trials, and also potential new review titles.
Schmidt S1 , Becker C1 , Jena S2 , Wullich B3 , Kunath F3
1 DGU, Germany; 2 DRKS, Germany; 3 Department of Urology,
University Hospital Erlangen, Germany
Background: The German Association of Urology (DGU)
initiated a project in 2010 to easily register clinical trials
within the German Clinical Trials Register (DRKS), named
DGU sub-register. The DRKS is a primary register of
the World Health Organization (WHO) International Clinical
Trials Registry Platform (ICTRP). Urology-related studies were
retrospectively included in the DGU sub-register. However, the
completeness of urologic trial registration in the DGU register
remains unclear. Objectives: To assess whether prostate
cancer studies conducted in Germany were registered in the
DGU sub-register. Methods: The WHO ICTRP database was
searched for all prostate cancer trials conducted in Germany.
The retrieved registration identifiers were used as search
terms to identify the associated trials registered in the DGU
sub-register. The increase in newly registered trials per year
was analyzed using linear regression. Results are presented
with 95% confidence intervals. A P value less than 0.05 was
considered statistically significant. Results: The search in
WHO ICTRP identified 872 records for 194 trials. The annual
number of registered trials identified in WHO ICTRP did not
increase significantly (P value 0.063). Of all the identified trials,
108 (56%) also had a DGU-DRKS identifier and 25 (13%) were
primarily registered in the DGU sub-register. While 67% of the
trials were registered in 2010 to 2015, 33% were registered
in 2009 or earlier. The annual increase was statistically
significant (0.89 more registered trials/year; 95% CI 0.18 to
1.59: P value 0.018). Conclusions: Only half of the prostate
cancer studies are displayed in the DGU sub-register. Since
the establishment of the DGU sub-register, trial registration
increased annually. Whether this development is due to
the availability of a user-friendly registration platform or
to an increased awareness of trial registration cannot be
determined. Furthermore, determining the true proportion of
registered studies remains a challenge. Results may be biased
by the completeness and quality of entries in the WHO ICTRP.
Keyword: ‘prostate cancer’, limitation: ‘recruitment location’
154
LRO 9.5
LRO 9.6
Prematurely discontinued randomized trials
are frequently labelled ‘completed’ in trial
registries: a systematic review
Epidemiology and reporting characteristics
of systematic review protocols
Schandelmaier S1 , Al Turki R1 , Olu KK1 , Frei R1 , Von Niederhäusern
B1 , Agarwal A2 , Briel M1
1
University Hospital, Basel, Switzerland; 2 McMaster University,
Hamilton, Canada
Background: Trial registries such as clinicaltrials.gov typically
include information about the status of a randomized
controlled trial (RCT), i.e. whether it is ongoing, completed,
or discontinued. Trial registries can be a useful resource
for systematic reviewers who are searching for unpublished
RCTs, such as ongoing or often discontinued RCTs. The
accuracy of trial status information in registries has never
been investigated. Objective: To examine whether trial
registries accurately label discontinued RCTs as ‘discontinued’
and provide reasons for RCT discontinuation; and to identify
factors associated with inaccurate trial status information.
Methods: Systematic reviews of published RCTs reported
as discontinued and registered. We systematically searched
MEDLINE and EMBASE (from 2010 to 2014). Pairs of reviewers
independently extracted data from publications and
corresponding registries using pre-piloted data extraction
forms. We performed multivariable regression to identify
risk factors for inaccurate trial status information. Results:
We included 173 discontinued RCTs that were registered
in various trial registries, most frequently in clinicaltrials.gov
(77%). RCTs were mostly discontinued due to slow recruitment
(62%), futility (19%), or harm (17%). Of the 173 discontinued
RCTs, 45% were accurately labelled as discontinued; the
remaining RCTs were wrongly labelled as completed (40%),
suspended (9%), on-going (5%), or not started yet (1%).
Most RCTs that were accurately labelled also provided the
accurate reason for discontinuation (57/77, 74%; Table 1).
Significantly, accurate registration of trial status became
more frequent over time (adjusted odds ratio 1.16 per year;
95% confidence interval 1.04 to 1.3) whereas trial features
such as industry sponsorship, multiple centers, or sample
size were not significantly associated with accurate trial
status information. Conclusion: Less than half of published
discontinued RCTs were accurately labelled as discontinued
in registries. Systematic reviewers searching trial registries
for eligible trials should view trial status information with
caution.
Shamseer L1 , Page M2 , Reid E3 , Turner L4 , Sampson M5 , Moher D1
1 Ottawa Hospital Research Institute; University of Ottawa, Canada;
2
School of Public Health and Preventive Medicine, Monash
University, Australia; 3 Department of Pharmacy, Vancouver
General Hospital, Canada; 4 Ottawa Hospital Research Institute,
Canada; 5 Children’s Hospital of Eastern Ontario, Canada
Background: Protocol development is a fundamental
preliminary step in the systematic review process, enabling a
priori consideration of review methods and preventing ad hoc
decision-making during the review process. When available,
protocols can facilitate detection of selective reporting when
compared to completed reviews. While protocols are a
mandatory component of Cochrane Reviews, most reviews
are done outside Cochrane and few report working from
protocols. The reporting completeness of systematic review
protocols has not previously been examined. Objectives: This
study will assess: the proportion of systematic reviews that
have been registered and/or have an available protocol; and
for those with a protocol, the epidemiological characteristics
and completeness of reporting. Methods: Three-hundred
systematic reviews indexed in MEDLINE in February 2014, 14%
of which are Cochrane Feviews, form the sample from which
protocols and registration information will be sought. We
will search PROPSERO to determine the number of registered
reviews (post-October 2013 only for Cochrane Reviews) and
will use a systematic search to identify whether a public
protocol exists. For all identified protocols, epidemiological
characteristics (e.g. number and country of authors, ICD-10
category) will be extracted and reporting characteristics will
be assessed based on items from the PRISMA extension
for protocols. Data will be extracted by a single assessor
after meeting a minimum threshold of agreement (80%)
during independent pilot-testing by two assessors. We will
summarise data as proportions for dichotomous items and
median and interquartile range for continuous items. We will
compare reporting between Cochrane versus non-Cochrane
protocols and registered versus non-registered protocols.
Results: Results will be available for presentation at the
Colloquium. Significance: This study will provide data on
the prevalence of review registration, availability of review
protocols, and the completeness of information reported in
SR protocols. This study will be the first of its kind to provide
evidence on the strengths and weaknesses of SR protocol
reporting.
Table 1
155
Missing data
LRO 10.1
How much study participant data is missing
from our trials?
Kirkham J1 , Dwan K2 , Blümle A3 , von Elm E4 , Williamson P1
1 University of Liverpool, United Kingdom; 2 Cochrane Editorial Unit,
United Kingdom; 3 German Cochrane Center, Germany; 4 Cochrane
Switzerland, Switzerland
Background: Study publication bias and outcome reporting
bias (ORB) have been recognised as two threats that can
affect the validity of systematic reviews due to problems
with missing outcome data. There is strong evidence of
an association between significant results and publication;
studies that report positive or significant results are more
likely to be published and outcomes that are statistically
significant have higher odds of being fully reported [1].
Objectives: To estimate the proportion of missing data due
to lack of publication of the studies and the proportion of
missing outcome data within a published study. Methods:
Data were taken from protocols of randomised controlled
trials submitted to the research ethics committee of the
University of Freiburg (Germany) between 2000 and 2002
and associated full published articles. The total amount
of published and unpublished outcome data from all study
participants was extracted and the proportion of missing data
from both unpublished and published studies computed.
Missing data from unpublished studies was taken from
planned sample sizes from study protocols. A sensitivity
analysis was undertaken to account for partially reported
outcome data. Results: From 259 studies in the study cohort,
167 were published and 92 were unpublished. Half (51%;
1,288,719 of 2,522,010) of the participant outcome data from
the 259 studies was missing; 39% of this missingness was
attributable to missing data from published studies and
12% from unpublished studies. The sensitivity analysis
revealed that the majority of missing data from published
studies was linked to data being reported in a way that
could not be included in a meta-analysis. Conclusions:
Missing participant outcomes data from both published and
unpublished studies is frequent. Preventative measures
and potential statistical solutions for reducing the impact
of missing data in meta-analyses will be discussed.
Reference
[1] Dwan K M, Gamble C, Williamson P R, Kirkham J J.
Systematic review of the empirical evidence of study
publication bias and outcome reporting bias - an updated
review. PLoS ONE 2013: 8: e66844.
LRO 10.2
Extent and handling of missing dichotomous
outcome data for participants in 672 trials
included in 100 Cochrane and non-Cochrane
systematic reviews
Kahale LA1 , Diab B1 , Mustafa R2 , Busse JW3 , Agarwal A3 , Waziry
R1 , Li L4 , Lopes LC5 , Koujanian S6 , Dakik A1 , Chang Y3 , Guyatt
GH3 , Akl EA1
1
American University of Beirut, Lebanon; 2 University of
Missouri-Kansas City, USA; 3 McMaster University, Canada; 4 Sichuan
University, China; 5 University of Sorocaba, Brazil; 6 Lebanese
American University, Lebanon
Background: Reports of randomized clinical trials (RCTs) do
not always specify whether participants belonging to certain
categories (e.g. those who withdraw consent, non-compliers)
were followed-up for the outcome(s) of interest or not (i.e.
missing participant data (MPD)). Systematic reviewers may,
therefore, have to make assumptions on whether data
from these participants is missing. Objectives: To describe
characteristics, reporting and handling of categories of
participants that could be potentially counted as having
MPD in RCTs included in Cochrane and non-Cochrane
systematic reviews (SRs). Methods: We selected a random
sample of 100 Cochrane and non-Cochrane SRs published
in 2012. Eligible SRs reported a group-level meta-analysis
of a patient-important dichotomous efficacy outcome,
with a statistically significant effect estimate.
Ten
reviewers, working in pairs, independently extracted
data from the RCTs included in eligible meta-analyses.
We focused on 10 categories that could be potentially
counted as MPD: ’participants deemed ineligible after
randomization’, ’did not receive any treatment’, ’withdrew
consent’, ’outcome not assessable’, ’explained and
unexplained lost to follow-up’, ’dead’, ’experienced adverse
events’, ’non-compliant’, ’discontinued prematurely’, and
’cross-over’. We specifically assessed: (1) the number of
participants in those categories; (2) whether trialists explicitly
reported on the follow-up status of those participants; and
(3) how trialists handled those participants in their analysis.
Results: We included a total of 672 RCTs included in 100
eligible Cochrane and non-Cochrane SRs. We are in the
process of data abstraction. We will present the findings of
the study at the Colloquium. Conclusions: Results will inform
a better understanding of trial reporting on the follow-up,
and handling of categories with potentially missing outcome
data, an issue of importance for systematic reviewers and for
standards of trial reporting.
156
LRO 10.3
LRO 10.4
Reporting and analysis of missing participant
continuous data in randomized controlled
trials (LOST-IT-II): a systematic survey
Reported versus actual analytical methods to
handle missing participant data in
meta-analysis: a survey of 100 systematic
reviews
Zhang Y1 , Alexander Kennedy S2 , Abu Bakar Aloweni F3 , Flórez
ID4 , Colunga L5 , Craig S6 , Zhang S7 , Li A1 , Cruz Lopes L8 , Fei Y9 ,
Alexander P1 , J. Riva J10 , Sadeghirad B1 , Agarwal A11 , Akl E12 ,
Schünemann HJ13 , Thabane L1 , Guyatt G13
1
University, Canada; 2 Department of Medicine, McMaster University,
Canada; 3 Singapore General Hospital, Singapore; 4 Universidad de
Antioquia, Medellin and McMaster University, Colombia, Canada;
5
Hospital Civil de Guadalajara, ”Fray Antonio Alcalde”, México;
6 Michael G DeGroote Institute for Pain Research and Care,
McMaster University, Canada; 7 Medical Affairs, GSK, Canada;
8
Universidade de Sorocaba, São Paulo, Brazil; 9 Beijing University
of Chinese Medicine, McMaster University, China, Canada;
10
Department of Family Medicine, McMaster University, Canada;
11
Faculty of Medicine, University of Toronto, Canada; 12 Clinical
Epidemiology Unit and Center for Systematic Reviews in Health
PolicyandSystemsResearch(SPARK),AmericanUniversityofBeirut,
Lebanon; 13 Department of Medicine and Clinical Epidemiology and
Biostatistics, McMaster University, Canada
Background: Missing participant data (MPD) can bias trial
results and conclusions if the missingness is associated
with the occurrence of the outcome of interest. No study
has summarized the reporting and analytic approaches
of MPD that authors have used for continuous outcomes
in randomized controlled trials (RCTs). Objective: Our
objectives are to assess: (1) how authors report MPD for
continuous outcomes; (2) the analytic methods used for
primary analysis and sensitivity analysis to address MPD;
and (3) the impact of these sensitivity analyses on the
conclusions. Method: We are conducting a systematic
survey of RCTs published in 2014 in core medical journals.
We include RCTs reporting at least one patient-important
outcome analyzed as a continuous variable. We randomly
sample RCTs aiming at a sample size of 200. We will calculate
the proportion of RCTs that explicitly reported whether MPD
occurred, percentage of randomized participants with MPD
for the continuous outcome, the methods used to handle
MPD in the primary and sensitivity analyses, and how often
the conclusions are interpreted considering the sensitivity
analyses conducted and whether consideration of MPD
changes the conclusion. We will conduct regression analyses
with independent variables general trial characteristics (e.g.
type of funding, type of intervention) and methodological
trial characteristics (e.g. allocation concealment, length of
follow-up), to establish factors associated with reporting,
analysis, and conclusions. Conclusion: Our methodological
survey will have important implications for both trialists and
users of trial evidence in interpretation of the findings from
RCTs with MPD on continuous outcomes.
Kahale L1 , Diab B1 , Mustafa R2 , Busse JW3 , Agarwal A3 , Waziry R1 ,
Li L4 , Lopes LC5 , Koujanian S6 , Dakik A1 , Chang Y3 , Guyatt GH3 ,
Akl EA1
1
American University of Beirut, Lebanon; 2 University of
Missouri-Kansas City, USA; 3 McMaster University, Canada; 4 Sichuan
University, China; 5 University of Sorocaba, Brazil; 6 Lebanese
American University, Lebanon
Background: Systematic review authors do not typically
report the analytical methods they used to handle missing
participant data (MPD) in meta-analyses. Moreover, when
methods are reported, the concordance with reviewers’
actual analyses is uncertain. Objectives: To determine which
analytical methods systematic review authors actually used
to deal with MPD in their meta-analyses of dichotomous
outcomes, and whether there is concordance with the
approach described in their methods section. Methods:
Eligible systematic reviews (SRs) reported a group-level
meta-analysis of a patient-important dichotomous efficacy
outcome, with a statistically significant effect estimate. We
acquired the full texts of all trials included in the meta-analyses
of interest. Ten reviewers working in pairs will independently
abstract statistical data relevant to the meta-analysis of
interest from both the SR (numerator, denominator) and
the corresponding RCTs (number randomized, number with
missing data, number of observed events). Then, we will
compare the SR data to the corresponding trial data. Based
on these comparisons, we will classify the analytical method
the SR authors actually used to deal with MPD into four
categories: ’complete case analysis’, ’making assumptions
for MPD’, ’using the trialists’ assumptions’, or ’excluding trials
with high rate of MPD’. Then, we will assess the consistency of
the method across trials included in the same meta-analysis,
and concordance with the method SR authors report to have
used. Results: We included a random sample of 100 Cochrane
and non-Cochrane SRs published in 2012. We identified a
total of 672 RCTs included in these SRs. We are in the process
of data abstraction. We will present the findings of the study
at the Colloquium. Conclusions: The findings of our study are
likely to reinforce the importance of transparency in reporting
and conducting SRs, particularly when dealing with MPD.
157
LRO 10.5
Prognostic models for cardiovascular disease
risk in the general population: a systematic
review
Damen J1 , Hooft L1 , Schuit E1 , Debray T1 , Collins GS2 , Tzoulaki I3 ,
Lassale CM4 , Siontis G5 , Black J6 , Heus P1 , van der Schouw YT7 ,
Peelen L7 , Moons K1
1 Dutch Cochrane Centre, Julius Center for Health Sciences and
Primary Care, University Medical Center Utrecht, The Netherlands;
2
Centre for Statistics in Medicine, NDORMS, Botnar Research
Centre, University of Oxford, United Kingdom; 3 Department of
Epidemiology and Biostatistics, Imperial College of Medicine,
London, United Kingdom; 4 Department of Epidemiology and
Biostatistics, School of Public Health, Imperial College London,
United Kingdom; 5 Department of Cardiology, Bern University
Hospital, Switzerland; 6 MRC Epidemiology Unit, University of
Cambridge School of Clinical Medicine, United Kingdom; 7 Julius
Center for Health Sciences and Primary Care, University Medical
Center Utrecht, The Netherlands
Background: Cardiovascular disease (CVD) is a leading cause
of morbidity and the leading cause of mortality worldwide.
Many prediction models have been developed to assess
individual CVD risk to allow targeting of preventive treatment.
Objectives: To give an overview of all prognostic models that
predict future risk of CVD in the general population, and to
describe predicted outcomes, study populations and included
predictors. Methods: In June 2013 a systematic search
was performed in MEDLINE and Embase to identify studies
that described the development, validation or incremental
value of a multivariable prognostic model predicting CVD
in the general population. Results: Out of 9671 papers
identified, 314 were included, describing the development
of 373 prognostic models, 519 external validations and
278 incremental value assessments. In total 132 models
(35%) were externally validated, of which 70 models (19%)
were validated by independent researchers. Most models
were developed in Europe (n = 258), and predicted risk of
coronary heart disease (n = 115, Table 1) over a 10-year
period (n = 211). Furthermore, most prevalent predictors
were age and smoking (n = 323 and n = 332 respectively,
Figure 1), frequently with separate models for males or females
(n = 256). Discrimination and calibration were reported for
only 62% and 52% of the external validations respectively.
Crucial clinical and methodological information was often
missing, and when reported, substantial heterogeneity in
predictor and outcome definitions was seen between models.
For 53 models the prediction time horizon was not reported
and for 56 models the intercepts or baseline hazards were
not reported, making it impossible to use for individual risk
prediction. Conclusion: In a similar manner to various other
clinical domains, there is an excess of prognostic models
predicting CVD in the general population. Additionally, the
usefulness of most models remains unclear, due to incomplete
presentation, lack of external validation, and heterogeneity in
predicted outcomes and study populations. Future research
should focus on validating, updating, meta-analyzing and
studying impact of models.
Table 1
158
Figure 1
LRO 10.6
Meta-analysis methods for better validation
of prediction models
Steyerberg E1
1
Erasmus MC, The Netherlands
Background: Prediction models are increasingly developed
for many diagnostic and prognostic endpoints, and
increasingly based on individual patient data from multiple
studies. Validation of predictions from such models is
important, but many researchers use inefficient designs
and analyses. Objectives: We aimed to evaluate the role
of meta-analytic methods in the validation of prediction
models. Methods: We considered 15 studies of patients
with traumatic brain injury (n = 11,026), where we predicted
6-month mortality. Prediction models were constructed in
each of the 15 studies, and in the pooled data set. Various
approaches to internal and external validation were explored.
Results: Logistic regression models included 10 predictors,
with good overall discriminatory ability (median apparent c
Figure 1
159
statistic in the 15 studies 0.81). For internal validation per
study, we confirmed that a random split-sample approach was
inefficient and unstable, in contrast to bootstrap resampling.
In the pooled data set, an internal-external validation
procedure provided insight in the substantial variability in
discriminatory ability (c statistic) by study, as well as variability
in calibration (study-specific intercept and calibration slope).
This variability could well be quantified by random-effects
meta-analysis modeling and I2 (I-squared) statistics (> 50%).
The predictive distribution of predictions for patients in future
studies could hence be estimated, and was much wider than
that based on a fixed-effect analysis in the pooled data set.
Conclusions: We conclude that meta-analytic methods are
very useful to obtain insight in the validity of prediction
models. Quantification of heterogeneity in predicted risks
should become the focus of validation studies, rather than
confirmation of performance.
tests and comparisons by each study) and quality assessment,
as well as to generate paper-based data extraction- and
reporting-forms. (See Fig. 1)
Results: We present the feasibility of a customized interactive
database using the example of an ongoing Cochrane DTA
review. Conclusions: Based on tools that are already
available, a custom-made interactive database to fulfill the
special needs of DTA-collaborators proved successful and
productive. This tool will be made available on the ACE
website to be used by other groups with similar needs.
LRO 11.2
The reference standard for diagnostic
imaging test accuracy
Reed M1
1
Canadian Association of Radiologists, Canada
Diagnostic test accuracy reviews
LRO 11.1
Generating an interactive database for
international collaboration on a Cochrane
diagnostic test accuracy-review
Roth D1 , Lee A2 , Hovhannisyan K3 , Hartmann A1 , Pace NL4 ,
Herkner H1
1 Medical University of Vienna, Department of Emergency
Medicine, Austria; 2 Department of Anaesthesia and Intensive
Care, The Chinese University of Hong Kong, Hong Kong;
3 Rigshospitalet, The Cochrane Anaesthesia Review Group,
Copenhagen, Denmark; 4 Department of Anesthesiology, University
of Utah, Salt Lake City, USA
Background: Reviews of diagnostic test accuracy (DTA)
are increasingly gaining importance within Cochrane, and
Cochrane tools, such as RevMan, are constantly being
updated to serve the requirements of conducting such types
of review. However, not all tasks can currently be solved using
standard Cochrane tools. Objectives: The authors of the
DTA-review ’Airway physical examination tests for detection
of difficult airway management in apparently normal patients’
of the Cochrane Anaesthesia, Critical and Emergency Care
Group (ACE) strived for a tool to facilitate data-collection
for their team scattered across four countries and three
continents. The need for such a tool was aggravated by
the fact, that this project represents a ’multidimensional’
review with multiple target conditions, a multitude of
tests, serving both as standard- and index-tests, and that
each individual study usually includes a large number of
comparisons. Methods: Based on the QUADAS 2 database
by Whiting et al, an MS Access-database to be hosted by a
filehosting-service was developed to collect data on measures
of DTA on multiple levels (multiple target conditions, index
Background: The reference standard for measuring
diagnostic test accuracy ideally should be error-free in
establishing the diagnosis of the condition of interest and
should be applicable to all of the population being tested.
Unfortunately it is virtually impossible to find such reference
standards to measure the accuracy of diagnostic imaging
(DI) tests. Objective: the objective of this presentation is
to describe and illustrate the principal barriers to finding
an ideal reference standard for assessing DI test accuracy.
Analysis: there are three main barriers to finding an ideal
reference standard for assessing the accuracy of DI tests. 1.
The high quality of current DI technology. Pathology is usually
considered to be the ideal reference standard for assessing
DI test accuracy. However, DI very often provides imaging of
pathology which is as good as anatomical pathology. Common
examples would be fractures, gallstones, appendicitis and
intracranial hemorrhage. 2. The reference standard cannot
ethically be applied to all study participants. One of the
most common uses for DI is to diagnose conditions, such
as tumours, injuries or abscesses, which require surgical
intervention. The findings at surgery are an excellent reference
standard. However, surgery can only ethically be performed
on those patients with a positive DI test. 3. There is no
practical reference standard for the condition being assessed.
Examples of this would include such common conditions
as pneumonia and pulmonary oedema for which DI is one
of the primary diagnostic tests. Conclusion: finding a
reference standard for assessing DI test accuracy is extremely
difficult. Methodologists and radiologists need to work
together to establish appropriate criteria for determining
reference standards for assessing DI test accuracy.
160
LRO 11.3
Methods for assess validity of diagnostic tests
in rare diseases
Canon L1 , Perez A1 , Avellaneda P1 , Serrano P1
1 Instituto de Evaluación Tecnológica en Salud, Colombia
Background: Rare diseases are those with a prevalence of
1 per 5000 individuals or, within the European Union, fewer
than 5 per 10,000 persons. According to Colombian law
there should be full access to diagnostic technologies for
rare diseases in our country. In order to prioritize them, 11
reviews of diagnostic validity were developed, upon which
this paper is based. Objectives: To describe the findings
of methodology for the development of diagnostic validity
reviews for rare diseases. Methods: Eleven assessments of
diagnostic utility tests were done in Colombia in 2014 to
perform this descriptive study. The most important aspects
in the development of reviews for low-prevalence diseases
were identified with little evidence available. Results: 72% of
assessed diseases had specific prevalence data for Colombia.
The built-in assessments evidence was 9% of systematic
literature reviews, 27.2% of cohort studies, 18% of diagnostic
test studies and 36.3% of descriptive studies. No evidence
was identified to answer the research question in 27% of the
11 assessments done. The quality of the studies was assessed
with AMSTAR (A MeaSurement Tool to Assess systematic
Reviews; 9%) and QUADAS 2 (Quality Assessment of Diagnostic
Accuracy Studies tool; 36%). Due to the study type 45% of the
evidence did not have quality assessment. In two of the 11
assessments done combined effect analysis was performed.
Conclusions: Despite the limited evidence for rare diseases,
systematic and reproducible evaluations can be done in order
to have an input for decision-making in the countries. The
prevalence and pathophysiology regarding rare diseases do
not allow the performance of studies of high methodological
quality, however, the evidence from descriptive studies is
quite useful in this context. Given the limitations of the
sparse evidence from observational studies and descriptive
designs it must be reported with high quality, which means a
challenge for the production of systematic literature reviews.
LRO 11.4
Summarising diagnostic accuracy using a
single parameter
Simmonds M1
for Reviews and Dissemination, University of York, United
Kingdom
1 Centre
Background: In meta-analyses of diagnostic test accuracy,
results are often presented as separate analyses of sensitivity
and specificity, but this ignores the correlation between these
parameters. More complex approaches such as the bivariate
approach or hierarchical summary receiver operating
characteristic (HSROC) curve model do account for the
correlation, but are more difficult to implement and interpret.
Objectives: To demonstrate how diagnostic accuracy may
be more simply summarised using a single parameter rather
than sensitivity and specificity. Summarising the diagnostic
accuracy of a test in a study with a single parameter
permits the use of standard meta-analysis techniques for
pooling results across studies, making interpretation simpler.
Four parameters will be considered: diagnostic odds ratio,
diagnostic standardised mean difference, Lehmann’s index
and Youden’s index. Methods: The four parameters will
be described, and shown to be related to the underlying
distribution of the diagnostic test. It will be shown how all four
parameters may easily be estimated using simple regression
models. The four new parameters will be compared,
and compared to existing methods, using commonly cited
examples of diagnostic test accuracy meta-analyses. Results:
As will be seen from various examples, the four single
parameter methods offer a generally robust alternative
to the HSROC model. Summary results and conclusions
can vary substantially between methods, demonstrating
that summaries of diagnostic accuracy may be sensitive
to assumptions about the underlying distribution of the
diagnostic test. Conclusions: Summarising diagnostic
accuracy using one of the proposed single parameter
methods offers an alternative approach to more complex
methods such as the HSROC model. Results of these
analyses can easily be presented on a forest plot, making
interpretation of results simpler and making interpretation of
heterogeneity and comparisons between different diagnostic
tests straightforward.
LRO 11.5
Using full information on cut-offs per study to
determine an optimal cut-off in diagnostic
test accuracy reviews
Rücker G1 , Steinhauser S1 , Schumacher M1
1
Medical Center, University of Freiburg, Germany
Background: In diagnostic test accuracy (DTA) reviews, it
is often assumed that each study reports only one pair of
specificity (Sp) and sensitivity (Se). The established bivariate
model considers the joint distribution of Sp and Se across
studies. However, in primary studies (Sp, Se) pairs are often
reported for two or more cut-offs, and the cut-off values
are reported as well. Objectives: To use this additional
information for modelling the distributions of the underlying
biomarker for diseased and non-diseased individuals and to
determine an optimal cut-off. Methods: We assume that
for some or all DTA studies in a meta-analysis a number of
cut-offs is reported, each with corresponding estimates of Sp
and Se. These provide direct information about the empirical
cumulative distribution function (ecdf) for both groups of
individuals. We propose a class of hierarchical models for the
distribution including study as a fixed or random factor. These
models lead to average ecdfs for both groups of individuals. As
the difference between these is the Youden index as a function
of the cut-off, an optimal cut-off can be found by maximising
161
this function. A summary receiver operating characteristic
(ROC) curve is estimated based on the distributions. Results:
The approach is demonstrated on three meta-analyses of
alcohol screening, procalcitonin as a marker for sepsis and
diagnosis of asthma. Conclusions: If there are a number of
studies reporting at least two cut-offs with (Sp, Se) per study,
we can determine an optimal cut-off and estimate a summary
ROC curve based on all available information from the primary
studies.
LRO 11.6
Time trends in summary estimates from
meta-analyses of diagnostic accuracy studies
Cohen JF1 , Korevaar DA1 , Wang J1 , Leeflang MM1 , Bossuyt PM1
1
The Netherlands
Background: Summary estimates from meta-analyses of
diagnostic accuracy studies can change over time, as evidence
accumulates. Objectives: To evaluate trends in changes
over time in summary estimates of diagnostic accuracy.
Methods: We included 54 meta-analyses with at least five
primary studies from 42 MEDLINE-indexed systematic reviews
published in English between September 2011 and January
2012, reflecting 706 diagnostic accuracy studies, with 338,109
participants in total. Within each meta-analysis, we ranked
studies by publication date and applied random-effects
cumulative meta-analysis, to follow how summary estimates
of diagnostic odds ratio (DOR), sensitivity and specificity
would have evolved over time. Time trends were assessed
by fitting a weighted linear regression model of the summary
accuracy estimate against rank of publication. A negative
slope indicates a decrease in summary accuracy estimates
over time. Results: Slopes ranged from -0.80 to 0.56 for DOR,
−0.44 to 0.37 for sensitivity and -0.37 to 0.36 for specificity.
We found a significant time trend in at least one accuracy
measure in 42 (78%, 95% confidence interval (CI) 65% to 87%)
meta-analyses. Among a total of 93 significant time trends in
DOR, sensitivity or specificity, 62 (67%, 95% CI 57% to 75%)
were negative, indicating that summary estimates of accuracy
tended to decrease over time. Conclusions: Changes in
summary estimates are relatively frequent in meta-analyses
of diagnostic accuracy studies, with estimates based on more
studies typically reflecting weaker diagnostic performance.
The results from early meta-analyses of diagnostic accuracy
studies should be considered with caution before drawing
conclusions about the performance of medical tests.
Decision-making: Policy and public
health
LRO 12.1
Evaluation of an organizational strategy for
evidence-informed decision making
Ward M1 , Bryant B1 , Dobbins M2 , Peirson L2
1 Peel Public Health, Canada; 2 McMaster University, Canada
Background: Peel Public Health is one of Canada’s
largest public health departments with 630 staff serving
1.3 million residents. In 2009 we set a 10-year strategy
for evidence-informed decision making (EIDM). It involved
development of a rapid review process, extensive work force
development, consistent funding, supportive infrastructure,
and highly visible leadership. Objectives: To conduct a
midpoint evaluation of the strategy to assess the use and
impact of research in program decisions. Methods: The
strategy was evaluated in five ways. 1. Survey of senior
leaders at three points about the impact of research in their
division. 2. Assessment of the practice outcome from each
of 40 rapid reviews. 3. Uptake of EIDM training by staff at
all levels of the organization. 4. Assessment of the change
in skills to find, appraise and apply research by an external
researcher. 5. Case study at three points to assess the
penetration of EIDM into the organization. Results: The main
barriers to research use in 2009 were lack of access to full text
articles, inconsistent ability to assess the quality of research,
and no time. By 2014 we belonged to a library consortium
and had two librarians, had trained 130 staff in EIDM, and had
mentored 40 specialist-manager teams in the rapid review
process. Skills in finding, appraising and applying research
were tested and had improved significantly. Program changes
arising from rapid reviews included five to stop a program, 20
to change a program, 12 to start a new program and nine to
maintain the status quo. The case study revealed high visibility
of the strategy for all staff, but minimal penetration to front
line. Conclusions: Creating a culture for research use can be
achieved by an organization. It requires strong and persistent
senior leadership, investment in necessary infrastructure
and funding, intensive staff training and mentorship, and
an intentional change management strategy. Facilitators
include close relationships with researchers, good quality
research syntheses and the availability of well-tested training
programs. Challenges include staff turnover, competing
priorities, and organizational fatigue.
162
LRO 12.2
LRO 12.3
Developing a conceptual framework to
support coverage decisions for vaccines
adoption: a Delphi consensus process
Influencing policy for promoting appropriate
prescribing and medicines use: training key
decision makers to use a database of
systematic reviews
González-Lorenzo M1 , Tirani M2 , Piatti A3 , Coppola L3 , Gramegna
M3 , Auxilia F2 , Demicheli V4 , Banzi R5 , Bertelè V5 , Moja L1
1 Clinical Epidemiology Unit, IRCCS Galeazzi Orthopaedic Institute,
Milan; Diparment of Biomedichal Sciences for Health, University
of Milan, Italy; 2 Deparment of Biomedichal Sciences for Health,
University of Milan, Italy; 3 Direzione Generale Salute, UO Governo
della Prevenzione e Tutela Sanitaria, Regione Lombardia, Milan,
Italy; 4 Servizio Regionale di Riferimento per l’Epidemiologia,
SSEpi-SeREMI - CochraneVaccines Field, Azienda Sanitaria Locale
ASL AL, Alessandria, Italy; 5 Laboratory of Regulatory Policies,
IRCCS-Mario Negri Institute for Pharmacological Research, Milan,
Italy
Background: Decision makers must make choices about
which vaccines to incorporate into the prevention programs
of a National Health Service. A transparent, unbiased, and
comprehensive framework based on evidence-based criteria
is necessary to guide decision-making on vaccine adoption.
We developed a potential framework to support vaccine
coverage decisions based on the DECIDE Evidence to Decision
framework. Objectives: This study aims to evaluate and
validate a framework to support vaccine coverage decisions,
and to explore alternative ways of presenting framework
criteria to support policy decision-making. Methods: We
adopted a Delphi consensus process to engage a group of
experienced healthcare professionals. We constructed a
questionnaire based on the six dimensions of the framework
(burden of disease, vaccine characteristics and impact of
immunisation programme, values and preferences, resource
use, equity and feasibility). This resulted in 81 structured
questions concerning the relevance of each proposed
criterion. We requested participants to rate these criteria on
a 9-point Likert scale ranging from 0 (not at all important) to
9 (extremely important). We then conducted a two-round
Delphi process through Internet and a discussion group.
Results: Fifty-nine participants from multidisciplinary areas
were invited by email; 46 participants accepted the invitation.
In the first round, 46 out of 81 criteria achieved consensus.
Among those lacking consensus, the major concerns were
related to criteria about mortality, morbidity, symptoms, and
differences between local and global data. We presented
results of the first round in a group discussion. In the
second round, 48 criteria were assessed, and 41 achieved
consensus. Conclusions: In order to optimize its application
and effectiveness, a vaccine framework should consist of
dimensions and criteria that are supported by evidence
summaries; these can be presented in graphical or tabular
displays. Additional research is required to test the framework
for coverage decisions on one or more vaccines, in order to
identify their critical elements, and their relative importance
in influencing decision-making.
Colquhoun H1 , Hellis E2 , Belanger D2 , Ens B2 , Hill S3 , Lowe D4 ,
Mayhew A5 , Taylor M6 , Grimshaw J7
1 University of Toronto, Canada; 2 Canadian Agency for Drugs and
Technologies in Health, Canada; 3 Co-ordinating Editor, Cochrane
Consumers and Communication Review Group/La Trobe University,
Australia; 4 La Trobe University, Australia; 5 Ottawa Hospital
Research Institute, Canada; 6 Australian Catholic University,
Australia; 7 Director Cochrane Canada/Ottawa Hospital Research
Institute/University of Ottawa/, Canada
Background: Health systems globally promote appropriate
prescribing by health care providers and safe and effective
medicines use by consumers. Rx for Change (a publicly
available database developed by the Cochrane Effective
Practice & Care, and Consumers & Communication Review
Groups) provides access to systematic reviews regarding best
practices for prescribing and using medicines. We developed a
training program, with knowledge user (KU) input, to promote
its wider use. Objectives: To describe the development,
delivery, and evaluation of an Rx for Change training
program for five medicine-focused organizations in Canada
and Australia. Methods: Development: Interviews with KUs
were conducted in each organization. A directed content
analysis of interview transcripts was performed and used
to develop generic and tailored training content. Delivery:
Workshops were offered to all organizations. Evaluation:
A survey measuring attitudes, skills, and intentions for
using Rx for Change and evidence for decision-making was
conducted pre-, post-, and three months after training
completion.
Results: Development: Seventeen KUs
participated in interviews. Themes for training content
included the scope of, navigation and strategies for using
Rx for Change (generic content) and practical examples
on incorporating evidence within their workplace context
(tailored content). Delivery: The program included an
information video, a 60-minute face-to-face workshop with
didactic, hands-on and interactive components, and two
post-training reminders. Forty-eight people attended one
of six workshops. Evaluation: ANOVA indicated statistically
significant improvements in: skills for using evidence in policy
decision-making F(2,117) = 8.034, P = 0.001; skills in using Rx
for Change, F(2,117) = 88.58, P = 0.000; and attitudes towards
Rx for Change F(2,117) = 7.079, P= 0.001. Confidence in skills
for using evidence and Rx for Change was maintained three
months after training (P ¡0.007; P ¡0.000). Conclusions:
Sustained improvements in skills for using evidence in policy
decision-making were reported by participants in a training
program developed with KU involvement.
163
LRO 12.4
LRO 12.5
Does presenting the findings of a review
change public health decision-makers’ view
on an intervention?
Translating health research to policy: taking
forward evidence-informed decision-making
in Nepal
Baker P1 , Francis D1 , Demant D2
1
Public Health, School of Public Health & Social Work, Queensland
University of Technology, Australia; 2 School of Public Health &
Social Work, Queensland University of Technology, Australia
Karki K1 , Subedi M2 , Karki A1 , Aryal K1 , Stewart M3 , Rana S4 ,
Dhimal M1
1 Nepal Health Research Council, Nepal; 2 Patan Academy of
Health Sciences, Nepal; 3 Ministry of Health and Population, Nepal;
4
London Borough of Merton, United Kingdom
Background: Knowledge translation and exchange strategies
are required to support the utilisation of Cochrane Reviews in
decision making.
Objectives: To analyse the views of public health practitioners
and whether they change after participating in a workshop
communicating the findings of a recently updated relevant
review. Methods: Audience response technology (clickers)
were used to enhance engagement and identify participants’
views on the topic. Participants from three Canadian public
health units were polled before and after the presentation.
The polling included an assessment of agreement with
the statement ’Community-wide interventions effectively
increase population levels of physical activity’. The review
found the interventions rarely provided an effect. Results:
Across the three units, 132 participants used clickers.
Initially, 77% of respondents stated they had confidence
in identifying an evidence-based program or service. Prior
to presentation 3% strongly agreed, and 36% agreed with
the statement ‘Community-wide interventions effectively
increase population levels of physical activity’.
Post
presentation, the agreement with the statement reduced
to 1.5% strongly agreeing and 3% agreeing. Eighteen per
cent voted neutral and the remaining 77.5% disagreed. A
small proportion (6%) of participants increased their belief
in the intervention (by 1 level on a 5 point scale), and
26% remained unchanged in their view. However, a large
proportion of the participants (68%) reduced their belief
by 1 or more level (Wilcoxon signed ranks test: Z= -6.850, P
value < 0.001). Those who had prior belief in community-wide
interventions had the greatest likelihood for change. Sixty per
cent agreed with the presentation findings, 30.5% remained
neutral and 9.5% disagreed. Conclusions: Presenting findings
of systematic reviews to stakeholders has the potential to
modify participant’s views. The use of clickers can identify
these changes and stimulate discussion. Further research is
necessary to identify whether long-term policy change occurs
from the presentation.
Background and Objectives: Knowledge derived from
research and experience is of limited value if not put into
practice. Currently, attention is increasingly focused on
innovative ways to introduce and scale-up evidence-informed
practices in the health sector. It is estimated that only 14%
of findings actually make the transition from research to
practice within two decades of their discovery. This research
is designed to understand factors that facilitate or hinder the
translation of evidence into policy in Nepal, and to identify
ways forward for improvement and developing mechanisms
to link health research to policy. Methods: Key informant
interviews will be conducted with researchers and policy
makers using separate interview guides. A minimum of
15 interviews each with researchers and policy-makers will
be undertaken. After preliminary analysis of interviews, a
separate mini-workshop will be conducted with each group
to explore issues raised further. Results: Our study is currently
in progress and we expect to have our results to present at
the Colloquium. We expect this research to identify key gaps
on both sides; researchers, who are the evidence-creators
and policy-makers, who are the users of evidence. We
expect the findings to guide further actions of evidence
synthesis and knowledge translation in Nepal and similar
low- and middle-income countries, and ultimately bridge the
knowledge gap that exists between researchers and policy
makers. This research is well accepted by the researchers
and policy makers during the interviews. Conclusions:
Regular assessment of gaps in knowledge translation and
the development of a mechanism of communication between
researchers and policy-makers will help to translate research
into evidence-informed decision-making.
LRO 12.6
A methodological approach for binary data to
determine the extent of treatment effect in
the early benefit assessment of drugs
Bender R1 , Skipka G1 , Wieseler B1 , Kaiser T1 , Thomas S1 ,
Windeler J1 , Lange S1
1
IQWiG, Germany
Background: At the beginning of 2011 the early benefit
assessment of new drugs was introduced in Germany with the
Act on the Reform of the Market for Medicinal Products
(AMNOG). The Federal Joint Committee (G-BA) generally
164
commissions the Institute for Quality and Efficiency in Health
Care (IQWiG) with this type of assessment, which examines
whether a new drug shows an added benefit over the current
standard therapy on the basis of a dossier submitted by the
pharmaceutical company. In this benefit assessment the
costs of the therapies are not taken into account. In the case
of an added benefit, the law requires this benefit to be
classified as minor, considerable, or major. Objectives:
To describe and discuss the approach for assessing the
extent of added benefit for binary data developed by IQWiG.
Methods: Based on the underlying law, outcomes are grouped
according to their relevance as follows: (1) mortality, (2)
serious (or severe) symptoms (or late complications); serious
(or severe) adverse events, health-related quality of life, (3)
non-serious (or non-severe) symptoms (or late complications),
non-serious (or non-severe) adverse event. Thresholds for
confidence intervals for the relative risk are derived by means
of meta-analytic power considerations for shifted hypotheses
based on suggested effects for each of the categories of added
benefit for each outcome group. Results: For the treatment
effect to be classified as a minor, considerable or major added
benefit, the methodological approach stipulates that the
(two-sided) 95% confidence interval of the effect must exceed
a specified distance to the zero effect. Conclusions: The
proposed approach provides a robust, transparent and thus
predictable foundation to determine minor, considerable
and major treatment effects on binary outcomes in the
early benefit assessment of new drugs in Germany. After
a decision on the added benefit of a new drug by G-BA
(Gemeinsame Bundesausschuss), the classification of added
benefit is used to inform pricing negotiations between the
umbrella organization of statutory health insurance and the
pharmaceutical companies.
165
Poster Session 1
Impact of Cochrane reviews
P1-003
Analysing the Altmetric scores
of articles from the Cochrane Database
of Systematic Reviews
Stewart G1 , Ruotsalainen J2
1 Wiley, United Kingdom; 2 Cochrane Occupational Safety and
Health Review Group, Finland
Background: In 2014, Wiley partnered with Altmetric; a
company that compiles article-level metrics. Altmetric
provides an alternative to the more traditional ways of
measuring the impact of scholarly work, by tracking
data derived from online activity and discussions about
individual scholarly papers collectively known by Altmetric
as ‘mentions’. Cochrane Reviews are given an Altmetric score
based on the number of mentions they receive. Wiley use
traditional metrics to provide an annual ‘Impact and usage’
report to every Cochrane Review Group (CRG) with the aim
of assisting groups in showing the impact of their published
reviews. The report does not currently include Altmetric
data and the use of Altmetric data within Cochrane is not
widespread. Objectives: To analyse Altmetric data and
establish whether more widespread use of the data would
be beneficial for CRGs and Cochrane authors. Methods: An
analysis of the number of mentions received by all published
Cochrane articles and a more detailed analysis of mentions
received by articles published by the Cochrane Occupational
Safety and Health (OSH) Group will be undertaken. The
online submission system, Archie, will be used to export the
publication details of all published Cochrane articles. The
Altmetric Explorer platform will be used to export the number
of mentions associated with all published Cochrane articles
and to analyse the nature of mentions received by articles from
the OSH group. Results: To date, 6040 Cochrane articles have
received an Altmetric score; 34 of these articles were published
by the OSH group. The results will show how CRGs compare
by score and weighting of mentions. Activity associated
with articles published by the OSH group will be analysed
additionally by demographics and source. Conclusions:
Conclusions will be drawn and presented from the analysis.
P1.004
Identifying Cochrane citation classics
Stewart G1 , Uthman O2 , Pentesco-Gilbert D1
1 Wiley, United Kingdom; 2 University of Warwick, United Kingdom
Background: The concept of ‘citation classics’ was first
mentioned in a 1977 essay by the creator of the Impact
Factor, Eugene Garfield (Garfield 1977). The first group of
citation classics contained the 500 papers most cited from
1961 to 1975. In October 2013, a paper published in PLoS
ONE identified and assessed a group of the 100 most cited
articles in systematic reviews and meta-analyses from 1977
to 2008 (Uthman 2013). The 100 most cited articles did
not contain a Cochrane Review despite Cochrane Reviews
being internationally recognised as the highest standard in
evidence-based health care. Objectives: To identify and
examine the characteristics of a group of citation classics from
the Cochrane Database of Systematic Reviews. A secondary
aim is to investigate how citation classics from the Cochrane
Database of Systematic Reviews compare with citation
classics from other systematic reviews and meta-analyses
published in the same time period. Methods: The Web of
Science Core Collection of Thomson ISI will be searched
to identify the most cited Cochrane Reviews from 2005 to
2014. Data on the year of publication, associated keywords,
average number of cites per year, the country and institution
of the corresponding author and the Cochrane Review Group
(CRG) responsible for producing the review will be gathered.
Results: Initial investigations have found that 7068 Cochrane
Reviews have been indexed in the Web of Science Core
Collection from 2005 to 2014. The highest cited Cochrane
Review in this time period has been cited 452 times. The
100th highest cited Cochrane Review has been cited 89 times.
The results of the data analysis will be made available at
the meeting. Conclusions: By identifying Cochrane citation
classics it is hoped that opportunities for future engagement
in topic areas and with institutions and individuals will be
discovered.
References
Garfield E. Introducing citation classics –human side of
scientific reports. Curr Contents 1977; 1: 5–7
Uthman OA, Okwundu CI, Wiysonge CS, Young T, Clarke A.
Citation classics in systematic reviews and meta-analyses:
who wrote the top 100 most cited articles? PLOS ONE 2013:
8:10
P1.006
Open access: collaboration between NIHR
and Wiley
Osborne R1 , Bailey S1
1 National Institute for Health Research, United Kingdom
Background: The UK National Institute for Health Research
(NIHR) provides GBP 1.4 million a year direct funding to
Cochrane Reviews through the Cochrane Programme Grants
Scheme, established to provide high-quality systematic
reviews of direct benefit to users of the National Health
Service. Objectives: To comply with NIHR Open Access
policy, the NIHR reached an agreement with Cochrane and
Wiley in 2013 to deposit all Cochrane Reviews produced
from the Cochrane Programme Grants to Europe PubMed
166
Central (PMC) 12 months after publication. Methods: The
submission of each Cochrane Review to Europe PMC is
reviewed and approved by the Cochrane Editorial Unit, as
detailed in the Cochrane Editorial and Publishing Policy
Resource (EPPR). Upon publication of a review on the
Cochrane Library, Wiley supply the standard review PDF
to NIHR for display on the NIHR Journals Library website.
Upon expiration of the 12-month embargo, the PDF is taken
down from the website, as it is free to access in The
Cochrane Library from that point onwards. Results: All
Cochrane Programme Grant Reviews published on or after
1 February 2013 meet the Wiley ‘green’ open access policy,
and have been placed on the NIHR Journals Library website
under a dedicated collection: http://www.journalslibrary.
nihr.ac.uk/nihr-research/cochrane-programme-grants. After
the initial 12 months they are free to access for all via the
Cochrane Library, and can be included in an institutional
repository or any repository mandated by the author’s funder,
such as PMC, 12 months after publication. Conclusions: ‘NIHR
investment in systematic reviews means our health and care
services have access to the best possible evidence to inform
decisions and choices’ (Professor Dame Sally Davies, CMO,
England). Our collaboration with Wiley has made alignment
with the NIHR Open Access Policy a reality. The Department
of Health and NIHR agree with the principal of Open Access to
the outputs of its research, recognising it can offer both social
and economic benefits, as well as aiding the development of
new research and stimulating wider economic growth of the
UK economy.
studied were likely to be beneficial, and they all recommended
additional research. Also, two of the reviews concluded that
the interventions were not helpful, one of which did not
recommend further studies and one review recommended
additional studies. In total, 38 of the reviews reported that
the evidence did not support either benefit or harm, two
did not recommend further studies and 36 recommended
additional studies. Therefore, 81 of the total of 84 completed
SRs recommended further research (Fig 1). Conclusions:
Forty-four Cochrane SRs investigating efficacy of CBT for
prevention and treatment of mental disorders concluded that
the interventions studied were likely to be beneficial, and they
all recommended additional research. Thirty-eight reviews
reported that the evidence did not support either benefit
or harm. On the basis of these results we recommend the
elaboration of high-quality primary studies in CBT in active
collaboration and consultation with International Societies of
CBT and Academy. This will represent a major component of
methodological advance in dissemination of Evidence Based
Psychology.
P1.007
Mapping the Cochrane Library for decision
making in Cognitive Behavior Therapy (CBT)
for prevention and treatment of mental
disorders
1
2
2
Melnik T , Neufeld CB , Ferreira IMF , Atallah AN
1
Brazilian Cochrane Center, Brazil; 2 University of Psychology
Ribeirão Preto - FFCLRP, Brazil
Background: Cognitive behavior therapy (CBT) is based on the
main assumption that how we perceive situations influences
how we feel emotionally and how we act in different situations.
CBT helps people identify their dysfunctional thoughts and
evaluate how realistic those thoughts are. The focus is on
learning how to change distorted thinking. The emphasis is
also consistently on solving problems and initiating behavioral
change. Objectives: We evaluated the conclusions from
Cochrane Systematic Reviews (SRs) of randomized controlled
trials in terms of their recommendations for clinical practice
and research for CBT for prevention and treatment of mental
disorders. Methods: In our cross-sectional study of systematic
reviews published in the Cochrane Library, we analyzed all the
completed systematic reviews published about the efficacy
and effectiveness of CBT in the prevention and treatment
of mental disorders. Results: We analyzed 84 completed
SRs. Of these, 44 (52.4%) concluded that the interventions
Mixed methods/implementation
1
P1.009
A novel methodological approach to
synthesising the findings from quantitative
and qualitative systematic reviews
Moore D1 , Gwernan-Jones R1 , Ford T1 , Garside R1 , Stein K1 ,
Rogers M1 , Thompson Coon J1
1
University of Exeter Medical School, United Kingdom
Background: There are few established methods for the
integration of findings from quantitative and qualitative
systematic reviews of interventions. This paper describes
a method of overarching synthesis that we developed to
bring together the findings from four systematic reviews
about the effectiveness and experience of school-based
interventions for attention-deficit/hyperactivity disorder
(ADHD). Methods: We approached the synthesis from two
perspectives: 1. we brought together the findings of
two qualitative syntheses using an inductive approach to
167
identify the contextual elements that might influence the
effectiveness of interventions; 2. we worked deductively from
quantitative findings about the effectiveness of interventions
and moderators of effect, to identify findings in our other
reviews that offered hypotheses about the relationships
between possible moderators and effectiveness. Results:
Findings from the inductive approach revealed a hierarchy
of contextual levels that influence the use of school-based
interventions.
Implications for the development and
evaluation of school-based ADHD interventions were raised
from the deductive findings which helped to further our
understanding of potential moderators of effectiveness.
Unlike existing methods of overarching synthesis, the
methods employed did not only explain the findings of
the effectiveness review, but rather developed a greater
understanding of the use of ADHD interventions in school
settings.
Conclusion: The addition of an overarching synthesis to this
series of systematic reviews has allowed comparison across
all reviews, and brought together a breadth of evidence
regarding the use of ADHD interventions in school settings.
The identified links between reviews are valuable because of
their relevance to future intervention programme design and
evaluation.
P1.010
Appropriate use of prescription opioids: how
to do a systematic review of strategies
Irvin E1 , Furlan A1 , Van Eerd D2 , Munhall C1 , Kim J3 , Danak S1
1
Cochrane Back Review Group, Canada; 2 Insitute for Work and
Health, Canada; 3 Cancer Care Ontario, Canada
Background: The World Health Organization (WHO) estimates
that worldwide 69 000 people die from opioid overdose each
year; an estimated 15 million people suffer from opioid
dependence (i.e. an addiction to opioids). Objectives:
To determine existing strategies, frameworks, collaborative
networks, and materials aimed at improving the appropriate
use of prescription opioids, and/or reducing the misuse,
abuse and diversion of these drugs. Methods: We conducted
a mixed methods review. We searched MEDLINE, EMBASE,
PsycINFO, and CINAHL, and identified grey literature by
searching websites for the following groups: regulatory
authorities for health-related professionals, government,
public health and health promotion agencies, prevention
and treatment organizations, workers’ compensation boards,
private insurance companies, and law enforcement agencies.
There were no language restrictions. Our interventions
of interest were the types of strategies developed to
improve the appropriate use of opioids. Our primary
outcomes were: appropriate opioid use for pain, reduction
of misuse, abuse, and/or addiction, overdose, diversion
and/or crime.
We engaged stakeholders from law
enforcement, government, public health, and vulnerable
groups. Results: We found a total of 3191 references
from the database searches and grey literature.
We
included 109 references: 26 empirical studies, 53 descriptive
studies, and 30 reviews. Some studies only described the
strategies, but did not have any empirical evidence to
support the implementation of the strategy. We classified
the strategies into 14 groups: educational, promotional,
campaign, framework, interactions, networking, community,
forum, actions, regulations/law enforcement, prescription
monitoring, programs, take-home naloxone, and others.
Conclusions: This mixed method review found 14 types
of strategies that were relevant and could be disseminated
to consumers. The empirical evidence may be examined
to provide the effectiveness of some of these strategies
on improving the appropriate use of prescription opioids.
Outputs from this review included proposed policy, safety
kits, and public awareness campaigns.
P1.011
Evidence mapping - do systematic reviews
report the inclusion of women and sex as a
moderator of intervention effects?
Williams JW1 , Duan-Porter W1 , Allen LaPointe N1 , Goldstein KM2 ,
Hughes JM3 , McDuffie JR2 , Klap R4 , Gierisch JM1
1
Duke University, USA; 2 Durham VA Medical Center, USA; 3 University
of North Carolina at Chapel Hill, USA; 4 Greater LA VA Medical Center,
USA
Background: Delivering effective healthcare for women is
a priority for policymakers and requires evidence about
effective interventions. Objectives: To evaluate reporting
on the inclusion of women and evaluation of sex-specific
intervention effects in systematic reviews (SR). Methods:
We engaged stakeholders to prioritize conditions and
interventions. We searched PubMED and the Cochrane
Library for English-language SR published since 2009, that
addressed a medication, behavioral, quality improvement
(QI), or exercise intervention for depression, diabetes mellitus,
fibromyalgia, chronic low back or knee pain. We summarized
data on the inclusion of women, intervention effects and
sex effects as a moderator using descriptive statistics.
Results are reported for depression; evaluation of the
other conditions is in progress. Results: We screened 629
citations for depression, and 79 SR were eligible. The most
commonly evaluated conditions were: major depression
(n = 54 studies), minor depression (n = 22), dysthymia
(n = 21), and treatment-resistant depression (n = 5).
Evaluated interventions included: psychotherapy (n = 47),
antidepressants (n = 34), relapse prevention strategies (n =
6), guided self-help (n = 4), exercise (n = 1) and QI (n = 1).
The proportion of women enrolled in the primary studies was
reported in 27 SR (34%) and in this subset, women constituted
≥ 50% of the sample in all pooled estimates. Sex-specific
effects were evaluated in 14 SR (18%), using meta-regression
(n = 7), patient level meta-analyses (n = 3), subgroup analyses
(n = 2), or qualitative synthesis (n = 2). Four SR reported no
sex effects for: therapy vs antidepressant (AD), therapy and AD
vs AD alone, self-help, and QI. Cognitive behavioral therapy
and antidepressants (i.e. SSRI and venlafaxine) were found
168
to be more effective for women than men. Sexual adverse
effects were more common in men than women treated
with paroxetine. Conclusions: The minority of SR described
the inclusion of women or evaluated sex as a moderator of
treatment effect. Some depression interventions appear more
effective for women. However, moderator effects were often
evaluated using suboptimal methods.
P1.012
Reporting of intervention fidelity
in systematic reviews of asthma education
Ospina M1 , Villa-Roel C2 , Rowe BH2
1
Alberta Health Services, Canada; 2 University of Alberta, Canada
Background: Intervention fidelity is the degree to which
an intervention is delivered as intended. Recent evidence
indicates that intervention fidelity should be part of
the appraisal of randomized controlled trials (RCTs) of
non-pharmacological interventions. It is not known how
systematic reviews (SRs) of non-pharmacological RCTs in
asthma education evaluate intervention fidelity and which
specific fidelity dimensions have been reported. Objectives:
To describe the reporting of intervention fidelity in SRs of RCTs
assessing the effectiveness of asthma education. Methods:
Relevant SRs of the effectiveness of asthma education in
children and adult populations were identified through
searches in seven electronic databases. Two independent
reviewers assessed eligibility of SRs and completed data
extraction. The Treatment Fidelity Assessment Grid was used
to evaluate the reporting of five dimensions of treatment
fidelity in the SRs: (1) fidelity to theory; (2) provider
training; (3) treatment implementation; (4) treatment receipt;
and (5) treatment enactment. Results were summarized
descriptively. Results: From 5218 citations, 24 SRs of asthma
education (10 Cochrane versus 14 non-Cochrane reviews)
were included. The majority of reviews (n = 20) assessed at
least one aspect of treatment fidelity of individual RCTs. The
dimensions of treatment fidelity most frequently assessed in
SRs were treatment implementation (n = 17) and provider
training (n = 11); treatment receipt (n = 6), theoretical fidelity
(n = 4) and treatment enactment (n = 4) were less frequently
reported. No statistical associations between these domains
and SRs characteristics (type, quality and direction of results)
were found. Conclusions: Current SRs of asthma education
reveal inconsistent reports of intervention fidelity in RCTs.
This limits the translation of results into clinical practice
and may influence the interpretation of effectiveness. Using
standardized models of fidelity assessment in SRs will ensure
that key features of non-pharmacological interventions
are understood, appraised appropriately, and examined in
relation with therapeutic change and outcome results.
P1.013
Methodological strategies for the synthesis of
complex interventions –oral health education
programmes for nursing staff and residents
Albrecht M1 , Kupfer R1 , Reißmann DR2 , Mühlhauser I1 , Köpke S3
1
MIN Faculty, Unit of Health Sciences and Education, University
of Hamburg, Germany; 2 Department of Prosthetic Dentistry,
University Medical Center Hamburg-Eppendorf, Germany; 3 Nursing
Research Unit, Institute of Social Medicine, University of Lübeck,
Germany
Background: Methodological challenges regarding the
synthesis of complex interventions have been discussed
extensively. Meta-analyses of randomized controlled trials
(RCTs) do not provide sufficient information concerning the
implementation of complex interventions such as education
programmes. Contextual and component-related factors
are indispensable for review users. So far this information is
rarely provided in Cochrane Reviews. Objectives: Produce a
description of intervention components and context factors
in a Cochrane Review on the effectiveness of oral health
promotion education in nursing homes. Methods: The
systematic review was prepared in accordance with the
Cochrane Handbook. Data retrieval on research processes
of the development, piloting and evaluation of the complex
interventions including description of single components
and context factors based on the reporting guideline
‘Criteria for Reporting the Development and Evaluation
of Complex Interventions in healthcare’ (CReDECI2). In
addition to RCTs, other publications on development,
piloting and process evaluations were identified and included
through additional systematic searches, author requests and
reference tracking. Results: All included studies (2 RCTs, 7
cluster-RCTs) were complex interventions using more than
one active component. Results for dental health indicate
that educational interventions may have positive short-term
effects on dental plaque, denture plaque, and inflammation
of oral mucosa (moderate quality evidence). Included studies
reported insufficient information on development and
evaluation processes of complex interventions. For example,
only two studies reported on reasons for the selection of
intervention components and no study included information
on relevant context factors (Figure 1). Conclusions: Oral
health educational interventions seem to positively influence
residents’ dental health in the short term. Conclusions about
effective components cannot be made due to insufficient
reporting. Facilitators, barriers and resources of the different
interventions are unknown, but are crucial for decisions on
implementation.
169
P1.014
Practical tools and guidance on conducting
systematic reviews of complex interventions
Viswanathan M1 , Guise J2 , Chang C3 , Butler M4 , Kondo K2 ,
Dickinson C2 , Arbues F2 , Motu’apuaka M2 , Pigott T5 , Tugwell P6
1
RTI-UNC Evidence-Based Practice Center, RTI International, USA;
2
Oregon Health Sciences University, USA; 3 Agency for Healthcare
Research and Quality, USA; 4 Minnesota Evidence-Bsed Practice
Center, University of Minnesota, USA; 5 Loyola University, Chicago,
USA; 6 University of Ottawa, Canada
Background: 1. The Agency for Healthcare Research
and Quality (AHRQ), under the aegis of the United States
Department of Health and Human Services, supports the
conduct of systematic reviews (SRs) on a wide array of clinical
topics. With a rapidly changing healthcare environment,
AHRQ’s efforts to conduct SRs of complex interventions
are taking on increasing importance. AHRQ is supporting
a meeting of experts in complex interventions to identify
practical tools and guidance for scoping reviews, creating
appropriate analytic frameworks, and analyzing data (June
2015, Washington DC). Objectives: To create practical
tools for and guidance on conducting systematic reviews
of complex interventions. Methods: Guidance will be issued
around best practice and applications of new methodological
approaches for analysis of complex interventions, based
on a group consensus process of international experts in
evidence reviews, research, and healthcare implementation
of complex interventions.
Results: SRs of complex
interventions can vary widely in purpose, data availability
and heterogeneity, and end-user expectations. Questions
from end-users may range from exploratory to confirmatory.
Underlying sources of complexity can arise from context,
intervention, implementation, and participant responses.
The analytic framework chosen to answer systematic review
questions will need to account for end-user needs, sources
of complexity, and causal or associative links. Analytic
approaches can vary in data requirements or underlying
causal assumptions. We summarize the guidance developed
through multiple workgroups. Specifically, we touch upon
context, frameworks, and tools for complex intervention;
address the range of questions posed in SRs; and the analytic
approaches available to answer these questions. We offer
a decision framework to select the analytic approach best
suited to the context of the systematic review and note
data requirements, expertise needed, statistical power, and
computing resources. Conclusions: The resulting guidance
will aid a global audience of systematic reviewers of complex
interventions in improving the utility and rigor of their reviews.
170
Education and training
P1.016
P1.015
Iranian biomedical researchers and their
contribution to developing systematic
reviews
Experience of capacity building for
evidence-based child health
Agarwal A1 , Singh M2 , Jaiswal N1 , Thumburu KK1 , Mathew JL2 ,
Malhotra S3 , Shafiq N3 , Dutta S2 , Chadha N4
1
ICMR Advanced Centre for Evidence-Based Child Health,
Postgraduate Institute of Medical Education and Research,
Chandigarh, India; 2 Advanced Pediatric Centre, Postgraduate
Institute of Medical Education and Research, Chandigarh, India;
3 Department of Pharmacology, Postgraduate Institute of Medical
Education and Research, Chandigarh, India; 4 Dr.Tulsi Das Library,
Postgraduate Institute of Medical Education and Research,
Chandigarh, India
Background: There is a growing need for evidence to inform
policy and practice in India. The Post Graduate Institute of
Medical Education and Research, Chandigarh under the aegis
of the Indian Council of Medical Research (ICMR), New Delhi,
established an advanced center for evidence-based child
health. This center conducted workshops and short courses
to inform healthcare providers and researchers about the
principles of systematic reviews (SRs). Objectives: To build
capacity, conduct SRs and promote practise and training of
evidence-based health care in children. Methods: Twelve
short courses on ‘How to practise evidence-based child
health’, six short courses on ‘How to teach evidence-based
medicine’ and nine workshops on ‘Protocol development’
were conducted in Chandigarh and two satellite centers,
two each in north and north-east India, by the tutors from
the ICMR advanced centre for evidence-based child health.
We followed the module developed by the Royal College of
Pediatrics and Child Health. We conducted pre-testing and
post-testing with similar questions during each course to
evaluate the knowledge of evidence-based child health. Short
courses included lectures followed by small group interactive
sessions on critical appraisal of randomized controlled trials,
diagnostic test accuracy and conduct of SRs. Results:
Six-hundred and fifty participants from different parts of
India were educated and introduced to Cochrane and SRs.
More than half of the participants were not aware of Cochrane
and SRs before the course. More than 50% indicated interest
in further training in SRs. Most participants preferred small
group interactive sessions. There was a significant increase
in post-test marks compared to pre-test. Conclusions:
Capacity-building workshops and short courses in India
are avenues for increasing awareness and contributions to
Cochrane and SRs. There is a growing need to continue to
train individuals and develop programmes to support the use
of evidence for policy making and clinical practice in India.
Mesgarpour B1 , Arabzadeh S2 , Kabiri P3 , Akhondzadeh S4 ,
Malekzadeh R5
1
National Institute for Medical Research Development (NIMAD),
Iran; 2 Deputy of Research and Technology, Ministry of Health
and Medical Education, Iran; 3 Department of Epidemiology &
Biostatistics, School of Public Health, Tehran University of Medical
Sciences, Iran; 4 Psychiatric Research Centre, Roozbeh Hospital,
Tehran University of Medical Sciences, Iran; 5 Digestive Disease
Research Institute, Tehran University of Medical Sciences, Iran
Background: Appropriate training and the availability of
key resources play a pivotal role in producing a systematic
review. Over the past 10 years, the Ministry of Health and
Medical Education in Iran has provided access to research
databases and journals and has conducted several workshops
on how to write systematic reviews for researchers in medical
universities. Objectives: This study aimed to identify
the systematic reviews/meta-analysis published by Iranian
biomedical researchers, establish the overall trend and
describe characteristics of such publications. Methods: A
sensitive search strategy was designed based on authors’
affiliation and the key words ‘systematic AND review’ or ‘meta
analysis’. The citation databases of Scopus and Web of Science
were searched from 1990 to March 2015. We also searched for
the Cochrane Database of Systematic Reviews as a source of
publication. We identified systematic review/meta-analysis
that affiliated by at least one author from Iran. Two
independent reviewers screened and evaluated retrieved
records to select relevant studies. We excluded study
protocols. Results: Our search resulted in the retrieval of
2596 records (1735 after removing duplicates), of which 912
records (850 papers and 62 conference proceedings) were
judged to be relevant. The number of papers increased from
one in 1998 to 219 in 2014. The majority of papers were
published in English (98%) and in non-Iranian journals (71%)
including 43 papers in the Cochrane Database of Systematic
Reviews. Out of 850 papers, 155 (18.2%) were conducted
to investigate a national issue. The various domains of
reviews were detected, and included treatment/prevention
(37.2%), epidemiology and prevalence (14.9%), risk/harm
assessment (10.2%), prognostic (9.9%), policy making (7.7%),
public health (7.6%), diagnostic (6.1%) and gene disease
association (4.8%). Conclusions: This study introduces the
fast growth of scientific communications of Iranian biomedical
researchers in developing systematic reviews. Further study
on the quality assessment of these publications is warranted
to identify potential gaps and educational demands for Iranian
researchers.
171
P1.017
P1.019
Two years’ experience for training appraisers
of guidelines using an AGREE II scoring guide
in Korea
Development of the first Academic League of
Evidence-based Medicine in Brazil. A
partnership between the Brazilian Cochrane
Centre and Universidade Federal de São
Paulo
Shin E1 , Chang S2 , Kim D3 , Jang J4 , Yeon J1 , Lee Y5
1
Korean Academy of Medical Sciences, South Korea; 2 Department
of Urology, Kyung Hee University School of Medicine, South Korea;
3 Department of Diagnostic Radiology, Yon Sei University School of
Medicine, South Korea; 4 Korean Academy of Medical Sciences,
Korea South; 5 Department of Legal Medicine, Seoul National
University College of Medicine, South Korea
Background: An effective education program had been
developed in Korea since 2013 to train qualified appraisers
of guidelines. To alleviate discrepancies among appraisers
(multidisciplinary physicians), a scoring guide that consists
of 92 guides for anchor points 1, 3, 5, and 7 in 23 items
of AGREE II instrument has been developed and applied
by the Executive Committee for clinical practice guidelines
(CPGs), the Korean Academy of Medical Sciences in Korea.
Objectives: To train and examine AGREE II scoring guide for
time-saving and increasing reliability for the assessment of
quality of CPGs among appraisers. Methods: Three hours
of an education program was provided to 29 physicians in
2013 and 20 physicians in 2014. There were three different
training subjects: understanding the AGREE II instrument and
a scoring guide; learning how to apply the scoring guide; and
how to evaluate each of the 23 items and assign a score.
The guideline for prevention and treatment of metabolic
syndrome in primary care and the guideline for the diagnosis
and treatment of Helicobacter pylori infection in Korea, 2013
Revised Edition, were assessed using a Korean AGREE II scoring
guide in 2013 and 2014. Each item was rated on a 7-point
scale by participants. Disagreement was defined by more
than four differences in score among appraisers. Results:
Item 5 in domain 2 (stakeholder involvement) showed the
highest disagreement (27.6%, 8 of 29), and five items (items
1, 4, 6, 7 and 15) showed no disagreement in 2013. Items18
and 19 in domain 5 (applicability) showed high disagreement
(35% and 40.0%), and eight items (item 1, 2, 6, 7, 8, 10,
12 and 17) showed no disagreement in 2014. Conclusions:
We demonstrated that the Korean AGREE II scoring guide
was a useful tool to train appraisers of CPGs; it provides a
capable time-saving method to teach how to rate with a low
disagreement rate.
Daou J1 , Atallah A1 , Barroso G1 , Costa I1 , Paroni R1 , Freitas R1 ,
Vitor S1 , Viel J1 , Klepp A1 , Baldan A1 , Martins J1 , de Mello S1 ,
Gomes D1 , Fonseca E1 , Gomes V1 , Protetti N1 , Hungria F1 , Porfirio
G1 , Gois A1 , Riera R1
1 Universidade Federal de São Paulo, Brazil
Background: The relationship between Universidade Federal
de São Paulo and the Brazilian Cochrane Centre (BBC),
which has some members as professors in the Institution,
made possible the development of the first Academic League
of Evidence-Based Medicine (EBML) in Brazil. Objectives:
The objective of the EBML is to allow the student make
a link between issues of great medical relevance and the
evidence in the literature about those issues, leading to
scientific knowledge and critical appraisal as well. These two
components will guide the student’s best clinical practice in
the future. Methods: The League’s activities include a weekly
meeting for discussions and monthly supervised shifts in the
emergency service. The topics for discussion include critical
evaluation of literature, how to write scientific papers and how
and where find the best evidence. Results: Eighteen months
after the League’s creation (35 meetings), the following
activities were accomplished: translation of abstracts of some
Cochrane Systematic Reviews, a partnership with Students
4 Best Evidence, development of five papers (methods
issues and case reports) for publication, an eight-hour
course of evidence-based medicine (EBM) for the local
academic community, creation of a social network page
to disseminate EBM and participation of EBML members in
the systematic review workshops from BCC. Conclusions:
The EBML seemed to be a useful tool for dissemination of
EBM among undergraduate students, for the creation of a
critical scientific thought. It allowed the students to have
more practice both medical and scientific, allowing them to
enter their internships safer in their actions and crucially in
the process of health decision-making with concepts of EBM.
172
P1.020
P1.022
Development and evaluation of
doktormitSDM, an approach to improving
SDM in daily clinical practice –a continuum of
increasing evidence
Multi-professional teaching of health
research methodology in undergraduate
programs. The case of Universidad de La
Frontera in Chile
Kasper J1 , Liethmann K2 , Klemperer D3 , Rumpsfeld M4 , Geiger F5
1
Department of Health and Caring Sciences,Faculty of Health
Sciences, University of Tromsø, Norway; 2 Unit of Health Sciences
and Education, University of Hamburg, Germany; 3 Ostbayerische
Technische Hochschule Regensburg, Germany; 4 Division Internal
Medicine in the UNN, Norway; 5 University Medical Center
Schleswig-Holstein, Dept. of Pediatrics, Kiel, Germany
Seron P1 , Orellana JJ1 , Baeza B1 , Velazquez M1
1
Universidad de La Frontera, Chile
Background: Proven training of health professionals’ in
shared decision making (SDM) competencies are rare. Sound
evaluation requires consideration of such measures’ complex
character. Objectives: We aim to structure our reporting
on a series of studies to develop short in situ training for
clinicians according to the complex interventions framework.
Methods: We started with a comprehensive literature search.
Moreover, theory and conceptual knowledge were worked up
with regard to didactic requirements. Core components such
as measurement-based video feedback or a video tutorial
were piloted in scientific workshops. A pretest with 10
clinicians assessed patient involvement with the MAPPIN’SDM
before and after the so called doktormitSDM training. Then,
in a multicenter RCT (N = 40), SDMmass was used as primary
endpoint, a compound measure for patient involvement.
To allow for broad scale implementation, doktormitSDM
didactic and contents were translated into an online format.
The tutorial addresses a population of 12,000 practitioners
counselling colon cancer screening decisions. We piloted
the new format and newly developed components as an
interactive video learning parkour. Results: The training
has been shown to respond to a lack of research and
meets specific needs by applying criteria of SDM and EBPI
in a theory-based didactic concept. Components and entire
version are proven feasible, time economic and motivating for
clinicians. Promising results of the pre/post study informed
the training’s revision and the RCT design. DoktormitSDM
turned out efficient in changing communication quality of
clinician-patient decision-making dyads. The RCT, however,
still revealed practical difficulties in delivery to trainees.
Physicians completing the online tutorial and passing a test at
the training platform Curacampus, receive a SDM certificate
and credits. Certified doctors are now authorized to invoice
for higher counselling rates. Conclusions: A promising
and malleable approach to improving SDM in medical
consultations has been developed and studied successfully.
Whether doktormitSDM can change daily practice is a matter
currently undergoing further investigations.
Background: Development of skills in research methodology
in undergraduate students is a challenge. The Medicine
Faculty of Universidad de La Frontera in Chile has
implemented integrated and student-centred curricula for
all nine programs. These curricula include a line about
management and research in health. In the first two
years, the students are distributed in multi-professional
groups to cover this subject. Objectives: Evaluate the
course: ‘Management and Research in Health III’, in the
last six years, from the student perspective. Methods:
At the end of course, the students evaluate the teaching
process through a validated questionnaire. This questionnaire
has seven dimensions: organization and responsibility,
update and clarity, motivation and student participation,
knowledge achieved, interpersonal relationship, evaluations,
and comprehensive training. Each dimension includes
questions that must be answered on a 1 to 5 scale, where
1 is equal to not achieved, and 5 completely achieved. A
descriptive analysis is made for each dimension and for the
course. Results: Results from 1764 questionnaires from 2009
until 2014 were included. The total evaluation through years
was 4.12. The year with the best score was 2013 with 4.54,
and the year with worst evaluation was 2009 with an score of
3.89. The best dimension was comprehensive training with
an 4.24 (standard deviation (SD) 0.81), and the dimension
with the worst score was knowledge achieved with 4.03 (SD
0.95). A trend of better evaluation through the years was
observed. Conclusions: The global students’ evaluation is
good, and highlights the value assigned to comprehensive
training dimension. These results show that one of the
objectives that originated this innovative approach has been
fulfilled.
P1.023
Effect of evidence-based medicine courses on
informatics competencies of medical college
students
Qin W1 , Chen Y2 , Zhou Q1 , Wang J1 , Chen J1
1 Department
of Evidence-Based Medicine and Clinical
Epidemiology West China Hospital Sichuan University, China;
2
China National Nuclear Corporation 416 Hospital, Chengdu, China
Background: Effective and appropriate use of information
and communication technologies is an essential competency
for all healthcare professionals. With the development of
information globalization, the ability to acquire, process
173
and use information is particularly important. Objectives:
To evaluate the effects of evidence-based medicine
(EBM) courses on informatics competencies of medical
college students. Methods: We included 88 fifth- and
seventh-year medical program and postgraduate medical
students who selected the EBM course. We used the
self-completion ‘informatics competencies questionnaire’ to
describe and measure students’ informatics competencies
pre-and post-EBM courses. The questionnaire contains
three parts: information acquisition capability, processing
capability, and usage capability. Results: Information
acquisition capability: students like obtaining information
through the Internet, library and communicating with
teachers and classmates. After the EBM courses, the students
could construct search strategies better, most of the students
(56.1%) could formulate answerable clinical questions, and
the postgraduate change obviously in expanding database
(86.4% to 97.6%, P value < 0.05). Information processing
capability: after the EBM courses, the proportion identifying
important information increased, and more students could
tell the true information from the false (79% to 89.9%, P
value < 0.05). Information usage capability: after the course,
more students could cite the article by article with right format
(27.1% to 45.8%, P value < 0.05). More postgraduates could
adopt the true information (35.5% to 48%, P value < 0.05).
Conclusions: The evidence-based medicine course has a
positive effect on the informatics competencies of medical
students.
and to provide additional support, if necessary. Results:
We hosted three fellowships in 2014, with a total of 16 lead
authors and 10 co-authors, representing 11 CRGs. Participant
feedback was positive. Factors enabling authors to progress
with their reviews included: dedicated time, ongoing technical
and methodological support, ability to work in author teams
and liaising with CRGs to solve immediate issues. Seven of
16 lead authors submitted reviews for peer review (Table 1).
The remaining five authors who had planned to submit by
February 2015 have not adhered to their timelines. Reasons
include lack of dedicated time when back at work; delays in
feedback from co-authors; and underestimating time required
to dedicate to the review. Of those that submitted for peer
review, additional issues include peer review process delays.
Conclusions: Providing fellowships for Cochrane authors to
have dedicated time to work on their reviews is an effective
strategy for review progress. However, further active support
is required post-fellowship as authors continue to experience
similar time-management problems.
P1.024
Providing time out through dedicated
fellowships: strategy to improve Cochrane
author support towards review completion
Durao S1 , Oliver J2 , Young T3 , Kredo T2
1
South African Cochrane Centre, South Africa; 2 South African
Cochrane Centre, South African Medical Research Council, South
Africa; 3 Centre for Evidence-based Health Care, Stellenbosch
University, South African Cochrane Centre, South African Medical
Research Council, South Africa
Background: A 2013 survey of 64 Cochrane authors in the
South African Cochrane Centre (SACC) reference countries
identified limited time and financial constraints as the main
barriers to completing reviews. These findings informed a
new training strategy to support authors through fellowships
for dedicated time to work on their reviews. Objectives:
To assess the progress of authors who received fellowships
to complete or update their reviews. Methods: Authors in
SACC reference countries could apply for fellowships. Eligible
authors required a published protocol or a review update
due, with identification and selection of studies completed.
The five-day programme consisted mainly of time to work on
reviews, with short daily methods sessions, ongoing support
from Centre and associated staff, and linking with Cochrane
Review Groups (CRGs). Participants completed a timeline that
the SACC uses for regular follow-up, for project management
P1.025
Curriculum reform at Pontificia Universidad
Católica de Chile School of Medicine: an
opportunity to review the teaching of
evidence-based medicine
Rivera S1
Pontificia Universidad Católica, Chile
1
Background: In recent years, our institution started a process
of curricular analysis, literature review, conducted general
needs assessment and targeted needs assessment of learners
taking into account: academic advance of students, new laws
for health care and patient’s rights, globalization and new
trends in medical education. Thus, we realized the curriculum
had not evolved enough and graduates might not be prepared
to face the future challenges of medical practice. One aspect
was to improve the teaching of evidence-based medicine.
Objectives: Evaluate our curriculum, make the necessary
changes, and adjust the undergraduate profile of our students
to the new requirements, including competencies related to
decision-making based on evidence. Methods: Kirkpatrick
Model for Curricular evaluation: the evaluation process began
in 2009, with the formation of various committees to analyze
the current curriculum, curriculum relevant international
174
175
universities, the requirements for physicians in the new health
scenarios and projections, pre joints with graduate medical
education trends and expert opinion on each of the topics.
The views of students, alumni, faculty and the information
obtained from the evaluation processes and curriculum
accreditation is sought. Results: We designed a new
curriculum for a new graduate profile by learning outcomes
in eight domains of competencies in a comprehensive
framework. One of these skills is related to the management,
communication of information and evidence for clinical
decision making (see Table 1). This six-year program will
be implemented in 2015, and includes improved curricular
sequence, content integration, melding basic science and
clinical concepts throughout the career, with development of
the humanities and professionalism, prioritizing early contact
with patients and new methodologies, and early teaching
of management of information, evidence and technology
in clinical decision making (see Figure 1). Conclusions:
Curriculum processes require analysis models to make a
proper diagnosis of learning needs, define a undergraduate
profile and implement a system evaluation to deliver
feedback.
P1.026
Informing Cochrane learning strategies:
effectiveness of e-learning strategies to
increase evidence-based health care (EBHC)
competencies
Rohwer A1 , Rehfuess E2 , Young T1
1 Centre for Evidence-Based Health Care, Stellenbosch University,
South Africa; 2 Institute for Informatics, Biometry and Epidemiology,
Ludwig-Maximilians University of Munich, Germany
Background: Electronic learning (e-learning) strategies have
been widely adopted by educators worldwide. It is a complex
intervention with multiple components and dimensions that
may interact differently in different contexts. Objectives:
To assess the effectiveness of evidence-based healthcare
(EBHC) e-learning on EBHC competencies in healthcare
professionals. Methods: We considered randomised and
non-randomised controlled trials, interrupted time series
and controlled before-and-after studies comparing EBHC
e-learning or blended learning to no learning, face-to-face,
blended or e-learning for healthcare professionals. We
searched MEDLINE, EMBASE, CENTRAL, CINHAL, ERIC,
SCOPUS, PsychInfo and Web of Knowledge. Two authors
independently screened search outputs, selected studies for
inclusion, extracted data and made risk of bias judgements.
We synthesised results narratively due to large amounts of
heterogeneity between studies. Results: After screening
5110 titles and abstracts, we identified 38 potentially eligible
studies, 19 of which met our inclusion criteria. Included
studies were individually and cluster RCTs and controlled
trials. Participants were undergraduate and postgraduate
students as well as practicing clinicians. Most studies
compared EBHC e-learning to lectures or no learning, some
compared blended learning to lectures or no learning. Overall,
studies were of moderate quality, but 11 studies had high
risk of attrition bias. Knowledge scores improved after
e-learning interventions when compared to no learning.
Knowledge scores improved in both groups when e-learning
was compared to face-to-face interventions, and there was
generally no difference in the improvement. One study that
compared a clinically integrated e-learning module to a pure
online module showed significantly more improvement in
knowledge scores with the former.
Conclusion: E-learning of EBHC is effective in increasing EBHC
knowledge and skills, but should be integrated into clinical
settings for maximum benefit and relevance. Cochrane can
draw on our findings to inform learning strategies to increase
capacity to access, assess and interpret Cochrane Reviews.
P1.027
Are Polish physicians familiar with the
Cochrane Collaboration and the Cochrane
Library?
Leśniak W1 , Bala MM1 , Jaeschke RR1 , Koperny M2
1 Jagiellonian University Medical College, Krakow;
Polish
Branch of the Nordic Cochrane Centre, Poland; 2 Regional
Sanitary-Epidemiological Station, Krakow, Poland
Background: In October 2014 the Polish Branch of the Nordic
Cochrane Centre was established. This entity plans a range of
activities, including raising awareness of Cochrane activities
and training in the use of Cochrane Reviews (CRs). Objectives:
The aim of the study was to assess the awareness of Cochrane
and its activities, use of the Cochrane Library (CL) and CRs in
Poland and the need for workshops on the use of systematic
reviews and literature searching. Methods: A website (using
one of the most popular portals for health professionals
in Poland) questionnaire-based study. Results: During a
four-week period 169 doctors completed the questionnaire:
52% were women; 3% had finished their medical studies
less than a year previously, 15% had finished within one to
five years, 15% within five to 10 years, 32% within 10 to
20 years, and 34% more than 20 years ago. Seventeen per
cent of participants work only as general practitioners, 8%
only as specialists in outpatient clinics, 61% in hospitals, and
14% at universities. Most participants (77.5%) had heard
about Cochrane, 70% stated that they knew about Cochrane
activies, 64% had read publications in CL, 49% had read full
texts of CR, 52% used the results of CR in clinical practice
(all these results were independent of the place of work
or on the time since graduation; there was a statistically
significant relationship between the use of CR in clinical
practice and reading the full texts of CRs (P value < 0.001)).
It is expected that these percentages will be lower in the
general population of Polish medical doctors than in the study
population, where participants were probably interested in
the topic. The frequencies of using CL were: 32% more than
once a month, 15% twice a month, 7% once a week, 4% more
than once a week, 42% never used it. Eighty-five per cent of
176
participants were willing to take part in workshops on the use
of CR in clinical practice and searching for clinical information
in databases. Conclusions: The awareness of Cochrane
activities and products and the use of CL is unsatisfactory in
Poland. There is a necessity for promoting use of CR through
training activities.
P1.029
Increasing utilization of online databases by
evidence-based medicine curricula among
medical students
Chiu Y1 , Kuo K1 , Weng Y2 , Chen C1 , Chen K1
1
Taipei Medical University, Taiwan; 2 Chang Gung Memorial
Hospital, Taiwan
Background: Evidence-based medicine (EBM) has been
identified as a core competence to help physicians improve
healthcare quality. One of the important steps to implement
EBM is to acquire evidence-based information. Objectives:
EBM curriculum was introduced for third-year undergraduate
medical students. The current study aimed to determine
whether education can enhance their usage of evidence-based
online databases. Methods: A structured questionnaire
survey was conducted for third-year medical students who
received medical curriculum of EBM in 2013. Students
completed a baseline questionnaire survey prior to the
beginning of curriculum and finished the same questionnaire
right after the end of curriculum. Pair-sample t test
was used to compare the values of means between the
pre- and post-curriculum questionnaires. Results: After
the curriculum, students were more likely to access the
evidence-based retrieval databases –including two Chinese
databases and eight English databases. Specifically, medical
students more often accessed the National Digital Library
of Theses and Dissertations in Taiwan (P value 0.046), the
Chinese Electronic Periodical Service (P value 0.013), the
Cumulative Index to Nursing & Allied Health Literature (P
value < 0.001), the Cochrane Library (P value < 0.001), MD
Consult/ Clinical Key (P value < 0.001), MEDLINE/PubMed
(P value < 0.001), ProQuest (P value < 0.001), UpToDate (P
value < 0.001), Micromedex (P value 0.011), and DynaMed (P
value 0.037). In contrast, the usage of Web portals was not
increased (P value 0.380). The common motivations to access
the online databases were self-learning and class assignment.
Conclusions: The medical curriculum of EBM was helpful for
accelerating the utilization of online evidence-based retrieval
databases. The data have clinical implications for medical
educators to disseminate the implementation of EBM.
Systematic review methods
P1.031
Progress of PROSPERO registering systematic
reviews
Ciapponi A1 , Glujovsky D1 , Bardach A1
1 Instituto de Efectividad Clı́nica y Sanitaria - IECS, Argentina
Background: PROSPERO is an international open access
electronic database of prospectively registered systematic
reviews in health and social care. A unique registration number
and key features from the review protocol are recorded
permanently in order to avoid unplanned duplication
and enable comparison of reported review methods with
what was planned in the protocol.
Objectives: To
assess the progress of PROSPERO registering systematic
reviews, in absolute terms and in relative terms compared
to systematic reviews published in MEDLINE. Methods:
First, we identified the monthly cumulative registrations
and the status of these registrations on PROSPERO
(http://www.crd.york.ac.uk/PROSPERO/) from inception date
(22 February 2011). Second, we identified the annual
publications of systematic reviews in PubMed using
the most specific filter developed by Montori et al.
(((MEDLINE [tiab]) OR systematic[tiab])) AND ((review[tiab])
OR meta-analysis[Publication Type]) and we also identified
PROSPERO registrations (((prospero) AND ((registr* or crd)))).
Results: From inception to 28 February 2015 (four years)
6026 registrations were recorded in PROSPERO, most of them
(87%) reporting an ongoing status (Table 1). We observed
an increasing trend in the number of records (Figure 1). The
proportion of systematic reviews published in MEDLINE that
reported a CRD record number is also growing but, remains
below 2% (Figure 2). Conclusions: PROSPERO registration is
growing but is still remarkably low. Hopefully the PRIMA-P
2015 statement will boost this trend further.
177
178
P1.032
The Embase project 1: crowd characteristics
and behaviour after 18 months
1 Cochrane Dementia and Cognitive Improvement Group, Oxford
3 York Health Economics Consortium, United Kingdom; 4 Cochrane
Background: The Embase project is managed by a consortium
made up of Metaxis Ltd, the Cochrane Dementia and
Cognitive Improvement Group and the York Health Economics
Consortium. The project’s objectives are to identify reports of
randomised trials in Embase and to submit those reports to
Cochrane’s Central Register of Controlled Trials (CENTRAL).
The project has been managed by this consortium since April
2013 and uses crowdsourcing for much of the screening.
Objectives: This poster will focus on presenting answers to
the following three questions about the screening crowd: 1.
What do we know about the crowd? 2. Do screeners become
more confident over time? 3. How engaged is the crowd?
Methods: By September 2015 the project will have generated
over 18 months’ worth of data. Previous publications have
focussed on crowd performance, but here we focus on the
characteristics and behaviour of the crowd. We will analyse
the data from the project’s sign up form, which all screeners
must complete. This includes demographic information
plus information about prior knowledge of randomised trial
design. We will assess screeners’ decision-making over time
by looking at: time per citation and the proportion of ‘Unsure’
classifications made. To determine screeners’ engagement
we will look at screeners’ average activity per month/year.
Results: The 18-month results will be presented in this poster.
Interim data suggest that the crowd is 57% female; that most
of the crowd is aged between 31 and 40 years old; and around
4% were not familiar with randomised trial design before
taking part. Conclusions: Crowdsourcing brings with it many
potential benefits, but is not a simple option. The more we can
learn about our participants, the more we can help make the
experience rewarding and fulfilling for them and retain their
involvement in this project and others that are now underway,
such as Project Transform.
P1.033
Experiences with Covidence in preparing a
comprehensive systematic review
Helfer B1 , Samara M1 , Leucht S1
1
Cochrane Schizophrenia Group, Germany
Background: Covidence is ‘a not-for-profit service dedicated
to improving the use of evidence in health-care decision
making’ (www.covidence.org). It is an on-line platform to
perform systematic reviews and meta-analyses. Objectives:
To access the usefulness of Covidence in performing
a comprehensive systematic review and meta-analysis.
Methods: In 2014 to 2015 we performed a systematic review
and meta-analysis of antidepressants added to antipsychotic
drugs for schizophrenia using Covidence. This included
comprehensive screening (approximately 6500 abstracts),
full text review (approximately 250 full text articles), data
extraction and assessment of risk of bias (approximately 100
included studies) and export of data into Review Manager
software (RevMan). All steps were performed by two
independent reviewers. Based on our own experiences with
the software and correspondence with the Covidence support
team, we analyzed the advantages and disadvantages of this
service. Results: Although initially plagued with numerous
technical issues and missing functionalities, thanks to
continuous improvement based on user feedback, Covidence
became an invaluable tool in preparing our systematic review
and meta-analysis. As of April 2015, some problems of
a technical nature still exist, especially when handling big
data-sets, but they are typically easy to overcome with the
quick help from the Covidence support team. An important
advantage of the software is that all data are safely stored
and automatically updated as entered and that Covidence
enables reviewers to work independently at the same time
and in different geographical locations. Moreover, Covidence
is currently free of charge and supports exporting data into
RevMan. Conclusions: Covidence can be a very useful tool for
systematic reviewers and meta-analysts and their teams. As
it is constantly improved, it holds promise for the future.
P1.035
Updating systematic reviews published in
Chinese: a cross-sectional study
Wei D1 , Zhang H2 , Li L3 , Li R2 , Chang X4 , Fang Z3 , Chen Y1 , Yang K1
1 School of Basic Medical Sciences,
Lanzhou University;
Key Laboratory of Evidence-Based Medicine and Knowledge
Translation of Gansu Province, Lanzhou, China; 2 The Second
Clinical Medical College of Lanzhou University, Lanzhou, China;
3
School of Basic Medical Sciences, Lanzhou University, Lanzhou,
China; 4 The First Clinical Medical College of Lanzhou University,
Lanzhou, China
Background: Updating systematic reviews (SR) is a useful
solution for reducing the impact of publication bias (or time
lag bias) on the results. Nevertheless, the authors of SRs do
not keep SRs up to date. Research shows that only 3% of
SRs published in peer-reviewed journals and 38% of Cochrane
Reviews have been updated. The quantity of SRs published in
Chinese is increasing every year. However, there is little study
that focuses on whether they are updated. Objectives: To
investigate the current situation regarding the updating of
systematic reviews published in Chinese. Methods: We used
the terms ‘systematic review’, ‘meta-analysis’, and ‘update’
to search two electronic databases, WANFANG Data and
China National Knowledge Infrastructure (CNKI). Meanwhile,
we performed a handsearch of the four Chinese journals
with ‘evidence-based’ in their titles. The previous version
of SRs were also identified. Two independent reviewers
179
screened the updating of SRs and extracted the data. The
data extracted included publication year and journal, search
strategy, etc. Results: Five updated SRs were included. Of
those, three (60%) were published in peer-reviewed journals,
and the two (40%) remaining SRs were published as an
abstract of conference and an academic dissertation. Only
one (20%) SR reported they had updated their own SR.
The range of updating period was from one to seven years.
The conclusion had changed in one (20%) SR. Conclusions:
A few SRs published in Chinese have been updated; the
period between initial publication and updating varies and is
generally long. In addition, generally the authors of SRs do
not consider updating their own SRs.
P1.036
It’s the noise that makes the poison: why
current methods for assessing chemicals for
toxicity provides so little evidence?
Mandrioli D1 , Silbergeld E1
1
Johns Hopkins Bloomberg School of Public Health, USA
Background: Toxicology as a field has not often produced
efficient and timely evidence for decision making in public
health. In fact, of the 87,000 chemicals registered for
commerce in the USA, only one-tenth have been tested
for potential harms. Several in vitro and in vivo tests
have been adopted, without appropriate validation, and
the process of hazard assessment is extremely slow and
largely based on non-transparent practices such as ‘expert
opinion’ and ‘weight of evidence’. In response to this, the
US National Research Council called for the adoption of
evidence-based methods and systematic reviews in regulatory
decision making. EPA, FDA and the European Food Safety
Agency have recently endorsed these methods in their
assessments of safety and risk. Objectives: Evaluating
the effects of the adoption of evidence-based methods in
toxicology for filtering the primary literature by developing
criteria to identify tests of low quality and high risk of bias
and thereby accelerating hazard assessment. Methods: We
compare current practices in toxicology with principles and
methods utilized in evidence-based medicine and health care,
with emphasis on Cochrane’s record. Several aspects are
analyzed: validity and quality of the evidence, transparency,
time for evaluation, unnecessary animal loss, consistency
of the evaluations. Results: Evidence-based toxicology
(EBT) methods could provide an effective filter of low-quality
studies and reduce the overall time and resources needed
for risk assessment in the future. EBT methods proposed
by OHAT and Navigation Guide represent the first step in
this direction, although further validation of the methods
is necessary. Adherence to Cochrane principles, as well
as transparent development of systematic methods, is a
fundamental prerequisite for future implementation of EBT.
Conclusions: The adoption of consistent principles and
methods is likely to enhance the validity, transparency,
efficiency and acceptance of toxicological evidence, with
benefits in terms of reducing the burden of animal used,
delays and costs for all stakeholders (researchers, consumers,
industry).
P1.039
Development of an algorithm
for choosing study designs
Peinemann F1 , Kleijnen J1
1
Maastricht University, The Netherlands
Background: The purpose of systematic reviews is to answer
distinct research questions on healthcare interventions. The
type of study design, which is scheduled for inclusion in
a systematic review, may be critical for its findings and
consequences. We did not find an existing algorithm
providing guidance for systematic reviewers to choose the
most appropriate study designs. Objectives: The aim
of this study was to develop an algorithm that provides
guidance and awareness for various design characteristics
suitable to answer distinct research questions in healthcare
interventions.
Methods: We addressed the following
questions. What is the preferred content and format of a
straightforward algorithm for integration of various study
designs in a systematic review that incorporates major
study characteristics as decision points and that allows
consideration of practical concerns? Can the algorithm hold
its promise as a useful guide for choosing appropriate study
designs by testing its feasibility and applying it to existing
systematic reviews? Results: We developed an algorithm that
incorporates length of follow-up (long, short) and frequency
of events (rare, frequent) as binary decision points. At the
end of these four pathways, each of five types of outcome
(death, disease, discomfort, disability, dissatisfaction) was
linked with at least one design label. The algorithm also
considers practical or ethical concerns as well as unavailable
best evidence, and it provides examples for explaining the
critical guidance points. We confirmed the usefulness of the
pathways by reproducing the study selection and its impact
on the conduct of four systematic reviews. Conclusions: The
algorithm facilitates decisions about inclusion of different
study types to those planning to prepare a systematic review.
The algorithm cannot be applied without the consideration
of disease-specific circumstances and aims of interventions.
However, we think that it may reduce the time burden on
review authors and may increase the scope and applicability
of systematic reviews.
180
P1.040
Importance of author inquiries
Peinemann F1
1 University of Cologne, Germany
Background: Systematic reviews may rely on study data
published in peer-reviewed journals. A good international
reputation and a high impact factor of a journal may
promise that the article contains correct and verified data.
We conducted a systematic review on drug intervention in
patients with high-risk neuroblastoma and included one
study for which we found some inconsistencies within the
main report published by the Journal of Clinical Oncology
(impact > 17). Methods: I found that it was difficult or not
possible to reproduce the results of the statistical analyses
of the included study. I did not know if I just needed help
to understand the data better, whether I faced minor typing
errors, or if I had discovered flaws serious enough to change
the conclusion of the study. Therefore, I located the authors
and asked them to reply to my inquiries by email. Results:
I am very grateful to the authors for leaving room for my
doubts. The authors took great efforts to re-evaluate the
analyses thoroughly. Finally, the resulting changes were
published as an erratum, which was attached to the online
version of the article. The difference of overall survival
between the treatment groups was not significant as opposed
to the earlier version. Conclusions: The inquiry of authors to
explain their analyses is important and can have an impact on
the study conclusion and possibly the treatment of patients.
Respectful appreciation of the work achieved by the authors
is a prerequisite. Highly skilled professional scientists may
reply appropriately and pave the way effectively for clearing
the data.
P1.041
The importance of registration for systematic
reviews/meta-analysis about traditional
Chinese medicine in Chinese
1
1
1
1
2
3
3
4
Wang J , Zhang L , Mu W , Liu Z , Hu J , Du L , Li Y , Shang H
1 Ianjin Branch of the Ministry of Education Virtual Research Center
for Evidence-Based Medicine, China; 2 Chinese Medicine Institution
of Beijing, Beijing Hospital of TCM, China; 3 Chinese Cochrane
Center, China; 4 Key Lab of Chinese Internal Medicine of the Ministry
of Education, Dongzhimen Hospital, Beijing University of Chinese
Medicine, China
Background: With the rapid development of evidence-based
medicine in the field of traditional Chinese medicine
(TCM), a lot of low quality systematic reviews have been
produced, which may burden the evidence and affect
decision-making. Objectives: In order to explain the
importance of registration, this research studies the current
situation regarding it. Methods: Systematic searching of CNKI,
VIP, WanFangdatabase, CBM, PubMed, Web of Science (WOS),
the Cochrane Library and the PROSPERO registry platform was
carried out to included all published SR/MAs about TCM in
both Chinese and English. General information was extracted
individualy and the report and methodological qualities were
assessed independently by PRISMA and AMSTAR checklists.
According to data from CNKI and WOS, cited information
was identified. Results: A total of 2460 SR/MAs have been
published since 1997 and only 10.85% were published before
2008. All 56 registered research with full text came from
the Cochrane Library. The number of SR/MAs written by
Chinese people was about 8.36 times (1958/234) more than
non-Chinese. However, the number of high methodological
and report quality SR/MAs in Chinese are less than 1/4
(19/80) in English. Although the Chinese SR/MAs involved 179
kinds of diseases, 28.72 %"i566/1971) concentrated on five
single diseases while only 18.20 %"i89/1971) in English which
brought high recurrence rate of same subject in Chinese.
70.98% Chinese SR/MAs were cited for 7990 times in total,
accounting for 5.71 times on average. While the ratio was
67.69% (331/489) in English with total 4903 citations and 9.98
times on average. Within these 331 cited English articles, 289
of them have been cited by Chinese writers in a total of 778
times, which equalized to 2.69 times per articles. Conclusions:
Although the number of TCM SRs increases rapidly, low quality,
disease-concentration, outcomes bias of the studies cannot
be ignored. Registration may helpful to control the overload
evidence and the misleading, especially important for articles
in Chinese, because TCM clinicians prefer to produce and
use the Chinese SR/MAs due to the language preference and
evidence accessibility.
P1.042
Does applicability make the difference? Two
systematic reviews on the same topic
Angelescu K1 , Nussbaumer B2 , Sieben W1 , Scheibler F1 ,
Sauerland S1
1
Institute for Quality and Efficiency in Health Care (IQWiG),
Germany; 2 Department for Evidence-Based Medicine and Clinical
Epidemiology, Danube University Krems, Austria
Background: Most guidelines recommend screening for and
treatment of asymptomatic bacteriuria (ASB) in pregnancy.
A Cochrane Review (CR) on antibiotic treatment of ASB
in pregnancy provides the basis of this recommendation.
However, an Institute for Quality and Efficiency in Health Care
(IQWiG) report on benefits and harms of screening for ASB
concluded that the benefit is not proven.
Objective: We compared the two systematic reviews to find
out why they have come to differing conclusions. Methods:
We compared PICO, study pool, risk of bias (RoB) assessment,
effect estimates, assessment of applicability, and conclusions.
Results: Differences in inclusion and exclusion criteria result
in different study pools. The IQWiG report included only 3
of 14 studies that were included in the CR. However, effect
estimates of pyelonephritis rates are roughly comparable
(CR: relative risk = 0.23, 95% confidence interval (CI) 0.13 to
0.41; IQWiG: odds ratio = 0.21, 95% CI 0.07 to 0.59). Studies
date back from the 1960s to the 1980s. Both reviews rate
181
the RoB of included studies as high. Both do not describe
a methodology to assess applicability. However, in the CR
applicability is discussed with respect to study medication
and microbiologic methods, but not generally rejected. The
IQWiG report regards interventions in 2 of 3 included studies
as not applicable to current health care. In conclusion, the
CR identifies preventive effects, but recommends interpreting
the results cautiously. The IQWiG report, which, unlike the CR,
comprises a dichotomised statement on benefit and harms,
regards the benefit of an ASB treatment as not proven due to
serious concerns regarding applicability.
Conclusion: Despite differences in detail, overall conclusions
regarding the effect of treatment do not differ much. Both
works see the need of further randomised trials. The main
difference is seen in the significance of applicability and
derived conclusions. To date there is no well-developed
methodology for assessing applicability. However, different
domains of applicability (population, setting, interventions,
outcomes, follow-up) should be assessed as recommended in
current EUnetHTA guidance.
P1.043
Exploring inconsistencies between
observational and experimental studies of
selenium and diabetes risk
Vinceti M1 , Filippini T1 , Del Giovane C1 , Crespi C2
1
University of Modena and Reggio Emilia, Italy; 2 Fielding School of
Public Health, UCLA, USA
Background: Observational and experimental epidemiologic
studies that have addressed the relation between intake of the
trace element selenium and cancer risk have yielded strongly
conflicting results, as recently reported by a Cochrane Review.
Most observational studies suggest an inverse association,
while randomized controlled trials (RCTs) have indicated a
null or direct relation. Little is known about the replication
of such inconsistencies when dealing with the risk of other
chronic disease. Objectives: We investigated the results
of observational and experimental studies linking selenium
exposure to the occurrence of type-2 diabetes. Methods: After
a literature search, we identified 12 observational studies
(eight cross-sectional and four cohort) and five RCTs. Using
a random-effects model, we computed the summary relative
risk (RR) of type-2 diabetes along with its 95% confidence
interval (CI) in subjects with the highest versus the lowest
selenium exposure category in observational studies, and in
subjects allocated to selenium compared to placebo in the
RCTs. Results: Summary RRs were 1.98 (95% CI 1.22 to
3.23) and 1.13 (95% CI 0.15 to 8.45) for cross-sectional studies
using serum and toenail selenium for exposure assessment,
respectively. Cohort studies based on toenail selenium yielded
a summary RR of 0.68 (0.72 to 0.98), while the only study
assessing dietary selenium intake gave a RR of 2.39 (1.32 to
4.32). For RCTs, summary RR was 1.10 (1.00 to 1.21) among
selenium-supplemented versus placebo. The distinctive
feature of the two observational studies (one cross-sectional
and one prospective) that failed to find an excess diabetes
risk associated with higher selenium exposure was that the
subjects were health professionals. Age, gender, study area
and other demographic characteristics did not appear to have
influenced the results. Conclusions: These results suggest
that the ability of observational studies to predict results of
RCTs when addressing the health effects of selenium may
differ on the basis of the outcome studied (diabetes versus
cancer) as well as the indicator used for exposure assessment
and the type of population under study.
P1.045
Project on a Framework for Rating Evidence
in Public Health (PRECEPT): structure of a
draft framework
Harder T1 , Takla A1 , de Carvalho Gomes H2 , Eckmanns T1 ,
Ellis S3 , Forland F4 , James R5 , Jansen A2 , Meerpohl J6 , Morgan A3 ,
Rehfuess E7 , Schuenemann H8 , Wichmann O1 , Zuiderent-Jerak T9
1
Robert Koch Institute, Germany; 2 European Centre for Disease
Prevention and Control (ECDC), Sweden; 3 National Institute for
Health and Care Excellence (NICE), United Kingdom; 4 Norwegian
Institute of Public Health, Norway; 5 Scottish Intercollegiate
Guidelines Network (SIGN), United Kingdom; 6 German Cochrane
Center, Germany; 7 University of Munich, Germany; 8 McMaster
University, Canada; 9 Linköping University, Sweden
Aims: The Project on a Framework for Rating Evidence in
Public Health (PRECEPT) is a collaboration between European
public health agencies and universities, established in 2012,
that aims to establish a framework for evaluating and grading
evidence in the field of infectious disease epidemiology,
prevention and control. PRECEPT is funded by the European
Centre for Disease Prevention and Control (ECDC). This
presentation describes the structure and workflow of a draft
framework.
Methods and Results: The PRECEPT framework is designed to
rate scientific evidence related to four domains that are of high
priority in infectious disease prevention and control: disease
incidence/prevalence, risk factors for disease, diagnostics and
intervention. The framework is grouped into six consecutive
working steps, starting from a complex public health question
and ending with an evidence statement for each relevant
domain. In step 1, approaches are described for identification
of relevant questions. In step 2, methodological guidance
is provided for the conduct of systematic reviews for these
questions. For the appraisal of methodological quality of
identified individual studies, 15 different quality appraisal
tools are proposed and an algorithm is given to match a
given study design with an appropriate tool (step 3). In
step 4, a generalized evidence grading scheme based on
the GRADE methodology is provided to rate the quality of
bodies of evidence for each domain. The evidence appraisal
process ends with the preparation of evidence profiles and
summary of finding tables (step 5) followed by preparation
of an evidence summary for communication of the results
(step 6). By applying this methodological framework, the user
should be able to evaluate and grade scientific evidence from
the four major domains in a transparent and reproducible
way. Outlook: The draft framework is currently being piloted
182
by applying it to examples from infectious disease prevention
and control. Further refinements of the methodology, as
well as application to other domains, and the inclusion of a
methodology for going from evidence to recommendations
are planned for future versions of the framework.
P1.046
The unique challenges of systematic reviews
and meta-analysis of cluster-randomized
trials
Chen Y1 , Yao L1 , Wang X1 , Wei D1 , Wang Q1 , Wang M1 , Yang K1
1
Lanzhou University/Chinese GRADE Center, China
Background: Cluster-randomized controlled trials (CRTs)
have greatly increased over the past 20 years and
have motivated the publication of an extension of the
CONSORT statement for this design because of its particular
methodological issues. However, there are few systematic
reviews (SRs) or meta-analysis of CRTs that have been
conducted or published. Objectives: To address key
questions when conducting SRs of CRTs. Methods: We
searched MEDLINE, Embase, the Cochrane Library, CNKI
(China National Knowledge Infrastructure/Chinese Academic
Journals full text Database), VIP (a Chinese full-text database),
WANFANG (a Chinese full-text database) and CBM (China
Biomedicine Database Disc) using the term cluster*, group*,
field*, community, communities, systematic review*, meta
analysis in titles on 16 January 2015. We also checked the
section about CRTs in the Cochrane Handbook for Systematic
Reviews of Interventions. Results: We identified 696 papers
but only four were relevant to SRs of CRTs. The main
challenges of systematic reviews and meta-analysis of CRTs
are how to: 1. develop highly sensitive search strategies
for SRs of CRTs; 2. assess the particular risk of bias of
CRTs; 3. deal with CRTs with different types of clusters; 4.
choose statistical methods to combine the results; 5. combine
the findings from CRTs; 6. conduct the subgroup analysis;
7. consider intracluster (or intraclass) correlation coefficient
(ICC); 8. GRADE the evidence, such as recruitment bias, in
CRTs; 9. explain the difference when cluster- and individually
randomized trials in an SR are inconsistent for the same topic;
10. generalize the findings. Conclusions: The SOC(systematic
reviews of cluster randomized trial) Working Group has already
explored highly sensitive search strategies for SRs of CRTs. We
expected to work together with methodologists, researchers
using CRTs and statisticians to develop a more detailed
guideline for systematic reviewers of CRTs.
P1.047
A checklist approach to evaluation of
scientific study protocols approval removes
bottleneck and improves methodological
quality: a randomized controlled trial
Kumar A1 , Kosambiya J2 , Reljic T1 , Chenneville T1 , Menezes L1 ,
Mhaskar R1 , Miladinovic B1 , Djulbegovic B1
1 Morsani College of Medicine, USF Health, Program for CER, Tampa,
USA; 2 Government Medical College, Surat, India
Background: Scientific Review Committee (SRC) members
mostly use qualitative methods for reaching decisions
(approval or disapproval) on study protocols. Checklists for
reporting scientific studies have helped improve transparency
and presentation of studies in the public reporting of research
findings. While the use of checklists has become prevalent in
the arena of publishing, their role in the SRC evaluation and
decision-making process has not been assessed.
Objective: To assess the efficacy of the SRC approval
process using an objective checklist in addition to the current
procedures versus standard operating procedures (SOP) alone
for making a decision on approval or disapproval of scientific
study protocols. Methods: Two committee members were
randomly assigned to review each protocol involving human
subjects using the checklist plus the SOP or SOP alone.
Checklists (e.g. CONSORT, STROBE etc.) recommended
by scientific journals were matched to study design. The
primary outcome was approval rate for checklist(s) versus no
checklist(s) group. Differences in disapproval rates between
the two groups were assessed using a Chi2 test. The
agreement between reviewers (checklist versus no checklist)
was assessed using the Kappa statistic. Results: A total
of 28 study protocols were reviewed in duplicate by 56
reviewers (10 participants were excluded due to incorrect
review). Characteristics of reviewers in compared groups were
similar (see Table 1). The relative risk reduction in disapproval
was 33% with use of checklist plus SOP compared with SOP
alone (absolute risk reduction = 25%; relative risk = 0.67;
95% CI 0.44 to 1.03). The agreement between two reviewers
(checklist versus no checklist) assessing the same protocol
was poor, resulting in kappa of 0.07 (95% CI 0 to 0.39).
Conclusion: Use of checklist along with SOP results in
lower disapproval compared with the SOP only. The lack
of statistical significance is likely due to small sample size.
Introduction of checklists in the study design phase ensures
better methodological quality of study protocols and assists
SRC members remove the bottleneck in approvals and serve
as facilitators of research.
183
review and meta-analysis, resulting in wasted information.
Objectives: To quantify the proportion of incomplete
outcome data reporting for randomized controlled trials
(RCTs) of medical interventions for open-angle glaucoma
(OAG). Methods: We searched electronic databases for RCTs of
medical interventions for OAG with no restrictions with respect
to date or language. Two individuals independently screened
records and abstracted data. For this project, we analyzed
the completeness of IOP and visual field (VF) data in included
trials. Discrepancies were resolved by consensus. We used the
most complete reporting of outcomes for each RCT. Results:
We identified 417 trials with 65,452 participants as of May 2014,
of which 406 (97%) trials and 87 (21%) trials reported IOP and
VF as an outcome, respectively (Table 1). Data from about
56,000 participants from 356 trials was ‘wasted’ due to lack
of VF outcome data (Table 1). For IOP, a surrogate outcome,
data from about 10,000 participants from 67 trials was ‘wasted’
due to lack of IOP outcome data. Conclusions: Only a small
proportion of all identified trials reported complete VF data
for meta-analysis, a patient-centered outcome. Investigators
have a moral obligation to their participants and colleagues
to be more vigilant about complete reporting outcomes and
not ‘waste’ data. The comparative effectiveness of glaucoma
interventions could be better determined with more complete
reporting on VF and other patient-centered outcomes.
P1.049
P1.048
Over 56,000 participants’ data ‘wasted’: an
example from randomized controlled trials of
medical interventions for open-angle
glaucoma
Law A1 , Lindsley K1 , Rouse B1 , Wormald R2 , Dickersin K1 , Li T1
1
Cochrane Eyes and Vision Group US Satellite, USA; 2 Cochrane Eyes
and Vision Group Editorial Base, United Kingdom
Background: A completely specified outcome requires the
following five elements: domain (e.g. intraocular pressure
(IOP)), specific measurement (e.g. applanation tonometry),
specific metric (e.g. change from baseline), method of
aggregation (e.g. mean), and time-point (e.g. three months).
To perform a meta-analysis of the effect of an intervention, one
also needs to know the estimate of treatment effect as well as
its precision (e.g. standard error) from each trial. Incomplete
reporting of an outcome prevents its full use in a systematic
Quality of published randomized controlled
trials (RCTs) in Thai healthcare journals
Pengput A1 , Pattanittum P2
1
Master Degree in Public Health Student, Biostatistics Program,
Faculty of Public Health, Khon Kaen University, Thailand;
2
Department of Biostatistics and Demography, Faculty of Public
Health, Khon Kaen University, Thailand
Background: A well-designed randomized controlled trial
(RCT) provides the most reliable evidence of the effectiveness
of interventions –the gold standard for clinical trials. Many
RCTs are published each year with either trivial or serious
flaws. Some serious flaws lead to misleading conclusions
or failure to provide important information to guide clinical
decision making. The more flaws, the lower the quality of
RCTs. A way to assess the quality of RCTs is to consider the
reporting quality; the CONSORT (Consolidated Standards of
Reporting Trials) statement is a tool. There is no research
that evaluates the reporting quality of RCTs published in Thai
healthcare Journals. Objectives: To describe the quality
184
of published RCTs in Thai healthcare journals. Methods: We
searched for Thai-language RCTs published in Thai healthcare
Journals from January 2008 to December 2012 by using an
Online Public Access Catalog (OPAC) of Khon Kaen University
and Thai University Library Network of OPAC. For each full
RCT text we assessed the reporting quality by means of a
questionnaire based on the revised version of the CONSORT
statement 2010 (a 25-main item checklist; consists of 37
items). We recorded the quality of reporting presented by
each item of the CONSORT checklist, and also the overall
quality of included RCTs. Results: The search identified 757
RCTs, 35 of which met the inclusion criteria. Result by each
item of CONSORT: 18 of 37 (48.6%) items were reported in at
least 75%. Some important items were inadequately reported;
item 7a sample size determination (40%), item 8a sequence
generation (57%), item 9a allocation concealment mechanism
(43%), item 11a blinding (57%), item 13a participant flow (9%),
items 17a and 17b reporting of estimated effect size and its
precision (14%, 13%). The least reported items were item 23
registration number and name of trial registry (1/35), item 24
where the full trial protocol can be accessed (1/35), item 12b
statistical methods for additional analyses (2/14). The overall
quality was moderate; 28 RCTs reported on 19 to 27 items.
Conclusions: The quality of reporting RCTs in Thai healthcare
journals needs to be improved.
P1.050
and sentences; - reporting of setting, recipient, provider,
procedure, materials and intensity; - use of descriptive aids
such as tables and access to supplementary documents. We
compared the trials’ description of their intervention between
the groups according to whether the intervention was found to
be effective, and whether it was published in a leading general
medical journal. Results: We have extracted intervention
content of the trials. Overall reporting of the content was
variable. Early results suggest that trials published after
2008 generally provide longer descriptions of the intervention
of interest than those published before 2008 (median word
length pre 2008 was 236, post 2008 it was 296). Conclusions:
To our knowledge, this is the first comparative study of
standards of historical and current intervention reporting.
Our results will identify areas for improvement. The result
may inform future guidelines.
Reference
Nieuwlaat R, Wilczynski N, Navarro T, Hobson N, Jeffery R,
Keepanasseril A, Agoritsas T, Mistry N, Iorio A, Jack S,
Sivaramalingam B, Iserman E, Mustafa RA, Jedraszewski D,
Cotoi C, Haynes RB. Interventions for enhancing medication
adherence. Cochrane Database of Systematic Reviews
2014, Issue 11. Art. No.: CD000011. DOI: 10.1002/14651858.
CD000011.pub4.
P1.051
Reporting of interventions: a comparative
case study before and after the publication of
the extended CONSORT guidelines for
reporting non-pharmacological trials
1
1
1
1
Candy B , Vickerstaff V , Jones L , King M
1 University College London (UCL), United Kingdom
Background: Limited description of interventions in
trial publications wastes resources and potentially harms
patients. Interventions cannot be replicated or assessed
for generalizability if they are poorly described. Even
if found to be ineffective, complete description of an
intervention is needed to reduce resource waste and inform
approaches. Recognition of the need to improve reporting is
not new. Both the 2008 extension to CONSORT statements
for non-pharmacological interventions, and the 2010 revised
CONSORT statement emphasized the importance of full
reporting of the details of an intervention. Objectives:
We explore whether intervention reporting is improving.
We compare descriptions of similar non-pharmacological
interventions in trials undertaken before and after 2008.
Methods: We reviewed 182 trials included in a Cochrane
Review on interventions for enhancing therapy adherence
(Nieuwlaat 2014). We included for increased similarity
only psychotherapeutic and educational multi-component
interventions. We split the trials into two groups: (1) those
published before 2008; and (2) those published after 2008. Two
authors independently assessed intervention description. We
compared description according to: - number of words
Misleading reporting in major fertility
journals: poor use of confidence intervals and
absolute differences
Glujovsky D1 , Sueldo C2 , Borghi C2 , Nicotra P2 , Andreucci S2 ,
Ciapponi A1
1
IECS - Instituto de Efectividad Clı́ nica y Sanitaria, Argentina;
2 CEGYR - Centro de Estudios en Genética y Reproducción, Argentina
Background: To get a correct interpretation of the findings in a
study, the authors should use correct measures to report them.
Although there are lots of items recommended by reporting
guidelines, some, specifically, could have a high-impact on
the key message for the reader. Some of them (i.e. P
value and relative risks) are more frequently used than others
(confidence intervals (CI) and interpretation of absolute risks
differences), and missing these others provides an incomplete
overview of the clinical scenario. Objectives: To evaluate the
proportion of randomized controlled trials (RCTs) published in
the top fertility journals that reported their results providing
the estimated effect size and its precision; and for binary
outcomes, the presentation of both absolute and relative
effect sizes. Additionally, we did a descriptive analysis of
the interpretation of those results. Methods: All the RCTs
published in top fertility journals in 2014 were analyzed
by pairs of independent reviewers, evaluating the titles and
abstracts of identified articles. As a second step, two randomly
selected independent reviewers, assessed each included
study to finally include them in the analysis and to do the
data extraction. Discrepancies were resolved by consensus.
185
We analyzed the abstract and the results section of the full
text separately to see if the authors mentioned the P value
and a CI for the main outcome and for secondary outcomes.
For binary outcomes, we evaluated if any relative measure
(i.e. RR, odds ratio or relative risk reduction) was used and if
any absolute measure (i.e. absolute risk reduction or number
needed to treat) was mentioned. For continuous measures,
we evaluated if the mean difference and its CI was used.
We analyzed whether any reference related to any clinically
relevant result was mentioned in the Materials and methods
section. We also evaluated the Discussion and Conclusions
and described the authors’ interpretation from the results
that they published. Results: While more than 90% reported
P value, less than 50% reported CI and absolute measures.
More results and conclusions will be shown at the Colloquium.
P1.052
Critical reporting elements for publications
of research and systematic reviews of
complex interventions
Guise J1 , Viswanathan M2 , Chang C3 , Butler M4 , Kondo K5 ,
Dickinson C1 , Arbues F5 , Motu’apuaka M5 , Pigott T6 , Tugwell P7
1 US Cochrane West, Oregon Health, Science University, USA;
2
Research Triangle Institute, UNC RTI Evidence-Based Practice
Center, USA; 3 Agency for Healthcare Research & Quality, USA;
4 University of Minnesota Evidence-Based Practice Center, USA;
5
Scientific Resource Center for the AHRQ Effective Health Care
Program, USA; 6 Loyola University Chicago, USA; 7 Cochrane Health
Equity Field/Campbell Equity Methods Group, Canada
Background: Complex interventions are widely used in
healthcare and public health and are increasingly the subject
of systematic reviews. Inconsistencies and shortcomings in
the reporting of complex interventions hamper systematic
review efforts. Objectives: To develop a standard list of
critical elements to report in publications of research and
reviews of complex interventions. Methods: We reviewed the
literature, interviewed international experts, and conducted
a modified Delphi consensus process among international
experts in evidence reviews, research, and implementation of
complex interventions to generate a final checklist of reporting
items for publications of complex interventions. Results:
Items considered for reporting of research and reviews of
complex interventions included: intervention characteristics
such as the number and type of components, adaptation
over time, active and optional components, independence
versus dependence among components, theoretical,
conceptual, and/or functional foundation for the intervention,
governance, skills and training of agents delivering
the intervention; population characteristics including the
intended recipient (e.g. individual, population, health
system), cultural, educational, demographic, medical, and
skills required to participate; setting/context characteristics
including organizational readiness to change or adoption
of intervention, key attributes of champions; comparator
characteristics including describing ‘usual care’ and rival
activities; and outcomes including expected outcomes,
unintended consequences, resource utilization, and cost. We
will report Delphi results including the final reporting elements
checklist. Conclusions: Space is limited in publications yet
sufficient details are required to enable wide-scale successful
adoption. This checklist may help authors with consistent
language and reporting of individual studies, help systematic
reviewers focus on key considerations, and inform future
research.
P1.053
Developing a Preferred Reporting Items for
Development of Evidence based Clinical
Practice Guideline in Traditional Medicine
(PRIDE-CPG-TM)
Choi J1 , Jun JH1 , Lee JA1 , Choi TY2 , Lee MS2 , Shin B3 , Choi J2 ,
Choi J2
1
Korea Institute of Oriental Medicine, South Korea; 2 Korea Institute
of Oriental Medicine, South Korea; 3 School of Korean Medicine,
Pusan National University, South Korea
Background: The development of evidence based clinical
practice guidelines (CPGs) is a challenge for all guideline
developers in traditional medicine. Using a publication
platform with reporting items may overcome these
challenges. Objectives: The aim of this study was to develop
a unified methods platform with preferred reporting items
for development of evidence-based (PRIDE) clinical practice
guidelines (CPG) in traditional medicine (TM). Methods:
We searched the literature and guideline handbooks, and
collected information from methodological conferences. A
group of experts edited drafts of the PRIDE, identifying for
each item of one or more exemplars of good reporting,
and developed text explaining the rationale and discussing
relevant evidence. To develop the PRIDE, we also used general
and various guideline checklists and drew on the experience
gained from Oriental medical doctors and methodological
experts. Results: We designed the Preferred Reporting
Items for Development of Evidence-based Clinical Practice
Guidelines in Traditional Medicine (PRIDE-CPG-TM), in the
form of a checklist with descriptions of items and examples,
to improve reporting of CPG in TM, and so facilitate their
interpretation and replication. The PRIDE included five items
and 40 sub-items. These items pertain to the development
methodology (22 items), overview of diseases and symptoms
(six items), recommendations (four items), implementation
and dissemination (five items) and the appendix (three items,
i.e. glossary, references etc.). Conclusions: The completeness
of descriptions in CPGs is very poor. Therefore, a complete
description of the recommendation of TM in CPGs is necessary
for physicians to be able to use the recommendations
in areas of clinical practice. The PRIDE will be useful
guidance for TM developers when developing evidence-based
guidelines.
186
Conclusion: Substantial room for improvement remains for
the reporting of surgical RCTs. A guideline that address
issues specific to surgical RCTs should be developed by
methodologists, journal editors and surgeons.
P1.056
P1.055
Huang T1 , Tam K2
Departmant of Nursing, College of Medicine and Nursing,
HungKuang University, Taiwan; 2 Department of Surgery, Taipei
Medical University, Shuang Ho Hospital, New Taipei, Taiwan
Comparison of the attention paid to reporting
guidelines and trial registration in nursing
journals in the last three years
1
How many reports you can trust? A
cross-sectional study to characterize the
quality of research reporting
1
1
1
2
1
Zhang L , Zhao P , Jia P , Chen Q , Zhang M
1
Chinese Evidence-Based Medicine Centre, West China Hospital,
Sichuan University, China; 2 West China School of Public Health,
Sichuan University, China
Background: Muir Gray once said ‘In the 19th century
health was transformed by clear, clean water. In the 21st
century health will be transformed by clean, clear knowledge.’
Nowadays, substantial articles are published in various
magazines. Unless research is adequately reported, the time
and resources invested in its conduct is wasted. Objectives:
To use the reporting guidelines Consolidated Standards of
Reporting Trials (CONSORT) and Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA) to compare
the quality of reports between 2004 and 2014. Methods:
We conducted a systematic review (SR) of 400 randomized
controlled human trials and 100 SRs published in MEDLINE,
Embase and the Cochrane Library. We randomly selected
eligible pieces of research until their number met our required
sample size. According to reporting guidelines CONSORT
and PRISMA, we assessed these randomized controlled trials
and SRs to one of three grades (completely compliant with
requirements; generally compliant with requirements, and
non-conformant with compliance requirements). Teams
of two reviewers will independently screen full texts of
reports for eligibility, and abstract data, using standardized,
pilot-tested extraction forms. We will conduct univariable
and multivariable logistic regression analyses to examine the
association of pre-specified study characteristics (such as
funding sources, clinical area, type of intervention, and so on)
with the quality of research reporting.
Progress and Discussion: Currently, we are screening these
trial and SR reports. Reporting guidelines such as CONSORT,
PRISMA aim to improve the quality of research reports, but
all are much less adopted and adhered to than they should
be. Adequate reports of research should describe clearly
which questions were addressed and why, what was done,
what was shown, and what the findings mean. Our findings
will contribute to a set of recommendations to optimize the
conduct and reporting of randomized controlled trials and
SRs with meta-analysis.
Background: Reporting guidelines listed at the website of
the EQUATOR network, such as CONSORT and PRISMA, are
known to improve reliability of medical research literature by
promoting transparent and accurate reporting of research
studies. Similarly, trial registration improved research
transparency and will ultimately strengthen the validity and
value of the scientific evidence base. Most biomedical journals
require authors to ensure that the manuscript includes all
the information recommended in the relevant reporting
guidelines, and support clinical trial registration policies by
the World Health Organization. However, the attention paid to
reporting guidelines and trial registration in nursing journals
is unclear. Objectives: The study is aimed to evaluate the
editorial policy of nursing journals in the basic requirements
for reporting guidelines and trial registration in the last three
years. Methods: The ‘Instructions for Authors’ of nursing
journals included in ISI Web of Science was reviewed for
evidence of an editorial policy on the reporting guidelines and
trial registration in March 2012 and March 2015. Results: Of
89 nursing journals examined in 2012, 25 (28.1%) required
authors to comply with recommendations in the relevant
reporting guidelines, and only seven (7.7%) required clinical
trial registration. In 2015, 107 nursing journals were evaluated,
43 (40.2%) required authors to follow the reporting guidelines,
and 13 (12.1%) do not consider the trials for publication unless
they have been registered prospectively before recruitment of
any participants. Conclusions: Three years have passed, most
nursing journals still do not require the authors to comply
with the recommendation of relevant reporting guidelines,
nor claim the need of trial registration prior to the start of
patient enrollment in a clinical study. For the purpose of
improved reliability in nursing research literature, there is
a need for nursing journals to post a requirement on the
reporting guidelines and trial registration.
P1.057
The information need in systematic review
and meta-analysis about acupuncture
Wang X1 , Wei D1 , Liu Y1 , Wang Q2 , Qu Q2 , Yang K1
1
Lanzhou University, Key Laboratory of Evidence-Based Medicine
and Knowledge Translation of Gansu Province; Chinese GRADE
Center, China; 2 School of Basic Medical Sciences, Lanzhou
University, China
187
Background: As acupuncture becomes more and more
popular around the world, the amount of research on
it, both primary studies and systematic reviews/metaanalysis(SR/MAs), is increasing rapidly.
However, the
reporting quality of SR/MAs is poor. The Standards for
Reporting Interventions in Clinical Trials of Acupuncture
(STRICTA) have shown positive work for reporting quality
of clinical trials of acupuncture. Thus, it might be useful to
develop an extension of PRISMA for acupuncture to improve
the reporting quality of SR/MAs of acupuncture. Objectives:
Our first step was to investigate the need for information in
SR/MAs of acupuncture from the perspectives of clinicians,
researchers, master students and doctors. Methods: Based
on a literature review and the PRISMA statement, we designed
a questionnaire, containing three parts: the demographic
information of respondents; the experience of respondents;
the importance of the proposed items. We selected five
cities from south, north, north-west and south-west China,
to conduct our survey in person, and interviewed those who
major in Chinese acupuncture with more than one year’s
experience. Results: A total of 269 questionnaires were
collected in 18 hospitals, medical universities and research
agencies, and 251 (93%) with complete data were used for
analysis. The average age of respondents was 33 years.
The length of most respondents’ service was less than five
years and they read between one and five relevant articles
per month. Various approaches for obtaining important
information were used by interviewees, and electronic
databases, search engines and academic conference were the
most used. The most popular types of literature were RCTs
(67%), observational studies (51%), basic research (46%), and
SR/MAs (37%). Fifty-six per cent of the respondents indicated
poor satisfaction with the completeness of information in the
literature. Conclusions: Clinicians, researchers and students
have a pressing need for complete and critical information
from SR/MAs in acupuncture.
P1.058
How many journals adopt reporting guidance
for RCTs, systematic reviews and Clinical
Practice Guidelines in ‘Instructions for
authors’? A cross-section survey from 150
medical journals
Yao L1 , Chen Y1 , Fu S1 , Wang X1 , Wang Q1 , Wei D1
1
Background: Empirical evidence suggests that active
implementation of the CONSORT and other reporting
guidance by journals can lead to improvements in the
reporting of trials and other studies. Objectives: To
investigate how many medical journals adopt reporting
standards for randomised controlled trials (RCTs), systematic
reviews (SRs) and Clinical Practice Guidelines (CPGs) in their
‘Instructions for authors’. Methods: We used highly sensitive
search filters were used to identify RCTs, SRs and CPGs
in PubMed. Then we selected the top 50 journals that
published each type of study and checked whether CONSORT,
PRISMA and reporting criteria for CPGs were adopted in
their instructions for authors. Results: The results showed
that 28 (56%) journals adopted CONSORT, 20 (40%) journals
adopted PRISMA. For CPGs, only one (2%) journal had
adopted Conference on Guideline Standardization (COGS) as a
reporting standard. Apart from that, two journals mentioned
in their instructions that when authors develop guidelines,
they should follow Appraisal of Guidelines for Research
and Evaluation (AGREE) and the Institute of Medicine (IOM)
standard. Conclusions: We suggest that medical journals that
publish CPGs should introduce relevant reporting standards in
their ‘Instructions for authors’. Guideline authors also need to
report their guidelines clearly, transparently and standardly.
P1.060
Reports on animal experiments involving
neoplasms in Chinese Journals: adherence to
ARRIVE guidelines
Liu Y1 , Zhao X2 , Mai Y2 , Li X2 , Wang J2 , Chen L2 , Mu J2 , Jin G2 , Gou
H2 , Sun W2 , Feng Y2
1
Evidence-Based Medicine Center, School of Basic Medical
Sciences, Lanzhou University, Key Laboratory of Evidence-Based
Medicine and Knowledge Translation of Gansu Province, China;
2
Evidence-Based Medicine Center, Lanzhou University, China
Background: The Animals in Research: Reporting In Vivo
Experiments (ARRIVE) guidelines were published in 2010 with
the aim of improving the quality of animal study reports. It
is not clear how well animal experiments on neoplasms in
China adhere to these reporting standards. Objectives: To
evaluate the reporting quality of animal experiments involving
neoplasms that were published between 2010 and 2012 in
Chinese journals and were supported by the National Natural
Science Foundation of China (NSFC). Methods: A literature
search of studies published between 2010 and 2012 was
performed using the Chinese Science Citation Database and
the Chinese Journal Full-Text Database. Data were extracted
into pre-prepared forms. Reporting quality was assessed
using the ARRIVE checklist (40 (39+1) items). Results: A
total of 396 animal studies were included in the analysis: 127
published in 2010, 140 published in 2011, and 129 published
in 2012. The range of ARRIVE score was 11 to 32. The
studies published in 2012 have a greater average ARRIVE
score than those published in 2010 (P value 0.034). There was
no significant difference between the 2010 and 2011 ARRIVE
scores (P value 0.452) or the 2011 and 2012 scores (P value
0.154). Conclusions: Animal experiments of neoplasms in
published in Chinese journals did not comprehensively report
the information recommended by the ARRIVE guidelines.
We strongly recommend researchers report this information
when reporting animal studies.
188
P1.061
The reporting of items generation and
consensus in reporting guidelines: a cross
sectional study
Chen Y1 , Wang X1 , Wang Q1 , Yao L1 , Wei D1 , Wang M1 , Yang K1
1 Lanzhou University/Chinese GRADE Center, China
Background: Empirical evidence suggests that active
implementation of the CONSORT and other reporting
guidelines by journals can promote improvements in the
reporting of trials and other studies. Objectives: To
investigate how experts generate items and reach consensus
when they develop reporting guidelines. Methods: We
searched the EQUATOR library (equator-network.org) on 20
March 2014. Each guideline was reviewed by two independent
authors. We only included reporting guidelines on clinical
research and systematic reviews. We excluded extension
versions and older versions. Results: We identified more than
200 reporting guidelines, but finally 18 met our criteria. The
results showed that 14 (78%) described methods of reporting
items generation, and 10 (56%) of them adopted more than
one approach. The top three methods were literature reviews
(13; 72%), adapted existing reporting guidelines (9; 50%),
and surveys and interviews (3; 17%). For items consensus,
13 (72%) described methods, 12 (67%) of them mainly used
consensus meeting, and four (22%) of them used modified
Delphi. Conclusions: Most reporting guidelines described
the methods about items generation and consensus in their
reporting guidelines, but few of them detailed the processes
and implementation.
Meta-analysis methods
P1.062
The Peto odds ratio viewed as a new effect
measure: derivation and application
Brockhaus AC1 , Bender R1 , Skipka G1
1 Institute for Quality and Efficiency in Health Care (IQWiG), Germany
Background: Meta-analysis has generally been accepted as a
fundamental tool for combining effect estimates from several
studies within the framework of systematic reviews. In the
case of binary data and rare events, the Peto odds ratio
(POR) method has become the relative effect estimator of
choice. However, as shown in several simulation studies,
the POR leads to biased estimates for the odds ratio (OR)
when treatment effects are large or the group size ratio is
not balanced. This leads to the hypothesis that the POR
estimate does not converge towards the true OR even for
rare events. Objectives: To investigate the limit of the POR
estimator for increasing sample size and its deviation from
the OR. Methods: We derived the limit of the POR estimator
for increasing sample size by application of the delta method.
We investigated in which data situations the POR limit is
sufficiently close to the true OR. Results: It was found that
the derived limit of the expected POR is not equivalent to the
OR, because it depends on the group size ratio. Thus, the
POR represents a different effect measure. We investigated
in which situations the POR is reasonably close to the OR
and derived the maximum effect size of the POR for different
group size ratios and tolerated amounts of bias, for which
the POR method results in an acceptable estimator of the
OR. Conclusions: The POR can be considered as a new effect
measure. The POR method can be used as a valid estimate of
the OR only in in the presented situations.
P1.063
Who should benefit from screening? Are we
using the right denominator?
Scheibler F1 , Rummer A1 , Sauerland S1 , Grosselfinger R1
1
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen
(IQWiG), Germany
Background: Many screening questions face the problem
that, due to low disease prevalence, studies need to
have enormous sample sizes in order to detect statistically
significant effects on patient-relevant outcomes. Methods:
We report on the results of an IQWiG benefit assessment
of neonatal pulse oximetry screening (POS) for detection
of critical congenital heart disease (cCHD). We compared
two different analytical strategies to assess the benefits of
POS: the ‘classical’ intention-to-screen (ITS) analysis using all
participants as denominator and the ‘alternative’ analysis
using only those participants who were affected by the
disease. Results: Only one concurrent controlled study
(de Wahl-Granelli 2009) could be included in the systematic
review. Based on all 155,567 newborns (ITS analysis), the study
failed to show a statistically significant effect of screening on
severe acidosis (OR 0.490 (0.217 to 1.109) P value 0.086). Using
only babies with cCHD as denominator (n = 160), the study
reported a significant effect on severe acidosis (OR: 0.268
(0.110 to 0.654) P value 0.003).
Discussion: In most screening trials, it is useful to analyze
results based on the ITS approach, because the screening
interventions themselves affect the prevalence of disease (e.g.
by detecting clinically insignificant cases). Thus, using only
participants with the disease as denominator might introduce
bias into the analysis. In the context of cCHD, however, all
affected newborns are destined to die when left untreated, so
bias is unlikely. Nevertheless, for comprehensive assessment
of a screening intervention, harms have to be examined within
the total target population.
Conclusion: It is obvious that the benefit of screening
primarily evolves from the treatment of affected people.
Therefore, if bias is unlikely, the demonstrated benefit
in diseased people may justify the implementation of the
program, even if the effect on the screened population
fails to demonstrate significance. Therefore, if possible
and reasonable, when evaluating screening interventions
an analysis of results using the affected population as
denominator seems to answer additional relevant questions.
189
Reference
P1.065
de Wahl-Granelli. Impact of pulse oximetry screening
on the detection of duct dependent congenital heart
disease: a Swedish prospective screening study in 39 821
newborns. BMJ 2009; 338:a3037. doi: http://dx.doi.org/
10.1136/bmj.a3037 (Published 09 January 2009).
Treatment benefit cannot be excluded on the
basis of mean difference alone
P1.064
Assessment of use and scope for use of IPD in
a random sample of 100 reviews from the
Cochrane Pregnancy and Childbirth group
Sudell M1 , Neilson J2 , Tudur-Smith C1
1
University of Liverpool, United Kingdom; 2 Cochrane Pregnancy &
Childbirth Group, United Kingdom
Background: Individual Patient Data (IPD) meta-analyses
have been described as the ‘gold standard’. However, they
are still relatively uncommon within the Cochrane Library
and there may be increased potential to improve the findings
of traditional meta-analyses by including IPD. Furthermore,
traditional reviews may be a useful vehicle for highlighting
the added value of IPD for future research. Objectives: To
assess the current use, and potential for use of IPD in current
Cochrane Reviews produced by the Cochrane Pregnancy and
Childbirth Group. Methods: A random sample of 100 of the
reviews available on the Cochrane Pregnancy and Childbirth
website were downloaded. Various aspects of the reviews
were recorded including number of included studies, number
of comparisons, risks of bias in included studies and number
of heterogeneous comparisons. Any reference to current
or future use of IPD was recorded, as well as planned and
executed subgroup analyses. A recommendation for whether
IPD would be valuable was given based on characteristics of
the reviews. Results: Ninety of the 100 reviews included at
least one study and 10 were empty. IPD was likely or possibly
in use in 11 of the 90 non-empty reviews, however the methods
were not well discussed. Seeking IPD was recommended for
64 of the non-empty reviews, and recommended subject to
advice from a clinician in a further seven reviews. Common
reasons for recommending that a review should seek IPD
included presence of heterogeneity that could be explored
with IPD to overcome potential bias due to selective reporting
or incomplete outcome data, and to overcome insufficient
data to complete all planned analyses. Conclusions: Our
investigation indicates that few Cochrane Pregnancy and
Childbirth reviews currently include IPD, but there is potential
to overcome, or at least minimise important issues, by
including IPD. However, from a clinical perspective it may not
always be justified to seek IPD, as the traditional approach
using aggregate data may be sufficient to answer the research
questions, and more resources, time and statistical expertise
are required for IPD meta-analyses.
Cates C1
1 Airways Group, United Kingdom
Background: The GRADE (Grading of Recommendations
Assessment, Development and Evaluation) recommendations
(for downgrading due to imprecision) encourage Cochrane
authors to be ‘confident’ that a treatment is clinically
unimportant if the mean difference and 95% confidence
interval (95% CI) fall short of the clinical minimally important
difference (CMID). Objectives: To demonstrate that a mean
difference alone cannot rule out important clinical benefit in
a population. Methods: A Cochrane Airways review will be
used to present data to show that a mean difference that is
clearly less than the CMID can lead to an important shift in the
population of those who respond to treatment (also known as
‘responders’ who individually exceed the CMID). A responder
analysis will be compared to the results of modelling the shift
in the population mean. Results: A statistically significant
increase in the proportion of responders cannot be ruled out by
a mean difference and 95% CI that are entirely below the CMID.
Conclusions: It is wrong to rule out a treatment as clinically
unimportant on the basis of the mean difference alone. If
the mean difference shows a statistically significant benefit
(however small that might be), then a responder analyses is
required to assess the size of the benefit in a population.
P1.066
Different mortality time-points in critical care
trials: current practice and influence on
effect estimates
Roth D1 , Heidinger B1 , Havel C1 , Arrich J1 , Gamper G2 , Herkner H1
1
Medical University of Vienna, Department of Emergency Medicine,
Austria; 2 Universitätsklinikum St. Pölten, Austria
Background: Mortality is frequently used as a primary
outcome in critical care trials. It is a patient-orientated
variable and robust against information bias. Mortality
incidence however needs to be measured at a defined
time-point. Practice of meta-analysis in critical care shows
that follow-up times of trials in critical care medicine may
differ substantially. This has substantial implications on the
potential pooling of such mortality estimates. Objectives:
Describe the current practice of mortality follow-up time
definitions in a representative sample of published critical
care randomised controlled trials (RCTs) and analyze the
influence of handling of different follow-up times on
pooled effect estimates. Methods: We searched CENTRAL,
EMBASE, MEDLINE, PASCAL Biomed, and PsycINFO for studies
published after 2000 using the Cochrane RCT-filter and a
critical care-filter. A random sample of 50% was drawn for
further title and abstract review. Study characteristics such as
sample size, type of intervention, disease spectrum, intensive
190
care unit setting, hospital setting, funding and patient
characteristics including age, gender, severity of illness scores
were extracted, as well as number and time-points of mortality
ascertainment within individual studies. Meta-regression
and multilevel mixed-effects linear regression was used to
analyze the influence of follow-up time on deviation of pooled
risk ratios from study-baseline, using the aforementioned
study-characteristics as co-variables, allowing for clustering
on level of study and time-point. Results: Search resulted
in 9246 studies, 4573 of which were chosen as a random
sample. After title-, abstract- and full text-review, 106 studies
representing 63,713 patients were included; 60 (57%) studies
reported only one time-point, 26 (25%) reported two, 14 (13%)
reported three, 4 (4%) reported four, and 2 (2%) reported five.
No influence of time-point on effect estimates was found
using the aforementioned methods. Conclusions: In a large
sample of critical care RCTs, almost half the studies reported
more than one mortality time-point. We found no influence of
different time-points on pooled effect estimates.
P1.067
Results in systematic reviews of anaesthesia
interventions that are reported as
statistically significant often contain risk of
type I error greater than 5%
Imberger G1 , Gluud C1 , Boylan J2 , Wetterslev J1
1 Copenhagen Trial Unit, Centre for Clinical Intervention Research,
Rigshospitalet, Copenhagen University Hospital, Denmark;
2
Department of Anaesthesia, St. Vincent’s University Hospital,
Dublin, Ireland
Background: TheGRADEWorkingGroupprovidesanapproach
toassessingthequalityofabodyofevidencethatisincreasingly
being used, involving explicit consideration of all sources
of uncertainty. One of these sources is imprecision, or
random error. Many currently published meta-analyses are
underpowered and likely to be updated in the future. When
data are sparse and there are repeated updates, the risk of
random error is increased. Trial sequential analysis (TSA) is
one of several methodologies that estimates this increased
risk in meta-analyses. With nominally statistically significant
meta-analyses of anaesthesia interventions, we used TSA to
estimate power and imprecision in the context of sparse data
and repeated updates. Methods: We searched for systematic
reviews with meta-analyses that investigated anaesthesia
interventions.We randomly selected 50 meta-analyses that
reported a statistically significant dichotomous outcome in
their abstract. We applied TSA to these meta-analyses, using
two main TSA approaches: relative risk reduction (RRR) 20%
and RRR corresponding to the border of the conventional
95% confidence interval (CI) closest to the null. We calculated
the power achieved by each included meta-analysis, using
each TSA approach, and we calculated the proportion that
maintained statistical significance when allowing for sparse
data and repeated updates. Results: In our sample, the
median number of trials included was eight (interquartile
range (IQR) 5 to 14), the median number of participants was
964 (IQR 523 to 1736) and the median number of events was
202 (IQR 96 to 443). Using both of our main TSA approaches,
only 12% (95% CI 5% to 25%) of the meta-analyses had power
≥ 80%, and only 32% (95% CI 20% to 47%) of the meta-analyses
preserved the risk of type I below 5%.
Conclusion: The majority of nominally statistically significant
meta-analyses of anesthesia interventions are underpowered,
and a large proportion do not maintain their risk of type 1
error below 5% if TSA monitoring boundaries are applied.
A consideration of the effect of sparse data and repeated
updates is needed when assessing imprecision in anaesthesia
meta-analyses.
P1.068
False positive findings in cumulative
meta-analysis with and without application
of trial sequential analysis
Imberger G1 , Thorlund K1 , Gluud C1 , Wetterslev J1
1
Copenhagen Trial Unit, Centre for Clinical Intervention Research,
Copenhagen University Hospital, Denmark
Background:
The majority of meta-analyses are
under-powered. The risk of type 1 error in early meta-analyses
is higher than the conventional 95% confidence interval (CI)
and the associated probability of 5% suggests. Several
techniques have been used with the goal of accounting
for increased random type I error in the context of sparse
data and repeated updates in meta-analysis. This study
aimed to explore the role of one of these techniques
–trial sequential analysis (TSA) –in assessing the reliability
of conclusions in early meta-analyses. Methods: We
screened the Cochrane Database of Systematic Reviews
and selected 100 meta-analyses that were large enough to
have demonstrated, to a reasonable level, that the given
intervention does not cause a clinically relevant effect on the
outcome in question. We conducted retrospective cumulative
meta-analysis using conventional techniques and measured
the proportion of false positives that would have occurred.
For these false positives, we performed TSA, mimicking how a
prospective analysis could have been performed had it been
done at the time of publication of each new trial. We used
three different TSA approaches to mimic this prospective
analysis. As a post-hoc analysis, we surveyed three years
of Cochrane Systematic Reviews and calculated the relative
risk of a meta-analysis being updated if it was not significant
relative to if it were significant. Results: Using conventional
retrospective cumulative meta-analysis, one or more false
positives were present in seven of the meta-analyses (7%;
95% CI 3% to 14%). Using the three TSA approaches, TSA
prevented the false positive type 1 error in 13 of the 14 times
the conventional threshold was crossed (93%, 95% CI 64% to
100%). Having a non-significant result made a meta-analysis
from a Cochrane Systematic Review in the years 2005 to
2007 1.57 times more likely to be updated (95% CI 0.92 to
2.68). Conclusion: TSA is a helpful statistical methodology
when assessing the reliability of early nominally statistically
significant findings in cumulative meta-analyses.
191
P1.069
P1.070
Proving clinical relevance of effects in
meta-analyses of binary data
From meta-analysis to practical decision
using virtual realistic population simulation:
an illustration on sudden death prevention in
type 2 diabetes
Skipka G1 , Bender R1 , Lange S1
1
Institute for Quality and Efficiency in Health Care (IQWiG), Germany
Background: Often, it is not sufficient to prove statistical
significance of effects. Rather, the clinical relevance of
effects has to be shown. Usually, clinical trials are powered
for statistical significance only. Within the framework of
systematic reviews the power (precision) is increased by
pooling results of multiple trials. Therefore, more stringent
conclusions towards clinical relevance can be drawn without
loss of power. Objectives: To develop an approach to
prove clinical relevance in meta-analyses of binary data.
Methods: The starting point is a specified true effect that
is considered clinically relevant. Also in order to prove
clinical relevance, the probability of statistical errors should
be controlled. Therefore, the precision of the estimates has
to be considered. This is fulfilled by determining a threshold
for the confidence interval (CI) obtained by meta-analysis
which corresponds to the testing of shifted hypotheses. This
threshold has to be precisely selected so that a specified
power is preserved. The main idea is that the power of the
meta-analysis should be identical to the power for proving
statistical significance in the individual studies included in
the meta-analysis. Balanced sample sizes are assumed. This
approach will be investigated for the effect measures relative
risk (RR) and odds ratio (OR). Results: Simulations show that
the resulting thresholds depend on the baseline risk to some
extent. The OR is substantially more severely affected than the
RR. Therefore, the RR is favoured. In addition, the baseline risk
dependency nearly vanishes if using the restricted maximum
likelihood approach (REML) instead of the Wald approach for
the calculation of the CIs. Conclusions: Clinical relevance can
be shown without loss of power by taking advantage of the
higher precision of meta-analyses. The thresholds for clinical
relevance are nearly independent of the baseline risk if the
REML-approach is applied. Thus, the determination of these
thresholds only depends on the higher precision gained by
meta-analyses.
Le H1 , Marchant I2 , Boissel J1 , Nony P1 , Kassaı̈ B1 , Cornu C1 ,
Gueyffier F1
1
UMR 5558, Université de Claude Bernard Lyon 1, France;
2 Universidad de Valparaı́ so, Chile
Background: Meta-analysis in reviews systematically results
from numerous randomized controlled trials and supplies
synthesis information for clinical practice. However, there is
no standard procedure to predict the impact of translating
these results at a population level. Objectives: We propose a
strategy to simulate public health impact from the results of
meta-analyses combined with a risk score applied on a virtual
realistic population (VRP) platform. We simulated a preventive
strategy for sudden death (SD), a major cardiovascular
event, in a French population with type 2 diabetes (T2D).
We illustrated how this approach could help to rationalize
health public decisions. Methods: - Generate a French
diabetic VRP. - Establish a SD risk score. - Review existing
meta-analyses on therapeutic efficacy in preventing SD in
T2D. - Apply these results on the generated VRP with drug
combination and risk-based patient selection. Results:
A French diabetic VRP of 176,187 was generated from a
8995-patient sample, giving an median of SD risk of 1.7% at
a 5-year time horizon. We estimated the numbers needed
to treat (NNTs) by a simultaneous treatment by statin and
ACE inhibitor at 221 people for the whole population, and at
108/104 women/men among individuals of the highest 10%
predicted SD risk. The corresponding untreated risks of SD
were 1.9% on the whole population, and 3.9% and 4.1% in
top risk deciles for women and men respectively (Table 1).
Conclusions: Our approach, which gathers effect models
(via meta-analyses and risk scores) and VRP simulation,
provides a powerful multi-component tool for valuing each
evidence-based component, better transposing clinical trial
results to practice, and facilitating clinical decision in both
public health and individual levels, on both medico-economic
aspects. Key words: health public simulation; meta-analysis;
practical decision making; sudden death; virtual realistic
population.
192
P1.072
A network meta-analysis of randomized
controlled trials of interventions for the
treatment of rotator cuff tears
Gagnier J1 , Beyene J2
1 University of Michigan, USA; 2 McMaster University, Canada
Background: Rotator cuff tears are common and increasing in
prevalence in our aging population. Rotator cuff tears can be
treated through various surgical procedures and rehabilitative
programs, many of which have not been compared directly to
each other. Using a network meta-analysis allows comparison
and facilitates pooling of a diverse population of randomized
trials across these approaches in ways that a traditional
meta-analysis does not. Objectives: Our aim is to perform
a network meta-analysis using evidence from randomized
trials on the relative effect of alternative approaches on
recovery in patients who have suffered from a rotator cuff
tear. Our secondary study endpoints included a repeat tear.
Methods: A network meta-analysis allows for simultaneous
consideration of the relative effectiveness of multiple
treatment alternatives. We began with systematic searches of
databases (including EMBASE and MEDLINE) and performed
hand searches of orthopaedic journals, bibliographies,
abstracts from orthopaedic conferences, and orthopaedic
textbooks, for all relevant material published from each
databases’ inception to 2015. Two authors independently
screened abstracts and manuscripts and extracted the
data, three evaluated the risk of bias in individual studies,
and two applied Grading of Recommendation Assessment,
Development and Evaluation (GRADE) criteria to bodies of
evidence. We included all randomized and quasi-randomized
trials comparing two (or more) treatment options for rotator
cuff tears in predominantly (i.e. > 80%) adult patients. We
calculated pooled estimates for all direct comparisons and
conducted a network meta-analysis combining direct and
indirect evidence for all comparisons. Results: We included
35 papers comparing two or more interventions for rotator
cuff tears. A total of four interventions were tested across
trials. We are currently performing the statistical analyses for
the network comparisons. Conclusions: This will be the first
network meta-analysis done for clinical trials of rotator cuff
tears. These findings will help guide clinical decision making
and the design of future randomized studies.
193
P1.073
Exploring the efficacy effectiveness gap
Koesters M1 , Holtrup A1 , Fiedler I1 , Becker T1
1
Department of Psychiatry II, Ulm University, Germany
Background: There is broad consensus that there is a
gap between the ‘efficacy’ of a drug proven in randomized
controlled trials (RCTs) and the ‘effectiveness’ of that drug in
everyday clinical practice. This consensus is mainly based on
anecdotal evidence. Objectives: This study systematically
reviewed and compared evidence from RCTs and nonrandomized and uncontrolled trials examining the
effectiveness of venlafaxine and duloxetine in the treatment
of depression. Methods: The systematic review used pre-post
effect sizes to compare treatment effects from RCTs and
non-RCTs examining the effects of duloxetine, venlafaxine
and placebo in the treatment of depression.
Data
were aggregated in random-effects models. Meta-regressions
were used to explore factors influencing the effect sizes.
Results: One-hundred and ninety relevant studies were
identified, and effect sizes for 114 acute-phase studies could
be calculated. All interventions, including placebo, showed
a statistically significant pre-post effect of more than two
standard deviations. Between-study heterogeneity was high.
However, there were no significant differences of effect sizes
from RCTs and non-RCTs, and metaregressions did not show
a significant influence of external validity measured by the
Downs and Black scale. Conclusions: The analyses could
not confirm the presence of an efficacy-effectiveness gap.It
is possible that the impression of such a gap existing is a
consequence of publication bias rather than being due to
RCT-practice differences.
P1.074
Application of Bayesian methods in public
health systematic reviews
Lewis MG1 , Nair S2
1
PhD student, India; 2 Director Public Health Evidence South Asia
(PHESA), India
Background: Bayesian meta-analysis has gained popularity
in the field of evidence synthesis.
Public health
interventions are targeted to highly heterogeneous
populations, multi-component interventions, multiple
outcomes, influenced by context and, moreover, the answer
to a question comes from a hierarchy of studies, which
is in contrast to the clinical interventions.
In such
situation Bayesian methods have a wide scope of application.
Objectives: To develop methods to incorporate Bayesian
methods into public health evidence synthesis. Methods:
Public health systematic reviews were identified which
included studies with different type of study designs for a
single question. Data were extracted from those studies
for the outcome(s) concerned. Bayesian meta-analysis was
performed with likelihood from the stronger studies and the
results of the weaker studies as prior. The analysis was
performed in WINBUGS software with 5000 iterations for
burning the sampler and 10,000 iterations for estimation.
Convergence was assessed using autocorrelation and density
plot. A simulation study was also performed with different
meta-analysis situations with repetitions of 1000 in each case.
Results: The concept has been applied in ten public health
systematic reviews, which included different study designs
to answer the same question. Different combinations of
priors and likelihood were drawn and odds ratio, relative
risk with 95% CrI, CI were calculated. The results were then
compared with the estimates of traditional meta-analysis
with the posterior estimates of Bayesian. Conclusions: The
present paper describes a mechanism to incorporate all the
levels of a body of evidence for a single question using the
Bayesian approach.
P1.075
An empirical study investigating the extent of
heterogeneity, inconsistency and potential
limitations in networks of interventions
published in 2013
Gianatsi M1 , Nikolakopoulou A1 , Chaimani A1 , Salanti G1
1
Department of Hygiene and Epidemiology, School of Medicine,
University of Ioannina, Greece
Background: Network meta-analysis (NMA) has become
an increasingly popular methodology and several Cochrane
Reviews comparing multiple interventions have been
published over the last years.
Despite the rapid
methodological development of NMA, the good practice
requirements of this advanced evidence synthesis tool are
often ignored or improperly applied. Objectives: To
update the current empirical evidence on characteristics
of published NMAs, including their methodological rigor
and quality of reporting, and to evaluate empirically the
extent of heterogeneity and inconsistency in networks of
interventions. Methods: We searched MEDLINE, the Cochrane
Library and EMBASE to identify NMAs published in 2013.
We extracted data on various network characteristics and
assessed how often researchers evaluated and reported the
assumptions underlying NMA appropriately. We documented
the methods employed to estimate and present relative effects
and treatment ranking and assessed the extent to which
authors discussed the potential for bias in their findings.
We estimated three indices that quantify the percentage of
variability attributed to heterogeneity, inconsistency, or both,
for each primary outcome. Results: Our search identified 72
networks that met all inclusion criteria. So far we have
extracted data from 40 networks: only 25% discussed how
plausible the assumption of NMA is. Inconsistency was
assessed statistically in less than half of the networks (43%),
while issues of bias (e.g. publication bias or attrition bias) were
discussed in about one in six networks. The I-squared statistic
for heterogeneity or inconsistency ranged between 25% and
75% in almost half of the networks. Results from the 72
194
networks will be presented at the Colloquium. Conclusions:
Assumptions and potential limitations of NMA are not widely
discussed in publications and this casts doubts about the
credibility of their conclusions. The majority of networks in
our database presented a moderate level of heterogeneity
and inconsistency.
Siersma V, Als-Nielsen B, Chen W, Hilden J, Gluud LL,
Gluud C. Multivariable modelling for meta-epidemiological
assessment of the association between trial quality and
treatment effects estimated in randomized clinical trials.
Stat Med 2007;26:2745–58.
P1.077
P1.076
metaepi: an R package for conducting
meta-epidemiologic studies
Eusebi P1 , Abraha I1 , Cozzolino F1 , Siersma V2
1
Regional Health Authority of Umbria, Italy;
Copenhagen, Denmark
2
University of
Background: Various factors in the conduct of clinical trials
may bias the intervention effect. On the basis of theoretical
and empirical considerations, Cochrane identifies several
factors to be assessed for risk of bias in randomized trials.
Empirical evidence comes from meta-epidemiology, where
the intervention effects from the studies in a collection
of meta-analyses are compared between trials with and
without a particular characteristic. Meta-epidemiological
studies can be performed with a two-step approach (Sterne
2002) or a one-step approach (Siersma 2007). Investigation
of heterogeneity, sensitivity analyses and graphics are key
points. Outcomes can be continuous or discrete. Objectives:
To develop an R package for standardizing the procedure
of a meta-epidemiologic study. Methods: Several functions
are developed in the R package allowing for: (1) fitting twoand one-step models; (2) performing meta-regression and
subgroup analyses; (3) drawing Baujat plots, forest plots and
funnel plots. The package’s functions are illustrated on data
from a meta-epidemiologic study investigating the impact
of deviations from intention-to-treat on the intervention
effect (provisionally accepted for publication). Results: The
functions of the package deploy the R utilities in manipulating
data, the lme4 package for mixed-effects modelling and
the base graphics system. The functions are easy to use
and allow for well-structured meta-epidemiologic analysis.
Conclusions: The R package metaepi is a convenient tool
for facilitating the statistical analysis in a meta-epidemiologic
study and can provide guidance for the steps required in
performing such challenging task.
Reference
Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett
C, Egger M. Statistical methods for assessing the
influence of study characteristics on treatment effects
in ‘meta-epidemiological’ research. Stat Med 2002;21:
1513–24.
Similarity of the treatment benefit between
children and adults: results of a
Janiaud P1 , Lajoinie A1 , Cour-Andlaueur F2 , Cornu C1 , Cochat P3 ,
Cucherat M1 , Gueyffier F1 , Kassai B2
1 University of Lyon, France; 2 EPICIME-Clinical Investigation
Center, INSERM CIC201/UMR5558, France; 3 CHU Lyon, Service
de Pharmacologie Clinique, France
Background: Therapeutic benefit is commonly extrapolated
from adult clinical trials to children without conducting trials
in children. It has been widely recognized that children
cannot be provided with safe and efficacious drugs compared
to those available for adults without involving them in
clinical trials. To our knowledge, extrapolation of data
from adults to children based on the results of randomized
controlled trials (RCTs) has not consistently been explored.
Objectives: Our main objective was to see whether the
therapeutic benefit observed in placebo controlled RCTs is
different between adults and children. Methods: We searched
three electronic databases for meta-analyses that included
double-blind, placebo–controlled RCTs with separate results
for adults and children. The selected reviews were classified
according to disease and drug used. The heterogeneity
of treatment response between adults and children was
measured using ratio of odds ratios (RORs). Results: We
selected 89 meta-analyses and calculated RORs for 124 drugs.
Heterogeneity in the direction of the treatment effect was
observed in one drug and heterogeneity in the quantity of the
treatment effect in 13 drugs (Figure 1), indicating significantly
different treatment effect in adults when compared with
children. RORs were not significantly different from 1 for
110 drugs. For 36 of these drugs, the treatment effect
was confirmed in both populations. Conclusions: We
found different treatment benefits estimated by clinical
trials performed in adults compared with those performed
in children for 14 out of 124 drugs. Our results also
show the potential influence of pharmacokinetics and
pharmacodynamics and the need to better report on the
rationale of dose adjustments in RCTs. A better reporting of
age range and dose adjustment would allow full exploration
of possible sources of clinical and statistical heterogeneity,
enabling the prediction of situations where there may be
differences between adults and children.
195
196
P1.078
Similarity of the benefit risk ratio between
children and adults: results of a
Janiaud P1 , Chiganne M1 , Hergibo F1 , Araujo I1 , Kassai B2
1
University of Lyon, France; 2 EPICIME-Clinical Investigation Center,
INSERM 7 CIC201/UMR5558, France
Background: Adverse events (AEs) are a major cause of
morbidity and pose a substantial burden on limited healthcare
resources. Drug efficacy and safety data are often extrapolated
from adults to children. Children can be more resilient to
AEs or more vulnerable. Objectives: Our objective was to
assess whether the benefit-risk ratio is different between
adults and children using evidence from placebo-controlled
randomized clinical trials (RCTs). Methods: We searched
three electronic databases for meta-analyses that included
double-blind, placebo-controlled RCTs with separate results
for adults and children. For the quantitative synthesis only
meta-analyses that had an efficacy primary outcome and
at least one safety outcome (adverse events (AEs) and/or
withdrawals due to AEs) were retained. The net efficacy
adjusted for risk odds ratio (NEAR-OR) was calculated for
adults and children using the expected frequency of patients
responding without harms. The best benefit risk profile
corresponds to the greatest number of patients to respond
favorably without suffering adverse events. The heterogeneity
of the benefit risk ratio between adults and children was
measured using the ratio of NEAR-ORs. Results: Only 20
out of 89 included meta-analyses, evaluating 25 drugs, had
data allowing the quantitative synthesis. For two drugs the
NEAR-OR changed direction compared to the efficacy OR in
children (Figure 1). For example for lamotrigine for drug
resistant epilepsy, the OR showed a significant treatment
benefit in children but the NEAR-OR showed significantly more
harms. For AEs outcomes, two NEAR-RORs were significantly
different from 1 (Figure 2). One drug showed a higher
treatment benefit without AEs for children and the other
showed a higher benefit risk ratio in adults. Conclusions:
We found a difference in the benefit risk ratio between adults
and children for two drugs. The adjustment of the treatment
benefit on risks led to a loss of benefit for children for two
drugs. The NEAR-OR is a new method which seems useful
when assessing the benefit risk ratio of drugs. It needs to
be more widely used in order to be validated and to provide
robust conclusions.
197
198
P1.080
The waste of research in the addiction field:
an example from a review on psychological
intervention for psychostimulant-use
disorders
Minozzi S1 , Mitrova Z1 , Saulle R1 , De Crescenzo F2 , Amato L1
1
Cochrane Drugs and Alcohol Group, Italy; 2 Institute of Psychiatry
and Psychology Catholic University of the Sacred Heart, Rome, Italy
Background: There are a fair number of reasons that have
been explored for waste in research, such as when the needs
of stakeholders are ignored, or when what is already known is
overlooked. We would highlight waste of research, due to the
great variety of interventions compared without consideration
of the existing evidence. Objectives: To assess the number
of comparisons for which meta-analyses can be done, and to
quantify the waste of studies that have to be excluded based on
the variety of the comparisons in the field of psychostimulant
addiction. Methods: Data will be extracted from an ongoing
Cochrane Systematic Review assessing the effectiveness of
psychological intervention for psychostimulant-use disorders.
Experimental and control interventions will be listed as well
as different types of comparisons performed. The number
of studies with comparisons similar enough to be pooled in
meta-analysis will be ascertained, as well as the types of
comparisons that will be reported finally in the ‘Summary
199
of findings’ table. The number of studies excluded will
be quantified and the types of comparisons wasted will
be described. Results: Preliminary results based on 93
studies included showed that 15 comparisons can be done,
with a median number of three studies (range 1 to 38).
Three comparisons included only one study and another
three only two studies.
High variability in outcomes
considered represents an additional limit to synthesizing
results. Conclusions: Our findings could help to highlight a
major issue of relevance of primary research in the field of
addiction, giving a hint in the debate on reasons for waste
of research and the potential role of Cochrane Reviews in
informing the research agenda.
P1.081
Methods for individual participant data
meta-analysis of relative treatment effects: a
methodologic overview
Debray T1 , Moons K1 , van Valkenhoef G2 , Efthimiou O3 , Hummel
N4 , Groenwold R1 , Reitsma H1
1
Julius Center for Health Sciences and Primary Care, The
Netherlands; 2 University Medical Center Groningen, The
Netherlands; 3 University of Ioannina, Greece; 4 Universität Bern,
Switzerland
Background: Individual participant data (IPD) meta-analysis
is an increasingly used approach for synthesizing and
investigating treatment effect estimates. Over the past
few years, numerous methods for conducting an IPD
meta-analysis (IPD-MA) have been proposed, often making
different assumptions and modeling choices while addressing
a similar research question. As a consequence, meta-analytic
models sometimes lead to conflicting results, which casts
doubt on their utility and validity. Objectives: To assist
researchers to choose appropriate methods and provide
recommendations on their implementation when planning
and conducting an IPD-MA. Methods: We conducted a
literature review to provide an overview of methods for
performing an IPD-MA using evidence from clinical trials
or non-randomized studies when investigating treatment
efficacy. Articles were eligible if they presented statistical
methods, results from empirical case studies or provided
methodological guidance. Results: A total of 153 records
(out of 3360 unique records identified) were included.
Conclusions: IPD-MA offer numerous advantages over
meta-analyses that are solely based on published AD. There
is substantial, although disperse, information on how IPD-MA
should be conducted. This review may help systematic
reviewers with a limited background in medical statistics
to identify relevant methods for their IPD-MA.
P1.082
Acupuncture therapy for diabetic peripheral
neuropathy: a network meta-analysis and
methodology research
Xiong W1 , Chen W1
1
Evidence-Based Chinese Medicine, Beijing University of Chinese
Medicine, China
Background:
Well-designed multicenter randomized
controlled clinical trials (RCTs), systematic reviews and
meta-analysis could just assess two interventions treating
the same disease. However, when there are more than
two treatments for the same disease, direct comparisons
between each treatment should not be forgotten. Network
meta-analysis (NMA) aims to compare and estimate the pair
wise effect sizes of variety of treatments for the same disease
simultaneously when there is a lack of sufficient evidence from
direct comparisons. Objectives: To evaluate the application
of NMA in the therapeutic evaluation of acupuncture therapy
for the treatment of diabetic peripheral neuropathy (DPN), and
to apply NMA to acupuncture therapy and traditional Chinese
medicine. Methods: We searched the following databases to
September 2014: the Cochrane Central Register of Controlled
Trials (CENTRAL), PubMed, CNKI, VIP, Wanfang, and SinoMed.
After including RCTs of acupuncture therapy for DPN, we
performed NMA and drew network plots through GeMTC R
package and WinBUGS software based on the Bayesian
statistical model and Markov Chain Monte Carlo method
(MCMC). Results: Our research involved seven treatments
(manual acupuncture, electro-acupuncture, tapping, needle
warming moxibustion, mecobalamin, no treatment and
vitamin B1 and B12) in 40 trials, and a total of 2602 participants.
The network plot drawn by R software through calling the
200
GeMTC R package has been appended to the attachments.
The Rank probability showed that four types of included
acupuncture therapy all had beneficial effects on global
improvement, especially tapping, which displayed the best
efficacy, and needle warming moxibustion which followed it.
However, the poor quality of the methodology may lead to
a need to be more cautious when interpreting these positive
findings. Conclusions: The results may provide valid certified
clinical evidences of acupuncture therapy for the treatment
of diabetic peripheral neuropathy as well as the feasible
solution for the methodological issues in the NMA, which can
help to establish a reference model therapeutic evaluation of
traditional Chinese medicine.
P1.083
Network meta-analyses in clinical practice
guidelines: a cross-sectional survey
Yao L1 , Fu S1 , Cui R1 , Deng W1 , Zhang P1
1
Background: The development of clinical practice guidelines
(CPGs) should be based on systematic reviews. Network
meta-analyses (NMAs) are useful in CPGs when there is no
evidence that compares interventions of interest directly
(indirect treatment comparison; ITC), or when combining
direct and indirect evidence (multiple treatment comparison;
MTC). This study is to analyze the proportion of NMAs used
in CPGs. Objectives: This study is to analyze the proportion
of NMAs used in CPGs. Methods: PubMed database was
searched to obtain NMAs; 100 NMAs were randomly selected
as the sample by SPSS 19.0, then we collected the articles
that cited the NMAs sample in Google and Web of Science.
Results: A total of 289 NMAs were retrieved from PubMed,
and 100 NMAs were sampled randomly. A total of 2026
articles had cited the NMAs sample, among which 50 (2%)
were CPGs, including 12 National Guideline Clearinghouse
(NGC) CPGs and two National Institute for Health and Care
Excellence (NICE) CPGs. All of the CPGs that cited NMAs
were conducted or updated in 2011 to 2013. Conclusions:
The value of NMAs could make explicit the decisions made
implicitly when multiple pairwise comparisons are presented
in CPGs. However a low proportion of NMAs were used in
CPGs. Further research on how to improve use of NMAs in
CPGs would be valuable.
P1.086
Cumulative meta-analysis of the efficacy of
acupuncture on conventional
therapy-induced side effects in oncology
patients
Pan Y1 , Shen X2 , Shi X3 , Zhang H3
1 Institute of Medical Psychology, School of Basic Medical Sciences,
Lanzhou University, China; 2 Institute of Epidemiology and Health
Statistics, School of Public Health, Lanzhou University, China;
3 Department of Internal Medicine, Gansu Rehabilitaition Center
Hospital, China
Background: Current meta-analyses have investigated the
effectiveness of acupuncture for management of conventional
therapy-induced side effects of cancer, however, the results
were inconsistent. Little is known about the cumulative
meta-analysis of acupuncture practice; it is important that
the cumulative effect of the various proposals is kept under
consideration.
Objective:
To evaluate empirically the cumulative
performance of acupuncture in meta-analysis on therapyinduced side effects of symptoms in cancer patients. Methods:
Databases: MEDLINE, EMBASE, PsycLIT, MANTIS, Science
Citation Index, Cochrane Controlled Trials Registry, Cochrane
Complementary Medicine Field Trials Register, Cochrane Pain,
Palliative Care and Supportive Care Specialized Register,
Cochrane Cancer Specialized Register and conference
abstracts were consulted up to January 2015. We evaluated
the quality of the included studies by AMSTAR criteria
and analyzed the data using the Stata software, version
10.0. Data were provided by investigators; odds ratios and
95% confidence intervals were calculated on dichotomous
data. Results: Five meta-analyses have been published
from 2006 to 2013, including a total of 5235 cancer
participants and 5685 controls. The effect size and confidence
intervals in pooled analysis failed in favor of acupuncture
group compared with control therapies (acupuncture versus
control groups, odds ratio (OR) = 1.00, 95% confidence
interval (CI): 0.87 to 1.33). Furthermore, the confidence
intervals of treatment effectiveness of total population can
be narrowed by accumulation in chronological order., the
impact of sample size on the comprehensive results can
be observed by accumulation according the sample size.
Conclusions: Cumulative meta-analysis methods can provide
more information than the traditional meta-analysis methods.
The shortage of meta-analysis may have caused a lack of
statistical significance, which could be explained by the small
sample size of the current study, acupuncture prescription,
primary endpoint, the possibility of heterogeneity still
exists, and the findings could not result in a reasonable
recommendation.
201
Trial registration
P1.087
Practical and conceptual issues of trial
registration
Freitas C1 , Pesavento T1 , Pedrosa M1 , Riera R1 , Torloni MR1 ,
Nogueira G1 , Porfirio G1 , Silva A1 , Atallah A1 , Logullo P1
1
Brazilian Cochrane Centre, Brazil
Background: Trial registration is a prerequisite for publication
of studies in respected scientific journals. Recent Brazilian
regulations also require the registration of some clinical
trials in the national registry (ReBEC), but there is little
information available about practical questions involved
in the registration process. Objectives: To discuss the
importance of trial registration and practical issues involved
in this process. Methods: Information was obtained from
trial registry platforms, reference lists, and websites (last
search: September, 2014) on the following topics: definition
of a trial, history, purpose and importance of trial registries,
which information should be registered and the process of
registration. Results: Clinical trial registration aims to avoid
publication bias. In Brazil, it is required by journals indexed
in LILACS, SciELO; worldwide it is required by ICMJE-affiliated
journals. Recent Brazilian laws require that all trials (phases
I to IV) involving new drugs to be marketed in the country
must be registered in the Brazilian Clinical Trials Registry
(ReBEC). The pros and cons of using different trial registration
platforms are discussed in an article accepted by the Sao
Paulo Medical Journal. Conclusions: Trial registration is
important and currently there are mechanisms to enforce
its implementation. Researchers should take into account
national laws and publication requirements when choosing
the platform on which they will register their trial.
P1.088
An audit of the data structures of four
international clinical trial registries
Ko H1 , Hunter K1 , Vu T1 , Suasa R1 , Smith E1 , Zhang L1 , Askie L1
1 NHMRC Clinical Trials Centre, University of Sydney, Australia
Background: Clinical Trial Registries (CTRs) are increasingly
used by systematic reviewers to identify ongoing and
completed clinical trials. CTRs have trial information on
at least 20 unique data items (i.e. WHO Trial Registration
Data Set (TRDS)). Whilst being compliant with the TRDS,
the content between registries may be inconsistent as CTRs
operate independently of each other. This presents difficulties
when extracting data from multiple CTRs. Objectives: To
characterize the content of the data fields in the 20 TRDS
items across different CTRs, and to identify data fields where
extracting consistent information was complex. Methods:
Data fields from 4 CTRs (ANZCTR, ClinicalTrials.gov, ISRCTN,
EuCTR) were audited in Dec 2014 as part of a larger project
analysing country-specific clinical trial activity. The 20 TDRS
items were assessed for their data quality, and how CTRs
varied regarding the number of data items per data field,
aggregated data items, and data item formats. Results:
All 4 CTRs collected the 20 TRDS items, but the content
and number of data fields used to collect these items was
not consistent across CTRs. Some TRDS items required
single data fields only e.g. registration number. The more
information-intensive items often had multiple, associated
data items or were aggregated into single data items, e.g.
intervention description, sponsor, and funder items differ
across CTRs. Data items were presented in different formats
e.g. one data item might have a combination of categorical,
binary, or free text data. Some non-TDRS data items were in
some CTRs but not others. Some CTRs have coding options
for certain data items that are not directly translatable in
other CTRs, e.g. intervention condition codes. Conclusions:
It is a complicated process for users to extract consistent data
across multiple CTRs. Due to variations between CTRs (re:
content, where information is kept, number of fields, types
of data and how information is coded), accurate information
extraction requires data field matching, formatting, cleaning,
and recoding. Further standardization of CTR data is needed
to improve the ease with which CTR data can be more fully
utilised.
P1.089
Australian New Zealand Clinical Trials
Registry (ANZCTR) user survey results from
2009 and 2015
Hunter K1 , Vu T1 , Sausa R1 , Tan-Koay A1 , Ko H1 , Askie L1
1 NHMRC Clinical Trials Centre, University of Sydney, Australia
Background: Clinical trial registries are an important resource
for researchers, consumers and clinicians. In order to
maximise their utility, it is important to obtain user feedback.
ANZCTR has conducted two user surveys since its inception in
2005. Objectives: To obtain user feedback on the relevance,
functionality and usefulness of the ANZCTR. Methods: There
were two survey periods: Sep 2008 to Mar 2009 and Nov 2014
to Mar 2015. Users were prompted to complete a survey upon
visiting the website. Both surveys were identical, containing 8
items covering user role, frequency and ease of website use,
type of information being sought and relevance of the registry.
Users were required to select appropriate option(s) from a list,
and a free text item was also included. In the first survey the
question about role was mandatory. Results: In the 2008/9
survey 16,356 users responded to the mandatory question
about their role, and a smaller proportion (∼2500) completed
some/all of the remaining optional questions. In the 2014/15
survey 2662 users responded to at least one question. Both
surveys indicated that the most common role was a registrant
(28.9%), followed by researcher (26.9%), consumer (19.5%),
clinician (10.1%), journal editor/staff (2.6%), funder (1.7%)
and regulator (1.1%). Most respondents were first time users
of the site (42.4%), sought general information about clinical
trials (19.4%), and were able to find what they were looking
for most/all of the time (61.2%). Most also agreed/strongly
202
agreed that prospective trial registration is a good idea (84.2%)
and that ANZCTR is an important national resource (80.2%).
A comparison between the two survey periods showed a
5.0% increase in the proportion of researchers using the
site and 6.2% increase in the proportion of respondents
who strongly agreed that ANZCTR is an important national
resource. Conclusions: The most common users of ANZCTR
were registrants and researchers. Whilst most respondents
were able to find what they were looking for, there is scope for
improvement of the search function and navigation interface
of the website. The vast majority agreed that ANZCTR is an
important national resource.
P1.090
Is it necessary to search trials registers?
Mann M1 , Weightman A2 , Coles B3
1 Cardiff University, United Kingdom; 2 Cochrane IRMG, United
Kingdom; 3 Cancer Research Wales/Cardiff University, United
Kingdom
Background: In order to comply with the Cochrane Handbook
and Methodological Expectations of Cochrane Intervention
Reviews (MECIR), Cochrane authors are obliged to search trials
registers where relevant to their review topic area. Searching
trial registers retrieves ongoing or unpublished studies, which
will reduce the risk of publication bias. Objectives: Our
study investigated the extent to which Cochrane authors are
searching trials registers. Furthermore, to identify whether
any of the included unpublished studies were retrieved from
a trials register search. Methods: We examined 27 reviews
flagged as ‘New’ in The Cochrane Library (Issue 3, 2015) 19
March 2015. We determined reviews where trials registers
had been searched. We extracted data to identify whether
any included studies in the reviews were found uniquely in
a trials register. Results: Twenty reviews (74%) included
a search of trials registers. Some reviews searched other
registers in addition to the registers suggested in MECIR as
mandatory. In our poster presentation, we will fully discuss
the findings as to how far registers provide a unique source
of unpublished but relevant studies. All review results will
be anonymous. Conclusions: Even though endorsement
of MECIR recommendations has been in place since 2013,
some authors of new reviews were not compliant with these
standards. Therefore, Cochrane Review Groups should be
more explicit in their expectations of authors about standards
for MECIR.
P1.091
The World Health Organization’s
International Clinical Trials Registry
Platform: a ‘one-stop shop’ for identifying
clinical trials conducted worldwide?
Corp N1 , Jordan J1
1 Research Institute for Primary Care & Health Sciences, Keele
Background: The World Health Organization (WHO)
International Clinical Trials Registry Platform (ICTRP) is a
central repository for clinical trials regularly updated from 15
primary and three partner registries worldwide, the former
meeting requirements of the International Committee of
Medical Journal Editors. As such, it has the potential to
be a valuable one-stop access point to identify on-going,
recently completed and unpublished trials, negating the need
for multiple searches across multiple registries. However,
whilst the WHO ICTRP includes the Clinicaltrials.gov registry,
the current methodological standards for Cochrane Reviews
stipulate the mandatory search of both, questioning its
functionality and utility. Objectives: This study aims to assess
the functionality and utility of the WHO ICTRP as a ‘one-stop
shop’ for identifying clinical trials by comparing search
facilities and outputs with those from the original primary
trials registers. Methods: Using specific musculoskeletal
conditions as examples, WHO ICTRP and its 15 primary
registries, will be searched individually to identify relevant
clinical trials: simple (condition only) and more complex
(e.g. stipulating interventions, trial end or registration dates)
searches will be undertaken and search outputs compared
for agreement. Results and Conclusions: We will present
results to clarify whether WHO’s ICTRP offers a valuable
‘one-stop shop’ for retrieving clinical trial information, or
whether the additional time implicit in searching the original
trials registries is time well spent in terms of acquiring a
comprehensive overview of registered trials. We will also
present an assessment of the search capabilities, information
attainable and handling of search outputs of each of the
included registries and discuss the implications for Cochrane
Reviews and other systematic reviews.
P1.092
The German Clinical Trials Register (DRKS)
and its data quality compared to other trial
registries
Jena S1 , Kunzweiler K1
University Medical Center, Freiburg, Germany
1
Background: Publicly accessible clinical trials registries are
crucial for providing information about planned, ongoing,
completed or even discontinued clinical trials for patients
and their doctors, scientists, health professionals, funding
organizations and the general public. National registries
such as the German Clinical Trials Register (DRKS) are an
indispensable tool to inform the public in the national
language. Practitioners, patients and laymen clamour
urgently for data available in national languages. These
registries help to: - conduct reviews, meta-analyses, and
medical guidelines using a basis of comprehensive and
complete trial information; - avoid misinformation and to
reduce bias in health care and health research; - utilize
resources reasonably so that clinical trials don’t have to
be reiterated, exposing patients to an unnecessary risk.
To meet these needs requires comprehensive, complete,
203
and consistent data in clinical trial registries. Publications
regarding data quality in clinicaltrials.gov (1) and the World
Health Organization (WHO) Meta-Portal (2) showed the data
quality is insufficient. Objectives: The trial data in DRKS will
be examined analogously for completeness, accurateness
and consistency. A lot of automatic and manual queries in
DRKS should lead to a high data quality, that stands out
from other databases. Methods: Data on the following four
topics will be extracted, coded and analyzed: intervention
of the trial, endpoints, contact details, study design. For all
publically accessible fields the filling degree will be calculated.
Results: Preliminary evaluations showed consistent and
nearly complete information in all analyzed fields. Further
analyses will be presented. Conclusions: High data quality
in trial registries is essential for clinical research and health
care, DRKS endeavours to contribute to this scientific task.
References
1. Zarin DA, Tse T, Williams R et al. The ClinicalTrials.gov
results database-update and key issues. N Engl J Med 2011:
364: S852–S860
2. Viergever RF, Ghersi D, Scherer RW. The quality of
registration of clinical trials. PLOS ONE 2011: 6: Se14701.
P1.093
Current malaria clinical trials activity on the
Pan African Clinical Trials Registry and
comparison to a historical cohort
Lutje V1 , Pienaar E2 , Kredo T2 , Abrams A2
1
Cochrane Infectious Diseases Group, United Kingdom; 2 South
African Cochrane Centre, South African Medical Research Council,
South Africa
Background: Ninety per cent of the estimated deaths due
to malaria occur in sub-Saharan Africa, with 77% of these
in children under five years. Although estimated African
malaria mortality rates decreased by 54% between 2000 and
2013, much needs to be done to ensure optimization of
life-saving malaria interventions. The Pan African Clinical
Trials Registry (www.pactr.org; PACTR), launched in 2007,
provides online information about completed, ongoing and
planned clinical trials in Africa, which can be used to apply
the best available strategies to treat and prevent disease, and
plan future research. Objectives: To analyse interventions
reported in malaria trials conducted in Africa and registered
on PACTR between 2007 and 2014, and to compare them
against those included in a cohort of pre-2007 trials, to
highlight trends and possible gaps in interventions. Methods:
We conducted a cross-sectional analysis of trials currently
registered on www.pactr.org, and extracted those reporting
malaria interventions. Data extraction included number of
trials, location, intervention studied, and participant age
range. Descriptive analysis was run in MS Excel. Results:
Of 388 studies registered on www.pactr.org from 1 May 2007
to December 2014, 51 trials reported on planned or ongoing
research on malaria interventions. Of these, 24 reported on
treatment, 26 on prevention and one on other interventions.
For the prevention trials, seven included drug interventions,
15 reported on vaccines, two on mosquito control, and two
on other methods. Compared to the older cohort, there is
an increase in the number of vaccines trials (from 15.7% to
57.6%) and a decrease in the number of mosquito control
trials (from 26.3% to 7.6%). Other comparative analyses are
ongoing. Conclusions: A comparison of African malaria trials
pre-2007 and from 2007 to 2014 revealed changes in research
trends. Searches of registers such as PACTR are indispensable
tools to allow up-to date research monitoring and appropriate
planning for future initiatives.
P1.094
Does registration will help systematic
reviews to be published in journals indexed
by Web of Science? A current investigation
based on the PROSPERO platform
Zhang Y1 , Peng Y1 , Sun X1 , Du L1 , Huang J1
1 The Chinese Cochrane Center, West China Hospital of Sichuan
University, China
Background and Objective: The PROSPERO platform is
an international prospective register platform established
by the University of York in 2011. Whether registration
will help systematic reviews to be published in journals
indexed by Web of Science is not known. The current
study aims to investigate published systematic reviews that
have been registered on this platform, based on the Web
of Science database. Methods: The PROSPERO platform
was searched for systematic reviews for which the review
status was ‘published’. The last search was performed on
11 January 2015. Two reviewers independently screened the
included systematic reviews according to the inclusion and
exclusion criteria, and extracted data. The country, journal,
time between the registration date and publication date, and
funding were all analyzed. Results: A total of 239 systematic
reviews were included in the data analysis. The average time
between the registration date and published date was almost
14 months. These studies were from 35 countries, amongst
which the UK published the most studies (n = 70), followed by
Canada and Australia. All these studies were published in 175
journals, of which, PLoS One published 19 studies and ranked
first, followed by the Health Technology Assessment and BMJ.
With regard to the funding, 130 studies were supported by
funders, while 109 studies were not supported by funders.
Conclusion: The current evidence indicates that published
systematic reviews that have been registered on the
PROSPERO platform are relatively few. Most of these studies
are from the developed countries. It is still necessary to
disseminate the importance of PROSPERO.
204
Outcomes/Safety and adverse
events
P1.095
Endoscopic and open release similarly safe
for the treatment of carpal tunnel syndrome.
A systematic review and meta-analysis
Nikolakopoulou A1 , Vasiliadis H2 , Shrier I3 , Lunn M4 , Brassington
R4 , Scholten R5 , Salanti G1
1
Department of Hygiene and Epidemiology, University of Ioannina
School of Medicine, Greece; 2 Department of Orthopaedics,
University of Ioannina, School of Medicine, Greece; 3 Centre for
Clinical Epidemiology, Lady Davis Institute for Medical Research,
Jewish General Hospital, McGill University, Canada; 4 Centre for
Neuromuscular Disease, National Hospital for Neurology and
Neurosurgery, Queen Square, London, UK; 5 Dutch Cochrane Centre
and Julius Center for Health Sciences and Primary Care, University
Medical Center Utrecht, The Netherlands
Background: Analysis of safety data using randomized
controlled trials is often challenging because events are
typically rare. Various statistical methods have been proposed
to model rare events in a frequentist setting. In a Bayesian
framework, rare events can be modeled using the binomial
likelihood. Evaluation of the comparative safety of endoscopic
(ECTR) and open carpal tunnel release (OCTR) poses an
additional challenge; the outcomes are often correlated
because of bilateral involvement. Moreover, statements
about the conclusiveness of evidence are difficult to draw
when outcomes are rare. Objectives: To evaluate whether
there is evidence supporting a higher risk after ECTR in
comparison to the conventional open release using a variety
of methods to address rare events and developing a model for
dichotomous correlated outcomes. Methods: We included
all randomized or quasi-randomized controlled trials in a
systematic review and synthesised them in a meta-analysis.
Safety was assessed by the incidence of major, minor and total
number of complications, recurrences, and re-operations.
Different methods to handle rare events and a Bayesian
model were compared. A bivariate random-effects model was
developed to account for the correlated events. We explored
various methods to approximate correlation coefficients
based on the number of patients with bilateral involvement.
We applied three different methods to decide whether the
evidence was conclusive. Results: The assessment of major
complications, reoperations and recurrence of symptoms
does not favor either of the interventions and all models
are consistent in their results. ECTR performs better in
terms of minor complications compared to OCTR. Despite the
fact that complications were rare, it is unlikely that future
evidence will alter these conclusions. Conclusions: Various
statistical analyses for correlated and rare events did not
suggest a significant benefit of one surgical technique over
the other. The strength of evidence in favor of a clinically
relevant difference is weak. Further studies are not likely to
detect a difference between ECTR and OCTR in terms of safety
outcomes.
P1.096
Scope and consistency of outcomes reported
in randomised trials included in Cochrane
systematic reviews of haemodialysis
Tong A1 , Williams G1 , Manns B2 , Hemmelgarn B2 , Wheeler D3 ,
Tugwell P4 , Winkelmayer W5 , Urquhart-Secord R1 , Craig J1
1 TheUniversityofSydney, Australia; 2 UniversityofCalgary, Canada;
3
University College London, United Kingdom; 4 University of Ottawa,
Canada; 5 Baylor College of Medicine, USA
Background: Inconsistent outcome selection and reporting in
clinical trials may limit their capacity to inform evidence-based
decision making and the ability to combine findings in
systematic reviews. Objectives: To assess the scope and
consistency of outcome selection and reporting in trials
included in systematic reviews of interventions for patients
on chronic haemodialysis. Methods: The Cochrane Database
of Systematic Reviews was searched from 1999 to January
2015 for published systematic reviews of interventions for
patients on chronic haemodialysis. The description and
frequency of outcomes reported within the source randomised
controlled trials were assessed. Results: The 18 systematic
reviews included 282 trials that reported over 518 outcomes.
There was considerable heterogeneity among the outcomes
reported. Across all trials, 320 outcomes occurred only once,
and 198 outcomes were identified twice. The five most
common outcome domains were mortality (50 trials, 18%),
blood pressure (46 trials, 16%), calcium (44 trials, 16%),
parathyroid hormone (41 trials, 15%) and quality of life
(39 trials, 14 %). There was considerable variation in how
the domains were measured, the time at which they were
measured, and the units and threshold changes that defined
an outcome. For example, blood pressure was reported as
pre-dialysis, post-dialysis, maximum, difference in pre- and
post-dialysis, ambulatory, resting, rise, and time to restore.
Conclusions: A wide array of outcomes are reported in trials
of interventions in haemodialysis, with large heterogeneity
in timing of measurements, units of measure, and threshold
changes that define an outcome. This highlights the need for
a well-defined set of standardized core outcomes to improve
the comparability of trial findings in meta-analysis and to
provide greater confidence around treatment decisions for
patients undergoing haemodialysis.
205
P1.097
P1.098
Quality assessment of oral health-related
quality-of-life questionnaires in children and
adolescents
Inconsistent evaluation of heparin-induced
thrombocytopenia in Cochrane systematic
reviews: are harm-benefit balance truly
being appraised?
Zaror C1 , Pardo Y2 , Martinez-Zapata MJ3 , Espinoza G4 ,
Muñoz-Millán P5 , Ferrer M2
1
CIGES, Faculty of Medicine, Universidad de la Frontera, Chile;
2 Health Services Research Group, IMIM (Hospital del Mar Medical
Research Institute), Spain; 3 Iberoamerican Cochrane Centre,
Institute of Biomedical Research (IIB Sant Pau), Spain; 4 Department
of Public Health, Faculty of Medicine, Universidad de la Frontera,
Chile; 5 Department of Pediatric Dentistry and Orthodontic, Faculty
of Dentistry, Universidad de la Frontera, Chile
Background: Oral health–related quality of life (OHRQoL)
is a multidimensional construct that includes a subjective
evaluation of the individual’s oral health, functional
well-being, expectations and satisfaction with care, and
sense of self. The development and validating of specific
instruments for children of different ages has allowed
better capture of effects related to children’s oral health
problems, but only a little is known about the quality of
them. Objectives: To perform a quality evaluation of
the available evidence on the development process and
metric properties of OHRQoL questionnaires in children and
adolescents. Methods: To identify all available OHRQoL
questionnaires a systematic literature search was performed
in MEDLINE, EMBASE, Cochrane, SciELO, and LILACS. The
search was complemented through manual review of the
references of included articles. Both selection and data
extraction were conducted by two researchers independently.
Two experts independently assessed all the articles identified
for one instrument and applied the Evaluating Measures
of Patient Reported Outcomes (EMPRO) tool, which was
designed to assess the quality of OHRQoL questionnaires
in a standardized way. An overall EMPRO score and six
attribute-specific scores were calculated (range, 0 to 100) to
describe the quality of instrument performance. Results: The
search revealed 2465 articles, of which 321 were duplicates,
with 177 full-texts examined after selection based on titles and
abstracts. Eleven articles were identified by handsearching.
Finally 18 questionnaires and 156 associated studies of their
metric properties were identified for evaluation with the
EMPRO tool. Eleven were generic instruments to measure
QHRQoL, one for patients with hypodontia, one for patients
with Down syndrome, one for patients with fixed appliances,
two for discomfort and two evaluate impact of OHRQoL on
parents and families. Conclusions: Knowing the quality of the
instruments used to assess the OHRQoL allow us to apply the
most appropriate for clinical practice and research. We plan
to present the complete results and definitive conclusions at
the Cochrane Colloquium 2015.
Junqueira D1 , Faria JC1 , Fontoura M1 , Menezes de Pádua CA1
1 Universidade Federal de Minas Gerais, Brazil
Background: Bleeding is the main recognised adverse
effect of heparins. However, other adverse effects, such
as heparin-induced thrombocytopenia (HIT) can also be
expected. HIT is an immune reaction, frequent in some
clinical scenarios, and associated with paradoxical increase
in the risk of thromboembolic complications. Objectives: To
assess the consistency of the assessment of HIT in Cochrane
Systematic Reviews (SRs) evaluating heparin interventions
for prophylaxis or treatment of thromboembolism. Methods:
The Cochrane Database of Systematic Reviews was searched
from inception to March 2015 for SRs including any
heparin (unfractionated heparin (UFH), or low molecular
weight heparin (LMWH)) for prevention or treatment of
thromboembolism. Detailed information obtained from the
SRs included: (1) statement of thrombocytopenia or HIT as
outcome (primary or secondary); (2) statement of a precise
definition for HIT (HIT is defined as a relative reduction in
platelet count of about 50% (even if the platelet count at
its lowest remains > 150 x 109/L) occurring within five to 14
days after the start of heparin therapy and confirmed by
antibody tests); (3) reporting of HIT in the Results section
qualitatively or quantitatively. Results: Fifteen SRs published
from 2002 to 2015 were identified. The reviews investigated
a wide range of clinical scenarios and most were assessed
as up-to-date after 2010 (66.6%). Six reviews (40%) specified
thrombocytopenia as a secondary outcome and three (20%)
cited HIT as secondary outcome. Only one review provided
a clear, though incomplete, definition of HIT. In the Results
section, one SR reported no cases of HIT without defining the
reaction, one presenting a definition for HIT and reported that
participants from the included trials were not investigated
for HIT, and another was an empty review. Conclusions:
HIT is a harmful effect of heparins not evaluated in Cochrane
Reviews. The assessment of the harm-benefit balance of an
intervention cannot be limited to the main recognised adverse
effect. Improvements are needed in order to reduce bias in
the assessment of HIT, and probably other adverse effects, in
Cochrane Reviews.
P1.099
Estimating the occurrence of
non-standardised adverse outcomes: the
case of lipodystrophy
Junqueira D1 , Lana LG1 , Perini E1 , Menezes de Pádua CA1
1
Universidade Federal de Minas Gerais, Brazil
Background: Retrieving appropriate adverse drug reaction
data from epidemiological studies is a challenging task, even
206
when well-established outcomes (e.g. HIT- heparin-induced
thrombocytopenia) are taken into account. Lipodystrophy
(LD) is a disfiguring adverse reaction to antiretroviral therapy
(ART), which has been reported without consensus on the
case definition. Objectives: To estimate the frequency of
LD among HIV-infected patients on ART in order to establish
occurrence estimates according to diagnostic methods and
case definitions reported in the literature. Methods: A
systematic review of non-randomised studies (NRS) was
carried out through a sensitive search of MEDLINE, CINAHL,
LILACS, EMBASE and International Pharmaceutical Abstracts
databases - up to June 2014. NRS reporting LD as a primary or
secondary outcome and comparing HIV-infected patients on
different ART for at least six months were considered eligible.
The main measure of occurrence was the incidence and
prevalence of LD. Results: Twenty studies (12 cross-sectional
and eight prospective) were included in the systematic review.
Most studies (80%) provided detailed information on the
definition of LD, describing the anatomical sites, and four
(20%) reported the outcome in vague and unspecific terms.
All the studies investigated morphological alterations in the
abdomen, face and limbs, and other anatomical sites. The
diagnostic methods varied between studies (n = 4; 20%),
with the concomitant evaluation of the physician and the
patient being the most used (n = 9; 45%). Considering the
reported nomenclature, the prevalence of LD ranged between
11.7% and 67.8%. The incidence of LD ranged between
8.1% and 52.0%. Conclusions: The very heterogeneous case
definitions used in the NRS precluded the establishment of
point estimates of LD. Instead, we were able to provide rough
interval estimates. This might be a first effort to demonstrate
that: (i) improvement in the reporting of LD is needed, and
(ii) systematic reviewers must be aware of the challenges in
estimating adverse drug reactions that require better clinical
definition and depend on diagnostic methods to be identified.
P1.100
Reporting health related quality of life in
critical care RCTs
Roth D1 , Tscherny K1 , Heidinger B1 , Havel C1 , Arrich J1 , Gamper
G2 , Herkner H1
1
Medical University of Vienna, Department of Emergency Medicine,
Austria; 2 Universitätsklinikum St. Pölten, Austria
Background: Health-related quality of life (HRQoL), defined
as ‘the functional effect of an illness and its treatment upon
a patient as perceived by the patient’, might provide better
information about the actual impact of a therapy; detect
less obvious or unexpected effects; and is among the most
relevant outcomes. A Cochrane Methods Group focusing
on patient-reported outcome measures has been formed in
1999, and analysis of the reporting of quality of life has
been recommended for all Cochrane Reviews since 2003.
The frequency of reporting HRQoL in randomized controlled
trials (RCTs) on critical care in general has however not been
yet evaluated. Objectives: To assess how many Cochrane
Reviews on critical care included HRQoL as an outcome,
evaluate the frequency of reporting of HRQoL in RCTs in
critical care medicine, and to analyze the differences in study
characteristics between those reporting HRQoL and those
not. Methods: To access a cohort of studies, the Cochrane
Database of Systematic Reviews was searched for reviews
on critical care published after 2000. All reviews in the
RCTs contained therein were included in our analysis. Study
characteristics such as sample size, type of intervention,
disease spectrum, intensive care unit (ICU) setting, hospital
setting, funding, effect size, number and type of outcomes,
risk of bias, were extracted. HRQoL-specific data included
whether HRQoL was reported, and which instrument was
used. Results: The reference lists of 136 reviews, 53 (39%)
of which mentioned HRQoL as an outcome, were reviewed.
A total of 468 individual studies representing 88,585 patients
were included; 39 (8%) reported HRQoL as an outcome, one
(0.2%) study as a primary outcome. Most commonly used
scores were the Glasgow Outcome Scale (8; 21%), Modified
Rankin Score (6; 15%), Rankin Score (4; 10%) and Barthel
Index (3; 8%). No differences were found between studies
reporting HRQoL and those not. Conclusions: Although
measuring HRQoL is recommended for all Cochrane Reviews,
only 39% defined it as an outcome. Reporting HRQoL is even
less common in individual studies of critical care, with only
8% of studies including measures of HRQoL.
P1.101
Psychometric evidence for outcome
measures used in patients undergoing total
hip arthroplasty: a systematic review
Gagnier J1 , Huang W1 , Mullins M1
1
University of Michigan, USA
Introduction: The number of clinical trials testing variations in
total hip arthroplasty (THA) are increasing. Research suggests
that these studies use a heterogeneous array of outcome
measures and that many of these measures have questionable
reliability and validity. This creates difficulty when attempting
to perform systematic reviews of these trials and in using
the results of clinical trials to inform decision making.
Objectives: Our objectives were to: (1) identify currently
available patient reported outcome (PRO) measures used in
THA patients; (2) appraise the methodological quality of the
studies that evaluate the identified instruments; (3) assess the
psychometric evidence of these instruments; and (4) provide
a summary of the overall evidence for and against each
included questionnaire. Methods: We searched MEDLINE,
EMBASE, CINAHL, SportDiscus, the Cochrane Library, Scopus
(1931 to 2014) and reviewed reference lists and contacted
experts to identify relevant articles. We included articles on
the development or evaluation of a psychometric property
of a PRO measure in those undergoing THA and published
in English. The study risk of bias was appraised using
the ‘consensus-based standards for the selection of health
status measurement instruments’ checklist and psychometric
evidence assessed using criteria proposed by Terwee et al.
207
Finally, we combined these assessments to arrive at
a synthesis of overall evidence for each PRO measure.
Results: We included 49 unique studies testing properties of 18
different PRO measures. The Western Ontario and McMaster
Universities Osteoarthritis (WOMAC) Index, the Oxford Hip
Score (OHS), and the Harris Hip Score had the most studies
assessing them. The OsteoArthritis Knee and Hip Quality
Of Life (OAKHQOL), OHS, Patient Specific Index (PASI), and
WOMAC had the highest overall evidence. Most included
studies were poorly reported and very few psychometric
properties were tested in the literature. Conclusion: We
found that very few instruments had sufficient psychometric
properties. We recommend that reporting and methodologic
guidelines be developed for the assessment of psychometric
properties of PRO measures.
P1.102
Psychometric evidence for outcome
measures used in patients undergoing total
knee arthroplasty: a systematic review
Gagnier J1 , Huang W1 , Mullins M1
1
University of Michigan, USA
Introduction: Total knee arthroplasty (TKA) is a viable
treatment option for improving pain and function due to
severe osteoarthritis. Current recommendations suggest
using patient reported outcome (PRO) data to assess TKA
outcomes. Many PRO instruments have been developed
and compared, but this research often has questionable
quality and does not adhere to methodologic and reporting
guidance. Objectives: (1) to identify currently available
PRO questionnaires used in TKA patients; (2) to assess the
risk of bias of studies that evaluate psychometric properties
of identified instruments; (3) to assess the psychometric
evidence of these instruments; and (4) to provide a summary
of the overall evidence across each PRO
measure. Methods: We searched MEDLINE, EMBASE, CINAHL,
SportDiscus, the Cochrane Library, Scopus (1931 to 2014),
reviewed reference lists and contacted experts to identify
relevant articles. We included articles on the development
or evaluation of a psychometric property of a PRO measure
in those undergoing TKA and published in English. The
study risk of bias was appraised using the ‘consensus-based
standards for the selection of health status measurement
instruments’ checklist and psychometric evidence assessed
using criteria proposed by Terwee et al. Finally, we combined
these assessments to arrive at a synthesis of overall evidence
for each PRO measure. Results: We included a total of 59
studies on 15 different PRO instruments, with Western Ontario
and McMaster Universities Osteoarthritis (WOMAC) index and
the Oxford Knee Score (OKS) being the most frequent. Most
instruments had poor evidence, and only the OKS and the
Osteoarthritis Knee and Hip Quality of Life questionnaire
(OAKHQOL) had any excellent evidence. OAKHQOL had the
highest overall performance with three properties having
positive evidence. Conclusions: Despite many studies on
PRO instruments for TKA, few are of good quality. As a whole,
additional studies are needed throughout the literature to
produce quality evaluations of PRO instruments for TKA. We
recommend that reporting and methodologic guidelines be
developed for the assessment of psychometric properties of
PRO measures.
P1.103
The higher quality, the more convincing: a
cross-sectional study of randomized
controlled human trials on drug safety
Zhang L1 , Chen Q2 , Zhao P1 , Jia P1 , Zhang M1
1
Chinese Evidence-Based Medicine Centre, West China Hospital,
Sichuan University, China; 2 West China School of Public Health,
Sichuan University, China
Background: Can drug safety analyses in randomized
controlled trials answer drug safety issues accurately?
Regrettably, for much research the answer is no. Findings
from drug safety analyses may be misleading, potentially
resulting in clinical threat and health decision making. Few
studies have investigated the reporting and conduct of drug
safety analyses and a number of important questions remain
unanswered. Objectives: To describe the methodological
quality of drug safety research in randomized controlled trials
with a modified version of the Cochrane ‘Risk of bias’ (RoB)
tool. Methods: We conducted a cross-sectional study of 165
eligible randomized controlled human trials on drug safety,
which we randomly selected from MEDLINE, Embase and the
Cochrane Central Register of Controlled Trials (CENTRAL) in
2014. Teams of two reviewers independently screened full
texts of reports for eligibility, and abstracted data, using
standardized, pilot-tested extraction forms. We assessed
the risk of bias in these studies According to a modified
version of the Cochrane RoB tool (we considered random
sequence generation; allocation concealment; blinding of
participants, caregivers, and outcome assessors; adjudication
of adverse events; prognostic balance between treatment
groups; and incomplete outcome data). We conducted a
statistical analysis to describe current situation about the
methodological quality of drug safety research in randomized
controlled trials. Results: Within these studies, only
20% specifically mentioned the randomized method, but
77% mentioned randomization without specifying methods.
Similar results appeared for allocation concealment; patients
and caregivers were blinded in 48%, but outcome assessors in
only 24%. There were few studies that made judgements
of adverse events. Conclusions: Currently, qualities of
randomized controlled trials about drug safety research are
uneven. Research of high quality with sufficient reports can
get more approvals. Our findings will contribute to a set of
recommendations to optimize the conduct and reporting of
drug safety analysis.
208
P1.104
When you conduct a systematic review about
drug safety, you will run into these issues
Zhang L1 , Chen Q2 , Zhao P1 , Jia P1 , Zhang M1
1 Chinese Evidence-Based Medicine Centre, West China Hospital,
Sichuan University, China; 2 Sichuan University West China School
of Public Health, China
Background: Safety analyses in randomized trials examine
whether adverse events were caused by treatment. However,
findings from safety analyses may be misleading, potentially
resulting in suboptimal clinical and health decision-making.
Few studies have investigated the reporting and conduct of
safety analyses and a number of important questions remain
unanswered. Objectives: To assess study characteristics
associated with the results (such as the quality and credibility
of research results) of safety analyses. Methods: We
conducted a systematic review of 165 eligible, randomized,
controlled human trials on drug safety, which we randomly
selected in MEDLINE, Embase and the Cochrane Central
Register of Controlled Trials (CENTRAL) until the number
of them met our required sample size. Teams of two
reviewers independently screened full texts of reports for
eligibility, and abstract data, using standardized, pilot-tested
extraction forms. We conducted univariable and multivariable
logistic regression analyses to examine the association of
pre-specified study characteristics (did it make an accurate
definition of safety prior; did it describe the frequency
clearly (i.e. times or person-times); did it accurately define
denominator of adverse events) with the effect of drug safety
analyses.
Progress and Discussion: Currently, we are extracting the
data of these publications. Although reports of controlled
trials are recorded sufficiently with regard to curative effect,
they are not reported to the same level for safety. A clear
understanding of safety analyses, as currently conducted and
reported in published randomized controlled trials, will reveal
both strengths and weaknesses of this practice. Our findings
will contribute to a set of recommendations to optimize the
conduct and reporting of safety analyses.
P1.105
Can assessors in a trial of psychotherapy be
successfully blinded?
Watanabe N1
Japanese Cochrane Branch, Japan
1
Background: Blinding is a key element of treatment
evaluation in systematic reviews, and is considered more
difficult to obtain in trials assessing psychosocial treatment
than in those on pharmacotherapy. Objectives: To evaluate
the concordance rate and kappa values for agreement
between the right allocation and the guessed allocation
by blinded assessors in a randomized controlled trial (RCT)
on psychotherapy. Factors associated with concordance
between actual assignments and guessed assignments are
also explored. Methods: Data on both actual and guessed
assignments by six blind assessors were obtained at fourand eight-week assessments in an RCT assessing the added
value of brief behavioral therapy for insomnia over treatment
as usual for residual depression and refractory insomnia
in 37 adults at three psychiatric outpatient departments.
The severity of depression was assessed by blind raters.
Neither the patients nor physicians of TAU were blind to
allocation. However, all patients were requested not to reveal
their allocated treatment to the assessors for the severity of
depression. After each assessment, an assessor guessed which
group the patient had been assigned to, making it possible
to examine whether the blinding was successful. Information
about the degree of confidence in and reasons for guessing
such was also gathered from the assessors. Results: The
concordance rates and kappa values for agreement between
the actual allocation and the allocation guessed by blind
assessor at each assessment were 56.7% and 0.15 at four
weeks and 70.2% and 0.41 at eight weeks, respectively. This
indicated that the blinding of the assessors was satisfactory.
With regard to factors associated with the right allocation, only
‘intuition’ of the assessors was statistically significant (P value
= 0.02). Conclusions: Assessors in a trial on psychotherapy
can be successfully blinded, where a sufficient effort is made.
Reference
Watanabe N, Furukawa TA, Shimodera S, et al. Psychother
Psychosom 2013; 82:401–403.
P1.107
Difficulty in combining results of primary
studies in the addiction field: need to
enhance consistency of reported outcomes
Minozzi S1 , Mitrova Z1 , Saulle R1 , De Crescenzo F2 , Amato L1
1
Cochrane Drugs and Alcohol Group, Italy; 2 Institute of Psychiatry
and Psychology Catholic University of the Sacred Heart, Rome, Italy
Background: The COMET initiative outlined the need for
the development and application of standardized sets of
outcomes agreed by all stakeholders, including patients,
known as a ‘core outcome set’. In order to allow the
results of trials to be compared, contrasted and combined
as appropriate, the defined core outcome set should be
measured and reported in all trials of a specific health area.
This need is particularly relevant in the field of addiction,
where there is a considerable heterogeneity in outcomes
and measures used. Objectives: To analyze the variety of
outcomes and measures used in randomized controlled trials
assessing the effectiveness of pharmacological intervention
for psychostimulant misuse, taken as an example to identify
the most used outcomes and measures. Methods: Cochrane
Systematic Reviews published by the Cochrane Drugs and
Alcohol Group up to the end of June 2015 and assessing
209
the effectiveness of psychosocial interventions for drug and
alcohol misuse will be analyzed. The following data will
be reported: number of reviews and of included studies,
a complete list of clinical outcomes and of different ways
to measure each outcome, outcomes for which pooling of
studies was possible and outcomes reported in the ‘Summary
of findings’ table(s). Results: Our preliminary results based
on nine reviews with 121 included trials showed that the
highest heterogeneity was found for drug use outcomes:
use of dichotomous versus continuous outcomes; point
abstinence versus continuous abstinence; different length
of abstinence period considered; different time points
for outcome assessment; self- reported versus objectives
measures. Conclusions will be presented at the Cochrane
Colloquium 2015. Conclusions: Our findings could be used in
the process of defining an agreed core outcome set
Consumer issues and shared
decision-making
P1.110
Synthesis of health system barriers and
facilitators to scaling up vaccination
communication interventions in Cameroon,
Mozambique and Nigeria
Muloliwa AM1 , Cartier Y2 , Ames H3 , Oku A4 , Bosch-Capblanch X5 ,
Cliff J6 , Glenton C3 , Hill S7 , Kaufman J7 , Oyo-Ita A4 , Rada G8 ,
Lewin S9
1 Provincial Directorate of Health, Nampula, Mozambique,
Mozambique; 2 International Union for Health Promotion and
Education, France; 3 The Norwegian Knowledge Centre for the
Health Services, Norway; 4 University of Calabar, Nigeria; 5 Swiss
Tropical and Public Health Institute, Switzerland; 6 Universidade
Eduardo Mondlane, Mozambique; 7 La Trobe University, Australia;
8 Catholic University of Chile, Chile; 9 The Norwegian Knowledge
Centre for the Health Services in Norway and Health Systems
Research Unit, South African Medical Research Council, South
Africa
Background: Communication between healthcare providers
and individuals and communities is a key factor affecting
childhood vaccination uptake worldwide.
Effective
communication is critical to successful, sustainable
vaccination programmes. However, there is limited evidence
specific to low income settings (LISs) to guide programme
managers in implementing communication interventions.
The ‘Communicate to vaccinate’ (COMMVAC 2) project aims to
build research evidence on the effects of communication with
parents and communities to improve childhood vaccination
uptake in LISs. Objectives: To synthesise evidence on health
system barriers and facilitators to scaling up vaccination
communication interventions in three countries where the
COMMVAC 2 project is being implemented: Cameroon,
Mozambique and Nigeria. Methods: We will identify eligible
documents through searches of relevant electronic databases;
consultation with the COMMVAC 2 partners, experts and key
agencies, such as the World Health Organization; and through
internet searching. We will include documents that focus
on factors affecting the scale-up of childhood vaccination
communication in the study countries. Data from these
documents will be combined with data from key informant
interviews in each country. Framework synthesis will be
used to understand how health system factors affect the
implementation of vaccination communication interventions
at scale.
Results: Data collection in countries and
consultation with experts is underway. Database searching
has been completed and screening is underway on 146 full
text documents covering Nigeria (n = 98), Cameroon (n =
20), Mozambique (n = 15) and a range of countries (n
= 13). Preliminary results will be presented during the
Cochrane Colloquium. Conclusions: Understanding the
factors associated with the effective implementation at scale
of communication interventions for childhood vaccination
will inform the selection and adaptation of interventions to
improve vaccination coverage.
P1.111
How evidence-based are Print- and Online
Mass-Media in Austria? A quantitative
analysis
Kerschner B1 , Wipplinger J1 , Klerings I1 , Gartlehner G1
1
Cochrane Austria, Danube University Krems, Austria
Background: Print and Online-Media are important sources
of health information for lay people. Previous studies suggest
that mass media often exaggerate medical facts, however so
far, no quantitative data on the degree of exaggeration exist
to our knowledge. Objectives: To quantify the degree of
deviation of medical news stories from the actual evidence
supporting these stories. Methods: Between May 2011 and
June 2014, the media watch blog Medizin-Transparent.at
covered 219 health topics that had been the subject of mass
media reports in Austria. We searched the WISO newspaper
database and Google for all Austrian print and online news
stories about these topics published within a time window
of three months of the original news stories. The strength
of evidence for these topics was determined in rapid review
processes and rated on a modified GRADE scale and was
subsequently compared to subjective ‘implied evidence’ in
newspapers. The latter was rated on a self-developed scale
on the basis of the modified GRADE scale by two raters
independently. Results: Our search yielded 990 media
articles in Austrian print and online media covering the 219
prespecified health topics. In 59% of these articles, the
evidence for medical facts was reported in a strongly distorted
way (exaggerated or understated), only 11% adequately
reflected the real strength of evidence. While 61% of articles
implied the highest level of evidence for the reported effects,
the real evidence was at the highest level for only 3% of articles.
Compared to quality media, tabloid media report in a more
distorted way. However, this is mainly due to a different choice
of covered subjects, since the degree of distorted reporting
210
did not differ significantly between the tabloid and quality
media for subjects covered by both. Conclusions: More than
half of Austrian newspaper and online media reports strongly
exaggerated or understated medical facts when compared to
the actual evidence for these facts. Tabloid media reported in
a slightly more distorted way than quality media. Our results
confirm previous reports about serious incorrect reporting of
medical facts in mass media.
P1.112
Exploring the understanding of consumers
and clinicians about network meta-analysis
results and their usefulness to make
decisions: an Italian survey
Del Giovane C1 , Tramacere I2 , Filippini G2 , D’Amico R1
1 Italian Cochrane Centre, Italy; 2 Fondazione I.R.C.C.S. Istituto
Neurologico Carlo Besta, Italy
Background: Network meta-analysis (NMA) provides a
summary of the best evidence already available in the
literature and can be extremely useful for different types
of stakeholders such as consumers and clinicians. Therefore,
it is important that this method is fully understood and
properly recognized, especially in multiple sclerosis (MS).
Objectives: To investigate: (1) to what extent consumers
and clinicians are aware of NMAs; (2) how they understand
and interpret the outputs of NMAs; and (3) how the way
the results of NMAs are summarized and presented could
be enhanced. Methods: The assessment was carried out
through a survey. We focused on a specific clinical condition,
MS, and the sample of the two types of stakeholders were
patients with MS and neurologists. A web-based questionnaire
was developed for each group. The questionnaires included
questions that intend to: explore the general knowledge and
understanding of the risks and benefits of health interventions
from randomized controlled trials, systematic reviews and
meta-analysis, and NMA; investigate the understanding,
usefulness and how to enhance graphical representations
of results from a NMA on MS. Text and graphs examples were
shown in the questionnaires. Results: The results will be
available and presented at the Cochrane Colloquium as the
survey will be launched in April 2015. Conclusions: We expect
to evaluate and improve the way the results are presented
and understood by patients who need to be well informed and
receive the best available therapy, and clinicians who need to
decide which treatment to use for a specific disease.
P1.113
What German hospital pharmacists want to
know: cross-sectional survey on information
behaviour and information needs
Bollig C1 , Günther J2 , Suter K3 , Hoppe-Tichy T4 , Antes G1 , Lang B1
1
Cochrane Germany, Germany; 2 PharmaFacts GmbH, Freiburg,
Germany; 3 Hospital-Pharmacy, University Hospital Basel,
Switzerland; 4 Pharmacy Department, University Hospital of
Heidelberg, Germany
Background: The main role of hospital pharmacists in
Germany is to prepare and dispense drugs, but they are
also essential contact people for physicians, nurses and
patients for questions related to pharmacotherapy. To
provide high-quality information, hospital pharmacists need
access to unbiased and up-to-date drug information sources.
Objectives: Investigation of information habits of hospital
pharmacists and identification of barriers limiting appropriate
information retrieval in daily routine and continuous
education. Methods: Since October 2013 Cochrane Germany
at the Medical Center-University of Freiburg and the Chair
of Scientific Journalism at the Technical University (TU) in
Dortmund (Germany) have investigated questions relating to
medical knowledge translation. A subproject at Cochrane
Germany analyses the information-seeking behaviour and
the information provision for German hospital pharmacists.
Therefore, approximately 1700 members of the German
Society of Hospital Pharmacists will be invited by e-mail
to complete a voluntary and anonymous online survey. The
questionnaire consists of open-ended and closed questions
about the importance of different categories of information
and the frequency of drug information questions in their daily
routine, the most commonly consulted sources, the amount
of time available and necessary, their desires concerning
access to drug information sources, and demographics about
their education and practice setting. Results: Results of the
survey will be available and analyzed in spring 2015 and will
be presented at the Cochrane Colloquium 2015. Conclusions:
Our questionnaire will identify whether language barriers, lack
of time or limited access are obstacles to hospital pharmacists
using drug information sources. Moreover, we will investigate
if the information sources used fulfil the requirements of
hospital pharmacists.
P1.114
HTA guide
Pertl D1
1 Gesundheit Österreich GmbH, Austria
Background: Gesundheit Österreich GmbH (GÖG), on behalf
of the Austrian Federal Ministry of Health, has drawn up
an internet guide on health technology assessment (HTA).
HTA goes back to the late 1970s, when political decision
makers voiced demands for well-founded, comprehensive
information on health technologies, possible courses of
211
action and their consequences. It is generally recognised
that a central aspect of HTA is to provide a scientific
evaluation of health technologies for the purpose of political
consultancy. On the basis of the best available evidence,
health technology is objectively assessed in a transparent and
systematic process, taking into account the specific context
in which it will be applied, in order to provide a basis for
decision-making by political actors. What is common to
all definitions is that HTA goes beyond assessing the mere
effectiveness of health technology. Since the beginning of
HTA in the late 1970s, various types of efforts have been
made to force HTA for decision-making. Objectives: The
objective of the HTA guide is to provide quick and easy access
to national and international HTA information for the general
public and decision makers. In addition, the HTA guide
aims to pool links of relevant institutions in the field of HTA.
Methods: The HTA guide is revised and updated regularly, the
information is available in English and German. New contents
are searched for via the internet. The identification of new
contents is also facilitated by links of the membership of GÖG
to several expert networks. In addition, interested institutions
or persons can contact the webmaster with regard to the
content or with technical questions. Results: The HTA guide
gives an overview of definitions of HTA, handbooks, guides
and glossaries, lists institutions, outcomes and sources of HTA
in Austria, in selected other countries and across countries. It
also provides links to HTA newsletters, education and training
offers, and free and for-pay databases. The HTA guide is
available under http://hta-guide.biqg.at/.
P1.115
Building partnerships with the public by
learning about Cochrane evidence
Nunn J1 , Synnot A1 , McDonald S2 , Allen K2 , Hill S1
1
Cochrane Consumers and Communication Review Group,
Australia; 2 Cochrane Australia, Australia
Background: One of the goals of Cochrane’s Strategy 2020
is to make Cochrane the ‘home of evidence’ and to build
greater recognition of our work amongst the public, including
health practitioners, consumers, policymakers and guideline
developers. In order to develop stronger partnerships with
the public, Cochrane Australia and the Consumers and
Communication Review Group are working together to deliver
a series of learning and development events for Australian
consumers. Objectives: To increase public understanding
of health research and the role of evidence in informing
health practice and policy. Methods: Building on successful
work in the UK, we are developing learning resources and
activities to support the public to understand and value
evidence from systematic reviews. We will take a partnership
approach, working with local health service providers to
create free learning and development opportunities for the
public, patients, consumers and volunteers. Specific learning
objectives include participants being able to explain the value
of systematic reviews, how this knowledge can be applied
and how they can support the work of Cochrane. The events
will use a series of learning resources, facilitated discussions
and activities in order to achieve specific learning outcomes.
Results: The learning and development opportunities will be
delivered in mid-2015. We have completed an initial scoping
exercise and identified potential partner organisations, who
are helping us shape both the content and the delivery model.
We will be able to report on the success and impact of the
project against planned objectives and measurable outcomes.
Conclusions: This work has implications for the way in which
Cochrane can support the public to understand and value the
evidence from Cochrane Reviews. We hope to build on this
work and create a sustainable and scalable model, working in
partnership with relevant organisations.
P1.116
Discussing the risks and benefits of opioid use
for chronic non-cancer pain (CNCP)
Harris J1
American College of Occupational and Environmental Medicine;
The Permanente Medical Group, USA
1
Background: The use of opioids for chronic non-cancer
pain (CNCP) has increased rapidly in the last 15 years.
However, the effectiveness of opioids for CNCP remains
unsupported by good quality evidence, and adverse effects on
multiple organ systems are increasingly documented. Opioids
are often used at high doses and/or in combination with
other psychoactive medications. CNCP has been regarded
as purely physical, although its etiology is unknown in
most cases, and it is often associated with psychiatric
conditions and emotional distress. Objectives: To create
a knowledge base and framework for informed discussion
of the risks and benefits of opioid and other medication
use for CNCP. Methods: A community-wide coalition of
public health, medical, education, law enforcement and
treatment professionals in Marin County, California (Rx Safe
Marin) developed evidence-based provider recommendations
for this discussion, and associated community education
resources. A parallel effort is underway at medical groups
in the County. The American College of Occupational and
Environmental Medicine has included recommendations for
this discussion in its recently updated opioid guideline as well.
Results: The evidence for the effectiveness of opioids for CNCP
and adverse effects were reviewed. A framework for periodic
informed consent and shared decision making discussions
was developed, that was similar in all three organizations.
Provider material includes a series of case studies and video
vignettes for communication about difficult issues and patient
concerns. Rx Safe Marin and associated provider groups are
monitoring adverse event rates and prescription rates, and
working on documentation of shared decision making and
informed consent. Conclusions: The wide spread effects
of opioids on multiple organ systems, beyond addiction,
dependence, overdose and death, balanced against the still
unproven effectiveness of opioids for CNCP should be carefully
212
considered in a shared decision-making/informed consent
framework when making treatment decisions.
P1.117
How willing are the public to pay for
anti-hypertensive drugs for primary
prevention of cardiovascular disease: a
survey in a Chinese city
Tang J1 , Wang W2 , An J3 , Hu Y4 , Cheng S3 , Griffiths S2
1 TheHongKongBranchoftheChineseCochraneCentre, China; 2 The
Chinese University of Hong Kong, China; 3 Baotou Medical College,
China; 4 Peking University Center for Evidence-Based Medicine and
Clinical Research, China
Background: Current recommendations on drug treatment of
hypertension for primary prevention of cardiovascular disease
are primarily determined by the evidence of effectiveness,
disregard the resources available and values of people, and
recommend a universally fixed-risk cutoff for initiating drug
treatment. The guidelines may have over-estimated the
willingness of the public to accept and pay for these drugs
and a fixed cutoff may not fit all populations. Moreover, the
public may have been misinformed and are unable to make
the right decision even if they are consulted.
Objective: To describe the gap between current policy and
what the public truly want. Methods: A cross-sectional survey
with face-to-face interviews of rural and urban residents in
northern China. Results: A total of 887 rural residents and
921 urban residents were interviewed with a response rate
of 97%. Of these, 95% (95% confidence interval (CI) 94% to
96%) said they would take anti-hypertensive drugs if they had
hypertension, although 91% (95% CI 89% to 92%) said they
did not have sufficient knowledge to make a decision; 78%
(95% CI 76% to 80%) believed that anti-hypertensive drugs
were primarily to lower blood pressure or relieve symptoms.
They over-estimated the cardiovascular risk of untreated
hypertension by approximately 12 times and the absolute
benefit of drug treatment by 20 times. Given the actual
absolute benefit of the drugs, only 23% (95% CI 21% to 25%)
were willing to pay the current annual cost of RMB 500 (USD
73.30) for these drugs. Given the current cost, they were, on
average, willing to pay for the drugs only when the five-year
cardiovascular disease risk was as high as 35% (95% CI 31%
to 38%) or even higher. Conclusion: The public in China are
significantly misinformed and considerably over estimate the
risk of hypertension and the benefit of treatment. The public’s
willingness to pay for anti-hypertensive drugs is much lower
than the current guidelines implicitly assume. The willingness
to pay should be considered, along with other factors, when
prescribing anti-hypertensive drugs to an individual patient
or making hypertension guidelines for a population.
P1.118
Cochrane initiatives in Nepal: reflections on
progress in improving evidence-informed
decision-making
Aryal K1 , Dhimal M1 , Stewart M2 , Dhakal P1 , Mehata S3 , Rana S4 ,
Tharyan P5 , Karki K1
1
Nepal Health Research Council, Nepal; 2 Ministry of Health and
Population, Nepal; 3 Nepal Health Sector Support Programme,
Nepal; 4 London Borough of Merton, United Kingdom; 5 Cochrane
South Asia, India
Background and Objectives: In 2014, the Nepal Health
Research Council (NHRC) and Cochrane South Asia began
collaborating formally in order to foster an evidence-enabling
environment within the Nepali health sector. This poster
reflects on progress and outlines future directions. Methods:
Critical reflection that incorporates the perspectives of
representatives from government, research institutions, and
non-governmental organizations (NGOs) in Nepal. Results:
The crucial impetus for collaboration was a workshop
attended by key NGO and governmental stakeholders,
including the Secretary for Health, at which the NHRC
was unanimously designated as the nodal agency for
Cochrane-related activities. Subsequent activities included
workshops for policy-makers and planners, training on
Cochrane methods for researchers, and identification of
research priority areas. The NHRC was subsequently invited
to provide technical support in developing the Third Nepal
Health Sector Plan (NHSP III), which outlines priorities for
2015 to 2020. As a result, the recently-developed NHSP III is
vastly more evidence-informed than the previous two plans.
Conclusions: We attribute our successes to date to: engaging
policy makers early; finding an effective nodal
agency that was acceptable to stakeholders; developing
and leveraging formal and informal networks; facilitating
the participation of key stakeholders at the Hyderabad
Colloquium; and establishing evidence-support teams that
seek out, synthesize, and present contextualized evidence
in digestible formats such as policy briefs, thereby gaining
buy-in from high-level stakeholders. Through 2015 we plan
to continue engaging government officials and to provide
training for in-country research teams. We also hope
to establish a branch of Cochrane South Asia in Nepal
by 2016.
P1.120
Clinical decision support system: an effective
pathway to promote shared decision making
between clinician and patient.
Jia P1 , Zhang L1 , Zhao P1 , Zhang M1
1
Chinese Cochrane Center, China
Background: A clinical decision support system (CDSS) has
been defined as a computerized system that uses case-based
213
reasoning to assist clinicians in assessing disease status, in
making a diagnosis, in selecting appropriate therapy or in
making other clinical decisions. Related studies showed
that CDSS can promote shared decision making and improve
clinical decisions. However, little is known about the ways
and factors that are most effective in promoting shared
decision making between clinicians and patients. Objectives:
To provide a rigorous, evidence-based assessment of the
ways and factors that are most effective in influencing
shared decision making between clinicians and patients.
Methods: We searched the MEDLINE, EMBASE, Cochrane
Library and four main Chinese databases including CBM, VIP,
CNKI and Wanfang using the search term ‘decision support
systems, clinical’, ‘decision-making, computer assisted’,
‘CDSS’, ‘shared decision making’, ‘shared determinant’ etc
from 1977 to March 2015. Microsoft Excel 2007 was used
to perform data extraction and analysis. Results: We
included a total of 340 studies in our study that ranged
from 1997 to 2015. The number of studies reached its peak
for 2010, with 56 studies accounting for 17.47% of the years’
publication. The CDSS can promote shared decision making
in five major ways: timely alerting (37.5%), transparency
interpreting (17.4%),scientific critiquing (8.6%), reasonable
assisting (8.0%), diagnosing and managing (6%).Several
practical factors contribute to the success of CDSS in
promoting shared decision making: (1) automatic provision
of decision support; (2) an intuitive user interface; (3) timely
decision support; (4) providing actionable recommendations
that are succinct and relevant to patient care; and (5)
knowledge skill of patient and clinician. Conclusions: There
are a great number of ways and factors that influence
the use of CDSS for promoting shared decision making
between clinicians and patients. Further research needs
to be conducted before there is widespread dissemination
into clinical practice.
P1.121
Patient participation in health care
in China: what ways promote patient
participation and engagement?
Jia P1 , Zhang L1 , Zhao P1 , Zhang M1
1
Background: Patient participant in health care has been a
concern since the complex social movements of the 1960s.
The doctor-patient dispute has been increasingly prominent
in recent years in China. Related studies showed that the
communication openness was seriously jeopardised by the
lack of trust between health workers and patients. Objectives:
To investigate the ways of promoting the participation
of the patient in health care in China. Methods: We
searched four Chinese databases including CBM, VIP, CNKI
and Wanfang using the search term ‘patient participant’,
‘patient involvement’ ‘health care’ and ‘clinical care’ from
2000 to February 2015. Results: Eighty relevant studies were
published from 2000 to 2015 but only 22 meet our criteria. The
number of studies relevant to patient participant in health care
was increasing from the year 2010 to 2014, with two, three, five
and 10 respectively. Five categories of ways were identified
that promote patient participant in health care: education by
clinicians (8), self-monitoring (5), communicating information
(3), family’s participation (2) and education by internet (2). The
participants were diabetics, children, women with cesareans
and patients of plastic surgery. Among the 20 studies, the
institutions of the first authors mostly were universities (65%),
hospitals (11%), and colleges (9%), respectively. Conclusions:
Patient participant in health care is still in its infancy in China.
Although participation in health care has been emphasized
for many years, there is not much evidence to support this.
Hopefully ‘participation in health care’ will be a focus in future
in China.
P1.122
Patient safety education: a cross-sectional
survey of knowledge, skill and attitude of
medical students on patient safety in China
Jia P1 , Zhao P1 , Zhang L1 , Zhang M1
1
Background: Reducing medical errors and improving patient
safety have become the priority of health care around
the world. China has been carrying out patient safety
education in medical students for the past 10 years. Medical
students should be able to recognize unsafe conditions, and
systematically report errors and near misses. Objectives:
To assess the knowledge, skill and attitude of medical
students on patient safety. Methods: This is a descriptive
and cross-section study. A questionnaire of 31 items was
developed based on the World Health Organization (WHO)
study for patient safety curriculum guide. The survey was
conducted in three medical universities of China in 2014. A
total of 183 third- and 221 fourth-year medical students were
included. Microsoft Excel 2007 and SPSS 17.0 were used to
perform data extraction and analysis. Results: The survey
yielded 500 questionnaires with an 81% response rate. The
average blank item in a total of 31 items was 0.96% of which the
item that was most often blank was ‘What will happen when a
medical error occurs?’. The study population consisted of 143
male and 262 female students. Their attitude to good patient
safety was positive, but the students had little knowledge of
patient safety especially regarding medical errors and how to
report them,. There were no statistical differences across the
three universities or grades. The only statistical difference was
for the item ‘I would like to discuss with others when I made
medical error’ between genders. Conclusions: The results
suggest that the knowledge and skill of patient safety from
medical students is very poor in China. The patient safety
culture should be established properly.
214
P1.123
P1.126
Uptake of systematic reviews and
meta-analyses in health policy briefs: an
analysis of references
Development of clinical research support
system for conditional coverage with
evidence development
Wang Q1 , Chen TY2 , Ding HF2 , Zhang H3 , Sun HH2 , Zhang JY4 , Li
N2 , Yao L1 , Wang XQ1 , Chen YL1 , Yang KH1
1
Evidence-based Medicine Center of Lanzhou University;Key
LaboratoryofEvidence-basedMedicineandKnowledgeTranslation
of Gansu Province, China; 2 The Second Clinical Medical College of
Lanzhou University, China; 3 School of Basic Medical Sciences of
Lanzhou University, China; 4 School of Public Health of Lanzhou
University, China
Son SK1 , Park DA1 , Lee MJ1 , Lee NR1 , Lee J1 , Yoon JE1 , Lim SW1
1
National Evidence-Based Healthcare Collaborating Agency, South
Korea
Background: The achievement of universal and equitable
access to healthcare, one of the health-related Millennium
Development Goals (MDGs), is most likely to be realized
through evidence-based health policies and actions.
Evidence-based health policy briefs are widely used by WHO
and health sectors of many countries, as the most important
way of presenting evidence to health policy-makers. They
provide potential options based on the best global evidence
(such as systematic reviews) and local evidence on priority
issues. Currently, there is no analysis of uptake of systematic
reviews and meta-analyses in the health policy briefs.
Objectives: To explore the uptake of systematic reviews and
meta-analyses in the health policy briefs based on the analysis
of references. Methods: Two researchers independently
handsearched the Health System Evidence (HSE) and World
Health Organization (WHO) databases before December 2014
and included health policy briefs in English and Chinese.
Two reviewers independently completed data extraction and
resolved disagreement by discussion. Results: We included
106 health policy briefs in WHO and HSE from 2001 to 2014.
All the policy briefs were published in English; 96% of policy
briefs (102) reported the references and the total number was
5683 (range: 6 to 247, mean: 56). The main three sources were:
websites (2315, 41%), journals (2165, 38%), and books (60,
1%). The main four types of references were: primary studies
(3379, 59%), systematic reviews (317, 6%), clinical practice
guidelines (29, 0.5%) and overviews of systematic reviews
(14, 0.2%). Among these cited systematic reviews, 172 were
Cochrane Reviews and 145 were non-Cochrane reviews. We
did not find any network meta-analysis. Conclusions: Most
health policy briefs reported references, but the proportion
of uptake of systematic reviews was low. We think health
brief developers should propose the evidence-based policy
options on the basis of up-to-date and high-quality systematic
reviews and meta-analysis. As a next step, we will explore the
details of cited systematic reviews on the topic and quality
and how many policy options were supported the systematic
reviews.
Background: Since 2014, the Korean government has
introduced Selective Reimbursement System to intensify
national health coverage for severe diseases including cancer,
cardiac, cerebrovascular, and rare diseases. Unfortunately, in
most cases, available scientific evidence that can be used as
an evidence-base for making policy decision is often limited.
Objectives: The purpose of this study was to suggest a
model for a clinical research support system for conditional
coverage with evidence development. Methods: A search
for relevant web pages, a quick literature review, and an
in-depth interview with experts were performed for this study.
Results: Criteria for prioritization of target technologies,
administration of research process, and funding source
should be developed to establish evidence development for
conditional coverage through clinical research. With respect
to the prioritization, the technologies that need additional
evidence due to uncertainty of safety and effectiveness and
have a high potential of being able to obtain evidence within
a short period should be proritized. In the administration of
research process, an independent and professional agency
including a ‘steering committee’ or ‘sub-committee’ that can
provide scientific advice and methodological guidance should
manage the clinical research processes. Funding issues are
key factors to be considered carefully in order to settle the
clinical research support system for conditional coverage
with evidence development in smoothly. Principally, it is
reasonable that the government should fund the study, as it is
in accordance with government policy support. Additionally,
various funding sources (e.g. research foundation etc) are
necessary in terms of system sustainability. Conclusions:
To guarantee a high quality of evidence obtained through
the clinical research, the importance of process management
must be acknowledged. The actual policy should be designed
through the consultation of stakeholders and agencies related
to the system, roles that should be performed by authorities
and the targeted technology.
P1.127
Is evidence-based medicine incorporated into
the analysis of European health systems?
Decsi T1 , Endrei D2 , Mihályi K3 , Lohner S1 , Boncz I4
1
Hungarian Branch of the German Cochrane Centre, University of
Pécs, Pécs, Hungary; 2 Clinical Centre, University of Pécs, Pécs,
Hungary; 3 Department of Paediatrics, University of Pécs, Pécs,
Hungary; 4 Faculty of Health Sciences, University of Pécs, Pécs,
Hungary
215
Background: There is an increasing requirement for policy
makers to ensure that their decisions are evidence-based.
In turn, researchers should conduct research addressing key
aspects of health policy questions, in order to render possible
an effective information transfer between researchers and
decision makers as well as an evidence-based approach to
policy making. Objectives: The aim of the present study was
to investigate the extent to which the idea of evidence-based
medicine (EBM) is incorporated into the documents describing
the functioning of European healthcare systems. Methods: An
electronic search was conducted in the documents available
in the European Observatory on Health Systems and Policies
database, using the text word ‘evidence’ and the total
number of the text word ‘evidence’ referring to EBM was
counted. These representations were then classified into
one of the following categories: (1) resources of health care;
(2) health technology assessment; (3) organisation of health
care; and (4) environment of health care within the society.
Results: Altogether thel documents mentioned the idea of
EBM 413 times; however, there were large differences in
both the absolute number of mentions and the incidence of
mentioning per page of the document. Two-hundred and
eight representations of the idea were assigned to one of the
four categories listed above. EBM was mentioned 57 times
in connection with resources of health care, 31 times with
health technology assessment, 56 times with organisation of
health care and 64 times in relation to the social environment
of health care. Conclusions: The results of the present
study indicate that the methodology of EBM has already been
widely used within the documents describing health systems
and policies in Europe. However, considerable differences
in the extent and depth of applying EBM methodology are
detectable in descriptive European health policy documents.
to the GRADE principles, taking into account the balance
between desirable and undesirable effects, quality of
evidence, values and preferences, and costs. Objectives:
To assess the possibility of adding recommendations to the
Cochrane-derived EVDs and to compare the formulation of
the recommendations with the guideline contents. Methods:
Treatment of menorrhagia, a common condition in primary
care, was chosen as the indicator guideline. We assessed
the treatment chapter of the guideline to find whether it was
possible to add a recommendation to every Cochrane-based
EVD linked to the chapter. The effect of the recommendation
on the usability of the guideline was evaluated. Results:
Altogether 13 Cochrane-based EVDs were linked to the
guideline. The Cochrane Reviews were published in 2000 to
2014 and all the EVDs referred to the latest published version.
One EVD already included a recommendation. According to
the evaluators, a further eight EVDs could be complemented
with a recommendation. In the remaining four EVDs the
addition of a recommendation was judged not to be feasible.
The results will be presented in table format explaining the
basis of these judgments. Conclusions: Recommendations
derived from concise evidence summaries and linked to the
most essential or critical parts of a point-of-care guideline can
provide valuable basis for clinical decision-making.
P1.129
Access of evidence-based online databases
among main and allied health professionals:
a nationwide survey in Taiwan
Chiu Y1 , Kuo K1 , Weng Y2 , Chen C1 , Chen K1 , Lo H1
1
Taipei Medical University, Taiwan; 2 Chang Gung Memorial
Hospital, Taiwan
P1.128
From evidence to recommendation:
Cochrane-based evidence summaries as
sources of recommendations in a
point-of-care guideline
Alenius H1 , Jousimaa J1 , Teikari M1 , Kunnamo I1
1
Duodecim Medical Publications Ltd, Finland
Background: Evidence-Based Medicine Guidelines (EBMG) is
a collection of concise point-of-care guidelines, originating in
Finland and published in a number of different language
versions. The guidelines are backed up by over 4000
Evidence Summaries (EVDs), of which over 50% are based
on Cochrane Reviews. The level of evidence (LoE) in the
EVDs is graded according to the principles of the GRADE
(Grading of Recommendations Assessment, Development
and Evaluation) group to be either high, moderate, low,
or very low (indicated by the letters A, B, C and D,
respectively). An increasing number of EVDs also include
a recommendation formulated by the EBMG editors. The
Strength of Recommendation (SRec) is graded to be either
strong or weak for or against an intervention according
Background: There are a variety of evidence-based online
databases, but few studies have examined whether main
and allied health professionals prefer different resources.
Objectives: The current study was to investigate the
preference in accessing the online databases between main
and allied health professionals of regional hospitals in
Taiwan. Methods: A constructed questionnaire survey was
conducted from January through April 2011 in nationwide
regional hospitals of Taiwan. Questionnaires were mailed
to main professionals (physicians and nurses) and allied
professionals (pharmacists, physical therapists, technicians,
and others), with 6160 valid returns collected. Results:
The most commonly-used databases for physicians were
MEDLINE/PubMed, the Cochrane Library, UpToDate, and
MD Consult. In addition, the commonly-used databases
for nurses and technicians were databases in Chinese,
including the Index to Chinese Periodical Literature, the
National Digital Library of Theses and Dissertations in
Taiwan, and the Chinese Electronic Periodical Service.
For pharmacists, the most commonly-used database was
Micromedex. Overall, the most common motivation for
accessing the online databases was self-learning, followed
by clinical practice, positional promotion, class assignment,
216
instruction preparation, research, medical accreditation and
insurance issues. Conclusions: This national survey depicts
the pattern of access to the online databases among various
health professionals. There were significant differences
between and within main and allied health professionals
in their use of online databases. The data provide clinical
implications for strategies to promote the accessing of
evidence-based information.
Knowledge translation and
communicating evidence
P1.130
Filtering the information overload from
systematic reviews with ‘Summary of
findings’ tables
Conway A1
1
HRB-TMRN and NUI Galway, Ireland
Background: This presentation will evaluate the ‘Summary
of findings’ (SoF) table as a knowledge translation tool
to communicate key evidence from systematic reviews to
clinicians. Systematic reviews are currently viewed as the
highest levels of evidence to support decision-making in
healthcare. However, barriers such as time limitations,
complexity and large volumes of research can impact
negatively on the ability and willingness of potential review
users to engage with full versions of systematic reviews.
SoF tables may filter the information overload for clinicians
enabling them to make better decisions. Objectives: To
evaluate the effectiveness of SoF tables on user understanding
of findings from systematic reviews. Methods: This systematic
review will include published and unpublished, randomised
trials and non-randomised trials, where the effects of exposure
to SoF tables of systematic reviews on one or more outcomes
is measured. The included trials will compare SoF tables to an
alternative method of dissemination of systematic reviews.
Results: To be confirmed. Conclusions: If the results indicate
that SoF tables support increased understanding, then the
review will benefit potential users of systematic reviews such
as clinicians, guideline developers, policy makers and other
stakeholders such as charitable organisations and individuals
or groups who inform the patient population and/or the
public. It may provide recommendations in a form in which
they can quickly access the key findings of future reviews.
It may also support these users in making decisions about
whether creating SoFs would be a good way of disseminating
review findings (and potentially other research findings)
within their own organisations.
P1.131
Moving from research to practice: training
students in translating and communicating
evidence
Junqueira D1
1
Universidade Federal de Minas Gerais, Cochrane Adverse Effects
Methods Group, Brazil
Background: Despite global calls for promoting the use of
research for informing decisions about health, the use of
evidence in policymaking and practice remains a challenge.
Objectives: This research aimed (1) to design knowledge
translation products to guide informed and shared clinical
decisions, and (2) to develop core competences, skills and
behaviors in evidence-based concepts for informed decisions
about health in a group of pharmacy undergraduate students.
Methods: Pharmacy students from Brazil developing their
undergraduate thesis were supervised to work on the subjects
‘health education’ and ‘evidence-based practice’ by designing
a knowledge translation product to inform the use of evidence,
considering local resources and costs. Students had no
basic training in research and followed a structured tutorial
of activities and learning materials focused on tools for
evidence-informed health care. Results: Four undergraduate
theses were developed in different topics: (1) metformin
for polycystic ovary syndrome, (2) treatment of sinusitis in
children, (3) antibiotics for acute otitis media in children,
and (4) beta-blockers for hypertension during pregnancy. All
of them were based on a Cochrane Systematic Review and
included a leaflet or infographic diagram as a final product.
During the oral presentation of the theses, students were able
to demonstrate critical understanding of the concepts for
informing decisions about health in order to influence future
professional environments. Conclusions: Undergraduate
training of differentiated professional who value the impact
that information and health education have for the success
of health interventions. Knowledge translation to consumers,
healthcare professionals, or policy makers may significantly
improve the curriculum of courses in the health area and play
an important role in designing expert contents to inform the
translation of research into practice and policy. Similar
experiences could support Cochrane activities developed
to support the use of evidence-based research by health
professionals and consumers.
P1.133
Treatment guidelines in specialist mental
healthcare: how important are they and how
to translate them into practice?
Fiedler I1 , Girlanda F1 , Becker T1 , Barbui C2 , Kösters M1
1
Ulm University, Ulm, Germany; 2 Department of Public Health and
Community Medicine, Section of Psychiatry, University of Verona,
Italy
217
Background: Clinical practice guidelines have become a
standard tool in the attempt to increase the quality of clinical
care. They provide a guide to best practice and a framework
within which clinical decisions can be made. Nevertheless,
knowledge on how to best implement guidelines into mental
healthcare is sparse.
Objectives: To investigate the
efficacy of guideline implementation on provider performance
and patient outcomes in mental healthcare settings, and
to explore how guidelines should be translated into
practice. Methods: We performed a systematic review
of randomised controlled trials (RCTs), controlled clinical
trials and before-and-after studies comparing guideline
implementation strategies versus usual care, and different
guideline implementation strategies in patients with severe
mental illness recruited in mental healthcare settings. We
conducted a meta-analysis of RCTs only. Results: Eighteen
studies met our inclusion criteria. Overall, a minority
of studies showed a positive statistically significant effect
of guideline implementation on provider performance or
patient outcomes, and often these studies employed a
non-randomised design.
The meta-analysis found no
statistically significant effect of guideline implementation in
terms of provider adherence to guidelines (odds ratio (OR) =
1.25, 95% CI 0.76 to 2.07; P value 0.38), whereas a statistically
significant trend in favour of guideline implementation on
patient outcomes was revealed (OR = 1.51, 95% CI 1.03
to 2.22; P value 0.04). The results did not show an
advantage of guideline implementation strategies performed
on professional and organisational level over strategies
that only included professional interventions. Conclusions:
Current evidence is not sufficient to draw firm conclusions
on the beneficial effects of guideline implementation on
provider performance and patient outcome. Regarding the
limited evidence, we were not able to determine the best
way to translate guidelines into practice. In the light of
numerous guidelines developed, there is a strong need for
further research on this topic by applying optimised methods
for the evaluation of guideline implementation.
P1.134
Bunches of evidence: displaying ‘Summary of
findings’ tables visually as ‘evidence flowers’
Jordan J1 , Babatunde O1 , van der Windt D1
1
Research Institute for Primary Care & Health Sciences, Keele
Background: ‘Summary of findings’ tables in Cochrane
Reviews can be difficult to interpret, particularly by patients
and clinicians, and this may limit their use in clinical
practice. Evidence flowers were originally developed as a
means of expressing the results of large evidence syntheses
to stakeholder groups in two research projects. In this
presentation we explore the feasibility of summarising the
information from ‘Summary of findings’ tables in the form
of evidence flowers. Objectives: To suggest a pictorial,
simple and novel method of displaying the information in
‘Summary of findings’ tables of Cochrane Reviews to improve
their accessibility to mixed audiences. Methods: Five recent
Cochrane Reviews that contain ‘Summary of findings’ tables
were purposively selected to represent a range of types of
reviews and findings, and evidence flowers were created to
represent their ‘Summary of findings’ tables. The petals of the
evidence flowers were coloured according to the Grading of
Recommendations Assessment, Development and Evaluation
(GRADE) evidence rating system and narrative summary of
the quantitative information on the petal was kept brief and
written in plain language. Effect sizes were indicated with a
three-star grading system. Results: The evidence flowers for
the five reviews will be presented alongside the corresponding
‘Summary of findings’ tables. As the evidence in ‘Summary
of findings’ tables is rated using the GRADE system, it was
straightforward to colour the petals of the flower. Deciding
on appropriate narrative summary and how to present the
quantitative information was more problematic. Agreement
on a standardised system for this is needed. Conclusions:
Evidence flowers are a novel and visually stimulating method
for presenting research evidence. We propose that they are
used in addition to ‘Summary of findings’ tables, making the
evidence more accessible to a variety of different audiences.
P1.135
Advocating for Cochrane evidence: packaging
Cochrane HIV/AIDS summaries
Oliver J1 , Kredo T1 , Galloway M1
1
Council, South Africa
Background: Advocating for evidence is a core approach for
Cochrane Strategy 2020. The South African Cochrane Centre
(SACC) aims to increase awareness of relevant Cochrane
HIV/AIDS Reviews. The Cochrane Library may be difficult for
consumers to access due to limited internet availability, and
navigating between reviews is time-consuming. Objectives:
To promote the use of Cochrane evidence by producing a
booklet of plain language summaries for the prevention,
testing and treatment of HIV/AIDS, and disseminating it in
the African region. Methods: We searched the Cochrane
Library to identify all HIV/AIDS reviews published between
2010 and 2014. We used the World Health Organization (WHO)
Guidelines for the Prevention and Treatment of HIV as a guide
to categorize the interventions. The booklet was produced in
print and electronic format (available on the SACC website).
The electronic version was disseminated to SACC contacts on
and around World AIDS Day 2014 and is ongoing. Stakeholders
were informed that printed copies were available. Results:
Our search of the Cochrane Library retrieved 58 HIV reviews
(14 on prevention; seven on testing and counselling; 30 on
treatment including adherence; three on nutritional support;
and four on health services and care). As part of our
promotion strategy we initially emailed the booklet to 1462
SACC contacts on World AIDS Day 2014 and it was included
on the Cochrane Library website. Our stakeholders included
218
provincial government HIV/AIDS departments, the National
AIDS Council, non-governmental and AIDS tre

Abstracts of the 23rd Cochrane Colloquium

Transcription

Similar documents

Jumpin` Jammi Jumpin` Jammin` Pyjama Party Pyjama Party

[email protected]

Reference Guide - Cochrane Library

Ask the Experts Candida

Tufts-New England Medical Center Evidence-based Practice

Readers Choice Award

Review Booster for Magento 2.0 - User Guide

FIRPA Presentation

The Full Picture - Army Technology