The peer review process: a primer for JNIS readers

Downloaded from http://jnis.bmj.com/ on July 6, 2015 - Published by group.bmj.com
Review
The peer review process: a primer for JNIS readers
Joshua A Hirsch,1 Laxmaiah Manchikanti,2,3 Felipe C Albuquerque,4
Thabele M Leslie-Mazwi,1 Michael H Lev,5 Italo Linfante,6 J Mocco,7
Ansaar T Rai,8 Pamela W Schaefer,5 Robert W Tarr9
For numbered affiliations see
end of article.
Correspondence to
Dr Joshua A Hirsch,
NeuroEndovascular Program,
Massachusetts General
Hospital, Harvard Medical
School, Boston, MA 02114,
USA; [email protected]
ABSTRACT
Peer review of scientific articles submitted for publication
has been such an integral component of innovation in
science and medicine that participants (be they readers,
reviewers, or editors) seldom consider its complexity. Not
surprisingly, much has been written about scientific peer
review. The aim of this report is to share some of the
elements of that discourse with readers of the Journal of
NeuroInterventional Surgery (JNIS).
Received 30 March 2015
Accepted 31 March 2015
INTRODUCTION
For more than 200 years, clinicians and researchers
have published their work in scientific and medical
journals.1–3 With changing production paradigms,
an ever-greater number of investigators worldwide
have contributed scientific articles. In 2010, Bastian
and colleagues4 noted that 75 trials and 11 systematic reviews were published every day. In 2012,
there were 28 000 scholarly, peer-reviewed journals
collectively publishing nearly 2 million articles a
year. The PubMed website indicates that almost 21
million articles have been published in biomedical
journals.5
In 2009, working with the British Medical Journal
(BMJ) group, the Society of NeuroInterventional
(NI) Surgery began publication of the Journal of
NeuroInterventional Surgery (JNIS). The main
purpose of JNIS was to be an academic, peerreviewed journal focusing on publishing scientific
articles to advance the field of NI surgery. As a result,
NI specialists look to JNIS for discussions of trials,
new techniques, and breakthroughs, among other
topics of interest. Therefore, we provide this assessment of the peer review process to advance the JNIS
mission.
HISTORY
To cite: Hirsch JA,
Manchikanti L,
Albuquerque FC, et al.
J NeuroIntervent Surg
Published Online First:
[please include Day Month
Year] doi:10.1136/
neurintsurg-2015-011781
In 1655 Denis de Sallo published the first collection of scientific essays in the Journal des Scavans.6
In this journal, De Sallo reported the findings of
others in an effort to communicate knowledge.
Peer review did not exist at that time.7 8 It was only
in 1731 that the Royal Society of Edinburgh published ‘Medical Essays and Observations’ using
what is now considered to be the first peer review
process.7 8 Manuscripts were mostly distributed, by
the editor, directly to individuals with expertise in
the articles’ subject matter with the purpose of
selecting articles for their innovation and interest.7
Like many advances in science and medicine, peer
review was not immediately appreciated as a breakthrough.8 An exemplar of that point is Albert
Einstein, who published over 300 journal articles
between 1901 and 1955, including his miracle year
of 1905 when he introduced new ways of understanding space, time, energy, momentum, light, and
the structure of matter. In the course of his entire
career he probably had only a single paper peer
reviewed, in 1936, for a submission he jointly
wrote about gravitational waves. The paper
received negative reviews, and Einstein rejected in
writing both the idea of an anonymous expert
reviewer selected by the editor and that expert’s
comments, eventually submitting the paper elsewhere.9 His response suggests someone unfamiliar
with the fundamental tenets of peer review. One
can speculate that had Einstein lived today his view
of peer review might have been somewhat
different.
The Royal Society of Edinburgh was careful in
making clear that the stamp of peer review did not
necessarily mean the work was better than
non-peer-reviewed publications. They thus provided a disclaimer stating that peer review did not
guarantee truthfulness or accuracy. They emphasized further that the purpose of the journal was
solely to disseminate creative and important
ideas.7 8 As today, the submitting authors were
responsible for the quality and veracity of their
own research.7 8
As the peer review process has been embraced
over time, its development has occurred in fits and
starts.6–8 10 Different journals apply varying
approaches to peer review. Until 1976, the Lancet
did not implement peer review. At that time,
Journal of the American Medical Association
(JAMA) sent their submissions to an internal review
panel and, on rare occasions, to outside experts.11
In contrast, the BMJ—with whom JNIS shares a
common publisher—has sent every non-editorial
submission to a recognized expert since 1893.10 By
the late 20th century, peer review has become
standard and is used by almost all biomedical
journals.8 10 11
PEER REVIEW
The modern peer review process has a variety of
constituents.8 10 11 Authors expect to receive a fair
and unbiased critique. Journals need to ensure the
integrity and accuracy of the material that they
publish. The lay public relies on the integrity of the
publication process to inform themselves and their
providers of innovative and optimal treatments.
The common thread for maintaining the quality
and integrity of research is the peer review process.
In essence, it provides a formal opportunity for
authors to gauge reaction to their work, as well as
allowing for revisions and subsequent correction of
errors before publication. At its best, peer review
Hirsch JA, et al. J NeuroIntervent Surg 2015;0:1–4. doi:10.1136/neurintsurg-2015-011781
1
Downloaded from http://jnis.bmj.com/ on July 6, 2015 - Published by group.bmj.com
Review
allows the editors and expert reviewers to apply with diligence
and care their unbiased knowledge, expertise, and interpretation.8 10–12
The peer review process, nevertheless, is not without its
flaws.8 Cantekin et al13 in an analysis of the peer review,
exposed a variety of problems leading to imperfection and bias.
Some of these include reviewer conflict of interest, breach of
confidentiality, and non-disclosures of funding sources.8 14–16
An additional problem that can arise in peer review is reluctance
on the part of reviewers or editors to accept papers that express
views contrary to currently held norms.15 17 18
The foundation of the selection process for articles submitted
to JNIS, and most other academic journals, is the work of volunteer reviewers. After agreeing to review an article, reviewers are
given a deadline to perform their analysis. Distinct comments
are solicited for both the authors and the editors. When the
associate editor is satisfied with the number and quality of the
revisions, he or she will make a recommendation to the editor
in chief (EIC). The EIC plays an important role in establishing
uniformity between the associate editors. The EIC makes one of
the following recommendations: accept, accept pending minor
revisions, accept pending major revisions, revise and resubmit,
and/or reject (box 1). If revisions are requested, they are
returned with an annotated pdf and a cover letter dealing with
the reviewer queries, typically, on a point-by-point basis. At
JNIS, reviewers are blinded to the authors’ identities, but the
editors are not. Other review formats exist, ranging from
blinded editors and reviewers, to completely open review.8
The majority of the scientific community considers peer
review to be valuable.7 8 10 11 19 In a 2009 survey of 4000
people involved in research, although 84% believed peer review
to be an important, arguably necessary, process for the dissemination of accurate scientific information, only 69% were satisfied
with current peer review systems.19 Despite the questions this
survey raised about perceptions of peer review, however, it is
noteworthy that fully 91% of respondents believed their last
paper had been improved by the peer review process.
PEER REVIEW BIAS
Bias may be defined as a systemic prejudice that prevents the
accurate and objective interpretation of scientific studies.8
Indeed, it has been suggested that the objectivity of peer review
has been affected in many different ways. These include, but are
not limited to, differences in belief or ideology, aversion to new
ideas,20 and conflicts of interest. Other potential concerns also
include cursory/superficial reviews by reviewers with decreasing
motivation and enthusiasm for reviewing over time,21 the occasional need for associate and chief editors to ‘over-ride’ the
unanimous recommendation of reviewers to reject papers that
may be ‘hidden gems’, and the poor ability to train reviewers to
improve the quality of their reviews and eliminate bias.22 23
The inability to allow dissenting views to be published in journals has become a major concern.8 18 Members of the NI community have had this angst several times in recent years.
Specifically, in 2009, when the vertebroplasty versus controlled
intervention trials were published, and again in 2013, when the
three inconclusive major intra-arterial stroke therapy trials were
published in the New England Journal of Medicine, content
experts were not given the opportunity to publish contemporaneous counterpoints. Rather, we were left to write short letters to
the editor that were frequently rejected—while at the same time
these studies became sensationalized by the lay media, to the
detriment of our patients.17
2
Box 1 Various steps in publication of a manuscript in
the Journal of NeuroInterventional Surgery (JNIS):
submission, peer review, and publication
1. Pre-submission phase (investigators)
▸ Research including data review and synthesis
▸ Initial manuscript preparation
▸ Internal review by authors
▸ Final manuscript signed off by all authors
▸ Submission to JNIS, including patient consent forms
2. Post-submission phase
I. Immediate post-submission phase ( journal)
▸ JNIS receives the manuscript
▸ Editorial manager logs and acknowledges receipt
▸ Quality check by editorial manager and editor in
chief: articles may be rejected at this point
II. Editors
▸ Editor in chief assigns manuscript to an associate
editor
▸ Editors review manuscript quality and confirm
relevance to the journal
▸ Editors decide on relevant experts to consider peer
review
III. Peer review
▸ Peer reviewers examine the manuscript for study
design/methodology, validity, accuracy, originality,
significance, quality of English, etc
▸ Peer reviewers will often ask for additional
comments, to have queries answered, for additional
references cited, and/or more rigorous statistical or
other additional expert review
IV. Associate editors
▸ Consider reviewer comments
▸ Make recommendation to editor in chief to reject/
revise/accept
3. Post associate editor recommendation
I. Editor in chief sends a letter with his/her decision about
the paper
▸ Authors respond to comments and suggestions
▸ Authors revise manuscript
▸ Authors resubmit with a cover letter dealing with
the reviewer comments/questions and a marked up
manuscript
II. Editors
▸ Final decision to accept/reject manuscript
4. Post-acceptance at JNIS
▸ Proofs with requested 48 h turnaround
▸ Publication online first
▸ Publication in print
Adapted and modified from Manchikanti et al.8
Bunner and Larson carried out pioneering work looking at
the quality of the peer review process from the perspective of
authors and editorial board members.14 Approaching 80%, a
large proportion of those who responded to the survey, indicated that they knew what to expect from the peer review
process, 19% indicated that they did not have a specific sense,
and approximately 3% indicated that they did not know what
to expect.14 For manuscripts that were accepted, 84% of
authors viewed their reviews favorably, whereas for manuscripts
that were rejected, 50% viewed them unfavorably. Perhaps more
Hirsch JA, et al. J NeuroIntervent Surg 2015;0:1–4. doi:10.1136/neurintsurg-2015-011781
Downloaded from http://jnis.bmj.com/ on July 6, 2015 - Published by group.bmj.com
Review
surprising, almost 30% of authors felt that reviewers did not
have an accurate understanding of the paper’s topic; an additional 34% felt the reviewer comments were insufficient, and
about 30% felt that the reviewers failed to provide suitable suggestions for revision. Most importantly, Bunner and Larson14
indicated that almost 50% felt the critiques were unfair and
biased.
TYPES OF PEER REVIEW BIAS
Put simply, reviewer bias is a violation of the ideal impartiality
that a reviewer or editor has in evaluating a submission.
Impartiality in this context is the ability for any reviewer to
interpret and apply evaluative criteria consistently for any given
manuscript.6–8 10–14 16 Peer review, however, can be difficult to
study systematically, as the process is typically subjective and
secretive.22
Gilbert et al24 in 1994 and Jayasinghe et al25 in 2003 showed
that specific classes of reviewers are systematically tougher or
softer24 25 on identical submissions.26 For example, reviewers
from the USA tend to be more lenient than their colleagues
from the UK, Germany, or Australia.8 Moreover, until quite
recently, there has been a well-studied, strong positive bias of
reviewers to favor positive, rather than negative study
results.27 28
Editors must pay specific attention to these issues for impartial assessment and publication.
Content-based bias
As the name suggests, this links directly to content of part or all
of the manuscript. Content-based bias may reflect reviewers and
editors preferring to see submissions that cite their own work.
Cognitive cronyism exists when favoritism is given to submissions that share similar schools of thought.8
Confirmation bias
Confirmation bias is the tendency to gather, interpret, and
remember evidence in ways that affirm previously held beliefs.
In the peer review system, confirmation bias is understood as
reviewer or editor bias against manuscripts where the thesis and/
or results disagree with the perspective of the reviewer or
editor.8 This type of bias goes towards the heart of the peer
review process, wherein evaluations are performed independently of “desires, value perspectives, cultural and institutional
norms and presuppositions, expedient alliances and other interest.” In a telling article, Ernst et al,29 found that reviewers who
had previously published materials in favor of what they termed
a controversial clinical intervention looked at a manuscript
where the data supported the use of that technique with a
friendlier eye than those who had produced manuscripts that
did not support the use of that technique.
bias raises potential downstream problems because it will—by its
nature—lead to errors in effect size measurements in later
meta-analyses.30 Publication bias is hypothesized to occur
because clinicians and researchers can disagree about the importance of research that produces negative outcomes.30 However,
the authors of this manuscript believe that, more recently, a
‘reverse’ form of publication bias may be at play in the neurointerventional literature, in that some notable recent NI published
trials have indeed highlighted negative results—gaining great
notoriety and providing considerable sensationalism.8 18
Bias of conflicts of interest
Perhaps the most self-evident forms of bias, a ‘conflict of interest’ occurs when participants in the publication process have a
personal interest that might inappropriately influence their judgment, regardless of whether or not their judgment is actually
affected.3 8 15 18 Editors and reviewers may be more openminded to research from friends, collaborators, or clinicians,
particularly if the report supports the professional world view
of the reviewer or editor. Conversely, editors and reviewers may
be more likely to reject a submission from a clinician whom
they dislike, who supports a competing idea, or who opined
negatively on a manuscript written by the current reviewer in
the past. Conflicts of interest can be difficult to detect.
FORMS OF PEER REVIEW
Double-blind peer review
Double-blind peer review is preferred primarily because of its
perceived objectivity and fairness.8 31 Challenges include the
difficulty of making a review truly double blind. In a survey of
1500 editors in chemistry, most respondents felt that double
blinding was pointless and, by implication, an unnecessary step,
because the particular content and the references chosen indirectly make the author’s identity clear. Moreover, a number of
empirical studies showed that reviewers can successfully identify
authors 25–40% of the time. In a subspecialized field like neurointerventional surgery, we believe this observation has validity.
Open peer review
With this approach, the authors and reviewers are known to
each other. This has the obvious benefit of increasing transparency and the possibility of speeding up the review process. Since
1999, the BMJ has asked potential reviewers to identify themselves and declare any competing interests. The reviewers are
not disclosed to BMJ readers at the time of publication. One
might consider this approach as a single-blind review. JNIS
worked with our BMJ publishers to develop the system we use
today—namely, double blind, without identifying reviewers and
authors to each other.
FRAUD AND PEER REVIEW
Conservatism bias
This is a bias against innovative or landmark research. In its
most dangerous form, conservatism bias threatens scientific progress in a very basic and understandable way.20 30 Specifically,
this type of bias can limit funding and public sharing of theories
that deviate from mainstream hegemony and result in authors
facing a higher burden of proof requirement to get their publications accepted.
Publication bias
Alleged publication bias is the tendency for journals to publish
research demonstrating positive, rather than negative, outcomes.8 18 27 28 If these assumptions are accurate, publication
When one considers the purpose and methodology of peer
review, one can argue that it is not necessarily designed to
detect fraud. Bearing that in mind, and recognizing that the critique may not be fair, the peer review approach, has been criticized by some participants for its inability to detect fraud.6 8 To
that point, Godlee et al32 intentionally introduced eight specific,
arguably detectable, weaknesses into a research article. They
then sent the flawed paper to 200 reviewers. Subsequent analysis of the reviewer comments showed that, on average only
two of the inserted weaknesses were identified.
Peer review cannot ensure that the paper does not have any
mistakes. Rather, alternative mechanisms of surveillance and veracity confirmation may occasionally need to play a role in
Hirsch JA, et al. J NeuroIntervent Surg 2015;0:1–4. doi:10.1136/neurintsurg-2015-011781
3
Downloaded from http://jnis.bmj.com/ on July 6, 2015 - Published by group.bmj.com
Review
ensuring the integrity of academic literature. Indeed, a researcher
was recently imprisoned for falsifying data in clinical trials.8
6
CONCLUSION
8
Peer review is a critical element of our routine scientific discourse. JNIS, like many journals, could not function in its
current format without it. It has served the medical/scientific
community well for over 100 years.6–8 13 16 For a process that is
so fundamental to our specialty and academic activities,
however, it is sobering to think how much is assumed and how
little we actually know about the biases and limitations inherent
to peer review. The editorial staff at JNIS believe in peer review
and favor the approach that we have crafted with the BMJ—
most notably, that the authors and reviewers are blinded to one
another, but not to the editorial staff.
Author affiliations
1
NeuroEndovascular Program, Massachusetts General Hospital, Harvard Medical
School, Boston, Massachusetts, USA
2
Pain Management Center of Paducah, Paducah, Kentucky, USA
3
Department of Anesthesiology and Perioperative Medicine, University of Louisville,
Louisville, Kentucky, USA
4
Barrow Neurological Institute, Phoenix, Arizona, USA
5
Department of Diagnostic Neuroradiology, Massachusetts General Hospital, Harvard
Medical School, Boston, Massachusetts, USA
6
Department of Interventional Neuroradiology, Miami Cardiac and Vascular Institute,
Miami, Florida, USA
7
Mount Sinai Health System, Icahn School of Medicine, New York, New York, USA
8
Department of Interventional Neuroradiology, West Virginia University,
Morgantown, West Virginia, USA
9
Department of Radiology, University Hospitals Case Medical Center, Cleveland,
Ohio, USA
Contributors JAH and LM composed the initial draft. All authors reviewed and
made editorial suggestions that resulted in the final draft.
Competing interests None declared.
7
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Provenance and peer review Commissioned; internally peer reviewed.
26
27
REFERENCES
1
2
3
4
5
4
Hirsch JA, Schaefer PW, Romero JM, et al. Comparative effectiveness research.
AJNR Am J Neuroradiol 2014;35:1677–80.
Gagliardi AR, Brouwers MC. Integrating guideline development and implementation:
analysis of guideline development manual instructions for generating
implementation advice. Implement Sci 2012;7:67.
Manchikanti L, Falco FJ, Singh V, et al. An update of comprehensive
evidence-based guidelines for interventional techniques in chronic spinal pain. Part
I: Introduction and general considerations. Pain Physician 2013;16(2 Suppl):S1–48.
Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews
a day: how will we ever keep up? PLoS Med 2010;7:e1000326.
US National Library of Medicine NIoH. Pubmed (cited 25 March 2015). http://www.
ncbi.nlm.nih.gov/pubmed
28
29
30
31
32
Benos DJ, Bashari E, Chaves JM, et al. The ups and downs of peer review.
Adv Physiol Educ 2007;31:145–52.
Kronick DA. Peer review in 18th-century scientific journalism. JAMA
1990;263:1321–2.
Manchikanti L, Kaye AD, Boswell MV, et al. Medical journal peer review: process
and bias. Pain Physician 2015;18:E1–14.
Nielsen M. Three myths about scientific peer review. 2009 (cited 25 March 2015).
http://michaelnielsen.org/blog/three-myths-about-scientific-peer-review/
Burnham JC. The evolution of editorial peer review. JAMA 1990;263:1323–9.
Rennie D. Editorial peer review: its development and rationale. In: Godlee F,
Jefferson T, eds. Peer review in health sciences. London: BMJ Books, 1999.
Verissimo D, Roberts DL. The academic welfare state: making peer-review count.
Trends Ecol Evol 2013;28:623–4.
Cantekin EI, McGuire TW, Potter RL. Biomedical information, peer review, and
conflict of interest as they influence public health. JAMA 1990;263:1427–30.
Bunner C, Larson EL. Assessing the quality of the peer review process: author and
editorial board member perspectives. Am J Infect Control 2012;40:701–4.
Manchikanti L, Hirsch JA, Cohen SP, et al. Assessment of methodologic quality of
randomized trials of interventional techniques: development of an interventional
pain management specific instrument. Pain Physician 2014;17:E263–90.
Twaij H, Oussedik S, Hoffmeyer P. Peer review. Bone Joint J 2014;96-B:436–41.
Hirsch JA, Hirsch AE, Zoarski GH, et al. Social responsibility in medical reporting.
J Neurointerv Surg 2010;2:217–18.
Manchikanti L, Candido KD, Kaye AD, et al. Randomized trial of epidural injections
for spinal stenosis published in the New England Journal of Medicine: further
confusion without clarification. Pain Physician 2014;17:E475–88.
UK RC. Report of the Research Councils UK Efficiency and Effectiveness of Peer
Review Project. 2006. (cited 25 March 2015). http://www.rcuk.ac.uk/RCUK-prod/
assets/documents/documents/rcukprreport.pdf.
Uzzi B, Mukherjee S, Stringer M, et al. Atypical combinations and scientific impact.
Science 2013;342:468–72.
Callaham M, McCulloch C. Longitudinal trends in the performance of scientific peer
reviewers. Ann Emerg Med 2011;57:141–8.
Couzin-Frankel J. Biomedical publishing. Secretive and subjective, peer review proves
resistant to study. Science 2013;341:1331.
Houry D, Green S, Callaham M. Does mentoring new peer reviewers improve review
quality? A randomized trial. BMC Med Educ 2012;12:83.
Gilbert JR, Williams ES, Lundberg GD. Is there gender bias in JAMA’s peer review
process? JAMA 1994;272:139–42.
Jayasinghe UW, Marsh HW, Bond N. A multilevel cross classified modelling
approach to peer review of grant proposals: the effects of assessor and researcher
attributes on assessor ratings. J R Stat Soc 2003;166:279–300.
Ross JS, Gross CP, Desai MM, et al. Effect of blinded peer review on abstract
acceptance. JAMA 2006;295:1675–80.
Emerson GB, Warme WJ, Wolf FM, et al. Testing for the presence of positiveoutcome bias in peer review: a randomized controlled trial. Arch Intern Med
2010;170:1934–9.
Lynch JR, Cunningham MR, Warme WJ, et al. Commercially funded and United
States-based research is more likely to be published; good-quality studies with
negative outcomes are not. J Bone Joint Surg Am Vol 2007;89:1010–18.
Ernst E, Resch KL, Uher EM. Reviewer bias. Ann Intern Med 1992;116:958.
Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:
e124.
Baggs JG, Broome ME, Dougherty MC, et al. Blinding in peer review: the
preferences of reviewers for nursing journals. J Adv Nurs 2008;64:131–8.
Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding
reviewers and asking them to sign their reports: a randomized controlled trial.
JAMA 1998;280:237–40.
Hirsch JA, et al. J NeuroIntervent Surg 2015;0:1–4. doi:10.1136/neurintsurg-2015-011781
Downloaded from http://jnis.bmj.com/ on July 6, 2015 - Published by group.bmj.com
The peer review process: a primer for JNIS
readers
Joshua A Hirsch, Laxmaiah Manchikanti, Felipe C Albuquerque, Thabele
M Leslie-Mazwi, Michael H Lev, Italo Linfante, J Mocco, Ansaar T Rai,
Pamela W Schaefer and Robert W Tarr
J NeuroIntervent Surg published online April 17, 2015
Updated information and services can be found at:
http://jnis.bmj.com/content/early/2015/04/17/neurintsurg-2015-01178
1
These include:
References
Email alerting
service
This article cites 28 articles, 5 of which you can access for free at:
http://jnis.bmj.com/content/early/2015/04/17/neurintsurg-2015-01178
1#BIBL
Receive free email alerts when new articles cite this article. Sign up in the
box at the top right corner of the online article.
Notes
To request permissions go to:
http://group.bmj.com/group/rights-licensing/permissions
To order reprints go to:
http://journals.bmj.com/cgi/reprintform
To subscribe to BMJ go to:
http://group.bmj.com/subscribe/