H A

HOW
H
TO
O CRITIC
CALLY APPRA
AISE QU
UANTITA
ATIVE
RES
SEARCH
H:
A Guide for Und
dergrad
duate So
ocial W
Work Stu
udents
C
Critically appraising
a
g research is an imp
portant sk
kill that so
ocial work students should
b
begin deve
eloping at the
t underg
graduate level.
l
Critiical appraiisal refers to system
matically
e
examining
g the qualiity and rig
gour of re
esearch. A
Assessing tthe validitty, reliabillity and
llimitations
s of research is cruc
cial becaus
se research
h informs practical decision-m
making,
iincluding clinical an
nd policy decisions.
d
This guid
de outliness a series of steps involved
iin effective
ely carrying out this process.
FIIRST ST
TEP: LOO
OKING FO
OR RESE
EARCH
S
Social worrk is a multidiscipl
m
linary, mu
ulti-metho
od field off study th
hereby add
ding an
e
extra challenge whe
en searchiing for rellevant artiicles in yo
our area o
of research
h. With
a
advanceme
ents in tec
chnology over
o
the pa
ast 10 yea
ars, onlinee literature
e review se
earches
h
have become more sophistica
ated and in
i many w
ways, easiier to carrry out. It is now
p
possible to
o filter you
ur search liimits enab
bling searcches by discipline, m
methodolog
gy (e.g.,
q
quantitativ
ve versus qualitativ
ve, meta-a
analyses, systematiic reviews), languag
ge, and
c
country off publicatio
on, among
gst others.. Universiities often offer free training s
sessions
w
with a librrarian to le
earn how to use diffferent dattabases (e..g., EBSCO
O, PsychN
Net) and
s
software to
t help build yourr reference
e library (e.g., End
dnote, Zo
otero). By taking
a
advantage of these resources,
r
you will save valuab
ble time an
nd future headache
es!
A key parrt of the critical
c
appraisal prrocess is to
u
understand the so
ource of the inforrmation o
or
““evidence” in your search. The literrature will
iinclude jou
urnal publications that
t
may or
o may no
ot
b
be peer-re
eviewed. Pe
eer review
w refers to
o a processs
w
where artticles are evaluated
d and crritiqued b
by
rresearcherrs and exp
perts in th
he same fie
eld prior tto
p
publication
n. It often comes as
s a surpris
se to man
ny
s
students that not all pub
blished arrticles arre
e
empirically
y rigorous.
T
The top 10
1 peer-re
eviewed social work journalls
a
according to “impactt factor” (i.e., a mea
asure of th
he
a
average number
n
off citations
s to recent articlees
p
published in science and soc
cial science journalss)
a
are listed in
i the box
x above. Itt is importtant to
b
be aware that
t
the quantity
q
of citations
s may
n
not be reflective of th
he quality, the rank
king of
1
Top
p Social Wo
ork Journals
s by
Impactt Factor 11) Children
n and Youth
Services Review 2
2) Social Se
ervice Review
w
3
3) Social W
Work
4
4) British J
Journal of So
ocial
Work
5
5) Journal o
of Social Wo
ork
Educatio
on
6
6) Research
h on Social W
Work
Practice
7
7) Families in Society
8
8) Health & Social Work
k
99) Social Work Research
110) Social Work in Health Care
Source: IISI (2007)
journals changes over time and is not always agreed upon by scholars. Known as
“grey literature” there are also additional sources of literature that might not be peerreviewed, but are equally important to review. Examples of grey literature sources
include government documents, policy reports, books, theses/dissertations (now
available in e-thesis form) and presented unpublished posters and papers. All
documents that you consider in your literature review require the passing of a
“critical fine-toothed comb” prior to including them in your review, which we
illuminate in this article. Also, remember to look at article bibliographies, especially if
you are studying a relatively new area of research. Finally, Google Scholar can also be
a helpful tool and has the advantage of including content in several languages as well
as indexed records with links to the primary source. However, do not rely strictly on
Google Scholar; it is a good place to begin your search but will not provide you with a
comprehensive or exhaustive literature review. For instance, some subjects are
covered more than others, and searches often contain “gaps” wherein sources (even
those from top-ranking journals) are missing (Jasco, 2005).
SECOND STEP: CRITICALLY APPRAISING RESEARCH
It is important to read research with a clear purpose or goal in mind. Start by
skimming the article before reading it thoroughly. You can learn a lot from the title,
abstract, methods and findings sections. When critically appraising research, you
should determine whether the article is relevant to your issue and setting, the
appropriateness of the design and methods, what the author’s conclusions are, and
how confident you are about the findings.
Here





are some points to consider:
What is the purpose of the study?
Are hypotheses clearly stated?
What is the appropriateness of the study design?
How reliable are the measures?
Does the study adequately control for differences between groups being
compared?
 Are the statistical methods appropriate?
 How valid are the findings?
 Can the findings be generalized to other populations?
Additionally, when critically appraising research, two important issues should always
be considered:
1. Why was the study done? What research questions were the
authors addressing?
The introduction of a research paper should state the research problem of interest.
For example, an article on early childhood trauma resulting from maltreatment might
state: Early childhood experiences of adversity often have long-term effects on
2
individuals. This statement should be followed by a brief review of the published
literature, such as: Evidence from child maltreatment literature indicates that early
childhood trauma has a critical and long lasting negative effect on psychological
outcomes. Unless it has already been covered in the introduction, the hypothesis that
the authors have decided to test should be clearly stated in the methods section of
the paper. If the hypothesis is presented in the negative, such as: The experience of
early childhood trauma resulting from maltreatment does not have long-term effects on
later life outcomes, it is known as a null hypothesis, or hypothesis of no difference.
The authors of a study rarely believe their null hypothesis when they embark on their
research, but they start by assuming there is no difference and try to disprove that
hypothesis.
2. What type of study was done? What was the research design?
First, decide whether the paper describes a primary study, a secondary study or a
program evaluation.
PRIMARY RESEARCH:
Primary research refers to original research and includes exploratory, descriptive,
and explanatory research.
Exploratory Research
Exploratory studies look at an area that has not yet been studied and/or defined. In
these studies, researchers want to develop initial ideas in order to develop a more
focused research question (Neuman & Robson, 2011). For example, an individual
may want to explore existing programs that aim to reduce aggressive behaviours in
children. Qualitative methods are often used in exploratory research, or as a first
step in informing a quantitative research question. Exploratory studies usually
precede explanatory studies that aim to identify causal relationships.
Descriptive Research
Descriptive studies are conducted in order to describe a population or subject of
interest. Thus, no direct cause-effect relationship is determined. Descriptive studies
are important to gain a clear picture of a population so that the need for effective
interventions, programs and policies can be established. For example, in the 2008
Canadian Incidence Study of Reported Abuse and Neglect (Trocmé et al., 2010), the
number (frequency) of characteristics of the population was collected (e.g., rates of
maltreatment by type, age, gender, and many other factors) in order to provide a
picture of child maltreatment in Canada. Descriptive research may employ
surveys/questionnaires and non-intrusive observation techniques:
Surveys/Questionnaires – A method of data collection whereby several people are
systematically asked the same questions and their answers are recorded and
analyzed (Neuman & Robson, 2011).
3
Non-Intrusive Observation Studies – A method of data collection that involves
systematically observing and recording conditions or behaviours, typically using an
observation rating form (Remler & Van Ryzin, 2011).
Explanatory Research
Explanatory studies, also known as experimental studies, focus on why events occur.
They are conducted to explain some kind of causation, or a cause-effect relationship
between two or more variables. In explanatory research, an intervention (or variable)
is directly manipulated. The effects are then observed on the experimental group and
compared to one or several other groups. In social science research, there are two
predominant forms of experimental design - randomized field experiments (commonly
known in the medical field as randomized controlled trials) and quasi-experiments:
Randomized Field Experiments - In a randomized field experiment (RFE)
participants are randomly assigned to either an intervention group (e.g. a new
therapeutic treatment) or a control group (e.g. no treatment or a standard treatment).
Randomization allows us to make estimates about causal relationships, as it ensures
that any differences that occur between groups occur by random chance, and not
because of other factors that could impact the dependent variable. For example,
suppose we wanted to compare depression scores following two types of treatment:
CBT or pharmacological therapy. If a sample of clients is randomly assigned to take
part in one treatment or the other, we would expect any differences between
individuals in the groups to be factors of random chance. As such, we could
attribute any difference in depression scores that we observed to be caused by the
treatment itself, and not by pre-existing characteristics of participants. An RFE
should answer the question: Is this intervention better than no intervention, or better
than a different intervention? RFEs are generally considered the “gold standard” of
research design. However, they are not often carried out due to ethical concerns (e.g.
withholding treatment), impracticality and high cost (Shadish, Cook, & Campbell,
2002). As such, they are found less commonly than descriptive studies in social
science research.
Quasi-experiments – Quasi-experimental research designs are a variation of
experimental design, but lack the researcher control or random assignment
characteristics of a true experiment (Remler & Van Ryzin, 2011). This type of design
is often used in social work research when randomized experiments are not possible
due to practical or ethical constraints. Quasi-experiments are the most common type
of research design in experimental research. These designs can include nonequivalent comparative groups design, simple time-series design, and multiple timeseries design (other research designs such as cross sectional studies and case-control
design studies are not elaborated here):

Non-equivalent comparative groups design - compares a group that appears
similar to the experimental group.
4


Simp
mple time-s
series des
sign - atte
empts to d
develop cau
usal infere
ences based on a
comparison of
o trends over mu
ultiple meeasuremen
nts beforre and affter an
interrvention is
s introduce
ed.
Multiple time
e-series de
esign – bo
oth an exp
perimentall group an
nd non-equ
uivalent
m
at multiplle points iin time be
efore and a
after an
comparison grroup are measured
interrvention is
s introduce
ed to the experimen
e
ntal group (Rubin & Babbie, 20
010).
S
SECONDA
ARY RESE
EARCH:
S
Secondary
y research attempts to draw co
onclusionss from reseearch or in
nformation
n
p
provided by
b others (R
Remler & Van Ryzin
n, 2011). T
This can in
nclude ana
alyzing sec
condary
d
data (such
h as panel survey da
ata), or sum
mmarizing
g existing p
published studies.
T
The latter type of stu
udy includ
des the
ffollowing:
Both the C
Cochrane an
nd Campbell
Collaborattions are inte
ernationally
reccognized org
ganizations, known for h
high
sstandard pro
ocedures used to conduct
sysstematic revviews summarizing existting
eviden
nce-based re
esearch.
N
Non-systematic rev
views –sum
mmarize
s
several priimary stud
dies
S
Systematiic reviews
s –summa
arize
s
studies acc
cording to
o a rigorous and
p
predefined
d methodollogy
ww
ww.cochrane
e.org
M
Meta-anallyses – inttegrate and
d
s
synthesize
e the nume
erical data
a from
s
several stu
udies.
www.cam
mpbellcollabo
oration.org
P
PROGRAM
M EVALUA
ATION:
P
Program evaluation
e
research is a metthod used
d to provid
de eviden
nce regard
ding the
e
effectivene
ess or op
peration of
o social programss and cliinical pra
actices. P
Program
e
evaluation
ns are con
ncerned wiith how programs a
are planneed, how w
well they o
operate,
a
and how effectively
e
they achie
eve their goals
g
(Mon
nette, Sulllivan & De
eJong, 199
94) and
c
can be cla
assified as summativ
ve and form
mative. Su
ummative evaluation
ns are con
ncerned
w
with the ultimate
u
su
uccess of a program
m and deccisions reegarding w
whether it should
c
continue or
o should have been chosen in the firrst place a
among altternative o
options.
F
For examp
ple, the Ch
hicago Ch
hild-Parentt Center p
preschool program w
was evalu
uated to
d
determine the effecttiveness off family su
upport an d increaseed parenta
al involvem
ment in
rreducing child
c
maltrreatment outcomes (Mersky, Topitzes & Reynold
ds, 2011). On the
o
other hand
d, formatiive evalua
ations are not conceerned with
h testing the succe
ess of a
p
program; they
t
focus
s instead on
o obtainin
ng inform
mation thatt is helpfu
ul in plann
ning the
p
program and
a
impro
oving implementatio
on and performancee (Rubin & Babbie, 2010).
P
Program ev
valuation is an esse
ential part of the devvelopmentt of social programs, and is
iimportant for fundin
ng institutiions and policy
p
mak
kers.
5
Summ
mary points
 When deciding whether
w
an article is rrelevant to your area o
of study, fiirst
establlish what sp
pecific prac
ctice questiion it addreesses.
 While peer-revie
ewed sociall work jou
urnals are highly reg
garded, ma
any
publis
shed papers may not be rigorou
us. Thus, it is importa
ant to be ab
ble
to crittically apprraise a wid
de variety o
of literaturee (e.g. grey
y literature
e &
peer-rreviewed).
 Quick
kly scan the
t
title, abstract, summary, methods and resu
ults
section
ns.
 Take note of what
w
“type”” of articlee you are reading ((i.e., prima
ary
researrch, second
dary researrch) as welll as the sp
pecific study
y design (e
e.g.
quasi--experimen
ntal) and ke
eep these in
n mind wh
hen drawing
g conclusio
ons
from the
t research
h questions
s.
This
s article is ad
dapted from
m:
Greenh
halgh, T. (1997). How to read a papeer. Education
n and Debatte, 3, 243-24
46.
Re
eferences
In
nstitute for Scientific
S
Infformation [IS
SI]. (2007). Eigenfactor.o
E
org ranking and mappin
ng scientific
knowlledge. Retrie
eved from: http://www.c
childwelfare .com/rankin
ngs.htm
Jasco, P. (200
05). As we may
m search – comparison
n of major feeatures of th
he Web of Sc
cience, Scopus and
Googlle Scholar citation based
d and citatio
on enhanced
d databases. Current Sciience, 89, 15
537-1547.
Retrie
eved from: www.ias.ac.in
w
n/currsci/no
ov102005/1
1537.pdf
M
Mersky, J., Topitzes,
T
J., & Reynolds, A. (2011). Maltreatmen
M
nt prevention
n through ea
arly childhoo
od
interv
vention: A co
onfirmatory evaluation
e
of
o the Chicag
go Child-Parrent Center p
preschool prrogram.
Childrren and Youth Services Review,
R
33, 1454-1463.
M
Monette, D. R.,
R Sullivan, T. J., & DeJ
Jong, C. R. (1994).
(
Appllied social re
esearch: Toolls for the hum
man
servicces. Fort Worrth, Texas: Holt.
H
N
Neuman, W. L. & Robson
n, K. (2011). Basics of so
ocial researcch: Qualitativ
ve and quan
ntitative apprroaches
dian edition). Toronto: Pe
earson Education Canad
da.
(Canad
R
Remler, D.K., & Van Ryziin, G.G. (2011). Researcch Methods iin Practice: S
Strategies for Description
n and
nd Oaks, CA
A: Sage.
Causattion. Thousan
R
Rubin, A. & Babbie,
B
E. (2
2010). Resea
arch methods for social w
work (7th ed. ). Belmont, California:
Brook
ks/Cole.
S
Shadish, W. R.
R et al. (200
02). Experim
mental and quasi-experim
mental designs for generral causal infference.
Boston: Houghton Mifflin.
T
Trocmé, T., Fallon,
F
B., MacLaurin,
M
B.,
B Sinha, V.,, Black, T., F
Fast, E.,…Holroyd, J. (2
2010). Canad
dian
Incide
ence Study of Reported Child
C
Abuse & Neglect 20
008 (CIS 200
08): Major Fiindings. Otta
awa:
Public
c Health Age
ency of Cana
ada.
6