Art, Science Technology of QDA

The Art, Science and Technology of
Qualitative Data Analysis
Prepared for the 2014 Canadian Evaluation Society Conference:
35 Years: Celebrating Contributions to Canadian Evaluation
by Werner Meier
http://www.RBMG.ca
June 16, 2014
Presentation Overview


Purpose: To demonstrate some techniques
for conducting qualitative data analysis with
the assistance of a computer software
program, i.e. Atlas.ti 7 (see: www.atlasti.com).
My Goal: To impart to you my enthusiasm
for using computer-assisted qualitative data
analysis software (CAQDAS) to conduct
evaluations.
September 22, 2014
www.rbmg.ca
2
Presentation Overview

A Three Part Presentation:

The Art 

The Science 

let’s take 15 min walk down memory lane;
let’s take 15 min to clarify some concepts and terms;
The Technology 
let’s see how it has evolved and what it can do.
…. of Qualitative Data Analysis (QDA)
September 22, 2014
www.rbmg.ca
3
Part 1
The Art of Qualitative Data Analysis
Can You Relate?
September 22, 2014
www.rbmg.ca
5
Is This Technique Familiar?
September 22, 2014
www.rbmg.ca
6
Margin Notes and Drawings

Known benefits?
Summarizing is a good way to keep track of essential
information while condensing lengthier passages.
 Clarifying ideas through a process of analysis, synthesis
and evaluation will increase understanding of the text.
 Making connections within the text and to other
documents will improve comprehension.
 Visualizing will help clarify complex concepts and ideas.
 Thoughtful reactions and questions about the text will
enhance critical analysis.


Any disadvantages?
September 22, 2014
www.rbmg.ca
7
Is This Technique Familiar?
Code:
Code:
Code: Benefits
Source:
Source:
Source: Jonathan LeMaster, Writing and Drawing
in the Margins, page 1
Fact, thought or quote:
“When readers engage in this active reading strategy, they are
clarifying, summarizing, and visualizing ideas presented in the
text.”
September 22, 2014
www.rbmg.ca
8
The Index Card System

Known benefits?
Facts, thoughts or quotations can be linked
to a series of codes that correspond to
the research topics.
 A flexible system that adapts easily to
changes/additions in research topics
and presentation sequence.
 Ensures that source references are well
documented for footnotes endnotes to
demonstrate the evidence base.


Any disadvantages?
September 22, 2014
www.rbmg.ca
9
Is This Technique Familiar?
How does the PH Initiative align with GoC priorities?
Interviewee Responses
Aligned
Yes
Cannot comment
This initiative is directly responding to many calls for strengthening the PH
system. (Many reports recommended this initiative, especially following SARS)
It does not command a large portion of the health system. Other areas are
important but PH capacity is fundamental. It strengthen the PH infrastructure
Created in response to the government’s response to SARS
• Enhanced scientific response; • Assist with hospitals; • Vaccine production
* No answer was provided.
I could not directly link it to the Economic Action Plan but health is a prerequisite of prosperity. We need to have a public health and healthy Canadians
to have a good economy. It's also related to the Health and Safety of the
Nation (GoC priority): For example, SARS had an economic impact on the city
of Toronto. Emergency Preparedness: We need to know how to respond to a
PH emergency, thus, public health needs the capacity to respond.
I would say that this initiative is due to the Naylor Report.
Responses for Stakeholder #1
Total n
Percent
Responses for Stakeholder #2
Total n
Percent
Overall
Total n
Percent
September 22, 2014
How aligned?
Knowledge
No
No
Transfer &
Resp. Resp.
Exchange
1
1
Health and
Emergency
Safety of the
Preparedness
Nation
1
1
1
1
2
50
3
4
75
4
6
67
www.rbmg.ca
1
1
1
0
2
0
2
4
50
2
6
33
0
2
0
2
4
50
2
6
33
1
1
1
1
1
1
2
50
1
4
25
2
6
33
1
2
50
2
4
50
3
6
50
0
2
0
0
4
0
0
6
0
10
The Excel Technique

Known benefits?
Easy transfer (cut/paste) of interview responses to
standard template.
 Can create separate respondent profile columns for
gender, age, occupation, country, etc. to allow sorting.
 A comprehensive list of mutually exclusive response
categories (column titles) is grounded in the response
data to open-ended questions.
 Easy to quantify responses to closed and open-ended
questions.


Any disadvantages?
September 22, 2014
www.rbmg.ca
11
1994 Program Evaluation Standards
Joint Committee on Standards for Educational Evaluation



U2 Evaluator Credibility: the people conducting the
evaluation must be trustworthy and competent to perform
the evaluation in order for the evaluation's findings to
achieve maximum credibility and acceptance.
A3 Described Purposes and Procedures: The purposes
and procedures of the evaluation should be monitored and
described in enough detail, so that they can be identified
and assessed.
A4 Defensible Information Sources: The sources of
information used in a program evaluation should be
described in enough detail, so that the adequacy of the
information can be assessed.
September 22, 2014
www.rbmg.ca
12
September 22, 2014
www.rbmg.ca
13
Part 2
The /\ Science
of Qualitative Data Analysis
Grounded Theory






Origin and Evolution:
Developed since 1967 primarily by sociologists Barney
Glaser and Anselm Strauss and applied in a broad range
of research disciplines.
Main Purpose:
To develop theories based on accepted standards of good
scientific method so as to understand social phenomena.
Main Characteristics:
Iterative research design, theoretical/purposive sampling,
empirical data collection, systematic data analysis,
inductive reasoning, and theoretical elaboration/testing.
September 22, 2014
www.rbmg.ca
15
Types of Research Approaches
Deductive Reasoning







Top – down
General to specific
Theory-based / driven
Issues/topics are known
Closed coding employed
‘Facts’ are gathered
Conclusions tend to
confirm / reject theory
Inductive Reasoning








September 22, 2014
www.rbmg.ca
Theoretical models
developed
Hypotheses tested
Patterns observed
Open coding employed
Issues/topics unknown
Better understanding
Specific to general
Bottom – up
16
Common Types of Analysis








Descriptive Analysis:
Presentation of raw data without interpretation.
Comparative Analysis:
Looking for similarities and differences across cases (see
Miles and Huberman 1994).
Inferential Analysis:
Conjecture about relationships between variables to
support hypothesis testing or an argument.
Triangulation Analysis:
Examination of findings for three lines of evidence
focused on the same topic/issue/outcome.
September 22, 2014
www.rbmg.ca
17
Evaluation Terms






Data Sources:
From where/whom the data was obtained, e.g.
documents, web site, community center, health clinic,
individual(s), etc.
Data Collection Technique:
How the data was obtained, e.g. content analysis,
observation, focus group, interview.
Line of Evidence
Different combinations of data sources and data
collection techniques.
September 22, 2014
www.rbmg.ca
18
Concentric Circles Method
Constant Comparison or Constructivist Grounded Theory
Data Sources
Data Collection Techniques
1. Reference Documents:
1: Content analysis:
GoC policy statements, Dept. RPP /
DPRs, program and project files;
performance data & reports
50 reference documents
2: E-survey: 300
organizations with follow-up
2. Stakeholders:
Health service delivery
providers across Canada
telephone interviews
3. Advisory Cmtte:
3: Tel. interviews:
Health Delivery and Human
Resources, including P/T
government representatives
w/ 25 committee members
4. Key Stakeholders:
4: F2F interviews:
Canadian Medical Association,
Canadian Nurses Association,
College of Surgeons and
Physicians, etc.
w/ 10 senior managers of key
health sector professional
associations
5. Client Programme:
5: F2F interviews:
Health Canada and HRSDC
w/ 4 program managers
September 22, 2014
www.rbmg.ca
19
2011 Program Evaluation Standards
Joint Committee on Standards for Educational Evaluation



U1 Evaluator Credibility: Evaluations should be
conducted by qualified people who establish and maintain
credibility in the evaluation context.
A5 Information Management: Evaluations should
employ systematic information collection, review,
verification, and storage methods.
A7 Explicit Evaluation Reasoning: Evaluation reasoning
leading from information and analyses to findings,
interpretations, conclusions, and judgments should be
clearly and completely documented.
September 22, 2014
www.rbmg.ca
20
Part 3
The Technology
of Computer-Assisted Qualitative Data
Analysis Software (CAQDAS)
EMERGENCE OF CAQDAS
Since 1988
HyperRESEARCH
Since 1989
Since 1993
September 22, 2014
1991
Since 1999
as NUD*IST
www.rbmg.ca
22
Evolution of CAQDAS





CDC EZ-Text - developed to assist health researchers at
the Centers for Disease Control and Prevention to create
and manage interview data using database technology.
C-I-SAID - Code-A-Text Integrated System for the
Analysis of Interviews and Dialogues was developed to
assist psychotherapists in clinical settings.
Saturate - developed using multi-user ‘cloud’ technology.
Dedoose - developed using multi-user ‘cloud’ technology;
web-based pay as you go.
CAQDAS Networking Project - Mission is to encourage the
independent use of CAQDAS packages.
September 22, 2014
www.rbmg.ca
23
Use of CAQDAS
in Social Science Research


Source: Sociological Abstracts database of
1,800 serial publications.
Searched: Journal article abstracts in English.
1990-2004: 31 references to CAQDAS products; 220 to
SPSS, SAS and Stata. (MacMillan and Koenig 2004);
 2004-2014: 94 references to CAQDAS products; 601 to
SPSS, SAS and Stata.
 CAQDAS Review Papers = 5; CAQDAS Methods Papers
= 18; Applied Research Papers using CAQDAS = 70
 2004-2008 < 10 / year; 2008-2014 > 10-20 / year

September 22, 2014
www.rbmg.ca
24
Use of CAQDAS in Evaluations
Literature Search Summary



Source: Scholars Portal – Peer reviewed Journals
w/”Evaluation” in the title = 15
Search String: CAQDAS products in Anywhere,
English, PubYear 2008 to 2014.
Findings: 67 articles referenced use of CAQDAS
products:
Nud*ist/NVivo = 37; Atlas.ti = 24; MaxQDA = 5;
QDAMiner = 1; Others = 0
 From 2008-2009 <10/year; 2010-2013 >10-15/year

September 22, 2014
www.rbmg.ca
25
Use of CAQDAS in Evaluations
Literature Search Summary



Source: EBSCOList - Education Source
Searched 2008-2014: Canadian Journal of
Program Evaluation published 211 Articles
Search String: CAQDAS products in Full Text,
English, PubYear 2008 to 2013.
 No (0) references to CAQDAS products;
 3 references to SPSS, SAS and Stata.
September 22, 2014
www.rbmg.ca
26
Use of CAQDAS in Evaluations
Literature Search Summary
Use of CAQDAS evidenced in Journal Articles:
1. Evaluation and Program Planning = 15
2. Assessment & Evaluation in Higher Educ. = 14
3. Evaluation = 9
4. Studies in Educational Evaluation = 7
5. American Journal of Evaluation = 7
6. Evaluation and the Health Professions = 4
7. Educational Research and Evaluation = 3
8. Journal of Evaluation in Clinical Practice = 3
9. Canadian Journal of Program Evaluation = 0

September 22, 2014
www.rbmg.ca
27
Use of CAQDAS in Evaluations
The Demonstration Effect Using Atlas.ti
September 22, 2014
www.rbmg.ca
28
Use of CAQDAS in Evaluations
The Demonstration Effect Using Atlas.ti
September 22, 2014
www.rbmg.ca
29
Use of CAQDAS in Evaluations
The Demonstration Effect Using Atlas.ti
September 22, 2014
www.rbmg.ca
30
Use of CAQDAS in Evaluations
The Demonstration Effect Using Atlas.ti
September 22, 2014
www.rbmg.ca
31
Use of CAQDAS in Evaluations
The Demonstration Effect – How to:
1. Create an electronic evaluation database using
secondary and primary data files;
2. Manage the evaluation database to reflect the lines of
evidence;
3. Create a closed coding structure based on the
Evaluation Matrix;
4. Search for and code text segments as evidence;
5. View and analyse the evidence;
6. Record the evaluation findings; and
7. Use data visualisation to demonstrate the evidence base
for your findings.
September 22, 2014
www.rbmg.ca
32
Primary documents (PD)
are secondary reference
documents and primary
evaluation data products
loaded into the Evaluation’s
Hermeneutic Unit (HU)
September 22, 2014
www.rbmg.ca
33
Evaluation database of
primary documents
available in My Library for
searching and coding
Primary documents
grouped by
lines of evidence
September 22, 2014
www.rbmg.ca
34
Search only one line of
evidence at a time
September 22, 2014
www.rbmg.ca
35
Derive from
the evaluation
framework
(deductively
created) or
emerge from
the text
(inductively
created).
September 22, 2014
www.rbmg.ca
36
Use the Evaluation
Framework/Matrix as
the basis for the closed
coding structure
Code by
evaluation
question
Or, code by
evaluation
Indicator
September 22, 2014
www.rbmg.ca
37
Use the
indicators to
comment and
define the
code when
using the
evaluation
question
September 22, 2014
www.rbmg.ca
38
This is the closed
coding structure
based on the TBS
Evaluation Criteria
plus other issues of
interest to the Dept.
September 22, 2014
www.rbmg.ca
39
Free/open codes of
interest can be added to
any evaluation issue
during the coding process
September 22, 2014
www.rbmg.ca
40
These are all open
codes of interest,
some assigned to
specific evaluation
criteria and others
floating
September 22, 2014
www.rbmg.ca
41
Segments of
text selected by
the evaluator to
be included in
the empirical
evidence base
September 22, 2014
www.rbmg.ca
42
Search &
code the
evidence
September 22, 2014
www.rbmg.ca
43
Quotations are
linked to codes
which are shown
in the margin of
the primary
source document
Quotations are always
shown in situ, i.e. where
they appear in the
primary document
September 22, 2014
www.rbmg.ca
Example of
text segment
assigned an
open code
44
List and review
all quotations for
any given code
and view in situ.
September 22, 2014
www.rbmg.ca
45
Quotations identified by
unique ID shown in
“Quotations Manager” with
evaluators’ description
and/or commentary among
team members
September 22, 2014
www.rbmg.ca
46
Graphical
representation of
the links between
codes and
quotations based
on the evaluators’
interpretation of
the evidence base
September 22, 2014
www.rbmg.ca
47
September 22, 2014
www.rbmg.ca
48
Visualise your evidence
base for any code within
the network
September 22, 2014
www.rbmg.ca
49
Compare your
evidence and
group text
segments based
on similar topics,
views, etc. and
assign
a label
September 22, 2014
www.rbmg.ca
50
September 22, 2014
www.rbmg.ca
43 text segments, i.e. bits of
evidence have been compared,
grouped and labelled creating new
nodes in the network; relations
among them and with the primary
code have been assigned;
visualization of the data improves
analysis and reporting.
51
Quotation-to-quotation
network (hyperlinks)
showing semantic relations
between quotations
September 22, 2014
www.rbmg.ca
52
Inferential analysis
generates a simplified
network; visualization
of the data improves
analysis and reporting.
September 22, 2014
www.rbmg.ca
53
Sort by lines of
evidence, group
similar interviewee
responses, analyse
and assign a label
to each group.
September 22, 2014
www.rbmg.ca
54
Inductive analysis
generates new
network nodes.
Semantic relations
with the primary
code are then
determined.
September 22, 2014
www.rbmg.ca
55
All three codes and related nodes
are merged into one graphic
display of the evidence for the
Evaluation Issue; visualization of
the data improves analysis and
reporting.
September 22, 2014
www.rbmg.ca
56
Memos have a variety of uses
depending on whether they are
linked to quotations, codes, or
other memos.
They can be written to comment
and reflect on quotations;
describe and summarise code
specific data; or, analyse,
interpret, integrate and generally
report on an Evaluation Issue.
September 22, 2014
www.rbmg.ca
57
Create a memo in the
‘Memo Manager’ for
each Evaluation Issue
Use this memo space to
document the evaluation
findings during the data
analysis and visualisation
process, ensuring that the
narrative reflects the graphic
display of the evidence.
September 22, 2014
www.rbmg.ca
58
When memos are systematically
linked to codes and supporting
quotations, it follows that the
reflections, analyses, and
interpretations contained in them
are grounded in the evaluation
evidence.
September 22, 2014
www.rbmg.ca
59
Demonstrating the evidence base
for any Evaluation Issue, related
codes or nodes is then, literally,
as easy as pushing a button.
Quotations can be reviewed ,
scrutinised , delinked and
categorised differently during the
QA/QC phase of the evaluation.
September 22, 2014
www.rbmg.ca
60
Generating an Evidence Binder is
also, literally, as easy as pushing
a button and easily convertible to
an MS Word document.
September 22, 2014
www.rbmg.ca
61
Can You Relate … Now?
September 22, 2014
www.rbmg.ca
62
The Art, Science and Technology of
Qualitative Data Analysis
Thank You
by Werner Meier
http://www.RBMG.ca
June 16, 2014