Enriched Instructor’s Manual NETA ENGAGEMENT to accompany Prepared by

Transcription

NETA ENGAGEMENT
Enriched Instructor’s Manual
to accompany
Prepared by
Timothy J. Haney
Mount Royal University
Copyright © 2014 by Nelson Education Ltd.
Page 1
NETA ENGAGEMENT: Enriched Instructor’s Manual to accompany
Fundamentals of Social Research, Third Canadian Edition
By Earl Babbie and Lucia Benaquisto
Copy Editor: June Trusty
Tanya Noel, Tamara Kelly, and Julie Clark of York University developed the pedagogical
model for Nelson Education’s Enriched Instructor Manual.
Contained on IRCD ISBN 0176616853
COPYRIGHT © 2014 by Nelson Education Ltd. Nelson is a registered trademark used herein
under licence. All rights reserved.
For more information, contact Nelson, 1120 Birchmount Road, Toronto, ON M1K 5G4. Or you
can visit our Internet site at www.nelson.com.
ALL RIGHTS RESERVED. No part of this work covered by the copyright hereon may be
reproduced or used in any form or by any means—graphic, electronic, or mechanical, including
photocopying, recording, taping, Web distribution or information storage and retrieval
systems—without the written permission of the publisher.
Page 2
Table of Contents
Preface to the Enriched Instructor’s Manual ......................................................................... 4
About NETA .................................................................................................................. 4
The Enriched Instructor’s Manual ................................................................................ 6
Additional Teaching Resources .................................................................................... 8
Introduction and Overview ................................................................................................... 9
Chapter 1 Human Inquiry and Science ............................................................................. 11
Chapter 2 Paradigms, Theory, and Research .................................................................... 18
Chapter 3 Ethical Issues for Social Researchers ................................................................ 23
Chapter 4 Research Design and the Logic of Causation ................................................... 27
Chapter 5 Conceptualization, Operationalization, and Measurement ............................ 32
Chapter 6 The Logic of Sampling ...................................................................................... 35
Chapter 7 Experiments ..................................................................................................... 39
Chapter 8 Survey Research ............................................................................................... 43
Chapter 9 Unobtrusive Research ...................................................................................... 49
Chapter 10 Field Research .................................................................................................. 53
Chapter 11 Qualitative Interviewing .................................................................................. 57
Chapter 12 Evaluation Research ........................................................................................ 61
Chapter 13 Qualitative Data Analysis ................................................................................ 64
Chapter 14 Quantitative Data Analysis .............................................................................. 68
Chapter 15 The Logic of Multivariate Analysis ................................................................... 72
Chapter 16 Social Statistics ................................................................................................ 77
Appendix A Sample Assignment 1: Research Design, Conceptualization,
and Causation.................................................................................................. 81
Appendix B Sample Assignment 2: Sampling and Quantitative Data
Collection and Analysis.................................................................................... 85
Appendix C Sample Assignment 3: Interview Research and Qualitative Analysis............. .. 88
Appendix D Sample Assignment 4: Content Analysis and Discourse Analysis................... .. 91
Page 3
Preface to the
Enriched Instructor’s Manual
About NETA
The Nelson Education Teaching Advantage (NETA) program delivers research-based instructor
resources that promote student engagement and higher-order thinking to enable the success of
Canadian students and educators.
Instructors today face many challenges. Resources are limited, time is scarce, and a new kind of
student has emerged: one who is juggling school with work, has gaps in his or her basic
knowledge, and is immersed in technology in a way that has led to a completely new style of
learning. In response, Nelson Education has gathered a group of dedicated instructors to advise
us on the creation of richer and more flexible ancillaries that respond to the needs of today’s
teaching environments.
The members of our editorial advisory board have experience across a variety of disciplines and
are recognized for their commitment to teaching. They include
Norman Althouse, Haskayne School of Business, University of Calgary
Brenda Chant-Smith, Department of Psychology, Trent University
Scott Follows, Manning School of Business Administration, Acadia University
Jon Houseman, Department of Biology, University of Ottawa
Glen Loppnow, Department of Chemistry, University of Alberta
Tanya Noel, Department of Biology, York University
Gary Poole, Director, Centre for Teaching and Academic Growth
and School of Population and Public Health, University of British Columbia
Dan Pratt, Department of Educational Studies, University of British Columbia
Mercedes Rowinsky-Geurts, Department of Languages and Literatures, Wilfrid Laurier
University
David DiBattista, Department of Psychology, Brock University
Roger Fisher, Ph.D.
Page 4
In consultation with the editorial advisory board, Nelson Education has completely rethought
the structure, approaches, and formats of our key textbook ancillaries. We’ve also increased
our investment in editorial support for our ancillary authors. The result is the Nelson Education
Teaching Advantage and its key components: NETA Engagement, NETA Assessment, and NETA
Presentation. Each component includes one or more ancillaries prepared according to our best
practices, and a document explaining the theory behind the practices.
NETA Engagement presents materials that help instructors deliver engaging content and
activities to their classes. Instead of instructor’s manuals that regurgitate chapter outlines and
key terms from the text, NETA Enriched Instructor’s Manuals (EIMs) provide genuine assistance
to teachers. The innovative pedagogical approach for NETA EIMs was developed by Tanya Noel,
Tamara Kelly, and Julie Clark of York University for Nelson’s Biology: Exploring the Diversity of
Life. This framework has proven so successful that it is being applied across a variety of
disciplines. NETA EIMs answer questions like What should students learn?, Why should
students care?, and What are some common student misconceptions and stumbling blocks?
EIMs not only identify the topics that cause students the most difficulty, but also describe
techniques and resources to help students master these concepts. Dr. Roger Fisher’s
Instructor’s Guide to Classroom Engagement (IGCE) accompanies every Enriched Instructor’s
Manual. The principles, research, and implementation of classroom engagement are set out in
detail in this essential document.
NETA Assessment relates to testing materials: not just Nelson’s test banks and computerized
test banks, but also in-text self-tests, study guides and Web quizzes, and homework programs
like CNOW. Under NETA Assessment, Nelson’s authors create multiple-choice questions that
reflect research-based best practices for constructing questions and testing higher-order
thinking. The program was developed by David DiBattista, a 3M National Teaching Fellow
whose recent research as a professor of psychology at Brock University has focused on
multiple-choice testing. All Nelson Test Bank authors receive training at workshops conducted
by Professor DiBattista, as do the copyeditors assigned to each Test Bank. A copy of Multiple
Choice Tests: Getting Beyond Remembering, Professor DiBattista’s guide to writing effective
tests, is included with every Nelson Test Bank/Computerized Test Bank package.
NETA Presentation has been developed to help instructors make the best use of PowerPoint® in
their classrooms. With a clean and uncluttered design developed by Maureen Stone of
StoneSoup Consulting, NETA Presentation features slides with improved readability, more
multimedia and graphic materials, activities to use in class, and tips for instructors on the Notes
page. A copy of NETA Guidelines for Classroom Presentations by Maureen Stone is included
with each set of PowerPoint slides.
Page 5
The Enriched Instructor’s Manual
This Enriched Instructor’s Manual is organized according to the textbook chapters and
addresses eight key educational concerns. It also includes elements of a traditional Instructor’s
Manual, including answers to problems, suggested answers to exercises and cases, etc.
The key educational concerns consist of the following:
1.
2.
3.
4.
5.
6.
7.
8.
Learning Outcomes
Key concepts
Student motivation
Barriers to learning
Engagement strategies
Assessment tools
Reflections on teaching
Additional resources
Each of these educational concerns is described in more detail below.
1. Learning Outcomes: What should students learn?
• The Learning Outcomes reflect back to the textbook’s Learning Outcomes and include
Bloom’s Revised Taxonomy descriptors.
2. Key Concepts: How does this chapter connect to the world of practice by (professionals in
the discipline)?
• This section identifies the key or threshold concepts covered in this chapter. Are there
concepts that are key to new directions in the discipline? Concepts that help the student
understand how the discipline works?
3. Student Motivation: Why should students care?
• Here, the guide underlines relevance for the student’s studies. Are there topics that
relate to earlier chapters or are the underpinnings for a future concept? Is there
relevance to everyday life or to a global issue?
• This section will highlight the ways that the topic is, in itself, engaging for students.
4. Barriers to Learning: What are some common student misconceptions and stumbling
blocks?
• This section identifies common misconceptions or difficult topics and helps instructors
to address them explicitly in lectures, through out-of-class work, and with in-class
activities (see below).
• Where the textbook takes on these misconceptions or helps to parse out difficult
concepts, references to particular pages or features in the book are provided.
Page 6
5. Engagement Strategies: What can I do in class?
• “What can I do in class?” comprises two sections:
a. Ways of engaging students at the outset. What is the first thing to do in class to
engage students? How should the instructor spend the first 10 minutes?
b. Other activities to connect and bridge concepts, to reveal misconceptions, to
further understanding of key concepts, etc. These activities may enhance
lectures or replace them.
• In each section, suggestions cover a broad range of in-class engagement activities: e.g.,
those that don’t require any extra work on the part of the instructor, and those that
require more instructor intervention; those that work best in small classes, and those
that work with larger groups.
• Activities that address particular student stumbling blocks or undermine misconceptions
are specifically identified. In each case, the Learning Outcome of the activity is flagged.
• Activities will vary by chapter, but may include
o Making explicit real-world links
o One-minute paper
o Think–pair–share
o Lecture boosters
o Cases and problem-based learning
o Small or large group discussions
o Clicker questions
o Collaborative team projects
o YouTube or other video clips and discussion
6. Assessment Tools: How will I know that my students have learned the Learning
Outcomes?
• This section includes references to other Nelson resources (e.g., test banks, CNOW,
TurningPoint questions, Web quizzes, etc.) that help assess student learning, as well as
suggestions for other tools and activities that can be used to assess students (e.g., a
one-minute essay to summarize what students have learned in class).
7. Reflections on Teaching: How can I assess my own “performance”?
o This section includes a checklist to encourage instructors to do self-assessment.
8. Additional Resources: What other resources are available?
• Resources include websites and articles on chapter-specific topics as well as additional
teaching support resources.
Page 7
Additional Teaching Resources
Additional teaching aids are available from Nelson Education in support of this textbook for
teaching social research methods in Canadian colleges and universities.
The following supplements are available on the Instructor’s Resource CD (ISBN 0176616853)
and the password-protected Faculty Resources Web page at www.nelson.com/babbie3Ce:
• NETA Assessment: The Test Bank was written by Andrea Kalmin of York University. It
includes over 600 multiple-choice questions written according to NETA guidelines for
effective construction and development of higher-order questions. Also included are more
than 300 true/false and more than 100 essay questions. Test Bank files are provided in
Word format for easy editing and in PDF format for convenient printing, whatever your
system.
The Computerized Test Bank by ExamView® includes all of the questions from the Test
Bank. The easy-to-use ExamView software is compatible with Microsoft Windows and Mac.
Create tests by selecting questions from the question bank, modifying these questions as
desired, and adding new questions that you write yourself. You can administer quizzes
online and export tests to WebCT, Blackboard, and other formats.
• NETA Presentation: Microsoft® PowerPoint® lecture slides for every chapter have been
created by Donald Swenson of Mount Royal University. An average of 60 slides per chapter
have been provided, many featuring key figures, tables, and photographs from
Fundamentals of Social Research, third Canadian edition.
• Image Library: This resource consists of digital copies of figures, short tables, and
photographs used in the book. Instructors may use these images to create their own
PowerPoint presentations.
• DayOne: Day One—Prof InClass is a PowerPoint presentation that you can customize to
orient your students to the class and their textbook at the beginning of the course.
Page 8
Introduction and Overview
Research methodology courses have the unfortunate reputation in most colleges and universities
for being dull, uninteresting, and unreasonably demanding. However, with the use of relevant
classroom examples, assignments that ask students to apply what they have learned in a
meaningful way, and a well-written text such as Fundamentals of Social Research, these courses
can be enjoyable and rewarding experiences for students. My own students often comment that
they anticipated the worst, but in the end, developed a love of methodology. For this reason, I am
excited to write the instructor’s manual for the third edition of Babbie and Benaquisto’s excellent
text. In this manual, I will outline many solutions to common problems encountered in methods
courses, to assist you to design your course in a way that will help students to learn and apply the
concepts presented in the text in a meaningful way.
In this manual, I discuss the key objectives, importance, and misconceptions about each
chapter in the text. Perhaps most importantly, I also provide examples of activities and
assignments that will elucidate the ideas that Babbie and Benaquisto present to students. Finally,
I provide information about resources, including films and websites, to both enhance and
augment the material in the text. I also connect the book’s learning objectives to the steps
presented in Bloom’s Taxonomy: Remember, Understand, Apply, Analyze, Evaluate, and Create,
with special attention paid to those near the top of the taxonomy—analysis, evaluation, and
creation.
Before presenting these suggestions and resources, however, it may be useful to discuss how
I approach teaching my own research methods course.
A Learn-By-Doing Approach to Teaching Research Methods
I have long believed that students should learn research methods by doing research, not simply
by answering questions about how social scientists conduct research. This view is consistent with
Nelson Education’s focus on active, rather than passive, learning (Fisher, Instructor’s Guide to
Classroom Engagement, page 9).
In my own course, I ask students to complete four assignments (see Appendixes A–D to this
manual), as well as one final exam. In Assignment 1, they are asked to pick a topic by
connecting two concepts from a list that I provide them (e.g., family income and educational
success). This is the topic that they will carry through the semester and use (in one form or
another) in all four assignments. In Assignment 2, students are asked to describe how they would
draw a sample from their target population. They construct a questionnaire and administer it to
the class. Students then enter data, generate simple bivariate tables, and draw some preliminary
conclusions based on their classroom data.
In Assignment 3, students generate an interview guide/schedule and then interview two of
their classmates about their selected topic. They transcribe and code their data before performing
qualitative data analysis. Finally, in Assignment 4, students perform unobtrusive research (both a
content analysis and discourse analysis) on some form of communication (e.g., television,
printed advertisements). In the content analysis, they come up with a coding scheme ahead of
time and they record occurrences of some phenomena using quantitative data collection methods.
Page 9
In the discourse analysis, they view the same form of communication, but instead using
qualitative methods, attempting to discern both manifest and latent content while deconstructing
assumptions.
In all, I have found these exercises to be useful and they help students to see how the ideas
presented in the Fundamentals of Social Research can be used to actually conduct their own
research. Student response to these assignments has been overwhelmingly positive. In my course
evaluations, students say things such as “I liked the format of the course (4 assignments, 1 final,
etc.)” and “I like the high weighting on assignments [sic] for a course like this…. I think you
learn better that way.” Although such an approach to the course requires a higher level of
commitment from the instructor (and a larger marking load), students seem to relish the
opportunity to put the course material to use immediately. This approach also helps students to
make the transition from passive receptors of knowledge to active creators of knowledge.
Concluding Remarks
I hope this manual will help you to adopt some new and creative ideas for your research methods
course. In the meantime, please do not hesitate to contact me if you have any questions about the
material provided here. I am confident that with many of the suggestions contained in this
manual, as well as the excellent material discussed in Fundamentals of Social Research, research
methods can be a rewarding, enjoyable, and useful course for both students and instructors.
Timothy J. Haney, Ph.D.
Department of Sociology and Anthropology
Mount Royal University
Calgary, Alberta
(403) 440-8659
E-mail:
[email protected]
Website: www.timhaneyphd.com
Page 10
CHAPTER 1
HUMAN INQUIRY AND SCIENCE
Chapter Summary
Chapter 1 presents the theoretical and epistemological underpinnings of social science inquiry,
addressing the timeless question: How have we come to know what we know? It does so by
covering a bit of social science history, including the difference between the premodern, modern,
and postmodern views of reality. Importantly, this first chapter discusses the focus of
sociologists on aggregates, or patterns, rather than on individual cases. Finally, it introduces
students to the language and logic of variables, types of explanation, and a brief introduction to
the ethics of social science research.
Essential Outcome: If nothing else, students should learn. . . .
Students reading this chapter should, at the very least, understand that there is significant debate
about what is “real” and what constitutes “knowledge” about the world. They should be aware
that this understanding has changed over time, is still in flux, and that not all social scientists
agree on what constitutes reality. They should also learn the importance of studying aggregates,
rather than solely individual cases. Finally, they should understand the dialectical relationship
between theory and data, known to social scientists as inductive and deductive reasoning.
Learning Outcomes: What should students learn?
After completing this chapter, students should be able to
1. Understand the various sources of knowledge, including tradition and authority, as well as
the potential sources of error in inquiry (inaccurate observations, overgeneralization,
selective observation, illogical reasoning). [Understand]
2. Contrast the premodern, modern, and postmodern views on what constitutes reality.
[Analyze]
3. Consider the difference between examining individual cases and aggregate patterns, and the
benefits and costs of both approaches. [Analyze]
4. Comprehend the use of variables (and their attributes) in social science research, including
the cross-tabulation of two dichotomous variables (presented in Figure 1-5) and the
difference between independent (cause) and dependent (effect) variables. Students should
also be able to derive their own examples of variables and attributes. [Create]
5. Examine the difference between induction and deduction, as well as the relationship between
the two. [Analyze]
Page 11
6. Understand the difference between quantitative and qualitative data, as well as the questions
that can be answered from either type of data. [Understand]
7. Consider the basic precepts of research ethics, including the importance of minimizing harm
to participants and of voluntary participation. Since everything we do in life could possibly
harm someone, all researchers must make value judgments, particularly about the degree of
harm to participants that should be allowed. [Evaluate]
8. Discuss why social scientists conduct research and discuss the competing goals and
objectives of social science research. [Evaluate]
Key Concepts: How does this chapter connect to the world of practice by social scientists?
This chapter introduces students to the foundational debates and issues related to social science
research. More important, students are asked to consider why we, as social scientists, do the
research that we do. In doing so, the chapter teaches students that our work is part of a historical
trajectory and is intricately connected to larger bodies of literature and theory. In short, the
chapter helps students emplace themselves within the social science enterprise.
Student Motivation: Why should students care?
Without the perspectives provided in this chapter, students will not be able to question and
critique the information that they are given. When they hear the ubiquitous “Research indicates
that…,” they should maintain a healthy skepticism until it becomes clear who or what was
studied (and how large the sample was), what logical process (induction or deduction) helped the
researcher to arrive at her or his conclusions, and what potential sources of error (such as
overgeneralization or selective observation) may have led to imprecise or problematic findings.
This chapter is the first step in helping students to become critical consumers of information.
Barriers to Learning: What are some common student misconceptions and stumbling
blocks?
1. Although the importance of studying aggregates, or patterns across large groups of people
rather than individual cases, is perhaps taken for granted by professional social scientists,
many students struggle to grasp this understanding. For example, when we discuss research
findings within the sociology of education, my own students will often raise their hands and
say, “But, that’s how my high school was.” Students should therefore consider that
sociologists generally (but not always) concern themselves with patterns, and that it is
common for these patterns to have several exceptions.
2. Another common misconception, especially for students who have a solid grounding in the
scientific method gleaned from the natural sciences, is the idea that an objective reality exists
and that it is the job of scientists to simply report and describe that reality. While many social
Page 12
scientists subscribe to this epistemological framework, many do not. Therefore, students
should consider the ways in which what we take as “real” is indeed shaped and constructed
through the research process.
3. When learning about inductive and deductive logic, students often assume that research must
take either one approach or the other. However, very little social science research is purely
inductive—observations with no pre-existing theory—or purely deductive—derived from
experience or principles with no previous observations or data. Rather, the two poles serve as
ideal types, or heuristic ways of making sense of the world that are probably too ideal or too
pure to reflect reality.
4. Finally, one common misconception among instructors, which is often passed down to
students, is that it is unethical to carry out research that may harm participants. While we all
agree that research should strive to cause no harm to subjects, the Tri-Council Policy
Statement: Ethical Conduct for Research Involving Humans (TCPS 2) (which governs what
is ethical and what is unethical for research in Canada) allows for some degree of harm and
risk. This is a topic that I enjoy discussing with my own students, and below I provide some
examples of the sorts of issues we debate.
Engagement Strategies: What can I do in class and/or online?
Since this chapter presents students with their first exposure to the logic of social research, there
are several possible discussion topics and activities that can be done in class:
1. I always begin the course by welcoming students to a foreign language class. After giving
them a moment to ponder that comment, I explain that social science research has a language
all its own, with terms such as positivism, social desirability bias, ontological, discourse, and
coding. I explain that once students know this basic language, it becomes easy to read
existing research and understand its findings. Moreover, an understanding of the language of
research methods is absolutely required if students hope (as some do) to one day generate
academic knowledge of their own.
2. When discussing sources of knowledge, I ask students to brainstorm (and tell me) what is
problematic about relying on tradition and authority. I ask them to think of examples of
“knowledge” that are passed through tradition and authority but that we know through
science are incorrect. If students struggle, one example of tradition might be that sugar causes
children to be hyperactive. Although this sort of myth is popularly believed, many such as
this one are untrue from a scientific point of view. An example of authority might be the
declaration by a famous physiologist that marijuana is dangerous. Although we are likely to
believe her because of her academic credentials and significant research experience, the
scientific knowledge in the field is, at best, inconclusive (and, actually, a bulk of the research
demonstrates that overdose is highly unlikely with marijuana—less likely indeed than with
alcohol).
As a way of keeping students’ attention, I often toss in examples related to sex as these
examples seem to resonate well with them. In thinking about authority as a source of
Page 13
knowledge, I tell them a story from my time in Grade 6. At that time, I remember a much
older student (Grade 7 or 8) telling my peer group that “masturbation makes you uglier.” My
peer group and I, knowing that we were younger and less educated, believed this student on
the basis of his age (which, along with education, can be a source of authority). Then, I ask
students if this bit of knowledge is in fact true, and most students, of course, argue
vehemently that it is not scientifically valid.
3. In discussing the difference between variables and attributes, I often write several variables
such as “Hair Colour,” “Frequency of Sexual Activity,” and “Political Views” on the board. I
then ask students (as a class) to provide the corresponding attributes. Almost immediately it
becomes apparent that there is no single way to derive attributes for these variables. For
example, frequency of sexual activity could be measured as the number of times a person has
sex in a week, a month, a year, or some other period of time. Students also quickly realize
that we need to further define sexual activity—which acts count and which do not? This
discussion helps lead to the issue of conceptualization and operationalization, which are
discussed in Chapter 4. They also frequently disagree over how to develop attributes for
political views, with some students arguing that “liberal” and “conservative” are specific
enough, and other students arguing that we really should be asking people views on specific
issues (such as gun control or military spending). This exercise helps students appreciate the
enormous complexity involved in designing a research project, and hopefully, it teaches them
that the findings derived from research depend heavily on the variable utilized in the
analysis.
4. It may be useful to ask students in class to consider the difference between independent and
dependent variables. I do this by providing an example of two variables, such as
race/ethnicity and income. Students are asked to consider which variable in this relationship
is likely to be the independent and which is likely to be the dependent variable. Since the
cause has to precede the effect, there is no logical way in which income can cause one’s race
or ethnicity (although it may have some nuanced effect on how a person self-identifies). I
then provide other examples, asking students to discuss why a particular variable would need
to be (or could not be) the dependent variable in the relationship.
5. One of the key points of the chapter is that sociologists study aggregates. As discussed
above, this is a difficult concept for many students to grasp. To help illuminate the idea, I
present students with a photograph of a dark highway, taken with open-exposure
photography. Visible are the lit-up lines of headlights, surrounded by blackness. I tell
students that this photo is a metaphor for the sorts of questions social scientists ask. We
might ask, for instance, why all of the headlights seem to follow the same path. We spend
less time asking why one individual vehicle did not follow the same path, instead driving far
off to one side. Although we might learn a great deal about the psychology of deviance by
studying that driver, it is more instructive to ask why all cars are following the same path.
Students respond that the lines are painted for them, so drivers follow the painted lanes.
In this context, I refer to the structure versus agency debate in sociology and discuss how the
lanes are one iteration of structure. While drivers have the ability to drive wherever on the
road they wish (on the shoulder, for instance), the structure that we have created ensures that
most drivers conform and stay within the lanes. Similarly, a sociologist may ask why, time
Page 14
and time again, people who pass in the hallway, always seem to pass one another on the
right-hand side of the hallway. In the aggregate, this pattern is very clear. Are there
individual deviations? Yes. And, it may be instructive to ask why these deviations happen.
But, sociologists are most concerned about how and why patterns develop and repeat
themselves over time.
6. In terms of the potential harm to research participants, I find it useful to locate the passages
in the Tri-Council Policy Statement that discuss harm, read them to students, and then ask
them if certain research projects (that I have made up) should be allowed. The TCPS 2 states
that “And while researchers should attempt to estimate the occurrence of the relevant harms,
this may be more difficult, or not possible, for new or emerging areas of research where no
prior experience, comparable research or publications exist” (p. 23). It goes on, stating, “In
their review, REBs (Research Ethics Boards) should be concerned with an assessment that
the potential research outcomes and potential benefits merit the risks” (p. 24). In other words,
risks to participants are allowable if they are indeed unavoidable and if the benefits of the
research outweigh the risks.
Having considered these points, I like to provide my students with some potential
scenarios. For example, they are asked to consider a study in which participants are asked
about the deaths of their parents or grandparents in order to determine whether those born in
Canada are less likely to cry (i.e., more stoic) than those born elsewhere. This study clearly
has some risk of causing emotional harm, but has a comparatively low benefit to the
participants or society (since it may or may not teach us about how different cultures deal
with sadness). Alternatively, I then ask them to consider pharmaceutical research. It is
common at medical schools in Canada for researchers to conduct studies on unproven drugs
that may indeed result in the deaths of some study participants. This is an enormous risk.
However, students are quick to point out that, although the potential for harm is great, the
potential benefit is great—both to research participants, who may be cured of a particular
ailment, and to society. Given that research ethics are discussed in greater detail in Chapter 3,
this discussion serves to capture the interest of students, but does not need to reach any
definitive conclusions.
7. Finally, I use this first chapter as a springboard to discuss with students why we do research.
This serves as the connection to the next chapter, where we spend more time discussing
positivism and competing intellectual traditions. In order to discuss the purpose of research, I
first introduce them to some of the ideas of Talcott Parsons, famous for his contributions to
functionalist theory. Parsons argued in The Structure of Social Action (1937) that the task of
social science was to illuminate the social world, much like a spotlight illuminates a dark
field or parking lot. We can begin to uncover “social facts” (as he called them), such as the
relationship between race and income. Eventually, with enough research, we might
illuminate the entire field. In other words, for Parsons, research was about uncovering
existing, external truths. (Implying value-neutrality on the part of the researcher.)
I then present students with two excerpts from critical educator Paolo Freire, who
famously wrote in Pedagogy of the Oppressed (1970) that “Washing one’s hands of the
conflict between the powerful and the powerless means to side with the powerful, not to be
neutral.” In other words, researchers and educators have a moral imperative not to simply
Page 15
study or learn about social problems and issues, but to actively work to correct them. This is
a controversial view, but one that generates substantial discussion among students. Yet, I
believe this is an important discussion to have in a research methodology course. Not only
should students learn how to do research, but they should be asked to consider why they are
conducting research and toward what ends their research will lead. In short, you can ask your
students:
Question: Should the goal of our research be the neutral reporting of results or changing
those facets of the world that we find problematic?
Assessment Tools: How will I know that my students have learned the Learning Outcomes?
As I mentioned in the Introduction, I have students complete four assignments in the course that
advance the same research topic through various stages of research design. In the first
assignment (see Assignment 1 in Appendix A), I ask them to complete a section on their
epistemological foundations. Specifically, I say:
Imagine you are going out into the field to speak with people and to attempt to answer
your questions. You will be surveying or interviewing people (right now it doesn’t matter
which one). It’s important to first consider issues of what you can observe, what is real,
and the political implications of your research. To do so, you’ll need to answer the
following questions. You do not need to label them as I do below (1, 2, etc.), but you will
need to make sure that each one is answered, and it will be helpful if you would address
them in roughly the same order:
1. What sort of epistemological approach best suits your research question?
2. Would this research project necessarily be positivist? Why or why not?
3. Would this research project be empirical in nature? Why or why not?
4. Could you undertake this research in a value-free way? Is that possible? Why or
why not? If not, what sorts of values, biases, or preconceptions may affect this
research? And, how might they affect it? Would you take a position of conscious
partiality? Why or why not?”
The answers to these questions incorporate knowledge that students gain from both Chapter 1
and Chapter 2 of the text, but I have found that they really make students think about how and
why they are conducting research. Indeed, my colleagues often comment that they hear students
wandering around our department suite mumbling about epistemology. While the questions
clearly make students think, I have also been surprised by the depth of thought and erudition that
I read in their assignments.
Page 16
Reflections on Teaching: How can I assess my own performance?
Because this is the first chapter of the textbook, it is not meant to provide students with definitive
answers to large epistemological questions. Rather, it is meant to introduce them to the practice
of social science research. As such, at this point in the semester, I am content if students are
asking big questions, such as “Is it irresponsible to do research that begins with the assumption
that gender inequality is a problem? Can we ever be totally free of biases? Would inductive or
deductive research give better results?” It is my hope that they will use the semester to try to
answer these questions, although I always warn students that these are things professional social
scientists spend years debating.
Additional Resources
1.
A visual representation of variables and attributes (as well as concepts and indicators) from
Caroline Persell’s Introduction to Sociology course at New York University, Sociology
Department, “Understanding Concepts, Variables, and Attributes.” 2003:
www.nyu.edu/classes/persell/aIntroNSF/Documents/ConceptsVariablesIndicators.html
2. YourTeacher.com, “Independent and Dependent Variables”:
www.youtube.com/watch?v=utNpSEEyMIU
3. Two books that give critical views on the creation and dissemination of knowledge:
Freire, Paolo. Pedagogy of the Oppressed. New York: Continuum. 1970.
Brown, Leslie, and Susan Strega, eds. Research as Resistance: Critical, Indigenous, and
Anti-Oppressive Approaches. Toronto: Canadian Scholars’ Press. 2005.
Page 17
CHAPTER 2
PARADIGMS, THEORY, AND RESEARCH
Chapter Summary
The second chapter distinguishes between common paradigms used in social science research
and more specific theoretical frameworks employed. It contrasts inductive and deductive theory
construction, before discussing the links between theory and research.
Students should be aware of the paradigms that we, as researchers, are operating within. Further,
students should understand how specific theoretical schools flow from the assumptions made by
particular paradigms. Finally, they should understand the process in which social scientists
engage as they move from conceptualization to operationalization to hypothesis construction.
1. Understand positivism as a paradigm, as well as its critiques. [Understand]
2. Explain the conflict, symbolic interactionist, feminist, and ethnomethodological
paradigms (which may be review from an introductory sociology course for many
students) and be able to conceive of research projects that flow from each paradigm.
[Apply]
3. Consider how theory leads to operationalization and, eventually, hypothesis construction.
[Understand]
4. Differentiate between well-written, testable hypothesis and those that are untestable.
[Evaluate]
5. Construct their own testable hypotheses. [Create]
The content presented in this chapter demonstrates the kinds of decisions (conceptualization,
operationalization, hypothesis construction) that take place before most research begins. Further,
it shows students how a researcher’s theoretical or epistemological foundations shape the
questions she or he asks, the expectations they maintain, and the ways in which they define and
measure the variables that they use in their analyses.
Page 18
Each paradigm has a built-in set of assumptions. It is impossible for students (or anyone else, for
that matter) to understand how a conclusion was reached without considering the paradigm from
which the authors were working. Once again, this chapter helps students to be more critical
consumers of research. It will also help them to examine the paradigms from which they are
working, as well as the assumptions that they are making. This awareness is a critical skill for
researchers.
Barriers to Learning: What are common student misconceptions and stumbling blocks?
Several core ideas are often difficult for my students to grasp, including the following:
1. Students often assume that theory is completely distinct from method, as the two are
normally offered in separate courses. However, this chapter makes it clear that people’s
theoretical perspectives influence their choice of method. This is especially true in
traditions that blend theory and method, such as ethnomethodology. My students are
often perplexed by the conceptual overlap between theory and method, as they have often
learned to compartmentalize and to see issues in either black or white.
2. Students also frequently come into this course assuming positivism is the only paradigm
adopted by social science researchers. In other words, they assume that there is an
external reality and that, using the methods of science, any observer can understand that
reality. This is, I think, attributable to the focus that high school science curricula
dedicate to studying the scientific method. Through discussing positivism (and its many
critiques, especially those from feminist theory) students begin to appreciate the strengths
of more constructionist ways of understanding the world. This is always a fruitful class
discussion, as I discuss with students how positivism is or was the hegemonic way of
imagining the world of knowledge. We also discuss the various facets of positivism,
including empiricism, utilizing the tools and language of science, value-freedom, and the
development of laws that can be transferred to many or all similar situations. I then
provide students with a fairly controversial research topic such as “women’s experiences
of sexual violence” and ask them if research can and should pursue this topic from a
positivist epistemological framework. The discussion helps students see that there is no
one “right” or “wrong” answer to this question, as many of their classmates will likely
disagree with their position.
3. Quite often, my students also assume that there is only one way to operationalize a
concept. For example, they assume that income must be in dollars earned per year. I then
pose several questions to them that muddy the water significantly, including
a.
b.
c.
d.
Why couldn’t we measure income as hourly wage?
In dollars per week?
Are we asking about before-tax or after-tax income?
Is nonstandard (off-the-books) employment counted?
Page 19
As you can see, there is no one single, intuitive way to measure income. I hope to impress
on students that operationalization decisions such as this must be understood if we are to
make sense of the conclusions that arise from research.
4. Finally, students often struggle in writing hypotheses. They often write double-barrelled
hypotheses that are nonfalsifiable or nontestable, such as “Women will have higher
grades than men and will also be more likely to get into graduate school.” Because this
hypothesis incorporates the relationship between three variables, it becomes untestable.
What if women do earn better grades but are not more likely to get into graduate school?
Have we supported the hypothesis or not? I normally take an opportunity to discuss with
students the logic of falsifiability and the ideas of social philosophers such as Karl Popper
and Thomas Kuhn. Students will also frequently write very vague hypotheses. In their
first assignment, which focuses on research design, students sometimes (even after our
conversations about this) write hypotheses such as “Income will affect political beliefs.”
While this may be true, it is more instructive to utilize the attributes of these variables
when writing a hypothesis. A better alternative might be “High-income individuals are
more likely to hold conservative political beliefs than low-income individuals.” This
hypothesis is better as it specifies a direction of the relationship, while also comparing
two groups (by use of than).
Several in-class strategies can be used to elucidate the key paradigms used in social science
research:
1. To help students think more clearly about the different paradigms used in social
science research, I often choose a topic (e.g., white-collar crime) and ask students
how a researcher from each of those paradigms might think about or research that
topic. For example, a positivist would assume that a certain, measurable amount of
white-collar crime exists and that it is possible, given the right methods and the right
access to a particular research setting, to uncover the reality of white-collar crime that
is normally hidden from view. By contrast, someone working from a conflict
paradigm might focus on how the wealthy use their advantage (such as working in
offices with large sums of money) to gain wealth at the expense of the poor or
middle-class. This exercise helps students to recognize the competing perspectives
used to explain any single phenomenon.
2. I also enjoy providing two concepts to the class (income and educational attainment,
for instance). Students then operationalize those two concepts and write a hypothesis
about the relationship between the variables they have come up with. I then ask each
group to write their hypothesis on the board. Each time, I am amazed by the diversity
of hypotheses that the class derives. It is immediately apparent that there is no one
single way to operationalize these concepts, nor is there one single way to
hypothesize about the relationship between them. This exercise also presents students
with an opportunity to critique each other’s hypotheses.
Page 20
3. For more practice operationalizing, I provide students with a concept such as “liberal
political views” and ask them to develop questions that could be asked of participants
to measure this concept. Some students ask participants whom they voted for. Others
want to ask about specific issues such as whether Canada should have joined its U.S.
allies in the Iraq war. I then point out that many of these questions (such as the
question about the war) are probably indicators of a concept. This often sparks a
debate about whether any one variable can truly capture the full meaning of a broad
concept such as “liberal political views.”
As part of their first course assignment (on research design), I ask students to pick two concepts
that they want to explore (e.g., views on the oil and gas industry in Alberta and family income).
They generate a research question based on these concepts, discuss which paradigm is best
suited for viewing their research, derive variables and attributes, operationalize their variables by
writing down questions that they would ask of research participants, and write testable
hypotheses. An assignment such as this indicates clearly whether students are grasping the
process that social scientists engage in when constructing research projects. This also provides a
good opportunity for the instructor to help students bridge the gap between hearing or reading
the material and actually engaging in the process of creating knowledge.
Hypothesis writing is one of the most challenging skills to learn, but also one of the best
indicators of instructor performance. A testable hypothesis requires that the students construct
two variables, with attributes, and arrange those variables into a hypothesis that makes a clear
comparison between two groups or two situations, while avoiding the common problems of
hypothesis writing such as double-barrelled hypotheses. If students are able to construct a
testable hypothesis on their own, they have sufficient mastery of the material.
1. Pffefer, Carla A., and Christabel L. Roglin. “Three Strategies for Teaching Research
Methods: A Case Study.” Teaching Sociology. Vol. 40, No. 4, October 2012, pp. 368–386.
2. Karl Popper was one of the earliest and best-known sociologists of knowledge. His work is
directly applicable to hypothesis writing, and specifically, to the requirement that hypotheses
be falsifiable: Popper, Karl. The Logic of Scientific Discovery. New York: Routledge. 2002.
3. The following text is widely considered to be the best existing treatise on the history of the
(social) scientific enterprise: Kuhn, Thomas. The Structure of Scientific Revolutions (4th
edition). Chicago: University of Chicago Press. 2012.
Page 21
4. Hypothesis writing: William M. K. Trochim, Research Methods Knowledge Base,
“Hypotheses”: www.socialresearchmethods.net/kb/hypothes.php
5. Explanation of positivism and its critiques: William M. K. Trochim, Research Methods
Knowledge Base, “Positivism & Post-Positivism”:
www.socialresearchmethods.net/kb/positvsm.php
Page 22
CHAPTER 3
ETHICAL ISSUES FOR SOCIAL RESEARCHERS
Chapter Summary
This chapter introduces students to the basic ethical precepts that researchers must consider
before undertaking research with human participants. In doing so, students learn some of the best
practices for conducting research, as well as some of the more infamous violations of research
ethics that have happened in the past.
After completing this chapter, students should understand the basic precepts of the Tri-Council
Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2) on research ethics
and should comprehend why human participants must give informed consent before participating
in research.
1. Discuss the TCPS 2, including its purpose and its general principles. [Understand]
2. Understand the basic ethical precepts that are codified in the TCPS 2. These include
respect for human dignity, respect for free and informed consent, respect for vulnerable
persons, respect for privacy and confidentiality, respect for justice and inclusiveness,
balancing harms and benefits, minimizing harm, and maximizing benefit. [Understand]
3. Comprehend several of the key concepts involved in conducting ethical research,
including the difference between anonymity and confidentiality, informed consent,
deception, and debriefing. [Apply]
4. Recall some of examples of violations of the rights of participants, including the Tearoom
Trade study and the obedience study. [Remember]
5. View a proposal for research and locate possible ethical pitfalls, suggesting alternative
methods that avoid these problems. [Evaluate]
6. Always remember that all research involving human subjects must receive ethical
clearance from a college or university’s Research Ethics Board. (REB). [Remember]
7. Design and carry out a research project that avoids common ethical pitfalls and is
acceptable according to TCPS 2 guidelines. [Create]
Page 23
Social scientists inhabit a world informed by the immediate knowledge of several gross
violations of the rights of participants, although many of those violations occurred before
policies such as the TCPS 2 came into existence. Avoiding repeats of many of the ethical
violations committed by our predecessors is a focus of many, if not all, contemporary social
scientists. In fact, in today’s social research culture, it is imperative that researchers understand
ethical precepts and go to great lengths to protect human participants to the greatest degree
possible. Without this understanding, students will be unable to participate in the process of
actively collecting and analyzing data from human participants.
1. This chapter points to one of the key reasons why students should care about research
methods: social scientists and Research Ethics Boards usually consider poorly designed
research to be unethical, as it wastes the time of participants. If participants are losing time
participating in a study that is so poorly designed it is unlikely to contribute any useful
knowledge, then the benefits to society do not outweigh the costs to participants.
2. Quite simply, without consideration of the ethical treatment of human participants, most of
the research that we do in the social sciences is not possible.
3. Understanding the material in this chapter will help students avoid ethical dilemmas that
could tarnish the reputation of the social sciences and make it more difficult for future
researchers to recruit human participants.
1. Students often misunderstand the difference between what is legally allowable and what are
ethical best practices in research. They will sometimes assume that just because a particular
practice (revealing participant names in published research, for instance) is not illegal, it is
also not ethically problematic. I often ask student which is a higher standard: law or ethics.
(For the reason stated above, the answer is ethics). This can create a lively discussion.
2. Students also often misunderstand the difference between anonymity and confidentiality of
participants. Anonymity, of course, is when even the researcher does not know, and cannot
figure out, the identities of participants. Confidentiality is when the researcher knows, or can
figure out, the identities of participants, but does not reveal this information to anyone.
3. Students will often assume that it is not ethically allowable under TCPS 2 guidelines to
deceive human participants. In fact, deception is allowable. For some researchers, especially
psychologists, deception is necessary in some circumstances in order to discern the actual
beliefs or reactions of participants in ways that are unaffected by preconceived notions of
what is actually being researched. Deception, however, requires a debriefing of participants,
Page 24
and researchers must first show that the research cannot be undertaken without such
deception. They also must normally have resources on hand (such as names of psychologists)
in the rare cases where participants are traumatized by the deception.
4. Finally, students will often assume that research cannot cause harm to participants. Much like
deception, research can involve some risk of harm to participants if the benefits of the
research (either to participants, to society, or both) are potentially great. Here I often use the
example of pharmaceutical trials. In many such trials, death is a very real risk (along with
other serious side effects). I ask students why this research is allowed to progress, and they
answer that the drugs may help the participants with an otherwise incurable disease, and the
potential benefits to society are great if the drug is effective. We then discuss the ways in
which researchers must demonstrate the potential harms and the potential benefits to the
Research Ethics Board, and the board decides if the risk/benefit equation is balanced enough
to allow the research to proceed.
1. Instructors can show film examples of some of the most ethically problematic research that
has occurred. Examples include the Milgram obedience experiment, the Zimbardo Stanford
prison study, and Laud Humphrey’s Tearoom Trade study. After showing one of these
examples in class, I often ask students why this research was unethical and, more
specifically, what precepts of the TCPS 2 this research violated. I also ask students if there is
a way this research could have been conducted more ethically.
2. My university has a sample consent form for participants. I show students this consent form
and we discuss why each piece of information is included in the form, as well as the risk we
would need to accept by excluding each piece of the consent form.
3. As part of Assignment 2: Sampling, Quantitative Data Collection, and Analysis (see
Appendix B to this manual), I ask students to craft their own consent form, making sure to
include all necessary elements.
4.
As an alternative in-class activity, instructors can put each of the principles of the TCPS
on the board (include respect for human dignity, respect for free and informed consent, respect
for vulnerable persons, respect for privacy and confidentiality, respect for justice and
inclusiveness, balancing harms and benefits, minimizing harm, and maximizing benefit, and so
forth). I then divide students into groups and ask each group to consider one of these key
concepts. I ask them to brainstorm a couple of threats that researchers must consider in each
area, as well as ways in which they can protect human participants. They write these ideas under
the appropriate concept on the board, and we discuss the topic as a full class.
Page 25
1. Besides demonstrating familiarity with key ethical concepts during class meetings, I suggest
asking students to reflect on ethics in the assignments that they submit for the course. In my
own assignment, I ask students to craft a consent form for the interview research that they
will carry out as part of the course, and I also ask them in the assignment to reflect on what
ethical safeguards such research requires.
2. Students could be asked (either in an assignment or on an exam) how research such as the
prison experiment or the obedience study could have been carried out more ethically. This
question asks them to not only be familiar with these violations of research ethics, but also
asks them to apply ethical precepts to these examples, integrating knowledge and practice.
3. As part of the second assignment (see Appendix B) that I give in my course, I ask students to
write a consent form for the research project of their choice (we first discuss our university’s
consent form template in class). If in their own projects, students are able to anticipate
potential ethical problems, they have successfully mastered the material contained in this
chapter.
If students have a solid theoretical grasp of key ethical concepts, such as informed consent, then
teaching has been successful. Beyond that, if students can successfully fill out the research ethics
application at your university, this is yet another measure demonstrating their mastery of the
material.
1. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2):
www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default
2. Online TCPS 2 Tutorial: Course on Research Ethics:
www.pre.ethics.gc.ca/eng/education/tutorial-didacticiel
3. YouTube, “Milgram Obedience Study”: www.youtube.com/watch?v=W147ybOdgpE
4. YouTube, “The Stanford Prison Study”: www.youtube.com/watch?v=sZwfNs1pqG0
5. Humphreys, Laud. Tearoom Trade: Impersonal Sex in Public Places. Aldine Transaction
Publishers. 1975.
6. Another excellent (and bestselling) book on the importance of research ethics, especially
informed consent, although it does relate more strongly to biomedical than to social research:
Skloot, Rebecca. The Immortal Life of Henrietta Lacks. New York: Random House. 2010.
Page 26
CHAPTER 4
RESEARCH DESIGN AND THE LOGIC OF CAUSATION
Chapter Summary
This chapter moves further yet from the theory and hypotheses into the actual process of
conducting research. It does so by describing various models of explanation and causation,
helping students to specify units of analysis, accounting for temporality, and finally, designing
and proposing the research.
Students should at the very least come away with an understanding of the three criteria for
causation (nonspuriousness, temporal ordering, and correlation). They should also understand the
importance of accounting for time through research design and should be able to distinguish
between cross-sectional and longitudinal designs, as well as the advantages and disadvantages of
each.
1. Understand the three purposes of research, including exploration, description, and
explanation. [Understand]
2. Differentiate between nomothetic and ideographic models of explanation. [Analyze]
3. Decide whether existing research (including their own) meets the three criteria that must
be met in order to establish causation: correlation, time order, and nonspuriousness.
[Evaluate]
4. Distinguish between necessary and sufficient conditions for causation. [Analyze]
5. Understand what a unit of analysis is and why it is important. Further, they should
understand the ecological fallacy and its potential for confounding conclusions drawn
from social research. [Understand]
6. Appreciate the difference between cross-sectional and longitudinal research designs, the
latter of which includes trend, cohort, and panel studies. [Understand]
7. Apply the various sequential steps involved in designing a research project, including
conceptualization, choice of method, operationalization, securing of ethics approval,
sampling, observations, data processing, analysis, and finally, application. [Apply]
Page 27
8. Propose their own research projects by applying the steps involved, including budgeting,
scheduling, and identifying a problem or objective in such a way that readers will be
convinced of the project’s importance. [Apply/Create]
This chapter introduces students to the logic of causation. Most (but not all) claims made in
social science research seek to establish causation, but doing so is difficult given the limits of
time, money, and measurement. Therefore, understanding of the concepts and ideas presented in
this chapter is crucial for making convincing arguments about cause and effect.
This chapter will help students to be more critical consumers of research. After reading it, they
should be able to critique existing research and the claims that it advances. For example, they
should be critical of research that makes claims about causation but does so with cross-sectional
data and variables measured contemporaneously. Furthermore, after reading Chapter 3, students
should also understand that poor-quality research is considered unethical (as it wastes the time of
human participants), so making claims about causation without the appropriate data for making
such claims could potentially be considered unethical.
1. Students are often quick to assume that correlation equates to causation—that because two
variables are related, one must cause the other. As most social scientists know, this is not
true.
Example: In explaining this to students, I use the example of ice cream sales and crime:
As ice cream sales increase, so do most types of crime. I then ask if we should outlaw ice
cream sales, and students of course respond “No!” They’re usually quick to point out that
we’ve missed a variable in our causal model—temperature or season affects both ice
cream sales and crime rates. In other words, we have a spurious relationship.
2. Students will also often assume that in order for causation to be established, X must cause Y
in all cases. In other words, every male in the sample would need to have a higher income
than every single female. This is simply not the case. I explain to students that in the social
sciences, we use a probabilistic approach to causation rather than a deterministic approach.
3. Lastly, students are often confused about the meaning of a study’s unit of analysis. As we
know, a researcher can draw conclusions only at the level of data that was analyzed. If we
have data on cities, we cannot draw conclusions about individual behaviour, for example
(doing so might mean committing the ecological fallacy).
Page 28
Example: As an example, we might find out through our research that schools with
lower-paid teachers have higher teacher turnover rates. Students are quick to conclude
that low-paid teachers are more likely to look for other jobs and eventually leave.
However, unless we have data on actual teachers (rather than schools), we cannot
conclude that low-paid teachers are the ones who are leaving. What if the teachers who
leave in the lower-paid schools are actually the higher-paid teachers? Statistically
speaking, it’s possible.
1. First, I like to give students examples of cross-sectional, trend, cohort, and panel studies.
I ask them, for example, why a researcher might want to conduct a panel (or longitudinal)
study rather than a cross-sectional one. This leads to excellent discussion about how a
researcher can capture change in research (and why capturing change is difficult, if not
impossible, in a cross-sectional research design). We also discuss why a cohort study
may be preferable in some circumstances to a repeated cross-sectional approach (again,
because it allows the researcher to see how individuals change over time as their
circumstances change).
2. I also enjoy discussing with students the ecological fallacy, or the notion that our
conclusions must be based on the level of data that we have collected. For example, it
may be that cities in Canada with a higher proportion of foreign-born (or immigrant)
residents have higher crime rates than cities with fewer foreign-born residents. I ask
students what they can conclude from this and usually they say that immigrants commit
more crime (although, as sociologists, they also usually point out that those immigrants
live in a society where their credentials are often not recognized, where it is difficult to
obtain a living-wage job, etc.). I point out to them, however, that our unit of analysis was
cities, not individuals. As a result, it’s problematic to make conclusions about individual
behaviour; even though cities with more immigrants may have higher crime rates, we do
not know who in those cities is committing more crime. It may be that immigrants in
neither type of city are the ones actually committing the crimes. This, we refer to as the
ecological fallacy and it is especially important for students to avoid this common
problem.
3. I enjoy discussing with my students the criteria for causation, especially
nonspuriousness. There are many examples of spuriousness, but the one I often use is the
connection between consumption of bottled water and better fetal health. I give students a
scenario in which public health researchers conclude that women who drink bottled
water (versus tap water) during pregnancy have healthier babies. I then ask them if we
should mandate that all pregnant women consume only bottled water. At first, they say
“Yes” but after some discussion, I ask them what other variables may be affecting both
who drinks bottled water to begin with and also who has healthier babies. Very soon,
students begin to talk about income or social class. Wealthier women are more likely to
buy bottled water, owing to the cost. However they are also more likely to be able to
afford organic foods, to take prenatal classes, to take prenatal vitamins, and so forth. In
Page 29
this example, students see that without the inclusion of this third variable (social class), it
is impossible to conclude anything about the effect of water consumption on fetal health.
4. Finally, I discuss with them the issue of time-ordering. For example, let us say that we
find a statistical correlation between two variables: attitude toward same-sex marriage
and income. It is tempting to conclude that having a higher income influences one’s
attitudes toward same-sex marriage (perhaps having a higher income brings you into
contact with more educated people whose attitudes rub off). However, I ask students to
consider the plausibility of the reverse: What if having a more liberal attitude toward
same-sex marriage introduces you to others who have similar attitudes (“birds of a
feather…”). Those networks could potentially lead to job opportunities that may increase
an individual’s income. The key point here is that we cannot know which variable has an
effect on which variable if they are measured contemporaneously, as in a cross-sectional
study. This is one scenario where a longitudinal research design would be very helpful.
I’ve always felt that the best way to assess student learning is to require them to design their own
research project. As part of Assignment 1 (see Appendix A), I ask students to consider questions
such as their unit of analysis and their research design (cross-sectional, longitudinal, etc.), and
require them to reflect on the advantages and drawbacks of these decisions. If students are able
to apply these concepts to a topic of their choosing, I can be confident that they understand the
concepts.
Probably the best indicator of a teacher’s effectiveness is the students’ ability to apply the
concepts learning in the reading and in class to a question or problem of their choosing. Further,
they should be able to discuss the concepts they have learned in a nuanced and sophisticated
manner. Doing this demonstrates a greater level of comprehension than simply asking students to
regurgitate textbook definitions of particular concepts (an approach that fails to capture higherlevel learning represented near the top of Bloom’s Taxonomy).
1. An excellent discussion of the three criteria for causality: William M. K. Trochim,
“Establishing Cause and Effect,” Research Methods Knowledge Base:
www.socialresearchmethods.net/kb/causeeff.php
2. Another description of the three criteria for causality (along with a discussion of reliability,
validity, and other issues), from the University of South Alabama: R. Burke Johnson,
“Validity of Research Results,” Educational Research:
www.southalabama.edu/coe/bset/johnson/lectures/lec8.htm
Page 30
3. A discussion of the difference between cross-sectional and longitudinal research designs
from the Institute for Work and Health, “Cross-Sectional vs. Longitudinal Studies,” At
Work, No. 55, Winter 2009:
www.iwh.on.ca/wrmb/cross-sectional-vs-longitudinal-studies
4. Charles Schallhorn, “Longitudinal v Cross-Sectional Studies,” YouTube:
www.youtube.com/watch?v=LL2CESAd8KA
5. University of Ottawa, “Idiographic and Nomothetic Approaches in Science,” Scientific
Paradigms in Population Health:
http://courseweb.edteched.uottawa.ca/pop8910/Notes/PhilosNomo.htm
Page 31
CHAPTER 5 CONCEPTUALIZATION, OPERATIONALIZATION, AND
MEASUREMENT
Chapter Summary
This chapter introduces students to the often challenging process of refining an idea through
conceptualization, operationalization, and eventually, measurement. They begin to understand
that there is no single way to conceptualize an idea, nor is there one single way to operationalize
a variable. There are multiple competing ways to do this, leading students to consider the role of
scales and indexes in measuring particular concepts.
Students should come away from this chapter with an understanding of how social scientists
conceptualize, measure variables, and ensure that their variables are valid measures.
1. Use conceptualization to bring their research project from the theory stage to the
measurement stage. [Apply]
2. Understand the role of indicators and dimensions in the conceptualization process.
[Understand]
3. Explain that variables can have different levels of measurement, including nominal,
ordinal, interval, and ratio. [Remember].
4. Determine the level of measurement of a given variable. [Analyze]
5. Understand some common methods for establishing reliability, as well as the different
types of validity. [Understand]
This chapter is especially important for those who do quantitative research, especially research
derived from survey data. Although conceptualization is done by most social scientists, other
ideas discussed here are more suited to particular methodologies. For example, level of
measurement must be known before performing any statistical tests (such as T-tests or
correlation), but is not a concept commonly used in qualitative approaches to research. Similarly,
Page 32
validity and reliability are discussed far more often with quantitative research than with
qualitative (due to their emphasis on measurement and replicability).
Quite simply, many of the statistical procedures that they will have to perform (if their school
requires a statistics course as part of their degree) require them to know the level of measurement
they are employing. For example, nominal-level variables cannot be used in correlation.
Similarly, interval or ratio-level variables cannot be utilized in chi-square tests. Not being aware
of this creates the potential that students will make mistakes in the interpretation of data. Further,
they should realize that there are multiple possible ways for measuring a potential variable, some
of which are more valid than others. This should get students thinking about how they can best
measure the concepts that they are interested in studying.
Since students have not read about survey research yet, they might be confused as to why
measurement matters so much. I explain to students that unless we properly conceptualize and
operationalize, we might end up learning about a concept or idea that we did not intend to
study—learning little about the concept or idea that we did want to learn about. Maintaining
validity (measuring what we think we are measuring) is important for drawing conclusions about
the social world that will be believed and taken seriously by the research community.
1. Students can be asked to create testable research questions, including a big-picture
originating question as well as the more specific research questions that they will aim to
answer. I then ask them to use the paradigms and theories available to social scientists
(e.g., feminist theory or poststructuralist theory) to frame their issue. Students can then be
asked to take the concepts they have decided to use and to generate variables with
attributes (See Assignment 1 in Appendix A).
2. In class, I also give students an idea such as “beliefs about animal rights.” I have them
conceptualize it to clarify that concept. They then operationalize it, making it into a
variable with at least two possible attributes. Through this process it becomes clear that
there is no one single way to conduct this process. Students often disagree about what
“animal rights” means, and will rarely devise the same way to measure and capture (for
instance) people’s attitudes toward the importance of animal rights.
Page 33
After reading this chapter, students should be able to take an idea (such as an opinion on capital
punishment), conceptualize it, operationalize it, and end up with a variable consisting of several
attributes. They may also decide that their concept is broad or multifaceted enough to require
several variables in order to measure it. In this case, they should be able to create the variables
and to discuss how those variables could be combined into an index or scale.
Appendix A at the back of this manual contains the first assignment that I give students in my
research methods course, an assignment on research design, conceptualization, and
operationalization. In it, there are several questions that I ask students to answer as they are
creating a research project of their own. These include “If you were asking questions of
participants (and you will be), how would you measure each of your variables? For example,
what sorts of things would you actually ask people? I also ask them to derive variables and list
their variables and attributes in a table. If students are able to engage in this process of applying
knowledge from the reading to their own projects, the objectives of the chapter have been met.
1. A good resource for understanding operationalization: Shuttleworth, Martyn.
Explorable.com. “Operationalization.” 2008: http://explorable.com/operationalization.html
2. A discussion of the levels of measurement that a variable can have, from West Virginia
University: Duval, Bob. Levels of Measurement:
www.polsci.wvu.edu/duval/ps601/Notes/Levels_of_Measure.html
3. This Internet page contains several video tutorials on levels of measurement: Sophia
Learning LLC. “Concept: Levels of Measurement”:
www.sophia.org/levels-of-measurement-concept
Page 34
CHAPTER 6
THE LOGIC OF SAMPLING
Chapter Summary
This chapter introduces students to the logic and method of sampling, with special attention paid
to probability sampling, random and systematic methods designed to derive representative
samples. It also discusses methods of nonprobability sampling and when those methods might be
more appropriate.
Students should learn why we use probability sampling, including when it is appropriate and
when it is not. They should also learn how to draw a sample in a systematic way, and understand
why most social scientists prefer these sampling methods. Finally, they should learn why larger
sample sizes generate more representative samples.
1. Comprehend some of the significant historical errors in sampling and how those errors
could have been avoided. [Understand]
2.
Understand the various approaches to nonprobability sampling, including purposive
sampling, snowball sampling, and quota sampling. [Understand]
3. Understand the logic of probability sampling and its various approaches, including the
importance of representativeness, random selection, and sampling error. [Understand]
4. Use sampling frames, sampling units, and sampling ratios to draw a simple probability
sample. [Apply]
5. Contrast simple random sampling, systematic sampling, stratified sampling, and
multistage cluster sampling (which are not mutually exclusive). [Evaluate]
6. Design and carry out an appropriate sampling strategy if given a target population and a
target sample size. [Create]
Social scientists understand that the conclusions we draw about the social world depend heavily
on who was included in the survey. If the sample was highly nonrepresentative of the population,
Page 35
we treat the conclusions drawn from that study with a high degree of skepticism. Therefore,
being able to draw a random sample is an important first step in generating findings and
conclusions that are generalizable to a larger population (the goal of much, although not all,
social science research).
Without a basic understanding of sampling strategies, students will be unsure how generalizable
their conclusions are to a larger population. They will also be unsure exactly to which group their
conclusions apply. Further, understanding how other social scientists go about sampling will help
them to be more critical consumers of research. Instead of simply believing that a finding in a
piece of published research is true, this chapter will help them first ask “Who was included in
that study and how were they selected?” which will help students to know how much confidence
to have in what they read.
The greatest difficulty for my students is grasping what social scientists mean when we say
something is “random.” We mean, of course, that any outcome has the exact same chance of
occurring as any other outcome. Students, however, use this term freely and in a way that is not
necessarily consistent with how social scientists envision it. For example, I often hear my
students say “We went out only once and then, after a week, he called me out of the blue. It was
so random.” To a social scientist, that phone call was far from random. There was not an equal
chance that any student could have called. Instead, the student who made the comment has a
fairly limited social network (e.g., she doesn’t know all students on the campus or all young
people in her city) and she had seen the caller only a week earlier. Therefore, this outcome did
not really occur randomly at all. Instead, when social scientists discuss random sampling, they
mean that any person in the population has the exact same chance of being included in the
sample as any other person in the population. In other words, a particular outcome has the same
chance of occurring as any other outcome. If students can understand the statistical use of
randomness, they should be able to comprehend the remainder of the sampling concepts in this
chapter.
In terms of in-class activity, there are several exciting and useful activities that instructors can
use to illuminate the idea of systematic and random sampling. For example:
1. When teaching this chapter, I utilize an exercise that reinforces the concept
representativeness. I like to bring in an opaque bucket of candy of various colours.
(Perhaps 5 or 6 different colours). Ahead of time, I count up how many candies are blue,
how many are red, and so forth, and I percentage them out. I do not share this information
with the students. Then, I approach a student in the class and I ask them to draw only 1
Page 36
candy. I ask the student to estimate, based on the colour of that candy, what percentage of
the candies in the bucket are the same colour as the one drawn. The student usually looks
stumped and says something such as “All of them?” I then explain to them that when you
study only 1 person, it is hard to learn anything about the entire population. At this point,
I have another student draw 5 candies. They may draw, for example, 2 blue, 1 red, 1
yellow, and 1 green. I then ask them to estimate the composition of the bucket. They
guess that about 40 percent of the candies in the bucket are blue, since that was the
percentage in the small sample. We then put the candies back into the bucket (sampling
with replacement) and repeat the exercise with samples of 10, 20, 30, and so forth. I write
down the proportion of candies that are blue (for instance) in each sample. When the
exercise is finished, I reveal the true colour composition of the candies in the bucket and
almost without fail, the larger samples provided a closer approximation to the actual
colour distribution. This demonstrates for students that larger samples become more
representative of the population because they contain a larger portion of the population
(i.e., larger sampling ratio). At the end, I let them eat the candy.
2. The above exercise can also be done using a deck of cards from the popular card game
Uno. For this variation, I take the deck and average out the cards in the deck. The mean
or average of all the cards is usually around 7. I then let students draw samples of 1 card,
3 cards, 5 cards, 10 cards, and so forth. After each draw, we take an average of the
sample. Just like in the candy example, they will see that larger samples are more
representative of the population because their sample mean more closely approximates
the population mean.
3. In order to demonstrate systematic sampling to them, I also show them a sampling frame
(list of all elements in the population) derived from my own research. This list is
essentially a list of households in a particular neighbourhood. Here is an example
sampling frame that uses households as the elements:
Street
Number
Street
Name
City,
Province
5809
Main St.
5821
Main St.
5939
Main St.
Toronto,
Ontario
Toronto,
Ontario
Toronto,
Ontario
etc.
etc.
etc.
Postal
Code
Occupant Name(s)
Phone Years at Owner
Number Address Occupied?
A1A 1A1
John and Jane Doe 486-4185
12
y
A1A 1A1
19
n
A1A 1A1
1
y
etc.
etc.
etc.
etc.
etc.
I then ask them how I could draw a representative sample of the households. Students
will usually suggest drawing every 5th or 10th element (household), which is an
appropriate strategy to use. But I then ask them if it’s important the elements are arranged
in order of address, in order of the name of householder, etc. (It doesn’t, unless the goal is
to make sure that several households on each street are drawn.) Then, we discuss where
on the list we would want to start—with the first? (That, of course means that the first
household on the list has a 100% chance of being drawn.) I then talk with them about
Page 37
random-number-generating programs (one of which I provide the link to below) that can
be used to select a starting position at random.
In Assignment 2 (in Appendix B: “Sampling and Quantitative Data Collection and Analysis”), I
have my students design a sampling strategy to generate a representative sample of our
university’s student population. Even though they will carry out their survey using students in
our class as the sample, I still have them create a sampling strategy and explain it to me. If
students discuss where they might obtain a sampling frame (list of all students), what the
elements are within that sampling frame, what their sampling ratio might be, and so forth, I am
able to conclude that they grasp the material associated with sampling.
As above, students should be able to design a sampling strategy to answer a research question
that they have derived (see Appendix B). This process includes
1. Deciding who comprises their target population
2. Considering what their sample frame might look like
3. Devising a method for randomly selecting a certain number of elements from their
sampling frame, based on their target sample size
If they are able to apply knowledge in this way, it is safe to assume that students have grasped
the key concepts involved in sampling.
1. Random number generator, to help with selecting a starting place at random: Random.org.
“What’s This Fuss about True Randomness?”: www.random.org
2. Discussion of simple random sampling from University of South Alabama: R. Burke
Johnson. Sampling: www.southalabama.edu/coe/bset/johnson/lectures/lec7.htm
3. A statistical analysis software package that can both generate random numbers and draw
simple random samples: Stata Corp. LP. Stata data analysis and statistical software:
www.stata.com
4. Sampling example similar to the candy example described above, from the Science
Education Resource Centre, Carleton University: Pedagogy in Action. “Reese’s Pieces
Activity: Sampling from a Population”:
http://serc.carleton.edu/sp/library/datasim/examples/reeses.html
Page 38
CHAPTER 7 EXPERIMENTS
Chapter Summary
This chapter introduces students to experimental methods commonly used in social research, and
to some of the common problems that occur in experimental research design. Since experimental
methods are used more commonly in psychology than in the other social sciences, this chapter
also introduces the natural experiment, a method commonly used in disciplines such as
sociology.
Students should understand the benefit of including both experimental and control groups. They
should also understand how the experimental research design can be used in situations more
commonly encountered by sociologists (e.g., natural experiments).
1. Understand which topics of social research require an experimental research design.
[Understand]
2.
Differentiate between situations requiring a pretest and posttest. [Evaluate]
3. Understand the difference between the experimental group and the control group.
[Understand]
4. Recognize the pitfalls of experimental design, including the Hawthorne effect, history,
maturation, experimental mortality, testing, and several others, when reading existing
research. [Apply]
5. Appreciate the importance of double-blind approaches and randomization for most
experiments. [Understand]
6. Use the “natural” experiment” approach to social research, and recognize situations
where this approach is useful. [Apply/Evaluate]
Experimental methods are some of the best ways to understand human behaviour in a
comparative fashion. However, many questions asked by social scientists cannot be answered
Page 39
through experimental methods, which require the manipulation of key variables, the division of a
sample into at least two groups, and so forth. Despite its limitations, the logic of experimental
design often works well for other types of research. For example, the Additional Resources
section below includes two examples of natural experiments—one conducted by economists and
one by sociologists.
Experimental methods are some of the most commonly used methods in the social sciences,
albeit more common in psychology than in other social science disciplines. Nevertheless, the
core concepts of experimental research methods (experimental group, control group, holding
variables constant, and so forth) can be applied to topics more commonly investigated by
sociologists, through the natural experiment and field experiment approaches.
Students usually understand experimental methodology fairly well, as many students have had a
previous psychology course. However, these students may enter the course assuming that
sociologists commonly divide people into experimental and control groups and then apply a
treatment, similar to psychologists. Therefore, the largest challenge in teaching experimental
research design in a sociology classroom involves teaching students how to extend this method
to other types of naturally occurring situations. This is why it is particularly important to cover
natural experiments, for instance, as this method is much more commonly used by sociologists
than are true experiments.
1. After discussing with students the principles of experimental research design, it may be
useful to begin discussing field experiments and the situations in which they are most
appropriate. In my course, I often show the two-part film True Colors (see Additional
Resources below). Originally aired on ABC’s Primetime Live in 1991, the news program
asked two men, John and Glenn, to relocate to a new city (St. Louis, Missouri) and look
for an apartment, apply for jobs, buy a car, and so forth. The men have similar
educational backgrounds, skill sets, cultural capital, and so forth (control variables).
However, John is white and Glenn is black.
Not surprisingly, the men are treated very differently in their social interactions. This
film, although dated, demonstrates experimental methodology. For example, the
researchers have held many things constant (educational background, age, clothing
choices) but have allowed one variable to vary between the two men (race or ethnicity).
As a result, any difference in treatment can be attributed to that variable. Students find
this example interesting and it generates excellent discussion about both research
methodology and racial discrimination.
Page 40
2. As a discussion piece for natural experiments, instructors may want to consult the article
detailed in item 3 in Additional Resources below. In this research project, we compared
two neighbourhoods, both devastated by Hurricane Katrina. One neighbourhood,
however, was a low-income, predominantly black neighbourhood. The other was a
wealthier and predominantly white neighbourhood. This presented a unique opportunity
for a natural experiment because it is so rare that those who are privileged would find
themselves in the same circumstances (a destroyed house and displaced neighbourhood)
as those who are less privileged, allowing for the ability to control for particular
circumstances. As a result, we were able to analyze how pre-existing differences in
income and race/ethnicity helped those from the more privileged neighbourhood secure
better outcomes from preparation for the disaster through eventual return and recovery.
In my course, I do not have students carry out an experiment (due to time and other logistical
constraints). However, it would be possible to have students carry out a field experiment, similar
to the one portrayed in True Colors. For example, students might wonder whether baristas at a
coffee shop are friendlier to a certain type of customer. This question could easily be answered
using a field experiment (which involves some manipulation) or a natural experiment (which
takes advantage of experiment-like conditions that already exist). If students are successful in
carrying out an experiment such as this (employing a control group, for example) and
interpreting results, they have successfully mastered the learning outcomes for the chapter.
It might also be useful to ask students to reflect on the advantages and disadvantages of
experimental research design on an exam. On their final exam, I ask my students a short-answer
or essay question similar to the following:
Question: Why do sociologists rarely use experimental research design? What types of
research design are more common in sociology, and why?
Students explain that the experimental research design usually (although not always) involves
manipulation, and sociologists are often concerned about preserving naturalism. This is why they
would more often use a method such as ethnography (or even survey or interview research,
which are perhaps not as useful for preserving naturalism but do not involve any manipulation).
If students are able to reflect on the advantages and disadvantages of experimental research
design, instructors can be confident that they have successfully taught the material.
1. Professor Cosmic. True Colors: Racial Discrimination in Everyday Life (originally aired by
ABC’s Primetime Live in 1991): A field experiment in labour market discrimination:
Part 1: www.youtube.com/watch?v=YyL5EcAwB9c
Page 41
Part 2: www.youtube.com/watch?v=gOS3BBmUxvs
2. The following article is an example of an economics field experiment. The authors created
several fictional job-seekers (equal in qualifications and educational background) and sent
résumés to job postings advertised in local newspapers. They allowed the name of the
participant to vary (“Emily” or “Greg,” meant to denote white names, and “Lakisha” or
“Jamal,” mean to denote black names). They found, again not surprisingly, that those with
“black-sounding” names received fewer callbacks, despite their equal qualifications. This
again exemplifies how those in the social sciences (but outside of experimental psychology)
can use experimental methods to learn about social phenomena such as racism:
Bertrand, Marianne, and Sendhil Mullainathan. “Are Emily and Greg More Employable
Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.”
American Economic Review, Vol. 94, No. 4, 2004: pp. 991–1013.
3. Example of a natural experiment:
Elliott, James R., Timothy J. Haney, and Petrice Sams-Abiodun. “Limits to Social
Capital: Comparing Network Assistance in Two New Orleans Neighborhoods Devastated
by Hurricane Katrina.” The Sociological Quarterly. Vol. 51, No. 4, 2010: pp. 624–648.
(This research is also described in Chapter 6, on page 196 of the textbook).
4. An excellent paper on the history and use of field experiments (co-authored by Steven Levitt,
one of the authors of the bestselling book Freakonomics):
Levitt, Steven D., and John A. List. “Field Experiments in Economics: The Past, the Present,
and the Future.” NBR (National Bureau of Economic Research) Working Paper 14356,
September 2008: www.nber.org/papers/w14356.pdf?new_window=1
Page 42
CHAPTER 8
SURVEY RESEARCH
Chapter Summary
Survey research is one of the most commonly used methods for data collection in social research,
especially for those who intend to perform quantitative analyses. This chapter introduces
students to the guiding principles of survey design and encourages them to consider how they
might write stronger questions, generate higher response rates, and approach potential
participants in a professional manner.
Students should be able to distinguish between well-written survey questions and poorly written
survey questions, and they should clearly comprehend the advantages and disadvantages of
different types of survey questions.
1.
Understand the types of questions typically used on survey forms and questionnaires,
including open-ended and closed-ended. [Understand]
2.
Avoid common survey question problems in their own work, such as double-barrelled
questions; lack of clarity, competency, and relevance; respondents’ unwillingness to
answer; and biased and negative items. [Apply]
3. Construct contingency questions, understanding how useful they are in survey research.
[Create/Understand]
4. Understand the process of sending out surveys, as well as issues surrounding response
rate. [Understand]
5. Appreciate the general guidelines for survey interviews, telephone interviews, and
Internet interviews. [Understand]
Survey research is nearly ubiquitous within the social sciences. Although qualitative
methodologies are gaining in popularity, survey research still provides one of the more cost-andtime-effective methods for collecting a large amount of data.
Page 43
Besides its utility for social scientists, survey research methods are widely used in the fields of
law, business, social work, and many others. No matter their intended career path, students will
possibly at some point be required to create a survey, administer the survey, and analyze data
derived from that survey. So the concepts discussed in this chapter are some of the most
important and applicable.
1. One common but unavoidable problem when teaching survey research (or any
methodology) is that students will not yet know what is done with survey data. That is,
they must be taught instrument design and data collection methods before methods of
analysis. At the same time, comprehending design and data collection would be easier if
students understood first what sort of data was generated from surveys. One solution
might be briefly showing them a quantitative dataset (for example, in Microsoft Excel),
with cases in rows and variables in columns. This will give them a broader picture of the
methodology and an idea of the end goal.
2. Probably the most common mistake that students make is the failure to consider all
possible options that should be provided when writing closed-ended questions. This
results in choices that are not exhaustive.
3. One other typical problem is that students will often create closed-ended questions where
open-ended questions may actually be preferable. Closed-ended questions provide
researchers with less information than open-ended questions. For example, students may
ask a research participant
“What is your current yearly income?”
$0–20,000
$20,000–$40,000
$40,000–$60,000
$60,000–$80,000
$80,000 or more
Aside from the fact these options are not mutually exclusive (someone earning exactly
$40,000 would fall into two of the categories, not just one), the bigger issue is that once
data are collected, it is impossible to make comparisons between people who earn
$40,000 and those who earn closer to $60,000 (since they are included in the same
category). Asking an open-ended question might be preferable in this situation, as it will
Page 44
give full information about a person’s income. These data can always be collapsed into
groups later, but it is better (from an analysis standpoint) to start with more complete
data.
4.
Students should consider the differences between open-ended and closed-ended survey
questions. While open-ended questions yield richer data, they do create a problem when
responses need to be coded. To demonstrate this to students, you may give them an
example like the following:
Question: How large is the business at which you work? _______
Participant responses could include “3000 hectares,” “Very large,” “200 employees,” and
“Multinational.” Coding these in a way that will differentiate large employers from small
ones could prove very difficult.
On the other hand, closed-ended questions are often too simple to properly capture
underlying concepts. One example might be:
Question: In your opinion, is there still racism in Canada?
[ ] Yes
[ ] No
[ ] Not sure
A persistent, multifaceted, pernicious social problem such as racism is probably too
complicated to capture by “Yes” or “No.” Further, what each participant considers to be
racism will vary considerably. As a result, students should think carefully about the
opportunities and costs of various kinds of survey questions.
1. First, I like to provide students with examples of non-ideal survey questions and have
them critique those questions. For example, I ask them “What do you think of the
proposed peace plan?” (Which is a bad question because it assumes participants know
which peace plan the survey is referring to, and also assumes they have some knowledge
about its specifics.) I also give them examples of double-barrelled questions, such as
“Should Canada abandon its space program and spend the money on domestic
programs?” (If respondents are given the choices “Yes” and “No,” which answer should a
participant mark who believes the space program should be abandoned, but wants the
money spent on some other cause?) Finally, I give them examples of leading questions
such as “Don’t you agree with the prime minister that proroguing Parliament was
necessary?” (The phrase “don’t you agree” makes it seem as if anyone who is reasonable
would agree with the premise of the question.) Other common problems we discuss
include willingness to answer (for example, questions asked of immigrants to Canada),
illogical choices, and so forth.
Page 45
2. Similarly, I also ask students to critique several examples of questions that have choices
that are either not exhaustive or not mutually exclusive. For example:
What is your current age?
20–30
40–50
55–75
These choices are not exhaustive, since an individual who is 35 or who is 53 would not
fall into any category.
What is your current age?
0-20
20-40
40-60
60-80
80 or older
These choices are not mutually exclusive, since an individual who is 40, for example,
would fall into two different categories.
Although the problems with these questions may seem to be obvious, many questions
suffer from the same problems but are less obvious. For example, when students
construct questions, many of their questions look something like the following:
What is your sexual orientation?
Heterosexual (straight)
Homosexual (gay or lesbian)
This question is not exhaustive, since many people do not identify themselves as either
straight or gay. Indeed, sexuality scholars now discuss many different sexual orientations,
including ones such as pansexual, which should be included as well.
Students seem to enjoy critiquing examples of survey questions. Indeed, I even show
them examples of questions from a survey that I once administered. They are quick to
notice problems in my questions, which not only makes me (as the instructor) seem more
human and fallible, but shows them that writing good survey questions is something with
which professional sociologists, not just students, struggle.
3. With my students, I discuss the feminist critiques of survey research and quantitative
methods. The main critiques of survey research are
a. That it can be exploitive of participants, viewing them more as “subjects” and less
as active participants in the research. Further, feminist critics worry that survey
researchers take information from participants but do not give back to the
participants in ways they appreciate or understand.
b. Feminist scholars worry that survey research dehumanizes participants by
insisting on the need for objectivity. Researchers are discouraged from getting to
know participants on a personal level.
Page 46
c. They worry that survey research relies on the “researcher knows best” paradigm,
in which there is a hierarchical relationship between “expert” and “subject.”
d. Feminist critics argue that the complexity of human experience cannot be
captured in “objective” survey choices such as “Agree” or “Disagree,” or even
more problematically, “Yes” or “No.”
e. Finally, they contend that the research produced from surveys is often stripped of
the researcher’s voice. Instead, the research speaks using an anonymous,
disembodied voice of authority, which both masks the researcher’s biases and
preconceptions while at the same time speaking from a masculine position of
authority.
Once students have learned the basics of survey researcher, I enjoy discussing these
five issues with them, seeing the extent to which they agree with the critiques, and
playing devil’s advocate to the positions that they take. This, I believe, creates a rich
learning environment and shows them that almost no methodology is free of criticism.
As with most material covered in a research methods course, it is probably most effective to
require students to construct research projects of their own. As a result, I ask my students to
design a survey instrument that they administer to the other 25 or so students in the class. This
exercise is included in Appendix B as Assignment 2. If students are able to construct and design
a survey that avoids many of the common problems in survey design (double-barrelled questions,
leading questions, and so forth), they clearly understand at least the basics of survey design and
administration.
At this point in the course, now that we are covering more concrete material, such as creating
survey instruments, I like to administer an anonymous student evaluation of my teaching. This
can be done using the popular and free online survey tool TooFAST (www.toofast.ca). I ask
students to answer such questions as “I now feel that I could design and carry out a survey
research project,” using such responses as “Strongly Agree/Agree/Unsure/Disagree/Strongly
Disagree),” “I understand which sorts of research questions and situations are appropriate for
survey research,” and so on. Since students can answer anonymously, this honest feedback helps
instructors to better understand what students comprehend clearly and where more attention
might be required.
1. Some samples of survey questions that could be used as examples: Constant Contact®, Inc.
Sample Survey Questions, Answers and Tips:
www.constantcontact.com/aka/docs/pdf/survey_sample_qa_tips.pdf
Page 47
2. A survey put together by the American Sociological Association to survey sociology majors,
National Survey of Seniors Majoring in Sociology:
www.asanet.org/images/research/docs/pdf/BandB_web_questionnaire.pdf
3. Some examples of bad survey questions, from Western Kentucky University: Examples of
Bad Questions & Suggestions of How to Fix Them!
www.wku.edu/~holli.drummond/first%20page/classes/strategies%20of%20social%20researc
h/example%20of%20bad%20survey%20questions.pdf
4. TooFAST is an excellent tool for carrying out online surveys, as well as anonymous
evaluations of teaching effectiveness. Even better, it is housed in Canada, so it is not subject
to the U.S. Patriot Act, as similar platforms (like Survey Monkey) are: https://www.toofast.ca
5. An excellent resource (still the classic) for survey construction:
Converse, Jean M., and Stanley Presser. Survey Questions: Handcrafting The Standardized
Questionnaire. Thousand Oaks, CA: Sage. 1986.
6. The literature on feminist methodologies contains some of the most vociferous critiques of
survey research. The following books provide two well known critiques:
Harding, Sandra. Feminism and Methodology. Bloomington, IN: Indiana University Press.
1987.
Sprague, Joey. Feminist Methodologies for Critical Researchers. Lanham, MD: Rowman and
Littlefield. 2005.
Page 48
CHAPTER 9
UNOBTRUSIVE RESEARCH
Chapter Summary
This chapter presents an introduction to unobtrusive methods such as content analysis, the
analysis of existing statistics, and historical/comparative research. These methods allow
researchers to investigate the social world without research ethics review and without recruiting
participants. They are especially useful for the analysis of publicly available communications
and documents.
Students should understand and appreciate why it might sometimes be necessary or preferable to
conduct research that does not involve direct contact with human participants. This can occur out
of a desire not to affect the outcome of the research or, just as commonly, because one’s research
question can be answered without doing so (e.g., questions about how messages are
communicated can often be answered by doing a content analysis of a popular television show,
for example). In these cases, unobtrusive methods are both easier and less expensive than
interviewing or surveying.
1. Understand how and why social scientists analyze existing statistics (such as Durkheim’s
study of suicide). [Understand]
2. Understand how and why sociologists undertake content analysis, including the process
of coding, as well as the analysis of manifest and latent content. [Understand]
3. Design and carry out a content analysis project. [Apply/Create]
4. Understand the process of comparative and historical research. [Understand]
Although it is common for social scientists to study individuals directly, many issues are best
learned about through the use of other methods. For example, if a sociologist wants to better
understand the social and economic conditions under which revolution occurs (or does not
occur), it is less useful to interview people (who may have never considered that question or may
have never experienced an uprising or revolution) than it is to study past epochs or locations to
systematically figure out what conditions tend to precede revolution. Although such an event is
Page 49
still orchestrated by people (and thus still sociologically relevant), this comparative and historical
method is often the best for answering these sorts of questions.
The media most appropriate for content analysis are widely available (written letters, television
programs, articles, and so forth). Further, they are accessible without applying to a Research
Ethics Board for approval. Therefore, these research projects can proceed with very little time or
expense. This is also true of historical methods and of the use of existing data; in both of these
methods, data are widely accessible and ethics board approval is normally not necessary.
Therefore, especially for students who are conducting research within the limitations of a course
and a semester schedule, these methods are particularly attractive and useful.
Students will often assume analysis of media must be done quantitatively. That is, they assume
that they must devise a structured table or rubric for recording instances that a particular
phenomenon occurred as they were watching or reading. As I explain to students, analysis of
content can be done this way, or it can be done as discourse analysis, which is the viewing of
content to see what themes emerge, who is creating those discourses, and for what specific
reasons.
In teaching unobtrusive methods such as content analysis, instructors can attempt to accomplish
several tasks in the classroom:
1. In my course, I introduce students to the kinds of questions that are asked by researchers
doing content analysis. For example,
Example Research Questions:
“Is Canadian music more politically cynical than American music?”
“Are gender roles portrayed differently in Family Guy than in The Simpsons?”
2. I walk students through what I believe are the basic steps of content analysis:
a. Operationalization
b. Creation of a codebook with variable names
c. Choosing parameters (for example, including only magazine
advertisements larger than a certain size)
d. Devising a sampling strategy
e. Creating a recording sheet to keep track of data;
f. Analyzing data
Page 50
3. Together, students and I walk through the above process using the example of magazine
advertisements (which I have copied from the Internet). We look at whether male or
female models are positioned in the front/back or foreground/background. We then
analyze our data quantitatively.
4. To show them that this process can be done qualitatively, for discourse analysis, we
analyze the text of Health Canada press releases related to the H1N1 flu pandemic of
2009. Students look for the messages and discourses that Health Canada is creating about
risk.
Although I do not have my students do an analysis of existing statistics or do a comparative or
historical project (although possible, both are very large undertakings), I do have students do a
content analysis (quantitative) and discourse analysis (qualitative). They do this as Assignment 4
in the course (see Appendix D). If they are able to devise and carry out both a content and
discourse analysis, they have indeed learned the outcomes (as the least the ones related to content
analysis).
Second, I ask them in an essay question on the final exam to differentiate unobtrusive
methods from other types of data collection methods (particularly surveying and interviewing),
and to discuss the advantages and disadvantages of each approach.
Like all methodologies, unobtrusive methods are best learned when they are actively used.
Therefore, it is best to have students design and carry out a small, manageable content analysis
(for example). If students are able to collect and analyze data systematically to draw conclusions
about some type of written or transmitted content, then teachers can assume that they have
successfully communicated to students the necessary and important ideas present in this chapter.
1. An excellent example of content analysis was recently published in the journal Gender and
Society. The author watched episodes of the popular television show Jeopardy! in order to
determine how men and women handle success. He found that women, through the tone of
their voice, are apologetic for their success (i.e., answering questions correctly), whereas men
are not. Through this research, the author demonstrates one of the micro-level mechanisms
by which gender inequality persists. This article is a great example of content analysis, for
possible class discussion or supplemental reading: Linneman, Thomas J. “Gender in
Jeopardy! Intonation Variation on a Television Game Show.” Gender and Society, Vol. 7,
No. 1, February 2013.
Page 51
2. The Writing Center at Colorado State University maintains this website dedicated to the
history, epistemology, and practice of content analysis: Writing at CSU. “Content Analysis:
http://writing.colostate.edu/guides/guide.cfm?guideid=61
3. Comparative and Historical Sociology section of the American Sociological Association
website: http://www2.asanet.org/sectionchs
Page 52
CHAPTER 10 FIELD RESEARCH
Chapter Summary
Field research encompasses a number of methodologies commonly used to observe people in
their natural environments. Some of these approaches, like ethnography, are designed primarily
to create knowledge about the social world. Others, such as participatory action research, are
designed to mobilize and empower individuals, first and foremost, and only secondarily, to
create knowledge.
Students should understand both the advantages and disadvantages of field methods, specifically
ethnographic methodologies. Within this methodological framework, they should understand the
grounded theory and institutional ethnography approaches to research.
1. Understand the advantages of qualitative field research, including preserving naturalism.
[Understand]
2. Contrast the advantages and disadvantages of being a full participant vis-à-vis a complete
observer within ethnographic research. [Analyze]
3.
Understand the case study approach to social research. [Understand]
4. Avoid reactivity during field research. [Apply]
5. Understand the grounded theory approach and how it differs from the extended case
method approach. [Understand/Evaluate]
6. Understand institutional ethnography and participatory action research. [Understand]
One of the advantages of field methods is the ability to capture the relationship between what
participants say they do and what they actually do. With surveying or interviewing, researchers
can capture only the perspectives and reflections of participants; they cannot capture how
participants behave. Therefore, field methods are viewed by many researchers as more
Page 53
naturalistic. That is, field methods capture participants in their natural environments, acting
(hopefully) in ways consistent with how they would normally act.
For students, field research presents one of the more engaging methodological approaches, as
this type of research can take place in bars, nightclubs, malls, airports, or just about any spaces
normally inhabited by people. For this reason, many of my students find field research much
more enjoyable to conduct than survey or interview research. And they are often surprised by the
rich description and depth of analysis they are able to produce.
My students have a difficult time understanding both institutional ethnography and grounded
theory, approaches that grew out of the gender and feminist methodology literature. Put simply,
institutional ethnography is the practice of asking participants to describe their everyday realities.
From those realities, it is possible to construct an understanding of the practices and policies that
affect them. In other words, it is a mechanism for understanding power relationships by starting
with those most affected by power relationships. Grounded theory is an inductive approach to
research, by which observations or data are compiled first, in order to generate a theoretical
understanding of some phenomenon.
1. The best way to understand field methods, such as ethnography, is probably for students
to read an ethnographic work. In a course that is perhaps only 13 or 15 weeks long, this
may be problematic, so I choose an ethnographic work and selectively extract excerpts to
share with my students to help them understand the advantages and disadvantages of the
ethnographic approach. Two of my favourites are Venkatesh’s Gang Leader for a Day
(2008) and Pascoe’s Dude You’re a Fag (2007) (see Additional Resources below).
Venkatesh’s book is full of insights into both the advantages and disadvantages of
ethnographic work. As one example, when the author visited a poor, predominantly black
neighbourhood of Chicago with a survey form containing questions such as “How does it
feel to be black and poor?” the respondent “looked at a few more pages of the
questionnaire. ‘You ain’t going to learn shit with this thing.’ He kept shaking his head
and then glanced toward some of the older men standing about, checking to see if they
shared his disbelief. Then he leaned in toward me and spoke quietly. ‘How’d you get to
do this if you don’t even know who we are, what we’re about?’ His tone wasn’t
accusatory as much as disappointed and perhaps bewildered” (p. 16). Passages such as
this help students understand some of the advantages of ethnographic work; it helps
Page 54
researchers to gain a fuller understanding of the lives of participants in very nuanced and
detailed ways that cannot be captured in a closed-ended survey question, or even in a
three-hour interview.
Venkatesh is clear about the drawbacks of his methodology, as well, providing good
discussion material for class. After the discussion, instructors can ask students quite
simply:
Question: After learning a bit about ethnography by hearing passages from one
ethnographic work, what do you think is the best way to learn about people,
especially people who are low-income, marginalized or disadvantaged (as was the
case in Venkatesh’s work)?
This question provokes excellent discussion among students, some who defend the
ethnographic method as more naturalistic, and others who believe it is too intrusive and
therefore has the potential to cause too much harm to participants. This sort of exercise
could be done with any ethnographic work, especially one where the author is
particularly reflexive (e.g., concerned about how the author’s position shapes research
findings).
2.
In teaching students how to observe others in a public place, it may be useful for them to
practise this skill. To do this, you could try having two students (volunteers) come to the
front of the class to read a script (I use a script from the movie version of Grapes of
Wrath, but what they read is entirely arbitrary). You can then ask the remaining students
to observe and to write down everything they can. Then, it is useful to ask them what they
observed. They tend to focus on body positioning and verbal inflection.
In my class, I ask “Did anyone notice that [student name] checked her cellphone four
times?” or “Did anyone notice the groundskeeper walk by the window. What was he
doing?” Students find that they were so focused on the two students that they blocked out
everything else in their social environment. This exercise, of course, demonstrates the
tendency among field researchers to focus on a particular topic or person of interest, but
this means filtering out other words and actions that might end up being just as
meaningful. In other words, field research involves a series of decisions about what to
pay close attention to and what to ignore.
In my own course, I do not have students conduct their own field research (due to time
limitations), as they do this frequently in other courses. However, this is certainly an option for
assessing students’ grasp of the material. Most Research Ethics Boards do not require review for
observation research conducted in public places, making this type of research easier and very
accessible for students within the limitations of a semester-long course.
Page 55
The best way to ensure that students understand field research is either to require them to
conduct their own field research, or alternatively, to ask examination questions that capture their
understanding of the method. As an example, I ask students on their final exam:
Exam Question: An ethnographer can choose the extent to which she or he is a full
observer or a full participant (or somewhere in between). Discuss the advantages and
disadvantages of each of these approaches to ethnographic research.
Students typically reflect on the extent to which a researcher can stay objective if he or she
becomes a full participant, and by contrast, the extent to which they can understand and interpret
what they see if they are a full observer. Therefore, this question is excellent for getting students
to think through how a researcher should negotiate his or her own position when commencing
ethnographic work.
1. Some books that provide good examples of field methods (specifically ethnographic
research) that may be useful for discussion with students:
Venkatesh, Sudhir. Gang Leader for a Day: A Rogue Sociologist Takes to the Streets. New
York: Penguin. 2008.
Pascoe, C. J. Dude You’re a Fag: Masculinity and Sexuality in High School. Berkeley:
University of California Press. 2007.
Tuchman, Gaye. Wannabe U: Inside the Corporate University. Chicago: University of
Chicago Press. 2009.
2. A good resource on institutional ethnography from Syracuse University:
http://faculty.maxwell.syr.edu/mdevault/default.htm
3. Website for Dr. Barney Glaser and the Grounded Theory Institute, which includes an
excellent description of the grounded theory approach to field research:
www.groundedtheory.com
4. McTaggart, Robin. Caledonia Centre for Social Development. “The 16 Tenets of
Participatory Action Research.” 1989: www.caledonia.org.uk/par.htm
Page 56
CHAPTER 11 QUALITATIVE INTERVIEWING
Chapter Summary
This chapter covers the logic underlying interviewing, an important method for generating
qualitative data. Indeed, interviewing is probably the most common qualitative method in use
within the social sciences. The chapter also covers variants of interviewing, such as oral history
and focus groups.
Students should understand and appreciate why researchers may use interviews to generate
qualitative data and how those data can be used to create knowledge about the social world.
1. Understand how the interviewing process is iterative and changes from one interview to
the next. [Understand]
2. Conduct interviews and focus groups. [Apply]
3. Contrast the advantages and disadvantages of focus groups with those of one-on-one
interviews. [Evaluate]
4. Understand the oral history approach to interviewing, as well as its limitations.
[Understand]
When social scientists advance research questions that involve mechanisms or explanations, they
require more nuanced methods than a survey can provide. In these instances, interviewing is
probably the best method for answering these questions. Interviews allow participants to explain,
in full detail, their behaviour, attitudes, and feelings.
Page 57
Interviews are probably the easiest and most practical method for exploring a topic in detail.
Unlike in a survey, interviews provide an opportunity to ask participants why they engage in
certain activities or hold certain beliefs. So, rather than simply answering the what questions,
interviews allow the researcher to begin answering the important why questions.
Before conducting an interview themselves, students typically underestimate how difficult it is to
conduct an interview and take notes. They are normally also not prepared for the speed at which
participants speak. Only by conducting an interview will students become aware of these factors.
Students may also assume that a focus group is naturally superior to one-on-one interviews
due to the sheer volume of data generated by a larger group of people. Researchers know this is
not always the case. Due to the “group-think” effect, the potential for dissenting views to remain
silent, the difficulty isolating whose voice belongs to whom on a tape, and the ethical
considerations involved in a potential loss of confidentiality (e.g., how does a researcher stop a
participant from speaking about the focus group after it is over?), there are some distinct
advantages to one-on-one interviewing.
There are several excellent in-class activities to help students understand and appreciate
interviewing:
1. In my course, I ask students to divide into groups of four students as part of Assignment 3
(see Appendix C), which covers interviewing and qualitative data analysis. Then each person
in the group interviews the other three students using a semi-structured interview format and
an interview guide that they have constructed ahead of time. The interviews take perhaps 10
to 15 minutes each. Students take notes, although they do not necessarily write out excerpts
verbatim. Students seem to enjoy interviewing one another, and come away with an
appreciation of how difficult it is to simultaneously conduct an interview, listen, and take
notes.
2. I do not ask students to record interviews that are conducted in class, given the equipment we
would need, as well as the ethical considerations (most Research Ethics Boards want to
review and grant approval for studies using voice or video recording), but I do ask them to
take notes. For this, I suggest they use a data recording sheet, such as the following.
Page 58
Data Recording Table
What you observe
Your reactions/thoughts
2. To hone their skills in interviewing, you may also ask students to watch a taped
interview. This will help students get the “feel” of an interview, to see how the
interviewer uses probing or prompting questions to dig deeper into the subject matter,
and to see how the interviewee may work to change the direction of the conversation.
There are many examples of interviews on YouTube, but one that I have used in my own
course is an interview between Diane Sawyer (an ABC news anchor) and U.S. President
Barack Obama. The interview can be accessed at
www.youtube.com/watch?v=_wsde7yjK3A
Following the interviews that students conduct in class (see item 1 above, as well as Assignment
3 in Appendix C), students then input their notes into a word processing program, code their
notes using colours, and analyze their data. If students are able to carry out these three short
interviews, code data, and analyze data, they have achieved the major learning outcomes in the
chapter.
Since a research methods course is almost by definition a “how to” course, the best way to help
students learn the material is to ask them to use the methods themselves. Therefore, assignments
that ask students to design an interview (or focus group) research project, construct an interview
guide, or carry out interviews will require students to demonstrate mastery of the material. If
students demonstrate that they are able not only to understand the material in the chapter but also
to apply it to their own topics of interest, instructors can be confident that their teaching has been
effective.
Page 59
1. Example of a focus group from the popular television show Frasier:
www.youtube.com/watch?v=UInbPO1Shus
2. A sample interview guide from Cornell University (most are much shorter than this one,
however): Profiles of Practitioners. “Sample Interview Guide”:
http://courses2.cit.cornell.edu/fit117/CP_I_InterviewGuide1.htm
3. For conducting telephone interviews, the following is an excellent resource:
Gwartney, Patricia A. The Telephone Interviewer’s Handbook: How to Conduct
Standardized Conversations. Hoboken, NJ: Jossey-Bass. 2007.
4. Examples of digital voice recorders for conducting interviews:
www.getolympus.com/us/en/audio/digital-recorders.html
Page 60
CHAPTER 12 EVALUATION RESEARCH
Chapter Summary
While some forms of research aim simply to create knowledge, other forms attempt to discern
which sorts of policy or social interventions are best and whether implemented interventions are
effective. Evaluation research, unlike what is often called “pure” research, is an attempt to use
social science methods to gain information on the effectiveness of social programs, policies, or
other interventions.
Students should understand how and why social scientists may want to use research methods to
evaluate policies or programs.
1. Understand the basic logic of evaluation research, including the need for measuring the
effects of interventions. [Understand]
2. Understand how to logically determine whether an intervention has had the desired
outcome. [Understand]
3. Design their own evaluation research. [Apply]
Social scientists often encounter opportunities to engage in more applied research, including
research that assesses the effects of particular policy changes or other interventions. This chapter
introduces students to the underlying logic of this type of work.
Most students taking sociology courses will not become professional sociologists employed by a
university. Those who stay within the social sciences will find work with government agencies
or nongovernment organizations. These organizations are often very applied, very policyoriented, and very concerned with being able to demonstrate clear effectiveness (or not) of
particular interventions to various stakeholders. This chapter may very well be the most
important one for students who want to seek employment in these types of fields.
Page 61
Students often find the material in this chapter confusing because, unlike survey research,
evaluation research is necessarily a method. It does have a rationale and logic of its own,
however, and evaluation research can employ many of the other methods discussed in this text.
To that end, evaluation research is more of an epistemological approach to research.
Additionally, the actual methods employed in evaluation research depend heavily on the issue
being studied and the particular type of intervention being invoked. As a result, it covers a very
broad array of methodological approaches.
As with most research method, a learn-by-doing approach is best for learning and understanding
evaluation research. For my own students, I provide the following vignette:
Dr. Harrison wants to find out whether changing the federal Employment Insurance leave
available for new mothers from one year of leave paid at 55 percent of one's wages to one
year of leave paid at 100 percent of one's wages will result in more women taking advantage
of the parental leave program. In small groups, design evaluation research that will help to
determine whether this would be the case. This includes designing the proper policy
intervention, determining on which group it will be tested, and deciding what methods you
will use to determine if the change likely would have the suspected effect.
After students finish the project in small groups, we discuss the groups' approaches together as a
class. Normally we find that they vary wildly in terms of how broadly the policy change should
be implemented as a test case, what groups should be studied in the evaluation, at what points in
time they should be studied, and so forth. This provides ample opportunity to discuss the
advantages and disadvantages to various approaches that could be employed.
In my course, I ask students on the final examination (as an essay question) to design their own
piece of evaluation research, on whatever topic or issue they choose. This must include:
1. Deciding on a design (experimental, quasi-experimental, etc.)
2. Deciding on variables
3. Specifying the intervention
4. Specifying the population
5. Deciding how he or she will know if the intervention has been successful
Although there are certainly other ways to assess student performance, if students can engage in
this process by creating their own evaluation research, it is clear they have learned the outcomes
presented above.
Page 62
Teaching evaluation research is difficult for many college and university professors, who tend to
do research that is more “pure” than “applied.” Even so, if students are able to grasp the
importance of social or policy interventions, as well as the necessity for using systematic
methods to evaluate success or failure of the intervention, the major objectives for teachers have
been met.
1. Students will want to know what type of career opportunities is available in applied research
fields. The following websites might provide some helpful information:
a.
b.
c.
d.
e.
f.
www.canadianresearch.org
www.publicservicecareers.org
www.idealist.org
www.careerjet.ca/social-science-research-jobs.html
http://ca.indeed.com/jobs?q=Social+Science&l=
www.servicecanada.gc.ca/eng/qc/job_futures/statistics/4169.shtml
2. Two excellent textbooks on evaluation research:
Daponte, Beth Osborne. Evaluation Essentials: Methods for Conducting Sound Research.
San Francisco: Jossey-Bass. 2008.
Clarke, Alan. Evaluation Research: An Introduction to Principles, Method, and Practice.
Thousand Oaks, CA: Sage. 1999.
Page 63
CHAPTER 13 QUALITATIVE DATA ANALYSIS
Chapter Summary
This chapter introduces students to the fundamental principles of qualitative data analysis. For
data derived from interviews, ethnography, or other qualitative methods, students should be
familiar with memoing, coding, and computer-aided data analysis.
Students should understand how to code qualitative data and search for patterns when analyzing
qualitative data.
1. Understand how social scientists analyze qualitative data, especially the iterative and
inductive nature of most qualitative analyses. [Understand]
2. Use the grounded theory method for analyzing qualitative data. [Analyze]
3. Code qualitative data, including the process of open coding used to establishing and
classifying ideas concepts and themes. [Apply]
4. Understand how social scientists use memoing to construct more valid analyses.
[Understand]
5. Conduct their own qualitative research project (i.e., interview research or ethnography),
when combined with the material learned in Chapters 10 and 11. [Create]
Most social science research produces such a large volume of data that it requires a systematic
way to sort, code, and analyze that data. This is very true of qualitative, as well as quantitative,
research.
Page 64
This chapter presents the logical extension of Chapter 10 (Field Research) and Chapter 11
(Qualitative Interviewing). Once qualitative data are collected, researchers need a systematic
method for engaging in analysis. The information provided in this chapter allows students to
engage in first-hand analysis of data collected through the qualitative methods that they learned
in earlier chapters.
1. Students will often assume that because qualitative data are nonnumerical, there is no
systematic way to analyze them. Instead, students will want to simply “eyeball” interview
transcripts. The content provided in this chapter can provide students with a method for
systematically analyzing qualitative data in order to reach conclusions that are both
reasonable and defensible, given the data collected.
2. Students will often have difficulty deciding on what concepts should be used to code their
data. To address this issue, it can be helpful to code a transcript together in class (see item
1 below), to show students what sorts of concepts they might want to learn more about
(although this does depend heavily on the topic of research being examined).
Several in-class strategies can be used to help students understand the basics of qualitative
analysis:
1. It may be useful to show students an example of a full interview transcript (several are
available on the Internet). Once the class has scanned the transcript, it is instructive to ask
them what concepts they see emerging from the transcript. When they identify a few
concepts (“alienation,” or “poverty,” for example) you can create a colour (in a word
processing program) for each code, and then highlight sections of the text in the
appropriate colour. This way, students see a very basic example of how text is coded
(without the use of more sophisticated computer programs such as Atlas.ti or NVivo).
2. My students often ask why transcripts contain signifiers for emotions and actions, such as
“(laughs)” and the use of brackets [ ] for inaudible talk. Students enjoy debating the
advantages and disadvantages of using these transcription devices, but normally reach the
conclusion that without them, we may lose track of the meaning that the participant
intended. For example, “(laughs)” at the end of an otherwise dry-sounding comment may
indicate sarcasm. Without that, the person doing the analysis of the data (who is not
always the same person doing the interviewing) may not understand that the comment
was intended to be sarcastic.
Page 65
3. After coding, I discuss with students how to actually go about analyzing data. One of the
easiest methods (if doing analysis in a word processing program) is to copy and paste
sections that are coded the same into the same file or portion of a file. This way, all
instances of “alienation” will be grouped together, allowing for easier analysis of this
particular concept in the final written document.
4.
Finally, I would stress that the best way for students to develop a feel for qualitative
methodology is to conduct their own interviews. In my course, I ask students to interview
three classmates about a topic of their choosing, using an interview guide that they have
written ahead of time (see the following section for more information). They also
transcribe, code, and analyze these data. Students consistently comment that interviewing
is best learned by doing, not simply by hearing their instructor lecture about it.
As part of the third assignment that I ask my own students to complete (see Assignment 3 in
Appendix C), I require them to conduct short interviews of three classmates on a topic of their
choosing. Students then transcribe (as well as they can, given that I do not permit them to taperecord the interviews), code for particular concepts/themes, and write up their findings and
conclusions. They do their transcribing and coding in a word processing program (for
simplicity), but if you have access to a qualitative data analysis program such as NVivo or
Atlas.ti, students will benefit from having some familiarity with these programs, as well. If
students can successfully code, analyze, and write an analysis based on their interview data, they
have successfully mastered the learning outcomes.
As with most topics covered in this text, the more insightful questions that students ask, the more
certain you can be that students comprehend the material. For example, when studying
qualitative data analysis, students will often ask questions such as “How can we be certain our
own biases aren't affecting the way we code the transcript?” This is by no means a simple
question for an instructor to answer, and can lead into complicated discussions of validity and
reliability, but questions such as these suggest that teaching has been successful—to be able to
critique qualitative analysis (as this question is implicitly doing) first requires an understanding
of it.
1. NVivo, a software program for coding and analyzing qualitative data:
www.qsrinternational.com/products_nvivo.aspx
2.
Atlas.ti, an alternative program for coding and analyzing qualitative data:
www.atlasti.com/index.html
Page 66
3. Transcript Divas Transcription Services is an example of a Canada-based transcription
service. Many of these services will take digital audio files and provide high-quality
interview transcripts. Students might benefit from learning about a service such as this:
http://transcriptdivas.ca
4. An excellent discussion of coding, with examples: Learning Domain, “Coding of Qualitative
Data”: www.learningdomain.com/Chapter8.Coding.pdf
Page 67
CHAPTER 14 QUANTITATIVE DATA ANALYSIS
Chapter Summary
This chapter introduces students to the fundamental principles of qualitative data analysis. For
data derived from interviews, ethnography, or other qualitative methods, students should be
familiar with memoing, coding, and computer-aided data analysis.
Students should understand how to code qualitative data and search for patterns when analyzing
qualitative data.
1.
Understand the purpose of the codebook for quantitative analysis and how it facilitates
data entry and analysis. [Understand]
2. Perform basic univariate data analysis, including central tendency, frequency
distributions, measures of dispersion, and continuous vs. discrete variables. [Create]
3. Understand multivariate data analysis and construct bivariate tables with percentaging.
[Understand /Create]
Analysis of quantitative data is very common in the social sciences and is perhaps one of the
easiest ways to generate findings that are reliable and generalizable. Further, it is perhaps the
most cost-and-time efficient type of analysis for demonstrating a relationship between two
variables.
Analysis of quantitative data is one of the key skills honed during the process of earning a social
science degree. It is also a skill set that is in demand by government and industry. For these
reasons, students should be keenly interested in learning the basics of quantitative analysis.
Page 68
1. The largest stumbling block for students learning quantitative analysis is undoubtedly fear
and apprehension. Many students are intimidated by mathematics and statistics, without
realizing that quantitative analysis requires excellent logical skills, but not necessarily
complicated mathematics.
2. Students also seem to have comparatively fewer problems devising and carrying out a
survey but more problems determining what to do with the data. In other words, research
design comes to students more intuitively than analysis. The challenge, therefore, is to teach
students how to do some simple quantitative analysis without overwhelming them.
There are several effective strategies for teaching the key ideas and concepts involved in
quantitative analysis:
1. First, it may be helpful to show students a cross-tabulation presented in a table to help them
understand how two groups can be compared. In the example table below, I present fictitious
data about respondents' intentions to wait until marriage to have sexual intercourse. So it
contains two variables: Intend to Wait (Yes/No) and Sex/Gender (Male/Female). This is very
similar to the bivariate analysis presented in Table 14-6 in the textbook.
Female
Male
Intend to wait
70%
40%
Do not intend to wait
30%
60%
Col. Total
(100%)
n = 1,653
(100%)
n = 1,740
Tables such as this allow the class to discuss several issues, including
a. Which variable is the independent variable and which is the dependent
b. Whether the independent variable should be presented in rows or columns
c. Which direction should the data be percentaged (across or down)
d. Which group is most likely to answer that they intend to wait until marriage (in this
case, females).
e. Why interpreting the first row or the second row is mathematically the same (i.e.,
because the two rows sum to 100%).
Page 69
Examples such as this also give the class an opportunity to discuss the issue of likelihood in
causal inference. Students can see very clearly that for gender to affect the intention to wait
for marriage, we do not necessarily expect every single female to intend to wait and every
single male to intend not to. Rather, students can see very clearly that patterns have
exceptions.
2. To illuminate the difference between the mean and the median (Figure 14-3), it may be
useful to discuss how the mean is influenced by outliers. To do this, I give students a small
set of income data (n = 5):
$20,000
$30,000
$50,000
$80,000
$100,000
I then ask students to calculate the mean ($56,000) and the median ($50,000). Then I ask
them what would happen to the mean if we added Bill Gates (income of $43,710,000,000) to
the distribution. Students quite quickly surmise that the mean increases drastically—so much
that it misrepresents the true “average” in the distribution. The median, however, changes
very little. For example, if the person earning $100,000 leaves the distribution and Bill Gates
enters the distribution, the median is unchanged ($50,000) even though the new mean would
be in the neighbourhood of $8 billion.
As part of my students' second assignment, I ask them to construct a survey, administer the
survey to the other 30 students in the classroom, enter data in Microsoft Excel (cases in rows,
variables in columns), and finally, create a cross-tabulation similar to the one in Table 14-6. I
also ask students to compute means and medians and to compare group means (for example, the
number of times that men and women go to the gym per month). If students can take raw data
and construct such a bivariate analysis, they have achieved the major objectives of the chapter.
If students are asked to perform analyses of their own, their successful calculation of means,
medians, modes, and their bivariate analyses should clearly demonstrate their mastery of the
material, and therefore, that the teaching of the material has been successful.
Page 70
1. An example of a bivariate analysis (cross-tabulation), with interpretation: Thomas Brown
and students. Hi-Tech Sociology, “The Adventure Begins”:
http://socquest.net/Q1/Q1ResEx/ResExQ1_print.html
2. A resource for calculating the mean, median, and mode (measures of central tendency):
Centre for Innovation in Mathematics Teaching, “Mean, Median, Mode, and Range”:
www.cimt.plymouth.ac.uk/projects/mepres/book8/bk8i5/bk8_5i2.htm
Page 71
CHAPTER 15 THE LOGIC OF MULTIVARIATE ANALYSIS
Chapter Summary
To avoid the problem of spuriousness and to recognize the complexity of the social world, most
analyses derived from quantitative data are multivariate. This chapter introduces students to the
logic of multivariate analyses and the basic strategies undertaken by researchers doing such
analyses.
Students should understand what it means to “control” for the effect of a variable when
analyzing the relationship between two other variables, as well as why this might be both
desirable and necessary.
1. Understand the elaboration model, which involves using data in order to establish
causality. This includes the use of a test variable (held constant), partial relationship
(spurious relationship), and a zero-order relationship (relationship when nothing is being
held constant). [Understand]
2. Understand replication, or the process by which a relationship between two variables
holds even after a test variable is added. [Understand]
3. Understand explanation, or a relationship between two variables that is “explained away”
by the test variable. [Understand]
4. Understand interpretation, which is a research outcome in which a test or control variable
is discovered to be the mediating factor through which an independent variable has its
effect on a dependent variable. [Understand]
5. To understand specification, which is when an initially observed relationship between
two variables is replicated among some subgroups created by the control variable but not
among others (e.g., among the elderly but not among children). [Understand]
6. Understand the basic model of measures of association, regression, statistical
significance, and chi-square tests. [Understand]
Page 72
Without conducting a multivariate analysis, it is difficult to tell whether a finding is simply a
result of spuriousness. That is, we cannot know whether the relationship is simply the result of an
omitted third variable. This chapter allows social scientists to make better causal arguments.
Although students do not often conduct their own multivariate analyses in an introductory
methods course (although some do), the content in this chapter will help them to be more critical
consumers of research. After taking this course, students should be able to read findings from
quantitative analysis, understand them, and critique them, and offer possible avenues for
improvement.
1. Students will often assume that because multivariate analysis is based on quantitative
data, they will need mathematical familiarity to be able to understand the material. As I
explain to students, multivariate analyses require reason and logic, but do not often
require a high degree of mathematical competency.
2. Despite its importance, students often struggle to grasp what it means, statistically
speaking, to control for a particular variable. It may be helpful to explain to them that
controlling for a third variable (that may partially explain the relationship between X and
Y) is akin to ensuring that any effect observed holds across categories of the control
variable. For instance, let us assume we are interested in the following empirical
relationship:
Educational Attainment (X1)  Income (Y)
If we want to find out how education affects income, there are other variables we first
want to take into account that could, theoretically speaking, affect both educational
attainment and income and could “explain away” any relationship between the two. One
of those variables (which we can call X2) could be gender. Therefore, by controlling for
gender, we are making sure that the effect of educational attainment on income is the
same for males and for females. If it is roughly the same, we can conclude that the
relationship holds. If it is not the same for men and women, we have evidence that the
relationship between education and income may have been spurious, as both are
predicted by or caused by gender.
Page 73
1. Given the potentially challenging material contained in this chapter, the best method for
demonstrating it to students is to talk through the examples presented in the text. For
example, Figure 15-3 demonstrates to students how relationships that seem to be apparent in
bivariate analyses can be altered by adding a third variable. It may also be useful to find
multivariate analyses published in existing research and to walk through this research with
students.
2. Students will often ask, with so many potential problems and explanation issues, how we can
have any confidence in the causal relationships outlined in existing research. Because this is
the chapter that asks students to begin critiquing multivariate analyses, it is also a good
juncture to introduce students to the peer-review process. So, I outline the process for them at
this point, including
a. Submission to a journal
b. Editor's decision regarding whether to send an article for review
c. Sent to two to four reviewers for double-blind review (can discuss advantages and
disadvantages of double-blind review). Review takes two to six months.
d. Reviewers recommend a decision, which can include: acceptance, revise and
resubmit, or rejection.
e. Author decides whether or not to resubmit (if invited)
f. Article accepted
g. Article sits in a queue waiting for publication
h. Entire process can take two years.
Students find this to be an interesting discussion. They understand how the process can
generate stronger research and can weed out problems in methodology or analysis, but at the
same time, they also argue that such a lengthy process impedes the timeliness of research
findings. I once had a student exclaim in exasperation, “That means we are always working
with findings that are more than two years old!” The student was disturbed that the creation
of knowledge was hampered by what she felt was an antiquated and overly restrictive review
process. Other students saw the value of the peer review process as being worth any resultant
delays in publication. Either way, this sort of debate is important to have as the course wraps
up and students begin considering what happens with the knowledge created using the
methodological approaches described in the text.
3. If instructors are asking students to collect survey data and to provide analyses of that survey
data, they may consider asking students to perform not only a bivariate analysis but also a
multivariate analysis using a control variable. Therefore, it is useful to walk students through
a sample bivariate and multivariate analysis in class. For example, if we use the example of
virginity pledges (pledging not to have sexual intercourse until marriage) as our independent
variable and actually delaying sexual intercourse until marriage (or not) as our dependent
variable, students might generate the following bivariate analysis.
Page 74
No Pledge
Pledge
No Delay
90%
33%
Delay
10%
66%
Col. Total
100%
100%
Here we see that those who took the pledge were indeed more likely to delay having sexual
intercourse until marriage. However, students can also expand such an analysis into a
multivariate analysis by including a control variable, such as gender.
Males
Females
No pledge
Pledge
No Pledge
Pledge
No delay
71%
0%
88%
33%
Delay
29%
100%
22%
67%
Col. Total
100%
100%
100%
100%
Here, by comparing males and females, we see an interesting pattern. Males followed
through on their pledge 100% of the time, whereas females followed through only 67% of the
time. So, although it is true that those who pledged were more likely to delay than those who
did not, it was indeed the males within the group that were the most likely to delay. In this
example, adding sex/gender as a control variable helps us to unpack the relationship further
and see that, in fact, men are more likely than women to follow through on the virginity
pledge. That said, students will also notice that the original bivariate relationship does hold,
even with the addition of the control variable: those who pledge tend to delay, and those who
do not take the pledge do not tend to delay.
The example in item 3 above is one that instructors could ask students to perform in an
assignment with original data that they have collected themselves. Indeed, it is similar to the type
of bivariate and multivariate analyses I ask students to do as part of their coursework (see
Assignment 2 in Appendix B), using survey data that they have collected from their classmates.
Page 75
If students are able to conduct such a multivariate analysis, they have learned the outcomes
presented in this chapter.
If students are able to understand the meaning of statistical control, which can be surmised from
examination questions, they grasp the most crucial learning outcome of the chapter. As an
alternative assessment strategy, instructors can have students design their own quantitative
analysis (see Assignment 2 in Appendix B) and can ask them to consider any other variables that
may partially explain their observed relationships. If students are able to think of variables that
should be included as control variables (and can explain why they should be included) they
maintain at least a basic understanding of the material.
1. An excellent description of control variables, with examples, from the Social Science
Research and Instructional Center, “Introducing a Control Variable (Multivariate Analysis)”:
www.ssric.org/trd/modules/siss/chapter3
2. A discussion of mediating and moderating variables from the University of Vermont,
“Mediating and Moderating Relationships”:
www.uvm.edu/~dhowell/StatPages/Methods/MedModCorrection.pdf
3. Probably the key text (a classic!) for understanding multivariate causal inference:
Mclendon, McKee. Multiple Regression and Causal Analysis. Long Grove, IL: Waveland
Press. 2002.
4. One of the best books for writing up findings, especially quantitative findings. It also
provides a scathing and rather humorous critique of the way social science research is
normally written:
Becker, Howard. Writing for Social Scientists. Chicago: University of Chicago Press. 1986.
Page 76
CHAPTER 16 SOCIAL STATISTICS
Chapter Summary
Researchers who collect quantitative data must be familiar with basic social statistics if they
hope to draw conclusions based on their data. This chapter introduces students to some very
simple statistics used by quantitative researchers, including basic descriptive and inferential
statistics. This includes measures of association, regression analysis, and an introduction to
statistical significance.
Students should at least understand the difference between descriptive and inferential statistics,
and which situations call for each type of statistical analysis.
1. Recall what a raw data matrix looks like, with cases in rows and variables in columns.
[Remember]
2. Create the appropriate table (cross-tabulation) to compare variables at different levels of
measurement (nominal, ordinal, etc.). [Apply]
3. Read and interpret a correlation matrix, similar to the one that appears in Table 16-5.
[Analyze]
4. Understand the logic of regression, including the basic regression equation (Y = bX + a)
and the regression line. [Understand]
5. Understand the meaning of statistical significance as it relates to the likelihood that an
observed relationship is due to sampling error. [Understand]
Social scientists must be able to decide on the proper method of analysis for quantitative data.
Many considerations go into this decision, including the level of measurement of the variables
involved (nominal, ordinal, etc.), the purpose of the analysis (description or inference), and the
number of variables involved (univariate, bivariate, or multivariate). This chapter provides
students with an introduction to some of the many statistical considerations that social scientists
must make.
Page 77
In many degree programs, students who complete the research methods course will also need to
complete a course in social statistics. Therefore, this chapter provides a bridge between the two
courses and will help students understand the logic of social statistics before taking any further
statistics courses. If a student’s degree program does not contain a statistics course, this chapter
provides students with a small window into statistical methods for analysis of quantitative data.
1. Many students will have difficulty differentiating between variables measured at the
nominal, ordinal, interval, and ratio levels. Without this knowledge, however, they will
be unable to determine which statistical methods can be used for analysis. They may
require more review of level of measurement in order to master the content in this
chapter.
2. Students will often struggle to understand statistical significance. This is primarily
because the term significant is used frequently in popular discourse as a synonym for
meaningful. Therefore, students often struggle to understand that significance has a
unique meaning when used statistically—as a way of gauging our confidence that an
observed relationship actually holds in the entire population and is not due to chance.
1. Regression analysis is difficult for students to understand unless they see and interpret
examples. For this reason, it may be useful to take a regression model from a journal
article, present it to students in class, and walk through the interpretations with them.
For this, I like to show students a regression table from the following article: Wharton,
Amy. “So Happy Together? The Impact of Gender Segregation on Men at Work.”
American Sociological Review, Vol. 52, No. 5, 1987: pp. 574–587.
Table 2 in the above article presents an ordinary least-squares regression model
predicting men’s satisfaction at work. Particularly, it helps us understand how gender
variables such as the gender composition of the workplace, having an employed or stayat-home spouse, and authority in the workplace contribute to men’s satisfaction with their
jobs. Students find this to be an engaging topic, and moreover, the regression coefficients
are easy to interpret and discuss.
2.
When discussing correlation (a type of social statistic used to understand the relationship
between two interval–ratio level variables), it is likewise useful to view and discuss the
example of a correlation matrix. One example appears in Table 2 of the following article:
Haney, Timothy J. “Broken Windows’ and Self-Esteem: Subjective Understandings of
Neighborhood Poverty and Disorder.” Social Science Research. Vol. 36, No. 3, 2007: pp.
968–994.
Page 78
3.
If instructors have access to a computer lab, it may be useful to run some simple
statistical procedures together as a class, with a set of very simple data provided by the
instructors (one that requires no recoding of variables or manipulation of data). This can
be done in Microsoft Excel or, if the lab has such a program, it can be doing using more
complicated statistical analysis programs such as SPSS. The simplest analyses to perform
are bivariate cross-tabulations and correlations, although running simple multiple
regression models is feasible as well. Students find this hands-on approach empowering,
although it is perhaps a lot to ask of students, given the sheer amount of content that must
be covered in a research methods course. Still, if proper parameters are provided (e.g., no
recoding necessary, a very small data set, and a small number of variables) it is
manageable. However, instructors should note that when performing regression in Excel,
the data set cannot contain any missing observations (it will generate an error message).
In more sophisticated programs such as SPSS, missing observations are deleted through
listwise deletion procedures.
Many instructors will not have access to data analysis software in an introductory research
methods course. So, given the limits of time and technology, it may not be fruitful to ask students
to calculate their own correlations or regression models. Rather, the best way to assess student
learning is to take such a table from an existing journal article (like the ones cited above) and ask
students specific questions about it. For example, in item 1 in the preceding section, an instructor
could ask students, “Are men who have stay-at-home spouses more or less satisfied with their
jobs than men who have spouses in the labour force?” Alternatively, on a final examination, an
instructor could ask, “Is job authority a significant predictor of work satisfaction for men?” The
first question tests their ability to determine the direction of a relationship, while the second
question tests their ability to determine statistical significance. Both questions allow instructors
to ensure that students can interpret regression (or correlation) output and therefore have
mastered the outcomes for the chapter.
Teaching higher-level statistical analysis procedures to an introductory research methods class is
one of the most challenging tasks that a methods teacher undertakes. Although students may not
have a complete and solid grasp of these procedures (especially how to perform them) by the end
of the course, they should develop a solid foundation that can be used for higher-level courses. If
students clearly understand the logic for multivariate statistical analysis (including the inclusion
of control variables), they most likely have the requisite skills to not only be more critical
consumers of research, but also to enroll in and succeed in higher-level statistics or research
methodology courses, either at the undergraduate or graduate level. This can be determined by
giving students tables such as the ones that appear in journal articles and requiring them to
interpret the results.
Page 79
1. The following is a good example of an article that uses multivariate statistical analysis
(regression) to explore a topic of interest to students. Although a bit dated, it is one of the
better articles that I have found for discussing regression and its uses:
Wharton, Amy. “So Happy Together? The Impact of Gender Segregation on Men at Work.”
American Sociological Review, Vol. 52, No. 5, 1987: pp. 574-587.
2. An article that provides a correlation matrix (Table 2) that might be useful for practice
interpretation:
Haney, Timothy J. “‘Broken Windows’ and Self-Esteem: Subjective Understandings of
Neighborhood Poverty and Disorder.” Social Science Research. Vol. 36, No. 3, 2007: pp.
968–994.
3. Columbia University, “How to Read a Regression Table”:
www.columbia.edu/~jfs2106/teaching/causality/readings/MeierRauchAppendix_small.
pdf
4. A discussion of how to best interpret a crosstabulation with a chi-square test: Michael,
Robert S. Indiana University. “Crosstabulation and Chi-Square”:
www.indiana.edu/~educy520/sec5982/week_12/chi_sq_summary011020.pdf
5. How to do linear regression in Excel: www.youtube.com/watch?v=ExfknNCvBYg
6. How to do regression in SPSS: www.youtube.com/watch?v=KNMaG5ceRw0
Page 80
Appendix A
Sample Assignment 1
R e s e a r c h De s i g n , C o n c e p t u a l i za t i on , a nd C a us a t i o n
In other courses you may have already designed and carried out an observation research project.
You may or may not have had control over the topic of your research. Here, however, you will
have a choice of a topic to move forward through the semester. To excel in this assignment, you
will need to answer each of the following questions in some detail. The assignment is
constructed in several stages. You must divide your sections into the sections that appear below,
including the corresponding letter (A, B, C, etc.).
A. Choose Concepts/Ideas
The concepts shown below are not perfect (they’re intentionally imperfect)—they are meant as
guides. If you have an alternative topic, please run it by me before proceeding with it. This
project must be something you can do by surveying/interviewing students on the our campus
(e.g., projects involving elderly people or children would be impossible). You will design it with
the assumption that you will be studying students on campus, although in actuality, your
classmates will be providing data for your analyses. You’ll begin by picking a topic using the
ideas below or an alternative that you’ve run by me. Given a number of limitations, some topics
may simply not be feasible. These are just ideas—you’ll have to refine, reword, and improve
them.
1. Parents’ educational attainment
2. Academic success
3. Gender
4. Opinions on the ideal division of domestic labour
5. Concern about the environment/climate change
6. Challenges of holding a full-time (or part-time) job
7. Family income
8. Political values
9. Treatment by employers
10. Career goals
11. Use of Facebook
12. Amount of studying
13. Time spent commuting
14. Use of public transportation
15. Getting the flu vaccine
16. Rent or own a home
17. Mode of transportation (bike, walk)
18. TV watching
19. Alcohol consumption
20. Cellphone use
21. Amount of sleep per night and grades
22. Community involvement or volunteering
Page 81
23. Purchasing of products made in Canada/the United States versus elsewhere
24. Hired help in the home (nannies, babysitters, and other domestic employees)
25. Time expected to finish bachelor’s degree
26. Where food is purchased (chain, locally owned, farmer’s market, etc.)
27. Views on political parties (Conservative, Liberal, NDP, Green, Bloc Québécois, Wild
Rose, other)
28. Post-university moving plans
29. Views on the oil and gas industry in Alberta
30. Gambling
31. Leisure activities
Here, you might simply say “I’m interested in the relationship between {concept} and
{concept}.” You might also add a couple sentences about what attracted you to this topic. Note
that these are just suggestions. There are many other concepts from which you could choose.
B. Research Question(s)
Derive an appropriate
1. Originating question
2. Research question
3. At least TWO specifying questions
C. Theory
Using the concepts you have laid out, discuss the theoretical framework that you are using. How
and why are your concepts connected? Here you should draw on your textbook, or alternatively,
from sociology texts available at the university library. You could draw on larger theoretical
schools or paradigms (poststructualism, feminist theories, symbolic interactionism, etc.) or you
could draw on theories specific to your research question or sub-area (gender/feminist theories,
environmental theories, etc.). In short, you should demonstrate how your question and your
research are part of a bigger picture.
D. Variables and Attributes
Name the variables that you would use in your analyses and list the possible attributes for each
variable. You could consider using the layout shown below.
Variable
Attributes
Variable 1
Attribute 1
Attribute 2
Attribute 3
Variable 2
…etc….
Attribute 1
Attribute 2
…etc…
Page 82
E. Operationalization
When you ask questions of participants, how will you measure each of your variables? (For
example, what sorts of things would you actually ask people?)
F. Hypotheses
Write a couple hypotheses using your variables and derived from your theory. Label your
hypotheses as follows:
H1:
H2:
In each hypothesis, underline your independent variable and boldface your dependent variable
and put your unit of analysis in italics.
G. Establishing Causation
1.
Would you adopt a longitudinal, cross-sectional, or experimental design to answer
your questions? Why?
2.
Would it lend itself better to quantitative or qualitative data analysis? Why? Why not
the other?
3.
Does your proposed project meet the three criteria for causation? How can you be
sure? Are there any of these criteria that you cannot yet be sure of? Make sure to list
the criteria and discuss how each does or does not meet these benchmarks.
4.
Would this project take a deterministic or probabilistic approach to establishing
causation? Would it try to establish causation at all? Explain.
5.
Would this project likely take a nomothetic or ideographic approach to explanation?
Why?
6.
Would you be using inductive or deductive logic in your research? What would be the
advantages and disadvantages of each approach?
H. Epistemological Foundations
Imagine you are going out into the field to speak with people and to attempt to answer your
questions. You will be surveying or interviewing people (right now it doesn’t matter which one).
It’s important first to consider issues of what you can observe, what is real, and the political
Page 83
implications of your research. To do so, you’ll need to answer the following questions. You do
not need to label them as I do below (1, 2, etc.), but you will need to make sure that each one is
answered, and it will be helpful if you address them in roughly the same order:
1.
What sort of epistemological approach best suits your research question?
2.
Would this research project necessarily be positivist? Why or why not?
3.
Would this research project be empirical in nature? Why or why not?
4.
Could you undertake this research in a value-free way? Is that possible? Why or why
not? If not, what sorts of values, biases, or preconceptions may affect this research?
And how might they affect it? Would you take a position of conscious partiality?
Why or why not?
Page 84
Appendix B
Sample Assignment 2
S a m pl i n g a n d Q ua n t i t a ti v e D a ta Co l l e c t i o n a n d A n a l y s i s
In your previous assignment, you began designing a research project. Now you will carry it out.
Note that there are several parts to this assignment, and Part A will be due [AT LEAST ONE
WEEK BEFORE THE FULL ASSIGNMENT] in class.
The final assignment that you submit on that date should include the sections that I list
below, although Sections A and E will be separate printouts.
A. Survey/Questionnaire Form
You will design a survey/questionnaire form appropriate for the topic and hypotheses you
proposed last week. This form will demonstrate your skill at question construction and ordering.
You will bring 30 copies to class on [DATE] to have your classmates fill out. Other directions
are as follows:
1.
2.
3.
4.
5.
You will be required to include about 8 questions (sub-questions count as one question).
You must use a combination of open-ended and closed-ended questions.
Your name must appear at the top of the form.
You must use at least one Likert scale
You must follow the guidelines for effective question construction and ordering laid out
in the textbook and in class lectures.
6. You must include questions that measure each of the variables you said you would use in
Assignment 1, including any variables that you thought might affect the relationship
between your independent and dependent variables (e.g., any variable that would cause a
spurious relationship).
You will bring 30 copies to class on [DATE], but you should also turn in a blank copy with your
final assignment on [FINAL DUE DATE].
B. Consent Form
Create a consent form that your participants (assuming you were administering this to students
on our campus) would be asked to sign. Be sure to include each of the necessary elements that
we discussed in class. Also make sure that you have addressed the applicable ethical issues
discussed in class. Also, consult the chapter on ethics in your textbook.
Page 85
C. Sampling Strategy
Assuming you were going to undertake this project on our campus, devise an appropriate
sampling strategy, making sure to note:
1. Which strategy you are using
2. What you would use as a sampling frame
3. How you would select elements from that frame
4. What your elements would be
5. Why this strategy is preferable to all other possible sampling strategies
6. What the disadvantages are to this strategy (if any)
Note: There is no “right” or “wrong” sampling strategy, but you must be able to defend why you
chose the strategy you did, and you must also discuss the weaknesses to that strategy. Simply
saying “I’m using convenience sampling” will earn no credit. Also note that convenience
sampling cannot be random.
D. Codebook
Determine how you will turn your categories/attributes into numerical data. For example,
“Strongly Agree” might become “5” and “Agree” might become “4,” etc. For Yes/No questions,
the usual format is 0 for No and 1 for Yes.
The best way to do this will probably be to take a blank copy of your survey/questionnaire
form and label each possible response with a number (see the example I have provided). For
open-ended questions, you will have to devise a strategy to place responses in categories and
then assign numbers to those categories. Make sure to explain how you made these decisions.
E. Data Entry
Enter the data into Microsoft Excel by including variable names in the top row. Your first case
(person) then becomes the second row, and so on. You will need to turn your questions into short
(about 8-digit) variable names. So, “opinion on same-sex marriage” could become “SameSex.”
Your table will look something like the following.
CaseNum
1
2
3
4
5
6
…
Gender
0
1
0
0
0
1
…
SameSex
1
3
4
2
1
0
…
PolViews
2
3
1
5
0
1
…
Please print your raw data in table form and hand it in with your assignment.
Page 86
F. Analysis
Now you’ll need to analyze your data. To do so, use the manipulations that we practised in class.
You should include at least one bivariate table to help you establish whether there is a
relationship between your independent and dependent variables. To answer other questions
(including your specifying questions) you might need additional analysis. This does not need to
be mathematically complicated—calculus is not required. Averages and percentages are fine.
You are the expert on your topic, so use the analyses that you think are appropriate for testing
your hypotheses and answering your questions. Keep in mind that I’m looking for creativity and
ingenuity, not perfection or mathematical genius.
G. Conclusions
In this section you should discuss whether you have found evidence that supports, or does not
support, each of your hypotheses. Make sure to reference specific findings that either lend
support or do not.
You should also address the extent to which these findings answer your questions
(particularly specifying questions and research question). You might also briefly discuss how it
helps us to begin to understand your originating question.
Lastly, you should discuss the limitations of your study. What problems did or do you have?
How could this study be improved? What problems arose that you did not anticipate?
Page 87
Appendix C
Sample Assignment 3
I n t e r v i e w Re s e a r c h a n d Q u a l i t a t i v e A n a l y s i s
In your previous assignment, you proposed and completed a survey research project. Now you
will have to adapt your topic to be a qualitative (interview) project. You will be interviewing
three classmates, although once again we will conduct the project as though you were recruiting
your participants from the larger student body.
Note that there are several parts to this assignment, and Part B will be due [DATE AT
LEAST ONE WEEK BEFORE THE FULL ASSIGNMENT’S DUE DATE] in class. The
full assignment will be due in class [DATE].
A. Reformulation of Research Questions
It should be evident from reading and lectures that qualitative researchers ask questions that are
very different from those asked by quantitative researchers. Accordingly, you’ll need to revise
the research questions that you proposed in the first assignment so that they are more appropriate
and useful for qualitative inquiry. (Note: Since Assignment 1 didn’t specify which sort of
questions you should create, some of your questions may already be useful for qualitative
inquiry, but you’ll need to make that decision). Your originating question probably will not
change, but it is likely that your research question and specifying questions will change. Either
way, make sure that your questions are worded so that they are appropriate for qualitative
inquiry.
Also, provide a short explanation why your new questions are more appropriate for
qualitative inquiry than your previous ones, drawing on both lectures and the textbook.
B. Interview Schedule or Guide
You will construct an interview schedule (e.g., a list of questions, including probes) and bring it
to class on [DATE]. Use the guidelines in the text and lecture notes to do this. In particular,
focus on the avoidance of leading questions. Use a combination of different types of question
(introductory, main-topic-related, process, and prompts/probes for each).
When you interview your classmates, try to write down everything they say. Direct quotes
are helpful when there’s something particularly useful that is said. Otherwise, rudimentary notes
will allow you to capture as much content as you possibly can—but you’ll have to write quickly!
Invariably, you will miss some things that are said (people can talk much faster than you can
write), but after the interview is over, try to fill in the missing pieces.
C. Ethics
In the previous assignment, you created a consent form that you would distribute to participants
if you were carrying research out on campus. You should discuss how and why your proposed
research will not violate any key ethical concepts. You can draw from the Tri-Council Policy
Page 88
Statement (posted on the blackboard), from the textbook chapter on ethics, and from class. Given
your topic, your sampling strategy, and other aspects of your project, what are the key ethical
issues and principles that you will need to work thorough? What sort of safeguards
can/will/would you take to avoid any potential ethical problems in your proposed study? Again,
assume you are interviewing students on campus (whom you do not know), not your classmates
(whom you actually will be interviewing).
D. Reflect on Interviews
Qualitative research is much more reflexive and dialectical than quantitative research. It is
important to reflect on your experience so that (if you were going to complete more interviews)
you could address any problems that crop up and refine your approach. Here you should discuss
the strengths and weaknesses of the interviews that you conducted with classmates. (Do not
name classmates by name. You should make up pseudonyms for each). In reflecting on your
interviews, you may want to utilize the following questions:
1. What observations might you have left out of your notes?
2. Were you able to establish rapport with your interviewees? With some more than others?
3. Were there any questions that participants had trouble answering? Did you need to use
any probes?
4. Would you say that you conducted a structured, semistructured, or unstructured
interview?
5. Did the setting or surroundings affect the interview at all? (For example, noise?)
E. Transcription
Type your field notes into a word processing document. You’ll probably find it easiest to keep
all three interviews in the same document. So, you might start with a heading for “Jane Doe” (or
whatever you call your first participant), include all notes for Jane, and then include a new
subheading for “Samantha Smith.” While you should include everything you wrote down, also
include anything that you remember but didn’t have time to write down initially (you can/should
make note of the fact that this was recalled after the fact). The sooner you transcribe after your
interviews are conducted, the better. Also, it benefits you to start early because this step can be
more time-consuming than one would expect.
F. Coding
Next you’ll want to locate patterns in the data. You will choose a colour for each concept or idea
that you’re interested in and that applies to your research question. Then you will use the
“Highlight” function in a word processing program to highlight each corresponding piece of data
in that colour. For example, “environmentalism” could be assigned green, so you will go through
your notes and highlight each item in green that relates to environmentalism. You can and should
also use the “Comment” feature in the word processing program to insert ideas into the text that
Page 89
relate to your analysis. If something in the notes reminds you of something that another
interviewee said, you can make note of that using the “Comment” feature.
Along with your assignment, please hand in your coded and transcribed notes, including any
inserted comments. If you print in black and white, try to note (using a highlighting pen) which
concepts/ideas are in which colour.
G. Analysis and Theory Generation
In this section, you will analyze your data. What patterns did you see that addressed your
research question(s)? Although you did not have to write formal hypotheses, did you find
anything that you either expected to find or that surprised you?
Most importantly, this is the section where you should and must use your observations (i.e.,
your interview data) to create or modify a theory about how the world works or about why
people do/say the things they do. This doesn’t require outside research because you are
generating theory inductively.
H. Conclusions and Limitations
1. What were the main conclusions and the main theoretical insights that can be drawn from
this project?
2. How were you able to avoid the five common mistakes or problems in qualitative
research? Or were you able to avoid them at all?
3. Are you able to generalize your findings to any other groups of people that you did not
actually talk to? If so, which groups?
4. What insights did you gain from this research (if any) that a quantitative approach
(survey/questionnaire) could not have given you?
5. Are there any insights that you may have gained by watching individuals (as in
ethnography) versus simply interviewing them? What would be the advantages and
disadvantages of each approach?
Page 90
Appendix D
Sample Assignment 4
C o n t e n t A n a l y s i s a n d Di s c o u r s e An a l y s i s
In the previous two assignments, you completed a quantitative analysis and a qualitative analysis
(congratulations!). Now, you’ll perform unobtrusive research. Moreover, in the first three
assignments, you proceeded as though you would be collecting data from the student body. This
assignment is different. Here, you will take your topic (Facebook use, commuting, etc.) and will
expand it to examine some form of written, spoken, or mediated communication.
This assignment will be due on [DATE] in class (during our final class meeting)
The assignment has two parts, a content analysis portion and a discourse analysis portion,
although you will use the same data source for both portions. Please divide your assignment just
as I have divided the one below (using roman numerals and letters).
The first thing you’ll need to do is to pick a medium of communication that will allow you to
analyze your topic. This might include Health Canada press releases, Facebook advertisements,
Family Guy episodes, magazine advertisements, Stephen Harper speeches, or any other medium
of communication that you believe will help you answer your questions and address your
research topic.
I. Content Analysis
A. Reformulation of Research Questions, Creation of Hypotheses
Much like in Assignment 3, you will need to reformulate your research question and
specifying questions to make them appropriate for the new type of data that you will be
analyzing. Also, create one or more (probably more) hypotheses or expectations.
B. Operationalize
Define the concepts you use in your hypotheses. For example, if you believe that the text
of a speech will make more references to war than to peace, define what you mean by a
reference to “war” versus a reference to “peace.”
C. Create a Codebook
Your codebook should have variable names and numerical codes for each possible
value/attribute.
D. Choose Your Parameters
Which media/speeches/ads/programs will you have to exclude (because of
access/availability or other limits)? Why have you made each of those decisions?
Page 91
E. Sampling
How will you choose which media/speeches/ads/programs you will include in your
analysis as part of your sample and which you will leave in your population (and not
actually study)? Is it possible to do this randomly, or is a convenience sample necessary.
Will your strategy need to be multistage cluster sampling, or will this not work in your
case? Make sure you justify why the strategy you’ve chosen is the best one possible.
F. Create a Data Recording Sheet
Just like in the quantitative analysis (Assignment 2), you should include case numbers in
rows and variable names in columns. When you have collected data, include this
completed sheet with your assignment.
G. Analyze Data
Analyze your data quantitatively. What are the numerical patterns that emerged out of
your data? For example, did you find that newspaper articles about the Haiti earthquake
used the term “chaos” 64 percent more often than they used the word “community”?
What do these numerical findings reveal about the nature of your topic? How do these
findings address your research question and contribute to our knowledge of the issue at
hand?
II. Discourse Analysis
Go back through your data sources and analyze each one closely, looking for the
following elements. You may not have time to view each article/image in-depth. If that’s
the case, pick a few that you find particularly compelling. You do not need to worry as
much about drawing a random sample here—pick the instances that are problematic,
interesting, or important.
A. Intended Audience
Who is the intended audience for these speeches/advertisements/programs, etc.? What
clues make you believe this?
B. Creator
Who is the creator of this type of discourse? In terms of power/wealth, what is the
position of the person/group who is creating this message? Logically, what would the
creator’s objective be? Is the creator of the message or discourse in a more powerful
position than the intended audience?
Page 92
C. Message
What message do the words/images convey? Discuss this at some length, analyzing the
intricacies of what is being said or conveyed.
D. Latent Content
What latent (hidden) content emerges from the words? What is being said or conveyed
that might not be visible unless you were thinking hard about the meaning? This is your
opportunity to deconstruct the discourse and analyze it with a fine-tooth comb. What are
the liberating or oppressive elements that are emerging from this discourse? What effect
will or does this discourse have on its intended audience? How would somebody in a
different position (with more or less power) create a discourse that is different from the
one you are viewing?
Page 93

Enriched Instructor’s Manual NETA ENGAGEMENT to accompany Prepared by

Transcription

Similar documents

2011 - The Couchiching Conservancy

AVS24 2.4 GHZ Wireless Audio/Video Sender

Our proud cat‐ski operators: Baldface, Selkirk Wilderness, Valhalla

here - Nelson Mandela Bay Business Chamber

HEMINEVRIN - Alcohol and Alcoholism

Copyright page - Katherine Don

No trophy could ever fully express our

Chaise Lounge