Casillas. (2015). - Claremont Graduate University

Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Claremont Graduate School Professional Development
team
 Collaborative work from lectures, co-presentations,
research grants, published papers:






Claremont Evaluation Center's
Professional Development
Workshop Series in Evaluation and
Applied Research Methods 2015
National Science Foundation
American Evaluation Association
CDC/AEA Summer Institute
Aotearoa New Zealand Evaluation Association
Culturally Responsive Evaluation Assessment (CREA)
Many others CRE practitioners
Wanda D. Casillas, Ph.D.
[email protected]
Montclair State University and
Deloitte Consulting, LLP
AGENDA
9:15 a.m. –10:45 a.m.
Workshop:Overview and discussion of cultural selves
 Interactive
 Opportunities to practice ways of speaking and thinking, challenging
workshop issues
 Reflective
 Opportunities to think about these workshop issues over the day
10:45 a.m. –11:00 a.m. Break
11:00 a.m. –12 noon
Workshop: Thinking about culture in evaluation
12 noon –1:30 p.m. Lunch Break
1:30 p.m. –3:00 p.m.
Workshop: Being culturally responsive in evaluation
 Prospective
 Opportunities to think about workshop issues further and work with
presenters as interested
3:00 p.m. –3:15 p.m.Break
3:15 p.m. –4:45 p.m.
Workshop Conclusion
1
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Recognize the relevance and value-addedness of culture in
evaluation theory and practice
 Discuss the evaluation process in the context of working with
underserved communities, in particular
 Identify strategies for assisting evaluators and agencies in becoming
culturally responsive
the way of life of a group of people, the complex of
shared concepts and patterns of learned behavior that
are handed down from one generation to the next
through the means of language and imitation.
(Barnouw, 1985)
the ever-changing values, traditions, social and political
relationships, and worldview created, shared and
transformed by a group of people bound together by a
combination of factors that include a common history,
geographic location, language, social class, and
religion…
(Nieto, 1999)
 Demographic, sociopolitical, contextual dimensions
 Characteristics as dynamic, multifaceted, learned,
created, influenced
 More than about race and ethnicity but inclusive of other
identifications (SenGupta, Hopson & Thompson-Robinson, 2004;
Nieto, 1999)
2
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
Instructions: Consider the most prominent cultural roles that you play.
Use the handout provided to record all of the roles that come to mind
(feel free to record more than 7 by creating more circles if needed).
Use the questions below to help you think about your roles.
Mother
How do you think about
yourself?
MexicanAmerican
Evaluator
Military
Wife
 How do you identify yourself on a daily basis?
 What are some ways you would describe yourself?
 How might others perceive you?
Scholar
Student
Woman
Think about all of the cultural
roles you play and how
playing these roles define how
you perceive yourself.
Identify influences in your
world view.
How do you think your world
view influences what you do
personally? Professionally?
1) What questions an evaluator asks and ultimately does not ask
 How you define yourself is the template by which you organize the
world
2) What an evaluator illuminates and ultimately minimizes
3) What evaluation approach is used and ultimately not used
 Culture and one’s cultural background influence our template(s)
4) What data are collected and ultimately overlooked
 We self-reference this template to help us make decisions in our
5) How interpretations are made and whose interpretations are held in
high or low esteem
everyday lives, and in our professional practice
6) What conclusions are drawn and what conclusions are not
considered
7) How results are presented and to whom such results are
disseminated.
3
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Summary of disciplines of workshop participants
 Summary of experiences of workshop participants
 Part I: Talk through theory and perspective of culturally
responsive evaluation
 Part II: Review principles or tenets that surface from concept
mapping study and literature review on CRE
 Part III: Introduce strategies for implementing principles and
theory
“What has frustrated me in the ways
multicultural programs have been evaluated is
that the people who do the evaluation generally
do not understand the nature of multicultural
work...The evaluators and their evaluations
often miss the point of what the program is
about and use inappropriate standards on
which to interpret the program on which to
make value judgments” (Stockdill, 1992:17)
4
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Role of primary inclusion of participants:
 “As Lincoln (1991) points out, most people who
evaluate social programs know very little about the
minority program participants’ world view, the
appropriateness of program interventions in meeting
their needs, or programs’ personal consequences for
these clients (Madison, 1992).
8/12/2015
 Using one set of methodologies in one cultural context
and applying to others:
“...evaluators must exercise great caution in trying to
apply the methodologies, models, and categories
devised in and for the developed world in Third World
countries...different views of reality and the nature of
change lead to different assumptions about appropriate
goals, treatment, and evaluation models (Cuthbert,
1985).
 Evaluation in rudimentary form in developing countries, even among
 Culture is either not taken into consideration or it is unknown how to
 Evaluation as auditing versus examination of how useful and/or
 Culture seems to be something to learn so that one can simply get
international organizations
appropriate a certain program is for the local community
 Western ideas of what "should" happen become the default priority
for organizations
be successfully taken into account
the evaluation done; it’s not really considered
 Often include cultural communities as passive participants
 Difficulties understanding the population's ways of thinking, living,
and understanding
5
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Continued flurry of evaluation discussion in conference meetings,
 Social location of evaluator
matters: lived experiences shape
assumptions and frames of
reference in evaluation process
 Avoiding ethnocentrism
periodicals, and special meetings, BUT:
means embracing multiple
cultural perspectives
 Limited understanding and consideration in evaluation
 Culture is central to the
mainstream
evaluation process
 Evaluators play roles in furthering
social change and justice: we are
“more than technicians” but have
duty to recognize power relations,
challenge systems of inequity
 Culturally and ethnically
diverse communities have
contributions to make in
redefining field of evaluation
 Growing, but still lack of courses and training opportunities that
pay attention to cultural context, relevance, responsiveness,
competence in evaluation
(Hopson, 2003)
 Approach/model used to guide evaluation
 System and culmination of evaluation principles
and strategies
 Theoretically and politically positioned
 Privileging lived experiences, esp. communities and
populations of color
 Avoiding the phenomenon of “evaluating down”
6
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
Decolonizing/ indigenous
positions, epistemologies,
and frameworks
Critical theories and
epistemologies of race
Social agenda and advocacy
theories, models and
approaches in evaluation
8/12/2015
A set of academic and interpersonal skills that
allow individuals to increase their understanding
and appreciation of cultural differences and
similarities within, among, and between groups.
This requires a willingness and ability to draw
on community-based values, traditions, and
customs, and to work with knowledgeable
persons of and from the community in
developing focused interventions,
communications and other supports.
(Orlandi, 1992)
(Hopson, 2009)
Cultural competence is a stance taken
toward culture, not a discrete status or simple
mastery of particular knowledge and skills. A
culturally competent evaluator is prepared to
engage with diverse segments of
communities to include cultural and
contextual dimensions important to the
evaluation.
Understand and recognize the larger context for programs or projects
Research and learn about the cultural group
Be aware of cultural labels and historical context
Identify potential historical inaccuracies
Design evaluation with participants in mind
Be culturally-specific in design
Use a multifaceted approach and appropriate methods
Collect data in culturally responsive ways
Allow for self-determination by stakeholders and program participants
Engage directly with participants through discussion
Engage stakeholders in general planning and in theory development
Public Statement on Cultural Competence in Evaluation
(American Evaluation Association, 2011)
Casillas, Hopson, & Gomez (2015)
7
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Debrief on definitions, influences of CRE
Build trust and facilitate communication
Allow for representativeness
Build diversity of organization/evaluation team
Access diversity from external sources
Being inclusive of diversity
 Q&A
Understand the evaluation audience and help the audience to
understand the evaluation purpose and process
 Time check
 Getting ready for Exercise #2 – outside experts
Make the evaluation accessible
 Handout and directions
Understand evaluator attributes that may affect professional practice
Casillas, Hopson, & Gomez (2015)
CRE
Framework
2
Engage
stakeholders
(adapted from Frierson, et.al, 2010)
1
Prepare for
the evaluation
9
Disseminate and
use the results
3
Identify purpose
of the evaluation
4
Frame the
right questions
Cultural
Responsive
5
Design the
evaluation
8
Analyze
the data
7
Collect
the data
6
Select and adapt
instrumentation
32
8
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
Typical
evaluation
activities
Culturally responsive
activities
Options for implementing CRE behaviors
Informant interviews
Group discussions
Feedback sessions
Engage Stakeholders
Analyze context
Explore communication styles
Prepare for
evaluation
Identify purpose
of the evaluation
Gain trust
Facilitate stakeholder ownership of the
evaluation
Multi-ethnic
Assemble evaluation team
Shared lived experiences
Informants
Acquire foundational
Interpreters
knowledge
Cultural guides
"critical friends of the evaluation"
Examine program theory for cultural
Process evaluation
sensitivity
Examine appropriateness of program goals
Progress evaluation
for population
Determine the effects of program
Summative evaluation
implementation on participants
Consensus on evaluation
purpose
8/12/2015
 Be informed by the sociocultural context of the evaluand,
including
 History
 Formal and informal power relationships
 Communication and relational styles
 Informant interviews, group discussions,
sessions
feedback
 Assemble an evaluation team whose collective lived experience
fits the context of the evaluand.
 Evaluator awareness of own
cultural values,
assumptions, prejudices, stereotypes
 Not merely about matching demographics
 Multi-ethnic, informants, interpreters, cultural
guides, “critical friends of the evaluation”
Derived from Frierson, Hood, Hughes, and Thomas (2010)
 Develop a stakeholder group representative of the population
served by program.
 Seek to include persons impacted by the program directly and
indirectly.
 Pay attention to issues of power, status and social class.
 Include multiple voices in meaningful preparation process and
activities.
 Create climate of trust, respect.
 Create Stakeholder Map (handout)
 Document, examine program implementation
 How well is the program connecting with its intended
consumers?
 Is the program operating in ways that are respectful of
cultural context?
 Are program resources equitably distributed?
 Document, examine progress toward goals
 Who is benefiting from the program, and are these
benefits equitably distributed? Who is burdened by the
program?
 Is program theory culturally sensitive?
 Evaluate overall effectiveness
 Capture cultural nuances
 Examine correlates of participant outcomes
9
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Use the CRE Framework to assist in design
 Reflections on embedding culturally competent
planning
evaluation in Indian Country
 Build understanding of values that underlie
programs and projects and create value-added
evaluative contribution
 Engage stakeholders in participatory manner
 Reflect and work as team to develop key
components of design
 Build ethic of participation and capacity building that values
community, relationships, respect
 Frame purpose by building conceptual picture/model
 Careful of “too sequential and narrative driven” logic
model
Typical
evaluation
activities
 How do your designs incorporate the first three steps of the CRE
 What challenges and opportunities do you see in integrating the
CRE framework?
Engage Stakeholders
framework?
Frame the right
questions
Design evaluation
Select/Adapt
instruments
Culturally responsive
activities
Options for implementing CRE
behaviors
Determine appropriate type of
evidence
Critically question evaluation Does the question limit results of the evaluation?
questions
Is there another possible approach to take?
How might different evaluation questions result in
different understandings of the program?
Identify appropriate design
Mixed method often yield best results
Data collection at multiple time points
Decide to identify, develop,
and/or adapt existing measures
Pilot test for appropriateness to
population
Translate when necessary
Forward-backward translation
Translation by committee
Multiple forward translation
Check instruments for semantic/content
equivalence
Derived from Frierson, Hood, Hughes, and Thomas (2010)
10
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Questions of relevance to significant stakeholders
 Determine what will be accepted as evidence
 Notice whose voices are heard in the choice of
questions and evidence.
 Reflect on how questions limit what can be
learned and how they might be posed differently.
 Notice how different questions may expand
understanding. Revise and refine questions.
 Can questions be answered with available
resources?
 Is there another possible approach to take?
8/12/2015
 Build design appropriate to both evaluation
questions and cultural context.
 Seek culturally appropriate mixed methods,
combining qualitative and quantitative
approaches.
 Try to collect data at multiple points in time,
extending the time frame of the evaluation as
needed.
 Construct control or comparison groups in ways
that respect cultural context and values.
 Identify, develop or adapt instruments for the local
context.
 Establish evidence of reliability and validity.
 Language and content of instruments should be
culturally sensitive.
 Use best translation practices, validating both
semantic and content equivalence.
 Forward/backward (FBT)
 Translation by committee (TBC)
 Multiple forward translation (MFT)
 Norms must be appropriate to the group(s)
involved in the program.
 Evaluation of undergraduate STEM research
program designed for students of color
 Deliberate design of evaluation team intimately
connected with program of study and background of
program, including similar lived experiences of
participants
 Questions were sensitive to lived experiences of
participants and focused on substance of participant
experiences
 Beyond attention to traditional issues of success but exploring
issues of persistence as students of color
 Attempt to address nuances and subtleties relative to
experiences and impact of program
11
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
Typical
evaluation
activities
 How do your designs incorporate the second three steps of the
 What challenges and opportunities do you see in integrating the
CRE framework?
Options for implementing CRE
behaviors
Storytelling
Chronicles
Parables
Poetry
Observation
Interviews
Focus groups
Revisionist histories
Qualitative
Engage Stakeholders
CRE framework?
Culturally responsive activities
Collect data
Quantitative
Training "people as instruments"
Determine accurate interpretations
Analyze Data
Multiple strategies for analyzing quantitative data
Results and
dissemination
Know what is being said/observed
Nonverbal Behaviors
Videotaping and reviewing
Know language
Know context
Share lived experience
Disaggregate variables
Cross tabulations
Community review results for "relevance gaps"
Widely disseminate
Use multiple modalities
Clearly present data
Report in modes appropriate for the "right people"
Derived from Frierson, Hood, Hughes, and Thomas (2010)
 Procedures used to collect both qualitative and
quantitative data must be responsive to cultural
context. E.g. storytelling, focus groups,
chronicles, interviews
 Nonverbal as well as verbal communications
provide keys to understanding.
 Train data collectors in culture as well as
technical procedures, videotape and review
 Recognize how cultural identifications of the
evaluation team affect what they can hear,
observe.
 Shared lived experience provides optimal
grounding for culturally-responsive data
collection.
 Understanding cultural context is necessary for
accurate interpretation.
 A cultural interpreter may be needed to capture
nuances of meaning.
 Stakeholder review panels can more accurately
capture the complexity of cultural context,
supporting accurate interpretation.
 Disaggregate data and cross-tabulate to examine
diversity within groups.
 Examine outliers, especially successful ones.
 Remember that data are given voice by those who
interpret them- shared lived experience, shared
language
12
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 Cultural responsiveness increases both the
truthfulness and utility of the results.
 Maximize community relevance of findings; invite
review by community members prior to
dissemination.
 Communication mechanisms must be
culturally responsive.
 Inform a wide range of stakeholders.
 Make use consistent with the purpose of the
evaluation.
 Consider community benefit and creating positive
change.
 Discussion of urban school-to-career
intervention program using culturally
responsive evaluation approach
 Input derived from school stakeholders on how best
to analyze and interpret data in ways that provided
meaning in particular contexts
 Findings disaggregated by gender and age to get
breakdown of career attitudes and beliefs for
participants
 Findings provided to numerous stakeholders in
particular ways (e.g. student findings presented in
student-friendly manner)
 Culture matters and affects every evaluation
 How do your designs incorporate the third three steps of the CRE
framework?
 CRE encourages the consideration of a variety of factors in
conducting evaluations in agencies and communities
 What challenges and opportunities do you see in integrating the
CRE framework?
 Consideration of cultural context and factors can provide a more
responsive and accurate evaluation
13
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 STEP 1: Articulate the steps that you engage in
 STEP 2: Identify steps where CRE principles, tenets, ideas align
or steps that could be augmented with CRE
 STEP 3: Operationalize what CRE “looks like” in that step using
our discussion today
PLANNING
 STEP 4: Adapt and use your hybrid approach!
Casillas, Hopson, & Gomez, 2015
Casillas, Hopson, & Gomez, 2015
14
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
8/12/2015
 What will you take away from this workshop?
 How will you use CRE in your current or anticipated work?
 What are your next steps in implementing CRE (if you have
any)?
 How can we help?
Stay in touch!
15