Casillas. (2015). - Claremont Graduate University



Casillas. (2015). - Claremont Graduate University
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Claremont Graduate School Professional Development
 Collaborative work from lectures, co-presentations,
research grants, published papers:
Claremont Evaluation Center's
Professional Development
Workshop Series in Evaluation and
Applied Research Methods 2015
National Science Foundation
American Evaluation Association
CDC/AEA Summer Institute
Aotearoa New Zealand Evaluation Association
Culturally Responsive Evaluation Assessment (CREA)
Many others CRE practitioners
Wanda D. Casillas, Ph.D.
[email protected]
Montclair State University and
Deloitte Consulting, LLP
9:15 a.m. –10:45 a.m.
Workshop:Overview and discussion of cultural selves
 Interactive
 Opportunities to practice ways of speaking and thinking, challenging
workshop issues
 Reflective
 Opportunities to think about these workshop issues over the day
10:45 a.m. –11:00 a.m. Break
11:00 a.m. –12 noon
Workshop: Thinking about culture in evaluation
12 noon –1:30 p.m. Lunch Break
1:30 p.m. –3:00 p.m.
Workshop: Being culturally responsive in evaluation
 Prospective
 Opportunities to think about workshop issues further and work with
presenters as interested
3:00 p.m. –3:15 p.m.Break
3:15 p.m. –4:45 p.m.
Workshop Conclusion
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Recognize the relevance and value-addedness of culture in
evaluation theory and practice
 Discuss the evaluation process in the context of working with
underserved communities, in particular
 Identify strategies for assisting evaluators and agencies in becoming
culturally responsive
the way of life of a group of people, the complex of
shared concepts and patterns of learned behavior that
are handed down from one generation to the next
through the means of language and imitation.
(Barnouw, 1985)
the ever-changing values, traditions, social and political
relationships, and worldview created, shared and
transformed by a group of people bound together by a
combination of factors that include a common history,
geographic location, language, social class, and
(Nieto, 1999)
 Demographic, sociopolitical, contextual dimensions
 Characteristics as dynamic, multifaceted, learned,
created, influenced
 More than about race and ethnicity but inclusive of other
identifications (SenGupta, Hopson & Thompson-Robinson, 2004;
Nieto, 1999)
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
Instructions: Consider the most prominent cultural roles that you play.
Use the handout provided to record all of the roles that come to mind
(feel free to record more than 7 by creating more circles if needed).
Use the questions below to help you think about your roles.
How do you think about
 How do you identify yourself on a daily basis?
 What are some ways you would describe yourself?
 How might others perceive you?
Think about all of the cultural
roles you play and how
playing these roles define how
you perceive yourself.
Identify influences in your
world view.
How do you think your world
view influences what you do
personally? Professionally?
1) What questions an evaluator asks and ultimately does not ask
 How you define yourself is the template by which you organize the
2) What an evaluator illuminates and ultimately minimizes
3) What evaluation approach is used and ultimately not used
 Culture and one’s cultural background influence our template(s)
4) What data are collected and ultimately overlooked
 We self-reference this template to help us make decisions in our
5) How interpretations are made and whose interpretations are held in
high or low esteem
everyday lives, and in our professional practice
6) What conclusions are drawn and what conclusions are not
7) How results are presented and to whom such results are
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Summary of disciplines of workshop participants
 Summary of experiences of workshop participants
 Part I: Talk through theory and perspective of culturally
responsive evaluation
 Part II: Review principles or tenets that surface from concept
mapping study and literature review on CRE
 Part III: Introduce strategies for implementing principles and
“What has frustrated me in the ways
multicultural programs have been evaluated is
that the people who do the evaluation generally
do not understand the nature of multicultural
work...The evaluators and their evaluations
often miss the point of what the program is
about and use inappropriate standards on
which to interpret the program on which to
make value judgments” (Stockdill, 1992:17)
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Role of primary inclusion of participants:
 “As Lincoln (1991) points out, most people who
evaluate social programs know very little about the
minority program participants’ world view, the
appropriateness of program interventions in meeting
their needs, or programs’ personal consequences for
these clients (Madison, 1992).
 Using one set of methodologies in one cultural context
and applying to others:
“...evaluators must exercise great caution in trying to
apply the methodologies, models, and categories
devised in and for the developed world in Third World
countries...different views of reality and the nature of
change lead to different assumptions about appropriate
goals, treatment, and evaluation models (Cuthbert,
 Evaluation in rudimentary form in developing countries, even among
 Culture is either not taken into consideration or it is unknown how to
 Evaluation as auditing versus examination of how useful and/or
 Culture seems to be something to learn so that one can simply get
international organizations
appropriate a certain program is for the local community
 Western ideas of what "should" happen become the default priority
for organizations
be successfully taken into account
the evaluation done; it’s not really considered
 Often include cultural communities as passive participants
 Difficulties understanding the population's ways of thinking, living,
and understanding
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Continued flurry of evaluation discussion in conference meetings,
 Social location of evaluator
matters: lived experiences shape
assumptions and frames of
reference in evaluation process
 Avoiding ethnocentrism
periodicals, and special meetings, BUT:
means embracing multiple
cultural perspectives
 Limited understanding and consideration in evaluation
 Culture is central to the
evaluation process
 Evaluators play roles in furthering
social change and justice: we are
“more than technicians” but have
duty to recognize power relations,
challenge systems of inequity
 Culturally and ethnically
diverse communities have
contributions to make in
redefining field of evaluation
 Growing, but still lack of courses and training opportunities that
pay attention to cultural context, relevance, responsiveness,
competence in evaluation
(Hopson, 2003)
 Approach/model used to guide evaluation
 System and culmination of evaluation principles
and strategies
 Theoretically and politically positioned
 Privileging lived experiences, esp. communities and
populations of color
 Avoiding the phenomenon of “evaluating down”
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
Decolonizing/ indigenous
positions, epistemologies,
and frameworks
Critical theories and
epistemologies of race
Social agenda and advocacy
theories, models and
approaches in evaluation
A set of academic and interpersonal skills that
allow individuals to increase their understanding
and appreciation of cultural differences and
similarities within, among, and between groups.
This requires a willingness and ability to draw
on community-based values, traditions, and
customs, and to work with knowledgeable
persons of and from the community in
developing focused interventions,
communications and other supports.
(Orlandi, 1992)
(Hopson, 2009)
Cultural competence is a stance taken
toward culture, not a discrete status or simple
mastery of particular knowledge and skills. A
culturally competent evaluator is prepared to
engage with diverse segments of
communities to include cultural and
contextual dimensions important to the
Understand and recognize the larger context for programs or projects
Research and learn about the cultural group
Be aware of cultural labels and historical context
Identify potential historical inaccuracies
Design evaluation with participants in mind
Be culturally-specific in design
Use a multifaceted approach and appropriate methods
Collect data in culturally responsive ways
Allow for self-determination by stakeholders and program participants
Engage directly with participants through discussion
Engage stakeholders in general planning and in theory development
Public Statement on Cultural Competence in Evaluation
(American Evaluation Association, 2011)
Casillas, Hopson, & Gomez (2015)
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Debrief on definitions, influences of CRE
Build trust and facilitate communication
Allow for representativeness
Build diversity of organization/evaluation team
Access diversity from external sources
Being inclusive of diversity
 Q&A
Understand the evaluation audience and help the audience to
understand the evaluation purpose and process
 Time check
 Getting ready for Exercise #2 – outside experts
Make the evaluation accessible
 Handout and directions
Understand evaluator attributes that may affect professional practice
Casillas, Hopson, & Gomez (2015)
(adapted from Frierson,, 2010)
Prepare for
the evaluation
Disseminate and
use the results
Identify purpose
of the evaluation
Frame the
right questions
Design the
the data
the data
Select and adapt
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
Culturally responsive
Options for implementing CRE behaviors
Informant interviews
Group discussions
Feedback sessions
Engage Stakeholders
Analyze context
Explore communication styles
Prepare for
Identify purpose
of the evaluation
Gain trust
Facilitate stakeholder ownership of the
Assemble evaluation team
Shared lived experiences
Acquire foundational
Cultural guides
"critical friends of the evaluation"
Examine program theory for cultural
Process evaluation
Examine appropriateness of program goals
Progress evaluation
for population
Determine the effects of program
Summative evaluation
implementation on participants
Consensus on evaluation
 Be informed by the sociocultural context of the evaluand,
 History
 Formal and informal power relationships
 Communication and relational styles
 Informant interviews, group discussions,
 Assemble an evaluation team whose collective lived experience
fits the context of the evaluand.
 Evaluator awareness of own
cultural values,
assumptions, prejudices, stereotypes
 Not merely about matching demographics
 Multi-ethnic, informants, interpreters, cultural
guides, “critical friends of the evaluation”
Derived from Frierson, Hood, Hughes, and Thomas (2010)
 Develop a stakeholder group representative of the population
served by program.
 Seek to include persons impacted by the program directly and
 Pay attention to issues of power, status and social class.
 Include multiple voices in meaningful preparation process and
 Create climate of trust, respect.
 Create Stakeholder Map (handout)
 Document, examine program implementation
 How well is the program connecting with its intended
 Is the program operating in ways that are respectful of
cultural context?
 Are program resources equitably distributed?
 Document, examine progress toward goals
 Who is benefiting from the program, and are these
benefits equitably distributed? Who is burdened by the
 Is program theory culturally sensitive?
 Evaluate overall effectiveness
 Capture cultural nuances
 Examine correlates of participant outcomes
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Use the CRE Framework to assist in design
 Reflections on embedding culturally competent
evaluation in Indian Country
 Build understanding of values that underlie
programs and projects and create value-added
evaluative contribution
 Engage stakeholders in participatory manner
 Reflect and work as team to develop key
components of design
 Build ethic of participation and capacity building that values
community, relationships, respect
 Frame purpose by building conceptual picture/model
 Careful of “too sequential and narrative driven” logic
 How do your designs incorporate the first three steps of the CRE
 What challenges and opportunities do you see in integrating the
CRE framework?
Engage Stakeholders
Frame the right
Design evaluation
Culturally responsive
Options for implementing CRE
Determine appropriate type of
Critically question evaluation Does the question limit results of the evaluation?
Is there another possible approach to take?
How might different evaluation questions result in
different understandings of the program?
Identify appropriate design
Mixed method often yield best results
Data collection at multiple time points
Decide to identify, develop,
and/or adapt existing measures
Pilot test for appropriateness to
Translate when necessary
Forward-backward translation
Translation by committee
Multiple forward translation
Check instruments for semantic/content
Derived from Frierson, Hood, Hughes, and Thomas (2010)
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Questions of relevance to significant stakeholders
 Determine what will be accepted as evidence
 Notice whose voices are heard in the choice of
questions and evidence.
 Reflect on how questions limit what can be
learned and how they might be posed differently.
 Notice how different questions may expand
understanding. Revise and refine questions.
 Can questions be answered with available
 Is there another possible approach to take?
 Build design appropriate to both evaluation
questions and cultural context.
 Seek culturally appropriate mixed methods,
combining qualitative and quantitative
 Try to collect data at multiple points in time,
extending the time frame of the evaluation as
 Construct control or comparison groups in ways
that respect cultural context and values.
 Identify, develop or adapt instruments for the local
 Establish evidence of reliability and validity.
 Language and content of instruments should be
culturally sensitive.
 Use best translation practices, validating both
semantic and content equivalence.
 Forward/backward (FBT)
 Translation by committee (TBC)
 Multiple forward translation (MFT)
 Norms must be appropriate to the group(s)
involved in the program.
 Evaluation of undergraduate STEM research
program designed for students of color
 Deliberate design of evaluation team intimately
connected with program of study and background of
program, including similar lived experiences of
 Questions were sensitive to lived experiences of
participants and focused on substance of participant
 Beyond attention to traditional issues of success but exploring
issues of persistence as students of color
 Attempt to address nuances and subtleties relative to
experiences and impact of program
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 How do your designs incorporate the second three steps of the
 What challenges and opportunities do you see in integrating the
CRE framework?
Options for implementing CRE
Focus groups
Revisionist histories
Engage Stakeholders
CRE framework?
Culturally responsive activities
Collect data
Training "people as instruments"
Determine accurate interpretations
Analyze Data
Multiple strategies for analyzing quantitative data
Results and
Know what is being said/observed
Nonverbal Behaviors
Videotaping and reviewing
Know language
Know context
Share lived experience
Disaggregate variables
Cross tabulations
Community review results for "relevance gaps"
Widely disseminate
Use multiple modalities
Clearly present data
Report in modes appropriate for the "right people"
Derived from Frierson, Hood, Hughes, and Thomas (2010)
 Procedures used to collect both qualitative and
quantitative data must be responsive to cultural
context. E.g. storytelling, focus groups,
chronicles, interviews
 Nonverbal as well as verbal communications
provide keys to understanding.
 Train data collectors in culture as well as
technical procedures, videotape and review
 Recognize how cultural identifications of the
evaluation team affect what they can hear,
 Shared lived experience provides optimal
grounding for culturally-responsive data
 Understanding cultural context is necessary for
accurate interpretation.
 A cultural interpreter may be needed to capture
nuances of meaning.
 Stakeholder review panels can more accurately
capture the complexity of cultural context,
supporting accurate interpretation.
 Disaggregate data and cross-tabulate to examine
diversity within groups.
 Examine outliers, especially successful ones.
 Remember that data are given voice by those who
interpret them- shared lived experience, shared
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 Cultural responsiveness increases both the
truthfulness and utility of the results.
 Maximize community relevance of findings; invite
review by community members prior to
 Communication mechanisms must be
culturally responsive.
 Inform a wide range of stakeholders.
 Make use consistent with the purpose of the
 Consider community benefit and creating positive
 Discussion of urban school-to-career
intervention program using culturally
responsive evaluation approach
 Input derived from school stakeholders on how best
to analyze and interpret data in ways that provided
meaning in particular contexts
 Findings disaggregated by gender and age to get
breakdown of career attitudes and beliefs for
 Findings provided to numerous stakeholders in
particular ways (e.g. student findings presented in
student-friendly manner)
 Culture matters and affects every evaluation
 How do your designs incorporate the third three steps of the CRE
 CRE encourages the consideration of a variety of factors in
conducting evaluations in agencies and communities
 What challenges and opportunities do you see in integrating the
CRE framework?
 Consideration of cultural context and factors can provide a more
responsive and accurate evaluation
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 STEP 1: Articulate the steps that you engage in
 STEP 2: Identify steps where CRE principles, tenets, ideas align
or steps that could be augmented with CRE
 STEP 3: Operationalize what CRE “looks like” in that step using
our discussion today
 STEP 4: Adapt and use your hybrid approach!
Casillas, Hopson, & Gomez, 2015
Casillas, Hopson, & Gomez, 2015
Casillas. (2015). Cultural responsiveness
in applied research and evaluation. CGU
 What will you take away from this workshop?
 How will you use CRE in your current or anticipated work?
 What are your next steps in implementing CRE (if you have
 How can we help?
Stay in touch!