A multi level Communication for Social Change initiative

Evaluating Soul City
A multi level
Communication for Social
Change initiative
Shereen Usdin
Soul City
A South African NGO working in partnership with
localNGOsin 8 SADC countriesto develop
indigenousCFSC interventionsin Southern Africa
We are complex beings…
“We do notgrow absolutely,chronologically.Wegrow
sometimesin 1 dimension and notin another;
unevenly.Wegrow partially.We are relative.We are
mature in one realm,childish in another.The past,
presentand future mingle and pullusbackward,
forward,orfixusin the present.We are made up of
layers,cells,constellations.”
AnaisNin*
*Fr
er
AAttheor
al
*From
om D.
D.Reel
Reel
er,,‘‘
heoryyof
ofSoci
Soci
alChange”
Change”,,CDRA
CDRA
The Soul City Model of Social
Change
Socio-economic + political national environment
• Healthy public policy
• Re-orientate services
d ia
e
M
s
Mas
isati
l
i
b
o
lM
Socia
Individual
• Knowledge
• Attitudes/Beliefs/
/Risk perception
• Skills
• Self-efficacy
• Support for policy action
• Subjective norms
• Link to services
Community
• Dialogue/debate
• Social norms
• Diffusion of innovations
• Community efficacy
• Community action
• Social networks
• Social cohesion
Advoc
acy
on
Experimental designs
• Group 1 getpilland controlGrp2,a placebo
• Investigatecause-and-effectrelationships
• Attribute effectto the intervention
SocialChange iscomplexand so
CFSC interventionsneed more
complex,nuanced evaluation
designs
Challenges: noisy confounders
As exposure to Soul
City increases …
Can we say that Soul
City causes increased
condom use?
Condom use increased
?????
People exposed to Soul City are also
likely to be exposed to other HIV
interventions.
More Challenges
•• Caus
Causal
aliittyyiissdi
diffffiicul
culttttoopr
prov
oveebecaus
becauseeal
almos
mostt
impossible
impossible to
to get
get aa pristine
pristine control
control group
group
•• Change
Changeiissnot
notuni
unidi
dirrect
ectiional
onaland
andpeopl
peopleemay
maywel
welll
rrel
elaps
apsee
•• MMai
aint
ntai
aini
ning
ngpos
posiittiivveebehav
behaviiour
oural
alssoocount
countss
•• Change
Changeoccur
occurssiinnaacompl
complex
exenv
enviirronment
onment((ffor
orex
exampl
ampleeaa
pr
pres
esiident
dentof
ofaacount
countrryyssay
ayiing
ngtthat
hatHI
HIVVdoes
doesnot
notcaus
causee
AI
AIDS
DS–i
–ittiisshar
harddttoomeas
measur
ureetthe
henegat
negatiivveeiimpact
mpactof
oftthi
hiss))
Things to consider…
• Changemaynothappen overnightand
evaluationsmaybetoo earlyto seeresults
• Often need to measure intermediate indicators
(dialogue/intention… )which maypredict
success
• Unlessyou test,itishard to linkdirectlyto
changein HIV incidence/prevalence
Study Design
5. Media Monitoring
and Analysis
2. Sentinel Site Studies
3.National
Qualitative Impact
Assessment
1. National Quant
Survey Individual
4. Partnership
Study
Community
6. Institutional
/ Organisational
Impact Study
7. Cost Effectiveness Study
Society
Evidence of impact attributed
to the intervention
•
• Qualitative
Qualitative evidence:
evidence:
––
––
––
Direct
Direct mention
mention // direct
direct attribution
attribution
Consistency
Consistency
Data
Data Triangulation
Triangulation
•
• Quantitative
Quantitative evidence:
evidence:
–– Simple
Simple descriptive
descriptive analysis
analysis
–– More
More complex
complex inferential
inferential analysis
analysis (regression
(regression
analysis
analysis allowing
allowing us
us to
to make
make stronger
stronger associations
associations
–– Dose
Dose response
response
Evidence of impact attributed to
the intervention
•
• Consistency
Consistency of
of results
results
–– Across
Across evaluations
evaluations of
of multiple
multiple SC
SC interventions
interventions over
over time
time
–– Across
Across studies
studies within
within evaluations
evaluations of
of one
one specific
specific intervention
intervention
•
• Triangulation
Triangulation
––
––
––
Different
Different Methods
Methods // designs
designs
Different
Different Researchers
Researchers for
for each
each study
study
Different
Different sources
sources of
of data
data (ngo’s;
(ngo’s; gov;
gov; civil
civil leaders
leaders etc)
etc)
Speedy
implementation
of the Domestic
Violence Act
"There were pressures [to implement the DVA], there were pressures from occasions
where people held marches and stuff like that.” [SA Police Services]
“a lot of articles [on domestic violence and the DVA] started to appear … they create
expectations in the community which makes the government deliver….” [Government’s
Victim Empowerment Unit, Domestic Violence Task Team]
“I think my work with Soul City and also some of the NGOs, the Network
for Violence Against Women, opens your eyes.” [Government Domestic Violence Task Team]
Direct lobbying
Participation in public protest against VAW
Community
mobilization
Initiated by the
partnership, and
associated with exposure
to the edutainment vehicle
%
“Soul City influenced us
to organize the march,
emotions were high … “
[Rural Female]
5
6
3
3
1 source
2 sources
4
1
2
0
No SC
3 sources
Comparison over the evaluation period:
press coverage and Soul City / NNVAW Domestic Violence press releases
DV related coverage
attributable to the SC /
NNVAW partnership
VAW: Non-DV press coverage
% of total coverage
Media Advocacy
VAW: DV press coverage
40
DV Press Releases
* 49% of print media monitored dealt
with / referred to DV
* Approximately 1 in 5 articles directly
referred to SC and/or the NNVAW
30
20
10
0
June
July
Aug
Sept
Oct
Nov
Dec
Some textured stuff is measurable
E.g. dialogue * subjective norms * social
cohesion * networking
Figure 2: Perceptions of social norms by exposure to Soul City media
Most people in your community
disagree that a man is right in
expecting a woman to have sex with
him without using a condom
%
80
60
40
No SC
exposure
1 SC media
source
2 SC media
sources
3 SC media
sources
"Disagree" - Evaluation Survey
But…..
“Noteverything thatcountscan be counted.
And noteverything thatcan be counted,counts.”–
AlbertEinstein*
**Fr
er
AAttheor
al
**From
om D.
D.Reel
Reel
er,,‘‘
heoryyof
ofSoci
Soci
alChange”
Change”,,CDRA
CDRA
“Markers”
• Sometimesyourtearscome down when you are
watching becausethepartthatyou seelookslike
reallyhow you live...
((MMal
ale,
e,Li
Limpopo,
mpopo,Rur
Rural
al,,2525-35)
35)
New Stuff: propensity score analysis
?
Exposure group
People assigned to groups based
on propensibility to be exposed so
they are “comparable / similar”
Knowledge
Attitudes
Subjective
norms
Intentions
Selfefficacy
Behaviour
“Control group”
Into the future….
–– More
More advanced
advanced methodologies
methodologies to
to investigate
investigate
change
change at
at community
community level
level through
through social
social
networks/capital/cohesion
networks/capital/cohesion
–– experimenting
experimenting with
with “less
“less complex”
complex” sentinel
sentinel
sites
sites @
@ schools
schools
–– Explore
Explore advances
advances in
in quantitative
quantitative analysis
analysis
Is
Is Soul
Soul City
City exposure
exposure over
over 10
10 years
years associated
associated with
with
social
social change
change and
and more
more positive
positive behaviours?
behaviours?
…
… especially
especially young
young people
people exposed
exposed to
to Soul
Soul Buddyz
Buddyz
(possible
(possible link
link with
with decrease
decrease in
in HIV
HIV rates
rates in
in latest
latest ANC
ANC
survey
survey in
in youngest
youngest age
age group)
group)
Analysis
Analysis of
of 33 year
year panel
panel study
study with
with same
same people
people
examining
examining impact
impact over
over time
time
Longitudinal
Longitudinal qualitative
qualitative studies
studies –– with
with children
children
Using
Using complexity
complexity theory
theory to
to analyse
analyse data
data