here - CPC Learning Network

Sustaining Early Childhood Development Impacts
A systema)c review of
impact evalua)on evidence
Jeffery Tanner, World Bank
Colombia University—CPC Webinar—March, 2016
Outline
u  What
we did
u  What we found
u  How you can use it
What we did—Methods
Inclusion Criteria
u 
All interventions (Bank and non-Bank) occurring during early childhood
u 
All outcomes occurring after early childhood (primary school/age 6)
u 
Middle- and Low-Income Countries
u 
Published 1990-2014 (including gray literature)
u 
Impact Evaluations with a credible counterfactual (passed risk of bias)
u 
These provide causal, attributable measures of project outcomes
Resulting studies
u 
115 relevant studies
u 
54 studies (24 projects, 21 countries) passed quality inspection
This becomes our dataset
What we did—Scope
6 Domains of HD outcomes
3 Factors Influencing Outcomes
u 
Physical Development
u 
Gender
u 
Cognitive Development
u 
Socioeconomic Status
u 
Language Development
u 
Time
u 
Socioemotional Development
u 
Schooling Outcomes
u 
Labor Market Outcomes
What did we find?
u 
We can do a better job supporting parents and families.
v 
Three-quarters of the programs that support parents are in the Latin American
and Caribbean Region
v 
More screening for disabilities, developmental delay and maternal depression;
More wellness check-ups, childcare
Ø 
Robust IE evidence that parent support for Health Care and Early Stimulation
leads to large and growing benefits
World Bank Support to Early Childhood Development and Lasting Effects of Early Childhood Interventions
5
What we found – Confirming priors
OUTCOMES
u 
Cognition: Early Stimulation for most measures, and CCTs, deworming and
breastfeeding improve general cognition
u 
Language: Early Stimulation and CCTs
u 
Schooling: Early Stimulation, Preschool, and CCTs work
u 
SES: Children from poorer families benefitted
INTERVENTIONS
u 
Early Stimulation works for most outcomes most of the time
u 
Health and Social Protection also fairly robust
u 
Earlier starts and longer exposure are associated with better outcomes
What we found – A few surprises
OUTCOMES
u 
Early stimulation can in fact improve labor market outcomes
u 
Can’t (yet) sustain physical benefits beyond early childhood, though evidence
on height is mixed
u 
Socioemotional effects tend to lag, peak in adolescence, then fade
u 
Most outcomes are gender-neutral. However, Early Childhod interventions are
particularly impactful for girls’ schooling
INTERVENTIONS
u 
Deworming can improve both Cognition and Language. So can MCH packages.
u 
Nutrition needs to be supported throughout and beyond the first thousand
days to have any lasting effects
Nutrition
Interventions
Must be in place
throughout
(and beyond)
the first thousand days
to be effective
What did we find?
u 
Key interventions should be sustained throughout the First Thousand Days (and
beyond).
Ø 
Nutrition unlikely to have sustained effects otherwise
Ø 
Water, Health Care also more beneficial if 0-3
v 
Only the Latin American and Caribbean Region consistently emphasized child development
interventions for those younger than 3 across sectors
World Bank Support to Early Childhood Development and Lasting Effects of Early Childhood Interventions
9
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
How You Can
Use it—
Gap Map by
Region
Where to
Study?
u 
MENA and EAP highly
under-represented
u 
Need more from
China, India, Africa
interior
u 
Little from LowerMiddle Income
countries
Be More strategic: Evidence
Ø  Gaps on Interventions & Outcomes
Only 1 IE directly measures ECD’s effect on reducing poverty—by measuring impact on
later jobs and incomes
<1/2 of the IEs explored ECD’s effects on shared prosperity—by measuring how
outcomes differed by gender or parents’ income
Only one each study on sustained effects of early childhood from sanitation, water or
hygiene interventions
Little or no evidence on long-run benefits to children whose mothers received common
interventions.
0:
nutrition counseling, pre-natal visits, prevention of depression 1:
breastfeeding promotion, attended delivery or family planning
12
Mother
Children
Intervention Categories
Cognitive
Development
1
Developm ent
Language
1
Developm ent
Socioem otional
3
5
8
Developm ent
Physical
4
4
1
2
5
10
8
8
Unique Studies
Num ber of
2
5
1
4
Unique Projects
Num ber of
Outcome Domains
2
1
3
2
Labor Market
1
3
2
Schooling
3
3
Counseling on adequate diet during
pregnancy
Micronutrients and iron-folic acid:
supplementation and fortification for
Supplemental feeding (preschool, centerbase and/or take home rations)
2
1
Growth monitoring and promotion - prevention
and treatment for acute malnutrition
T herapeutic zinc supplementation for
diarrhea
Complementary feeding
Exclusive breastfeeding promotion
Counseling on optimal feeding practices:
1
1
1
1
5
1
1
1
1
2
1
1
7
1
1
1
1
5
1
1
1
1
4
1
1
1
1
Micronutrients and fortification for children
3
1
15
3
Access to health care
Prevention and
maternal depression
Water and sanitation
Well child visits, growth monitoring,
screening for developmental delays
Planning for family size and spacing
Deworming
1
4
1
1
Adequate sanitation
Hygiene or hand-washing
Access to safe water
Immunization
Disease prevention
Attended delivery
Antenatal visits
1
3
18
5
1
6
2
6
2
Pregnancy, delivery
and postnatal
1
6
6
4
2
4
1
6
1
3
1
1
1
1
4
10
2
2
1
5
6
2
1
Disease treatment
7
1
1
2
1
1
1
Parent support program
Quality teaching, programming or curricula
1
1
Stimulation
Pre-school Infrastructure
Early childhood and
pre-primary programs
Transition to quality
primary schools
Birth registration
Parental leave and adequate childcare or daycare
4
4
1
3
3
2
Conditional
10
1
55
25
1
1
1
1
1
19
1
1
1
1
27
16
1
Unconditional/targeted income support
(child grants, etc)
2
1
15
14
Social assistance
transfer programs
1
19
7
4
Orphans and fostering
21
15
Fill in the
Gaps!
6
Governance reflecting
Women's political reservation
ECD interests
Child protection
interventions
Number of Unique Studies
16
How You Can
Use it—
Policy or regulation in nutrition, health, education and social
protection
Number of Unique Projects
Gap Map
Interventions
x Outcomes
Nutrition
Health
Water, Sanitation
Education
Social Protection
Governance
Intervention Type
Summing up
u 
What did we do?
• 
u 
u 
A systematic review of credible IE evidence of the effect of early childhood
interventions on later life outcomes in developing countries
What did we find?
• 
Early Stimulation works. Social Protection and Health may too
• 
Nutrition needs full 1st 1000 days
• 
Sustaining improvements to physical development is tough
• 
Poor most often benefit: Implications for/against targeting?
How to Use It?
§ 
Deciding between interventions, Designing selected interventions
§ 
Rich Resources; explore from multiple angles
§ 
MINE THE GAPS
From surviving to thriving…
…Will require coordination between actors
u 
Sustained interventions from birth to age 3
Leverage existing early childhood entry points across sectors
u 
Support for parents in early stimulation
Actively engage parents as allies
u 
Strategic selection of operations and impact evaluations by region &
sector
Build on LAC experience
World Bank Support to Early Childhood Development and Lasting Effects of Early Childhood Interventions
15
Thank you.
Systematic Review:
http://ieg.worldbankgroup.org/Data/reports/
ImpactsofInterventions_EarlyChildhoodonLaterOutcomes.pdf
ECD Evaluation page:
http://ieg.worldbankgroup.org/evaluations/wb-support-early-childhood-development
Challenges and Cautions
for Systematic Reviews
Everything that is known; Not everything that is knowable
u 
u 
u 
Representativeness of Interventions—non-random selection
• 
Overrepresentation of easily evaluable interventions
• 
Focus on “reduced form” studies excludes those with intermediate
outcomes
• 
Lack of evidence does not imply no effect
Representativeness of Impact Evaluations
• 
Includes only existing studies
• 
Publication bias (file drawer bias)
Interpretation of Results
• 
IEs measure partial equilibrium; general equilibrium may be different
• 
Null results must be interpreted carefully—we never “accept” zero
• 
External validity—changes to time, place, or scale may affect results
Why little meta-analysis or meta-regression?
u 
Bin sizes too small
u 
Methods underdeveloped—Highly nested structure of the data
u 
Multiple outcome measures per
u 
Domain with multiple domains per
u 
Study with multiple studies per intervention
u 
With multiple interventions per country
…all of which are unbalanced—and heterogeneous
u 
But simple meta-analysis generally shows results (early stimulation) or null
results (physical development) where they would be expected by visual
inspection of the data.
Quick Tutorial
Getting the most from the ECD Systematic Review
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
How You Can Use It – Decide, Design: YMMV
Example: Tanzania
Make an argument based on likelihood of LT impacts on Schooling:
Early Stimulation
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
Neighbors with Evidence?
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
How You Can
Use it—
Gap Map by
Region
u 
Nothing from
Tanzania, but there
is material from
neighbors Kenya and
Mozambique
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
What about those ECD
interventions in Kenya, Moz?
How You Can
Use it—
Gap Map by
Region—low
quality
studies
u 
See 1 low-quality study
from Tanzania
Look it up in
u 
Appendix F—citation
information;
u 
Online database—why it
was rated low quality
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
Look up that study in
Appendix F
Find out why it didn’t pass in the
database
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
What is known about schooling
outcomes from other areas?
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
What are some options for
measuring schooling
outcomes?
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
These resources are useful
for post-early childhood
outcomes, but what about
more proximate effects?
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Hmm…Would it contribute significantly to
Ø  Tables, Forest plots, Vignettes
global knowledge to start a LT impact
Gap Maps: Intervention x Outcome, Region
evaluation on this project?
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
Intervention Categories
How You Can
Use it—
Nutrition
Mother
…and few on
Cognition or
Language
Education
Health
Social Protection
Disease Treatment
Vitamins, micronutrients, or fortified food for
pregnant women
2
1
Exclusive breastfeeding
1
1
Micronutrients and fortified food for children
1
1
Optimal feeding practices/complementary
feeding
2
3
Supplementary feeding (preschool, centerbase and/or take home rations)
3
Quality early childhood and pre-primary
program
E mp l o y me n t a n d L ab o rM arket
O u t c o me
Physical
Development
Schooling
Outcome
8
2
8
5
1
8
1
1
1
2
2
4
5
3
3
3
4
10
1
1
2
2
4
6
Pre-school Infrastructure
1
1
1
1
2
2
Family planning
1
1
1
1
Well child visits, screening for
developmental delays
3
3
4
6
Immunization
1
1
1
1
4
Deworming
1
2
1
1
1
Conditional
Cash Transfers
(Incentives for human
Unconditional/targeted income support
capital investments)
3
4
1
1
Child protection
2
1
(child grants, etc)
Socioemotional
Development
2
6
Parent support program
2
2
Stimulation
7
6
Water and sanitation
1
1
Governance (Women's political reservation)
1
1
20
19
Number of Unique Studies
10
Number of
Unique Studies
1
2
1
1
3
4
6
1
1
1
4
Childcare/daycare
Multi-sector
But there are
none on Labor
outcomes
Disease Prevention
Language
Development
Hygiene and hand-washing
Other
There are relatively
more IEs on Schooling
outcomes for Early
Stim, but still few in
absolute number
Early
Learning/Childcare
Pregnancy, delivery
and postnatal
Intervention Type
Gap Map
Interventions
x Outcomes
Children
Outcome Domains
Cognitive
Development
10
1
1
2
2
3
6
2
1
1
18
2
1
15
27
18
1
54
How You Can Use It – Decide, Design: YMMV
u 
Decide between interventions
u 
Design of chosen intervention
Resources
u 
Sections are Cross-referenced, summarized, stand alone
u 
Chapters address an outcome domain and subcategories, organized by intervention type
Ø 
Tables, Forest plots, Vignettes
u 
Gap Maps: Intervention x Outcome, Region
u 
Challenges and suggestions for doing a long-term IE on ECD
u 
Appendices:
u 
Ø 
A: Details on the 21 interventions, by country
Ø 
B: Details on outcome instruments
Ø 
E: Anthology of other ECD Systematic Reviews
Ø 
F: List of IEs that didn’t pass the quality rating
Full Coding Database on Web
I’ve heard setting up an IE for the long haul can
be a challenge. What should I look out for?
Nutrition
Health
Water, Sanitation
Education
Social Protection
Governance
Intervention Type
Mother
Children
Pregnancy, delivery
and postnatal
Disease treatment
Access to health care
Prevention and
maternal depression
Water and sanitation
Parent support program
Stimulation
Early childhood and
pre-primary programs
Transition to quality
primary schools
Birth registration
Intervention Categories
2
Cognitive
Development
1
1
Developm ent
Language
3
1
Developm ent
Socioem otional
2
3
5
8
Developm ent
Physical
4
4
1
2
5
10
8
8
Unique Studies
Num ber of
2
5
1
4
Unique Projects
Num ber of
Outcome Domains
1
3
2
Labor Market
3
3
Schooling
2
Counseling on adequate diet during
pregnancy
Micronutrients and iron-folic acid:
supplementation and fortification for
Supplemental feeding (preschool, centerbase and/or take home rations)
1
Growth monitoring and promotion - prevention
and treatment for acute malnutrition
T herapeutic zinc supplementation for
diarrhea
Complementary feeding
Exclusive breastfeeding promotion
Counseling on optimal feeding practices:
1
1
6
1
1
3
1
1
2
3
1
1
5
1
1
1
6
18
15
1
2
1
1
7
1
1
1
1
2
5
3
5
1
1
1
1
4
1
1
1
1
Micronutrients and fortification for children
1
6
6
4
2
Attended delivery
Planning for family size and spacing
Deworming
1
4
1
1
Disease prevention
Well child visits, growth monitoring,
screening for developmental delays
Adequate sanitation
Hygiene or hand-washing
Access to safe water
Immunization
1
1
4
10
2
2
Antenatal visits
5
6
2
1
4
1
1
1
1
1
7
1
1
1
1
1
2
Quality teaching, programming or curricula
1
1
Pre-school Infrastructure
Parental leave and adequate childcare or daycare
10
25
1
55
1
3
1
6
1
4
4
19
3
3
16
2
Conditional
1
27
1
1
14
1
1
1
7
15
1
2
19
1
1
15
Social assistance
transfer programs
Unconditional/targeted income support
(child grants, etc)
21
4
Orphans and fostering
16
6
Governance reflecting
Women's political reservation
ECD interests
Child protection
interventions
Number of Unique Studies
Policy or regulation in nutrition, health, education and social
protection
Number of Unique Projects