Common Outcomes and Measures for Children

Otonabee Valley (OV) Family Hub
Common Outcomes and Measures for
Children and Families
Project Report
January 14, 2013
Table of Contents
Table of Contents .................................................................................................................. 1
List of Tables ......................................................................................................................... 1
List of Figures ........................................................................................................................ 1
Acknowledgements ............................................................................................................... 2
1. Executive Summary ........................................................................................................... 3
2. Introduction ...................................................................................................................... 5
3. Otonabee Valley Family Hub Common Outcomes and Measures for Children and Families
Roundtable ........................................................................................................................... 5
3.1 Small group output and outcomes comparison ................................................................... 6
3.2 Themed group discussion results ......................................................................................... 8
3.3 Closing reflections and roundtable evaluation ................................................................... 10
4. Next Step Recommendations ........................................................................................... 12
4.1 Describe the service - who is using what services? ............................................................ 12
4.2 Describe the service impacts - gather rich data to enhance our understanding of client
and staff experiences. ............................................................................................................... 12
4.3 Establish an OV Common Outcomes Working Group to implement next step
recommendations. .................................................................................................................... 12
4.4 Explore the re-activation and role(s) of the OV Neighbourhood Committee. ................... 12
4.5 Describe the service - track referrals. ................................................................................. 12
4.6 Develop and test an OV Common Outcomes Theory of Change. ...................................... 12
4.7 Recommendation timeline. ................................................................................................ 13
4.8 Present OV recommendations to Peterborough Children’s Planning Table. ..................... 13
Appendix 1: Annotated Bibliography - Common Outcomes and Measures for Children and
Families (with an emphasis on community-based services)................................................... 14
Appendix 2: Otonabee Valley Family Hub Output and Outcomes Inventory .......................... 31
Appendix 3: Output and Outcomes List from Service Providers - OV Family Hub................... 48
Appendix 4: Small Group Output and Outcome ‘Elements’ by Category ............................... 54
Appendix 5: Roundtable Participants ................................................................................... 55
Appendix 6: Themed Group Discussion Notes ...................................................................... 56
List of Tables
Table 1: Small Group Output and Outcomes Comparison - Finding Common Themes ............. 7
List of Figures
Figure 1: Roundtable Evaluation Results .............................................................................. 11
1|P a g e
Acknowledgements
Thank-you to the lead group that has overseen the common outcomes and measures project
since late spring 2012 – your wisdom and guidance has been invaluable: Teresa Burke (Nursery
Two Child Care), Maria Castiglione (Learning Disabilities Association of Peterborough), Karen
Chomniak (Peterborough County-City Health Unit), Alex Cranfield (Five Counties Children’s
Centre), Nancy Fischer (City of Peterborough), Simon Dadds & Erin Hentig (Haliburton-Kawartha
Children’s Aid Society), Joanne McCarthy (Kinark Child and Family Services), Mary-Ann Meagher
& Emmy Ruttle (Peterborough Family Resource Centre), and Lloyd Schoenmaker (Otonabee
Valley Public School). I’d like to also take this opportunity to thank all of the service providers
who took the time to contribute to the common outcomes and measures for children and
families inventory during the summer of 2012 – without your feedback and knowledge, the
November roundtable would not have been possible. Thank-you to the participants in the
November 19, 2012 roundtable. Your participation marked the culminating event in this
common outcomes and measures for children and families project – with your input we can
move forward together!
Todd Barr
Trent Centre for Community-Based Education
January 11, 2013
2|P a g e
1. Executive Summary
In late April 2012, the Trent Centre for Community-Based Education (TCCBE) was hired by the
Early Intervention/Best Start Committee of Peterborough to initiate conversations about
working together to improve child and family outcomes in the Otonabee Valley (OV) Family Hub,
OV School community and neighbourhood. The hope was to be able to document this process
so it could be applied in other Peterborough City and County Child and Family Centres.
Guided by a small lead group of agencies and staff at the OV School Family Hub, the OV
Common Outcomes and Measures for Children and Families Project included three activities:
a) A literature review related to outcomes and measures for children and families, with an
emphasis on community based services;
b) An inventory of OV School service provider programs, outputs & outcomes for children
and families;
c) A community roundtable to discuss the above results and figure out common outcomes
and measures for children and families;
After the roundtable, the lead group for the project met to discuss next step recommendations
and a timeline for consideration by the Early Intervention/Best Start Committee of
Peterborough. They are as follows:
1. Describe the service - who is using what services?
Form a working group to implement a pilot common registration system for tracking
“who is using what services” at the OV Family Hub. Client consent issues and
methodological suggestion of collecting first and last name, postal code and year of
birth must be considered.
2. Describe the service impacts - gather rich data to enhance our understanding of client
and staff experiences
Design and deliver client focus group(s) and staff focus group(s) to better understand
the value of the OV Family Hub at the grassroots level. The purpose of these focus
groups would be to a) receive feedback and stories; and b) contribute to an OV Common
Outcomes Theory of Change (described below).
3. Establish an OV Common Outcomes Working Group to implement next step
recommendations.
There is a need for continued, community-based oversight of future OV Family Hub
Common Outcomes and Measures for Children and Families activities. It is important to
3|P a g e
provide working group membership opportunities for service providers, OV program
participants and neighbourhood residents.
4.
Explore the re-activation and role(s) of the OV Neighbourhood Committee.
The lead group recommends that OV School Principal and OV Family Hub Coordinator
revisit the Terms of Reference for the OV Family Hub “Neighbourhood Committee” and
implement a plan for re-activation and re-vitalization ASAP.
5.
Describe the service - track referrals
Form a working group to implement a pilot referral-tracking system that a) tracks
referrals and b) tracks successful referrals.
6.
Develop and test an OV Common Outcomes Theory of Change
The lead group believes that recommendations 1 and 2 above, combined with the
results from the literature review, outputs and outcomes inventory and roundtable, will
allow OV Family Hub participants to develop an OV Common Outcomes Theory of
Change (ToC)1 and work together to “test” the ToC by beginning to track and measure
child and family outcomes.
7.
Recommendation timeline
The lead group suggests the following timeline for implementation of next step
recommendations:
Establish OV Common Outcomes Working Group
Winter 2013
Implement pilot common registration system
Winter to fall 2013
Develop and test OV Common Outcomes Theory of Change
Fall 2013 onwards
Design and deliver service impact focus groups
Spring 2013
Implement pilot referral tracking system
Winter to summer 2013-14
Re-activate and re-vitalize OV Neighbourhood Committee
Spring 2013
8. Present OV recommendations to Peterborough Children’s Planning Table
The lead group recommends presenting the above seven recommendations to the
Peterborough Children’s Planning Table for support and endorsement.
1
An overarching and flexible framework (usually a one-page visual) that describes short, medium and long-range
outcomes for children and families participating in the OV Family Hub…and maps activities, plans, barriers and
opportunities related to those goals. Having a one-page visual is a useful tool when many participants are working
on parts of a greater whole.
4|P a g e
2. Introduction
In late April 2012, the Trent Centre for Community-Based Education (TCCBE) was hired by the
Early Intervention/Best Start Committee of Peterborough to initiate conversations about
working together to improve child and family outcomes in the Otonabee Valley (OV) Family Hub,
OV School community and neighbourhood. The hope was to be able to document this process
so it could be applied in other Peterborough City and County Child and Family Centres.
Guided by a small lead group of agencies
and staff at the OV School, the OV Common
Outcomes and Measures for Children and
Families Project included four main
activities:
1. A literature review related to
outcomes and measures for children
and families, with an emphasis on
community based services
(Appendix 1) – spring 2012;
2. An inventory of OV School service provider programs, outputs & outcomes for children
and families (Appendices 2 & 3)2 – spring-summer 2012;
3. A community roundtable to discuss the above results and figure out common outcomes
and measures for children and families – November 19, 2012;
4. Present final report and recommendations to Early Intervention/Best Start Committee
of Peterborough - January 14, 2013.
As mentioned above, results from the literature review and outputs and outcomes inventory
are listed in Appendices 1 through 3. What follows below is the results from the November 19
roundtable and subsequent lead group recommendations for the Early intervention/Best Start
Committee of Peterborough.
3. Otonabee Valley Family Hub Common Outcomes and Measures for Children
and Families Roundtable
On November 19, 2012, nineteen individuals participated in the Otonabee Valley (OV) Family
Hub Common Outcomes and Measures for Children and Families Roundtable. The goals for the
day were:
2
Appendix 2 is the complete inventory; Appendix 3 is the summary of inventory that was used to guide the
November 19 roundtable discussions.
5|P a g e
1. Identify information we are currently collecting, or would like to collect to improve
outcomes for children and families accessing the programs and services offered by the
OV Family Hub and outreach activities in the school and surrounding neighbourhood;
2. Build capacity for doing outcomes measurement together;
3. Build understanding between Otonabee Valley Family Hub service providers, OV School
community and neighbourhood and the broader network of child and family service
providers and educators in Peterborough City and County;
4. Spend time together doing activities that re-energize us & inspire everyone to work
together!
The roundtable started with participant introductions
and a discussion of the goals for the day. The facilitator
(Todd Barr) then presented results from the OV
Common Outcomes and Measures for Children and
Families Project so far (literature review and OV
common outputs and outcomes for children and
families inventory).
3.1 Small group output and outcomes comparison
After the above presentation and discussion,
roundtable participants were divided into two random
groups and asked to sort two identical sets of outputs
and outcomes (“elements” - generated from the
summer 2012 inventory) into themed piles – with a
maximum of six piles per group. Participants were
provided with outputs and outcomes only – agencies
or programs were not identified. Three guiding
questions for this sorting activity were:
1. Are there any outputs or outcomes to add?
2. Are any outputs or outcomes missing?
3. Is there anything to set aside for now?
After a short break, roundtable participants came back together to compare results from the
two random groups (Table 1) and identify common themes.
6|P a g e
Group #1
Table 1: Small Group Output and Outcomes Comparison - Finding Common Themes
Common
themes
Program outputs
(26)*
Volumes of
service
(quantitative)
(26)
Adult skills
(11)
Volumes of
service/program
outputs (52)
Group #2
Adult-related
Family &
outcomes (9)
social
relationships
(9)
[Program]
feedback
outcomes
(6)
Childrelated
outcomes
(9)
Access
and
barriers
(5)
Adult-related
outcomes/skills
(20)
Relationships
(10)/Selfesteem &
values (4)
[Program]
Feedback and
evaluation
(10)
Family and
social
relationships
(23)
Child
development
(3)
Program
and
service
feedback
(16)
Childrelated
outcomes
(12)
Participant
goal-setting
(6)
* The number in brackets represents the total number of individual outputs and outcomes (elements) sorted into
that particular pile. See Appendix 4 for a list of small group output and outcome ‘elements’ by category.
As indicated in Table 1, seven preliminary common outputs and outcome groupings were
identified:
1. Volumes of service/program outputs (e.g. attendance)
2. Adult-related outcomes/skills
3. Program and service feedback
4. Child-related outcomes
5. Family and social relationships
6. Access and barriers
7. Participant goal-setting
7|P a g e
Discussion:
When discussing the “program feedback and evaluation” piles, both groups realized that some
of the individual outputs and outcomes (elements) in those piles were related to “child
development” and “parenting,” so these elements were re-sorted to the appropriate pile.
Group #1 decided to merge “adult related outcomes” and “skills” because the adult outcomes
were all skills-related. Group #1 also decided to merge “relationships” and “self-esteem &
values” because these two themes fit with group #2’s “family and social relationships” theme.
The larger group was not surprised at the number of elements in “volumes of service/program
outputs” because it is fairly easy to capture and required by many funders.
Another interesting discussion point was that both groups merged certain themes because they
were only allowed a maximum of six themes. For example: “Access and barriers” was a group
#1 theme that was merged into other themes and “participant goal-setting” was merged into
the “family and social relationships” theme by group #2.
Overall, the group was pleased with the results because similar themes emerged between the
two randomly-assigned groups.
3.2 Themed group discussion results
With the results of the comparison activity in mind,
roundtable participants were asked to prioritize four
common output and outcome themes for further
discussion.
The group agreed that if a theme had a high number of
individual elements (i.e. this is an outcome that service
providers and others are focusing on/hoping to focus on
at the OV Family Hub more than another outcome), that
it did not necessarily mean that it would be easy to
figure out ways to measure that outcome together!
With this in mind, the four initial prioritized themes for
further discussion were:
1. Volumes of service/program outputs (e.g. attendance)
2. Adult-related outcomes/skills
8|P a g e
3. Program and service feedback3
4. Child-related outcomes
Participants self-selected into one of the above four themed groups and were provided with
the following question to guide their discussions: “How can we measure/track this outcome
together given realistic resources?”
Highlights4 from the themed group discussions:
Volumes of service/program outputs: Key measures are unique client counts and attendance.
To avoid duplicating clients would need a common identifier – perhaps a card – either manual
or electronic.
Adult-related outcomes/skills: Explore the idea of a common pre- and post- questionnaire. Start
in registered programs where there is consistent attendance rather than drop-in programs
where you don’t necessarily see a family regularly. Look beyond program satisfaction surveys to
trying to measure behaviour change.
Program and service feedback: Try to reach out to current, past and people who only come
once to programs for feedback. Create year over year data and compare. Stories and focus
groups can be rich sources of information.
Child-related outcomes: Consistent use of existing inventories/screening tools by all service
providers. The three discussed were:
1. Triple P (Positive Parenting) measuring behavioural development;
2. Nipising District Developmental Screen which measures the development of speech,
expression, etc. and is a helpful tool to compare between what parents are seeing and
what service providers have noticed in a child’s development;
3. Early Development Instrument (EDI) scores done on all SK students in the school system
by teachers.
Overall themed groups: Many agencies/programs work with goals but not all use formal
measurement tools.
3
Editor’s note: There appears to be two different understandings of “feedback and evaluation.” One
understanding is feedback related to program delivery (e.g. quality of instruction, accessibility of workshop
location, etc.). The other understanding appears to be related to general techniques for obtaining feedback and
evaluation overall.
4
Complete themed group discussion notes are provided in Appendix 5.
9|P a g e
3.3 Closing reflections and roundtable evaluation
To close the day, the group discussed next steps overall,
participated in a brief closing reflection activity and
completed a brief roundtable evaluation.
Next steps
 We need to have a conversation about client
consent when it comes to evaluation – especially
shared evaluation between agencies/programs.
How do others do this? Perhaps we could look at
other models for ideas?
o Consider common intake to manage
shared consent issues.
 Is there some best practice learning from other
hubs related to screening tools (in Peterborough and beyond)?
 How do we get rich information from families? Try focus groups and use facilitators
from a variety of agencies. Share facilitators so they don’t evaluate their own programs.
 Do we need a broader output and outcomes inventory? It depends on what the “hub”
wants. This brings up the issue of hub governance and moving forward. Who decides on
a plan of action?
 Need for greater awareness about what’s going on at OV Family hub.
Closing reflection (participants were invited to share a few words out loud):












Good to hear how people use the hub
Great work
Excited to evaluate
Excited
Intrigued
Appreciation of everyone’s time
Common measures do exist
Optimistic
Challenged
Excited – lots of opportunities
Energized by the team
Want to know about other agencies
10 | P a g e
 Optimistic – this is where collaboration starts
 Encouraged
Roundtable evaluation
Participants were asked to complete a brief evaluation survey based on five criteria:
Participatory, Useful, Fun, Refreshments and Logistics. A score of 10 meant that it was the best
possible experience with little room for improvement and a score of zero being the absolute
worst experience with lots of room for improvement. Participants were also asked to write any
comments related to their scoring. Figure 1 contains a summary of participant scores and
participant comments follow afterward.
Participatory
10
8
6
4
Logistics
Useful
2
0
Refreshments
Fun
Roundtable Evaluation Results
Figure 1: Roundtable Evaluation Results
The majority of individuals found the session participatory, useful and fun. At least one
individual wanted to see how it related to their work before they deemed the roundtable
‘useful.’ For the majority, refreshments and logistics also appear to have been handled well.
Participant Comments: Useful
 Not sure yet
 Useful for general information but not necessarily for my specific program
Participant Comments: Logistics, Refreshments and Overall
 Thank-you!
 Organized, understandable, logical process
 Great work Todd; Fantastic – thanks
11 | P a g e
4. Next Step Recommendations
After the roundtable, the lead group for the project met to discuss next step recommendations
and a timeline for consideration by the Early Intervention/Best Start Committee of
Peterborough.
4.1 Describe the service - who is using what services?
Form a working group to implement a pilot common registration system for tracking “who is
using what services” at the OV Family Hub. Client consent issues and methodological suggestion
of collecting first and last name, postal code and year of birth must be considered.
4.2 Describe the service impacts - gather rich data to enhance our understanding of
client and staff experiences.
Design and deliver client focus group(s) and staff focus group(s) to better understand the value
of the OV Family Hub at the grassroots level.5 The purpose of these focus groups would be to a)
receive feedback and stories; and b) contribute to a OV Common Outcomes Theory of Change
(described below).
4.3 Establish an OV Common Outcomes Working Group to implement next step
recommendations.
There is a need for continued, community-based oversight of future OV Family Hub Common
Outcomes and Measures for Children and Families activities. It is important to provide working
group membership opportunities for service providers, OV program participants and
neighbourhood residents.
4.4 Explore the re-activation and role(s) of the OV Neighbourhood Committee.
The lead group recommends that OV School Principal and OV Family Hub Coordinator revisit
the Terms of Reference for the OV Family Hub “Neighbourhood Committee” and implement a
plan for re-activation and re-vitalization ASAP.
4.5 Describe the service - track referrals.
Form a working group to implement a pilot referral-tracking system that a) tracks referrals and
b) tracks successful referrals.
4.6 Develop and test an OV Common Outcomes Theory of Change.
The lead group believes that recommendations 4.1 and 4.2 above, combined with the results
from the literature review, outputs and outcomes inventory and roundtable, will allow OV
5
Guiding questions include: What is the value of the Hub to you? What is good about it? What needs
improvement? What have you learned from your participation in the Hub? What was your reason for coming? Did
you get what you need? What has your child learned?
12 | P a g e
Family Hub participants to develop an OV Common Outcomes Theory of Change (ToC)6 and
work together to “test” the ToC by beginning to track and measure child and family outcomes.
4.7 Recommendation timeline.
The lead group suggests the following timeline for implementation of next step
recommendations:
Establish OV Common Outcomes Working Group
Winter 2013
Implement pilot common registration system
Winter to fall 2013
Develop and test OV Common Outcomes Theory of Change
Fall 2013 onwards
Design and deliver service impact focus groups
Spring 2013
Implement pilot referral tracking system
Winter to summer 2013-14
Re-activate and re-vitalize OV Neighbourhood Committee
Spring 2013
4.8 Present OV recommendations to Peterborough Children’s Planning Table.
The lead group recommends presenting the above seven recommendations to the
Peterborough Children’s Planning Table for support and endorsement.
6
An overarching and flexible framework (usually a one-page visual) that describes short, medium and long-range
outcomes for children and families participating in the OV Family Hub…and maps activities, plans, barriers and
opportunities related to those goals. Having a one-page visual is a useful tool when many participants are working
on parts of a greater whole.
13 | P a g e
Appendix 1: Annotated Bibliography - Common Outcomes and Measures for
Children and Families (with an emphasis on community-based services)
June 2012
A. Method
Google and Google Scholar were used to find sources, along with bibliographies from the
sources themselves. Keywords used included any combination of the following: outcomes,
measures, children, families, community-based, common evaluation framework, service
integration, hubs. Research was global. Only articles written or previously translated in English
were assessed. There were close to 40 articles initially set aside for review, with some being
discarded because of duplication of subject material or irrelevance to the topic. All sources
were considered regardless of the year published. A total of 22 articles were used in this
literature review.
This annotated bibliography is divided into three sections that arose while the sources were
being reviewed:



Community-Based Evaluation and Practice
Child and Family (and Neighbourhood) Outcome Measures
Service Integration
Each source is reviewed as objectively as possible, with a two-page synthesis and analysis
section at the end of the document. All sources are listed alphabetically in Appendix 1.
B. Annotated Bibliography
B1. Community-Based Evaluation & Practice
Corter, C. and J. Pelletier (2004). The rise and stall of parent and community involvement in
school. Orbit. 34: 7-12.
The authors argue for a more critical look at parent and family involvement in schools. While
this idea is popular and promoted widely, and that boosting student learning and development
are the primary aims, it is argued that the goals for parental and community involvement and
the consequent training/orientation/practices to achieve those goals do not always match up.
And that the current popularity of parent and community involvement is due to the increased
pressures on children and families: single parenting, widening income gaps, immigration and
working mothers to name a few. The authors outline various academic and non-academic
benefits (e.g. stronger student achievement and learning, community-building) and costs (e.g.
compromising classroom ethics, negative effect on family life with increased pressure from
schools) to parental and community involvement…and focus on ‘mixed results’ examples of
school parent councils and the United Kingdom’s national approach to bettering early
14 | P a g e
childhood educational outcomes. In the end, the authors advocate for targeted involvement
strategies that are demonstrably connected to goals/benefits. For example: making sure school
councils have clearly articulated goals and purpose, using other parents and community
members to reach out to dis-engaged parents and supporting joint parent-teacher training so
parent involvement in the classroom is more informed.
Scharfe, E. (2011). "Benefits of mother goose: influence of a community-based program on
parent-child attachment relationships in typical families." Child Welfare 90(5): 9-26.
The author argues that participants in a Peterborough-based Parent-Child Mother Goose
(PCMG) program - a relatively inexpensive, community-based program – benefited equally to
those who are involved in professional services to “remedy parent-child attachment
difficulties.” Families who participated in the PCMG program also experienced higher levels of
parent-child attachment than families that were on the program waiting list (i.e. they did not
participate in PCMG during the time of the study). Since this study was conducted with
relatively stable, middle-class families who are often capable of asking for, and receiving
parenting assistance, the author suggests that a similar study conducted with high risk families
(who are less likely to seek out parenting support) would yield even higher positive results. The
author also highlights that the PCMG program teaches practical skills for good parenting vs. a
professional approach that focuses on the benefits of a program for children – an approach
where high-risk parents might feel judged. The author used the Relationship Scales
Questionnaire to measure mother’s attachment in close friendships and romantic relationships,
the security scale from the Waters and Deane Attachment Q-sort to measure child attachment
security, and Parenting Sense of Competence was used to assess the mother’s feelings of
efficacy and satisfaction with parenting.
Stoecker, R. (2005). Evaluation. Research Methods for Community Change: A Project-Based
Approach. Toledo, Sage: 181-209.
It is important to consider evaluation at the beginning and throughout a project or program,
not just at the end. This kind of “developmental” evaluation enables corrective learning along
the way. Evaluation choice #1: participatory or external? Participatory evaluation is usually
undertaken when there is unity among stakeholders, a shared desire for reflection and
improvement, and commitment to the time required. External evaluation is often chosen when
there isn’t unity among stakeholders, and/or a funder or the available time demands it. Groups
often choose a hybrid of these two approaches. Evaluation choice #2: outcome or process?
Outcome evaluation studies what happens at the end of a project; process evaluation studies
what happens during a project. Evaluation is an integrated process combining participatory
research with popular education, community organizing and participatory planning and applies
to all stages of a typical research project: determining the question (based on project goals),
choosing the methods (how will we know we’ve achieved the goals?), gathering the data
(sometimes easier said than done), analyzing the data (participants interpret the data;
facilitator suggests ‘creative tensions’) and presenting the data (ideally to project participants
before funders so a plan of action can be added to the evaluation). Beware of stakeholder
15 | P a g e
concerns about negative results, lack of control (especially for clients/constituents), and costs
(innovation and streamlining is key when budgets are tight). An integrated evaluation process
can attend to these concerns.
B2. Child and Family (and Neighbourhood) Outcome Measures
Aber, L., J. Berg, et al. (2010). Using Child Indicators to Influence Policy: A Comparative Case
Study. From Child Welfare to Child Well-Being. S. B. Kamerman. 1: 189-215.
This book chapter is a comparative case study of child indicators for policy change in Columbia,
France and South Africa from 1990 to present – specifically education. Study goal #1 was to
identify critical features of education-related child indicator systems and their use in influencing
policy – presenting features common across countries and those unique to individual countries.
Study goal #2 was to make recommendations about how to improve education-related child
indicators for policymaking. A common philosophical approach shared by all three countries
was a focus on children’s rights as a basis for policy-making. Despite this shared philosophical
approach, there were considerable differences related to educational policy objectives.
Identified indicators were divided into three categories roughly according to the policy-making
process: input (e.g. financial investment), process (e.g. student-to-educator ratio) and
outcomes (e.g. graduation rates). Indicators were also analyzed according to content, timing,
aggregation and quality (p. 205) – and whether or not indicators met with the “five purposes of
indicators for policy-makers” (i.e. for background info, to monitor needs & establish trends, for
goal-setting, to create accountability standards and to evaluate policy). The authors also argued
that despite an international/comparative/commonality focus, the utility of indicators in the
policymaking process must also be viewed in the context of national values, goals and priorities
for child policy. Finally, the authors “…recommend that national child indicator systems
privilege: methodological rigor over comprehensiveness; within-domain comprehensiveness
over across-domain comprehensiveness; and the use of policy to influence the design and
quality of child indicator systems (as well as the use of child indicator systems to improve the
design and positive impact of policy on children’s welfare and well-being).”
Ben-Arieh, A. (2010). From Child Welfare to Children Well-Being: The Child Indicators
Perspective. Children’s Well-Being: Indicators and Research 1. S. B. K. e. al., Springer
Science+Business Media: 9-22.
This book chapter chronicles the development of the child indicators movement since the early
1970’s. The “ecology of child development” is one theoretical approach to child welfare.
Bronfenbrenner’s bio-ecological model of human development has four dynamic and
interdependent concentric circles of environmental influence: Micro-system is strongest
influence (e.g. parents, school), followed by meso-system (i.e. connections between element
sin micro-system) then exo-system (societal context in which families live e.g. parents
workplace, conditions of neighbourhood) and finally the macro-system (i.e. wider societal
context like economic conditions, policies, global developments). Barriers and facilitators within
16 | P a g e
this model become the indicators of child well-being. The United Nation’s Convention on the
Rights of the Child (CRC) offers a second model through which to frame child well-being
indicators and is based on its four core principles: non-discrimination, best interest of the child,
survival and development and respecting the view of the child. A third and final model offered
by the author states the importance of the “new” sociology of childhood – recognizing that
childhood as an important developmental stage in addition to indicators related to a child’s
future. The author then goes on to discuss the importance of child-based indicators alongside
adult-generated and more objective, statistical-based indicators…and in some cases, moving to
child as the unit of measurement altogether (as opposed to families). Three main types of data
are outlined: administrative data, census and surveys and social research (longitudinal and ad
hoc). The article wraps up with a brief six-stage continuum of the shift from from a “child
welfare” perspective to a “child well-being” perspective, a call for a Child Welfare Composite
Index (local, national, international) and a ten-point summary of the state of the child indicators
movement today – including much of what has been stated above with the addition of a policyoriented focus.
Connell, J. P. and A. C. Kubisch (1998). Applying a Theory of Change Approach to the Evaluation
of Comprehensive Community Initiatives: Progress, Prospects, and Problems United
States of America, The Aspen Institute.
A theory of change is a set of short, medium and long-term outcomes combined with strategic
activities and indicators that will let individuals know when certain outcomes have been
achieved. A theory of change should be plausible (i.e. will the proposed activities lead to the
desired outcomes?), doable (i.e. will we have the resources to accomplish our goals?) and
testable (i.e. can progress be measured?) and be developed starting with long-range outcomes
and moving towards short-term outcomes and activities. The authors provide suggestions on
the do’s and don’ts for creating and implementing a theory of change. For example, in a
comprehensive community initiative (CCI) with multiple service providers and stakeholders,
don’t have strict standards of theory articulation too early in the process as it can undermine
participation and stifle dynamic energy. The authors also state that while it may be difficult to
negotiate multiple “theories of change” amongst service providers and stakeholders, trying to
find common ground with outcomes in the early stages of a CCI will lessen tension and possible
fragmentation when the time comes for resource allocation and evaluation. With this in mind,
the authors provide three ideas for consideration when measuring activities and outcomes in
CCI’s: a) Measurement of a CCI's activities is as important as measurement of its outcomes
(and ideally the results of one can be linked to the results of the other); b) The measurement
discussion must resolve the issue, "How good is good enough?" and c) The process of surfacing
and articulating the theory of change will provide important information about measuring
activities and outcomes – especially in terms of evaluation design and timing.
Coulton, C. J. (1995). Using Community-Level Indicators of Children’s Well-Being in
Comprehensive Community Initiatives. New Approaches to Evaluating Community
Initiatives: Concepts, Methods, Concepts. J. I. Connell, A. C. Kubisch, L. B. Schorr and H.
17 | P a g e
Weiss. Washington, D.C., The Aspen Institute: Roundtable on Comprehensive
Community Initiatives for Children and Families: 173-199.
The author begins by discussing the importance of local communities as units of analysis and
the different ways of understanding what a ‘community’ or ‘neighbourhood’ is. The author then
goes on to describe two types of community-level indicators (with many examples): outcomesrelated (i.e. measuring the status of children on various social, health, and developmental
outcomes) and contextually-related (i.e. the ways in which the community affects individualized
outcomes…such as the age distribution and number of affluent families in a neighbourhood).
Various methodological considerations for community-level data collection are addressed:
assignment of geographical location, small area limitations, reporting bias and error, small area
population estimates and change, the need for standardization and corruption of indicators.
The author argues frequently that the broader regional/city context must be considered when
trying to evaluate comprehensive community initiatives, along with a long-term perspective
encompassing the past and future. Indicator analysis can lead to identification of change
patterns over time in a single neighbourhood, comparisons across neighbourhoods (which also
allows for a regional perspective to emerge), and for clustering of similar neighbourhoods
which can lead to more efficient sharing of community development practices amongst
comprehensive community initiatives. The author advocates for community resident and leader
involvement in the process of neighbourhood indicator development and analysis, along with
government agencies and others who are mandated to collect data. Collaboration amongst
stakeholders with common goals will be a source for overcoming challenges with local-level
data usage like protecting confidentiality.
Coulton, C. J. and R. L. Fischer (2010). Using Early Childhood Wellbeing Indicators to Influence
Local Policy and Services. From Child Welfare to Child Well-Being. S. B. K. e. al., Springer
Science+Business Media. 1: 101-116.
This chapter focuses on the development and application of child well-being indicators at the
local level (i.e. regions, counties, cities, towns or neighborhoods), with a focus mainly on the
USA context. The chapter begins with background information on the importance of local
context for children and the relationship of community indicators to concerns about child
wellbeing within a local context. Local indicators that reveal pockets of concern or disparities in
child wellbeing can be used to target resources to areas where they are needed most. The
authors detail possible reasons for these differences: selection, contextual effects, institutional
and political resources, built environment, and spatial proximity. Next, the chapter reviews a
number of methodological issues and challenges that characterize local indicators work. Then,
the chapter discusses the infrastructure that is needed to sustain local indicators work,
including both producing child wellbeing indicators and seeing to it that they are used to
address policy and program. Finally, the authors provide several case studies that demonstrate
how child wellbeing indicators have been developed and applied in selected locales.
Sets of local indicators can be crafted to the needs of the local decision makers and augmented
over time as new strategies and needs arise. After the initial upfront investment of time and
resources in the creation of such capacity, the efforts required to maintain them are
18 | P a g e
proportionally less. This article illustrates the ways in which specific indicator data related to
child well-being have been used by local decision makers to change policies, design programs,
monitor progress and enhance community service strategies.
Dempsey, I. and D. Keen (2008). "A Review of Processes and Outcomes in Family-Centered
Services for Children With a Disability." Topics in Early Childhood Special Education
28(1): 42-52.
This article reviews present research regarding the relationship between the provision of
family-centered services and the achievement of positive outcomes for children and their
parents; it discusses the links between help-giving practices and child and family outcomes for
families who have a member with a disability and summarizes the extant research in an
accessible format while identifying areas for future research. Family-centered philosophy
developed from both help-giving and empowerment literatures. Because the philosophy has
been widely adopted as best practice in a variety of disciplines, it is crucial that important gaps
in the research backing for family-centered support are addressed. This article suggests that
further research focused on the processes and outcomes of family-centered practice is needed.
The authors emphasize using evidence-based practice to enhance integrity, providing
information to practitioners that will be immediately useful, and highlight the need for the
testing of a coherent model to explain how the work of family-centered practitioners affects
families and their children. Another suggested approach is to facilitate the testing process of
family-centered philosophy. The article continues to review case studies that consider service
delivery processes, achieved outcomes for parent and child, and moderating variables. From
this, the authors assert that for outcomes to be optimized, help-giving needs to do much more
than foster a respectful and empathetic relationship between professionals and parents.
Fisher, H., P. Pecora, et al. (1999). Improving the Quality of Children's Services: A
Working Paper on Outcomes-Based Models of Service Delivery and Managed Care.
Englewood, Colorado, USA, The Casey Outcomes and Decision-Making Project: 135.
In the paper’s executive summary, the authors caution against the sole use of outcomes as a
basis for service delivery, stating that other things such as service quality, governance and cost
effectiveness - in the context of child and family outcomes, values and principles - are
important for determining how well programs and services are serving children and families.
The authors then discuss a conceptual framework for child welfare reform put forward by User,
Gibbs and Wildfire. This conceptual framework contains four domains of focus: public policy
context, program management and structure, program operations and program impact. That
said, the authors advocate for a system that is driven by service values and principles, which in
turn drive outcome domains (e.g. safety, permanence and improvements in functioning and
well-being), domain-specific outcomes, interventions (i.e. a specific service) and measurable
indicators. Later in the paper, the authors argue that if such a community-based system is
developed and cost-effective services are reported on, it can also begin to drive public policy
and related funding. The main body of the paper begins with an outline of the issues facing
outcomes-based systems (including eight ‘pressure-points’ for the increased focus on
19 | P a g e
outcomes) and continues with an overview of the opportunities for using outcomes-based
frameworks. The paper then provides many concrete suggestions for developing an outcomesbased framework for delivering community-based (or ‘managed care’) child welfare services,
including a set of ‘definitions’ for different terms used in outcomes-oriented frameworks, an
example of a 24-indicator framework, eight key steps for developing outcome measures and
nine steps for moving into an outcomes-based model of service delivery. This builds on an
earlier report in the same series titled Assessing Outcomes in Child Welfare Services: Principles,
Concepts, and a Framework of Core Outcome Indicators, which “…presents philosophical and
conceptual principles for child welfare services and a suggested framework of core indicators
for an outcomes-based model of service delivery.” The paper finishes with a brief discussion
about decision-making implications for outcome-based service delivery, including the danger of
mis-attribution for outcomes not achieved (e.g. low post-service employment rates being linked
to service delivery instead of a downturn in the economy), and the potential of increased
involvement of families in planning and decision-making processes from start to finish.
Kramer, M., M. Parkhurst, et al. (2009). Breakthroughs in Shared Measurement and Social
Impact. Boston, MA, FSG Social Impact Advisors: 54.
This paper examines twenty efforts to develop shared approaches to performance, outcome, or
impact measurement across multiple organizations. The results demonstrate three different
breakthroughs in shared measurement: Shared Measurement Platforms, a common online
platform for data capture and analysis, including field specific performance or outcome
indicators (e.g. Success Measures Data System); Comparative Performance Systems, a common
online platform for data capture and analysis in which all participants within a field use the
same measures, uniformly defined and collected (e.g. Cultural Data Project); Adaptive Learning
Systems, an ongoing participatory process that enables all participants to collectively measure,
learn, coordinate, and improve performance (e.g. aligning goals among different organizations
when dealing with aspects of a single complex issue). The authors claim many funders face a
difficult choice when it comes to understanding the results of their grants: they may hire a
third-party evaluator to study the grant-funded program and incur an additional expense, or
they may accept a grantee’s self-report, which often lacks hard data and objective analysis. The
performance and outcome measurement systems described in this paper offer a third
alternative that falls between those extremes. It is suggested that these systems offer timely
performance and outcome data about funded programs while imposing a minimal cost and
reporting burden. Recognizing that no single initiative can solve major social problems, these
breakthroughs offer ways to increase the efficiency, knowledge, and effectiveness of the entire
system of interrelated organizations that affect complex social issues. Rather than measure
whether a single grant has achieved impact, Adaptive Learning Systems provide a collaborative
process for all participating organizations to learn, support each other’s efforts, and improve
over time. The research of these authors suggest that the idea of measuring performance
across multiple organizations — and funders’ willingness to support those efforts — is still very
new, but that ultimately shared outcome measures help create transparency and accountability
and thereby enable the identification and development of higher performing organizations.
20 | P a g e
Peters, R. D., K. Petrunka, et al. (2003). "The Better Beginnings, Better Futures Project: A
Universal, Comprehensive, Community-Based Prevention Approach for Primary School
Children and Their Families." Journal of Clinical Child and Adolescent Psychology 32(2):
215-227.
The article outlines a study of three Better Beginnings, Better Futures Project sites from 1993 to
1998 (Better Beginnings is “…“[a] 25-year longitudinal prevention policy research
demonstration project to provide information on the effectiveness of prevention as a policy for
children”). The research “…[e]valuated a community-based, universal project designed to
prevent emotional and behavioral problems and promote general development in young
children, while also attempting to improve family and neighborhood characteristics, to link
effectively with existing services, and to involve local residents in project development and
implementation.” The study involved three sites in the Province of Ontario and two ‘control’
sites where no Better Beginnings project was implemented. Each site involved participation by
schools, child and family service providers, neighbourhood residents and other stakeholders in
the provision of a wide range of activities. Extensive evaluation instruments were used to
measure children’s mental health and development, parent and family functioning, community
characteristics, site-specific project development and program model analysis and economic
analysis. Results showed some improvements in different areas across all Project sites and
some improvements were site-specific (e.g. “…teacher ratings of children’s overanxious
emotional problems showed a significant cross-site pattern of decline, that is, across all three
Better Beginnings sites…”). Other areas that were measured showed no significant
improvement (e.g. “… [t]here were no patterns of improvement on any of the measures of
cognitive development or on measures of reading or mathematics achievement”). Overall, the
authors highlight the fact that intensive, multi-year and often classroom-based interventions
generated the most positive results – particularly related to improving social-emotional
functioning of young children. Also, regular home visits to parents and systematic parent
involvement in school-based activities at one site were likely contributors to higher
improvements in parent, family, school and neighbourhood measures. The authors point out
that although costs for this site were the highest of the three, compared to other intervention
programs, all three Better Beginnings sites were quite modest in cost. Finally, the authors point
out that each Better Beginnings site is locally developed and operated with significant local
resident involvement in all decisions – a process that is not insignificant in terms of having
indirectly positive effects on child, family and neighbourhood health.
Trocmé, N. (2003). The importance of process in developing outcome measures. National
Outcomes Symposium - Centre of Excellence for Child Welfare. Ottawa, Ontario,
Canada.
Despite the call for outcomes measurement in the past 25 years, child welfare services continue
to be delivered primarily by evidence of need not evidence of service effectiveness. That said,
evaluation research is often an unaffordable luxury hindered by cost and concerns about “what
gets measured, gets funded.” Determining common child welfare objectives in Canada (from
which indicators can be developed) has proven to be extremely difficult. Competing local
21 | P a g e
objectives have revealed tensions that can be summarized in three over-lapping common
objectives: Child Protection, Family and Community Support and Child Well-Being. An effective
outcome measurement system must track each objective while at the same time recognizing
the “ecological” (i.e. interrelated) dimensions of child maltreatment. The three main uses and
purposes of outcomes measurement are case management (practitioners), program
management (administrators) and program treatment outcome research (researchers). An
incremental multi-level outcomes-development strategy should begin with an emphasis on
making use of systems-based indicators that can be easily collected and standardized. Use of
multiple indicators can help overcome limitations of understanding any one indicator
measurement. A less centralized strategy is recommended for developing clinical assessment
and case management tools – with better information sharing supporting practitioner
transition from case planning to assessment and case planning. Although the author calls for an
incremental outcome development strategy that allows clinicians, managers and researchers to
separately develop the tools most appropriate to their needs, there is the danger that these
initiatives will lead to approaches that are not complementary and cannot eventually be
integrated. With this in mind, it is important to keep the bigger picture in mind – the idea that
all stakeholders are working towards common objectives for positive child and family welfare.
Trocmé, N., B. MacLaurin, et al. (2009). National Child Welfare Outcomes Indicator Matrix
(NOM). Montreal, QC, Canada, Centre for Research on Children and Families, McGill
University: 8.
The purpose of a National Child Welfare Outcomes Indicator Matrix (NOM) is to develop a set
of indicators to track outcomes related to the 200,000 children and youth who come into
contact with child welfare authorities every year across Canada and the 67,000 children and
youth who are, at any one day of the year living in out-of-home care. The NOM provides a
framework for tracking outcomes for children and families receiving child welfare services that
can be used as a common set of indicators across jurisdictions. The NOM includes four nested
domains: child safety, child well-being, permanence, and family and community support that
connect with ten indicators. These ten indicators were selected based on data that is readily
available, non-identifying, aggregated client data and should not be examined in isolation. The
NOM is intended for use by child-welfare managers and policy-makers and is not designed for
clinical decision-making. The ten indicators are: recurrence of maltreatment and serious injuries
and deaths (safety), school performance and child behaviour (well-being), out-of-home
placement, permanency status and moves in care (permanence), and family moves, ethnocultural placement matching and parenting (family and community support). There is some
criticism of current Management Information Systems that are guided by financial accounting
concerns vs. a Child Tracking System which would link service events to children, youth and
families for more nuanced statistical reporting.
Underwood, K., I. Killoran, et al. (2010). Have a Voice Project Report (Best Start Strategy –
Parent Outcomes). Toronto, Ryerson University: 40.
22 | P a g e
This report documents the experiences of families using Best Start services and programs in
three Ontario communities (Timiskaming, Rural Lambton and Chatham-Kent, and East
Hamilton). Each community emphasizes a different approach to service delivery: hub, virtual
hub (through a Community Liaison Worker) and adjacent (i.e. service providers in different
locations). Using Appreciative Inquiry (to avoid a deficit-based approach), researchers consulted
relevant literature, conducted an inventory of existing evaluation methods and data and
conducted primary research using focus groups and questionnaires. Findings are divided into
two main categories: “facilitators” for parents using services and parents’ perceived service
“outcomes.” “Facilitator” areas included: program/staff quality, range of programs/services,
and accessibility. “Outcomes” were divided into: support (e.g. financial, personal, parenting),
child development, children’s education, parent-family bonding, social/personal benefits, and
transitions to new environments. The research findings (i.e. parent respondents) did not focus
on the different service delivery models (e.g. comparative effectiveness, organizational
processes and practices).
B3. Service Integration
Brown, K. and K. White (2006). Exploring the evidence base for Integrated Children’s Services.
Edinburgh, Scottish Executive Education Department.
This report identifies that there is a large body of literature on the success factors for, and
barriers to effective child and family service integration, and the devastating impacts of nonintegrated service. While this may seem encouraging towards implementing service integration,
the author suggests that progress is being hindered by a) the many definitions and
understandings about what “service integration” means (Annex A on page 22 provides a list of
terminology that relates to varying notions of service integration) and b) the lack of reliable
outcomes research related to the effect of service integration (e.g. does service integration lead
to better grades in school or better parent-child attachment?). The author also points out that
there is little evidence on the economic effectiveness of service integration. For measuring
outcomes and economic effectiveness, a long time frame is needed, along with an evaluation
research methodology that limits improper attribution of effect to service integration. The
author points out that the majority of evidence is from the service provider perspective and is
focused on the process of integration vs. outcomes for children and families.
Corter, C., J. Bertrand, et al. (2002). TORONTO FIRST DUTY STARTING GATE REPORT:
Implementing Integrated Foundations for Early Childhood. Toronto: 84.
This report chronicles the first two ‘development’ years of the Toronto First Duty Project – a
demonstration project for child and family service integration that began in 1999. The report
provides an overview of child and family services internationally, in Canada, provincially (i.e.
Ontario) and locally in Toronto around the time the report was written (2002). The report then
goes on to describe the goals of the Toronto First Duty Project and provides detailed progress
reports from all five Duty First pilot sites. Overall findings recognize the difficulty and
23 | P a g e
complexity of program and service integration (due to many factors like inertia and a changing
political landscape) – within and across unique pilot sites. Overall findings also indicate that
despite slow progress, stakeholder and funder willingness to affect positive change over the
long haul is energizing.
Corter, C., J. Bertrand, et al. (2007). TORONTO FIRST DUTY PHASE 1 FINAL REPORT: Evidencebased Understanding of Integrated Foundations for Early Childhood. Toronto: 99.
This report chronicles Phase 1 implementation of the Toronto First Duty Project (TFD) - from
2002 to 2005. A clear purpose and vision for the project is presented:
…[T]o provide early learning and care for every child…at the same time support[ing]
parents to work or study and in their parenting role. Toronto First Duty (TFD) envisions
regulated child care, kindergarten and family support programs consolidated into a
single, accessible program delivery platform that is located in primary schools and
coordinated with early intervention and family health services.
Beginning with another review of the international, Canadian, provincial and local contexts for
child and family service integration, the report outlines a comparative, case-study evaluation
plan for the five different TFD pilot sites divided into three main categories: a) program, policy
and services; b) children and parents; and c) community and public awareness. Data collection
methods and tools are also outlined – from literature reviews, to interviews and surveys to the
use of more quantitative measures such as “intake and tracking” system data and creating and
monitoring common “indicators of integration change” across the pilot sites in the five
categories outlined in the TFD terms of reference: the early learning environment, the early
childhood staff team, governance, seamless access, and parent participation. Highlights from
“program, policy and services” evaluation findings included: service integration does not cost
more than traditional service delivery and provides more flexible and stable options to children
and families; integration from a human resources perspective is more successful when staff
have the time to develop common goals and share information, wisdom and professional
development opportunities; joining existing services and programs is more difficult than
starting new ones; including heavily regulated child care is one of the most challenging aspects
of service integration; and that leadership for organizational change is essential – especially
when local, unique contexts can vary greatly. For parents, the report authors state that the TFD
approach appeared to “increase their capacity and confidence in helping their children learn
and in communicating with the school and teachers in kindergarten.” (p.81) Parents also
reported a high level of satisfaction with TFD programs and with the concept of service
integration. Report authors stated that the evaluation was “…not designed to directly test
[children’s] outcomes.” (p.81) That said, the Early Development Instrument (EDI) was used as
the primary tool for measuring the impact of service integration. Report authors stated that
‘already good’ levels of impact were raised even higher as a result of service integration. In
relation to community and public awareness, intake and tracking data revealed that the TFD
sites were successful in attracting similar children and families to the child and family groups
outlined in their demographic profiles. With Phase 1 completed, TFD moved into a
24 | P a g e
‘dissemination’ phase - providing practical, high level/overview service integration resources
(http://www.toronto.ca/firstduty/) and continuing “integration” learning at one pilot site for
three more years (Bruce WoodGreen Early Learning Centre).
Corter, C., J. Pelletier, et al. (2009). Toronto First Duty Phase 2, 2006-2008: Final Research
Report. Toronto, Atkinson Centre for Society and Child Development, Institute of Child
Study/Department of Human Development and Applied Psychology, Ontario Institute
for Studies in Education/University of Toronto: 59.
A summary report of the full 2009 report provided the following findings highlights: a) An
integrated staff team improves program quality; b) An integrated approach to early childhood
services encourages programming appropriate to the developmental needs of young children;
c) Integrated early childhood service delivery reduces daily stress and hassles in family life; and
d) Integrated early childhood delivery facilitates parent participation in their child’s early
learning. The summary report also outlined implementation challenges for each key finding in
light of the Ontario Government’s new Full Day Learning initiative for 4 and 5-year-olds. The
authors call for provincial-level innovation and collaboration to sustain unique, local Toronto
First Duty-like approaches for combining education, child care and family supports in Ontario
communities.
Janmohamed, Z., J. Pelletier, et al. (2011). Toronto First Duty, Phase 3: The Bruce WoodGreen
Case Study. Toronto, ON, Atkinson Centre for Society and Child Development,
OISE/University of Toronto.
This report is essentially a check-in on how the Ontario Government’s Full Day Learning
Program for 4 and 5-year-olds is progressing. The authors focus on the tensions resulting from
the pairing of Kindergarten Teachers with Early Childhood Educators in full-day learning
environments – with research results echoing the findings from the Toronto First Duty Phase
Two Report in 2009 (see above). Challenges for these two different types of professional
working together included: lack of common goals, difference in pay and benefits, and perceived
duplication of roles despite different training. Keys to success included time for joint program
planning and training, having strong school and program leadership and increased access to
supports and resources via local and external curriculum specialists.
Kaga, Y., J. Bennett, et al. (2010). Caring and Learning Together: A cross-national study on the
integration of early childhood care and education within education. Paris, France,
UNESCO.
The report begins by outlining the differences between early childhood care and education
(ECCE) – and that most countries offer these services via ‘split-systems’ (i.e. one Ministry
responsible for childcare and one Ministry responsible for education). “Differences between
services in welfare and education in key areas such as access, regulation, funding and
workforce, lead to problems of inequality and lack of continuity for children, parents and
workers.” Two main strategies have been employed to counteract these problems: more
25 | P a g e
coordination and integration. Evidence suggests that coordination across Ministries works
when ECCE efforts are aimed at a target population, but are less successful when applied
towards an overall, comprehensive ECCE system. Integration occurs when ECCE is delivered
through one Ministry. The report documents integration examples through educational
ministries and social service ministries – stating that success has less to do with where ECCE is
located and more to do with having a holistic approach to child development and education.
The report documents examples of the history, processes and consequences of ECCE
integration. Interestingly, benefits of ECCE integration are framed in terms of “benefits sought”
(or goals)…and that if the following goals are set, they have a greater likelihood of being
achieved: 1) universal entitlement, 2) affordable access, 3) a unified and well educated
workforce, 4) enhancing learning for all ages, and 5) smoother transitions for young children.
The authors also outline many operational benefits of ECCE integration, including “…rethinking
the purpose, provision and practice of ECCE across all age groups, including children both under
and over 3 years; changed perceptions of ECCE among the workforce, parents and the wider
public, including greater recognition of its pedagogical value; a higher valuation given to staff
working in ECCE; the creation of a stronger ECCE system that enjoys parity with and can
influence compulsory education; greater coherence in policy; the reduction or elimination of
inequalities between services for children under and over 3 years; and increased resourcing for
ECCE through merging administrations and eliminating duplication.” Potential negative
consequences of ECCE integration are also why countries that maintain ‘split-systems’ resist the
idea of integration: separate and long-standing differences in culture between childcare and
education sectors, fear that childcare will be overwhelmed by the education sector, and fears
about the cost of integration. To affect positive, efficient and long-lasting change, political will
at all levels of government, childcare and education sectors are necessary along with a concrete
strategy for change. Assessment of integration ‘success’ for children and families was based on
varying perspectives on ECCE attendance rates (e.g. rural/urban, socio-economic effect of
universal access, etc).
Ontario, G. o. (2011). Building our Best Future: Realizing the Vision of Ontario Best Start Child
and Family Centres - An Update. M. o. C. a. Y. Services.
This report is an update to the Ontario Government’s Best Start initiative – initiated in 2004 to
create a framework for an integrated child and family service system through the development
of Ontario Best Start Child and Family Centres. The report chronicles recent consultation efforts
including a summary of input from parents & caregivers (they want: streamlined, accessible,
efficient and consistent service delivery), and service providers & administrators (they want:
equitable, easily accessible and high quality services for clients, definitions of ‘service
integration’ and ‘seamless service’ to guide outcome measurement, the ability to develop
localized solutions, to increase integration as long as it doesn’t disrupt what’s working). The
report outlines a working vision for Best Start Centres along with seven guiding principles,
including a focus on having a family-centred approach, providing intentional support to families
(i.e. understanding why the service is being provided and having a sense of the broader support
system), and improving outcomes for families and children. There is a government commitment
to provide ‘best practice’ research and monitoring results on the various aspects of service
26 | P a g e
integration and to support an innovation fund for local community projects related to service
integration. The report also mentions an initiative for three Ministries to work in a more
integrated way related to speech and language services for children from birth to the end of
grade three. The report concludes with the idea of developing an Outcomes Index to “…more
broadly measure and monitor child, family, community and social outcomes as well as broader
system functioning outcomes.”
C. Synthesis and Analysis (in no particular order)
Outputs, outcomes and the difficulty of attribution
Because it is easy to confuse program outputs and outcomes, and there are so many different
interpretations of desired child welfare “outcomes,” it will be critical to define things at the
outset of the OV common evaluation framework project. For example, is reducing length of stay
for children in out-of-home care an output or outcome? What about attendance rates? It is also
important to choose evaluation outcomes and indicators that are relatively easy to
measure…and to try and focus on clusters of indicators that can be analysed and discussed
altogether – as one indicator alone does not often indicate attribution of a particular service
intervention.
Keeping purpose, community-based and “what’s doable” in perspective
Trocmé’s 2003 keynote address at the National Outcomes Symposium - Centre of Excellence for
Child Welfare provides an anchoring perspective on the reasons why we might want to
measure child and family outcomes (e.g. for manager, case worker or researcher). Indeed, most
sources that were reviewed provided differing purposes for evaluation research like measuring
outcomes, understanding neighbourhood effects, providing information to funders/investors
and the quality of service integration. Taking things one step further, Stoecker’s chapter on
community-based research as evaluation provokes the reader into considering a strong
purpose-oriented foundation for their work, along with tangible step-by-step, communitybased suggestions for doing the research. That said, it was saddening that some child and
family service evaluations were not always rooted in community-based, outcomes-oriented
activities despite the fact that overall child welfare objectives and philosophies appeared
similar for all cases. That said, despite the existence of over-arching models of child welfare
outcome domains and indicators (e.g. the National Outcomes Matrix) and calls for community
engagement, it is important for local communities to focus on producing evidence that’s useful
with the resources that are available…all the while looking for opportunities to promote a
provincial (and national?) agenda for long-term sustainability.
Criticisms of service integration
Although service integration is not the main focus of this contract, I felt that it was important to
review some literature on the topic because service integration is a key component of the
Ontario Government’s Early Years Strategy. Most literature focused on, or criticized, how most
evaluations focus on the process of service integration (how) vs. improved outcomes as a result
of service integration (why). As an example, the 2006 Toronto First Duty “Guide to Early
27 | P a g e
Childhood Service Integration” focuses largely on the process of integration and less on
measuring common outcomes. Conversely, the Better Beginnings project in Ontario was
focused on child outcomes from the get-go and as a result, has produced positive service
integration results in those communities.
Clarity is key
The importance of communicating one’s message clearly and briefly cannot be overstated.
There is a proliferation of material related to this topic and if it’s not presented in an accessible
format, it will be useless to busy professionals and community leaders. Similarly, the process for
developing a “living” common outcomes framework must also be clear, mutually-agreeable and
well-documented so everyone involved understands what is happening (e.g. having a logic or
theory of change model). There must also be many avenues for ongoing feedback and dialogue.
Tools for tracking outcomes
When the Otonabee Valley community decides its future direction with respect to a common
outcomes framework, the sources reviewed for this bibliography contain a wealth of
information about specific tools and indicators used to track various outcomes related to child
success – and possible individuals that might be able to help with localized development. For
example, the 2008 Dempsey article showcases “…[i]ndependent, moderating and dependent
variables, and instruments reported in [35] family-centered research studies.”
28 | P a g e
Annotated Bibliography Sources
Aber, L., J. Berg, et al. (2010). Using Child Indicators to Influence Policy: A Comparative Case
Study. From Child Welfare to Child Well-Being. S. B. K. e. al., Springer Science+Business
Media. 1: 189-215.
Ben-Arieh, A. (2010). From Child Welfare to Children Well-Being: The Child Indicators
Perspective. Children’s Well-Being: Indicators and Research 1. S. B. K. e. al., Springer
Science+Business Media: 9-22.
Brown, K., & White, K. (2006). Exploring the evidence base for Integrated Children’s Services.
Edinburgh: Scottish Executive Education Department.
Connell, J. P. and A. C. Kubisch (1998). Applying a Theory of Change Approach to the Evaluation
of Comprehensive Community Initiatives: Progress, Prospects, and Problems United
States of America, The Aspen Institute.
Corter, C., J. Bertrand, et al. (2002). TORONTO FIRST DUTY STARTING GATE REPORT:
Implementing Integrated Foundations for Early Childhood. Toronto: 84.
Corter, C. and J. Pelletier (2004). The rise and stall of parent and community involvement in
school. Orbit. 34: 7-12.
Corter, C., J. Bertrand, et al. (2007). TORONTO FIRST DUTY PHASE 1 FINAL REPORT: Evidencebased Understanding of Integrated Foundations for Early Childhood. Toronto: 99.
Corter, C., J. Pelletier, et al. (2009). Toronto First Duty Phase 2, 2006-2008: Final Research
Report. Toronto, Atkinson Centre for Society and Child Development, Institute of Child
Study/Department of Human Development and Applied Psychology, Ontario Institute
for Studies in Education/University of Toronto: 59.
Coulton, C. J. (1995). Using Community-Level Indicators of Children’s Well-Being in
Comprehensive Community Initiatives. New Approaches to Evaluating Community
Initiatives: Concepts, Methods, Concepts. J. I. Connell, A. C. Kubisch, L. B. Schorr and H.
Weiss. Washington, D.C., The Aspen Institute: Roundtable on Comprehensive
Community Initiatives for Children and Famlies: 173-199.
Coulton, C. J. and R. L. Fischer (2010). Using Early Childhood Wellbeing Indicators to Influence
Local Policy and Services. From Child Welfare to Child Well-Being. S. B. K. e. al., Springer
Science+Business Media. 1: 101-116.
Dempsey, I. and D. Keen (2008). "A Review of Processes and Outcomes in Family-Centered
Services for Children With a Disability." Topics in Early Childhood Special Education
28(1): 42-52.
29 | P a g e
Fisher, H., P. Pecora, et al. (1999). Improving the Quality of Children's Services: A Working Paper
on Outcomes-Based Models of Service Delivery and Managed Care. Englewood,
Colorado, USA, The Casey Outcomes and Decision-Making Project: 135.
Janmohamed, Z., J. Pelletier, et al. (2011). Toronto First Duty, Phase 3: The Bruce WoodGreen
Case Study. Toronto, ON, Atkinson Centre for Society and Child Development,
OISE/University of Toronto.
Kaga, Y., J. Bennett, et al. (2010). Caring and Learning Together: A cross-national study on the
integration of early childhood care and education within education. Paris, France,
UNESCO.
Kramer, M., M. Parkhurst, et al. (2009). Breakthroughs in Shared Measurement and Social
Impact. Boston, MA, FSG Social Impact Advisors: 54.
Ontario, G. o. (2011). Building our Best Future: Realizing the Vision of Ontario Best Start Child
and Family Centres - An Update. M. o. C. a. Y. Services.
Peters, R. D., K. Petrunka, et al. (2003). "The Better Beginnings, Better Futures Project: A
Universal, Comprehensive, Community-Based Prevention Approach for Primary School
Children and Their Families." Journal of Clinical Child and Adolescent Psychology 32(2):
215-227.
Scharfe, E. (2011). "Benefits of mother goose: influence of a community-based program on
parent-child attachment relationships in typical families." Child Welfare 90(5): 9-26.
Stoecker, R. (2005). Evaluation. Research Methods for Community Change: A Project-Based
Approach. Toledo, Sage: 181-209.
Trocmé, N. (2003). The importance of process in developing outcome measures. National
Outcomes Symposium - Centre of Excellence for Child Welfare. Ottawa, Ontario,
Canada.
Trocmé, N., B. MacLaurin, et al. (2009). National Child Welfare Outcomes Indicator Matrix
(NOM). Montreal, QC, Canada, Centre for Research on Children and Families, McGill
University: 8.
Underwood, K., I. Killoran, et al. (2010). Have a Voice Project Report (Best Start Strategy Parent Outcomes). Toronto, Ryerson University: 40.
30 | P a g e
Appendix 2: Otonabee Valley Family Hub Output and Outcomes Inventory
As of October 24, 2012
Introduction
The following service providers were/are involved in the OV Family Hub between fall 2011 and fall 2012. The list was generated from
the OV Family Hub Annual Report 2011-2012 and multiple issues of the Parenting Education Community Guide issued by the
Peterborough Family Resource Centre (fall 2011 to fall 2012). Target group categories (for each activity) were taken from the
Parenting Education Community Guide. This list is a “point-in-time snapshot” of OV Family Hub activities - it is not a complete list! It
is meant as a starting point for initiating conversations about working together to improve child and family outcomes in the OV Family
Hub, OV School community and neighbourhood.
Peterborough Family Resource Centre
“…providing information and support to expectant parents, families with children from birth to age 6, and child care
providers”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
Long Term
(how much did
Outcomes
Outcome
Italicized added by Todd
we do?)
Staff from
Peterborough Family
# of children and
Parents/caregivers
Parents/caregivers Children are
community
Resource Centre offers
families attending attend programs
knowledge of
ready to be fully
agencies/school registered programs for
programs
services is
participating
boards
parents/caregivers:
increased
members of
Parents and children
society
experience positive
 You and Your Baby # of referrals to
PRFC staff
other
services
interactions
Parents/caregivers
 Songs and Signs
are referred to
for Babies and
services
Families
# of program
Program
Toddlers
appropriate to their
sessions
offered
feedback/improvement
 Toddlers and
needs
Best Start
Tantrums
Planning Table
# of agencies
 Every Parent’s
Parents/caregivers
providing services
Survival Guide
access service in a
 Esso Family Math
timely manner
# of families who
 PALS - Family
self-identify as
Literacy Program
Parents/caregivers
living in conditions
 The Gentle Art of
develop increased
of risk
31 | P a g e




Infant Massage
Fighting &
Aggression: Help
your child get along
with others
Families Connect
Parent-Child
Mother Goose
Steps and Stages
(includes Well-Baby
Clinic)
sense of parenting
competence
Parents identify a
decreased sense of
isolation
Family Play to Learn
programs (non-registered)
Community agency
registered programs e.g.
collective kitchen, KHCAS,
Wee Watch
Community agencies offer
enhanced supports to
Peterborough Family
Resource Centre programs
e.g. 5CCC resource
teacher, LDAP resource
facilitation
Peterborough Family
Resource Centre offers
services in outreach
locations
Target group(s): Prenatal,
Infant, Toddler/Preschool,
32 | P a g e
Parenting, School-Age,
Information/Support
Groups
Preliminary OV Outcome Cluster(s):
Notes: PFRC programs requiring registration are done via OV hub registration form. Mandatory ‘session program data’ form
captures: attendance by participant type; number of participants turned away; redeemed participant vouchers; referrals by service
provider (i.e. who a client was referred to); transportation stats; misc. items including # of Nippising screens, presentations, and
workshops delivered; types of NPF resources accessed; in-kind staff contributions (hours, agency), and early literacy stats (hours of
service, who was served). End-of-course evaluation survey is also used for each program. It is unclear how the end-of-session
survey and ‘session program data’ form measure achievement of some expected outcomes. PFRC produces an OV Hub Annual
Report that includes program and service updates, attendance statistics and quotes from past participants. “Parents and community
partners have been involved in focus groups, design of the logic model and opportunity to give feedback at the advisory committee
level.” A logic model for OV hub ‘programs and services’ and ‘service integration’ was provided (pasted above), along with a Brighter
Futures 2012-2014 logic model.
Five Counties Children’s Centre
“…helping children with physical, communication and developmental problems”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
(how much did
Outcomes
we do?)
Parents,
Music and Movement for
# of clients
Achievement of
children, Five
Preschoolers (5 x one hour
family/child goals
Counties staff, sessions; 3 hours weekly).
# of visits
OV school
Program
staff
Participation in this program
feedback/improvement
# of paid hours
is by referral through Five
Counties.
Long Term
Outcome
Target group:
Toddler/Preschool
Preliminary OV Outcome Cluster(s):
Notes: Participant intake is through regular Five Counties registration system (including a new child/family history form - in addition
to referral form - that is inputted into a database). Participant goals are set and monitored through the development of an
individualized “care plan” with parents/caregivers. An end-of-program survey is also used. An internal report is generated periodically
and includes staff feedback.
33 | P a g e
Input
Activity
Parents,
children, Five
Counties staff,
OV school
staff, PFRC
staff
Preschool Resource
Services (2 hour visit monthly
to PFRC’s Play to Learn
Program).
Output/Indicator
(how much did
we do?)
It would be
interesting to
track…
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
# of referrals
Drop-in participation – no
referral necessary. This
service can be a bridge to
referral.
Information
request topics
Target group: Parenting
Preliminary OV Outcome Cluster(s):
Notes: The Nippising District Developmental Screen is available for parents/caregivers and service providers to assist with
developmental delay diagnosis (available online). Participation in this service has been very successful – lots of engagement and
proof that the hub model works (i.e. Five Counties staff in fewer places more frequently vs. old model of mobile service delivery).
Input
Activity
Parents,
children, Five
Counties staff,
OV school staff
Speech & Language Services
(3 hours weekly (started in
May 2012).
Participation in this program
is by referral through Five
Counties.
Output/Indicator
(how much did
we do?)
# of clients
# of visits
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
See below for
description of
‘FOCUS’ tool
development.
# of paid hours
Target groups:
Toddler/Preschool, SchoolAge
Preliminary OV Outcome Cluster(s):
34 | P a g e
Notes: Participant intake is through regular Five Counties registration system (including a new child/family history form - in addition
to referral form - that is inputted into a database). Assessment is done using standardized speech and language assessment tools.
Participant goals are set and monitored through the development of an individualized “care plan” with parents/caregivers. Postassessment ‘FOCUS’ outcome measurement tool is in final stages of development and will likely be implemented this fall (tool
available online). Tool focuses on child development related to: speech, expressive language, pragmatics, attention, intelligibility,
social/play, independence, and coping strategies/emotions. Information will be inputted into a provincially-centralized database.
Input
Activity
Children,
service
provider staff,
OV school staff
Early Learning School
(interaction with students JK
to 3rd grade)
Participants are students
enrolled at OV school.
Output/Indicator
(how much did
we do?)
Multiple (e.g.
types of support
needs, wait times
for service
approvals)
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
Individualized goals
with percentage
achievement rating
Individualized
goals with
percentage
achievement
rating
Individualized
goals with
percentage
achievement
rating
Target group: School-Age
Preliminary OV Outcome Cluster(s):
Notes: This is a speech therapy demonstration pilot project (41 children) between KPRDSB, Five Counties, Community Care Access
Centre and Ministries of Children and Youth Services, Ministry of Health and Ministry of Education. Assessment is done using
standardized speech and language assessment tools. Data collection is done via spreadsheet and submitted to the Province.
Service provider client intake is separate. Service providers and OV school staff check-in regularly with one another and all attend
regular meetings with the Province.
City of Peterborough - Get REAL Drop-In Program for Youth
“…offering fun recreational programming.”
Input
Activity
Output/Indicator
(how much did
we do?)
Youth, Get
Get REAL
# of youth
REAL staff, OV This drop-in program
attending drop-in
school staff
provides youth (ages 13-18)
with a fun, safe place to hang
out weekly with staff mentors
(ages 19-27). Activities
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
It would be
interesting to
track…
Greater self35 | P a g e
include:
 Access to mentors
 Sport activities including:
volley ball, basketball,
soccer, dodge ball,
parachute games and
activities, floor hockey.
 Cooking classes monthly
where youth choose what
they’d like to cook
 Weekly snacks to provide
healthy options for youth
 Weekly craft and board
game activities
 Special lessons like
music lessons (piano &
guitar), yoga, dance, selfdefense, etc.
esteem
Creating close
relationships with
friends and staff
Strengthening
personal morals
and values
Target group: Teen
Preliminary OV Outcome Cluster(s):
Notes: MSExcel attendance sheet; Have used evaluation surveys from other youth agencies in the past – these are no longer
available.
Kawartha-Haliburton Children's Aid Society
“…ensuring the best interests, protection and well-being of children.”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
Long Term
(how much did
Outcomes
Outcome
we do?)
Parents,
Standard Triple P (multi# of participants
It would be
KHCAS staff, session parenting resource
interesting to
OV school
group; open to KHCAS clients # of participants
track…
staff
only - registration required)
indicating an
understanding of
How families used
Target group: Parenting
course material
new parenting
techniques and
36 | P a g e
# of referrals to
behavioural and
home support
programs (i.e. at
KHCAS or in the
home)
impacts
(Supporting clients
to track…) clientdefined goals.
Self-esteem for
parents and
children (b/c/ not
necessarily
connected to good
grades)
Preliminary OV Outcome Cluster(s):
Notes: End-of-program surveys for clients and service providers are used – NOT from Triple P. In addition to asking about general
satisfaction and program feedback, client survey also asks how the course supports problem-solving and healthy partner
relationships. KHCAS is creating a client evaluation survey for lower literacy levels. No evaluation report is available. KHCAS
suggested delivery of this program at OV hub.
Parents,
children, Five
Counties
staff, OV
school staff,
PFRC staff
Establishing Effective
Routines with Children (3 x 2.5
hour visits to PFRC’s Play and
Learn Program)
Drop-in; no KHCAS open file
necessary. This service can
be a bridge to referral.
A parenting resource group
where CAS provides supplies
to create child routine charts –
no KHCAS open file needed.
Drop-in…3 x 2.5 hour visits to
Play to Learn program.
Target group: Parenting
# of referrals to
behavioural and
home support
programs (i.e. at
KHCAS or in the
home)
It would be
interesting to
track…
It would be interesting
to track…
It would be
interesting to
track…
Participant satisfaction
Impacts resulting from
use of child routine
charts
How families used
new parenting
techniques
(Supporting clients
to track…) clientdefined goals
# of participants
Self-esteem for
parents and
children (b/c/ not
necessarily
connected to good
37 | P a g e
grades)
Preliminary OV Outcome Cluster(s):
Notes: No formal evaluation is conducted but program was very popular - in the first two months of delivery KHCAS provided
supplies for 100 routine charts. PFRC invited CAS to deliver program at OV hub.
Kinark Child and Family Services
“Caring, helping, healing - so children and youth can live socially and emotionally healthy lives.”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
(how much did
Outcomes*
we do?)
Standard Triple P (multiParenting Scale:
session parenting resource
 Laxness
group; open to Kinark clients
 Verbosity
only - registration required)
 Over reactivity
Had hoped to offer Standard
Triple P at OV school but had
to re-locate the group because
of scheduling conflicts.
Target group: Parenting
Long Term
Outcome
Scores are clinical
and non-clinical –
the goal is to move
from clinical to
non-clinical while
participating in the
course – with no
further intervention
required. 80% of
clients who start in
clinical move to the
non-clinical range.
Strengths and
Difficulties
Questionnaire
(child behaviour):
38 | P a g e
 Pro social Skill
 Hyperactivity
 Conduct
problems
 Emotional
behaviour
 Peer
relationships
Parents rate child
behaviour in terms
of impact. 80% of
children whose
parents participate
in the course move
to the normal
range of impact
(levels are: normal,
borderline and
abnormal).
Preliminary OV Outcome Cluster(s):
Notes: Kinark uses Triple P evaluation tools for Standard Triple P course. Completed surveys are delivered to Kinark’s corporate
office for database input. Outcomes for level 4 are noted above. *80% is for all five Kinark locations/regions – stats are broken-down
locally on an as-needed basis by Kinark Peterborough staff. The goal is to have local evaluation stats/feedback feed into staff
planning. Kinark also uses other Triple P pre- and post-evaluation survey tools for other Triple P courses they deliver. Tools are
varied and relate to parent experience/parenting (including depression/anxiety, relationship quality, discipline, and
communication/parenting style), child strengths and difficulties & client satisfaction - survey use depends on focus(es) of the course
being offered.
Learning Disabilities Association of Peterborough (LDAP)
“…committed to enabling individuals with learning disabilities to reach their full potential.”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
(how much did
Outcomes
we do?)
Long Term
Outcome
39 | P a g e
Parents,
children,
LDAP staff,
OV school
staff, PFRC
staff
Resource Facilitation (2.5 hour
drop-in visit 2x/weekly to
PFRC open or registered
programs)
It would be
interesting to
track…
Attendance
Resource Facilitation offers
components of LDAP’s Power
to Achieve Program for
parents. Resource facilitation
can be a bridge to referral.
Purpose: Raise awareness
about learning disabilities and
ADHD and support child
advocacy; Resource
Facilitation offers a one-to-one
approach – about 20 minutes
per parent. Often the parents
had questions and this leads
to information provision &
troubleshooting/strategies for
dealing with learning
disabilities and ADHD.
It would be interesting
to track…
Impact of intervention
for parents and their
children
Target group: Parenting
Preliminary OV Outcome Cluster(s):
Notes: Delivered in 2010-11 - about 23 parents accessed the service. Not delivered in 2011-12 due to resource limitations. In
general: an end-of-program survey is used for the Power to Achieve Program (questions relate to client satisfaction, program
feedback, learning disability awareness and parent skill development).
Nursery Two Child Care
“…providing a challenging, creative program and to promote the overall growth and development of young children.”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
Long Term
(how much did
Outcomes
Outcome
we do?)
40 | P a g e
Children,
Nursery Two
staff, OV
school staff
Before and After School Age
Program (ages 6-12)
# of program
participants
Hours are 7:30am-8:50am and
3:15pm-5:30pm. Program
takes place in one
Kindergarten Room and the
school gym. Program operates
Monday to Friday, September
to June.
Target group: School Age
Preliminary OV Outcome Cluster(s):
Notes: Enrolment has steadily increased since opening in September 2011. “As an Organization (as a whole not just at Otonabee
Valley) we use the following tools: child assessment tools, environmental assessment tools, parent surveys, staff surveys, child
surveys. At this point in time we have only used staff and parent surveys at our Otonabee Valley location.”
Follow-up request for parent and staff survey templates sent June 19, 2012 and August 23, 2012
Input
Activity
Children,
Nursery Two
staff, OV
school staff
Extended Day Program
(JK/SK)
Output/Indicator
(how much did
we do?)
# of program
participants
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
Target group: School Age
Preliminary OV Outcome Cluster(s):
Notes: “As an Organization (as a whole not just at Otonabee Valley) we use the following tools: child assessment tools,
environmental assessment tools, parent surveys, staff surveys, child surveys. At this point in time we have only used staff and parent
surveys at our Otonabee Valley location.”
Follow-up request for parent and staff survey templates sent June 19, 2012 and August 23, 2012
Peterborough County-City Health Unit
41 | P a g e
“Through community programs and by developing better health policies…working to improve the living conditions of those
in need for the benefit of all.”
Input
Activity
Output/Indicator Initial Outcomes
Intermediate
Long Term
(how much did
Outcomes
Outcome
we do?)
Parents,
Infant Toddler Development
It would be
It would be interesting
children,
Program
interesting to
to track…
Health Unit
track…
staff, OV
Providing informal consultation
Knowledge levels of
school staff,
on child development to
# of referrals
parents (about child
PFRC staff
parents attending one of
development and
PFRC’s drop-in groups (i.e.
community resources)
provide information, refer to
other programs and resources
Program
in the community, or to accept
feedback/improvement
a parent referral to the Infant
Toddler Development
Program). Can be a bridge to
other referrals too.
Target group: Parenting
Preliminary OV Outcome Cluster(s):
Notes: Suggestion to have an overall hub evaluation plan – especially for drop-in programs that “add-on” to PFRC programs. It’s
difficult for Health Unit staff to justify two hours at drop-in vs. time spent doing home visit (primary method for service delivery).
Input
Activity
Parents,
Health Unit
staff, OV
school staff
Collective Kitchen (1x/month;
June 2011 to June 2012);
Come Cook With Us (6 x 2
hour sessions)
Come Cook with Us is a
component of the
Output/Indicator
(how much did
we do?)
Key participant
comments
Attendance rates
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
Learning…
to prepare new
recipes
new cooking skills
how to cook food
42 | P a g e
Peterborough County-City
Health Unit Food Security
Health Promotion Project. This
project will be addressing
barriers to healthy eating that
are faced by individuals and
families living on low incomes.
The overall goal of the Food
Security Health Promotion
Project is: To partner with
community members and
organizations to develop a
coordinated system of food
security programs that
ensures all community
members have access to
affordable, nutritious and
personally acceptable foods.
The objectives of Come Cook
With Us are:
 To prepare new foods and
try new foods at home.
 To practice new cooking
methods.
 To store, thaw, prepare
and reheat foods safely.
 To share ideas on menu
planning, shopping, and
favorite meals.
 To make healthy meals
based on Canada’s Food
Guide to Healthy Eating
and share healthy eating
safely
to prepare healthy
foods that are new
to use the information
and skills learned to
prepare a meal at
home
to purchase healthier
foods than before
attending the class
-Able to afford to cook
the recipes learned at
the program
-Program
feedback/improvement
43 | P a g e

tips.
To give out ingredients and
vouchers so that
participants can afford to
try the meal at home.
Target groups: Teens,
Parenting
Preliminary OV Outcome Cluster(s):
Notes: Come Cook With Us uses a weekly/monthly report tracking table template and end-of-course evaluation survey. Re:
community involvement: “…[i]ssues are discussed at class and group is part of the problem solving process. Other issues may be
decided by team and shared with group for discussion and agreement.”
Input
Activity
Output/Indicator
(how much did
we do?)
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
Raising Sexually Healthy
Children (was offered in a
single workshop format)
Sex education and sexual
abuse prevention for children
and caregivers.
Target groups: School Age,
Teen, Parenting
Preliminary OV Outcome Cluster(s):
Notes: Service no longer delivered in workshop format. Health Unit has switched to one-on-one consultations and train-the-trainer
approach (e.g. training teachers to deliver curriculum).
Wee Watch Enriched Home Child Care
“…providing reliable, quality, licensed home daycare since 1984.”
Input
Activity
Output/Indicator Initial Outcomes
(how much did
Intermediate
Outcomes
Long Term
Outcome
44 | P a g e
we do?)
Care
providers,
children, Wee
Watch staff,
OV school staff
Monthly Provider Playgroup (in
2011-12: third Wednesday of
every month)
It would be
interesting to
track…
It would be
interesting to
track…
Open to all Wee Watchaffiliated daycare providers and
their children. Crafts, activities,
snack, circle story time,
connecting with Wee Watch
themes (e.g. Terry Fox)
Role modeling
impacts on daycare
providers (crafts,
child behaviour
support, snack
ideas, supporting
provider-defined
goals)
Child development
(e.g. fine motor,
gross motor,
cognitive,
math/science,
language,
manners)
Target group:
Information/Support Groups,
Infant, Toddler/Preschool
Impacts of social
experience that
monthly playgroup
provides for
children
Child safety (e.g.
street-proofing,
avoiding child
abuse)
Overcoming
barriers to
transportation
Preliminary OV Outcome Cluster(s):
Notes: Transportation was a stumbling block when the playgroups were centralized – the hub model has proved to be more effective
for two local providers but transportation is still an issue. In general: Wee Watch intake is centralized through head office via online
intake form. Each child has a ‘developmental’ book. Wee Watch providers also use a daily log for each child. Wee Watch providers
complete an annual evaluation survey that gets compiled by head office. It is unclear as to how the evaluation information is used at
the local level and whether it is used for policy/management purposes and/or program improvement.
Input
Activity
Output/Indicator
(how much did
we do?)
Initial Outcomes
Intermediate
Outcomes
Long Term
Outcome
45 | P a g e
Care
providers,
children, Wee
Watch staff,
OV school staff
Professional Development
(infrequent evening and
weekend use of OV hub space)
Professional development for
Wee Watch providers;
providers and parents; often
through other service providers
but sometimes through Wee
Watch too.
Preliminary OV Outcome Cluster(s):
Notes: List of Wee Watch-led PD topics unavailable at this time.
Community Counseling & Resource Centre
Input
Activity
Initial Outcomes
Intermediate
Outcomes
Credit Counselling (1 x two
hour visit to PFRC’s Play to
Learn Program; drop-in)
It would be
interesting to
track…
It would be
interesting to
track…
Service provides:
What piece of
information made a
difference in terms
of improving a
client’s personal
finances? Following
up for positive
impact stories…
The impact of
providing unbiased information
a. General information about
money management
issues, budgeting,
consumer/collection/credit
issues, banking, interest,
financial literacy, etc.
Output/Indicator
(how much did
we do?)
Long Term
Outcome
The impact of
trustworthy
outreach and safe
space for dialogue
b. Provide a booklet to each
participant that we publish
called: Help You Help
Yourself, A guide to making
the most of your money
c. Bring other handouts or
46 | P a g e
information sharing tools
topical to the season I am
visiting. (e.g. November I
might bring Holiday
Spending handouts or
Reducing Heating Costs)
d. Create awareness of safe,
reliable online resources
e. General opportunity for
families to learn more about
our services and where to
turn for help
Program can be a bridge to
referral.
Target groups: Parenting,
Information/Support Groups
Preliminary OV Outcome Cluster(s):
Notes: “I love the hubs!” An opportunity to let people know that there is somewhere to go if they have a financial problem and to
bring in un-biased information about people’s rights, safe borrowing information, etc. In general: CCRC client intake allows for the
setting and tracking of individualized goals. Credit Counselling evaluation form (for a standard training session) is primarily for
program improvement except for general question about helpful learning moments. CCRC would like to develop a shorter evaluation
form for the drop-in.
47 | P a g e
Appendix 3: Output and Outcomes List from Service Providers - OV Family Hub
Output and Outcomes List from Service Providers - OV Family Hub
as of: October 24, 2012
Italicized words represent possible outputs and outcomes to be tracked (output and outcome tracking 'hopes and dreams')
Note: The following list of service providers, outputs and outcomes were/are involved in the OV Family Hub between fall 2011 and fall 2012.
The list of service providers was generated from the OV Family Hub Annual Report 2011-2012 and multiple issues of the Parenting Education
Community Guide issued by the Peterborough Family Resource Centre (fall 2011 to fall 2012). This list is a “point-in-time snapshot” of service
providers involved in OV Family Hub activities - it is not a complete list! It is meant as a starting point for initiating conversations about working
together to improve child and family outcomes in the OV Family Hub, OV School community and neighbourhood.
Service providers included in this version of the inventory: Five Counties Children's Centre, City of Peterborough - Recreation Department (Get
REAL program), Kawartha-Haliburton Children's Aid Society, Kinark Child and Family Services, Learning Disabilities Association of Peterborough,
Nursery Two Child Care, Peterborough County-City Health Unit, Peterborough Family Resource Centre, Wee Watch Enriched Home Child Care,
and Community Counselling Resource Centre.
Output or Outcome Description
Outputs
# of children and families attending programs
Target Group
Name of Service
Service
Provider
All
Multiple
PFRC
# of referrals to other services
All
Multiple
PFRC
# of program sessions offered
All
Multiple
PFRC
# of agencies providing services
# of families who self-identify as living in conditions of risk
All
All
Multiple
Multiple
PFRC
PFRC
48 | P a g e
# of clients
Toddler/Preschool
# of visits
Toddler/Preschool
# of paid hours
# of referrals
Information request topics
Toddler/Preschool
Parenting
Parenting
Toddler/Preschool,
School-Age
Toddler/Preschool,
School-Age
Toddler/Preschool,
School-Age
# of clients
# of visits
# of paid hours
Multiple (e.g. types of support needs, wait times for service
approvals)
# of youth attending drop-in
# of participants
# of participants indicating an understanding of course
material
# of referrals to behavioural and home support programs
Music and Movement for
Preschoolers
Music and Movement for
Preschoolers
Music and Movement for
Preschoolers
Preschool Resource Services
Preschool Resource Services
Five Counties
Five Counties
Five Counties
Five Counties
Five Counties
Speech & Language Services
Five Counties
Speech & Language Services
Five Counties
Speech & Language Services
Five Counties
School-Age
Teen
Parenting
Early Learning School
Get REAL
Standard Triple P
Five Counties
City of Ptbo
KHCAS
Parenting
Parenting
Standard Triple P
Standard Triple P
Establishing Effective Routines with
Children
Establishing Effective Routines with
Children
Resource Facilitation
Before and After School Age
Program
Extended Day Program (JK/SK)
KHCAS
KHCAS
# of referrals to behavioural and home support programs
Parenting
# of participants
Attendance
Parenting
Parenting
# of program participants
# of program participants
School Age
School Age
KHCAS
KHCAS
LDAP
Nursery Two
Nursery Two
49 | P a g e
# of referrals
Parenting
Key participant comments
Teens, Parenting
Attendance rates
Initial Outcomes
Parents/caregivers attend programs
Parents and children experience positive interactions
Program feedback/improvement
Teens, Parenting
Achievement of family/child goals
Toddler/Preschool
Program feedback/improvement
Individualized goals with percentage achievement rating
Toddler/Preschool
School-Age
Participant satisfaction
Parenting
Impacts resulting from use of child routine charts
Impact of intervention for parents and their children
Knowledge levels of parents (about child development and
community resources)
Parenting
Parenting
Program feedback/improvement
Parenting
All
All
All
Parenting
Learning…to prepare new recipesnew cooking skillshow to
cook food safelyto prepare healthy foods that are newto use
the information and skills learned to prepare a meal at hometo
purchase healthier foods than before attending the class
Teens, Parenting
Infant Toddler Development
Program
Collective Kitchen; Come Cook With
Us
Collective Kitchen; Come Cook With
Us
Health Unit
Health Unit
Health Unit
Multiple
Multiple
Multiple
Music and Movement for
Preschoolers
Music and Movement for
Preschoolers
Early Learning School
Establishing Effective Routines with
Children
Establishing Effective Routines with
Children
Resource Facilitation
Infant Toddler Development
Program
Infant Toddler Development
Program
PFRC
PFRC
PFRC
Come Cook with Us
Health Unit
Five Counties
Five Counties
Five Counties
KHCAS
KHCAS
LDAP
Health Unit
Health Unit
50 | P a g e
Able to afford to cook the recipes learned at the program
Program feedback/improvement
Teens, Parenting
Teens, Parenting
Role modeling impacts on daycare providers (crafts, child
behaviour support, snack ideas, supporting provider-defined
goals)
Information/Support
Groups, Infant,
Toddler/Preschool
Monthly Provider Playgroup
Wee Watch
Impacts of social experience that monthly playgroup provides
for children
Information/Support
Groups, Infant,
Toddler/Preschool
Monthly Provider Playgroup
Wee Watch
Overcoming barriers to transportation
Information/Support
Groups, Infant,
Toddler/Preschool
Monthly Provider Playgroup
Wee Watch
Parenting,
Information/Support
Groups
Credit Counselling
CCRC
All
Multiple
PFRC
All
All
Multiple
Multiple
PFRC
PFRC
All
All
Multiple
Multiple
PFRC
PFRC
Toddler/Preschool,
School-Age, Teen
School-Age
Teen
Teen
Speech & Language Services
Early Learning School
Get REAL
Get REAL
Five Counties
Five Counties
City of Ptbo
City of Ptbo
What piece of information made a difference in terms of
improving a client’s personal finances? Following up for
positive impact stories…
Intermediate Outcomes
Parents/caregivers knowledge of services is increased
Parents/caregivers are referred to services appropriate to their
needs
Parents/caregivers access service in a timely manner
Parents/caregivers develop increased sense of parenting
competence
Parents identify a decreased sense of isolation
Child development related to: speech, expressive language,
pragmatics, attention, intelligibility, social/play, independence,
and coping strategies/emotions.
Individualized goals with percentage achievement rating
Greater self-esteem
Creating close relationships with friends and staff
Come Cook with Us
Come Cook with Us
Health Unit
Health Unit
51 | P a g e
Strengthening personal morals and values
Teen
City of Ptbo
Parenting
Get REAL
Standard Triple P; Establishing
Effective Routines with Children
Standard Triple P; Establishing
Effective Routines with Children
Standard Triple P; Establishing
Effective Routines with Children
How families used new parenting techniques and impacts
Parenting
(Supporting clients to track…) client-defined goals
Self-esteem for parents and children (b/c/ not necessarily
connected to good grades)
Parenting
Parenting Scale:
• Laxness
• Verbosity
• Over reactivity
Parenting
Standard Triple P
Kinark
Strengths and Difficulties Questionnaire (child behaviour):
• Pro social Skill
• Hyperactivity
• Conduct problems
• Emotional behaviour
• Peer relationships
Parenting
Standard Triple P
Kinark
Child development (e.g. fine motor, gross motor, cognitive,
math/science, language, manners)
Information/Support
Groups, Infant,
Toddler/Preschool
Monthly Provider Playgroup
Wee Watch
Child safety (e.g. street-proofing, avoiding child abuse)
Information/Support
Groups, Infant,
Toddler/Preschool
Monthly Provider Playgroup
Wee Watch
The impact of providing un-biased information
Parenting,
Information/Support
Groups
Credit Counselling
CCRC
The impact of trustworthy outreach and safe space for
dialogue
Parenting,
Information/Support
Groups
Credit Counselling
CCRC
KHCAS
KHCAS
KHCAS
52 | P a g e
Long-Term Outcomes
Children are ready to be fully participating members of society
Individualized goals with percentage achievement rating
All
School-Age
Multiple
Early Learning School
PFRC
Five Counties
53 | P a g e
Appendix 4: Small Group Output and Outcome ‘Elements’ by Category
To be added.
54 | P a g e
Appendix 5: Roundtable Participants
First Name
Hallie
Todd
Holly
Alex
Simon
Nancy
Molly
Last Name
Atter
Barr
Raymond
Cranfield
Dadds
Fischer
Gall
Chelsey
Nancy
Suzanne
Nicola
Mathew
Mary-Ann
Sandra
Emmy
Kellie
Kathy
Gray
Jackson
LaRue
Lyle
Martin
Meagher
Robinson
Ruttle
Walden
Warner
Organization
Peterborough County-City Health Unit
Trent Centre for Community-Based Education
Peterborough Free Methodist Church
Five Counties Children's Centre
Kawartha-Haliburton Children's Aid Society
City of Peterborough – Social Services
OV School - Breakfast program
City of Peterborough - Recreation Department
(Get REAL program for youth)
Community Counselling & Resource Centre
Living Hope & Come Cook With Us
Peterborough Family Resource Centre
Neighbours in Action (PPRN)
Peterborough Family Resource Centre
City of Peterborough – Children’s Services
Peterborough Family Resource Centre
Peterborough Family Resource Centre
Peterborough Family Resource Centre
55 | P a g e
Appendix 6: Themed Group Discussion Notes
“How can we measure/track this outcome together given realistic resources?”
1. Volumes of service/program outputs (e.g. attendance)
 Key measures are unique client counts and attendance
 To avoid duplicating clients, we would need a common identifier – perhaps a card –
either manual or electronic.
 Need to work through shared client consent – for attendance data only.
 Have one agency to house the data and only share aggregate data.
 Tracking referrals is very different for different agencies. Some referrals are made
directly to the agency via the parent or just as an aggregate for a group
 Could just count aggregate referrals out to start with
 Ask clients how they heard about the program and share the info with other agencies
 Many of these questions and the consent issue could be built into a common intake
process
 Language varies a lot between agencies – need common definition and common
processes
 Re # of paid hours: It’s measure of efficiency. It’s up to each agency to do their own
evaluation but if information is beneficial to hub sustainability then share with the hub
2. Adult-related outcomes/skills
 Explore the idea of a common pre- and post-questionnaire
 Start in registered programs where there is consistent attendance rather than drop-in
programs where you don’t necessarily see a family regularly
 Use technology to make the process easier. (e.g. electronic surveys, communication
mechanisms)
56 | P a g e
 Look beyond program satisfaction surveys to trying to measure behaviour change
 Each organization could ask one simple pre and post “One thing I will try,” and then
follow up three months later. Or after a certain amount of time, ask “Tell me about how
you used the information in the workshop, program, etc.?”
 Need to develop client consent mechanism prior to client engagement. It’s not
insurmountable…School for Young Moms has an example
 Design evaluations that try to pull out a key piece of knowledge, skill, information (i.e.
what was the purpose of the intervention and did it meet its purpose based on parent
report?) For example: For a workshop on child safety, can participants identify at least
one specific new learning like…crib bumpers are not recommended?
 Can we better tie our programs together? For example: in our Cooking Program can we
bring Credit Counselling in to do food budgeting?
 We could start by looking at families who participate in programming from numerous
service providers. Is there a bigger impact for families who attend more than one
program?
 Measure increase in knowledge of services (by parents)
 Link to research on resiliency and having a network within community/neighbourhood
3. Program and service feedback7
 Have a quiz or a test. This may be used in a traditional course like food safety where
certification is required
 Written, electronic or verbal survey. Participant offered choices to participate, with
participants asked open ended questions like: I learned…/the most important thing I
learned was…/I wish I had learned…
 Cookies for comments
 Want to track usage of multiple programs
7
Editor’s note: There appears to be two different understandings of “feedback and evaluation.” One
understanding is feedback related to program delivery (e.g. quality of instruction, accessibility of workshop
location, etc.). The other understanding appears to be related to general techniques for obtaining feedback and
evaluation overall.
57 | P a g e
 Common intake survey
 Common exit or discharge survey
 Try to connect with participants six months or one year after they leave
 Use email when possible
 Share expertise between agencies (i.e. I can facilitate a focus group for you and you for
me). Create a ‘speakers bureau’ of facilitators
 Try to reach out to current & past clients and people who only come once to programs
for feedback
 Create year over year data and compare
 Would like to track 18 year-olds when they exit
 Stories and focus groups can be rich sources of information
 Look at the time, resources available and purpose of your evaluation before deciding on
a strategy; do a cost/benefit analysis; are there any power dynamics at play?
4. Child-related outcomes
 The group discussed consistent use of existing inventories/screening tools by all service
providers. The three discussed were:
o Triple P (Positive Parenting) measuring behavioural development;
o Nippising District Developmental Screen which measures the development of
speech, expression, etc. and is a helpful tool to compare between what parents
are seeing and what service providers have noticed in a child’s development;
o Early Development Instrument (EDI) scores done on all SK students in the school
system by teachers.
 One form for client consent for information gathered on evaluations that are to be
shared between agencies
 Check with other hubs outside the region to see what tools they are using to measure
common child-related outcomes
 Tap into evaluation databases that already exist (e.g. Kinark).
58 | P a g e