WYMAN`S NATIONAL NETWORK Wyman`s National Network

Running Head: WYMAN’S NATIONAL NETWORK
Wyman’s National Network:
Bringing Evidence-Based Practice to Scale
COA Innovative Practices Award Case Study Submission
Emily Black
Wyman Center
WYMAN’S NATIONAL NETWORK
Abstract
Wyman Center is a nonprofit organization with a mission to enable teens from economically
disadvantaged circumstances to lead successful lives and build strong communities. Wyman's
Teen Outreach Program® (TOP®) is an evidence-based best practice youth development
program that provides critical supports and opportunities to young people during adolescence.
Based on success in St. Louis, Wyman recognized the potential for replicating TOP® on a
national scale and committed to building a National Network of partners and providers who can
deliver TOP® - and other evidence-based practices – with fidelity and at such scale that teens
and communities at-risk are positively impacted by the proven outcomes. To realize this goal,
Wyman employed an innovative and comprehensive replication approach which advances
promising practices through four key principles: fidelity, flexibility, affordability and sustainability.
In July of 2009, Wyman’s National Network launched, beginning partnerships with a variety of
organizations with the need, desire, and capability to reach large numbers of teens at-risk with
effective programming. The innovative capacity building strategy includes partner identification
and recruitment, intensive program training, periodic on-site program observations and audits,
and technical assistance by Wyman staff. In 2013, the National Network has the capacity to
reach 50,100 teens in 32 states and Washington D.C.
WYMAN’S NATIONAL NETWORK
Background
Wyman Center is a nonprofit organization with a mission to enable teens from
economically disadvantaged circumstances to lead successful lives and build strong
communities. With a commitment to delivering real results, Wyman bridges theory and practice,
using evidence-based and evidence-informed approaches to positively impact the lives of young
people – providing the supports and opportunities necessary to thrive in life, work and learning.
Wyman's Teen Outreach Program® (TOP®) is an evidence-based best practice youth
development program that provides critical supports and opportunities to young people during
adolescence. TOP® is grounded in contemporary research, which promotes the positive
development through curriculum-guided, interactive group discussions, positive adult guidance,
and community service learning. It is specifically designed for young people in 6th to 12th grade
and lowers risk of negative behaviors1 (e.g. course failure, school suspension, pregnancy, etc.)
while delivering on three essential goals: healthy behaviors, life skills and a sense of purpose
(see Appendix, TOP® Logic Model).
As one of Wyman’s most successful, scalable and affordable programs, TOP® helps
teens acquire the skills needed for positive outcomes and successful transitions into adulthood.
Wyman recognized the potential impact in replicating this program on a national scale and
committed to building a National Network of partners and providers who can deliver TOP® - and
other evidence-based practices - with fidelity to teens at-risk nationwide.
Goals
Wyman’s goal is to build a National Network of partners that supports faithful and
affordable replication of evidence-based practices at such scale that teens and communities atrisk are positively impacted by the programs’ proven outcomes. Beginning with Wyman’s Teen
Outreach Program®, Wyman aims to reach 250,000 teens cumulatively by 2015. As the National
1
Philliber Research, 2001. “Who Benefits Most from a Broadly Targeted Prevention Program?”
WYMAN’S NATIONAL NETWORK
Network grows, and partners begin to ramp up their reach, intermediary measures include the
number of partners, and the number of states in which TOP® is currently being replicated.
Approach
To realize this goal, Wyman is employing an innovative and comprehensive replication
approach which advances promising practices through four key principles: fidelity, flexibility,
affordability and sustainability.
To ensure the fidelity of the program as it expands, National Network partners must
agree to adhere to strict program delivery, data collection and evaluation guidelines. Wyman’s
approach to fidelity and certification process is designed to assure faithful replication with fidelity
monitoring tools from training to implementation, supporting partners to maintain certification
status, sustain funding, and deliver high quality programs (See Appendix, Approach to Fidelity).
The replication strategy balances the need for program fidelity with a degree of
customization or tailoring to the local culture and context in which the partner operates. Wyman
leverages the experience and reach of partners and allows them to tailor the program to the
specific needs of the teens they serve. For example, the TOP® framework is flexible enough to
be used in the school setting, in after-school programs, or an out-of-school enrichment program.
Wyman’s National Network offers a highly affordable program for teen pregnancy
prevention and youth development. The cost per participant in TOP® is significantly less than
the cost of the only other comprehensive youth development program with strong evidence of
pregnancy risk reduction.2 The average cost of $620 includes the assumed salary of facilitators,
which may already be covered in school or agency budgets, or may not exist if volunteers are
used as facilitators, thus making the costs even lower and more sustainable (See Appendix,
Economic Benefits).
To grow to scale (defined as minimum of 65-100 partners), Wyman is investing in
people, technology, marketing, and government relations, using “build, learn and grow”
2
The Brookings Institution, 2007. “Cost Effective Investments in Children”
WYMAN’S NATIONAL NETWORK
investments to expand systems and processes and add other resources necessary to enable
sustainable, affordable scale.
Implementation
Wyman’s National Network in July 2009. Wyman began identifying appropriate funding
and pursuing partnerships with a variety of organizations with the need, desire, and capability to
reach large numbers of teens at-risk with effective programming. The innovative capacity
building strategy includes regional partner identification and recruitment, intensive program
training, periodic on-site program observations and audits, and technical assistance by Wyman
staff.
Partner Identification and Recruitment
Primary target partners for scale are school systems and large regional nonprofits and/or
government agencies. These organizations are large and cost-effective, and can pick up many
of the functions required to ensure delivery with fidelity. Secondary target partners are small to
mid-sized local nonprofits with the capacity to serve several hundred hard-to-reach (due to
location or other barriers) youth annually who are at-risk and in need of services. Replication partners must be willing to follow the TOP® model of fidelity, and have
secured or identified a funding source which will cover start-up and first year of operational
expenses (See Appendix, Partner Screening Tool). Wyman’s National Network regularly
identifies and shares funding opportunities in the field of health, juvenile justice, and child
welfare, with potential partners. Once funding is secured and contractual obligations are met,
training and delivery begin. Training and Monitoring for Fidelity
The network is evaluated and maintained by Wyman and training is a vital first step that
infuses program content in a consistent way to all partners. Wyman’s “train the trainer” model
WYMAN’S NATIONAL NETWORK
teaches partners how to deliver an evidence-based program with fidelity. Wyman conducts fiveday comprehensive training sessions with experiential learning elements and "teach-back"
opportunities. The training is designed to support partners’ ability to develop deeper skills in
training staff, in monitoring and coaching for quality, and in overall strategies in youth
development. It provides partners with skill-building lessons that will make them more effective
in all areas of program management, delivery and evaluation, teen engagement, and program
facilitation. Oftentimes partners and their program providers are inexperienced working with the
structure and disciplined approach of a curriculum like TOP®. Wyman’s training and support
helps them increase their skills in delivering the type of program that will lead to significant and
sustainable results. Training occurs prior to program implementation and as needed.
To measure program impact, each Wyman partner measures teen progress through a
pre-survey at the beginning of the program year and a post-survey at the end of the program
year. The teen survey measures such outcomes as course failure, suspension, pregnancy,
attitude, physical and emotional safety, and engagement with community. Facilitators are
surveyed at the mid-point and end of the program year, regarding implementation of the
program (e.g. dosage, number of teens, etc.). Metrics from teen and facilitator reports determine
evidence based outcomes and patterns for successful implementation (See Appendix, TOP
Club and Partner Survey Metrics). TOP®’s requirements with regard to data collection and
reporting further instill both the knowledge and a level of discipline and analysis that help
partners maximize the impact of their efforts. The National Network’s proprietary systems also
enable partners to generate instant reports that they may use for program evaluation, grant
seeking, and related purposes. When partners understand and put into practice this disciplined
approach, they quickly recognize the benefits and seek to adopt similar approaches with all of
their youth development programs.
Field visits, audits and technical assistance by Wyman staff help partners hone their
skills, identify and address issues quickly, and ensure continuous quality improvement at the
WYMAN’S NATIONAL NETWORK
point of program delivery. Each certified partner receives eight hours of technical assistance per
year in order to foster quality program replication. This service is included in their annual
membership fee. Wyman follows up with TOP® providers through partner audits, which may
consist of additional club observation, data, reports, or review of records to ensure efficiency of
systems. Certified partner audits are based on Wyman discretion, and are conducted as
needed. All TOP® partners are certified by Wyman and undergo recertification on a bi-annual
basis. This ensures that the partner organization maintains high standards, meet program
requirements, and deliver quality programming. Required certification site visits ensure fidelity to
program and contract agreements. Specific criteria determine if a partner becomes re-certified,
provisionally certified or de-certified (See Appendix, Certification Criteria Rubric). Site visits
occur during the first year of implementation and every two years thereafter.
In addition, to share best practices, field observations occur when Wyman determines
that a certified partner’s successful program or systems provide learning opportunities for the
National Network.
Community of Practice
National Network partners are part of a community of practice through which they may
interact with peers who are committed to delivering excellence. As seen in the California
Institute of Mental Health (CIMH)3 Community Development Team Model, “forming peer-to-peer
networks to support easy exchange of information about implementation challenges and
solutions” is a key component of an effective evidence-based program implementation
approach. To facilitate this approach The National Network is bound together through an online
community. Certified replication partners are able to connect directly to Wyman with questions,
comments or concerns, and benefit by receiving updates on curriculum, share lessons learned
3
Todd Sosna, PhD, and Lynn Marsenich, LCSW, Community Development Team Model, Supporting the Model Adherent Implementation of Programs and Practices, The California Institute of Mental Health, October 2006
WYMAN’S NATIONAL NETWORK
and program delivery tips, access information on funding sources and other resources, and
download a library of useful materials to support implementation. As a result, they learn from
their participation in the National Network and are able to develop workable solutions to issues
and challenges through their interaction with their peers across the country.
Through TOP®’s community service learning requirement, partners become more
engaged with other community organizations and develop stronger affiliations and
collaborations with other service providers, funders, policy makers and local officials. This has
the added benefit of raising awareness of youth development needs and forming proactive
collaborations across a broad spectrum of community organizations that can promote and
advance a “pro-teen” culture and attitude in the local community.
Results
Over the past three and a half years, Wyman’s National Network has grown to include
57 certified replication partners that include school districts, nonprofit organizations, public
health agencies, and other youth-serving community and government agencies. These partners,
in turn, work with an estimated 275-300 providers whose program facilitators run 550 registered
TOP® clubs in 32 states and the District of Columbia (See Appendix, National Network Map).
During the 2011-12 school year, Wyman completed 20 certification site visits, five audits,
three field observations, and five on-site trainings or technical assistance visits aimed at helping
partners improve both their administrative and programmatic skills. In 2013 Wyman will conduct
8 trainings with up to 15 participants at each session, up to a 120 total participants. Our current
partners and our projected new 2013 partners will train as many as 2,500 TOP® Facilitators,
who have the capacity to serve 50,100 teens in 2013.
In terms of training quality and feedback Wyman is achieving program goals. 90% of
trainers answered all the post-test knowledge-based questions correctly. In addition the average
participant confidence score was 3.53/4.0; the average satisfaction score related to training
quality was 3.98/4.0.
WYMAN’S NATIONAL NETWORK
Additionally, Wyman’s TOP® is currently the subject of eight separate, ongoing research
projects studying both the process and outcomes of this program. As these projects reach their
conclusions, findings and recommendations will be incorporated as appropriate.
Identification and Mitigation of Risk Factors/Obstacles
TOP® certification serves to help identify and address, in a timely manner, many
challenges and ensures that the partners’ commitment to fidelity, and their ability to achieve it,
continues to grow. However, based on experience to date, the National Network’s major
challenges include the following:
Many partners have little or no experience in implementing evidence-based programs
such as TOP®. Often they have the right idea and focus, but lack the capacity, tools and
organizational structure to implement a proven practice (like TOP®) in a way that is sustainable.
Consequently, they may have difficulty with the need for a rigorous approach to fidelity. For
example, some partners have suggested that a seven- or eight-month program would “fit” better
with their school schedule or other priorities; even though the TOP® research has shown that a
nine-month “dosage” is needed to achieve the desired impacts. Other partners have struggled
with the need to provide a minimum of 20 hours of community service learning, citing a lack of
staff support, community coordination problems, safety issues, and other concerns. Still others
have struggled to maintain a minimum facilitator to participant ratio of 1:25.
The administrative requirements on National Network partners have proved challenging.
Some partners have found that “start-up” can be labor intensive. Partners are responsible for
ensuring that all program providers submit their data into the online system on a regular basis.
Many of these providers have limited experience collecting and reporting the type of data
required. Some even have little or no experience using online systems and require extra
instruction on the use of the technology. Other administrative issues concern the need to obtain
permissions and other forms from participants and their families, ensuring that required online
WYMAN’S NATIONAL NETWORK
surveys are completed by participants and facilitators in a timely manner, and understanding
both the process and the value of the data and the reports that are used for program quality
assurance and improvement. All of these components are important to maintaining fidelity and
achieving desired outcomes for teens.
To address these issues, Wyman is working with partners to provide them with more
technical assistance and more frequent site visits than anticipated in the original business
model. As Fixsen notes, ongoing consultation and coaching, staff and program evaluation, and
fidelity assessments are important “drivers” that help create and support partner behavior and
lead to sustainable and effective delivery of evidence-based practices.4 During site visits and
observations, program quality is generally excellent, even as partners continue to work to meet
administrative requirements. The National Network is adding additional training sessions and
encouraging greater use of the online Community of Practice website to assist partners and
strengthen their effectiveness.
Conclusion
As Wyman continues to grow the National Network, the primary focus remains on
identifying partner needs and helping them improve their effectiveness. Wyman remains in
“learning and adaptive” mode, using the lessons gathered from trainings, site visits and field
observations, program audits and other partner feedback to guide thinking and program
enhancements. Wyman’s National Network aims to create a collective voice that will drive
resources for sustainability and on-going capacity building, to effectively support teens at-risk
nationwide.
4
Fixsen et al, 2005 Running Head: WYMAN’S NATIONAL NETWORK
Appendix: Supporting Documents
11
Running Head: WYMAN’S NATIONAL NETWORK
12
WYMAN’S NATIONAL NETWORK
13
WYMAN’S NATIONAL NETWORK
14
WYMAN’S NATIONAL NETWORK
15
WYMAN’S NATIONAL NETWORK
16
WYMAN’S NATIONAL NETWORK
17