Document 6589616

Sep/Oct
2014
No. 12
Impact Evaluation Round Table
As part of its support to impact evaluation in the ILO, EVAL is creating
community of practice for those who are technically involved in
and implementing impact evaluations. Beginning in October, EVAL
organize several informal round table discussions of ongoing ILO
evaluations or impact evaluation methodologies currently being
developed. Anyone working on such activities is welcome to contact
join the group and/or to present their work for a round
table discussion.
i-eval
EVAL highlights
Making Impact Evaluation Work
Organizations and their donors are increasingly preoccupied with capturing knowledge about “what works” in
international development. The ever-expanding numbers of authoritative evaluations, research reports and studies
make this a very complicated task. ILO EVAL is expanding its efforts to filter through the available array of research and
resources to distil the most pertinent and credible insights and thereby enhance its contribution to identifying the impact
of ILOs work. Additionally, EVAL is dedicated to identifying knowledge gaps to better determine what further research,
studies, evaluations or impact assessments are required.
Planned collaboration between the ILO Evaluation Office and the World Bank is aimed to help us provide clarity on what
impact assessments are (or should be) related to the world of work, labour and jobs -- and how these differ from and
complement other processes of monitoring and evaluation. This collaboration, in conjunction with discussion with
technical experts inside and outside the ILO, will lay the basis for a renewed effort by EVAL to support and coordinate
impact evaluation in the ILO in a more rigorous manner.
EVAL has stepped up its efforts to support the proper use of impact evaluation in the organization, through EVAL
Guidance Note 13: Impact evaluation, peer review and exchange sessions and - more recently - a stock-taking review of
current impact evaluation practices in the ILO, see EVAL’s latest Think Piece. The Think Piece shows that impact
evaluation in the ILO has several strengths, but rigorous standards need to be applied, including in-depth needs analyses,
better initial preparation, and evaluability analysis before conducting an impact study. This is necessary to ensure a high
standard effort and also to justify the exorbitant cost which is associated with these complex and longer duration
evaluations. EVAL looks forward to our next roundtable on impact evaluation where we will report on progress made.
Guy Thijs, Director
ILO, Evaluation Office
Visit our website:
http://www.ilo.ch/eval/lang--en/index.htm
navigation
Regional Article: Europe – Departmental Article: Bureau of Employers' Activities
Innovation & Research – Learning Activities and Events
Blogs - Evaluation Newsletters
EVAL to
Flash news
EVAL is pleased to share the twelfth edition of i-eval Flash news with
you. Through this quarterly electronic bulletin we provide readers with
updates, news and information on publications and upcoming events
related to evaluation. You are invited to alert us about any news item
that you wish to include in the next issue at [email protected].
QUICK
a
designing
will
impact
i-eval Flash news
– Sep/Oct 2014, page 2
Innovation & Research
Completed Studies
Think Piece No. 6 – Impact Evaluation in the ILO:
Stock-taking of current practice, Oct. 2014. Written
by Dr. Achim Engelhardt, this Think Piece presents
information Dr. Engelhardt collected in a stock-taking
exercise aimed to lay the
groundwork for future EVAL
involvement in this area. The
study provides an overview of
past
and
existing
impact
evaluation work in the ILO, and
examines standards followed in
the UN development community.
Methodologies are also reviewed, and the piece
concludes with some recommendations for future
action.
Think Piece No. 7 - Delivering decent work results:
Implications of a meta-analysis of ILO Decent Work
Country Programmes, Oct. 2014
– This Think Piece was written by
Mr John Martin and aims to
synthesize the findings of his
2014 meta-analysis of internal
Decent
Work
Country
Programme reviews (CPRs), as
well as findings from his 2011
meta-analysis of independent high-level evaluations
of country programmes. Recommendations are put
forward for how better to approach internal country
programme reviews and make these useful to
broader DWCP planning and implementation.
to conflicts, disasters and other crisis situations and
requested the Office to make the necessary
organizational arrangements to strengthen its
capacity to deal with such situations. Soon after, the
ILO’s Evaluation Advisory Committee held its 16th
meeting and expressed its desire for EVAL to carry
out an evaluation of TC in fragile states in conflict and
post conflict areas. (forthcoming 2015).
Synthesis review of labour protection: As in previous
years, EVAL will manage a synthesis review of this
recurrent discussion topic to be conducted by an
external research team (forthcoming 2015).
New Books on Evaluation
Qualitative research and evaluation Methods 4th ed. –
Michael Quinn Patton, Sage, November 2014. “This
book contains hundreds of
examples and stories illuminating
all aspects of qualitative inquiry.
Patton has created the most
comprehensive,
systematic
review of qualitative methods
available. Key Features: Explores,
compares and contrasts 16
different theoretical and philosophical approaches to
qualitative inquiry; offers strategies for enhancing
quality and credibility of qualitative findings; unravels
the complexities of mixed methods and triangulation;
and explains the issues and approaches to fieldwork.”
On-going and planned studies
External quality assessment of independent project
evaluation reports 2012-13: This annual study will
use the scoring instrument as in previous years and
will review a sample of ILO-managed, independent
project evaluations from the reporting period. Due to
be published in November 2014, the study is
conducted by the International Organizational
Development, Ltd, UK and is part of the regular
externally-generated
quality
controls
EVAL
commissions on independent project evaluations.
Thematic evaluation on fragile states: In March
2014, during its 320th session, the Governing Body
discussed ILO technical cooperation in response
Agricultural Policy Monitoring and Evaluation 2014
Organization for Economic Development (OECD),
2014. “This edition covers OECD member countries
up-to-date estimates of support to agriculture. It is
complemented
by
country
profiles on agricultural policy. A
number of new country-level
frameworks
for agricultural
policies will become operational
in 2014, and multilateral trade
negotiations may have future
bearing on agricultural trade.”
i-eval Flash news
– Sep/Oct 2014, page 3
News from the Departments
Bureau for Employers’ Activities
(ACT/EMP)
Evaluation of Outcome 9: Employers have strong,
independent and representative organizations
ILO’s Bureau of Employers' Activities (ACT/EMP)
received funding from partnerships with both
Sweden (SP) and Norway (NP) to support the
implementation of the Outcome 9 strategy. The
funding has been used to support Outcome 9 targets,
or Decent Work Country Programme Outcomes
(CPOs), focussed on strengthening the capacity of
employers’ organizations in 19 countries across all
regions. The development of Outcome 9 Global
Products (GPs) was also supported through NP and
SP funding.
The evaluations, both the Swedish (final evaluation)
and Norwegian programmes (mid-term evaluation),
covered the period from April
2012 to March 2014. The purpose
was to examine the CPOs achieved
and GPs produced under Outcome
9 and to assess the contribution of
the
programmes
towards
achieving Outcome 9’s strategy.
The focus of the Norwegian funded programme was
to strengthen the institutional capacity of
employers’ organisations in policy development and
dialogue at national, regional and interregional levels,
and to assist them to develop new and improved
services to strengthen membership recruitment and
retention. The Swedish funded programme sought to
enhance the capacity of employers’ organizations to
participate in the policy dialogue on sustainable
enterprise development for job-rich growth and
provide opportunities to do more in-depth policy
work that could contribute to more effective national
policy dialogues.
NP and SP funding also contributed to the
development of Outcome 9 Global Products (GPs)
which supported activities at the country level. A key
product developed under NP was a global survey on
women in business and management and creation of
a resource network of employers to share good
practices and advocacy on advancing women in
business and gender equality. The SP contributed to
further improvements to the Enabling Environment
for Sustainable Enterprise (EESE) toolkit, which helps
employers’ organizations identify constraints in the
development of sustainable enterprises and
formulate advocacy and reform proposals for
dialogue with government.
Main Findings
The outcome evaluation findings confirm the
relevance of the NP and SP funded interventions as
well as those of the resulting CPOs and GPs. Strong
mutual linkages were observed between Outcome 9
and the DWCPs. A fruitful collaboration environment
had been established with other ILO Outcomes in the
development of the Outcome 9 Global Products. The
interventions reviewed were found to be logically
coherent and had realistic objectives in terms of CPO
(target) achievement.
 NP and SP funding contributed substantially to the
overall achievement of CPO targets. In addition,
the NP and SP funded global products reached
most of their set objectives. Both SP and NP
funded interventions contributed to the potential
for development change.
 Outcome 9 indicators and related measurement
statements were identified as a useful tool for
aggregating results achieved at a departmental
level. However, the indicators are not suited for
monitoring gradual increases in the capacity of a
partner EO over a longer period of time.
 The gender dimension was an integral part of NP
funded activities. Under SP funding, while the
gender dimension EESE tool had been
strengthened as part of the support to GPs,
challenges in mainstreaming gender equality in
EESE processes at the country level still exist. In
several cases the gender dimension was absent in
business agendas.
The NP and SP funded interventions indicated that
continued support from the ILO to sustain results at
the country level will be needed. In regard to the
GP’s, the tools for assessing various dimensions of EO
capacity and advocacy efforts are excellent. There is
potential for these GP elements to benefit a much
wider spectrum of agencies within the development
community in the future.
Lessons learned
The specific country context in which an EO partner
operates must be taken into account in the design of
support in order to be able to respond flexibly in a
dynamic and changing environment. It is important
to work with EOs which are representative of the
i-eval Flash news
– Sep/Oct 2014, page 4
private sector and can apply a broad perspective on
the needs of business environment reform.
More
focus
is
needed to support
the sustainability of
capacity
building
activities with EOs.
EO membership is a
critical factor in
strengthening
the
financial capacity and representativeness of EOs.
Additionally, it was emphasized that initiatives to
promote gender equality need to be strengthened as
they often remain as fairly “isolated islands” within
EOs.
Recommendations: The following recommendations
from the outcome evaluation summarize future
action to be taken:
On donor support: Continue to support the creation
of strong, independent and representative EOs. Give
priority to building up the membership base and
strengthening EO capacity for analysis and advocacy
related to business environment reform and gender
equality.
On capacity development: Experiment with the
methods aimed at capacity development. Learn from
others through partnerships between EOs in different
countries. Consider partnerships with various local
institutions, including local economic think-tanks.
Project evaluations from ACT/EMP
2010-2015
(summaries are available through the hyperlink, full
reports from [email protected]
GLO/11/59/SID
GLO/12/56/NOR
INT/06/62/NET
INT/06/63/NET
INT/06/55/NOR
GLO/14/59/NOR
RAB/12/50/NOR
Sweden-ILO Partnership Programme
Phase II 2012-18 Outcome 18 –
Maritime Labour Convention (Global
Product) - Final Evaluation (2014)
ILO Norway/Sweden Partnership
Programmes: Outcome 9: Employers
have strong, independent and
representative organization
Strengthening the capacity of
employers' and workers'
organizations to be effective
partners in social dialogue - Final
Evaluation (2010)
Mainstreaming tripartism across the
Netherlands/ILO Cooperation
Programme (NICP) and product
development for employers' and
workers' organizations - Final
Evaluation (2010)
ILO-Norway Framework Agreement:
Child Labour Component - Final
Evaluation (2009)
Outcome 9: Employers'
organizations thematic funding for
2014-15 Norway-ILO Partnership Final evaluation (Forthcoming 2015)
Developing the capacity of
employers' organizations in the Arab
Region through effective policy and
social dialogue - Final Evaluation
(Forthcoming 2015)
Monitoring and evaluation: Improve the monitoring
and evaluation of EO capacity development over a
longer timeframe, including the use of data on EO
membership development.
Mainstreaming of gender equality: Identify
opportunities to create successful gender initiatives
that can be replicated in the various organisational
structures of EOs and through EO advocacy processes
such as the national business agendas.
Evaluation Office
Our named changed
with Office Directive
IGDS No. 74, V.2,
6/2014
i-eval Flash news – Sep/Oct 2014, page 5
Regional News Lessons Learned from Evaluation
in Europe & Central Asia
TC project evaluations generate a wealth of findings
and knowledge, including useful perspectives that
can improve organizational processes and practices.
In the Europe and Central Asia Region some findings
and practical lessons have emerged covering
employment, social protection, labour migration,
social dialogue and other topics which might be
relevant to broader contexts and useful to various
evaluation stakeholders and teams across the ILO.
The following selected highlights of such findings
come from 2012-2013 project evaluations.
Quality of project Logical Frameworks:
Certain shortcomings in project design have been
noted on several occasions as posing a challenge to
the evaluation exercise. Logical frameworks, or
logframes, are typically mentioned as an area for
improvement in EVAL meta-analyses of project
evaluations as well as some internal studies EVAL has
undertaken. Some of these specific deficits are
related to the definition and expression of the
objectives and the appropriateness and usefulness of
the indicators.
The most common
drawbacks are linked
to the formulation of
indicators and result
statements at different levels of the
project results chain.
There is sometimes
confusion between
the level of results
expected at the outcome, output and activity levels.
Indicators are in some cases formulated as outcomes,
or even as activities, thus not being appropriate to
the nature of the change they aim to capture. In a
few cases, the evaluators reported that they had to
develop their own “additional” indicators during the
desk research in order to be able to start the active
research phase and interviews.
Asking the so-called “If – Then” question is
considered to be a good way to test the logic behind
the chain of results underlying the project approach
at the design stage. Since elaboration of new projects
is a time-consuming and multiple-stakeholder
process, new project proposals are often being
circulated for inputs in more than two rounds over an
extended period of time. Technical perspectives may
be prevailing over more formal programming
perspectives in terms of formulation of results. It is
also often not possible to effectively trace and
harmonize all the inputs coming from various
stakeholders, which is further complicated by time
deficits weighing on officials in charge of project
formulation. Having a single final reviewing entity
for quality control might be helpful in this regard. It
was felt that PARDEV’s role could be strengthened to
better streamline internal capacities, and to offer
real-time advice and support to the project design
teams.
Performance against the main evaluation criteria
Over the two-year period under review, relevance
and effectiveness of projects was assessed as
satisfactory or good. All the projects evaluated in
2012-2013 were considered as being highly relevant
to the national development goals, corresponding
accurately to national development frameworks, and
the ILO’s strategic framework and agreed-upon
country priorities. Efficiency was also assessed
satisfactorily with a particular acknowledgment for
the high quality of expertise and technical resources
that the ILO makes available to the projects and through the projects - to the constituents, including
quality research, analytical products and training.
Sustainability, however, was typically assessed less
positively. This might be successfully addressed if
project design teams introduced some standard
sustainability components that would take effect
during the last stages of implementation. Some
elements of follow-up on outcomes (e.g., tracing the
use of the most significant results of past
interventions) could be built into ILO cooperation
frameworks in the countries (DWCPs or Programmes
of Cooperation) which often have longer timeframes
and are based on long-term commitments, thus
offering a good basis for continuity.
Tripartite constituent involvement and engagement
Constituents’ engagement in the evaluation process
is being ensured at all stages and this reflects the
tripartite nature of the organization, its values and
methods of work. Such engagement is not only useful
for obtaining feedback on project performance, but it
also serves other important objectives. Having
constituents participate in evaluation emphasizes
equality of status among all the different parties and
stakeholders. It also ensures that constituents take
i-eval Flash news – Sep/Oct 2014, page 6
greater responsibility both for the process and the
results: the stronger the involvement, the stronger
the uptake of results and ownership of project
outcomes.
Also very important in Europe and Central Asia is the
language challenge. In order to facilitate
communication during the evaluation process
between the ILO, the evaluators and constituents it is
necessary to deal effectively with language barriers.
Translation of essential evaluation documents from
ILO is typically needed in everyday communication.
This aspect should be accounted for in evaluation
planning as it may have an effect on the timelines
needed to complete an evaluation. Average time
needed for consultations on the TOR, for example,
could be longer than in other regions, so making
relevant and adequate provisions for this in planning
and implementation would improve evaluation
processes at the national level.
covered by such projects can vary from two to ten in
some cases. There are specific aspects to consider in
evaluations for these larger and more complex
projects. They require a special set of higher level
evaluation skills that reflect the versatility and
technical knowledge required for multi-country and
themed projects. Additionally, evaluations of this
type may also require more time, which would have
specific implications for evaluation budgets that need
to be estimated at the project design phase.
When these complex regional projects are
implemented it is well worth emphasizing that each
country will probably have slightly different outputs,
means of action and varying activities, all of which
make designing a coherent and comprehensive
evaluation plan very difficult. At the project design
stages these concerns should be considered.
In conclusion, it is very important to underline that
the evaluation function is closely linked to learning
and knowledge management, so revisiting the
findings periodically can improve utilization of
monitoring and evaluation processes and findings.
Project evaluations from Europe in 2012-2013
(summaries are available through the hyperlink, full
reports from [email protected]
ALB/11/01/EEC
RER/07/08/AUT
RER/08/05/EEC
Finally, because the ILO does not have a country
office in many countries of the region, where the bulk
of technical cooperation is taking place, the role of
the ILO National Coordinators (NCs) in ensuring
smooth communication at the country level should
be recognized and emphasized. It should be noted in
this context that a move towards stronger ILO
presence in the countries would correspond to a
number of evaluation recommendations (including
the High-level Independent evaluation of DWCP for
Kyrgyzstan 2010).
Evaluation of multi-country projects
There is an increasing trend to design multi-country
projects, which can be additionally grouped by
themes, such as migration, youth employment, and
social dialogue, etc. The number of countries
RER/11/04/EEC
RER/11/51/FRG
RER/09/04/EEC
RER/09/05/FIN
SRB/10/01/UND
RBSA Funded
Human resources in Albania project Midterm Evaluation
Consolidating
the
legal
and
institutional foundations of social
dialogue in the Western Balkans–
Final Evaluation
Increasing protection of migrant
workers in the Russian Federation and
South Caucasus - Final Evaluation
Promoting integration of migrant
domestic workers in Europe - Final
Evaluation
Combating child labour in Central
Asia: Commitment becomes action
(PROACT CAR Phase III) - Final
Evaluation
Effective governance of labour
migration and its skills dimensions Final Evaluation
From the crisis towards decent and
safe jobs in Central Asia – Midterm
and Final Evaluation
Strengthening capacity for inclusive
local development in Serbia - Final
Joint Evaluation
Review of RBSA projects - Yugoslav
Republic of Macedonia 2010-2011 Final Evaluation
i-eval Flash news
– Sep/Oct 2014, page 7
ILO Evaluation Guidance
i-eval Resource Kit - ILO policy guidelines, 2nd ed.


Revision plans
Guidance Note 13: Impact evaluation Work is now being planned for a revision of the impact
guidance note. A new EVAL Think Piece on impact evaluation, see page three, undertook a stocktaking exercise on impact evaluation and as discussions continue, the guidance on this will be
updated and revised, and reported in the newsletter.
EVAL is also currently planning to revise guidance on how to conduct Decent Work Country Programme internal
reviews, following discussions within the Evaluation Network,and findings and recommendations from the new
Think Piece on DWCP internal reviews discussed on page three.
EVAL welcomes your input for improvements on these two topics. Please send to [email protected].
Evaluation Learning Activities & Events
ILO Evaluation Learning Activities in Turin
Independent evaluation of the International Training Center of the
ILO Academies
The Governing Board of the ITC-ILO requested an evaluation of the ITC’s Academies, ten of which took place between 2011
and 2013 and represented two recurring series of five academies. EVAL managed the evaluation in collaboration with the ITC.
The evaluation was welcomed by ITC management and reflected well on this aspect of ITC’s activities. The evaluation
demonstrated that the Academies have strong relevance to the ILO’s Strategic Programme Framework.
Impact evaluation of technical cooperation and development projects - Nov 3-7, 2014
Target group: Technical specialists in charge of monitoring and evaluation of development projects and programmes.
Description: The participants will be guided to perform an impact evaluation for a selected project (case study) based on
the learning contents pursued in the course.
Training course to certify evaluation managers - The fourth training session for EVAL’s Evaluation Manager Certification
took place in Bangkok in October 2014. Both the Asia and the Africa regional training sessions which took place, in addition to
the Turin trainings, were considered successful events and EVAL will continue to offer these courses to build the skills of
evaluation managers. Once an official completes the course, then a pilot evaluation experience (practicum) is undertaken.
After this has been successfully completed, the official will then be considered for certification as a recognized ILO Evaluation
Manager. Click here to read the last quarterly report on the EMCP training. Registration information can be found here.
EVAL congratulates the newly certified Evaluation Managers
Rose Anang
Maria Borsos
Matthieu Cognac
Darryl Crossman
Gugsa Farice
Sergio Iriarte
Oktav Pasaribu
Anne Schalper
Eszter Szabo
Andrés Yuren
New UNEG publication: Integrating Human Rights and Gender
Equality in Evaluations – August 2014 - This report was produced
by UNEG as an in-depth guidance handbook to serve as a field guide to
improve human rights and gender equality responsive evaluation
throughout the UN system. It is aimed at increasing knowledge on the
application of these two approaches in evaluation processes but also at
raising awareness on their specific relevance and significance for UN work. It complements the UNEG's Handbook
"Integrating Human Rights and Gender Equality in Evaluation: Towards UNEG Guidance," an abridged version that outlines
practical steps on how to prepare, conduct and use HR & GE responsive evaluations. The present document deepens each of
these aspects, and provides additional theoretical and applied information, tools and suggestions.
i-eval Flash news
– Sep/Oct 2014, page 8
External Knowledge Sharing, Courses and Webinars

Advocacy planning, Monitoring and Evaluation course – International NGO Training and Research Centre
(INTRAC), Oxford, UK – December 1 thru December 3 2014. Building on INTRAC’s extensive experience in both
monitoring and evaluation and in advocacy, this course will support practitioners with how to ensure M&E into
advocacy initiatives and into organizational learning and accountability. The course content draws on real-life
case studies from INTRAC’s work in supporting organizations to develop advocacy strategies and from carrying
out evaluations of advocacy initiatives. It will provide a series of practical tools which can help participants to
develop and improve the advocacy M&E systems in accordance with their own capacity and needs.

Claremont Graduate University Partners with UNICEF and the Rockefeller Foundation in partnership with
IOCE and DevInfo, offer a series of live webinars on "Emerging Practices in Development Evaluation". This
includes a range of guest speakers with experience in development evaluation. Please see the full program
here. These webcasts are free and open to the general public.

e-Learning programme on Development Evaluation – UNICEF
and IOCE, under the EvalPartners initiative, offer
an introductory. It is composed of an introductory class, independent units and a create-your-own-course option,
as well as fixed courses such as: Introductory class on Development Evaluation; Equity-Focused and GenderResponsive Evaluations; National Evaluation Capacity Development for Country-led M&E Systems; and Emerging
Practices in Development Evaluations, for example. The next cycle begins 27 January 2015 - 18 May 2015 and
interested participants can register here.

2nd International Conference on Evaluating Climate Change and Development
– Nov 4-6, Washington D.C.
The Climate-Eval Community of Practice, hosted by the Global Environment Facility Independent Evaluation Office
(GEF IEO) and its partners are pleased to announce the 2nd International Conference on Evaluating Climate
Change and Development, scheduled for November 4-6, 2014, in Washington D.C. The conference will bring
together evaluation professionals, academics, policymakers and other stakeholders from developed and developing
countries with the purpose of fostering the interdisciplinary exchange of ideas and methods to evaluate climate
change and sustainable development.
Blogs on evaluation World Bank Blog on Impact Evaluation
American Evaluation Association Blog
Evidence matters blog – Impact 3iE (new)
Foundation Strategy Group (FSG) Blog
Better Evaluation Blog
John Gargani’s EVAL Blog
Genuine Evaluation
Evaluation Capacity Development Group
Intelligent measurement
Design, Monitoring and Evaluation (new)
IDB Development effectiveness blog (new)
African Development Bank eval- blog (new)
Other evaluation newsletters
OECD DAC Evaluation News
OIOS Inspection & Evaluation
Evaluation Office (EVAL)
International Labour Office
CH-1211 Geneva 22, Switzerland
Email: [email protected]
Editor-in-Chief: Guy Thijs, Director
Executive Editor: Janet Neubecker
UN Women Newsletter
IFAD Evaluation News
EVAL Partners Newsletter