The Mid-Term PPA Independent Progress Review For the

The Mid-Term PPA
Independent Progress Review
For the Consortium of Restless
Development (lead agency), War Child
and Youth Business International
Willem van Eekelen and Jill Edbrooke
12 October 2012
Page 1 of 131
Table of Contents
Acronyms and abbreviations ................................................................................................................... 3
Acknowledgements ................................................................................................................................. 3
Executive summary ................................................................................................................................. 4
1 Introduction .......................................................................................................................................... 7
1.1 Purpose of the evaluation ............................................................................................................. 7
1.2 Scope of the evaluation................................................................................................................. 7
1.3 Focus of the evaluation ................................................................................................................. 7
1.4 Organisational context .................................................................................................................. 7
1.5 Logic and assumptions supporting DFID PPA funded project and/or programme activities ........ 8
1.6 Overview of PPA funded activities ................................................................................................ 9
1.7 Relationship of DFID PPA funded activities to other programme activities ................................ 10
2 Evaluation methodology ..................................................................................................................... 11
2.1 Evaluation plan ............................................................................................................................ 11
2.2 Research problems encountered ................................................................................................ 14
2.3 Strengths and weaknesses of selected evaluation design and research methods in retrospect15
3 Findings .............................................................................................................................................. 16
3.1 Results ........................................................................................................................................ 16
3.2 Relevance ................................................................................................................................... 19
3.3 Effectiveness ............................................................................................................................... 22
3.4 Efficiency and value for money assessment ............................................................................... 31
3.5 Impact and value for money of PPA funding .............................................................................. 37
4 Conclusions ........................................................................................................................................ 39
4.1 Summary of achievements against evaluation criteria ............................................................... 39
4.2 Summary of achievements against rationale for PPA funding.................................................... 46
4.3 Summary of problems and issues encountered ......................................................................... 47
4.4 Overall impact and value for money of PPA funded activities .................................................... 48
5 Utility................................................................................................................................................... 49
6 Lessons learned ................................................................................................................................. 49
6.1 Policy level .................................................................................................................................. 49
6.2 Sector level ................................................................................................................................. 50
6.3 PPA fund level ............................................................................................................................. 50
6.4 Organisational level..................................................................................................................... 51
7 Recommendations ............................................................................................................................. 51
/
Annex A: PPA IPR terms of reference .................................................................................................. 53
Annex B: Evaluation research schedule and timescales ...................................................................... 59
Annex C: Data collection tools .............................................................................................................. 60
Annex D: List of people consulted ........................................................................................................ 64
Annex E: List of data sources ............................................................................................................... 74
Annex F: Bibliography ........................................................................................................................... 74
Annex G1: Country programme report 1: Restless Development, Uganda .......................................... 83
Annex G2: Country programme report 2: Restless Development, Zambia .......................................... 97
Annex G3: Country programme report 3: War Child, Uganda ............................................................ 110
Annex G4: Country programme report 4: KYBT, member of YBI, Kenya ........................................... 120
Annex H: Details of the evaluation team ............................................................................................. 128
Annex I: PPA organisation’s management response to report’s findings (postsubmission)………….............................................................................................................. ...........129
Page 2 of 131
Acronyms and abbreviations
CD
CEO
CHH
Consortium
CSO
DRC
HIV
ICS
JFFLS
KYBT
MEL
NGO
OMS
PCI
PPA
REFLECT
SRH
War Child
YBI
Country Director
Chief Executive Officer
Child-headed household
The consortium of Restless Development, War Child UK, and Youth Business
International
Civil Society Organisation
Democratic Republic of Congo
Human Immunodeficiency Virus
International Citizen Service
Junior Farmer Field and Life Schools, a FAO-funded programme with
agricultural and life skills components
Kenya Youth Business Trust
Monitoring, Evaluation and Learning
Non-governmental organisation
Operations Management System
Pastoralist Communication Initiative
Programme Partnership Arrangement
Regenerated Freirean Literacy through Empowering Community Techniques,
a participatory approach to adult learning, community mobilisation and social
change
Sexual and reproductive health
War Child UK, which is legally distinct from War Child Canada and War Child
Holland.
Youth Business International
Acknowledgements
I would like to thank:
•
•
•
The children, youths and adults who helped me understand the context in which the
consortium partners operate, and the ways in which the consortium’s work impacts on
their lives.
The consortium’s counterparts who work for governments, donor agencies, peer NGOs
and local civil society organisations. They clarified the nature and impact of the
consortium’s relations with their government departments, civil society organisations,
donor agencies and companies.
The consortium’s staff in the Uganda, Zambia, Kenya and the UK. They remained kind
and welcoming.
Page 3 of 131
Executive summary
This is an Independent Progress Review (IPR) of a PPA consortium that consists of three
NGOs: Restless Development (lead agency), War Child and Youth Business International
(YBI). The IPR’s purpose is threefold:
1. To verify, and supplement where necessary, the consortium’s PPA-related reporting (i.e.
the Annual Review, changing lives case studies and additionality report);
2. To independently evaluate the impact that DFID funding has had on the consortium, its
partners and its ultimate beneficiaries, and to assess the value for money of the funding;
and
3. To assess the extent to which the consortium has acted upon the comments provided by
DFID as part of the Annual Review Process.
This review’s reference documents are the PPA logframe, the PPA business case and the
PPA Annual Reporting. In addition, Coffey’s Evaluation Strategy provides guidance with
respect to the IPR’s approach and methods. The methods include documentation reviews,
unstructured and semi-structured interviews, focus group discussions, number and narrative
trailing, swim lanes, and a cost-benefit analysis.
The three members of this consortium came together for the purpose of this PPA. They have
no history of cooperation, use different approaches to achieve their respective goals and
have no geographical overlap, but the organisations share:
•
•
•
A focus on young people;
A desire to enable young people to tackle the issues that most affect their lives (with
emphasis on youth protection, sexual and reproductive health, livelihoods and
entrepreneurship); and
An awareness that their work generates evidence-based insights that could and
sometimes already does benefit the wider international development sector.
The PPA funding – nearly £2.8 million per year - is allocated to consortium management
(10%), joint initiatives (10%), and each of the three consortium partners (80%, divided in
direct proportion of the organisations’ respective turnovers and utilised to cover a wide range
of capacity-related, programme and core costs.)
First part of the IPR: to verify, and supplement where necessary, the Consortium’s
PPA-related reporting (i.e. the Annual Review, changing lives case studies and
additionality report). This consortium’s PPA-related reporting has been very positive (all
milestones met or exceeded), comprehensive-yet-concise and truthful, with only the
following qualifications:
•
•
This PPA’s ‘Best Case Scenario’ – the very new Zambia-based Youth Accountability
Model – lacks crucial contextual information (e.g. the upcoming elections, when
measuring progress in the frequency of contacts between youths and political decisionmakers) and misinterprets several statistics (e.g. presenting clinical statistics as societal
statistics). This case study is based on an external evaluation that does not deal
adequately with a potential social desirability bias, which means that the results may
have been exaggerated. The Youth Accountability Project has great potential, but is not
(yet) the consortium’s ‘best case.’
A few of the beneficiary numbers are somewhat inflated, due to a certain amount of
double-counting.
Page 4 of 131
Second part of the IPR: to independently evaluate the impact that DFID funding has
had on the Consortium, its partners and its ultimate beneficiaries, and to assess the
value for money of the funding. This report covers both the performance of the individual
organisations and the joint initiatives. To do justice to the diverse ways in which the PPA
funding has been utilised, the IPR assesses each of the consortium members’ overall
organisational capacity.
For Restless Development, the IPR conducted assessments in the UK, Uganda and
Zambia, and found Restless Development to have an unusually strong tradition of employee
engagement; a very strong brand identity; strong organisational systems, policies and
processes (some of which are new and now need to be consolidated); very considerable
impact at programme level; and a policy and advocacy voice that is rapidly gaining in
strength and already louder than one would expect for an organisation of this size. The
assessment’s main recommendations relate to:
•
A reduction of the organisation’s institutional funding dependency. In the long run, this
dependence on institutional donor funding could have an impact on its organisational
independence.
•
The design of its CSO capacity development programmes. Restless Development could
strengthen these programmes by focusing its support on the organisations rather than on
the people therein, and by basing its support on better-framed capacity assessments.
For War Child, the IPR conducted assessments in the UK and Uganda, and found an
organisation that taps into a range of different funding sources and utilises this funding to
successfully reach and help protect particularly poor and disadvantaged rural individuals who are
not reached by any other organisation. The assessment’s main recommendations relate to:
•
The level of formalisation. War Child has reached the stage where the consistency of its
performance would benefit from stronger systems, structure and division of
responsibilities, policies, standards, processes, reporting, monitoring and reflection.
Utilising PPA funds, War Child is already working to make the organisation more
systematically accountable and verifiably effective and efficient.
•
War Child’s unique niche. In schools, War Child is one of many support organisations.
Out of school, War Child is often the only organisation with a meaningful presence. In
terms of programme impact and as a source of evidence to inform policies and ‘next
practice’ programming, War Child’s out-of-school work outperforms its in-school work.
This seems worth keeping in mind when designing future programmes.
For YBI, the IPR conducted an assessment of YBI’s Network Team in London (‘YBI’), and of
its partner in Kenya (‘KYBT’). As a membership organisation, YBI has a strong but limited
set of products and services. Learning is highly systematised and geared towards
transforming the organisation into a knowledge-based service provider that identifies,
develops and promotes good practice in the field of youth entrepreneurship, and that
enables its members to build their respective capacities to replicate such good practice. The
assessment’s main recommendations relate to:
•
Its target groups. YBI could do more to achieve its strategy-dictated percentage of
members from developing countries, and to achieve a gender balance amongst
beneficiaries covered by these members.
•
Its governance. YBI rightly emphasises the importance of youth involvement in decisionmaking, and would benefit from strengthening such involvement internally.
For the consortium, the IPR assessed the consortium’s Annual Review Process (above),
the consortium’s follow-up on DFID’s comments (below), and the consortium benefits and
dynamics. The assessment found a vibrant and engaging consortium with considerable
cross-learning. The consortium would benefit from:
•
Logframe outcomes and impact indicators that explicitly reflect programme reach (i.e. a
number of communities in which the consortium operates) rather than national averages.
Page 5 of 131
•
Investments in even more robust reporting systems, to minimise both double-counting
and underreporting.
Third part of the IPR: to assess the extent to which the consortium has acted upon the
comments provided by DFID as part of the Annual Review Process. In its feedback,
DFID made two types of requests:
•
•
Requests for clarification (e.g. ‘what will the new target for life skills training be?’). In its
response to DFID, the consortium has met these requests adequately.
Request to change things (e.g. ‘output 2 targets – suggest these are revised upwards’).
The consortium has agreed to make these changes.
DFID’s feedback and the consortium’s response were written prior to the IPR mid-term
review. Most of it remains valid in light of this review’s findings, but the IPR suggests two
changes to the consortium’s response:
•
•
DFID asked the consortium to increase the target beneficiary numbers, and the
consortium has agreed. In view of the IPR findings, it seems better for the consortium to
assess the extent of its double counting, to eliminate it, and then to produce new
estimates.
DFID asked the consortium to increase the reach of its CSOs capacity building efforts,
and the consortium has agreed. In view of the IPR findings, it seems better to focus on
impact rather than reach, by focusing the consortium’s support on organisations rather
than on the people therein, and by basing its support on a better-framed capacity
assessment.
The IPR report identifies a range of strengths, as well as several weaknesses that could all
potentially be addressed in the second half of the PPA period, and arrives at the bottom line
conclusion that, overall, this consortium provides very good value for money already.
Page 6 of 131
1 Introduction
1.1 Purpose of the evaluation
As per the Coffey Evaluation Strategy (Appendix 8.1, page 1), the purpose of the IPR is
threefold:
1. To assess the extent to which the consortium has acted upon the comments provided by
DFID as part of the Annual Review Process;
2. To verify, and supplement where necessary, the consortium’s PPA-related reporting (i.e.
the Annual Review, changing lives case studies and additionality report); and
3. To independently evaluate the impact that DFID funding has had on the consortium, its
partners and its ultimate beneficiaries, and to assess the value for money of the funding.
1.2 Scope of the evaluation
The consortium partners have utilised the PPA funding to cover a wide range of capacityrelated, programme and core costs. To do justice to the diverse ways in which the PPA
funding has been utilised, I have assessed each of the organisations’ overall capacity – and
changes therein - to consistently reach relevant results efficiently and effectively.
1.3 Focus of the evaluation
This evaluation’s points of reference are the PPA logframe and business case, and DFID’s
overall PPA interests. This means that I was most interested in consortium-based work, and
in work that is relatively new and possibly inspired by the PPA.
1.4 Organisational context
The consortium consists of three organisations:
1. Restless Development, the consortium’s lead agency, is a youth-led development
agency that aims to:
•
Enable young people to contribute to development processes that result in
government policy and practice that is both beneficial for and accountable to young
people;
•
Enable young people to take up productive livelihoods and employment opportunities
that contribute to their household income and the economies of their communities
and countries; and
•
Ensure that young people’s sexual and reproductive practices are safe and lead to
healthy lives.
Restless Development implements programmes in India, Nepal, Sierra Leone, South
Africa, Tanzania, Uganda, Zambia, Zimbabwe and the UK. In addition to its direct
programming, Restless works towards developing a generation of young people that
take leadership roles; building a strong youth sector; and utilising the evidence its work
creates for the purposes of sharing, learning and shaping policy and practice.
2. War Child UK (from now on ‘War Child’) implements programmes that have several foci
– formal and informal education, livelihoods, capacity building – which are all related to
War Child’s ultimate aim of ensuring the protection of particularly disadvantaged children
and youths who are not reached by other organisations. War Child works in Iraq,
Afghanistan, the Democratic Republic of Congo, Uganda and the Central African
Republic.
3. Youth Business International (YBI) is a network of independent organisations in
developed and developing countries, 1 which all aim to help young people start and grow
1
Bangladesh, India, Jamaica, Kenya, Nigeria, South Africa and Sri Lanka.
Page 7 of 131
their own businesses. The services YBI’s members provide to these underserved
entrepreneurial youths are typically a combination of financing, mentoring and training.
YBI supports its members by providing services, tools and sometimes financial support
that are all meant to increase the members’ capacity to fulfil their respective mandates
and to build shareable evidence in the process. In this report, ‘YBI’ stands for the YBI
Network Team, which is the team that is based in London and the formal PPA partner.
The organisations came together for the purpose of this PPA. They did not have a previous
history of cooperation; use different approaches to achieve their respective goals; and there
is no geographical overlap; 2 but the organisations share:
•
•
•
A focus on young people;
A desire to enable young people to tackle the issues that most affect their lives; and
An awareness that their work generates evidence that could and sometimes already
does benefit the wider international development sector.
The PPA funding is allocated to joint initiatives (10%), consortium management (10%) and
each of the three consortium partners (80%, divided in direct proportion of the organisations’
respective turnovers). This report covers both the performance of the individual
organisations and the joint initiatives.
1.5 Logic and assumptions supporting DFID PPA funded project and/or
programme activities
Because of the wide-ranging utilisation of PPA funding, this section covers the organisations’
overall (albeit largely implicit) theory of change.
Restless Development and War Child
Respectively, at least 50 and 85 per cent of the work of Restless Development and War
Child takes the shape of the direct implementation of programmes that largely evolve around
behavioural change. Both organisations design their programmes on the implicit basis that
these programmes serve three types of purposes:
•
•
•
Individual gains. Behavioural change requires a lengthy process that individuals and
groups will only go through if they
o have and accept the knowledge needed to make sensible choices;
o have the actual ability to make such choices;
o see and recognise good examples; and
o benefit from peer support and a period in which new behaviour is
consolidated and leads to rewards.
Restless Development uses multi-year behavioural change trajectories in which small
groups of full time volunteers live in and with communities. 3 War Child uses local
facilitators who use a range of approaches to engage and empower community groups.
Community gains. The resultant behavioural change will contribute to a positive spiral
in which actions of individuals and small groups may consolidate peace and provide
protection in fragile environments; or in which healthy individual choices lead to better
health throughout a community; or in which more consistently vocal youths leads to more
responsive duty bearers.
Sector gains. This final step is the theory of change that is presented in the PPA
business case. The idea is that the consortium partners utilise PPA funding to
2
This may change as the consortium is working together on defining a joint intervention approach to livelihoods
in both Uganda and Tanzania.
3
Restless Development recruits local, national and international volunteers. Most volunteers are national.
Restless Development refers to its volunteers as ‘volunteer development professionals.’
Page 8 of 131
o
o
o
test and refine their most promising evidence-based development models;
connect this evidence to policy-makers to influence best practices and attract
funds from other donors; and
support governments and other agencies to adopt innovative approaches
where they prove relevant and effective.
Restless Development has invested years in implementing, reflecting on and refining its
models, and some of the evidence this has generated is now sufficiently robust to use for
advocacy purposes. War Child is refining existing models rather than developing new ones,
and their modifications may well prove worthy of replication – but it will take a few more
years before War Child’s evidence of success is sufficiently convincing for other
stakeholders to replicate or learn from.
YBI
YBI’s theory of change is that local, high-impact organisations most effectively increase
young people’s access to the opportunities of entrepreneurship, and that impact will be
maximised through coordinated global activity to build capability, to scale and to learn and
exchange. In the spirit of this theory of change, YBI’s core proposition is to build both a
community of practice in youth entrepreneurship and a community of knowledge:
•
•
YBI’s network infrastructure aims to improve the impact of support available on the
ground to in-need young entrepreneurs. In part this means building the capability of incountry initiatives to provide quality support efficiently and effectively. For this purpose,
YBI is developing and deploying dedicated tools and services in core areas, such as
mentoring, entrepreneur training, and technology for members and potentially for the
sector at large.
Good practice requires good evidence, and YBI is increasingly oriented to focus on
Monitoring, Evaluation and Learning to maximise network potential to understand what
works (and what does not work) in youth entrepreneurship support, to lead learning
across the field, and to influence related national and international policies.
1.6 Overview of PPA funded activities
2010/11 2011/12 2012/13 2013/14
Joint annual income of consortium
partners (£1000)
PPA funding (£1000)
PPA funding as percentage of total income
Other DFID funding (£1000)
10,963
14,057
15,330
16,867
0
2,755
2,755
2,755
0
19%
18%
16%
1,841
3,158
1,786
1,786
In preparation for and in the early days of this PPA, the consortium has invested in thinking
and discussions around maximising the PPA’s value to the consortium organisations. This
led to four PPA-related goals:
1. To over-deliver against the targets outlined in the logframe and agreement;
2. To strengthen the organisations’ income position and to ensure that their financial health
is independent from future PPA funding;
3. To increase the profile of the work and be recognised as the go-to experts in their
respective fields;
4. To increase the organisational capacity of each of the consortium partners.
Page 9 of 131
As PPA funding has been treated as unrestricted income and is not separately recognisable,
it is not possible to calculate proportions of PPA funding over different types of purposes.
In the first 18-months of the PPA, the consortium has already achieved considerable
success in relation to each of these goals. These achievements were possible, in part, by
PPA spending on:
•
•
•
Strengthening the partners’ systems, through the recruitment of War Child’s first-ever
M&E manager and a range of investments to improve YBI’s service delivery to the
network.
Strengthening and diversifying the partners’ funding base, by recruiting an additional
member to the network expansion team of YBI and a first-ever Trusts and Foundations
fundraiser at Restless Development and War Child.
Conducting five pieces of research that are of interest to the consortium and beyond.
Such research is meant to strengthen the performance of the consortium partners and to
generate evidence that may be utilised for the partners’ policy messages. For example,
two youth entrepreneurship research pieces have been conducted in the context of the
consortium’s ‘Joint Working Projects’: an ‘Entrepreneurship Literature Review’ and a
background report, framework and toolkit on maximising impact of youth
entrepreneurship support in a variety of contexts (in which the PPA additionality is
explicitly acknowledged.)
In addition, the consortium utilised PPA funding to cover core and programme costs.
1.7 Relationship of DFID PPA funded activities to other programme
activities
To an extent, the PPA has made other programme activities possible or easier, by covering
some of their expenses and by covering core costs. In addition:
•
•
•
Restless Development’s PPA utilisation is in the process of reducing programme
funding risks by investing in the diversification of its funding base.
War Child’s PPA utilisation increases the scale of its programmes (by increasing its
funding base) and works towards consistent and verifiable programme performance by
developing an organisation-wide M&E system. For War Child, the main value for money
of the PPA funding is not about the size or depth of programmes but about strengthening
War Child’s capacity to deliver these programmes, and to manage the finances behind
these programmes, with low risks and to consistently and verifiably high standards.
YBI has allocated PPA-funded days of several of its staff members to fundraising,
network expansion, the development and provision of services (i.e. mentoring, MEL and
OMS), research and policy work, brand development, and the development of its
Performance and Learning function.
Moreover, the expenses covered by PPA funding may have freed up funding that is now
spent elsewhere, in an undocumented and untraceable reallocation process. As such, PPA
funding is likely to also have:
•
•
Expanded the scale of programmes. YBI is exploring membership possibilities in
Tanzania and Uganda, and may not have done this without the PPA.
Expanded the scope of programmes. In Zambia, the PPA contribution amounts to
40% of Restless Development’s funding, and without it Restless Development would
probably have been unable to diversify into out-of-school youths and civic participation.
Page 10 of 131
•
•
Expanded the depth of programmes. YBI has invested heavily in the capacity building
of its member in Kenya (KYBT) and this has strengthened all dimensions of KYBT’s
performance (e.g. processes, mentoring, monitoring).
Accelerated the speed and rigour of the implementation of organisational
strategies. The strategies of each of the consortium partners have been developed prior
to and thus independent of the PPA. Each of these strategies has transformative
elements which require unrestricted funding.
2 Evaluation methodology
2.1 Evaluation plan
2.1.1 Evaluation questions
The four key evaluation questions have been:
1. To what extent has the consortium acted upon the comments provided by DFID?
2. To what extent has the consortium’s PPA-related reporting been correct and
comprehensive?
3. What has been the impact that the consortium partners in general and the DFID funding
in particular has had on the consortium, its partners and its ultimate beneficiaries?
4. What has been the value for money of the consortium operations in general and of the
PPA funding in particular?
2.1.2 Evaluation design (and rationale for design)
Formal requirements
To fulfil the IPR’s formal requirements, the evaluation was conducted:
•
On the basis of this PPA’s logframe and business case; and
•
To the extent possible, within the framework provided by Coffey’s Evaluation Strategy. I
deviated from this guidance only in cases where the consortium had made PPA-related
investments in types of capacity building that are not explicitly covered by Coffey’s
evaluation strategy (e.g. relationship development, strengthening of governance, risk
mitigation measures, organisational structure).
•
In the spirit of the OECD DAC Quality Standards for Development Evaluation and,
whenever I engaged with children, compliant with UNICEF’s child-focused ethical
evaluation guidelines.
Units of analysis
The units of analysis follow the logframe, which means that the starting points were the
individual beneficiaries and the southern organisations that the consortium partners worked
with. From these basic units, I followed two types of trails:
•
•
Number trails. By following the reporting trail from a randomly selected individual to the
worldwide PPA report, I verified that a single beneficiary is included in the aggregate
beneficiary numbers once and only once. Annex C3 provides an example of a number
trail.
Narrative trails. By following the reporting trail from Annual Review statements and
case studies, I verified that these statements and case studies were based on data that
were valid and reliable, that had been interpreted sensibly and reported on truthfully.
Annex C4 provides an example of a narrative trail.
Sampling of country programmes
The consortium partners have utilised the PPA funding to cover a wide range of capacityrelated, programme and core costs. The implication is that the scope of this evaluation is
Page 11 of 131
each of the organisations’ overall capacity – and changes therein - to consistently reach
relevant results efficiently and effectively. To be able to conduct overall organisational
capacity assessments, this IPR consisted of four components:
1. For Restless Development: a headquarters and two country programme assessments;
2. For War Child: a headquarters and a country programme assessment;
3. For YBI: organisational assessments of the International Network Team in London and
one of the YBI members;
4. For the consortium: an assessment of consortium benefits and dynamics, the
consortium’s Annual Review Process, and the consortium’s follow-up on DFID’s
comments.
The country programme assessments took place in Uganda (Restless Development and
War Child), Zambia (Restless Development) and Kenya (KYBT, a YBI member). I selected
these countries on the basis of two criteria:
1. The size, complexity, relevance (for DFID) and range of the in-country
programmes, to allow me to see programmes in each of the logframe sectors and at
various levels of maturity. This excluded the young and mono-sector operations in the
Central African Republic, for example, or YBI’s members in Western Europe, Australia
and North America.
2. The proximity of the countries to each other, to limit the visits’ carbon footprint to the
extent possible. This meant that I assessed War Child’s operations in East Africa instead
of in Asia.
In my original plans a third criterion was the presence of PPA-specific programmes. Because
of the way the consortium partners have utilised the PPA funding (as described above), this
criterion proved irrelevant.
Sampling of beneficiaries
My original bid for this IPR included two beneficiary surveys. As the consortium partners had
recently conducted a number of surveys already (including several high quality ones) the
consortium team and I agreed that further surveys would not be the best possible value for
money. We therefore took the surveys out of the work plan (which reduced this IPR’s costs)
and instead focused on small scale testing of the Annual Review and previous independent
evaluation findings.
In Uganda and Zambia, I used disproportionate stratified random multistage cluster
sampling. In Kenya the programme population is much smaller and I used blind pencil tip
sampling to select mentors and quota sampling to ensure that I spoke with (potential)
beneficiaries. Annex C1 explains what these sampling techniques entail, and how and why I
applied them.
2.1.3 Research methodology and data collection strategy
Guidance from Coffey’s Evaluation Strategy
For the sake of comparability across PPA evaluations, I employed the methods suggested in
Coffey’s Evaluation Strategy wherever this was possible. This means that I:
•
Reviewed each of the documents listed in Coffey’s Annex 8.1, section 7.1. Annexes E
and F provide a full list of data and documents reviewed in the course of this assignment.
•
Conducted formal interviews, facilitated workshops and focus group discussions, and
had informal conversations with all stakeholders listed in Coffey’s Annex 8.1, section 7.2.
Annex D provides a full list of people consulted.
Page 12 of 131
•
Conducted a cost-benefit analysis where this was possible without making ‘heroic
assumptions,’ as per section 2.2 of Coffey’s Annex 12. The equations and assumptions
used are provided and explained in Annex C2.
Additional methods
Coffey’s evaluation guidance afforded the evaluators considerable flexibility, which allowed
me to select additional methods that are not mentioned in the Evaluation Strategy.
Specifically, I employed:
•
Trailing of numbers and narratives, as described above; and
•
Swim lanes to disaggregate and analyse the steps of programme implementation and
reporting processes. These exercises answered questions related to the three Es of
Economy (will this process consistently lead to the best possible procurement
decisions?), Effectiveness (does this process achieve what it aims to do?) and Efficiency
(is it possible to take steps out of this process?).
2.1.4 Analytical framework
Where possible, I have used the frameworks provided by Coffey’s Evaluation Strategy. For
example, the section titled Summary achievements against evaluation criteria follows the 13
evaluation criteria and colour-coded rating definitions that Coffey provided in its Appendix
5.2.
Coffey’s guidance does not cover all issues that affected the consortium’s performance.
Where I felt that additional assessments were needed to do justice to the consortium’s
efforts to build its capacity, I have used Kaplan’s dimensions of NGO capacity, and
specifically looked at:
•
•
The line of sight from activities and projects to a sensible set of organisational values,
vision and strategy. The line of sight ensures that the achievements are consistently
within the remit of the organisation.
Organisational structure, systems, policies and procedures, which jointly enable an
organisation to actually make things happen. 4
Both dimensions are reported on in the ‘effectiveness’ part of the Value for Money section of
this report.
The two other organisational capacity frameworks that we have considered in the design
stage were: 5
•
•
The EFQM framework of capacity. This framework looks at the ‘enablers’ that facilitate
the achievement of ‘results’, and at the way ‘results’ are monitored and used to
strengthen the organisation. We felt that the framework would be appropriate because of
its emphasis on partnerships (and this consortium itself is a telling illustration of the
importance that the organisations attach to partnerships) and its attention to investments
in issues related to HR, policies and processes, which are all things that the consortium
partners have invested in. We decided against it because the framework is only a little
better than Kaplan’s (because of EFQM’s comprehensiveness and standardisation) but a
lot more demanding on staff and the consultant’s time.
The ECDPM framework. We decided against this framework because the ECDPM
dimensions of capacity (including the ability to ‘act’ and ‘be coherent’) are very different
4
For Kaplan’s other dimensions of organisational capacity (i.e. external relations, acquisition of knowledge and
skills, and financial and material resources) I have followed Coffey’s guidelines.
5
‘We’ is the consortium, this assessment’s peer evaluator (Jill Edbrooke) and I (Willem van Eekelen, the lead
consultant and author of this report).
Page 13 of 131
from the issues covered by Coffey’s Evaluation Strategy, and it would have been difficult
to report on the ECDPM assessment’s findings within the IPR reporting format.
2.1.5 Approach to quality assurance of research
To ensure that the research and its findings were robust I have, in every stage of the
evaluation:
•
•
•
Shared and discussed every new tentative finding with the staff involved, and agreed on
any additional piece of evidence I would need to review to bolster and/or modify this
finding.
Followed each programme assessment with an immediate on-site face to face debriefing
and discussion. In these debriefings, I covered both the findings and the way in which I
had come to these findings.
Written and sent a sub-report within a week after having finalised each of the seven
evaluation components, after which each of the consortium members had the opportunity
– in stages if needed – to discuss the contents and to provide me with additional and/or
counter-evidence.
In addition, I benefited from the advice and coaching of a peer evaluator - Jill Edbrooke –
who reflected, from a distance, on the methodology and findings.
2.2 Research problems encountered
Documentation
The consortium partners have not consistently and adequately documented and filed all
relevant discussions and decision trails. Often, this merely meant a little extra effort, 6 but in a
few cases this has caused the assessment to be incomplete. The clearest example of this is
this evaluation’s inability to rigorously assess the additionality of the PPA funding. The
problem – and this is a problem that many PPA holders will share – is that unrestricted
funding is used to cover a range of core costs. Without PPA, these expenses would have
been covered by other funding sources and this means that there are other expenses that
would not have been covered and/or that fundraisers would have been distracted by the
need to get the basics covered. However, in absence of a ‘with PPA’ and ‘without PPA’
budget, or a documented process of budget decisions, it is impossible to say what these
other expenses would have been, and what the PPA additionality therefore is.
Timing
Restless Development’s Zambia country programme assessment took place in the first week
of the summer holiday. This meant that I was unable to meet with Restless Development’s
volunteers, and that the number of school-going children I have managed to interview is
lower than I would have liked. Similarly, I have heard a lot, from staff, volunteers and
beneficiaries, about the many activities that are organised with and for the different types of
community groups – but I have had the useful pleasure of actually attending only one of
these activities.
6
KYBT’s procurement documentation, for example, was not filed in a single location but did prove to be traceable
and complete.
Page 14 of 131
2.3 Strengths and weaknesses of selected evaluation design and
research methods in retrospect
Key strength: tracing a randomly selected individual to the aggregate worldwide
figure
This is not covered by Coffey’s guidance but it works wonderfully well as a logframe
verification technique. In each location (field office, country office, and London office) the
exercise takes no longer than 30 minutes. You start with a name of a person you have
randomly selected and have met with, and then check every piece of documentation that
goes upwards. In the course of this quick and simple process, you learn all there is to know
about the quality of the final numbers – you know if one (and only one) of the ‘23,497
beneficiaries’ is named Mary Mtonga, who you have met. In addition, you gain a wealth of
insight in relation to:
•
•
•
The organisation’s ability to utilise data for learning purposes. Organisations could
potentially learn a great deal in the course of the gathering process.
The logic and accessibility of the filing systems. This helps to understand the way
organisations deal with issues such as accountability and institutional memory.
The way different functions and offices communicate. Different functions are involved in
the course of the data aggregation process (e.g. programmes, M&E, policy and
advocacy, grant management) and the verification process serves to detect great
interdepartmental relations – or the opposite.
Key weakness: representativeness
The assessments I have conducted in Kenya, Zambia and Uganda are not representative of
some of the other work that each of the three organisations is doing. For example:
•
In the north of Uganda, Restless Development uses a system of community volunteers in
a sparsely populated region. My findings in relation to these community volunteers may
not be relevant or representative for the work Restless Development does in the east of
the country, where the programmes use national and international volunteers in a more
densely populated region.
•
The War Child programmes I have seen in Uganda are by and large rural programmes.
In other countries, War Child is implementing urban programmes, with very different
objectives and designs.
•
I have only assessed one YBI member: the one in Kenya, and I have only looked at this
member’s programme in Kibera. Other YBI members operate at much larger scale and
with much more advanced entrepreneurs, and my findings can therefore not be
extrapolated to the overall YBI membership.
Key risk: social desirability bias and externalities
Within each evaluation component, the first few interviews were very open and exploratory.
In the course of these first interviews and as issues and similarities emerged, the agenda
gradually took shape and this enabled subsequent interviews to become increasingly
focused and designed to minimise social desirability and maximise my ability to detect
externalities. I used:
•
•
The stimulated recall technique. Instead of referring to an NGO programme, I retraced
the steps that have led to certain behaviour (e.g. safe sex) or a certain situation (e.g.
being an entrepreneur). This helps to find programme impact without running the risk of
socially desirable responses, which might be the result of referring to the programme
itself.
A mind-mapping technique in which I gently probed people about their learning in
relation to the programme, and their application of this learning. The reason for this is
that people often changed their behaviour without being aware that this is the
Page 15 of 131
•
•
•
•
consequence of any particular ‘moment of learning.’ This technique brings applications of
learning that participants are not entirely conscious of to the surface.
‘What if’ questions to check programme-induced learning. ‘What would you do if this-orthat happens?’ This technique tends to work better than direct questions about learning.
‘Normalisation of behaviour.’ If I felt that shame or pride was impeding the conversation, I
provided en passant examples of a spectrum of behaviour, without passing value
judgements. This helped with SRH-related conversations in particular.
Counter-bias questions, to neutralise potential bias (‘Surely life as a raider had its
advantages too, right? What where these advantages?’).
The ‘method of silence’, where I asked a question and then said nothing at all until the
interviewee starts talking.
Beforehand, I discussed the local cultural environment with staff and key informants and
chose techniques that befitted the context. For example:
•
If group discussions were vulnerable to power play, I used the nominal group technique
(where people speak in turn rather than at will).
•
If children were likely to be reluctant to answer the questions of the ‘foreigner’, I asked
somebody else to lead the conversation.
•
If people were reluctant to reflect critically on knowledge, behaviour or exposure to
programmes, I used the ‘positive/negative’ technique where I first ask for positive
reflections, and then for critical reflections (sometimes followed by the ‘method of silence’
that I mentioned before).
Personal bias
I am an economist and perhaps too figure-focused. This bias may have showed itself in the
discussions around additionality. There is ample qualitative evidence – in the form of retreat
notes, minutes of meetings and email exchanges – that shows that the consortium partners
have thought very carefully about the best and most strategic possible ways in which to
utilise the PPA funding. The consortium partners would have liked me to assign more weight
to this evidence, and be less hung up about my inability to unambiguously confirm that these
discussions and intensions translated in actual expenditure streams (which are hard to
assess because of the unrestricted nature of the funding, which renders it untagged and thus
unrecognisable.)
3 Findings
For easy reference, I start each section in this chapter with a text box that provides the
guidance and definitions of Coffey’s Evaluation Strategy.
3.1 Results
3.1.1 Performance assessment against logframe
Coffey on the performance assessment against logframe
The extent to which grantees have delivered on outputs and achieved the changes indicated in their
logframes. In the first annual review this will largely assess outputs, while subsequent reviews will be
able to increasingly assess outcomes. The assessment will be of the whole organization or of the
part of an organization’s programme covered by the PPA.
Page 16 of 131
Appropriateness of the logframe and indicators
Impact indicators. I recommend that the impact indicators are reformulated in order to stay
within the realm of the consortium’s control. 7
Outcome indicators. The baselines cannot be compared with the milestones and targets as
the former are national averages and the latter are target group averages. For YBI, the
outcome indicators are inappropriate as their outcomes are measured as the success rates
of beneficiaries (e.g. ‘70% of the entrepreneurs are still in business after three years’) and
not as “the percentage of young people employed or setting up a successful business who
are able to contribute to household income” (outcome indicator 1.1).
Output indicators. The output indicators are ambitious but achievable.
Recommendation 1
The logframe’s impact indicators should be reformulated to reflect programme reach (i.e.
the people and communities in which the consortium operates) rather than national and
global targets.
The logframe’s current outcome baselines are national averages and should be replaced –
if possible at this stage – with baseline figures for the target populations
The remainder of this section covers output indicators only.
The quality of data sources which informed reporting against the logframe
In the Annual Review the consortium states that nearly all output indicators have been
achieved or exceeded. I tested this by following the number aggregation trail from named
individuals who had received support that is in line with the logframe indicators, to the
worldwide aggregate numbers (see Annex C3 for an overview of the steps) and can confirm
that this is correct, even after correcting for the following:
•
•
Restless Development numbers include very minor double-counting in Uganda (where
the reporting process is very thorough and fully documented) and significant doublecounting in Zambia (where the reporting process is less thorough and not fully
documented). 8
YBI’s partner in Kenya reports correctly, though it defines success as ‘receiving a loan’
rather than as ‘running a profitable business.’ It is appropriate for YBI to claim credit for
the work of its Kenyan member as the organisation would not be implementing this
programme without YBI’s fundraising efforts and a range of intensive support services
(e.g. consultancy services to improve operations, OMS, mentoring, coaching). Attribution
of beneficiary numbers to YBI’s work with other YBI members is less straightforward.
7
This recommendation follows the traditional logframe logic and is not in line with DFID guidance on logframes.
In its 2009 ‘How to note; Guidance on using the revised Logical Framework,’ DFID indicates that it does not
expect the consortium to achieve the impact level indicators (i.e. page 11: “Indicators at goal level should be
‘impact’ measures” and “The goal is not intended to be achieved solely by the project.”)
8
The underpinning system errors suggest that the double-counting in Restless Development Uganda are
incidental and lower than 5%, while the double-counting in Restless Development Zambia is likely to exceed 20%
as overlapping groups were both counted in full. Note that Restless Development Zambia is in the process of
strengthening its reporting system and beneficiary aggregation process. Ideally, the new system should include
college students as a separate category to avoid the numbers of this high-opportunity group to be combined, as
is currently the case, with the numbers of the particularly disadvantaged group of out-of-school youths.
Page 17 of 131
•
War Child Uganda underreported its beneficiaries by only considering ‘primary
beneficiaries’, even though some of the ‘secondary beneficiaries’ had the exact same
entitlements and benefits in terms of both training and access to farm land and produce. 9
Recommendation 2
All three organisations would benefit from more robust reporting systems, and should be
careful with statements for which evidence appears insufficiently robust. 10
Assessment of grantees progress in addressing DFID’s feedback
On 5 July 2012, DFID sent the consortium a letter titled ‘Feedback on 2011/12 PPA Annual
Report.’ In the opening paragraph, this feedback says that “In general this is a strong report.
It includes a lot of information and demonstrates that you have taken on board the earlier
feedback we provided.”
In the remainder of the feedback, DFID makes two types of requests:
•
•
Requests for clarification (e.g. ‘what will the new target for life skills training be?’). In its
response to DFID, the consortium has provided this clarification.
Request to change things (e.g. ‘output 2 targets – suggest these are revised upwards’).
The consortium has agreed to make these changes.
DFID’s feedback and the consortium’s response were written prior to the IPR mid-term
review, and most of it remains valid in light of this review’s findings (e.g. there is indeed
scope for cross-organisational learning in relation to, say, internal audits). However,
comparing my findings with the consortium’s feedback to the DFID response, I suggest two
changes to the consortium’s response:
•
•
DFID asked the consortium to increase the target beneficiary numbers, and the
consortium has agreed. In view of my findings, it seems better for the consortium to work
towards assessing the extent of its double counting, eliminate it, and then produce new
estimates.
DFID asked the consortium to increase the reach of its CSOs capacity building efforts,
and the consortium has agreed. In view of my findings, it seems better to focus on
impact rather than reach, by focusing the consortium’s support on organisations rather
than on the people therein, and by basing its support on a better-framed capacity
assessment.
3.1.2 Intended and unintended effects on poor and marginalised groups and civil
society
Coffey on ‘improving lives’
An assessment of the extent and the manner of changes in the lives of poor and marginalized
people as a result of the changes achieved, and the extent to which these changes are likely to be
sustained. It is recognised that agency reporting in this area is likely to be illustrative of changes,
9
The reason for being categorised as ‘secondary beneficiaries’ is that they had joined the groups after the initial
list of ‘primary beneficiaries’ had been closed.
10
For example: ‘In India, every rupee invested by YBI yields a 10-fold turnover’ (PPA submission document,
page 8); ‘approximately 1,000 national Volunteer Development Professionals reach […] 400,000 young people on
a weekly basis.’ (PPA Annual Review, page 22). Also relevant in this context: the consortium reports £273,177
worth of pro-bono support from the private sector (page 19 of the Annual Review), but has not provided – in the
Annual Review or to me - an indication of the calculations that this amount is based on.
Page 18 of 131
rather than comprehensive across the portfolio.
At the level of the individual beneficiaries, the impact of the work of each of the three
consortium partners is verifiably significant in each of the countries I have visited. This is, in
part, because the programmes of the consortium partners zoom in on carefully selected –
and typically hard-to-reach – groups and individuals, and each of them receives
considerable and tailor-made attention for a significant period of time.
This tailor-made and individualised approach of KYBT, and of some of War Child’s
programmes, means that the cost per beneficiary is relatively high and that the number of
people reached is modest and will not have a notable effect on ‘groups’ or ‘civil society’. 11
Within the remits of these KYBT and War Child programmes, this is unproblematic: they do
not have society-wide ambitions and instead focus on specific types of individuals
(respectively underserved entrepreneurial youths and ‘children who are not reached by other
organisations’).
This is different in the case of Restless Development’s volunteering and the War Child
‘REFLECT’ programmes. 12 These programmes work, intensively and for a period of several
years, with groups of 25 to 30 people. The group members learn a range of life skills that
lead to behavioural change. In Zambia, Restless Development’s programme in teacher
training colleges leads to healthier behaviour among students and trickles down to the
schools in which these college graduates subsequently teach (as was recently reaffirmed in
an external evaluation that traced graduate students). In northern Uganda, the Restless
Development and War Child programmes both reinforce the fragile peace by using mothers
as agents for change (War Child) and by organising activities such as ‘Sports for Peace,’
where previously hostile communities meet each other in the context of sports (Restless
Development). The costs of these programmes are modest, because of the use of full-time
volunteers (Restless Development) and local facilitators (War Child). A number of external
evaluations and my in-country assessments both confirmed that these programmes bring
significant benefits and that this behavioural change has an impact on others.
3.2 Relevance
3.2.1 Representativeness
Coffey on representativeness
The degree to which the supported civil society organisations represent and respond to the needs
and priorities of their constituencies, (including where relevant the poorest and most marginalized).
This will include an assessment of whether the planned interventions, as described in the logframe,
continue to respond to these needs and priorities.
Profiles of trustees, staff and volunteers. Restless Development states that ‘we are who
we serve’ and this is largely true in terms of age, as most staff members are young. It is also
partly true in terms of socio-economic profile: many peer educators and all community
volunteers come from the country’s poorer communities. Conversely and sensibly, Restless
staff and its national and international volunteers are recruited and promoted on the basis of
drive, competencies and potential, irrespective of their socio-economic backgrounds.
11
KYBT’s Nairobi-based programme aims to reach 100 entrepreneurial slum-dwellers in three years, at a unit
cost of £2,550. War Child Uganda programme has similarly small projects (such as its CHH-focused livelihoods
programme and its support for disabled children in Paipir Primary School) but also a few programmes that reach
a much larger number of people.
12
REFLECT stands for ‘Regenerated Freirean Literacy through Empowering Community Techniques,’ a
participatory approach to adult learning, community mobilisation and social change.
Page 19 of 131
YBI is youth-focused but not youth-led. There is no overlap between those who YBI aims to
support (young prospective and actual entrepreneurs) and those who govern YBI. There is
only one representative from a developing country, only two women, no youths and no
starting entrepreneurs in the Board of Trustees. Input from young entrepreneurs was
provided through a recent beneficiary survey and a range of in-country events. This input is
not formally tracked or processed.
Recommendation 3
YBI rightly emphasises the importance of youth involvement in decision-making, and would
benefit from operationalising such involvement internally.
War Child does not have any child- or youth-representation within its governance system
either, but the organisation does ensure its target groups are visible and heard. When
recruiting for the new CEO, for example, War Child made sure to ask questions that had
come from children in war-affected countries.
Understanding and responding to the needs and priorities of the target groups.
Restless Development and War Child have years of hands-on experience with vulnerable
and disadvantaged groups. This shows. The programme designs show an understanding of
behavioural change dynamics (multi-method, intensive, face to face and over a long period
of time) and of the risks of dependency. Neither organisation believes in turn-key projects.
Instead, they provide encouragement and only modest support as people and communities
acquire and use new skills, plan for post-project agricultural cycles, build their own buildings
and set up HIV resource centres (to give just a few examples). YBI does not have this
hands-on experience because it is a member organisation and YBI’s primary focus group is
not the young entrepreneurs but the organisations that serve them – but its members do.
Representing the needs and priorities of the target groups in the wider sector.
Increasingly and as a formal part of its strategy, Restless Development uses its hands-on
evidence in its advocacy endeavours. These endeavours are targeting youths directly
(through radio programmes, for example) as well as regional, national and international
policy makers. Although its policy and advocacy work is relatively new, at least as a formal
part of the organisation’s strategic objectives, Restless Development’s advocacy efforts
already show a verifiable track record of success. 13 War Child and YBI have not developed
their advocacy work to the same extent, but they, too, have a few verifiable (albeit
comparatively minor) successes to report.
3.2.2 Targeting
Coffey on targeting
The extent to which the interventions target the poorest and most marginalized, and the extent to
which they target in such a way as to achieve maximum benefit. These targeting strategies are likely
to be mutually exclusive, and the assessment will reflect on the way in which the balance between
them has been struck. This will include an assessment of whether the targeting continues to be
relevant.
The consortium partners target particularly disadvantaged and underserved children and
youths. For Restless Development and War Child, and for the YBI partner in Kenya, the first
13
The Annexed country reports provide a number of examples of such success.
Page 20 of 131
step towards actually finding their target groups is approaching the local authorities for
advice. This is sensible because it prevents the creation of parallel systems. The
government authorities appreciate the attitude, and regularly compare it with peer NGOs:
•
•
•
•
‘Sometimes we see a GIZ vehicle whizzing by, without stopping. Restless would not do
this.’ 14
‘Restless works with us, and this is good. Marie Stopes entered the Parish and started a
SRH campaign, without checking with the health clinic.’ 15
‘Maybe it is a matter of personal skills, but the engagement of Restless is very different
from the engagement of Save the Children.’ 16
‘ACF and the Samaritan Purse were nearly kicked out of the district, because their work
was so poor – Restless never had this sort of problem.’ 17
The authorities decide in which schools, areas, parishes, sub-countries and regions the
organisations implement their programmes. 18 The choices are typically inspired by the
scarcity of other NGOs and governmental programmes, and by the organisations’ mandate
to operate in disadvantaged communities (typically rural in the case of Restless
Development, urban in the case of YBI’s Kenyan partner, and a mixture of the two in the
case of War Child). 19
In school, Restless Development and War Child are sometimes two of many organisations –
one Ugandan school teacher listed nine organisations that all provided support, and some of
this support was very similar in nature (e.g. three organisations provided seeds for the
school gardens.)
Out of school, War Child and Restless Development are often the only organisation that
reach some of the most vulnerable and disadvantaged children in a meaningful way.
Within the communities, some programmes are inherently self-targeting as ‘people who have
other things to do don’t join. They are not interested in the possibilities these groups
provide.’ 20 Other programmes are designed in a manner that ensures appropriate selfselection: KYBT’s programme, for example, ensures that the applicants are genuinely
interested by only making the application forms available a week after its outreach events,
and by insisting on the completion of a four-day training course and a business plan before
providing loans. In each step of the process people drop off and the people who remain tend
to be truly motivated. 21 As a membership agency, YBI’s targeting cannot be quite so
thorough: YBI supports organisations that support ‘under-served young entrepreneurs’ but
does not define the term beyond ‘lacking collateral’ 22 and the organisation does not currently
meet its self-defined targets around gender balance and representation in developing
countries. 23
14
Mary, Napak District.
The man working in the clinic we visited – I forgot to note down his name.
16
Eko Edward, Assistant Chief Administrative Officer, Moroto District.
17
John and Paul, Napak District.
18
In the case of KYBT, the Comic Relief grant stipulated that all beneficiaries have to live in the Kibera slums, but
it is the District Youth Officer that directs the organisation to the actual areas for their outreach activities.
19
The work Restless Development is doing in Zambia’s teacher training colleges seems to be the exception to
the rural rule, but even this programme is indirectly rural in nature: once graduated, most of these students will
teach in rural primary schools and, as an external evaluation confirmed, generally convey some of the HIVrelated messages to their students and the communities in which they live.
20
A response from a group member, when asked why others from his ‘manyata’ had not joined the group.
21
Though there are still people who do not utilise the loan in line with the business plan, and who do not repay
their loans.
22
The member agencies themselves often have much more tightly defined target groups.
23
Specifically, and as per Appendix III of the YBI Strategic Plan 2011-2014:
•
“The overall gender balance […] of those running the businesses will be between 40:60 and 60:40.” In the
case of the PPA countries, YBI is not meeting this requirement: in the logframe overview of 14 May 2012 YBI
15
Page 21 of 131
Recommendation 4
To align itself more fully with the consortium spirit and the DFID priorities, YBI could do
more to achieve its strategy-dictated percentage of members from developing countries,
and to achieve a gender balance amongst beneficiaries covered by these members.
3.3 Effectiveness
3.3.1 Learning
Coffey on learning
The extent to which grantees learn from their work, and integrate the learning into improved
programming, as well as the extent to which others (civil society, governmental and international
organisations) make use of this learning in altered policy and practice.
Restless Development’s learning takes place at three levels. With a single example of
each:
•
Within countries. Restless Development gathers and aggregates the insights and
numbers of its monthly reports from and coaching sessions with volunteers. The
organisation utilises this evidence for programme design modification and advocacy
messages.
•
Across countries. Restless Development has cross-country system of internal audits.
The Zambian Finance Manager conducted the internal audit in Tanzania, for example,
and this exercise resulted in mutual benefits and learning.
•
Globally. Restless Development engages in systematic staff consultation on policies and
other things of importance to organisational performance. These many and sometimes
time-consuming consultations pay themselves back later: once something - the
organisation’s brand, set of values, strategy or policy – is ready, there is a strong sense
of ownership and buy-in throughout the organisation. This buy-in is not just because
there are consultations but because of the nature of the consultations: they are limited in
time and designed to maintain interest and momentum. 24
Restless Development is and will continue to be a downstream programme-driven
organisation. This programme work provides the credibility and the evidence that makes the
organisation potentially valuable for the wider international development sector. Restless
Development recognises this, and its current strategy explicitly aims to ‘share.’ A recent
example of this is the organisation’s work on teenage pregnancies that generated insights
reports to have supported 305 men and 167 women, which is a 66:34 ratio. (See the Excel sheet titled
PQD_PPA_11-12 status_UG & ZAM_14May12.)
•
“The proportion of YBI members that are from developing or developed countries will be within 15% of the
global balance between developed and developing countries.” This is not currently the case with the current
members (55:45 compared to 78:22). The pipeline members will not resolve this imbalance: only 66% of
these countries is classified as ‘developing’ according to the YBI’s criterion, bringing the total to 58:42). The
discrepancy is caused by the challenge of identifying high-calibre potential members in developing countries,
and by the costs of conducting feasibility studies in countries in which Accenture does not have a presence
(and therefore does not provide pro bono support).
24
The brand development process, for example, was driven by people throughout the worldwide organisation. It
was a staggered process that went from ‘do you have ideas?’ to ‘these are the 25 highest-scoring ideas’, et
cetera, until the new brand was launched in a presentation, by each CD simultaneously, about what had led to
the brand and what it needed to stand for, followed, at the end, by the actual name and logo. Then there was a
feedback round in which, for example, uninspired turned into uninspired, as the latter conveyed a more
unambiguous message. The buy-in to the new brand has been impressive: everybody from the senior managers
to the volunteers could explain what the brand and its accompanying words and slogans stood for.
Page 22 of 131
which Restless Development shared with the wider education sector in Zambia to, in the
words of Permanent Secretary of the Ministry of Education, “inform our next steps in
combating teenage pregnancies in the education sector.”
Recommendation 5
In the past period Restless Development has done a lot of learning and this learning has
caused a lot of change. The second half of the PPA period, the organisation should be
dedicated to a consolidation of these various changes.
War Child has traditionally been a hands-on direct programme organisation, with relatively
little attention to systems, structure and division of responsibilities, policies, standards,
processes, reporting, monitoring or reflection. The implication was that much time was
dedicated to reactive trouble-shooting, that reporting times were typically stressful, and that
the capacity of different country offices varied widely. Realising the importance of
strengthening its organisational capacity, War Child is now investing in new (PPA-funded)
positions and systems. In doing so, and at least within the London office, War Child benefits
from its very open mentality: when the organisation identifies a need, it encourages its staff
to go out and learn from other organisations. As long as War Child’s learning systems and
institutional memory remain insufficiently robust, the organisation is unlikely to retain and
utilise all this learning.
War Child has started to share its learning with other actors within the international
development sector. 25 Once War Child creates systems that capture its considerable
learning, the organisation may well develop into a ‘go to’ agency for advice on child
protection issues in fragile environments.
Recommendation 6
War Child would benefit from more explicit information exchange, learning, staff
development and filing systems, and from more diverse and intensive contacts among
colleagues in War Child’s field, country and London offices.
Progress in these areas would strengthen War Child’s institutional knowledge and
memory and, with this, the organisation’s potential to more fully extract and utilise its
learning and to deepen its contributions to wider policy and practice-sharing forums.
YBI’s learning is highly systematised and geared towards transforming into a knowledgebased service provider that identifies, develops and promotes good practice in the field of
youth entrepreneurship, and that enables its members to build their respective capacities to
replicate such good practice. With the help of pro bono experts, YBI continuously monitors
its own progress on this trajectory and adjusts its actions accordingly. It is too early to judge
the extent of success of this transformation as “being a leader in terms of data gathering and
processing is paramount – but for this kind of stuff the time lines are much longer than the
PPA time lines.” 26
In addition to its own learning, YBI strengthens the capacity of its members. Its Kenyan
member, which very nearly collapsed in 2009 and 2010, managed to survive and to develop
25
E.g. In the Wilton Park Conference, War Child’s plea in relation to article 39 of the Convention of the Child’s
Rights made it into the conference conclusions.
26
Andrew Devenport, CEO YBI, conversation on 14.6.2012.
Page 23 of 131
its products and processes as a direct consequence of YBI’s investments in its
organisational learning and capacity.
YBI does not have a history or ambitions in relation to policy advocacy in the wider
international development sector. The organisation does regularly present its Salesforce
application, which Salesforce sees as a show case example of a Salesforce application for
the use of multiple actors in multiple countries.
The consortium partners actively engage with and learn from each other. This is because
they operate a PPA-inspired joint research fund; because they have a system of periodic
learning-focused meetings; and mostly just because people find it easy to ask each other for
tools and suggestions, now that there is a formal link among the organisations: “I just picked
up the phone and asked them for their Finance Policy – I couldn’t do this with other
organisations.” 27
3.3.2 Innovation
Coffey on innovation
The extent to which grantees develop, test, and achieve the adoption by others of new knowledge,
such as in techniques, approaches, and design of interventions. Innovation is a special type of
learning. It is distinguished from learning in general by novelty.
The consortium has been commissioning PPA-funded research that may open the door to
innovation. For example: a review of literature related to youth entrepreneurship support
models found research gaps that the consortium may be able to fill. In addition, there are
many examples where the consortium partners introduced innovation to a country or region,
or have recognised and supported the innovation of other stakeholders. For example:
Restless Development:
helped to roll out UNICEF’s innovative U-Report (an SMS-based civil participation and
learning instrument) in Uganda; 28
•
is replicating, in different locations, a highly empowering action research model that was
first developed by Pastoralist Consultants International and subsequently modified by
Restless Development to become entirely youth-led; 29
•
introduced a youth-led model, implemented by a full-time volunteer in nearly all teacher
training colleges in Zambia, that aims to transforms SRH-related attitudes and practices
of the students and, indirectly, of the pupils they will subsequently teach. Restless
Development will replicate this model in Tanzania, if the Government of Tanzania gives
the organisation permission to do so.
•
War Child:
•
supported FAO by serving as one of the NGOs that are replicating FAO’s “Junior Farmer
Field and Life Schools” (JFFLS) in Uganda, after it had been developed and tested in
Zimbabwe and Mozambique;
•
produced a video animation to facilitate discussions in REFLECT groups, and other
organisations expressed an interest in replicating its use;
27
Helen Ord, Director of Finance, HR and Administration, War Child, conversation on 09.07.2012.
War Child has been part of this roll-out process too.
29
See Kizito A (2012) Strength, Creativity and Livelihoods of Karimojong Youth, Restless Development Uganda,
Pastoralist Communication Initiative and the Institute of Development Studies, Jinja. This study followed much
the same approach as a PCI report titled Brocklesby, MA (2009) Raising Voice – Securing a Livelihood,
Pastoralist Communication Initiative, Addis Ababa.
28
Page 24 of 131
•
introduced child helplines to Kinshasa in DRC and Herat in Afghanistan.
YBI:
•
offers members a best practice mentoring tool that guides and supports YBI members
when designing and implementing a locally appropriate mentoring programme;
•
offers members a cloud-based OMS of which Salesforce said that it is “one of our most
exciting and most-referenced international Salesforce Foundation implementations to
date. The ambition of the YBI multi-country implementation, and the method the YBI
team are taking to project manage the implementation, is ground-breaking and creating
an inspiring model for other non-profits to learn from.” 30 Seven member organisations
(and more will follow) have moved their data and process infrastructure onto this OMS.
This saves time and improved the way these members manage and use data to improve
operational performance.
These examples fall short of Coffey’s definition of ‘radical innovation’ as they are not new on
a global scale in the way REFLECT, Plumpy’nut and ARVs have once been new and there is
little evidence that other organisations have started to replicate the practices that the
consortium partners have introduced. However, they are examples of incremental innovation
in the environments in which the consortium partners operate, and this is acknowledged as
such: “Restless is into innovation, and pushes it in the face of the ministry.” 31
3.3.3 Partnership working
Coffey on partnerships
The extent to which partnerships are made with others (civil society, the private sector,
governmental and international organisations) that enhance the effectiveness and impact of
interventions and encourage sustainability. Partnerships that build sustainability might include:
1. leveraging funds for continuation,
2. securing policy adoption of an intervention or approach,
3. building capacity of southern actors to deliver a service or to monitor service delivery.
The three organisations all have strong networks that include peer organisations, donor
agencies and the corporate sector. In this section, I merely look at the way the organisations
utilise these networks for the three purposes listed by Coffey – but I note that this is only part
of the picture and that the networks serve a range of other purposes too (e.g. mutual
learning, joint ventures, staff exchange).
Leveraging funds for continuation
One of the four PPA-related consortium goals is to strengthen the organisations’ income
position and to ensure that their financial health is independent of future PPA funding. The
Annual Review confirms that this is working out and that the PPA funding enables “the
consortium to leverage additional financial investment for the advancement of young
people’s role in poverty reduction.” (PPA Annual Review, page 5). This has happened in two
ways:
1. All three organisations have invested PPA money towards their fundraising strategies (a
combination of additional staff and additional thinking as captured in, for example,
Restless Development’s first-ever Global Business Plan with a commitment to strong
diversification of partners and funding). This is paying off.
2. The consortium partners used PPA funding to meet match-funding requirements.
30
31
Isabel Kelly, international director of the Salesforce Foundation, March 2011.
Mr Remmy Mukonka, the National HIV/Aids Coordinator of the Ministry of Education in Zambia.
Page 25 of 131
In addition, the consortium partners mention the PPA in funding applications, as an
illustration that the organisations are credible and worth investing in. They believe that this
strengthens their applications. Interviews with KPMG 32 and Danida 33 did not confirm this: the
interviewees in both conversations were able to list many things that had attracted them to
Restless Development, but the strategic support of the British Government was not one of
these attractions.
Securing policy adoption of an intervention or approach
Historically, Restless Development is an action-oriented organisation, and it is only recently
that ‘learning and sharing’ has become an explicit part of its strategy. Restless Development
is quickly mastering the art of inviting others to absorb and utilise the organisation’s evidence
and research findings. The Zambia office has perhaps managed to advance most rapidly.
This office is already putting a well-acknowledged stamp on the work of regional and
national authorities. 34 Other country offices and the Restless Development Headquarters are
following and engaging increasingly meaningfully with governments and multilaterals, and in
inter-agency forums.
In the next few years, the volunteering model of Restless Development, and its role model
function as ‘the professional face of youth,’ are likely to inspire other stakeholders. Once this
happens, other organisations may replicate the models and systems of Restless
Development, and the sector’s decision-makers may adopt some of the organisation’s policy
messages. Once this happens, the impact of Restless Development’s concepts will no
longer be directly proportional to the growth of Restless Development itself.
YBI is “not a policy or lobbying organisation; our core business is to support organisations to
support young entrepreneurs - but we will use our learning in this context to engage with and
influence the wider sector.” 35 YBI currently does this on the basis of its own services and
products (and specifically of its tailored Salesforce OMS, which YBI regularly presents to
other stakeholders), and on the basis of internal and external research. In the next few
years, YBI aims to strengthen its empirical research until it is “world class,” 36 and this PPA’s
final review may already be in the position to confirm that YBI has reached this point.
War Child, too, has done learning that is worth sharing. The most impressive example of this
comes in the form of 3,200 northern Ugandan child safety report cards, which found that one
of the most common truisms on child safety – that homes are safer than institutional care –
is not always true. For now, War Child recognises that its modest size, in combination with
the limited resources it feels able to dedicate to advocacy work and advisory services (15%,
compared to 85% of the organisational attention that is to be focused on programme work)
makes it difficult to open doors at high levels. Therefore, for the time being, War Child tends
to work through coalitions.
32
KPMG’s Richard Bennison (COO Europe) and Roisin Murphy (Senior Manager, Corporate Social
Responsibility), telephone interview on 12.07.2012.
33
Mr Ib Petersen, State Secretary for Danida, telephone interview on 29.08.2012.
34
th
The latest example of this is a workshop that the Ministry of Education brought together, on August 17 2012,
to discuss Restless Development’s study on teenage pregnancies “with a view to inform our next steps in
combating teenage pregnancies in the education sector.” From the opening speech of the Permanent Secretary
of the Ministry of Education of Zambia, 17 August 2012. The country report for Restless Development Zambia
gives a number of other examples of the organisation’s policy contributions.
35
Maarten Rooney, Finance Director and Chief Operating Officer, YBI, conversation on 29.06.2012.
36
The current strategy’s fifth strategic aim is to “Significantly develop ability to gather, analyse and report our
data. World class empirical research will enable YBI to demonstrate the value of its work to policy makers as well
as to current and future resourcing partners.”
Page 26 of 131
Building capacity of southern actors to deliver a service or to monitor service
delivery.
YBI specialises in capacity building of NGOs that provide support to youth entrepreneurship.
In the case of its Kenya partner (which is the only one I visited), YBI has done so very
effectively indeed. KYBT’s service provision, monitoring, process engineering: KYBT has
developed it all with a lot and diverse types of YBI support (e.g. mentoring tools, tailor-made
software, training, remote coaching, onsite support, linking KYBT up with comparable
organisations in other countries). YBI is closely and systematically monitoring progress.
Restless Development also supports CSOs, but the only programme I have looked at (the
programme in northern Uganda) is not yet sufficiently robust. 37 In the next phase of this
programme, Restless Development would benefit from a capacity assessment tool that
would give a better sense of its counterparts’ organisational needs. Currently, training is
often provided on the basis of stated capacity gaps, rather than on the basis of the most
pressing capacity gaps. The capacity building typically takes the shape of training days,
which is only one way of developing capacity, and this training is conducted without
meaningful engagement with the trainees’ peers, manager and staff (which is important as it
increases the likelihood that the acquired skills are going to be utilised and passed on, and
that the learning is institutional rather than merely individual.)
Recommendation 7
Restless Development could strengthen its CSO capacity building programme by focusing
its support on the organisations rather than on the people therein, and by basing its
support on a better-framed capacity assessment.
War Child is working to develop the OVC-related capacity of district and sub-county
government authorities in northern Uganda. The contacts are frequent but challenging. In the
five days in which I conducted the assessment I met with a few able and motivated public
servants, but I also saw ample evidence of disengagement and a chronic lack of resources.
It is a tough programme and success is uncertain as:
37
A much larger and international CSO capacity building programme – also largely training-focused is funded by the Staying Alive Foundation. I have not looked at this programme, but a March 2012
external evaluation by Sophie Bray suggests that its impact has been as follows:
Page 27 of 131
•
•
•
Not all public sector officials I talked with had a good understanding of what the capacity
building programme is meant to achieve.
This disengagement and lack of resources comes from the very top: the state ministry is
unable to secure funding for OVCs, and the national priorities of the Uganda
Government do not mention them.
The poor performance of the OVC-related authorities is not merely capacity-related, but
also has to do with incentives and parallel agendas.
The final review could perhaps give an indication of the usefulness of this programme. Even
if it fails there is no real long term alternative to continuing such efforts as, ultimately, the
Ugandan government will have to take over the responsibilities that are currently allocated to
the NGO community. In the years to come, War Child will continue to struggle to find the
balance between ensuring that the authorities do their part of the work (which may require
providing government-set allowances, fuel and motorcycle maintenance services) and
keeping NGO dependency to a minimum.
3.3.4 Sustainability
Coffey on sustainability
Coffey does not cover sustainability as a separate issue. Instead, Coffey says that sustainability “has
been integrated with the „effectiveness‟ and „results‟ criteria, recognising that sustainability refers to
the continuity of results that typically have been achieved due to an effective approach to delivery.”
(Coffey Evaluation Strategy, section 2.2.1, page 7.)
Coffey asks evaluators to test the “sustainability hypothesis”, which is that “direct service delivery is
localised and unsustainable, whereas civil society holding government to account leads to broader
and more sustainable results.” (Coffey Evaluation Strategy, Annex 4, page 2).
In absence of further guidance, I have:
•
Assessed financial sustainability: the extent to which the consortium partners are likely to be
able to continue operations with or without further PPA funding.
•
Assessed impact sustainability: the extent to which programme impact is likely to last
beyond the life of the programme.
•
Compared some of this assessment’s findings with Coffey’s sustainability hypothesis.
Financial sustainability
The PPA income amounts to 19% of the consortium’s total income.
Page 28 of 131
Income
Annual income of consortium partners (£)
Restless Development
War Child
YBI
PPA funding (£)
38
Restless Development
War Child
YBI
PPA funding as % of organisational income
Restless Development
War Child
YBI
Consortium
2011/12
2012/13
(Indicative)
6,904,774
3,652,103
3,500,000
14,056,877
7,225,139
4,404,581
3,700,000
15,329,720
1,588,791
696,010
470,638
2,755,439
1,588,791
696,010
470,638
2,755,439
23%
19%
13%
19%
22%
16%
13%
18%
The PPA is the only DFID funding that YBI and War Child receive. Without causing
disproportionate dependency, this funding accelerates their growth, and the implementation
of their respective strategies.
Restless Development has also had other sources of DFID funding that, at the start of the
PPA, increased their total proportion of DfID funding to over 50% of the organisation’s
income. This is no longer the case, as the CSCF grants in Zambia and Uganda have now
come to an end.
Aside from DfID funding, Restless Development faces a ‘contract agency’ risk: the risk of
being so dependent on institutional donor funding so that it compromises the organisation’s
independence. In the short run this focus on institutional funding may be a wise choice, as
institutional fundraising is typically more efficient than the penny box, and the identity of the
organisation is very strong indeed. In the longer run, the organisation’s independence would
benefit from a larger proportion of non-institutional funding.
The organisation recognises the need to diversify its funding portfolio. Its fundraising
strategy is implemented by in-country fundraising efforts and a London-based team of
seven, and includes investments in trusts and foundations, a regional fundraising position in
Tanzania, a focus on the private sector and more active engagement with non-British
bilaterals. Successes have been very quick in coming – in her first year on the job, Restless’
Trusts and Foundations Coordinator has secured some £0.5m. This is impressive and
suggests that there are large un- and underutilised funding pools that Restless Development
could tap into. The successes are not entirely surprising as Restless Development is very
well-connected (through its Chair, for example) and already has a successful track record of
pro bono support. 39
38
The Restless Development allocation includes 10% of the total grant for grant management purposes, and
10% of the total grant for consortium-wide research purposes.
39
Restless’ branding has been developed by Fig Tree; many of its internal audits are done through AFID; and
KPMG has provided 39 placements in the past year. The strong ability to attract pro bono services was also
noted in the Due Diligence report: “Across a number of areas of the Grantee’s work, we observed creative and
innovative use of technical placements and pro bono services from skilled professionals to enhance the
Grantee’s access to experienced staff and to resource specific projects cost-effectively within limited resources.
For example, the rebranding exercise was guided by a PR and marketing firm pro- bono and they have a number
of experienced individuals working on technical placements for expenses only. The Grantee appears to make
good use of organisations such as Management Accountants for Non Governmental Organisations (MANGO) to
resource such placements.” (PPA Consortium_Best practice_2011)
Page 29 of 131
Wisely, Restless Development does not merely seek to maximise central income streams,
but also employs its fundraising efforts to empower the country offices and diversify their
funding portfolios. At the moment, in-country fundraising (which should grow to 50% of their
total funding requirements) is almost exclusively focused on the in-country presence of
international donor agencies, but in the longer run the private sector will also part of the
target audience. In this context, Restless Development is currently doing scoping studies in
India and South Africa and in Tanzania. The hope is to find companies that they could build
partnerships with, and get funding from.
Impact sustainability
The consortium partners do not follow a ‘drive-in-drive-out’ approach to development.
Instead, Restless Development and War Child use multi-year behavioural change
trajectories, and YBI builds long-term relations with its members.
The focus on sustainability is explicit and multidimensional. Following the four ‘elements of
sustainability’ of Restless Development, 40 and adding a fifth one:
Element of sustainability
Programme-level
sustainability, where
groups and activities
continue after project
closure.
Verified example
War Child’s beneficiaries of an OVC vegetable growing and
life skills programmes are acutely aware of the possible
discontinuation of the programme and “we talk about this all
the time, and we are going to put money aside for next year’s
seeds.” 41
Scaling up, where projects
are tried and tested, and
then expand.
Restless Development piloted HIV-related volunteering in
Zambia’s teacher training colleges. The concept worked well,
and the organisation is now implementing this programme in
all-but-one of Zambia’s teacher training colleges.
Integration, where
government authorities
integrate things into the
regular operations.
Persuaded by Restless Development Zambia, the Ministry of
Education says it is close to incorporating HIV-related
monitoring in the otherwise rather academic M&E school
visits. This may cause schools to prioritise their HIV-related
work, even after Restless Development’s withdrawal from
Zambia’s school system. 42
Replication, where
partners copy
programmes and
implement them more or
less independently.
YBI develops the capacity of its partners to replicate
(modified versions of) YBI’s mentoring system.
Development of people,
who serve as future
leaders and role models.
Restless Development Uganda implements its northern
programme through locally recruited ‘community volunteers.’
These volunteers are not merely a means to reach
communities but also form a target group themselves, as role
models and tomorrow’s community leaders. The results are
impressive: the volunteers are knowledgeable, wellrespected, turned to for advice, and more ambitious now than
40
They are explained in Restless Development’s Programme Quality Compendium, Chapter 7, page 13.
Catarina Longoli, Community Member, Panyangara and Loletyo, interview on 28 July 2012.
42
This example illustrates the long timelines required for public authorities to incorporate new practice. Restless
Development already used this example in a 2009 guide on Programme Sustainability, to illustrate one of its four
‘dimensions of sustainability’ (‘integration into institutional structures’) – and it still has not actually happened yet.
41
Page 30 of 131
they had been before the programme.
Not everything always turns out to be sustainable. For the time being, and to give just one
example of each programme visit:
•
Restless Development’s Youth Accountability Model in Zambia is promising but, for now
and the foreseeable future the community groups believe that “when the volunteers
leave it is as if Restless is dead.” 43
The peace in the Moroto district of northern Uganda remains fragile and requires the
ongoing support of Restless Development and the wider international development
sector.
The disabled children that War Child supports in Uganda are likely to move back home
without finalising their education if War Child withdraws its support, and the stigma
surrounding their disabilities is likely to return.
Kenya’s KYBT will collapse if YBI discontinues its support at any time in the foreseeable
future, and many of the businesses KYBT is currently supporting will not survive.
•
•
•
Moreover and even in the case of mature or ‘graduated’ programmes, the sustainability of
impact remains uncertain in regions that are as volatile as the ones that the consortium
partners work in. All that the consortium partners can do, and are doing, to maximise the
chance and extent of impact is to work towards empowering the stakeholders they aim to
support, and to work towards government take-up of their programmes.
Sustainability hypothesis
Coffey’s sustainability hypothesis is that “direct service delivery is localised and
unsustainable, whereas civil society holding government to account leads to broader and
more sustainable results.” (Coffey Evaluation Strategy, Annex 4, on page 2 and several
other places).
This rights-based premise is often true, and this is nowhere clearer than in the field of youth.
The bottom line is that, under stable circumstances, no youth-focused programme will ever
have sustainable impact as long as duty bearers do not recognise that anything other than
formal schooling falls under their responsibilities. The reason is obvious: “being young” is
inherently transient and individuals leave and enter this category on an ongoing basis.
However:
•
A fragile peace, as there is in northern Uganda and several of War Child’s operational
environments, requires civil society efforts to consolidate this peace.
Direct service delivery creates evidence that can subsequently be used in rights-based
advocacy work. Doing exactly this is one of Restless Development’s strengths.
Not all direct service programming falls within the scope of any duty bearer. YBI’s work is
an example of this: it is not a ‘right’ of young entrepreneurs to be mentored – but that
does not make it any less useful.
•
•
3.4 Efficiency and value for money assessment
Coffey on Value for Money
Whether the project or programme has achieved the best combination of economy (‘doing things at
the right price’), efficiency (‘doing things the right way’) and effectiveness (‘doing the right things’).
43
Bernard Chishimba, group member of Keembe Community, Chibombo District.
Page 31 of 131
3.4.1 Economy
Coffey on Economy
Relates to the amount of resources or inputs (usually measured in financial cost) which are required
to achieve a given output. Fewer – ‘cheaper’ – resources or inputs represents greater economy (i.e.
spending less).
Procurement.
The consortium’s procurement protocol is strong in some areas but not of consistently high
quality in all offices of all consortium partners. Within Restless Development, the usual
checks and balances are in place and internal audit reports have been followed up on. YBI
has, in response to the PPA-related due diligence assessment, put in place a simple
procurement protocol that covers UK-based procurement. It would be useful for War Child
Uganda and YBI’s member in Kenya to do something similar, as a quick review showed that
the procurement-related documents of both organisations are not currently filed in a single
format and in a single and easily traceable location.
In the remaining period, the organisations might want to explore possibilities to achieve
economies of scale through joint procurement practices.
Mentality. Restless Development in particular is very careful not to incur unnecessary
expenses and not to create risks. This translates into modest salaries and an admirably costconscious mentality, 44 but also to sub-optimal solutions and regulations that seem overly
stringent in an organisation that does not have a history of abuse of funds and in which staff
does not display a money-hungry mentality. Currently, for example:
•
receipt-hunting is sometimes disproportionately time consuming; 45
•
the requirement to return all redundant assets to Uganda’s Head Office costs fuel and
vehicle space and fails to utilise recycling opportunities that would be valuable locally
(e.g. old tyres could be turned into sandals);
And until very recently the entire Restless Development office in London ran on a monthly
£26-costing broadband. 46
3.4.2 Effectiveness
Coffey on Effectiveness
The extent to which objectives are met. The question that would need to be asked is: ‘how far have
the project or programme’s outputs and impacts contributed to it achieving its objectives? An
example would be: did the teacher placement programme improve the quality of the school
curriculum or raise achievement? In the section titled “Results” I cover the end-of-pipeline
effectiveness. In this section, I cover the organisational capacity that underpins its effectiveness.
44
By means of illustration: the accounts show small staff donations to Restless that represent unused parts of
allowances that have been paid by other organisations (e.g. X gets Y from organisation Z, uses only half of it, and
gives the other half to Restless Development). This is unusual – in fact, I have not come across this practice
before.
45
For example: the work done in relation to a missing lunch receipt amounted to well over an hour of senior
management time.
46
Perhaps a consequence of the same mentality and equally in need of change: War Child’s London office
operates on a single telephone line, with every phone ringing every time somebody phones.
Page 32 of 131
Restless Development
Line of sight. Restless Development has an extraordinary line of sight. The organisation’s
values, vision, global strategy, work plans and activities are closely aligned, and this is not
an accident: all the many people I have asked, from senior managers to drivers, could tell
me about the organisation’s values and could, at least in broad terms, link these values to
the organisation’s actual work. This clarity of the line of site is a consequence of the
organisation’s ‘bottom line attitude’ (nicely illustrated by the values of Hands, Head, Heart
and Voice and by its monitoring mantra of What, Why, So what, What now.) This clear line of
sight, and the resultant focus, has a strong positive impact on the organisation’s
effectiveness.
Structure, systems, policies and processes. Restless Development is unafraid of
allocating significant responsibilities to relatively young and inexperienced people. To
mitigate the related risks, the organisation invests heavily in its systems, policies and
processes. In the past few years, the organisation has modified its structure; carefully
designed its monitoring and internal audit systems; and developed its global policies 47 and
processes with ample input from many parts of the organisation.
The second half of the PPA period could be utilised to test and consolidate these changes,
and to kick-start the bits that do not yet work satisfactorily (such as its full cost recovery
system). The final review may already be in the position to assess whether any of the
innovation-related risks materialise – such as the implications of the ‘global salary scale’ that
sets basket-based salaries worldwide 48 for the organisation’s ability to attract scarce skills in
high-paying countries, and the robustness of the organisation’s many matrix lines.
War Child
Line of sight. Only the people in relatively senior positions were able to tell me about the
organisation’s values and strategy. War Child’s programmes are all directly or indirectly
focused on child protection, but vary widely within this remit and do not all follow the same
programme paradigm (e.g. some are firmly rights-based, other are equally firmly needsbased).
Structure, systems, policies and processes. In the first half of the PPA period, War Child
has been reducing risks and increasing value for money. In 2010, the organisation had
negative reserves, different financial and reporting systems in different countries, and
insufficiently rigorous monitoring of financial reporting. By mid-2012, these and many other
issues have either been resolved or are well on their way to being resolved.
In the second half of the PPA period, War Child will have time to consolidate these various
types of progress (e.g. the reserves are healthy and now they need to stabilise; the
templates are sound and now need to be rolled out). In addition, War Child would benefit
from a strengthening of the capacity of the London office. There are straightforward
improvements such as War Child’s infrastructure, which needs an upgrade as an
organisation of this size cannot be covered by a single telephone line and excel sheets for
everything. There are also issues related to a ‘small organisation’s culture’ that do not befit
an organisation that is of War Child’s current size and growing. Efficiency gains can be
achieved by delineating responsibilities more clearly, reducing the directors’ operational
tasks; and strengthening planning, systems and processes. Addressing these issues would
mean less starting-and-stopping of initiatives and would leave directors more time for
strategic thinking. This is possible without losing the important asset of a sense of
community.
47
Including a global Finance Policy that includes a chapter on Value for Money.
I.e. every employee in Role X is able to afford the same basket of products as any other employee in Role X,
irrespective of the country of operation.
48
Page 33 of 131
Even with all the recent and upcoming progress, and for a number of reasons that War Child
has little control over, the performance of War Child will remain inherently vulnerable.
Specifically:
•
•
•
In-country progress takes time. In the environments in which War Child is working,
appropriate local partners are hard to find and even the best ones face serious capacity
constraints. These partnerships take years to develop and claims of fast progress –
made by any organisation working in such environments - are likely to prove overly
optimistic. The implication is that it is wise for an organisation such as War Child to make
long term and increasingly substantial commitments in only a few countries – and in this
context I question the current strategy’s aim to increase the number of countries of
operation from five to eight. In essence: value for money is easiest to achieve in
countries in which the organisation already has had a multiyear presence.
The organisation will never be a smooth machine because the environments in which it
operates do not allow for this. In the field of finances, for example, the talent pool in War
Child’s countries of operation is typically shallow, accountancy knowledge is weak and
accountancy tools are poorly understood. Once Finance Officers are duly trained, they
are likely to be in high demand and leave.
The previous point relates to another disadvantage of a small organisation and War
Child’s streamlined team: departing staff members leave gaps. Systems, processes and
institutional memory might reduce the length and duration of such gaps – but only to an
extent. Rapid
turnover aggravates this lack of continuity.
YBI
Line of sight.
YBI’s line of sight is clear in relation to its own transformative processes and its members,
which are YBI’s direct beneficiaries. The line of sight is less clear in relation to its ultimate
beneficiaries, whose profiles and service requirements differ widely. The YBI Strategic Plan
illustrates this: it is mostly about the network size, identity, resources and services; less
about its members’ work; and least about the ultimate beneficiaries. This is not meant as
criticism, as it is the nature of most network organisations and certainly of a network
organisation that:
•
Does not have direct control over the work done by its members;
•
Works with members that differ greatly in size and focus;
•
Covers both the global North and the global South; and therefore
•
Does not define its members or ultimate beneficiaries very narrowly. The organisation
supports organisations that support entrepreneurship among ‘underserved’ youths – and
these youths range from uneducated slum dwellers in Sub-Sahara Africa to university
graduates in the West, whose only constraint might be that they are not yet able to offer
collateral.
Structure, systems, policies and processes. YBI has recently introduced a full cost
recovery system that is a bit more advanced than its consortium partners’ equivalents. This
is one of several measures that YBI has recently taken to strengthen the cost-effectiveness
of its operations. In addition, one of the YBI Network Team’s key foci is to strengthen this
member-level costs-effectiveness.
This is challenging because YBI’s members are legally independent, geographically far from
YBI’s London office and not obliged to utilise YBI’s services. It is also crucial because,
ultimately, YBI’s cost-effectiveness largely depends on the cost-effectiveness of its
members. The OMS is an example of the impact the network can have on the costeffectiveness of its members: it is a state of the art back office system that strengthens the
Page 34 of 131
cost-effectiveness of YBI’s individual members, and that these members could not
independently afford (no economies of scale) or arrange (lack of specialist knowledge).
3.4.3 Efficiency
Coffey on Efficiency
The relationship between output, in terms of goods, services or other results, and the resources
used to produce them. The question that would need to be asked is: ‘how economically have the
various inputs been converted into outputs, outcomes and impacts? Could more effects have been
obtained with the same budget? In other words ‘doing the right things at the right price’.
Introductory note
Organisations around the world are in competition with each other and therefore commonly
create the impression that the impact of their programmes is consistent, immediate and
enormous. The reality is that new programmes are risky and may either fail or take years of
trial and error before reaching maximum impact. Judging such innovative programmes on
the basis of their initial efficiency and effectiveness criteria kills innovation.
The programme portfolio of Restless Development in Zambia illustrates this point. Its most
mature programmes – which are the HIV-related college and school programmes - follow a
well-designed model. Internal and external evaluations show them to be highly effective, and
the evidence I gathered confirm their findings. 49 Conversely, the organisation’s youngest
programmes seem less effective. When working with health centres and out-of-school
youths, the genuine buy-in of the duty bearers is important but my conversations with a few
of them suggest that this buy-in is yet not consistent and firm. The same applies to the
communities themselves: it sometimes takes a lot of time and energy to get community
groups sufficiently interested to meaningfully engage.
None of my findings in relation to these younger programmes worry me as Restless
Development’s track record suggests that its programmes strengthen over time, and that
they eventually generate evidence that is used for advocacy work that may ultimately
strengthen the capacity and behaviour of the duty bearers.
Restless Development
There is something fundamentally low-cost about the work of Restless Development. The
organisation works through national volunteers who only receive a small monthly stipend,
and through international volunteers who pay their own costs. If the volunteering system
works well (and it does, in the case of Restless Development, for reasons described in the
next section on effectiveness) it is hard to think of ways that could reduce the unit costs –
regardless of how these costs may be measured.
The policy advocacy work of Restless Development is very efficient too: nearly all of it (and
the country reports provide a number of examples) is based on evidence that is created in
the course of Restless’ ongoing work.
49
To give just one of many examples (and see the country report of other examples): Mr Wilfred Chilala, District
Guidance and Counselling Officer of Chibombo District, talked about one of the schools in his district, where the
excessive number of teenage pregnancies showed a marked reduction after Restless Development started its
onsite work.
Page 35 of 131
Recommendation 8 and 9
Restless Development might want to consider the investment required to resolve two
time-consuming inefficiencies:
•
The organisation’s financial cycle. This cycle follows the academic year, which sets
Restless Development apart from most of the rest of the world. Changing this to the
normal calendar year would, in the long run, enhance the organisation’s finance- and
reporting-related efficiency.
•
The multi-system finance system. Currently, finances are managed through a
combination of Excel sheets and Pastel, with a third system covering fundraising
work.
War Child
War Child is not a ‘save-a-life-for-50p’ organisation. The War Child website says that “Our
projects are providing intensive support to some of the most marginalised and vulnerable
children in the world. We’re cost-effective, but not cheap. We focus on quality, not quantity.”
Many organisations specialise in one or only a few products or services (rural roads, school
feeding, health clinics) and produce or provide them at much lower unit costs than War Child
will ever do. These organisations are unlikely to reach War Child’s target group. To reach
this target group of particularly disadvantaged and often ‘invisible’ children and youths, War
Child’s entry point is not any particular product or service but the individual – the particularly
vulnerable girl or boy, unreached by any other organisation – and seeks to facilitate deep
and lasting progress in her or his life. This requires a costly and individualised approach,
without which there would probably be much less meaningful impact. By means of
illustration: a kilogram of second-hand clothes would be cheaper if War Child bought several
containers rather than a few small bundles, but trade in clothes does not befit all livelihood
beneficiaries and a one-size-fits-all approach would result in high economies of scale but low
impact on the ground.
War Child’s tailor-made approach is costly but sound for an organisation that focuses on
such a hard-to-reach target group. Within this approach, I have only seen a few very minor
examples of unnecessary expenditures or other inefficiencies (see the War Child country
report for details). The only example worth highlighting in this report is the use of the time of
War Child’s senior staff: more could be done to ensure that this time is spent wisely (and
with this I mean strategically).
YBI and KYBT
A range of recent developments illustrates that YBI values efficiency. Examples of this are
that:
•
•
•
YBI has developed, and is now closely monitoring, an operating model that distributes
responsibilities in a way in which all staff are fully utilised.
A new policy defines the levels of support that YBI is able to provide members in each of
YBI’s four size-related member categories (essentially ensuring that YBI investments are
linked to operational scale).
YBI utilises a lot of very carefully tailored pro bono support. 50 Without this support,
accreditation processes, research and several other parts of YBI’s work would cost much
more.
But YBI is a membership organisation, and the efficiency of its work partly depends on the
efficiency of its members. The only member programme I have assessed is KYBT’s Nairobi50
The Annual Review monetises this support on the basis of daily fees that range from £500 to £700.
Page 36 of 131
based programme, 51 which is a £255k programme that aims to help 100 entrepreneurial
youths from the Kibera slums. A cost-benefit calculation shows that, in a very optimistic best
case scenario, £1 invested leads to an undiscounted untaxed three-year net increase in
earning of £1.44. 52 This is not an efficient programme, and does not yet present the best
possible value for money. 53 The only funding agency that might be interested in funding
KYBT would either not make this type of calculations or see strategic long term value in the
existence of KYBT. This is the case with YBI, which is the reason that YBI invests so
disproportionately heavily in KYBT’s current operations. The PPA’s final review will come too
soon to assess whether or not this investment will enable KYBT to develop into an
organisation that is able to deliver appropriate value for money.
Recommendation 10
Spinning off the lending activities to a specialised lending institution would enable KYBT
to achieve economies of scale and to concentrate on KYBT’s core competence: training
and mentoring of starting entrepreneurs. KYBT would have to pay this organisation or
company a substantial premium because of the risks related to working with start-up
businesses in disadvantaged communities – but this premium would be several times
lower than KYBT’s own overhead costs per borrower.
3.5 Impact and value for money of PPA funding
3.5.1 Attributable impacts of PPA funding on results, relevance, effectiveness and
efficiency
To the credit of the three organisations, the consortium’s work is not inspired but facilitated
by DFID funding, as they did not develop their PPA proposals to befit DFID’s agenda but to
further their own. Consequently, the PPA funding is, together with all other funding, utilised
to achieve the strategies of the three organisations – and because of the size of the PPA
funding the consortium partners are able to progress towards achieving these strategies
relatively swiftly.
In absence of a ‘with PPA’ and ‘without PPA’ budget, it is not possible to verify what parts of
the operations would not have been funded in absence of a PPA. What I can verify is that:
•
The funding has been used to diversify funding portfolios. Restless Development and
War Child each created a fundraising position that aims to tap into the funding pool of
trusts and foundations. Both organisations have achieved fast and considerable
success. 54
51
From conversations with the YBI Network Team I understood that almost all YBI member organisations are
more sustainable than KYBT, and that no other YBI member is as dependent on YBI fundraising efforts. YBI is
aware that KYBT is not yet cost-effective or financially sustainable, and the latter is one of the reasons that KYBT
is not a fully accredited member of the YBI network.
52
This cost-benefit analysis’ calculations and assumptions are provided in Annex C2.
53
There are two microfinance specialists among KYBT’s mentors. They see value in the KYBT’s mentoring
model and are happy to provide pro bono support. However, when I asked them to compare the unit costs per
successful entrepreneur (defined as still being in business after three year) they calculated that KYBT operated at
8 to 10 times the unit cost that the microfinance companies they worked for would incur when focusing on the
exact same start-up target group. Another YBI member – the one in India – faced a similar lack of
competitiveness and therefore refocused from subsistence enterprises to growth-oriented micro-enterprises
(which they refer to as the ‘missing middle’).
54
YBI already had a healthy and impressively diverse funding base that includes governments (DFID, USAID),
funding charities (Comic Relief, Big Lottery) and - appropriate, considering YBI’s mandate – the private sector
(Credit Suisse, Barclays, Accenture). The organisation continues to strengthen its funding position, and does so
partly with PPA-funded hours.
Page 37 of 131
•
•
•
Restless Development and – proportionately more prominently – War Child have created
strategic PPA-funded positions in the fields of M&E, finances and HR. These positions
aim to strengthen systems, policies and processes, which are all likely to outlive the PPA
period. The final review will be able to assess the extent to which these systems, policies
and processes have strengthened the organisations’ operations.
The funding covers some core costs that are typically hard to cover by any type of
programme-specific funding (e.g. rent, insurance).
The funding has made it possible to conduct research that the consortium-partners felt
would be of consortium-wide relevance.
3.5.2 Value for money assessment of PPA funding
The consortium has allocated 80 per cent of the PPA funding to the three consortium
partners, in proportion to their 2010 budgets. They have utilised the PPA funding to cover a
wide range of capacity-related, programme and core costs. To the extent I was able to do
so, I have provided overviews of the way this funding has been spent in the sections titled
‘Overview of PPA funded activities’ and ‘Relationship of DFID PPA funded activities and
other programme activities.’ Overall, I believe this PPA represents good value for money as:
•
•
•
•
Each of the consortium partners has a sensible strategy in place. The PPA Business
Case builds on these strategies, and the PPA funding has been utilised in line with these
strategies.
Without the PPA these organisations would not have been able to implement their
strategies with quite the same rigour and speed, because each of them has
transformative elements which require unrestricted funding.
When utilising PPA and all other funding, the consortium partners are impact-oriented
and cost-conscious.
It is possible to further strengthen the consistently high level of Economy, Efficiency and
Effectiveness, as per the various recommendations that appear throughout this report in
general and in the section on ‘Recommendations’ in particular.
The consortium members agreed to allocate ten per cent of the PPA funding – £275k per
year - to Restless Development, to cover consortium management costs. 55 The true costs of
managing this consortium are much less than these annual £275k and the remainder serves
as unrestricted funding for Restless Development, for which the logic of the previous
paragraph applies.
Ten per cent of the PPA funding is earmarked for joint projects – mostly research projects that are selected on the basis of their consortium-wide relevance. The final evaluation will be
able to assess the extent to which this research has been utilised by each of the partners.
For now, I assume that these research projects form good value for money. I base this on
the fact that the research projects were designed carefully and jointly agreed on, that the
research reports are relevant and of good quality, and that the organisations have a track
record of following up on learning.
55
Specifically, the consortium, with Restless Development as its lead agent, has:
Set up management systems to effectively manage PPA consortium. These systems include a PPA Steering
Committee, a PPA Working Group, a Technical Leads group, and quarterly progress reporting.
2. Followed up, though not to the fullest possible extent, on the due diligence findings.
3. Set up a learning system that is part of regular Steering Committee, Working Group and Technical Leads
meetings.
Restless Development carefully documents this work.
1.
Page 38 of 131
4 Conclusions
4.1 Summary of achievements against evaluation criteria
This section follows the structure and colour coding principles of Appendix 5.2 of the Coffey
Evaluation Strategy, with green, amber and red representing a ‘high’, a ‘medium’ and a
‘poor’ performing organisation.
The previous sections and, with much more detail, the country programme reports of
Appendix G provide evidence of the statements made in this section. In a few cases, the
consortium partners might be able to support each other to ‘graduate’ from one rating to the
next.
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
Relevance
•
•
•
The organisation provides
evidence of a clearly
articulated targeting strategy
addressing the balance
between maximising impact
and addressing the needs of
the most poor and
marginalised
The organisation shows
evidence of continuously reevaluating their
interventions, and making
appropriate changes, to
ensure that they respond to
the needs of the target
population
•
•
•
Distinctive
offering
The organisation provides
externally verifiable evidence of
a distinctive competence in one
or more areas, whether it has
evolved over the course of the
funding, and demonstrates how
this has added value to the
Targeting is done in close consultation
with public authorities. This avoids
parallel systems and duplication of efforts,
and increases the likelihood of postprogramme continuity.
In each of the country programmes I have
assessed, target audiences are clearly
defined and fall within any definition of
‘the most poor and marginalised.’ These
target groups are often hard to reach and
face multiple vulnerabilities. Therefore,
when balancing between maximising
impact and addressing the needs of the
most poor and marginalised, the
consortium members (and War Child and
YBI in particular) favour impact on
individuals over size of groups.
Conversations with target beneficiaries
during four country programme visits
confirmed that the actual target audiences
are consistently in line with the formal
target audiences of Restless
Development, War Child and KYBT (YBI’s
member in Kenya).
For all three organisations, it is easy to
find examples of modifications in the
design of interventions. These
modifications are a consequence of
ongoing programme monitoring,
evaluation and learning (MEL). The
underpinning MEL systems of Restless
Development and YBI are stronger than
the one of War Child. War Child is
working to change this in the coming
period.
I have verified that, to give just a single
particularly meaningful example of each
consortium partner:
•
Restless Development has only just
formalised its commitment to utilise the
evidence its programmes generate for the
purpose of advocacy work – and is
Page 39 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
sector or industry as a whole
•
•
Learning to
improve
organisational
capacity
The organisation provides
evidence that it has used
learning to improve key
competences which have
become integrated into its
interventions and organisational
practice, leading to increased
organisational effectiveness with
demonstrable results
already achieving significant results.
War Child reaches, as it claims to reach
and in a meaningful manner, particularly
poor and disadvantaged rural individuals
who are not reached by any other
organisation.
Organisations that facilitate and foster
youth entrepreneurship benefit from YBI’s
support services. This benefit ranges from
operational strengthening to survival.
To give just one of many examples for each
consortium member:
•
A key competency of Restless
Development is its use of onsite
volunteers. The use of community
volunteers is relatively new, and the
consequence of the observation that
national and international volunteering
systems provide insufficiently continuity
for some of the more challenging
programmes.
•
A key competency of War Child is its
ability to tap into the music industry’s
charitable heart. The organisation came
to realise that the music industry is
generous but volatile – and this caused
War Child to decide, recently and with the
help of PPA funding, to develop its
capacity to fundraise among trusts and
foundations.
•
YBI is all about providing support to
organisations that facilitate and foster
youth entrepreneurship. A review of YBI
time allocations showed that the
organisation could achieve better value
for money by investing more heavily in
larger and better-established members.
To formalise this concept of more or less
proportional investments, YBI categorised
its members in four size-based groups.
Restless Development and YBI have sound
learning systems; War Child has a learning
mentality that will be more consistently utilised
once it has implemented an underpinning
learning system, which the PPA-funded MEL
Advisor is currently developing.
Learning to
improve
contextual
knowledge
Restless Development and YBI
•
The organisation
demonstrates that the design
and targeting of its
interventions derive from
systematic learning about
the context [my emphasis]
•
The organisation can show
To give just one of many examples for each
consortium member:
•
Restless Development conducted an
innovative piece of action research that
was implemented by community
members with various profiles and widely
different levels of education. The result is
a booklet that provides an unusually
Page 40 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
that learning during the
course of the intervention[s]
has resulted in significantly
improved results.
•
War Child
The organisation shows some
impact of learning about the
context and/or learning during
the course of the intervention[s]
on the design and targeting of its
interventions
•
sharp insight in the lives of the
Karamojons, and that has been utilised to
strengthen the organisation’s programme
in this district.
War Child was implementing a REFLECT
programme with groups of women in
Sudok sub-country of northern Uganda,
when six women in the wider community
committed suicide in quick succession. To
reduce the risk of further suicides, War
Child swiftly changed its target group from
its usual REFLECT groups to the entire
sub-county population. I am unable to
confirm attribution, but understood that
there were no more suicides after War
Child’s high-speed intervention.
YBI’s membership is based on an
accreditation process that requires an
independent assessment of a prospective
member. This contextual assessment
strengthens YBI’s ability to provide
services that befit the context of each
member.
War Child could progress to a green rating in
the remainder of this PPA period by
strengthening its learning systems.
Learning to
share with
others
Restless Development
The organisation can show clear
and verifiable examples of how
learning generated from its
interventions has significantly
improved results and has been
used by others in the
sector/industry
War Child and YBI
•
The organisation provides
examples of learning
generated from its
interventions which have
been generalised from the
intervention context for the
benefit of the sector/industry
more generally
•
The organisation describes a
clear or improving strategy
for communicating the
learning
Although Restless Development has only
recently formalised its commitment to ‘share,’
the results are already tangible. A recent
example of this is its programme experience,
in combination with a study, on the issue of
teenage pregnancies in Zambia, which
Restless Development shared with the wider
education sector. These insights will, in the
words of Permanent Secretary of the Ministry
of Education, “inform our next steps in
combating teenage pregnancies in the
education sector.”
YBI and War Child both provide examples of
learning from its interventions in sector
forums, such as:
•
YBI’s regular presentations of its
Salesforce application.
•
War Child’s contributions to the Wilton
Park Conference, where War Child’s plea
in relation to article 39 of the Convention
of the Child’s Rights made it into the
56
conference conclusions.
The final review could assess if YBI and War
Child have progressed to a green rating
56
“Children who have been neglected, abused or exploited should receive special help to physically and
psychologically recover and reintegrate into society. Particular attention should be paid to restoring the health,
self-respect and dignity of the child.” War Child wishes for a UN Formal Comment to this article.
Page 41 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
because its evidence-based sharing has, by
then, “been used by others in the
sector/industry.”
Incremental
innovation
The organisation provides some
evidence of the development
and testing of existing
knowledge to new contexts, and
can show how this has been
disseminated more widely
One of a few examples for each consortium
member is that:
•
•
•
Restless Development modified the
‘financial fitness’ programme of Zanaco
Bank in Zambia, and tested it in its own
youth-focused programmes. The findings
serve as input for Zambia’s plans to
integrate ‘Financial Health’ into its school
57
curriculum.
War Child Uganda was one of a few
organisations that adopted and adapted
FAO’s JFFLS model, which originates in
Zimbabwe and Mozambique, to befit the
context of northern Uganda. War Child
did this successfully: an FAO Programme
Officer felt that War Child had “achieved
more in a single year of JFFLS than many
of its peers had achieved in two years,”
and mentioned that FAO was utilising
War Child’s evidence to refine its design
58
for future cycles.
One of YBI’s core assets is a Salesforce
OMS that YBI makes available to
interested members. For every member,
YBI and the member jointly explore the
specific needs of the member, and YBI
modifies the OMS accordingly. Salesforce
has recognised the value of this approach
for other membership organisations, and
has regularly invited YBI to present its
application at Salesforce conferences.
The final PPA review will be able to assess if
the organisations have progressed to a green
rating, which requires that:
•
The organisation provides evidence of the
development and testing of existing
knowledge to new contexts that has led to
a demonstrable and significant
improvement in their interventions or
organisational capacity. [Coffey’s
emphasis]
•
The organisation provides evidence of the
extent to which it has been taken up by
others.
Radical
innovation
The organisation provides little
evidence of the development
I have not seen evidence of the development
and testing of genuinely new knowledge.
57
To this end and as one of several pieces of evidence, Restless Development was invited, by the Parliamentary
Sub-Committee on Labour, Youth and Sports, to present its findings on its Financial Fitness programme.
58
Godfrey Ocan, FAO Programme Officer, interview on 30 July 2012.
Page 42 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
and testing of new knowledge
Coffey’s guidelines give this a red colour
coding, but I have not followed this as it is
inappropriate for two reasons:
1. Radical innovation is rare. Very few
organisations ever manage to create
innovation that meaningfully changes the
global sector’s practice.
2. If it happens at all, it would take more
than the 18 months that this IPR report
covers.
Partnership
approach
•
The organisation provides
evidence of a well-developed
[…] partnership approach
with verifiable benefits for
results and sustainability
The organisation provides
evidence of mutual
accountability within the
partnership
The three consortium partners are
fundamentally open organisations that
maintain, utilise and contribute to large
networks. Conversations with contact persons
at government authorities, donor agencies,
inter-agency working group members and
private sector companies all showed
appreciation and typically provided a range of
verifiable examples of mutual benefits.
Restless Development
•
The organisation provides
evidence of an M&E / impact
assessment framework
which ensures that results
(in terms of changes in
people’s lives and civil
society more broadly) and
learning are captured,
shared and taken up by the
organisation and the sector
more broadly.
•
The organisation provides
evidence of impact
assessments which have
generated learning that has
had an impact on the
organisation and the sector
more broadly.
Restless Development has an elaborate
monthly monitoring system for its volunteer
programmes. The organisation systematically
feeds local information to regional, national
and international levels for reflection, learning
and follow up. The process generates
evidence that is utilised downward (to mitigate
risks, utilise opportunities and modify
programmes) as well as upwards (to provide
the basis for sector policy contributions).
•
M&E and
impact
assessment
War Child
The organisation is able to
provide evidence of
improvements to its M&E or
impact assessment systems,
which have enabled it to improve
the capture, analysis, use and
sharing of information on
changes or lessons learned
In January 2012, War Child recruited an M&E
Manager. He – Hur Hassnain - is currently
developing an M&E system in close
consultation with programme and M&E staff in
War Child’s country offices, and this has
already resulted in more reliable and regular
reporting flows.
I did not colour-code YBI’s performance on
this dimension as the definitions do not befit a
membership organisation and do not do
justice to YBI’s efforts in this field.
YBI has recently developed new and highly
formalised MEL systems for itself and for its
members. The internal systems have been
designed carefully, enjoy both formal and
59
informal organisational buy-in, and the final
PPA review is likely to find that the systems
have been implemented successfully.
59
Formal: YBI’s strategic aim is to “Significantly develop ability to gather, analyse and report our data.”
Informal: independently from each other and unprompted by me, all YBI interviewees emphasised the importance
of M&E for the future of YBI.
Page 43 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
The final review will also be able to assess
YBI’s success in strengthening the MEL
systems and practice of its members. On the
one hand, M&E practice has traditionally been
relatively weak within most of YBI partners,
and it will be hard to change their culture. On
the other hand and recognising the
challenges, the investments in the
membership system have been careful and
staggered. They include an OMS that YBI
makes available to its members, distance
coaching, M&E training, and a recent
Accenture-funded facility of “Global YBI MEL
design and implementation support on your
monitoring, evaluation and learning
60
programme.” It may work, and if it does then
it is worth promoting the model for replication
amongst other membership organisations.
Cost
effectiveness
•
•
•
Performance
against the
The organisation is able to
demonstrate a
comprehensive and granular
understanding of its costs
and cost drivers
The organisation is able to
demonstrate good
understanding of its costs
and able to make efficiencies
as a result
The organisation is able to
provide comprehensive and
robust quantitative and
qualitative evidence of its
cost effectiveness
A red rating would suggest that the
organisations have ‘little or no understanding
of its costs and cost drivers’ and this is not
true. Senior staff members could generally list
and talk about the main cost drivers; they
could explain how programme choices were
partly based on these cost drivers, and could
provide examples to this effect; and with
every programme I have assessed I saw
evidence of people working to keep costs to a
minimum.
However, this rating is amber because of the
third criterion, as:
•
I have only seen evidence of attempts to
calculate unit costs from Restless
Development and YBI, and have seen no
evidence of attempts to conduct costbenefit analyses.
•
Programmes are monitored and
evaluated as stand-alone entities, and I
have only seen one example of an
exercise that compared the cost61
effectiveness across programmes.
•
Progress towards a full cost recovery
model is slow and neither Restless
Development nor War Child has managed
to persuade all its employees to take the
first step of maintaining quality work
sheets.
Coffey does not define this and instead writes that “Scoring against this criterion
will be clarified when further details of the new Annual Review Process are
60
A PowerPoint presentation titled Accenture Grant 2012 Member Briefing, page 20.
This was at Restless Development Uganda, where the senior management team compared the inputs and
outcomes of its three focus areas to aid discussions about the best possible focus for the future.
61
Page 44 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
logframe
announced by DFID.” (Coffey Evaluation Strategy, Appendix 5.2, page 18.) I have
not seen this clarification and therefore merely note that the first set of output
milestones has by and large been met or exceeded.
Improving
lives
Restless Development
The organisation provides robust
evidence of how its interventions
have directly or indirectly
resulted in long-term and
sustainable changes in the lives
of the poor and marginalised
War Child and YBI
The organisations is able to
show some verifiable examples
of how the lives of the poor and
marginalised have been directly
or indirectly improved
Changes in
civil society
The organisations is able to
show some verifiable examples
of how its interventions have
directly or indirectly resulted in
sustainable changes to civil
society (i.e. people doing things
for themselves, civil society
holding government to account)
and can clearly demonstrate how
this will improve the lives of the
poor and marginalised
Explanation
War Child and YBI were both able to provide
me with case studies of people whose lives
had been improved as a consequence of their
work. More tellingly, I randomly selected
female and male beneficiaries from
particularly poor and disadvantaged
communities in Uganda and Kenya, and
found that their lives have been improved as a
consequence of the work of these two
organisations.
The same applies to Restless Development
but, in addition, Restless Development is
tracking the impact of its work more
systematically and comprehensively than the
other two organisations. This is not to say that
the organisation is more effective than either
War Child or YBI, but merely that the
evidence it generates is more robust.
Restless Development and War Child both
document examples of sustainable changes
to civil society, and I was able to verify some
of these examples. Of these two
organisations, Restless Development is
closest to a green rating. To achieve this
rating, the evidence that Restless
Development is gathering needs to go beyond
the tracking of behavioural change amongst
individuals, and include an assessment of the
way changes in civil society would cause or
contribute to the sustainability of any such
behavioural change, after the Restless
Development programmes have come to an
62
end.
As a membership organisation, YBI’s remit is
to change civil society by strengthening the
work of its civil society members. Until
recently, the ways in which YBI monitored its
impact (through, mainly, its 2011 member
63
survey) was insufficiently robust. Since this
62
An example of an impact evaluation in relation to civil society rather than behavioural change was of its Youth
Accountability Model in Zambia. This study was not among the organisation’s strongest evaluations, and the way
this study was turned into the PPA’s ‘best case’ case study was not entirely sound.
63
An example of the research design bias of this survey: the report stated that “with a response rate of 20 per
cent overall, there are 95 per cent statistical confidence limits of ± 3 percentage points.” (survey, page 10) This
assumes that the respondents and non-respondents have the same profile, which is unlikely to be the case
because of the loyalty effect (i.e. you are more likely to respond if the service has been useful to you.) There are
ways of testing this (by for example exerting extra effort to get a second batch of feedback, and then comparing
the early and the late responses to check whether or not there is an initial response bias) but no such tests were
employed. The same effect applies within some questions. The question about the usefulness of one’s mentor
suggests that more frequent contacts leads to more positive impact, but the more positive response could also be
Page 45 of 131
Criteria and
sub-criteria
Coffey’s category of
performance
Explanation
survey, YBI has started a process that is likely
to strengthen its own MEL and the MEL of its
members, and this may well cause YBI to
progress YBI to a green rating in the final
review.
4.2 Summary of achievements against rationale for PPA funding
Rationale
Achievements
The PPA Business Case states that “the
consortium’s joint theory of change
rests on their combined knowledge and
assumptions and states that a
fundamental shift needs to take place in
the way young people are valued as
partners in development and growth.
[…]Through the PPA, the consortium
will address this through investing in
programme outputs that in combination,
contribute to outcomes and impact in
the following ways [see next four rows]”
Change takes time and it is too soon to assess the
consortium’s success in achieving this change. The next
four rows provide a few very preliminary observations in
relation to the ways in which the consortium plans to
achieve this change.
1. Demonstrate scalable youth-led
interventions that directly impact on
poverty alleviation and drive wealth
creation, directly leading to the
outcomes.
I have seen evidence of:
•
•
Scalable youth-led interventions in the fields of HIV,
SRH, civic accountability and financial literacy. The
latter may lead to poverty alleviation, but it is too early
to confirm this.
Scalable (but not youth-led) Interventions that impact
on poverty alleviation and wealth creation.
I have not seen evidence of interventions that combine all
criteria (i.e. scalable, youth-led and impacting directly on
wealth creation. YBI’s work comes close but is not youthled. Restless Development and War Child have both
integrated a livelihoods dimension into their life skills
programmes but the direct impact on poverty alleviation is
not very significant, and I have not seen evidence of trickle
down or multiplier effects. War Child has a small livelihood
programme for OVCs but all grants redistribute rather than
create wealth.
2. Build the capacity of structures that
can support young people, ensuring
replication of those interventions at
a larger scale.
YBI does exactly this in Kenya, and is looking to expand to
Uganda and Tanzania. Restless Development has
programmes that support CSOs as well, and has
established inter-agency Youth Working Groups, in the UK
explained by mere appreciation of the pro bono time invested by the mentor, rather than the usefulness of this
time. An example of the reporting bias is the sheer enthusiasm with figures are presented, even if these figures
are less than entirely impressive, and the lack of explanation of seemingly random monetisation [“In 2010 YBI
mentors spent approximately 500,000 hours in total working with their young entrepreneurs, estimated to be
equivalent in financial value to USD 20 million.” (survey, page 14)].
Page 46 of 131
Rationale
Achievements
and elsewhere, that are praised for their approach and
64
speedy effectiveness.
3. Support the implementation of
better policies for young people,
increasing responsibility for and
scale of youth focused
interventions, leading to systemic
delivery of the outcomes.
I have seen considerable evidence of Restless
Development supporting public authorities with evidencebased policy development and implementation (see
country report Restless Development Zambia). I have
seen, but not verified, similar examples from other
countries.
War Child’s policy contributions to public authorities are
less frequent, but I saw two particularly impressive
examples:
1. The organisation is supporting the Ugandan Ministry of
Gender, Labour and Social Development with an OVCfocused district-based capacity development
programme. The final PPA review may be able to
assess this programme’s success. For now, I am
impressed by the rights-based ambitions of this
project: if it works, it will have a long term impact.
2. 200 young people collected the views of 3,200
northern Ugandan children, and found that one of the
most common truisms on child safety – that homes are
safer than institutional care – is not always true. War
Child shared the results with the district authorities,
and the final PPA review may be able to assess what
impact this has had.
YBI does not have an explicit policy advocacy role, and
does not systematically support the work of public
authorities.
4. Develop the evidence base for
investing in youth as an asset in
development to better facilitate the
outputs 3 and 4 (of the logframe),
and the outcomes (in the logframe)
at a global scale.
The consortium partners generate evidence in the course
of their work and are, at different levels of rigour, utilising
this to further the agenda of youth empowerment. In
addition, the organisations conduct both independent and
consortium-based research that furthers this same
agenda. This is a relatively new field of work, and the
evidence of its impact is, for now, promising but anecdotal.
4.3 Summary of problems and issues encountered
The key challenges encountered in this mid-term IPR are that:
•
Restless Development, War Child and YBI are three distinct youth-focused
organisations, with overlapping and complementary visions. They have come together
for the purpose of this PPA. They are different in terms of vision, strategy, underpinning
assumptions, target groups, structure, policies, processes, competencies, resources and
so forth. It is only partially possible to discuss them ‘as a consortium.’ I have assessed
each of them independently, and only highlighted their work as a consortium where they
64
“UNFPA tends to work with government institutions, but progress is sometimes very slow. We could not help
notice how fast the Youth Working Group of Restless was moving, so we joined.” Brian Kironde, National
Programme Officer for Adolescent Reproductive Health, UNFPA Kampala, interview on 20.07.2012.
Page 47 of 131
•
•
did in fact work as a consortium (as has been the case in their approach to research, and
in their regular exchange of contacts, tools and tips).
The consortium did not have budgets ‘with PPA’ and ‘without PPA’ and the consortium
has used the PPA funding to cover a wide range of costs. This meant that it was often
impossible to confirm PPA additionality by means of budgetary allocations. To give one
of many examples: PPA funding has been utilised to recruit person A, which freed up
some of person B’s time, who has therefore been able to attend an event where she has
talked with the head of a donor agency, which led to an invitation for a meeting, which in
turn led to additional funding. This claim is entirely plausible, but it is impossible for me to
confirm that none of this would have happened without the PPA.
There is some discrepancy between the DFID and Coffey guidance. Where this is the
case, I have chosen the advice that suited me best. For example, section 4.1 is titled
Summary of achievements against evaluation criteria. Appendix 5.2 of Coffey’s guidance
provides 13 evaluation criteria, with colour-coded rating scales for each of them. I have
followed these criteria instead of the three bullet points provided by DFID, because:
o Coffey’s 13 criteria were available well before DFID’s three bullets, and I had
used them as reference during the first part of the assessment.
o Coffey’s 13 criteria provided me with a well-defined framework within which to
assess and write, while DFID’s three bullets did not.
o I tried but was unable to cover, for each of the three organisations, the many
components that DFID’s third bullets asked me to cover: “Narrative on the validity
of the assumptions underpinning the design, implementation, management and
partnership arrangements of the grantees.” (Note that this requires a rather
fundamental and complex assessment for four focus areas, times three
organisations, times several assumptions per focus area per organisation)
4.4 Overall impact and value for money of PPA funded activities
The bottom line
The organisations have ambitious strategies that are of the type that require unrestricted
funding (e.g. for innovation and for capacity building in general and system building in
particular). They are on track to achieve their strategies. Without the PPA, progress is likely
to have been slower and less wide-ranging.
A few details
The consortium employed part of its PPA funding to:
•
•
•
Conduct research that is meant to strengthen the performance of the consortium
partners and to generate evidence that may be utilised for the purpose of policy
messages.
Create positions that will strengthen the partners’ systems; and
Create positions that will diversify the funding base into trusts and foundations.
The final review will be able to assess the extent of success of the first two investments. The
potential value for money is high, but the risks are considerable too (e.g. research might
remain underutilised, and introducing new systems to an organisation is never easy and
sometimes fails). The fundraisers are good value for money already: they have successfully
tapped into the market of trusts and foundations (Restless Development gained some £500k
with two fundraisers, and War Child already secured £108k, with a good chance of reaching
£195k by the end of the year. 65
65
£500k: provided in briefings at Restless Development, in July; £108k and £195k: provided by Martin Holliday,
War Child’s Trust Fundraiser, on 10.09.2012.
Page 48 of 131
Some of the PPA funding has covered core costs. It is easy to see what it is spent on, but it
is difficult to assess what this means. For example: the PPA funding covers part of an
insurance contract. What does this mean? This is a necessary expense, so it either
substitutes other funding (but it is not possible to assess what that funding has been spent
on now) or it alleviates the burden of having to find funding to cover this insurance from
fundraisers (but it is not possible to assess what donor contracts these fundraisers have now
invested their time in instead). In absence of the direct attribution of success, I looked at the
organisations and found that:
•
This is the PPA’s only youth-focused consortium and its three organisations implement
programmes that jointly fill the niche of youth protection, well-being, livelihoods and
entrepreneurship. Unit costs for the consortium’s hard-to-reach target groups are
relatively high, but few other organisations reach them to similarly significant effect. 66
•
The organisations invest in collecting the evidence that these programmes generate. On
the basis of this evidence, they strengthen their programmes and produce policy advice
that they have started to provide to the wider sector. Already, the consortium is able to
showcase verifiable examples where this policy advice has been taken up by the wider
development sector (by means of illustration: in Zambia alone this has been the case in
the field of HIV monitoring, reducing teenage pregnancies and financial literacy).
5 Utility
For the consortium partners. I hope this report will inspire the consortium partners to work
towards reaching Coffey’s definition of a ‘high performing organisation’ on even more of the
13 evaluation criteria than is currently the case.
For the country programmes. The recommendations made in Annexes G1, 2, 3 and 4
might be a little useful for the country programmes.
For DFID. I hope that DFID will find this report credible, encouraging, and reason to continue
its PPA contribution to the consortium. I also hope that DFID will take note of the two
observations I make in section 6.3, as only DFID is in the position to follow up on them.
6 Lessons learned
6.1 Policy level
On Coffey’s sustainability hypothesis. The current policy climate looks down on direct
programme delivery. This is illustrated by Coffey’s sustainability hypothesis, which is that
“direct service delivery is localised and unsustainable, whereas civil society holding
government to account leads to broader and more sustainable results.” (Coffey Evaluation
Strategy, Annex 4, page 2).
This assessment reinforces the position that:
•
Because of the long run potential, it is worth investing in programmes that aim to enable
the duty bearer to fulfil its duties and to equip the right holders to claim their rights.
However, in some environments the failure rate of such programmes is high, partly
66
The highest unit costs I have found amounts to £2,550 per beneficiary for KYBT’s start-up microbusiness,
which is 8-10 times higher than the unit cost microfinance companies would incur. Other unit costs are much
lower - War Child spends some £100 per beneficiary of an agricultural and life skills programme (though this
does not include the indirect costs, such as the FAO seeds, FAO overhead, SIDA overhead etc).
Page 49 of 131
•
•
•
because the duty bearer’s key constraint may not be capacity but lack of priority or
parallel agendas.
The consolidation of a fragile peace, as there is in several of War Child’s operational
environments, often requires the direct service delivery of a range of external
organisations.
Direct service delivery creates evidence that can subsequently be used in rights-based
advocacy work. Doing exactly this is one of Restless Development’s strengths (and it
may be possible for Restless Development to support the other two consortium partners
to fast-track their learning in this regards.)
Not all direct service programming falls within the scope of any duty bearer. YBI’s work is
an example of this: it is not a ‘right’ of young entrepreneurs to be mentored – but this
does not make it any less useful.
6.2 Sector level
Behavioural change in relation to sexual and reproductive health. This assessment
reinforces the position that behavioural change is difficult to realise, but that well-designed
programme activities that are rooted in a lengthy onsite presence stand a relatively good
chance. Even in such programmes it is easy to find evidence that awareness and safe
behaviour are not the same: I came across several stories of volunteers who spent their
volunteering time raising awareness on SRH issues and then faced or caused an unwanted
pregnancy. 67
Truisms in child protection. Evidence can falsify truisms. An exercise that gathered the
safety perceptions of over 3,000 children in northern Uganda found that one of the most
common truisms on child safety – that homes are safer than institutional care – is not always
true.
Specialisation in entrepreneurship support. Microfinance institutions and support
organisations require very different competencies and combining the two is likely to cause
inefficiencies. The assessment of KYBT’s operations confirmed this: its lending would be
much more efficient if KYBT subcontracted a microfinance institution to do this, and focused
its own energy on mentoring and training services.
6.3 PPA fund level
On innovation. There is tension between the requirements to innovate and the pressure to
show prompt, impressive and consistent results. Reality is that new programmes are risky
and may either fail or take years of trial and error before reaching maximum impact. Judging
such innovative programmes on the basis of their initial efficiency and effectiveness criteria –
as the IPR framework forces the assessors to do - kills radical innovation.
On social desirability. The Annual Review format pushes PPA agencies to make
statements that lack credibility. In the case of this particular PPA, the clearest example of
this is that the consortium credits the PPA for providing the organisations with the means
and credibility to engage in a range of private sector partnerships. This does not do justice to
the persistence and creativity of the consortium partners. KPMG works with Restless
Development and YBI because KPMG likes the work of these organisations, and there is no
detectable link between their support and the PPA. The Deutsche Bank did not choose War
Child as its Charity of the Year because of War Child’s DFID funding but because of the
67
In the field of teenage pregnancies the international development sector has a widely accepted belief in the
approaches being used, though decades of using these approaches have not achieved dramatic results.
Restless Development’s research in relation to teenage pregnancies in Zambia has the potential to bring
something new to the sector.
Page 50 of 131
organisation’s persuasive proposition and a guitar-playing rock star at the entrance on voting
day.
6.4 Organisational level
The lessons learned are covered throughout this report. In essence, the consortium consists
of three organisations that are in various stages of developing and consolidating their
systems and processes, many of which are strong already. Their mandates are distinct but
all focused broadly on disadvantaged youths. They implement activities that do not overlap
in geography but overlap partly in nature, and this allows for cross-learning that is indeed
happening – though not yet to the fullest extent possible. Their impact varies widely per
country and programme, but is often significant.
7 Recommendations
I have made recommendations throughout this report and the annexed country programme
assessments. In this section, I limit myself to two key recommendations per organisation.
The consortium
On the logframe. I hope that DFID will allow the consortium to revise its logframe’s impact
indicators to reflect the consortium’s programme reach, rather than unattributable global
targets.
On reporting systems. All three organisations would benefit from more robust reporting
systems, and should be careful with statements which seem insufficiently evidence-based.
Restless Development
On institutional funding dependency. Restless Development is very dependent on
institutional donor funding and in the long run this could potentially impact on its
organisational independence.
On CSO capacity development. Restless Development could strengthen its CSO capacity
building programme by focusing its support on the organisations rather than on the people
therein, and by basing its support on better-framed capacity assessments.
War Child
On a minimum level of formalisation. War Child has traditionally been a hands-on direct
programme organisation, with relatively little attention to systems, structure and division of
responsibilities, policies, standards, processes, reporting, monitoring or reflection. The
implication was that much time was dedicated to reactive trouble-shooting, that reporting
times were typically stressful, that the capacity of different country offices varied widely, and
that the impact was probably significant but often unverified. In recent times and with the
support of PPA funding, War Child has started to become more systematically accountable
and verifiably effective and efficient. War Child is mid-way through this trajectory, and should
maintain momentum.
On what makes War Child special. In schools, War Child is one of many support
organisations. Out of school, War Child is often the only organisation with a meaningful
presence. In terms of programme impact and as an evidence-generating tool, War Child’s
Page 51 of 131
out-of-school work outperforms its in-school work. This seems worth keeping in mind when
designing future programmes. 68
YBI
On target groups. YBI could do more to achieve its strategy-dictated percentage of
members from developing countries, and to achieve a gender balance amongst beneficiaries
covered by these members.
On walking the walk. YBI rightly emphasises the importance of youth involvement in
decision-making, and would benefit from operationalising such involvement internally.
68
To a lesser extent, this recommendation applies to Restless Development too.
Page 52 of 131
Annex A: PPA IPR terms of reference
Restless Development seeks qualified individual(s), group(s) or company to carry out
an Independent Progress Review (IPR) of its PPA funding globally and in selected
country programmes.
COUNTRY: London with travel to field locations
DURATION: June 11th – September 28th 2012
REPORTING TO: International Senior Manager
1. Background Information
Since April 1, 2011, the United Kingdom’s Department for International Development
(DFID) has
provided Programme Partnership Arrangement (PPA) funding for Restless
Development as the lead partner of a consortium that is implementing projects in over 60
countries. The consortium partners are War Child UK and Youth Business International.
Activities supported by DFID in these countries are focused around youth livelihood
development, adolescent sexual and reproductive health, youth civic participation, and in conflict-affected countries - child protection.
To assess consortium performance at the mid-term point of the programme term (2011 –
2013), Restless Development will commission an Independent Progress Review (IPR) of
this grant and submit a report of findings to DFID by mid-October 2012. To that end,
Restless Development would like to invite applications from a coordinating consultant or
group, to assess the consortium’s work at the global level, alongside a small number of
more in-depth country level reviews. Restless Development anticipates draft reports from
the consultant to be completed by the end of September 2012 to comply with this
deadline.
The Programme Partnership Arrangements (PPA) and Global Poverty Action Fund
(GPAF) are two of DFID’s principal funding mechanisms and will provide £480 million to
approximately 150 CSOs between 2011 and 2013. The current political climate and
results-based agenda demand a rigorous assessment of the effectiveness of funds
disbursed to ensure that they are managed to provide value for money.
2. Purpose
The main purpose of the IPR is to provide an independent assessment of the relevance,
effectiveness, efficiency, and results of DFID-funded consortium activities at the global
and country-specific levels. It will also verify information contained in the annual reporting
process between Restless Development and DFID, while reviewing the quality and
process of reporting within consortium organisations in general.
The IPR at mid-term evaluation stage will also develop recommendations for Restless
Development, War Child UK and Youth Business International to guide the remaining 18
months of the funding period. Its findings and conclusions will be shared with DFID and
DFID’s evaluation manager Coffey International, the Head Offices of the consortium
organisations, Management teams of the reviewed country programmes, and other
interested partners.
Page 53 of 131
3. Scope of work
Scope
The evaluation will focus on the following areas:
• Youth livelihood development (International offices x 3, Uganda, Zambia, Kenya)
• Adolescent sexual and reproductive health (International office x1, Uganda,
Zambia)
• Youth civic participation (International office x 2, Uganda, Zambia, DRC)
• Child protection (International office x1, DRC)
The international offices of Restless Development, Youth Business International and War
Child are all located in central London.
In addition to general recommendations, the evaluation will develop two to three
recommendations per focus area that can be implemented within the remaining 18
months of funding.
Relevance: Doing the right activities
• To what extent do consortium country programmes, as described in the PPA
logframe, represent and respond to the needs and priorities of constituencies and
target beneficiaries?
• To what degree have consortium activities balanced achieving the greatest impact
while targeting the most vulnerable and marginalised?
• To what extent does DFID funding influence and impact consortium targeting
strategy? (Additionality)
• Going forward, in what ways can the consortium improve the design and
implementation of these activities to achieve greater relevance?
Effectiveness: Doing activities in the right way
• What are the distinctive offerings of the consortium - globally and within the
selected countries - and how does it complement or add value to DFID’s
portfolio??
• In what ways does the PPA mechanism and overall approach taken by DFID
enable the consortium to provide this type of distinctive expertise and service?
(Additionality)
• Going forward, how can the consortium leverage its strengths to improve the
development outcomes of its programmes?
Efficiency: Doing activities at the right cost (e.g. value for money)
• To what extent can the consortium demonstrate cost effectiveness, including an
understanding of programme costs, the factors driving those costs, and linkages
to performance and ability to achieve efficiency gains?
• To what extent has the consortium delivered results at improved efficiency due
specifically to receipt of DFID funds? (Additionality)
• Going forward, in what ways could the consortium reduce costs without sacrificing
quality?
Results: Doing activities that produce positive change not otherwise achievable
• To what degree are consortium organisations progressing towards their outcomes
as described in the PPA logframe?
• To what extent have the consortium organisations directly impacted the lives of
beneficiaries, positively and negatively, as described by the PPA logframe?
(Additionality)
Page 54 of 131
•
To what extent have the consortium organisations caused positive change as
described by the PPA logframe? (Additionality)
Cross-cutting themes: additionality, organisational learning, innovation, and
sustainability
In evaluating each of the four categories of assessment, attention should also be given to
measuring additionality and to highlighting cases of organizational learning, innovation, and
sustainability.
•
Additionality. Particular emphasis will be placed on measuring additionality,
defined by DFID as “the additional benefits that are directly attributable to the
activities delivered by the project.” The additionality of funding demonstrates how
the results arising from an intervention would not have occurred in the absence of
the intervention. Discussion of additionality should also highlight whether the
PPA mechanism in particular has a multiplier effect on overall project reach,
quality, and impact.
•
Learning and innovation. Examples from each country program should be
provided and include learning that improves capacity (e.g. organizational
development) and learning that improves contextual knowledge (e.g. learning
about the situation of the target population). The degree to which these types of
learning have and could improve the consortium’s programming, as well as the
structure and use of the organisations’ monitoring and evaluation systems, should
be described
•
Sustainability. The review should assess the extent to which program activities
are sustainable across each of the four criteria. This should include an
examination of the outcome of the uptake of the consortium’s learning and
innovation by others. It should also include the nature of partnerships built with
civil society, governmental and international organisations and their impact on
sustainability e.g. through leveraging additional funds, securing policy adoption of
an intervention, or building capacity of partners.
4. Methodology
The methods to be used in the IPR include:
•
Desk study and document review: The evaluation team shall review proposals,
reports and other documents associated with Restless Development and the
consortium organisations’ work with DFID under the PPA.
•
Interviews and workshops with key stakeholders: The evaluation team will
conduct interviews with consortium staff, civil society and government partners and
volunteers and beneficiaries at Head Office and in the field.
•
Beneficiary focus groups and surveys: For the assessment of country
programmes, the evaluation team will meet with beneficiaries and community
representatives of the target population. This can include household surveys, focus
groups, and interviews.
•
Field visits and other travel: Field visits will be made to each of the countries under
review, as specified in the scope section above.
Page 55 of 131
The consultant or consulting firm commissioned to carry out the IPR and Restless
Development’s Senior Manager for Programme Quality are jointly responsible for choosing
the methods that are the most appropriate for the purpose of this evaluation. As part of their
expressions of interest, interested consultants or consulting firms should present a statement
of their proposed evaluation methods including a description of data collection instruments
and procedures, information sources and procedures for analysing the data. Final data
collection tools will be signed off by the Senior Manager for Programme Quality prior to field
work.
5. IPR Consultant
The IPR shall be carried out by a suitably-qualified and experienced consultant or
consulting firm (referred to as “IPR consultant” in the following). The consultant profile
should include:
•
•
•
•
•
A specialist with a minimum of seven years of experience in programme/project
delivery in an international development context
Up to date knowledge of DFID's priorities, operating environment and the value for
money agenda.
Experience with results-based monitoring and evaluation and familiarity with
different methodologies for evaluation, including factors involved in evaluating
'influencing' work.
An understanding of the specific sectors that consortium partners work in with
respect to the PPA. These include youth livelihood development (with a focus on
entrepreneurship), adolescent sexual and reproductive health, governance and
conflict resolution.
Ability to produce concise, readable and analytical reports
While IPR consultants may be nominated by the stakeholders listed above, they shall
have no past connection with the object of this IPR process, they will not act as
representatives of any party and will remain independent and impartial.
6. Management Arrangements
Restless Development’s Director of Programme Quality will be responsible for the
recruitment of and initial briefing to the IPR consultant. The International Senior Manager
will be the point of contact within the consortium for the duration of the IPR process,
including provision of logistical and technical support, and organisation of meetings and
interviews.
7. Deliverables and Timeframe
The IPR consultant will submit three reports and offer a presentation to relevant staff from
the consortium. Consultants will also provide a short presentation summarizing key
findings for country management teams at the end of each field visit.
•
Inception report (June 25th): Following the desk review and prior to beginning field
work, the evaluation team will produce an inception report subject to approval by
Senior Manager for Programme Quality. This report will detail a draft work plan with a
summary of the primary information needs, a breakdown of the methodologies to be
used, and a work plan/schedule. With respect to methodologies, the evaluation team
will provide a description of how data will be collected and a sampling framework,
data sources, and drafts of suggested data collection tools such as questionnaires
and interview guides. It should highlight the strategy for addressing additionality and
Page 56 of 131
efficiency (value for money) analysis. This includes reviewing the type and detail of
data available for analysis, the analytical approach selected, and a justification for
that approach. The report should specify which countries and regions the team will
visit. Once the report is finalized and accepted, the evaluation team must submit a
request for any change in strategy or approach to the International Senior Manager.
•
Draft report v1 (September 7th, 2012): A draft evaluation report will be submitted to
the International Senior Manager and selected representatives of each consortium
organisation, who will review the draft and provide feedback within one week of
receipt of the draft report. The report will follow the guidelines in the UNEG Quality
Checklist for Evaluation Reports and cover the following areas: Executive summary
(5 pages); Evaluation purpose, objective(s), and scope; Evaluation methodology;
Findings; Conclusions with recommendations and lessons for the programmes going
forward; Appendices, including evaluation terms of reference, maps, sample
framework, and bibliography.
•
Draft report v2 (September 21st, 2012): A second draft of the evaluation report
(incorporating feedback from the initial review) will be submitted to the International
Senior Manager and circulated to the Senior Management Teams ahead of the
consultants presentation to the SMT on Tuesday, September 25th.
•
Final report (28th September, 2012): Any comments from the Senior Management
Teams will be addressed and edits incorporated ahead of this final report
submission.
All material collected in the undertaking of the evaluation process should be lodged with the
Senior Manager for Programme Quality prior to the termination of the contract.
•
Presentation of findings (June/July and September): At the end of the field research,
the evaluation team will present key findings to management teams in each of the
countries visited. After the draft report v2 is submitted on September 21st, the
evaluation team will present their findings to representatives of the consortium
organisations’ Senior Management Teams and facilitate a discussion about the use
of PPA funds during the next phase of DFID support. This meeting is scheduled for
Tuesday September 25th.
8. Expressions of interest
Those interested please submit a bid to Restless Development through the e-mail address
below, stating clearly the e-mail and /or phone number of the person we should contact.
Bids must include the following:
• 3-4 page outline of evaluation framework
• Proposed evaluation budget. This should be in Pounds Sterling and
inclusive of fees and expenses, but exclusive of international flights, visas
and international accommodation
• CVs and evidence of evaluation experience for each team member
Proposals must be submitted before midnight on Monday May 7th
Shortlisted consultants will be interviewed on May 16th or 17th with a decision made shortly
after
Start date is anticipated as June 11th
Please state ‘PPA Independent Progress Review’ in the email title.
Page 57 of 131
Mail: [email protected]
Tel: +91-11-46053183
Page 58 of 131
Annex B: Evaluation research schedule and timescales
July
August
September
Action
Discuss and finalise inception report
Verify PPA reporting (UK, part 1)
Conduct organisational capacity assessments (part
1)
Visit Uganda, Zambia and Kenya
Verify PPA reporting (UK, part 2)
Conduct organisational capacity assessments (part
2)
Assess follow up plans on DFID’s comments
Draft report
Submit first draft report
Receive feedback on first draft report
Discuss and process feedback
Submit second draft report
Present report to Senior Management Team
Process feedback of Senior Management Team
Submit final report
28
2
915
1622
2329
305
612
1319
2026
272
39
1016
1723
2430
3-15
3-15
19-17
272
272
272
27-7
7
14
1720
21
25
2628
28
Page 59 of 131
Annex C: Data collection tools
Annex C1: Sampling
In Uganda and Zambia, I used disproportionate stratified random multistage cluster
sampling:
•
•
•
•
•
Cluster sampling because of the poor infrastructure, to avoid long hours of travelling:
instead of selecting individuals randomly, we selected clusters (such as a school or a
village) randomly, and then sampled randomly within the cluster.
Multistage sampling because within a cluster (a school, for example) I would sample
once more (and focus on a single class, for example).
Stratified sampling because it is important to get representatives from different groups,
such as girls and boys, or children in-school and children out-of-school. This ensured
that, for example, I talked with female volunteers in a region where 80% of the volunteers
were male.
Disproportionate sampling because DFID is more interested in some groups than in
other groups. Specifically: DFID is interested in the extent to which programmes reach
particularly disadvantaged groups. As such, I focused more on girls than on boys, more
on out-of-school youths than on in-school youths, and on children and youths with
disabilities wherever they participated, or could potentially have participated, in the
consortium partners’ programmes.
Random sampling, because within these clusters and within these stratified groups, I
would sample randomly, either using the blind pencil tip method if there are lists, or a
systematic technique such as inviting every fifth child in the class for an interview.
In Kenya, the programme population is much smaller and I used blind pencil tip sampling to
select mentors and quota sampling to ensure that I spoke with (potential) beneficiaries who:
•
•
•
•
•
Registered their interest in joining the KYBT programme but did not follow up;
Started KYBT training but did not complete it;
Completed the training but did not submit a business plan;
Submitted a business plan but did not receive a loan; and people who
Went through the entire process and received a loan.
In each of these groups I ensured to speak to both men and women, in equal numbers.
Annex C2: Assumptions and calculations of KYBT’s cost benefit
analysis
If we assume a very successful three-year Comic Relief programme, with a total budget of
£255,000 and the aim to assist 100 young entrepreneurs, in which:
•
•
The cumulative loans per person have an average value of £800 (Sh 100,000), which is
entirely invested in fixed and working capital (K) and in which the fixed assets do not
require depreciation.
The average business creates a gross annual profit (revenue minus running costs, or RC) of £1,452, based on an average daily profit of Sh 700, a six-day work week, an
average of a month per year of illness or other causes of absenteeism, an average of Sh
2,000 for premise rent, and an exchange rate of 125 shillings to the British pound. This
assumes considerable business growth in the course of the three years (partly unaided
growth and partly the consequence of additional loans) as the current average daily profit
is much lower than Sh 700: the people I talked with reported an average daily income
per person of some 400.
Page 60 of 131
•
There is no VAT or any other tax (i.e. T = 0) and no rate of discount (i.e. i = 0).
There is no instant drop-out, a 3-year business survival rate of 80% and half the average
three-year income for the businesses that do not reach this 3-year limit (which means
that the collective profit is 0.9 times the typical R-C). 69
The KYBT cost per beneficiary (I) is £1,610, which is the average cost of £2,250, minus
the average repayment of £640 (which assumes a 20% default rate, so 0.8*£800).
•
•
Then the cost-benefit equation - [3*0.9(R - C) - K] / I - would result in every £1 invested
leading to an undiscounted untaxed £2.20 in business income [i.e. (3*0.9*700*48*6100000)/125/1610].
This income is not the same as the net increase in earnings, as nearly all KYBT beneficiaries
are in the category of ‘economically active poor’, which means that nearly all of them have
opportunity cost (i.e. would have been able to generate an income in absence of a KYBT
loan; O.) 70 The inclusion of an average opportunity cost (i.e. alternative income) of KSh200
in the calculation shows that a £1 investment leads to a net increase in earning of £1.44 [i.e.
3*0.9(R – C - O) - K] / I - [i.e. (3*0.9*(700-200)*48*6-100000)/125/1610].
In reality, the equation will be closer to:
T
∑
t =0
0.9( R − C − T − O − K )t )
(1 + i ) t
T
It
∑ (1 + i)
t =0
t
As:
•
•
•
Assets do depreciate, which means that K1, K2 and K3 are not 0.
The rate of discount is not 0 (in the context of Kibera the ‘catastrophe risk element’ could
suggest that it should be at least 10%).
There are tax-type costs in Kibera (T), even though the area operates largely outside of
the formal economy.
Moreover,
There is a percentage of instant drop-out as the environment of Kibera is volatile and the
KYBT beneficiaries face multiple vulnerabilities. The current portfolio already shows a
percentage of instant drop-out.
•
KYBT’s beneficiaries frequently face acute crises that require whatever money can be
found. This means that K does not always lead to any R, which drives the average R
down to well below Sh700 per day. There are already examples of this in the current
KYBT portfolio.
•
This means that £1 invested generates a maximum of £1.44 in net increase in earning, and
is very likely to generate considerably less than that.
Annex C3: Verifying number trails
By following the reporting trail from a randomly selected individual to the worldwide PPA
report, I verified that a single beneficiary is included in the aggregate beneficiary numbers
once and only once. This is an example of the steps of such a verification process:
69
Measuring beyond three years would be unrealistic in the volatile environment of Kibera.
I base this on conversations with 14 Kibera inhabitants, all of whom had previously indicated an interest in
KYBT and all of whom I asked about their current and past income-generating activities.
70
Page 61 of 131
Select a project site.
In the field office:
•
Can I see the list of project sites please?
•
Select one site, using the blind pencil tip method.
•
Can I visit the Kalopajak youth group in the Lokuwas
Parish of Matany sub-county please?
Randomly select a single
beneficiary and confirm
this person’s reporting
category (e.g. out-ofschool female under 24.)
In an interview:
Confirm name and background.
•
Ask questions (including counter-bias questions to
minimise the risk of a social desirability bias) related to
the nature and source of knowledge, behaviour or
support received, and about alternative scenarios (e.g.
‘what would you do if …’).
If this person benefited from the project in the way reported
by the PPA logframe, she should appear on the list (see
next step). If she did not benefit from the project in the way
reported by the PPA logframe, she should not appear on the
list.
Confirm that this person
appears, and only appears
once, on the local
beneficiary list.
In the field or country office: confirm that this person
appears once and only once on the beneficiary list for this
Parish, and is categorised correctly. There is a problem with
the reporting trail if she does not appear or appears more
than once. If she appears once, the trail is correct. If this is
the case: note down the overall and category number of
beneficiaries on this list (e.g. 27 beneficiaries, 3 of whom
are out-of-school females under 24).
Confirm that the numbers
of this local list appear,
and appear only once, on
the regional beneficiary
list.
In the field or country office: verify that the regional numbers
of beneficiaries is the sum of the project site numbers (e.g.
the 27 beneficiaries of this parish are part of the regional
sum total of 425 beneficiaries, and the 3 out-of-school
females under 24 of this parish are part of the regional sum
total of 39 out-of-school females under 24).
Confirm that the numbers
of the regional list appear,
and appear only once, on
the national list.
In the country office: verify that the national numbers of
beneficiaries is the sum of the regional numbers (e.g. the
425 beneficiaries of this region are part of the national sum
total of 3,259 beneficiaries, and the 39 out-of-school
females under 24 of this region are part of the national sum
total of 348 out-of-school females under 24).
Confirm that the numbers
of the national list appear,
and appear only once, on
the organisation’s
worldwide list.
In the London office: verify that the national numbers of
beneficiaries is the sum of the national numbers (e.g. the
3,259 beneficiaries of this country are part of the worldwide
sum total of 11,539 beneficiaries, and the 348 out-of-school
females under 24 of this country are part of the worldwide
sum total of 1,252 out-of-school females under 24).
Confirm that the numbers
of the organisation’s
worldwide list appear, and
appear only once, on the
PPA beneficiary list.
In the London office of Restless Development: verify that
the consortium’s worldwide numbers of beneficiaries is the
sum of the organisation-specific numbers (e.g. the 11,539
beneficiaries of this organisation are part of the worldwide
sum total of 32,211 beneficiaries, and the 1,252 out-ofschool females under 24 of this organisation are part of the
•
Page 62 of 131
consortium sum total of 9,158 female youths).
Annex C4: Verifying narrative trails
By following the reporting trail from Annual Review statements and case studies, I assessed
whether these statements and case studies were based on data that were valid and reliable,
that had been interpreted sensibly and reported on truthfully. For example, the consortium’s
‘Best case scenario case study’ is about the Youth-led Accountability Model for Enhancing
Access to Health Services in Zambia. This case study is based on an external evaluation
report titled Evaluation of the Youth Accountability Model for Enhancing Access to Youth
Friendly Health Services.
•
•
•
Comparing the case study with the evaluation report showed that the case study:
o Omitted important contextual information. One example is that the case study
reports on the improved communication between youths and decision-makers
and on the increased frequency of government officials’ community visits - but
fails to mention that these improvements were observed during election
period.
o Misinterpreted statistics. One example is that the case study reports that
there has been a “significant increase in youth accessing condoms from
health centres (20% to 84%) and treatment for STIs and antenatal care (33%
to 65%). The 84% is based on clinical records, which means that this
percentage relates to youths who are in the health centres already, and says
nothing about the percentage of access to the health services. The 65% is the
percentage of youths that reported that it is “fairly easy to access treatment
for STIs at their local clinic” which is different from actually accessing such
treatment (which 7% of the respondents have done).
Assessing the quality of the data showed that the evaluation did not deal adequately with
a potential social desirability bias, which means that the results may have been
exaggerated. One example is that 65 respondents reported to have listened to the radio
show, but none of the respondents was able to name a single topic that these radio
shows had been discussing. This is unlikely and suggests that people might have given
answers that they felt would please the interviewer. Restless picked up on this and has
introduced ‘radio listener groups’ in colleges and elsewhere. A conversation with two
members of such a listener group illustrated the challenging work environment in which
Restless operates: these two youths had missed several shows because they had not
always been able to arrange a radio.
Comparing the findings with my own field observations, I was able to confirm some but
not all of the evaluation’s conclusions. One example of each is that I could confirm there
was regular contact between youth groups and political decision-makers, but could not
confirm that the health centres’ ‘youth friendly corners’ had indeed lowered the barrier to
entry for youths, as neither of the two health clinics had such a youth friendly corner (but
note that not being able to confirm the evaluation’s findings does not mean that the
evaluation’s findings are not valid).
Page 63 of 131
Annex D: List of people consulted
Name, first name
Organisation
Role
Type of contact
Date
Aanyu, Susan
Karamoja Employee
Savings
UNICEF, Kampala
First loan circle-Manager
Interview
23.07.2012
Youth Partnerships Consultant
Interview
20.07.2012
Rupa Parish, Rupa subcounty
War Child UK
FAO
War Child UK, Pader
office
n/a
Chairwomen, community group
Interview
25.07.2012
Advocacy Officer
Programme Assistant
Project Officer
Conversation
Interview
Debriefing and discussion
10.07.2012
30.07.2012
01.07.2012
KYBT beneficiary
13-14.08.2012
Matany Sub-County
War Child UK, Kaabong
office
War Child UK, Pader
office
n/a
Community Development Officer
Senior Project Officer – Livelihoods
Interview in office and,
later, visit to business
premise and another
interview
Group discussion
Focus group discussion
and debriefing
Debriefing and discussion
14.08.2012
Partnerships and Capacity Building Officer
Visit to market stand and
interview
Telephone interview
Focus group discussion
and debriefing
Several conversations
Community group member
Interview
25.07.2012
Office Assistant
Debriefing and discussion
01.07.2012
Abdulrahim, Shaban
Ahmed
Achilla, Regina
Adams, Kate
Adey, Michael
Adoch Ketty,
Stella
Adoyo, Peter
Agatha Achia
Akello, Beatrice
Akello, Paska
Akiny, Ridah
Akinyi, Emily
Aloka, Mary
n/a
War Child UK, Kaabong
office
AloyliousTembeiza Restless Development,
Moroto office
Amaese, Lomoe
Rupa Parish, Rupa subcounty
Amony, Grace
War Child UK, Pader
office
Project Officer
KYBT beneficiary
Potential KYBT beneficiary
Sr. Project Officer – GBV
24.07.2012
31.07.2012
01.07.2012
15.08.2012
31.07.2012
23-26.07.2012
Page 64 of 131
Name, first name
Organisation
Role
Type of contact
Date
Amugosi,
Josephine
Anamela, Likando
Andanje, Brian
n/a
KYBT beneficiary
14.08.2012
Faulo Advisory
Ex-volunteer
KYBT mentor
Askew, Karl
Restless Development
UK
n/a
Finance Director
Visit to premise and
interview
Focus group discussion
Visit to work place and
interview
Face to face
Potential KYBT beneficiary
Telephone interview
15.08.2012
War Child UK, Pader
office
Christian College
KPMG
HR Officer
Interview
01.07.2012
Member of HIV Committee
COO Europe
09/08/2012
12.07.2012
n/a
Napak District
Potential KYBT beneficiary
District Youth Counsellor
Focus group discussion
Telephone interview,
together with Roisin
Murphy
Telephone interview
Interview
09.07.2012
10.07.2012
8&10/08/2012
Community Development Assistant
Conversation
Conversation
Long conversations;
accompanied me for two
days
Interview
Ex-volunteer
Interview
10/08/2012
Fundraising Partnership Manager
Fundraising past, present
and pipeline
Interview
Three focus group
discussions (girls with
disabilities, boys with
8/08/2012
Ayata, Reuben
Owuor
Baluka, Martha
Banda, Eduard
Bennison, Richard
Boruyu, Rose
Bosco, Hon Agillu
John
Bouvier, Anne
Bowcutt, Sara
Cadogan, Tom
War Child UK
War Child UK
Restless Development
Zambia, Head and
Lusaka Offices
Chaila, Mr
Keembe Community,
Chibombo District
Chamasonde,
Keembe Community,
Chisoko
Chibombo District
Chibomba, Cooper Restless Development
Zambia, Head Office
Chilala, Wilfred
Chibombo District
Children with and
Paipir Primary School,
without disabilities Pader
Senior Grants Manager
Deputy Director of Corporate and Donor Fundraising
Country Director
District Guidance and Counselling Officer
Boarding pupils
10/08/2012
14.08.2012
04.07.2012
15.08.2012
24.07.2012
10/08/2012
09/08/2012
01.07.2012
Page 65 of 131
Name, first name
Chishimba,
Bernard
Chuna Moses
Kapoloni
Daniel
Dengel, Mariko
Devenport,
Andrew
Dickens, Okello
Edward, Eko
Etii, Tom
Forsgate, Victoria
Francis, Ed
Gale, Helen
Goodey, Beth
Greenhalf, Jessica
Group of girls and
young women
Group of pupils
Handia, Caroline
Harris, Jessica
Organisation
Role
Type of contact
Date
Keembe Community,
Chibombo District
Napak District
Group member
10/08/2012
Assistant Chief Administrative Officer
disabilities, mixed group
without disabilities)
Interview, and facilitator for
other interviews
Interview
War Child UK
Rupa Parish, Rupa subcounty
YBI
Volunteer
Community group member
Conversation
Interview
10.07.2012
25.07.2012
CEO
Conversations
Paipir Primary School,
Pader
Moroto District
War Child UK, roaming
Teacher
Interview
14.6.2012 and
03.08.2012
01.07.2012
Assistant Chief Administrative Officer
District Coordinator
23.07.2012
31.07.2012
Restless Development
UK
Restless Development
UK
YBI
International Programmes Manager
Interview
Focus group discussion
and debriefing
Face to face
Consultant Director
Face to face
Many
Executive Director, Research and Policy
Conversations
Training and Capacity Building Manager
Face to face
14.06.2012
and
09.10.2012
04.07.2012
Country Director
Group discussion and
several conversations
Focus group discussion
and garden demonstration
Focus group discussion
and garden demonstration
Interview
Face to face
Restless Development
UK
Restless Development,
Jinja office
Community Members,
Panyangara and Loletyo
Panyangara primary
school, classes P4-7
Zanaco Bank
Restless Development
Beneficiaries of War Child’s out-of-school Junior
Farmer Field and Live School
Orphaned pupils, and beneficiaries of War Child’s inschool Junior Farmer Field and Live School
Manager Social Responsibility
Director Investments and Partnerships
24.07.2012
04.07.2012
20-21.07.2012
28.07.2012
28.07.2012
08/08/2012
04.07.2012
Page 66 of 131
Name, first name
Role
Type of contact
Date
Chief Executive
Face to face
03.07.2012
M&E Manager, London office (on mission in
Uganda)
Chair of Trustees
Travel companion,
conversations
Face to face
28.0701.08.2012
03.07.2012
Trusts Fundraiser
Project Officer
10.07.2012
31.07.2012
Community Development Officer
Conversation
Focus group discussion
and debriefing
Interview
Senior Programme Manager U-Report
Volunteers
Job observation
Group discussion
20.07.2012
25.07.2012
Restless Development
UK
UNFPA, Kampala
Senior Manager Investments and Partnerships
Face to face
04.07.2012
Programme Assistant
Interview
20.07.2012
Programme Coordinator
Several conversations
26.07.2012
Potential KYBT beneficiary
Telephone interview
Interview
15.08.2012
20.07.2012
Kamau, Evan
Restless Development,
Moroto office
n/a
Ministry of Gender,
Labour and Social
Development
KYBT
13.08.2012
Kamugisha,
Moses
Restless Development,
Jinja office
Finance and Admin Manager
Interview and going
through some of the books
Group discussion and
Interview
Hartley, Nik
Hassnaisn, Hur
Hayman, Martin
Holliday, Martin
Imuret, Regina
Irina, Negaga
James Powel
Jeremy, Lomer
Francis and
Lokure, David
Dengel
Jones, Gemma
Juliet, Kwabano
Mujuni
Juliet, Nakazibwe
Jumu, Laila Rajab
Kaboggoza, James
Ssembatya
Organisation
UK
Restless Development
UK
War Child UK, London
office
Restless Development
UK
War Child UK
War Child UK, Kaabong
office
Department of
Community
Development Kaabong
Town Council
UNICEF, Kampala
Lorukumo Village, Rupa
sub-county
Assistant Commissioner, Youth and Children Department
Accountant
30.07.2012
21.07.2012
Page 67 of 131
Name, first name
Organisation
Role
Type of contact
Date
Kantolo, Howard
Karatziola, Tiffany
Deputy Head
Programme Coordinator
Interview
Interview
09/08/2012
10/08/2012
Kavivi, Faith
Chilata Basic School
Restless Development
Zambia, Head Office
KYBT
Field Officer
13.08.2012
Kekana, Thabo
Kendi, Priscilla
KYBT
Ex-volunteer
Field Officer
Interview and help with
meetings with beneficiaries
Focus group discussion
Interview and help with
meetings with beneficiaries
Kiplagat, Dr Ken
Kironde, Brian
Kitara, Christopher
Kolibi, Susan
Kunda, Chola
Kuno, Kapelikori
Kuvuva, Mary
Kwambwa,
Miyanda
Ledward, Kate
Livingstone
Kamugisha
Lobur, Choriba
Lodungu, John
Adiaka, Paul and
Agan, Mary
Logiel, Anna
10/08/2012
13.08.2012
Okoth &Kiplagat
Advocates
UNFPA, Kampala
Member, KYBT board of trustees
15.08.2012
National Programme Officer, Adolescent Reproductive
Health
Interview
20.07.2012
War Child UK, Pader
office
Musupo Parish,
Kaitkekile
Restless Development
Zambia, Head Office
Rupa Parish, Rupa subcounty
n/a
Embassy of Ireland
Operation Assistant
Debriefing and discussion
01.07.2012
Restless Development
Interview
25.07.2012
Monitoring and Evaluation Coordinator
Reporting trail verification
10/08/2012
Community group member
Interview
25.07.2012
Potential KYBT beneficiary
Senior Education Advisor
Telephone interview
Interview, with CD present
15.08.2012
08/08/2012
Restless Development
UK
War Child UK, roaming
Senior Finance Manager
Face to face
04.07.2012
M&E Advisor
27-30.07.2012
Rupa Parish, Rupa subcounty
Napak District
Community group member
Travel companion,
conversations
Interview
Community Services Department
Group discussion
24.07.2012
Association for
Head of Office
Interview
25.07.2012
25.07.2012
Page 68 of 131
Name, first name
Lokello, Joseph
Lokora, Paul
Longoli, Catarina
Longoli, Jennifer
Lopusimoe, Elia
Lorika, Moses
Lotak, Paul,
Lotee, Maria,
Angella, Sarah,
Lotee, Anna,
Kolibi, Christine,
Lotiang, Moses,
Loukon, Mark and
Moru, Mary
Lunga, Thomas
Makina, Veronica
Mark Joseph
Mark Longole
Mark, Simon
Organisation
Traditional Healers and
Health
Panyangara primary
school, class P4
Narube REFLECT
Group, Kathile Sub
County
Community Member,
Panyangara and Loletyo
Moroto District
Community Member,
Panyangara and Loletyo
Torurukae Savings and
loan association
Kalopajak youth group,
Lokuwas Parish Matany
sub-county
Office of the Member of
Parliament
Christian College
Restless Development,
Moroto office
Restless Development,
Moroto office
Kaabong Town Council
Role
Type of contact
Date
Pupil and, strangely, beneficiary of War Child’s outof-school Junior Farmer Field and Live School
Facilitator
Interview by Hur
28.07.2012
Short interview
30.07.2012
Beneficiary of War Child’s out-of-school Junior
Farmer Field and Live School
Community Development Officer Youth
Community Facilitator, War Child UK in Uganda
Interview and interpretation
services
Interview
Interview
28.07.2012
Second loan circle- Chairperson
Interview
23.07.2012
Community members
Group discussion
25.07.2012
Head of Office
Interview
10/08/2012
Member of HIV Committee
Finance Officer
Focus group discussion
Interview
09/08/2012
23-26.07.2012
Assistant Programme Coordinator
Several conversations
23-26.07.2012
Responsible for child protection
He showed up at a War
Child meeting to show its
logbook – and participated
in War Child staff meeting
30.07.2012
23.07.2012
28.07.2012
Page 69 of 131
Name, first name
Organisation
Role
Type of contact
Date
Maubo, Desmond
Restless Development
Zambia, Head Office
Restless Development
UK
n/a
Department of
Community
Development, Kaabong
Town Council
Ministry of Education,
Science and Vocational
Training
Restless Development
Zambia, Head Office
Restless Development
UK
Panyangara primary
school, class P5
KPMG
Assistant Programme Coordinator
Planning meeting
10/08/2012
Marketing and Communications Manager
Face to face
04.07.2012
Potential KYBT beneficiary
Assistant CDO
Telephone interview
Interview
15.08.2012
30.07.2012
HIV/Aids Coordinator
Interview
08/08/2012
Finance and Administration Manager
Budget and audit
discussion
Face to face
10/08/2012
Interview by Hur
28.07.2012
12.07.2012
Muronzi, Farai
Restless Development,
Jinja office
Senior Manager
Telephone interview,
together with Richard
Bennison
Group discussion and
several conversations
Mwa, David
War Child UK, Pader
office
Restless Development
Zambia, Lusaka Office
Restless Development
Zambia, Head Office
M&E Officer
Debriefing and discussion
21.07.2012
and
03.08.2012
01.07.2012
Programme Coordinator
Interview
11/08/2012
Senior Manager (until July 2012)
9-10/08/2012
Napak District
Private Business
Education Officer
Entrepreneur
Long conversations;
accompanied me for two
days
Interview
Interview
Maurer, Susanne
Miheso, Vincent
Moses, Lochokio
Mukonka, Remmy
Mukonkela,
Mukonki
Munford, Robyn
Munyes, Joseph
Murphy, Roisin
Mwape, Ben
Mwiinga, Harriet
Nakoya, Joyce
Namongo,
Director Programme Quality
Pupil and, strangely, beneficiary of War Child’s outof-school Junior Farmer Field and Live School
Senior Manager, Corporate Social Responsibility
Many
24.07.2012
30.07.2012
Page 70 of 131
Name, first name
Organisation
Role
Type of contact
Date
Paipir Primary School,
Pader
Restless Development,
Jinja office
KYBT
Head Teacher
Interview
01.07.2012
Advocacy and Campaigns Manager
21-27.07.2012
Newell, Natalie
Restless Development,
Jinja office
Programme Quality and Learning Manager
Group discussion and
several conversations
Presentation of the
Salesforce OMS
Group discussion,
interview and
documentation review
Njoroge, Henry
Nyaga, Alex
Xtranet technologies
Parapet group of
companies
FAO
Matany Sub-County
Chairman, KYBT board of trustees
Member, KYBT board of trustees
Programme Officer
Teacher, responsible for life skills
Interview
Group discussion
30.07.2012
24.07.2012
War Child UK, Kaabong
satellite office
Project Officer
30.07.2012
War Child UK, Kaabong
office
KYBT
Project Officer
War Child UK, Pader
office
Department Probation
Driver
Interview and guide to
REFLECT community
group
Focus group discussion
and debriefing
Long conversations and
help with meetings with
KYBT mentors and
beneficiaries
Interview
CDO.Probation Officer
Interview
31.07.2012
War Child UK, Pader
office
Association for
Traditional Healers and
Finance, Admin and Logistics Assistant
Debriefing and discussion
01.07.2012
Programme Officer
Interview
25.07.2012
Madalena
Naphy, Bernard
Naula, Mariam
Ndururi, Richard
Ocan, Godfrey
Ochen, Chilla
Charles
Odongo, Teddy
Odur, Patrick
Ogolo, Eunice
Ogwang, Ronald
Ogwaria,
Laurence
Okello, Laurence
Okengo, Francis
Intern IT
Project Manager
13.08.2012
21.07.2012
and
03.08.2012
15.08.2012
15.08.2012
31.07.2012
13.08.2012
01.07.2012
Page 71 of 131
Name, first name
Okongo, Mr
Olanya, Francis
Olupot, James
Peter
Omae, Everline
Omondi, Philip
Ondeko, Roselidah
Ongweso,
Modesta
Opar, Elector
Opio, Anthony
Opira, Joseph
Ord, Helen
Otema, Geoffrey
Oteng’, John
Otyang, Paul
Oyello, Bob
Patel, Amisha
Petersen, Ib
Ploeg, Sietske van
der
Restless
Organisation
Health
Panyangara primary
school
War Child UK, Kotido
office
Role
Type of contact
Date
War Child’s in-school facilitator
Interview and review of
documentation
Planning support and
travel companion,
conversations
Interview
28.07.2012
Telephone interview
Telephone interview
Interview
Interview and help with
meetings with KYBT
mentors
15.08.2012
15.08.2012
20.07.2012
13-15.08.2012
Livelihood Senior Project Officer, War Child UK in
Uganda
Karamoja Integrated
Development
Programme (KIDEP)
n/a
n/a
UNFPA, Kampala
KYBT
Programme Manager
Ministry of youth affairs
War Child UK, Pader
office
War Child UK, Kaabong
office
War Child UK
War Child UK, Pader
office
Ministry of youth affairs
Rupa Parish, Rupa subcounty
Matany Sub-County
Restless Development
UK
Danida
YBI
Member, KYBT loan selection panel
Project Officer
Restless Development
Potential KYBT beneficiary
Potential KYBT beneficiary
Senior GBV Coordinator
Training and Mentoring Manager
Finance Officer
Debriefing and discussion
27.07.2012
23.07.2012
14.08.2012
01.07.2012
Director of Finance, HR and Administration
Finance Assistant
Focus group discussion
and debriefing
Conversation
Debriefing and discussion
31.07.2012
KYBT volunteer mentor
Community group member
Interview
14.08.2012
25.07.2012
Headmaster
Trustee (and before this Restless’ contact person at
DFID)
State Secretary
Programme Manager
Group discussion
Face to face
24.07.2012
03.07.2012
Telephone interview
Conversations
29.08.2012
Several dates
Several
Debriefing and discussion
10/08/2012
09-10.07.2012
01.07.2012
Page 72 of 131
Name, first name
Organisation
Development
Zambia staff
Rooney, Maarten
Saidy, Ebrima
Zambia, Head Office
Sayo, Mairo Jairo
Sianga, John
Stevens,
Catherine
Teko, John
Temo, Evalyn and
Amei, Santina
Tshimba
Tunabaitamu,
Tamara
Walterfang, Gerald
Wambua,
Katherine
Wangeci, Lydiah
Wanjohi, Michael
Waria
Weisbaum,
Amanda
Williams, Rob
Williams, Sophie
Wina Wina
York, Laura
Role
Type of contact
Date
YBI
War Child UK, Kampala
and roaming
Finance Director and COO
Country Director
Conversations
Travel companion,
conversations
n/a
Matuka Community
Health Centre, Kapiri
Mposhi District
Restless Development
UK
Matany Sub-County
Matany Parish
Potential KYBT beneficiary
Health Officer
Telephone interview
Interview
Several dates
20.07.2012
and 30.0701.08.2012
15.08.2012
09/08/2012
Director People and Performance
Face to face
04.07.2012
Sub-County Chief
Restless Development
Group discussion
Interview
24.07.2012
24.07.2012
Christian College
Restless Development,
Jinja office
Viwango
Peer Educator
Fundraising and Coordination Coordinator
09/08/2012
21.07.2012
n/a
KYBT beneficiary
Faulo
n/a
Christian College
War Child UK
KYBT mentor
Potential KYBT beneficiary
Peer Educator
Programmes Director
Focus group discussion
Group discussion and
interview
Visit to work place and
interview
Visit to premise and
interview
Interview
Interview
Focus group discussion
Conversation
War Child UK
War Child UK
Microcredit bank
War Child UK
CEO
Programme Development Coordinator
Head of Office
Programmes Finance Manager
Conversation
Conversation
Interview
Conversation
09.07.2012
10.07.2012
10/08/2012
09.07.2012
KYBT Mentor
14.08.2012
14.08.2012
14.08.2012
14.08.2012
09/08/2012
09.07.2012
Page 73 of 131
Annex E: List of data sources
The only documents I have consulted and have not listed in the bibliography are the various
number sheets related to the number trailing (as explained in Annex C3).
Annex F: Bibliography
Accenture, 2012. PPA Consortium Joint Intervention Feasibility Project: Final Deliverable,
Restless Development, War Child, YBI
Consortium, 5 July 2012. The consortium’s response to the PPA Annual Report Feedback
Letter.
ADP GAP AsIs Survey Questions. Undated.
ADP GAP GMG Recommendations, 2012
Aston University/University of Strathclyde: What Capacity Development Services are Most
Effective in Improving Young Entrepreneurs’ Business Performance and Building their
Human Capital: a Literature Review, 2012. Final report
Bond, 2012. M&E: How to Guide. Louisa Gosling
Bond, 2012. Value for Money: What it means for UK NGOs.
Chinyama, MC, 2012. Request for Permission – Invitation for a National HIV Specialist to
Become Restless Development Board Member. Ministry of Education, Science and
Vocational Training, Zambia
Committee on Labour, Youth and Sport. 2011. Report for the First Session of the Eleventh
National Assembly, appointed on 21 October 2011, National Assembly of Zambia, Zambia
Consolidated picture: Percentage achievements as goals of consortium, 2011
Consortium, 2011. Memorandum of Understanding between Restless Development, War
Child and Youth Business International
Consortium, 2012. PQD_PPA_11-12 status_UG & ZAM_14May12 (this lists the numbers of
all consortium members in all areas of the PPA logframe). Unpublished, UK
Consortium: consolidated picture 2012
Denison J, Torpey K, Tsui S and Bratt J, 2009. Evaluation of the Students Partnership
Worldwide’s (SPW) School HIV/AIDS Education Program (SHEP); final report, Family Health
International for SPW (which was the previous name of Restless Development), Zambia
DFID, 2009. How to note: Guidance on using the revised Logical Framework, DFID, London
DFID, 2011/2012. CSO Youth Working Group Consultancy. Final Report Practical
Participation
DFID, 2012. PPA Annual Report Feedback Letter.
Page 74 of 131
DFID, 2012. Project Completion Tool, Zambia
DFID/Restless Development, 2012. Memorandum of Understanding between DFID and
Restless Development
Empowerment and Livelihood for Adolescents, 2011. Impact Report, Uganda.
Executive Summary and Recommendations: What evidence is there to suggest that
engaging young people in development enhances or limits development outcomes across
different contexts and in different geographical locations?
Foreign & Commonwealth Office; Wilton Park, 2012. Protecting children affected by armed
conflict: advancing the agenda of the last 10 years
Gaps in Research Themes
Independent Commission for Aid Impact, 2011. Approach to effectiveness and value for
money.
International Aid Transparency Initiative (IATA), 2008. ACCRA statement.
Kaabong Town Council, undated. OVC Community Mapping Report for Komuria East Ward.
Unpublished, Kaabong
Kaabong Town Council, undated. OVC Community Mapping Report for Kaabong Central
Ward. Unpublished, Kaabong
Kasuta, ET, Zulu J, Chilwalo M and Sichone G,2012. Improving the Capacity of Young
People to Engage in Decision-Making and Lead Comprehensive SRH Education in Zambia;
external evaluation. Restless Development, Zambia
Kenya Youth Business Trust, 2011. Accreditation letter. Kenya.
Kenya Youth Business Trust, 2011. Assessment Report. Kenya
Kenya Youth Business Trust, 2011. Progress Report for Accreditation Committee Meeting in
November 2011. Unpublished. Kenya
Kenya Youth Business Trust, 2011. Progress report for the Accreditation Committee
Meeting.
Kenya Youth Business Trust, 2011. YBI Interim Report, Kenya Blitz Phase 1. Routes
Business Cartographers Ltd. Kenya
Kenya Youth Business Trust, 2012. Dorine Achieng case study. Kenya
Kenya Youth Business Trust, 2012. Progress Report for Accreditation Committee Meeting in
April 2012. Unpublished. Keyna
Kenya Youth Business Trust, 2012. Progress Report. Unpublished. Kenya
Kenya Youth Business Trust, undated. Comic Relief Entrepreneur Swimlane Process flow
chart. Kenya
Page 75 of 131
Kenya Youth Business Trust, undated. IDB Mentor Swimlane Process flow chart. Kenya
Kizito, A, 2012. Strength, Creativity and Livelihoods of Karimojong Youth. Restless
Development Uganda, Pastoralist Communication Initiative and the Institute of Development
Studies, Jinja
KPMG, 2011. Pre-Grant Due Diligence Assessment
Making Cents: snapshot including links
Ministry of Education Zambia, 2012. Memorandum of Understanding between the
Government of the Republic of Zambia and Restless Development, concerning the
implementation of the Restless Development programme in Zambia. Ministry of Education
Zambia, Lusaka
Ministry of Education, Republic of Zambia, 2012. Speech by the Permanent Secretary of the
Ministry of Education on the teenage pregnancy information dissemination workshop.
Unpublished. Zambia.
Ministry of Education, Science and Vocational Training, undated. Monitoring Instrument for
Implementation of HIV and AIDS and Lifeskills Activities in Schools. Unpublished draft
document, Zambia
ODI, 2011.Maximising impact of youth entrepreneurship support in different contexts, Karen
Ellis and Carolin Williams
ODI, 2012. Toolkit: Maximising impact of youth entrepreneurship support in different
contexts. Karen Ellis and Carolin Williams
OECD, 2011. Overview of LDCs: DAC List of ODA Recipients.
Potential collaborative work
PPA Business Case and Summary, 2011. PPA with the consortium of Restless
Development, War Child and YBI
PPA Consortium Steering Committee Agenda, 2012
PPA Consortium Steering Committee minutes, 2012
PPA Consortium, 2011. Best practice
PPA Consortium: Youth Engagement in Development: Evidence of engagement of youth
links to development outcomes
PPA output indicator table
PPA Submission, 2011. Annex A
PPA Submission, 2011. Annex B
PPA Submission, 2011. Section 1 narrative
PPA Working Group minutes, 2012
Page 76 of 131
PPA, 2011. Logical Framework: consortium
PQD – PPQ Monitoring Tool, 2011. Q2
Restless Development and Institute of Economic and Social Research, University of Zambia,
2011. External evaluation of the Teacher AIDS Action Programme: A case study of southern
and central provinces. Zambia
Restless Development and Youth Business International, 2011. Sub-Grant Agreement
Restless Development, 2008. CSCF Proposal Narrative, Uganda.
Restless Development, 2008-2012. Efficiency Study Data Capture. Unpublished. Zambia
Restless Development, 2011. Budget Restless Development Zambia. Unpublished, Zambia
Restless Development, 2011. Case Study, Uganda.
Restless Development, 2011. Case Study, Zambia
Restless Development, 2011. Coffey Draft Evaluation Strategy Consultation Workshop for
PPA Grantees. UK
Restless Development, 2011. Consortium meeting agenda. UK
Restless Development, 2011. Consortium Working, DFID PPA slides. UK
Restless Development, 2011. CSCF Annual Report, Uganda.
Restless Development, 2011. CSCF Project Completion Report (PCR). Unpublished,
London
Restless Development, 2011. Finance and Administration Unit Operational Plan 2011-12.
unpublished, Zambia
Restless Development, 2011. Fundraising Strategy, Uganda
Restless Development, 2011. Good Practice Manual on HIV and AIDS awareness raising
and impact. Restless Development: in partnership with Irish Aid, Lusaka
Restless Development, 2011. IP case study.
Restless Development, 2011. Overall Vision for PQ Directorate slides.
Restless Development, 2011. Overview of draft evaluation strategy: DFID PPA recipients.
UK
Restless Development, 2011. PPA Working Group – Minutes and Action Points
Restless Development, 2011. PPA Working Group – Proposal Finalisation Minutes
Restless Development, 2011. Prioritising Proposal, WFP NUSAF
Restless Development, 2011. Report on YEP, Uganda
Page 77 of 131
Restless Development, 2011. Tactic and Key Performance Indicator spreadsheet
Restless Development, 2011. Zanaco Annual Report. Restless Development and Zanaco,
Zambia
Restless Development, 2011/2012. Donor budget. Unpublished. Zambia
Restless Development, 2011: Global Strategy 2011-2015.
Restless Development, 2012. 2012 Internal Audit Report Uganda. Unpublished, Uganda
Restless Development, 2012. 2012 Internal Audit Report Zambia. Unpublished, Zambia
Restless Development, 2012. Additionality Report Table, Appendix.
Restless Development, 2012. Additionality Report, Part F
Restless Development, 2012. Additionality Report, Part G: Changing Lives Best Case Study,
Zambia
Restless Development, 2012. Additionality Report, Part G: Changing Lives Typical Case
Study, Tanzania
Restless Development, 2012. Additionality Report, Part G: Changing Lives Worst Case
Study, Zimbabwe
Restless Development, 2012. Advocacy & Influence presentation.
Restless Development, 2012. Anti-Bribery and Corruption Policy. Unpublished, Zambia
Restless Development, 2012. Anti-Bribery and Corruption Policy. Unpublished. Zambia
Restless Development, 2012. CSCF Project Completion Report DFID (PCR). Unpublished.
Zambia
Restless Development, 2012. Efficiency Study, Unpublished. Uganda and Zambia
Restless Development, 2012. Efficiency Study, Uganda, Unpublished
Restless Development, 2012. Efficiency Study, Zambia, Unpublished
Restless Development, 2012. Evaluation of the Youth Accountability Model for Enhancing
Access to Youth Friendly Health Services; a case study of 4 pilot communities (Keembe,
Kafulamase, Matuka and Mindolo) in the Central and Copperbelt Provinces of Zambia.
Restless Development and the Zambia Governance Foundation, Zambia
Restless Development, 2012. Feedback from Pamela. Zambia
Restless Development, 2012. Feedback to country programmes. Unpublished. Zambia
Restless Development, 2012. Finance and Administration Unit Operational Plan 2011-12.
Unpublished. Zambia
Page 78 of 131
Restless Development, 2012. Finance and Administrative Procedures: Recent Changes.
Unpublished. Zambia
Restless Development, 2012. Global marketing communications strategy 2012-2015. Slides.
Restless Development, 2012. Launching the New National Youth Policy Together.
Restless Development, 2012. Log Frame. Unpublished. Zambia
Restless Development, 2012. Performance Management, self review of Tendai Chiweshe.
Zambia
Restless Development, 2012. PPA Annual Report
Restless Development, 2012. PPA Risk Register
Restless Development, 2012. PQD PPA 2011/12 Status. Unpublished. Uganda / Zambia
Restless Development, 2012. Programme Staff Finance Handbook 2012/13. Unpublished,
Zambia
Restless Development, 2012. Programme Staff Finance Handbook. Unpublished. Zambia
Restless Development, 2012. Recommendation Tracker (this monitors progress against the
2012 internal audit report). Unpublished, Zambia
Restless Development, 2012. Restless Development Zambia Performance Management.
Unpublished, Zambia
Restless Development, 2012. Strategic Fundraising Assessment Tool, Uganda
Restless Development, 2012. Submission the Parliamentary Sessional Committee for
Labour, Youth and Sport for the First Session of the Eleventh National Assembly of Zambia.
Restless Development, Zambia
Restless Development, 2012. Template for partners
Restless Development, 2012. TIKAMBE Programme Model for Civic Participation and
Shaping Policy and Practice. Zambia
Restless Development, 2012. Volunteer Finance Handbook 2012/13. Unpublished, Zambia
Restless Development, 2012. Volunteer Finance Handbook. Unpublished. Zambia
Restless Development, 2012. Zambia Fundraising Strategy and figures. Unpublished,
Zambia
Restless Development, undated. Recommendation Tracker. Unpublished. Zambia
Restless Development, undated. Reporting Framework 1112. Unpublished, Uganda
Restless Development/Unicef, undated. Teenage Pregnancy Desk Review in Zambia.
Unpublished. Zambia
Page 79 of 131
Restless Development/War Child, 2011. PPA due diligence follow up discussion notes
Restless Development/YBI, 2011. Final Due Diligence Report.
Restless Development/YBI, 2011. PPA due diligence follow up discussion notes
Save the Children, Emergency Communication Toolkit
The “Most Significant Change” (MSC) Technique: A Guide to its Use. Rick Davies and Jess
Dart
UN, 2011. The Millennium Development Goals Report.
UNFPA, 2012. Progress Report, Jan-Mar. Uganda
UNICEF (Ahmed Shaban Adbulrahim), undated. The Youth Leadership, Communication and
Advocacy: Training Guide for the Uganda Youth. Uganda.
UNICEF report 2012. Template of the PCA Guide: Programme Cooperation Agreement with
(War Child UK): Template for programmatic reporting
U-Report, 2012. The views of youth & education about sex in schools (poll summary).
Uganda
VITOL Charitable Foundation, 2012. Interim and Final Reporting guidelines
War Child, 2011. Provision of Emergency Services for Sexual Gender Based Violence
(SGBV) Prevention & Response in Kaabong District, North Eastern Uganda
War Child UK, 2011. PPA Monitoring tool. WCUK Q4 Report
War Child, 2011. CSF Q1 Report
War Child, 2011. OVC Quarterly Reporting Format
War Child, 2011. UNFPA Financial Report, Uganda
War Child, 2011-2015. Uganda Country Strategy. Uganda
War Child, 2012. Organogram.
War Child, 2012. PPA donor reporting budget – year 2
War Child, 2012. Q3 Progress Report
War Child, 2012. Uganda Programmes Outline. Uganda
War Child, 2012. Uganda Programmes Overview, June. Uganda
War Child, 2012. UNFPA GBV Year 2 Q1, Uganda
War Child, 2012. UNFPA Work Plan Monitoring Tool Year 2, Q2. Uganda
War Child, 2012. VFM Brainstorm.
Page 80 of 131
War Child, undated. Established child helplines in different countries
YBI (The Prince’s YBI), Undated. Introducing Performance & Learning. Unpublished.
YBI (The Prince’s YBI), Undated. Operational Overview. Unpublished. YBI with 2 third party
experts: London School of Economics and Innovation for Poverty Action (IPA). Also advised
by the World Bank.
YBI, 2010. Annual Report
YBI, 2011. Accreditation Committee Decisions. Unpublished.
YBI, 2011. Accreditation Framework Appendix 2.
YBI, 2011. Accreditation Process: Guidelines for YBI Members.
YBI, 2011. Assessment Report February: Kenya Youth Business Trust. Unpublished. Kenya
YBI, 2011. Members’ Manual. Unpublished. Kenya
YBI, 2011. Memorandum of Understanding Annex 1 (accredited). Unpublished. Kenya
YBI, 2011. Memorandum of Understanding plus annex outlining the terms of member status.
Unpublished. Kenya
YBI, 2011. Milestones spreadsheet. Kenya
YBI, 2011. Strategic Plan 2011 – 2014.
YBI, 2011. Youth entrepreneurship: Closing the gap
YBI, 2012. Accenture grant prioritization. GAP GMG recommendation
YBI, 2012. Accenture Grant: member briefing
YBI, 2012. Growth analysis: Benchmarking report. Unpublished. Kenya
YBI, 2012. Network team organisational chart.
YBI, 2012. Operating model
YBI, 2012. Presentation at GEM annual meeting: NES questions
YBI, 2012. Process Overview: Identify to Accredit slide.
YBI, 2012. Process Overview: Resource to Report slide.
YBI, 2012. The IPR’s initial overview of findings.
YBI, Rating Members slides.
YBI, undated. PPA monitoring report Q2
YBI. 2012. Mentoring Toolkit. The Prince’s Charities.
YBI. Q2 2012. Growth Report. The Prince’s Charities.
Page 81 of 131
Youth Working Group Consultancy, 2011/12. Practical Participation, Alex Farrow, Tim
Davies
Youth Working Group Meeting, minutes. 2012. Uganda
Youth Working Group, Membership Details 2012.
Page 82 of 131
Annex G1: Country programme report 1: Restless Development,
Uganda
Final version, 24 August 2012
Contents
Context .............................................................................................................................. 84
Assignment ....................................................................................................................... 84
Results ............................................................................................................................... 84
Performance assessment against logframe ..................................................................... 84
Effects on poor and marginalised groups and civil society ............................................... 85
Relevance .......................................................................................................................... 86
Representativeness ......................................................................................................... 86
Targeting strategy ........................................................................................................... 88
Effectiveness ..................................................................................................................... 89
Learning .......................................................................................................................... 89
Learning that improves Restless Development’s own capacity .................................... 89
Learning that provides contextual knowledge ............................................................... 90
Learning that can be shared with others ...................................................................... 90
Innovation ........................................................................................................................ 90
Partnership working ......................................................................................................... 91
Partnering with the government ................................................................................... 91
Inter-agency cooperation ............................................................................................. 92
Sustainability ................................................................................................................... 92
Efficiency and Value for Money assessment .................................................................. 93
Impact and value for money of PPA funding .................................................................. 95
Attributable impact of PPA funding on results, relevance, effectiveness and efficiency.... 95
Value for money assessment of PPA funding .................................................................. 95
Page 83 of 131
Context
Restless Development (‘Restless’, from now on) follows different models in different regions.
This report only covers the Restless operations in the Moroto and Napak districts of Uganda,
and its findings may not be relevant for Restless operations elsewhere.
In Moroto and Napak, Restless’ main programme employs groups of three ‘community
volunteers,’ who are situated in their own community or in a community nearby. In the
course of some six months these trained volunteers roll out a life skills awareness and peace
building programme to communities that are typically poor and relatively isolated. In addition,
Restless is implementing a capacity building programme for civil society organisations
(CSOs).
Assignment
This assessment is part of a PPA mid-term evaluation that covers the work of Restless, War
Child and Youth Business International. The assessment took six days, excluding travelling
time.
I first spent a day in Kampala, with the Country Director and some of her contacts in UNICEF
and UNFPA. I then visited Restless’ main office in Jinja, and its office in Moroto. I met with
children, youths and parents; district, sub-county and parish authorities; school staff;
Restless staff and facilitators; and a few CSOs. There were formal interviews and informal
conversations with individuals and groups; lots of documents; and a bit of observation (i.e. a
weekly staff meeting and a volunteer’s monthly reporting session). I would like to thank all
the many people who have helped me understand Restless’ work in Uganda – and
particularly Mariam and Jessica, for the many patient hours of conversation during our
travels.
This report follows the standard IPR structure.
Results
Performance assessment against logframe
Impact level. The impact indicators are related to national prevalence rates of HIV, and
beyond the control of Restless Development.
Outcome levels. It is not yet possible to assess whether the outcome targets will be
achieved.
Output levels. The consortium states, in the annual review, that the various outputs have
been achieved. I tested the number aggregation by tracking the reporting on a single
individual, who I had met in a randomly selected manyata. This exercise suggests that the
reported numbers are correct 71 as:
•
•
Her name was included only once in the beneficiary list that covered her manyata.
This list was included only once in the aggregate northern Ugandan figures.
71
In the same exercise I did see an example of double-counting, of a beneficiary who was counted twice
because she participated in two livelihoods groups. This is not common. I also note that the figures are based on
initial groups, and do not account for additions to or departure from these groups. This is acceptable, as the
additional detail that weekly tracking would provide would not be worth the additional hassle of such tracking.
Page 84 of 131
•
•
This aggregate number was included only once in the national figure of Uganda.
The individual remained a single person, categorised correctly as an ‘out of school
female under 24’.
Once back in the UK, I would like to verify that the Ugandan numbers are correctly
aggregated into worldwide numbers. If this is the case, I conclude that Restless’ part of the
logframe figures is reported on correctly. However, I also note that the logframe numbers do
not match all numbers that appear in the Annual Review narrative. Specifically: the Ugandan
numbers (and common sense) suggests that it is an exaggeration to say that “approximately
1,000 national Volunteer Development Professionals reach […] 400,000 young people on a
weekly basis.” (Annual Review, page 22.)
Recommendation 1
In Uganda, Restless Development tags its beneficiaries appropriately (e.g. ‘under-24 out-ofschool female’) and maintains rigor throughout its aggregation process from individual
beneficiaries to country-wide figures. At the worldwide consortium level, these figures feed
into:
•
A logframe that would be stronger if its impact figures are explicitly linked to target
groups; and
•
An Annual Review narrative that generally reports correctly but sometimes presents
figures that do not seem to be based on Restless’ rigorous beneficiary aggregation
process.
Recommendation 2
Each month’s reporting cycle starts with a conversation between each group of volunteers
and a Restless staff member. Because of the limited transport options, many of the
Karimojong volunteers arrive early and may wait for hours. Instead of hours of waiting, these
could be hours of active engagement among peers from different areas – and all that is
needed is a set of unsupervised exercises (e.g. peer coaching, problem-solving techniques,
trying out tools).
Effects on poor and marginalised groups and civil society
Effects on poor and marginalised groups
We are now much more respectful towards one another. Christine, community member,
reflecting on the impact of Restless’ peace building activities.
In the Moroto and Napak districts, behavioural change messages (use latrines / boil water /
get tested / grow vegetables) come from many different authorities and organisations.
Restless’ volunteers build on these messages by repeating them and initiating action (let’s
build them / boil it / get tested – I’ll go first / start a garden – I’ll provide the seeds). From
conversations with community members I got the impression that these volunteers often
manage to move these messages from passive knowledge to active behavioural change,
and that these messages cover a wide range of issues. 72 The volunteers’ success lies in
72
The issues that community members mentioned were in the fields of sanitation, SRH, GBV, education,
vegetable-growing, saving, conflict resolution (‘it is quieter now, in this manyata’) and peace consolidation. Note
that:
•
This speedy assignment does not allow for a quantitative assessment that includes surveying the action
of neighbouring manyatas to confirm likely attribution.
Page 85 of 131
their ongoing onsite presence: Restless Development’s community volunteer concept is the
very opposite of the much more common ‘dump and run’ approach towards behavioural
change.
Effects on the community volunteers
I used to be a drunk. Now I am a Restless volunteer, a different person. Mette, who is about
to finish her second volunteering assignment.
Perhaps even more profound than Restless’ success in effecting behavioural change is its
impact on the volunteers themselves. In the north of Uganda, these volunteers are recruited
locally, and more often than not they dropped out of secondary education. They tend to save
some of their modest allowances and regularly have plans to return to school after the
volunteering has come to an end. They are clearly respected in their respective
communities, 73 play a role model function, and in the course of their volunteering they gain:
•
•
•
•
creativity (“I challenged them: ‘next time come with a friend,’ and now the group is full’”),
skills (“Before this, I could only grow sorghum and maize”),
confidence (“Talking in front of a group? No problem.”) and
prospects (partly because of the previous points and partly because of Restless’ end-ofvolunteering CV and other support). 74
Effects on civil society organisations
Without Restless, we would not be at this
level. Board member of an active lending circle.
Uuuuuhmmmm – I do not remember just
now. Manager of a Restless CSO trainee, asked
what this trainee’s recent four-day training course
had been about.
Restless trains managers and staff of a number of civil society organisations. Some of these
organisations are active, and other ones seem to be largely dormant. Restless has made the
choice not to discard organisations that do not show clear follow up to Restless’ training
activities as “Restless is not an ‘easy wins’ organisation.” 75
This is admirable, but progress is important, and Restless could do more to encourage and
foster such progress.
•
Quite a few community members I talked with said things such as ‘“Restless was the first organisation in
this parish to talk about sexual and reproductive health.” This contradicts with the information I got when
talking with sub-county authorities and the staff in a health clinic. This is probably the consequence of a
social desirability bias.
•
There is a lot that can be changed, but the ultimate bottom line will remain the same for the years to
come: people want to be cattle owners.
73
I saw no evidence of jealousy or indifference.
74
To a much more limited extent, I saw similar evidence when talking with the Chairs of the committees that the
volunteers had formed. “Being the chairwoman makes me proud, and the skills that I learn are useful for myself
and something that I am sharing with others.” A female group member said that it was “good” to have a
chairwoman, and a male group member said it was “not a problem because we are all equal.”
75
“What will Restless do with a partner such as this one?” “Well: we won’t give up easily. Go deep. What are the
real issues? Could it be the board? If so, could we train the board? If it is the manager: how could we help her?
We do not drop organisations so quickly. It would also be unwise because this is a small place, and there are not
that many organisations.”
Page 86 of 131
Recommendation 3
Restless’ CSO programme could learn from the volunteering model. Specifically:
•
•
Restless takes time to prepare the communities in which Restless will place its
volunteers. Similarly, the CSO programme could start with longer visits, capacity-related
check lists and work observation. This would give a better sense of organisational needs.
Currently, training is often provided on the basis of observed capacity gaps, rather than
on the basis of the most pressing capacity gaps. 76
Restless involves the various levels of public authority, schools, health centres and the
community itself in the course of each volunteering cycle. Similarly, the CSO programme
could benefit from systematic engagement with the trainees’ peers, manager and staff.
This would increase the likelihood that the acquired skills are going to be utilised and
passed on, and that the learning is institutional rather than merely individual. Currently,
managers do not always know what their staff was trained on, and vice versa.
Relevance
Representativeness 77
We are who we serve. One of Restless Development’s slogans.
“We listen to them because they are from us – but with education.” Community group
member.
Responding to the needs and priorities of youths affected by conflict in Karamoja. The
profile of every target person I have met squarely fitted within this description. The support
provided is consistently perceived as relevant and useful, by the target group, and the
impact is positive and significant.
Programme suggestions, made by the target group and public authorities, were by and large
related to providing things such as bicycles, umbrellas, ploughs, school rooms, more seeds,
drama kits, and sport equipment. These requests were made less frequently than in
comparable evaluations that I have recently conducted, which suggests that Restless is
relatively good in communicating what it does and does not do. Catering for such requests
may not be wise as it would mean that the image and programme focus of the organisation
would change. Such changes would please the target group in the short run, but render
Restless’ programmes less effective in the long run.
Representing youths affected by conflict. First, the community volunteers do not just
‘represent’ but are Karimojong youths affected by conflict. Second and more widely visible:
one of Restless Development’s pieces of advocacy work is its report on innovative action
research that was conducted and reported on by young Karimojongs with very diverse
backgrounds. Such direct and powerful representation is rare in the international
development sector.
76
I have sent Jessica a book titled ‘Capacity Development in Practice.’ The first chapter is particularly good
reading, in preparation for a strengthened CSO capacity building programme.
77
Coffey’s note on this heading is that it is about “The degree to which the supported civil society organisations
[this refers to Restless] represent and respond to the needs and priorities of their constituencies, (including where
relevant the poorest and most marginalized).”
Page 87 of 131
This report is useful when designing future programmes in the region, but also highly
appealing because of its truly bottom up approach. I, for example, use this report as the
basis for a livelihoods exercise for my MSc students in International Development, and I
have no doubt that others will think of other applications.
The only reservation I have is the proportion of female volunteers: there is an 80-20
imbalance at the moment. Different people have different explanations for this imbalance: 1.
There are not many women who have been sufficiently well educated; 2. Women’s
husbands or fathers do not allow them to volunteer; and 3. They do not accept women who
are pregnant or have small babies. Whatever the reason: this imbalance requires continuous
reflection and efforts.
Representing the organisation. Restless is not a banner-raising and sticker-pasting
organisation. On the road side, the Restless logo does not feature among the many signs of
international organisations, and the logo is not visible in the communities it works with. This
is good, as all these NGO and UN signs waste money and reinforce the image of extreme
donor dependency and lack of ownership of one’s own future. If members of the target group
know Restless Development, it is because of the work the organisation does, because of the
onsite presence of its volunteers, and because its senior team takes the effort to pass by in
person 78 – not because of plaques and billboards.
Targeting strategy
Most organisations invite the elders for peace building, and they assume that these elders
have control over their youths – who are the ones who do the actual raiding. Restless targets
these youths directly. John and Paul, Napak district officials.
Restless volunteers are a means to reach communities, but they also form a target group
themselves. They need to have finished part of their secondary education, and be able to
work in English. Beyond this, Restless recruits on the basis of drive and potential. When I
asked four volunteers what they would be doing if they had not been selected, their answers
illustrated the impact their role has had on them:
•
•
•
•
I’d be drunk somewhere.
I’d be gathering fire wood.
I’d be advocating for peace, and growing vegetables.
I’d be part of a group.
Communities. Restless selects its communities in close consultation with the district and
sub-county authorities. They live in Parishes that are generally not covered, or only covered
in passing, by other organisations. 79 Within these communities, the operational model is
inherently self-targeting: ‘people who have other things to do don’t join. They are not
interested in the possibilities these groups provide.’ 80
78
Juliet corrected the driver when he missed a turn, in a vast area that looked all the same to me. This illustrates
that even senior staff members visit communities. This was also confirmed by the people in this particular
community who asked me, at the end of my visit, to give their regards ‘to Juliet and Mark’ (who is the second
most senior person in the Moroto office).
79
This is generally but not consistently true. I also talked with a woman who had chosen not to join a group as
she was in ‘a different group’ – which turned out to be initiated by another NGO and to have a garden as well.
Other criteria: there needs to be an access road, and telephone coverage.
80
A response from a group member, when asked why others from his manyata had not joined the group.
Page 88 of 131
CSOs. Restless targets CSOs that are small, local, and relevant to youth. Their levels of
functionality differ widely.
Effectiveness
Learning
One of Restless’ five strategic pillars is ‘sharing and learning.’ Restless staff knows this, and
takes it seriously. 81
Learning that improves Restless Development’s own capacity
Restless welcomes improvements. This is partly the consequence of a mind-set and partly
the result of learning systems. These systems include:
•
•
•
•
The staff appraisal processes.
Systematic staff consultation on policies and other things of importance to the
organisational performance. It struck me that even something as dull (sorry Moses!) as
the Finance Manual received feedback from people in non-financial roles. These many
and sometimes time-consuming consultations pay themselves back later: once Restless’
brand / values / strategy / policies are ready, there is a strong sense of ownership and
buy-in. 82
The monthly cycle that starts with a volunteer reporting and coaching session 83 that
revolved around What, Why, So what, What now?, and ends with a verification and
follow up visit. In between this start and end point the information is aggregated upward
and discussed at various levels.
The volunteer mid-term gathering, in which volunteers reflect on and learn from the first
half of their assignments.
There are many small improvements as a consequence of this learning, 84 as well as a few
bigger things such as:
•
•
The development of Restless Development’s brand identity, which was a time-intensive
consultative learning and development process that ultimately led to a more widely
shared brand awareness and ownership than I have seen anywhere else.
The split of the M&E function into an M and an E function, with the former allocated to
the programme team, in order to foster awareness and ownership of progress.
This learning is not just internally, within Uganda: I have also seen examples where
Ugandan feedback was incorporated in Restless’ worldwide affairs. 85
81
The other pillars are Direct delivery, Generation of leadership, Building a strong youth sector, and Shaping
policy and practice. Not all, but most of the staff members I asked about these five pillars could name them.
82
It is not just because there are consultations but because of the nature of the consultations: they are not
boring. The brand development process, for example, was very much driven by the people in the country offices.
It was a staggered process, that went from ‘do you have ideas?’ to ‘these are the 25 highest-scoring ideas’, et
cetera, until the new brand was launched in a presentation, by each CD simultaneously, about what had led to
the brand and what it needed to stand for, followed, at the end, by the actual name and logo. It gave a lot of
ownership. Then there was a feedback round. When Restless Uganda tested it amongst stakeholders it became
clear that the ‘un’ had to be striped through (uninspired), as the previously used box-system did not really work.
83
This is normally managed by the Restless ‘intern’, who is not an intern at all but a paid entry-level employee.
Restless should change this title, as it does not reflect reality.
84
For example: Restless has just introduced a colourful 15-page manual, as part of the volunteer training
programme, in an attempt to ‘make it stick better’; the finance unit lists all monthly tasks on the white board, to
avoid tasks being overlooked.
85
I am not sure if the attribution is exclusive, but understood that the global finance policy has a bribery section
because of Ugandan feedback; and that the strike-through in uninspired etc was a Ugandan suggestion too.
Page 89 of 131
Learning that provides contextual knowledge
When we returned to Moroto, we analysed the material. […] It shows how people’s
livelihoods are progressing. It helps us understand the opportunities that people value and
those that are just things to do because of hunger. From Restless’ action research report, titled
“Strength, Creativity and Livelihoods of Karimojong Youth”, page 42.
Some of the learning in relation to contextual knowledge is gained through Restless
recruitment: many of Restless’ staff members are former volunteers and have direct
knowledge of the contexts in which volunteers operate.
In addition Restless’ previously mentioned action research will soon be replicated in different
districts. Again, the research will be conducted by young people, with different levels of
education but all from the communities in which they conduct their research. I know of no
better way to generate contextual knowledge (see text box).
Learning that can be shared with others
Restless could lead us when we conduct youth-related research. Shaban, UNICEF.
Restless is learning on an ongoing basis and is systematically gathering a wealth of
information, but this information is insufficiently thoroughly analysed and too little of it
reaches other organisations.
This is unsurprising. Historically, Restless is an action-oriented organisation, and it is only
recently that ‘sharing’ is an explicit part of its strategy. The organisation now has to learn
how best to do this. The action research paper shows that Restless has the potential, but
this potential can be more fully utilised. Once Restless has learned to do this, I think that its
volunteering model and its role model function as ‘the professional face of youth’ will inspire
other stakeholders. Once this happens, the impact of Restless’ concepts will no longer be
directly proportional to the growth of Restless Development itself. 86
Innovation
Restless organised a widely attended Youth Conference. This was good stuff: UNFPA had
not been able to do this; the government had not been able to do this; and now Restless
could! UNFPA saw that this was a bandwagon to jump on. Brian Kironde, National Programme
Officer for Adolescent Reproductive Health, UNFPA Kampala.
Restless Development recognises worthwhile innovation, and supports it. Restless’ role in
the roll-out of U-Report is an example of this. In addition, Restless sometimes innovates
itself.
Establishing this Youth Working Group (and leading it in the typical ‘Restless’ way, with lots
of interaction and post-its); conducting this local-led action research in Karamoja;
86
Currently, and in the words of Juliet, the situation is one of ‘Restless is good at this, let’s use them’ rather than
‘Restless is good at this, let’s replicate the model.’
Page 90 of 131
contributions to Restless’ brand and policies: these are all examples of innovation. None of
these examples are genuinely new on a global scale – but they are new in the environment
in which Restless Development operates.
Partnership working
Partnering with the government
Restless went to the CBS [Community Based Services] on their first day, to register. ‘We
want to work with youth.’ ‘Where do you want to work?’ ‘It depends on you.’ So we looked at
the gaps and allocated a few locations, and Restless has subsequently integrated its work
within the strategies of the district. And they take the Karimojongs seriously, too: they
advertise for their volunteers within Karamoja, and the staff is mixed. Jennifer Longoli,
Community Development Officer Youth, Moroto District.
Ugandan district and sub-county authorities appreciate Restless’ work and help with things
such as the identification of Parishes and the selection of volunteers. This is an
achievement, as Restless’ entry point is not a favourable one. First, there is an implicit
hierarchy in the minds of public authorities: ‘hardware’ organisations that build schools and
give solar panels stand at the top, and ‘software’ organisations that build skills and create
awareness stand well below them. Within the group of ‘software’ organisations, the
organisations that provide education and vocational training score higher than the
organisations that work on ‘soft skills’, like Restless. Therefore, by virtue of its remit,
Restless has inherently low status. Second, Restless is rather less generous than some
other organisations, and does not routinely provide public officials with ‘allowances.’
The reason that Ugandan authorities are by and large positive about the work of Restless is
a consequence of Restless’ ongoing efforts to coordinate with district and sub-county levels
of government. Not all organisations do this:
•
•
•
•
‘Sometimes we see a GIZ vehicle whizzing by, without stopping. Restless would not do
this.’ 87
‘Restless works with us, and this is good. Marie Stopes entered the Parish and started a
SRH campaign, without checking with the health clinic.’ 88
‘Maybe it is a matter of personal skills, but the engagement of Restless is very different
from the engagement of Save the Children.’ 89
‘ACF and the Samaritan Purse were nearly kicked out of the district, because their work
was so poor – Restless never had this sort of problem.’ 90
The coordination and cooperation with public authorities will occasionally be tiring and
demoralising. Some officials are interested and engaged, but many others accept Restless’
progress reports without critical questions, additional requests, or meaningful feedback. 91
But Restless engages, and this creates goodwill.
87
Mary, Napak District.
The man working in the clinic we visited – I forgot to note down his name.
89
Eko Edward, Assistant Chief Administrative Officer, Moroto District.
90
John and Paul, Napak District.
91
At first I did not understand how this matches with the regular recommendation to conduct ‘joint monitoring
visits,’ but I then understood that the government-set allowances, payable to all government officials who are
asked to leave their office, might be an incentive.
88
Page 91 of 131
Inter-agency cooperation
If KIDEP would be looking for peer educators, the organisation would not have to look
elsewhere – they would just approach Restless. Olupot James Peter, KIDEP
Restless plays a useful inter-agency role. A particularly impressive example of such a role is
that Restless was a co-founder and is currently the lead agency of the Youth Working Group
of Uganda. This group quickly gained members and popularity, and strengthened interagency cooperation. When Museveni declared that SRH education is something for parents,
not for schools, Restless helped to persuade the members to counter this statement by
conveying evidence-based messages to the wider population, rather than by issuing
antagonistic press statements. This is impressive, and illustrates that Restless is already
punching above its weight.
Restless supports individual agencies too. “UNICEF wanted to engage more with youths and
mapped youth organisations in Uganda. Restless had been very helpful in this exercise.
Restless had lots of experience with youth participation, and allowed UNICEF access to
Restless’ network. Restless has been an important provider of U-reporters too.” (Shaban,
UNICEF)
Sustainability
Restless Development is not a donor-driven organisation. The organisation believes that
cash-for-work programmes is really ‘distributing hand-outs’, which ‘goes against our values’ so the organisation chose not to pursue this particular WFP funding option. This is admirable
and important. Having said this: financial sustainability is the biggest risk that Restless
Uganda is currently facing. Specifically:
•
•
•
The organisation is overly reliant on DFID and Comic Relief. The organisation explicitly
acknowledges this, and the funding strategy seeks to reduce this dependency. There
seems to be ample opportunity for this. However:
Restless engages a lot with other agencies, but much less with the people who are
responsible for the funding flows. The implication is, for example, that UNICEF is most
appreciative of Restless’ work and support – but Restless does not feature on the chart
of UNICEF’s ‘implementing partners.’ Moreover:
With rare exceptions, 92 Restless Development sends its reports in time. This is
important, particularly in the coming year or so, as Restless needs to prove itself as an
organisation that is able to scale up. It is also a challenge, especially now that the
Uganda office’s fundraising manager is on maternity leave. It requires a lot of awareness
and capacity building among the grass-root level implementers, and even then these
reports require a lot of editing.
A Kampala desk for the Country Director and other Restless staff members is useful (though
not without the usual and considerable risks that are part and parcel of remote
management), but only if it is combined with rigorous prioritisation. The opportunities are
very diverse, and each option could potentially cost a great deal of time. To illustrate this
diversity, there are:
92
Specifically, the reports to the AIDS Commission are often a few days late.
Page 92 of 131
•
•
•
•
Traditional donor agencies. Restless’ support to UNICEF’s work has created
considerable goodwill. This goodwill now needs to trickle down (or up) to those who
make funding decisions.
Peer organisations. Marie Stopes recently got a lot of DFID money, which it will partly
spend on partnerships. Positive signals were given (“you do great work; you should get
funded’) and the two organisations complement each other nicely, with Marie Stopes
having clinics and Restless having a large network.
Private sector companies. Amarex is Restless’ first potential in-country private sector
donor.
Individuals and networks. Alumni are underutilised, as direct or indirect sources of
funding.
Recommendation 4
Restless needs to copy its evidence-based mentality onto fundraising. By keeping track of
time invested in different funding activities (such as visits, proposals, cold calls, networks)
and targets (such as multilaterals, bilaterals, companies, individuals), Restless will gradually
learn where to focus its attention. Without rigorous prioritisation, the organisation will spread
itself too thinly.
Efficiency and Value for Money assessment
Restless is just a drop at the moment. Eko
Edward, Assistant Chief Administrative Officer,
Moroto District.
I see the youths that are involved. They have
the interest and become part of the Restless
community – I meet them, sometimes.
Restless is the only organisation that does
this. […] Other organisations do not have
this. Joyce, Education Officer, Napak District.
Economy. 93
•
The internal audit raised a number of minor issues which have all been addressed
swiftly. The result is that finance and procurement policies are sound, known, applied,
and sometimes so very rigorous that adherence is a rather time consuming matter.
•
The recently introduced Full Cost Recovery model and accompanying time sheets have
not yet been operationalised.
•
Restless Development operates at a modest scale, which means that the organisation
does not benefit from economies of scale (e.g. a computer will be more expensive for
Restless Development than for, say, UNICEF).
•
The accounts show small staff donations to Restless. These are mostly unused floats
(and returning this is good practice) but partly also unused parts of allowances that have
been paid by other organisations (and I have never before seen anybody returning this to
one’s organisation). This says something about the mentality of Restless’ staff.
Recommendation 5
To achieve economies of scale, Restless might benefit from joint procurement with other
organisations.
93
‘Economy’ refers to the costs of inputs and resources of an intervention. The extent of ‘economy’ is typically
measured by looking at procurement procedures and by calculating unit costs.
Page 93 of 131
Recommendation 6
In absence of any evidence of abuse of funds or of a money-hungry mentality, some of
Restless’ rules and regulations seem overly stringent. Currently, for example:
•
receipt-hunting is sometimes disproportionately time consuming;
•
the requirement to return all redundant assets to the Head Office costs fuel and vehicle
space and fails to utilise recycling opportunities that would be valuable locally (e.g. old
tyres could be turned into sandals);
•
physical monthly visits, receipts-in-hand, of the Moroto-based Finance Officer to Jinja
seems excessive; and
•
The Moroto field office would benefit from holding one-month reserves that could ensure
that volunteer allowances can be paid even if the town’s only bank is short on cash.
Recommendation 7
A single finance system would save time. Currently, finances are managed through a
combination of Excel sheets and Pastel.
Efficiency. 94 I know of no more efficient way of delivering ongoing behavioural change
messages than through Restless’ deployment of community volunteers. I am unable to make
a similar statement in relation to national or international volunteers, but assume that costs
are higher and without proportional increase in utility.
Recommendation 8
Restless is considering moving into urban areas. It seems to me that this is not necessary.
Restless is “just a drop,” but it is a drop with a very powerful model. Roll it out, make it
known, get it replicated – and then worry about urban areas.
Recommendation 9
Four out of the six members of Restless’ Management Committee are expatriates. There are
two drawbacks of this:
•
•
Expatriate employees are more expensive than locally recruited employees. These costs
are unusually modest, and amount to nothing more than an annual flight and a 10-20%
salary supplement.
Potential frustration. Restless empowers its staff members and encourages upward
mobility and, in this context, an expat-dominated Management Committee sends an
incorrect message.
International staff exchange (and especially South-South exchange) is valuable and
reinforces a collective identity, but any such exchange is best done for limited periods and at
different levels within the organisation.
Effectiveness. 95 In Restless Development, two complementary characteristics help to
ensure Value for Money.
•
Restless has an extraordinarily clear ‘line of sight.’ There are strong and direct links
between the organisation’s values, strategies (global, national and departmental),
programme aims and activities. Employees are aware of this. Almost all people I asked
94
‘Efficiency’ refers to how much you get out in relation to what you put in. It’s about maximising an output for a
given input, or minimising input for an output.
95
‘Effectiveness’ refers to how far a programme achieves its intended outcomes, using qualitative and
quantitative assessments of change.
Page 94 of 131
were able to list the organisation’s values and explain the thinking behind them. People
mentioned this when explaining why they were not salary-driven, and when reflecting on
Restless’ advocacy and direct programming activities. Even at the level of volunteers I
noticed the ability to link and use the top level brand identity (and an “uninspired”
Restless shirt) in day-to-day life skills activities. Unlike Oxfam, CAFOD, Christian Aid,
Islamic Relief, Save the Children and many other NGOs, Restless Development’s brand
name carries an important, empowering and non-patronising message.
•
Restless compares the Value for Money of its different approaches. Uganda’s
Management Committee has recently looked at each of their sectors of operation and,
for each sector, compared the impact with Restless’ financial and HR investments. For
each sector of operation they looked at the current extent of success and the life cycle
moment. 96 Such reflections, and the comparison of alternatives in general, is an
essential component of achieving Value for Money.
Impact and value for money of PPA funding
Attributable impact of PPA funding on results, relevance, effectiveness
and efficiency
The PPA has been a mixed blessing for Restless’ operations in Uganda. On the one hand,
the unrestricted money was welcome and is largely used as match funding and to cover
some of the core costs. 97 On the other hand, the PPA has meant that the organisation had to
terminate much of its CSCF programme in the east of Uganda, as PPA holders are not
eligible for CSCF funding.
Value for money assessment of PPA funding
Covering core costs (e.g. the office’s insurance, new tyres) does not represent the best
possible value for money. Recognising this, Restless Development is working towards a full
cost recovery system. The final evaluation will be in the position to assess the extent of
success.
Recommendation 10
Covering core costs is not the best possible use of strategic PPA funding. Restless
Development Uganda should move fast on its full cost recovery system, and utilise PPA
funding more strategically.
96
The conclusions were, in essence:
Restless has ample evidence of good practice in the field of SRH, and this evidence is ready for use in
advocacy activities.
•
Restless’ work has an impact in the field of livelihoods, but Restless does not yet have a sufficiently robust
track record to advocate for certain models.
•
Restless’ performance in the field of civic participation is not yet sufficiently strong. To move this sector
forward Restless needs innovative pilots, such as the action research and the Zing-funded ‘Transforming
Youth Programme.’
•
Restless may not be well-placed for CSO capacity building work, unless it is specifically in the context of
youth participation.
The evidence I gathered in the course of this assignment confirms these conclusions.
97
I note that the Finance function in London reports that the full cost recovery model was piloted in Uganda, in
2009-2010, but that I did not see much evidence of this. The time sheets required for full cost recovery were
insufficiently detailed and not used for any sort of analysis. The office has recently introduced a new type of time
sheet, which may prove to work better.
•
Page 95 of 131
End of country report.
Page 96 of 131
Annex G2: Country programme report 2: Restless Development,
Zambia
Final version, 19 August 2012
Contents
Assignment ....................................................................................................................... 98
Results ............................................................................................................................... 98
Performance assessment against logframe ..................................................................... 98
Effects on poor and marginalised groups and civil society ............................................... 99
Relevance ........................................................................................................................ 102
Representativeness ....................................................................................................... 102
Targeting strategy ......................................................................................................... 102
Effectiveness ................................................................................................................... 103
Learning ........................................................................................................................ 103
Learning that improves Restless Development’s own capacity .................................. 103
Learning that provides contextual knowledge ............................................................. 103
Learning that can be shared with others .................................................................... 104
Innovation ...................................................................................................................... 104
Partnership working ....................................................................................................... 104
Partnerships with schools and colleges...................................................................... 105
Partnering with the government ................................................................................. 105
Partnering with donor agencies .................................................................................. 106
Sustainability ................................................................................................................. 107
Efficiency and Value for Money assessment ................................................................. 107
Impact and value for money of PPA funding ................................................................ 109
Attributable impact of PPA funding ................................................................................ 109
Value for money assessment of PPA funding ................................................................ 109
Page 97 of 131
Assignment
This assessment is part of a PPA mid-term evaluation that covers the work of Restless
Development, War Child and Youth Business International . The assessment took a little
over three days.
I first spent a day in Lusaka, with the Country Director and his primary contacts in the Irish
Embassy, the Ministry of Education and Zanaco Bank. I then visited Restless Development’s
offices in Kabwe and Lusaka; and across Serenje, Mkushi, Kapiri Mposhi and Chibombo
Districts a teacher training college; a school (but without meeting any children as the
holidays had started); a health centre; an outreach office for a Member of Parliament; a
microfinance office; and a Community Development Assistant. The Chibombo District
Education Guidance and Counselling Officer kindly passed by Restless’ Kabwe office, and
on Friday evening I had the good company of two ex-volunteers, who are now part of
Restless’ active national alumni network. The conversations with them were important, not
least because I did not get to speak to any current volunteers, as the volunteering season
had ended.
Harriet and Tom, respectively Restless’ Senior Manager and Country Director, spent many
hours on the road with me. This led to lengthy and most helpful conversations. I would like to
thank them, and all others who have helped me understand the Restless work in Zambia.
This report follows the standard IPR structure.
Results
Performance assessment against logframe
One of the main pieces of feedback from DFID’s initial review of our draft report was that we
had to be extremely conscious of the quality, rigour and accuracy of the evaluations stated
as evidence of our achievements. A Restless document titled Feedback to CPs_01Jun12, page 4.
The output levels are very significantly overachieved, but I note that:
•
•
The aggregation of numbers of beneficiaries is done manually and the calculations are
not filed. The only way to verify these calculations is to do them again. The lists that the
calculations are based on are the attendance lists of activities and group member lists.
These lists partly overlap but are both counted in full. This means that the aggregate
figures include some double-counting.
The Zambia office follows the worldwide reporting template. This template does not
recognise college students as a separate category. The Zambia office revolves this by
categorising college students as ‘out of school youth.’ This is incorrect as it combines
two very different groups (i.e. particularly disadvantaged out-of-school youths and highopportunity college students).
Recommendations 1 and 2
•
The reporting template should include college students as a separate category to avoid
the numbers of a high-opportunity group to be combined with the numbers of a
particularly disadvantaged group.
•
The Zambia office’s new reporting system should include an aggregation trail that is filed
in its entirety for M&E purposes. This system should cross-check group and activity lists
Page 98 of 131
to avoid double counting. If the organisation follows up on this recommendation,
Zambia’s beneficiary numbers for 2012-2013 will be lower than the numbers for 20112012.
Effects on poor and marginalised groups and civil society
Effects on poor and marginalised groups
Face to face contact is not the same as a poster or radio show. Mr John Sianga, Health Officer
at the Matuka Community Health Centre, Kapiri Mposhi District, comparing Restless’ work with the
work of other organisations.
Zambia is full of posters, paintings, stickers, radio programmes, SMS campaigns and other
messaging that aims to reduce HIV prevalence rates. None of these types of communication
is likely to be as effective as Restless’ Behavioural Change Communication (BCC) model.
This model focuses on onsite and ongoing face-to-face contact between the target group
and volunteers. These volunteers are ‘peers’, in a way: they are not much older than the
people in this target group themselves. 98
Restless’ BCC operations target school pupils and, more recently, out-of-school youths. In
addition and unique within Restless’ worldwide operations, Restless Zambia places
volunteers in nearly all the country’s teacher training colleges. 99 In these colleges, volunteers
work with the HIV Committee to convey messages directly to the students.
The college volunteers also form a group of some 40 peer educators. These peer educators
receive five days of additional training. In their first year in college, the peer educators pilot
their skills in schools in the vicinity of the college. In their second year, during the teacher
practice, they set up a resource centre in each of their respective schools, conduct HIV
sessions with their pupils, and take these pupils to the wider community for sessions of
drama and dance, mixed with HIV-related messages.
A 2011 survey among former teacher trainees who now teach in primary schools reports that
86% (92) of the respondents indicated that they only had a single sexual partner at the time
of responding, and that 76% (77) of the respondents stated that they had only had one
sexual partner in the previous 12 months. 92% (n=98) reported using condoms, and 81%
(85) of the ex-students have gone for Voluntary Counselling and Testing (VCT). In relation to
their role as teachers: 71 out of 83 ex-students who had attended the Sexual and
Reproductive Health (SRH) sessions of Restless volunteers reported that they were now
offering lessons in SRH in their respective schools. 100
Restless’ interactive model is a genuine BCC model 101 rather than the generally less
effective one-directional messages, and the duration of the engagement seems sufficiently
long to help the recipients through a process that may eventually lead to behaviour
98
This closeness in age was repeatedly cited as the single most important asset of the Restless volunteers.
This is the ‘Teacher AIDS Action Programme’, or ‘TAAP.’
100
All figures are taken from Milapo, N and Macwan’gi, M; External Evaluation of the Teacher AIDS Action
Programme (TAAP); a case study of southern and central provinces, Institute of Economic and Social Research
(INESOR), University of Zambia, Lusaka, November 2011.
101
BCC refers to “an interactive process with communities […] to develop tailored messages and approaches
using a variety of communication channels to develop positive behaviours; promote and sustain individual,
community and societal behaviour change; and maintain appropriate behaviours.” FHI (2002) Behaviour Change
Communication (BCC) For HIV/AIDS. A Strategic Framework. Arlington: Family Health International.
99
Page 99 of 131
change. 102 This design and the findings of the 2011 survey jointly lead me to believe that this
model is likely to be effective, provided that grinding poverty does not prevent people from
making healthy choices. 103 A conversation with two peer educators took out my remaining
doubt: they confirmed a strong commitment and determination to continue their HIV work in
the years to come, even when I asked them questions with a strong counter-bias (‘but come
on, you don’t even get paid for this – surely you have better things to do, next year?!’).
The college and school programmes have been ongoing for a number of years and have
considerable impact. 104
The Youth Accountability Model is much newer. This project was used as the PPA ‘Best
Case Scenario’ and this was an unwise choice as the project is in its early stages of
transferring skills, and yet has to produce significant results. 105 This PPA case study is
102
There are several models of behavioural change. An example by means of illustration is the Stages of Change
Model, from Prochaska, J. O. & Di Clemente, C. C. (1982) Transtheoretical therapy: Toward a more integrative
model of change. Psychotherapy: Theory, Research and Practice, 19(3), pp. 276-288:
103
I.e. ‘Why would I worry about getting AIDS after several years if I can’t feed my siblings tomorrow?’ Note that
this link between poverty and HIV is the rationale behind Restless’ Financial Fitness programme.
104
See J.A. Denison et al (2011) Do Peer Educators Make a Difference? An Evaluation of a Youth-led HIV
prevention Model in Schools’,Health Education Research 27 (3). Several of these programme’s design elements
were cited as ‘weaknesses’ when I interviewed college HIV Committee members and peer educators. I
understand their point of view but think these weaknesses are in fact illustrations of smart design. For example:
•
The HIV Committee members complained that Restless had stopped investing in the college’s resource
centre, and now the college had to do this itself. This termination is not necessitated by a shortage of
funding, as the HIV Committee appeared to think, but a deliberate and gradual transfer of responsibilities
from Restless to the college.
•
The peer educators complained that Restless expected them to set up a resource centre in their practice
school but did not provide them with any material. This ‘forced us to be creative; we went to the health centre
to get things, and we produced a lot of stuff ourselves.’ This reinforced a sense of ownership, which will
benefit these resource centres more than a few pre-produced booklets and posters.
There were several other examples, all along the same lines of making lives more difficult in the short term, but
increasing ownership in the longer term. All this intelligent design does not mean that there are no problems. As
with all remotely managed programmes, the school and college programmes require risk mitigation measures
that only work if the counterparts – the schools and colleges – actively gather and communicate early warning
signals. This does not always happen and this means that, for example, breaches of the volunteers’ Code of
Conduct are not always noticed in time.
105
In the community that I visited the volunteer and youth group’s Chair struggled to get youths to attend
sessions: the reported activity group size ranged from 3 to 20, in a population of an estimated 200 youths (note
that this same problem is reported in sheet 12 of the TIKAMBE external evaluation report; note also that Restless
counts this group as 25 beneficiaries). Moreover,
•
The two self-assessments that the group had conducted and the group’s subsequent lobbying efforts had not
resulted in any progress - largely because the issues raised are a matter of national policy rather than local
initiative;
Page 100 of 131
based on an external evaluation, but omits crucial contextual information 106 and, in some
cases, misinterprets statistics. 107 The external evaluation itself does not deal adequately with
a potential social desirability bias, which means that the results may have been
exaggerated. 108 None of this suggests that the Youth Accountability Project is not worth the
investment – but it confirms that this type of initiatives takes years to bear significant fruits.
Effects on the volunteers
I always wanted to study economics, but the volunteering caused me to refocus onto social
work. Mr Thabo Kekana, ex-volunteer, 2007.
The vast majority of Restless volunteers complete their assignment, and the impact of the
experience is profound. They gain skills and confidence, and overcome fear (‘the students
were so intimidating in the beginning, and nobody would come to my sessions’). Many of
them remain committed to the principle of volunteering and the experience causes some
volunteers to change their career aspirations (see the quote above).
A much smaller portion of volunteers does not last until the end. Restless reports a drop out
percentage of between 10 and 20%. Some international volunteers dropped out almost
immediately after arriving, or discontinued their assignment because they got sick or found it
impossible to adapt to their host environment. The presence of a Zambia-based British
member of staff to support these international volunteers – Tiffany - has reduced this
retention problem. National volunteers regularly leave because they find employment
elsewhere, or are asked to leave after they break their Code of Conduct. 109 In each of these
cases, work continuity is compromised and the unit cost per volunteer increases.
Many of Restless Development’s employees worldwide are ex-volunteers. Their loyalty and
enthusiasm is very noticeable. Mr Mukonka of the Ministry of Education and Mrs Kwambwa
•
The youth group had not formally registered itself and could therefore not yet access the support of the
government’s Youth Empowerment Fund or secure microcredit, even though the microcredit representative
regularly attends the youth activities and said that he ‘would be happy to take a risk by giving them a loan, if
only they register and produce a good business plan.’ Restless has noted this registration-related challenge
and will incorporate ‘self-registration support’ in future rounds of working with out-of-school youths. The
health centre’s youth-friendly corner did not exist; and
•
The two most active members of the group had both left the area.
The Community Development Assistant reported not to have the time to take over Restless’ responsibilities, and
this has effectively meant that ‘when the volunteers leave it is as if Restless is dead’ (Bernard Chishimba, group
member of Keembe Community, Chibombo District).
106
One example is that the case study reports on the improved communication between youths and decisionmakers and on the increased frequency of government officials’ community visits - but fails to mention that these
improvements were observed during election period.
107
One example is that the case study reports that there has been a “significant increase in youth accessing
condoms from health centres (20% to 84%) and treatment for STIs and antenatal care (33% to 65%). The 84% is
based on clinical records, which means that this percentage relates to youths who are in the health centres
already, and says nothing about the percentage of access to the health services. The 65% is the percentage of
youths that reported that it is “fairly easy to access treatment for STIs at their local clinic” which is different from
actually accessing such treatment (which 7% of the respondents have done).
108
One example is that 65 respondents reported to have listened to the radio show, but none of the respondents
was able to name a single topic that these radio shows had been discussing. This is unlikely and suggests that
people might have given answers that they felt would please the interviewer. Restless picked up on this and has
introduced ‘radio listener groups’ in colleges and elsewhere. A conversation with two members of such a listener
group illustrated the challenging work environment in which Restless operates: these two youths had missed
several shows because they had not always been able to arrange a radio.
109
Something that was particularly common in the college I visited, and may be related to the college’s rather
strict rules in relation to boy-girl contact.
Page 101 of 131
of the Irish Embassy gave me unprompted and separate accounts of the impressive
personal growth they had witnessed as they saw ex-volunteers grow within the organisation.
Relevance
Representativeness 110
I have been on Restless’ radio show twice, from the Phoenix Studio in Lusaka. Chisoko
Chamasonde, Chair of the Keembe community group, Chibombo District.
Restless’ profiles. Restless Development states that ‘we are who we serve’ and this is
partly true. Many peer educators and all community volunteers come from the country’s
poorer communities. Similarly, a portion of the ICS volunteers are from a disadvantaged
(British) background. Conversely and sensibly, Restless staff and its national and non-ICS
international volunteers are recruited and promoted on the basis of drive, competencies and
potential, irrespective of their socio-economic backgrounds.
Understanding and responding to the needs and priorities of Zambia’s youths. One of
Restless’ primary strengths is the organisation’s years of hands-on work experience with
vulnerable and disadvantaged groups. This shows. The programme designs show an
understanding of behavioural change dynamics (multi-method, intensive, face to face and
over a long period of time) and of the risks of dependency. Restless does not believe in turnkey projects. Instead, it provides encouragement and only modest support as communities
build their own buildings; as peer educators set up resource centres; and as colleges
develop their own capacity to integrate HIV-related messages into their curriculum.
Representing the needs and priorities of Zambia’s youths in the wider sector. Restless
Development uses its hands-on work in its advocacy endeavours. These endeavours are
targeting Zambia’s young population (through its Phoenix radio programmes, for example)
as well as the country’s policy makers. Its advocacy vis-à-vis the government shows a track
record of success (and one that is readily confirmed by government authorities – there are a
few examples in a later section).
Targeting strategy
We would feel guilty if we did not take this forward... there is very little HIV awareness in the
communities in which we work and if we are not going to raise awareness then who will?
Waria and Tshimba, two Restless peer educators. Both had just returned from their teacher practice,
and invited me to visit the resource centres they had set up.
Restless Development Zambia selects its sites of operation in close consultation with the
relevant authorities. The choices are inspired by two desires: to operate in disadvantaged
communities and, for purposes of scale and sustainability, to work with the next generation
of teachers. Both choices are sound and the way Restless operationalises these choices is
impressive: the project sites are invariably rural and relatively poor, and the organisation
covers all-but-one of Zambia’s teacher training colleges.
110
Coffey’s note on this heading is that it is about “The degree to which the supported civil society organisations
[this refers to Restless] represent and respond to the needs and priorities of their constituencies, (including where
relevant the poorest and most marginalized).”
Page 102 of 131
The Zambia country programme has recently adopted the model of ‘community volunteers.’
Such volunteers are not only a means to reach communities (and ensure that the work does
not collapse when the national and international volunteers leave the area) but also form a
target group themselves, as role models and tomorrow’s community leaders. Unfortunately, I
have not been able to talk with any community volunteers in Zambia, but from the Uganda
evaluation I have learned that the results can be impressive.
Effectiveness
Learning
Learning that improves Restless Development’s own capacity
We learned that the ICS volunteers are most useful if we use them in our Financial Fitness
programme, as this programme does not require the same cultural sensitivities as the SRH
programme. Tiffany Karatziola, Programme Coordinator.
Learning is ongoing and in all Restless programmes there are ample examples of intelligent
initial design and subsequent evidence-based modifications. The community volunteers,
mentioned in the previous paragraph, illustrate this: Restless recruited them after finding that
Restless’ conventional approach of placing short term national and international volunteers
did not provide sufficient continuity in a programme that requires years of continuous work to
bear maximum fruit. This learning is both ad hoc (a mentality) and systematic. Examples of
the latter include:
•
•
•
Regular meetings between managers and their staff - and a dashboard on the wall that
names and shames the managers who have not done this (!).
Thorough induction processes. The recent three-week induction process for Restless’
new Senior Manager, for example, did not only cover the strategic and operational
issues for Restless Zambia and the worldwide guidance (e.g. values, global strategy and
worldwide policies) but also included phone calls with peers in other countries of
operation. Such phone calls create relations that are further developed in periodic crosscountry meetings, and these relations prevent country offices from re-inventing the
wheel. There are many examples along the lines of ‘I asked the Uganda office for their
entrepreneurship manual, which saved me a lot of work.’ 111
Restless’ system of internal audits. The Zambian Finance Officer has conducted the
internal audit in Tanzania, for example, and this exercise resulted in mutual benefits and
learning.
Learning that provides contextual knowledge
This is the toughest college to work in. This is where we send our strongest volunteers to.
Harriet Mwiinga, former Senior Manager.
Restless Development does not employ a one-size-fits-all approach. Instead, its work in
different colleges and schools is designed to befit the context. This was illustrated by my
111
This induction is useful but costly, in terms of staff time. Partly in this context, staff retention is important. This
is a problem, sometimes – as last year’s four Finance and Administration Managers illustrate.
Page 103 of 131
Restless guide who mentioned, when visiting a college and in passing, a range of things that
were specific to this particular college – from the need for visiting women to wear skirts to
follow the college requirements, to the need for Restless to assign the very strongest
volunteers to this college in view of previous drop out, which had partly been caused by this
college’s challenging work environment.
Some of Restless’ learning is hands-on and ongoing. This learning is framed by the
monitoring cycles that each Restless volunteer goes through every month, which have the
triple purpose of reporting, learning, and serving as an early warning system. Other learning
is more strategic, such as in the case of Restless’ various external programme evaluations,
some of which are not required by donor agencies and conducted solely for the purpose of
learning.
Learning that can be shared with others
This manual will work as a reference guide for policy makers and a starting ground for CSOs
in terms of checking where they are, and are not, hitting […] good practice criteria. Restless
Development’s Good Practice Manual on HIV and AIDS awareness raising and impact, 2011, page 7.
‘Sharing and Learning’ is one of Restless five ‘strategic approaches.’ The Zambia office’s
history with such sharing is mostly related to its engagement with the Zambian government
(see later) and its participation in inter-agency working groups. More recently, Restless
started generating and sharing more strategic learning with peer agencies – as is illustrated
by its recent publication of a Good Practice Guide on HIV related interventions.
Innovation
Restless is into innovation, and pushes it in the face of the ministry. Mr Remmy Mukonka, the
National HIV/Aids Coordinator of the Ministry of Education.
Restless’ Zambia office introduced:
•
•
•
A teacher college programme. Restless Tanzania will replicate this ‘Teacher AIDS Action
Programme’ if the Government of Tanzania gives the organisation permission to do so;
The Youth Accountability Project; and
The Financial Fitness Programme.
All three programmes were new for Restless worldwide. The first two were also new to
Zambia.
Partnership working
We are heading towards a new strategy, and could easily get rid of RD [Restless
Development]. But we won’t, as RD has proven to be a very good partner. Ms Miyanda
Kwambwa, the Senior Education Advisor of the Irish Embassy.
Page 104 of 131
Partnerships with schools and colleges
It is difficult for me to talk with them about sex. The age difference is too large. It is different
with the volunteers: they are much closer to their age. Howard Kantolo, Deputy Head, Chilata
Basic School.
The partnerships with schools and colleges are good and undistorted by financial incentives.
The colleges and schools place these volunteers because they find them useful, and
because the work of these volunteers is in line with the Ministry of Education’s policy that
dictates that HIV is to be part of every school’s curriculum.
Occasionally, where there is initial resistance to Restless’ ABC approach, 112 Restless
activates the local Headman (i.e. traditional leader) and/or government structures. This
generally works: only one out of the 14 colleges – the country’s only Catholic college - did
not allow Restless’ volunteers to work with the students. Even if the school or college is
supportive, Restless will make sure to work with the traditional and government authorities.
Their support helps to neutralise the occasional criticism (‘they teach our children about
sex!’).
Partnering with the government
Any organisation doing anything in the field of education must work through the Ministry of
Education. No parallel structures. Ms Miyanda Kwambwa, the Senior Education Advisor of the
Irish Embassy.
Restless Development has a presence in Zambia because, in 2002, the Permanent
Secretary of the Ministry of Education had heard about Restless’ work in Tanzania and had
invited Restless into the country. 113 Contact with different layers and types of government
has been frequent ever since, and the Ministry of Education has recently (July 2012)
nominated the Director for Human Resources and Administration to represent the ministry in
the Board of Restless Development Zambia.
That does not mean that contact is easy. The current Memorandum of Understanding took
three years to renew. But the relationship is worth spending time on, these four examples
illustrate:
•
•
The October 2011 Report of the Committee on Labour, Youth and Sport, for the first
session of the 11th National Assembly, lists 17 recommendations. One of them is as
follows: “Your Committee observes that the approach by the Restless Development
Programme was unique in that it used the youths to work with the fellow youths. The
active participation of youths in the welfare of others was very encouraging and should
be emulated by other organisations. Your Committee recommends that there should be
more interaction between the youths and the leadership (traditional and political).”
Restless is now working with this Sub-Committee to ensure that this recommendation is
actually followed up on.
One follow up activity is already in progress: the Ministry of Youth and Sports is
developing a voluntary ‘National Youth Service’ and Restless Development is one of the
112
Abstinence, Be faithful, Condoms.
I might have to take this out as Tom is not entirely certain and the evidence is ambiguous. I might check this
with Nick, who was closely involved at the time.
113
Page 105 of 131
•
•
•
NGOs that is providing technical support – obviously based on Restless’ own work,
which is based on a model that is very similar to what the ministry’s model might turn out
to be.
Together with the Chibombo District authorities, 114 Restless has developed an HIVrelated school monitoring tool. The Ministry of Education’s HIV Coordinator 115 has taken
the initiative forward, and feels confident that the ministry will soon run a pilot. The final
evaluation might want to follow up on this.
The Ministry of Education is likely to mainstream Financial Health into the curriculum.
When asked about the roll out timelines, some are more optimistic (‘next year’) than
others (‘next year? Forget it.’). Either way: the intention is there, and this was at least
partly inspired by a Financial Fitness presentation that Restless had been invited to give
to the Parliamentary Sub-Committee on Labour, Youth and Sports.
Restless conducted a desk-based study on teenage pregnancies. The Ministry of
Education was impressed and organised a workshop to discuss the findings; to review
and strengthen the recommendations; and to create the foundations for their
operationalisation. The workshop took place on 17 August 2012, with Irish Aid, Sida,
UNFPA, UNICEF and 30 other NGOs working in education and health sectors in
attendance. The attribution of this workshop was straightforward: the Permanent
Secretary to the Ministry of Education started her introductory speech with the
observation that “[…]Restless Development undertook a desk review from around the
country and today you are sharing your findings with us here with a view to inform our
next steps in combating teenage pregnancies in the education sector.” In the closing
note, the Deputy Permanent Secretary noted that Restless had given Zambia a “great
opportunity for strengthening coordination between key Health and Education
management units and improving NGO design on ‘effectiveness’ in addressing
pregnancy prevalence.”
There are ample examples of inter-agency cooperation. For example:
•
•
•
Different organisations play a role in the Restless TIKAMBE radio shows, which all
evolve around various education policies (e.g. re-entry policy, HIV policy, anti-gender
based violence policy);
Three organisations (Kara Counselling, New Start Centre and Corridors of Hope) provide
mobile VSTs that Restless mobilises communities for; and
Restless supports the annual ‘Career Expo’, co-organised by the Ministry of Education
and an NGO named Junior Achievement. Restless arranges the Financial Fitness
section of the programme, and provides Restless alumni to run the ‘career profiles’ of the
expo’s visitors.
Most of this cooperation is a consequence of Restless’ participation in the inter-agency ‘Civil
Society Health Forum’ and the ‘Project Coordination Committee’ (which Restless chaired in
2010-11).
Partnering with donor agencies
Restless Development Zambia prefers a few long term partnerships over many one-off
donations. This works: Restless does not get funding from many in-country donor agencies
(see next section) but the funding relations it does have are strong. When visiting Irish Aid
and Zanaco, the people I talked with emphasised that they were thoroughly content with
Restless’ work and had no intention of discontinuing the support. Substance, focus, impact,
reporting: it all received praise.
114
115
Specifically: Mr Wilfred Chilala, the District Guidance and Counselling Officer of Chibombo District.
Mr Remmy Mukonka.
Page 106 of 131
Sustainability
The PPA contribution amounts to 40% of the country office’s total funding. The remaining
funding is granted by four major sources (i.e. DFID’s ICS, UNICEF, Irish Aid and – new – the
Zambian Governance Foundation for Civil Society), and there are a few minor contributions
(the Zambia Governance Foundation, UNFPA, Zanaco and Comic Relief). A few
discontinuations would jeopardise the organisation’s operations in Zambia, but this seems
unlikely as the funding relationships are strong and have a multiyear history.
Restless Development’s pipeline includes corporate companies, a few bilateral and
multilateral agencies and a basket fund. Restless used PPA funds to cover the costs related
to accessing this basked fund - the Zambia Governance Foundation. The foundation has
already provided seed funding, and Restless Development is in currently in the final shortlist
of agencies that the foundation is considering for multiyear funding relations. If Restless
Development is indeed selected, this basket fund will reduce Restless’ PPA dependency and
increase the organisation’s annual budget in Zambia.
Efficiency and Value for Money assessment
Introductory note. Organisations around the world are in competition with each other and
therefore commonly provide donor agencies with overambitious logframes that create the
impression that programme impact is immediate and enormous. These organisations then
commonly exaggerate their impact in reports and case studies. The fact of the matter is that
a programme such as the Youth Accountability Project is new and risky, and Restless is
finding its way forward by trial and error. Judging such innovative programmes on the basis
of their initial efficiency and effectiveness criteria means killing innovation.
Economy. 116 The usual checks and balances are in place and the recent internal audit
report has been followed up on. More fundamentally, Restless Development Zambia looks at
its various options with an eye to achieve maximum value for money, and selects
programmes accordingly.
Efficiency. 117 Restless Development’s volunteers receive a modest allowance 118 and are
placed in schools and colleges on a closed-wallet basis (and these schools and colleges are
expected to provide accommodation). Many of the volunteers’ trainers are Restless partners,
who make their contributions on the basis of goodwill rather than financial incentives. After
the training, the volunteers are utilised for a range of purposes – they do onsite grassroots
work, train teachers and peer educators, provide support to the district Guidance and
Counselling Officer, provide evidence that feed into advocacy messages, and so forth. This
all causes Restless’ use of national and community volunteers to be highly cost-effective.
The costs of international and ICS volunteers are much higher, but these costs are absorbed
by either the volunteers themselves or the ICS grant.
The policy advocacy work of Restless is very efficient too. The HIV monitoring tool, the
current support Restless provides to the Ministry of Youth and Sports, the work done in the
116
‘Economy’ refers to the costs of inputs and resources of an intervention. The extent of ‘economy’ is typically
measured by looking at procurement procedures and by calculating unit costs.
117
‘Efficiency’ refers to how much you get out in relation to what you put in. It’s about maximising an output for a
given input, or minimising input for an output.
118
I suspect that one of the reasons for Restless’ ability to recruit good people while paying salaries that are
below the NGO market rates is that the organisation often recruits from amongst ex-volunteers, whose salary
expectations are modest.
Page 107 of 131
context of the TIKAMBE radio shows: nearly all advocacy work is based on evidence that is
created in the course of Restless’ ongoing work.
Recommendation 3
The volunteers would be even more effective if they arrive at the start of the teaching
season. This is not always the case.
Some of the inefficiencies require Headquarter action. Two particularly time-consuming
inefficiencies are:
•
•
The Restless financial cycle. This cycle follows the academic year, which sets Restless
apart from most of the rest of the world. Changing this to the normal calendar year
would, in the long run, prove to be a cost-effective thing to do.
The multi-system finance system. Currently, finances are managed through a
combination of Excel sheets and Pastel, with a third system covering fundraising work.
Effectiveness. 119
The volunteers did amazing things. Waria and Tshimba, two peer educators.
All Restless’ programmes share two things:
1. An extraordinary line of sight. Restless staff and volunteers all know the organisation’s
values (I asked Kennedy, the driver, and he knew all four of them). These values and the
global strategy directly impact on programme design. Such line of sight is most unusual,
and the resultant focus has a strong positive impact on the organisation’s effectiveness.
2. Very consistent cooperation with Zambia’s public authorities. Restless does not and
would never build parallel structures. This is wise, as parallel structures may be the most
effective option in the short run, but in the longer run such structures are counterproductive.
Newer programmes yet have to prove themselves, but Restless’ more mature programmes
(which themselves took 5-6 years to prove themselves) give hope that these new
programmes will eventually prove to be useful too. These more mature models - Restless’
college and school programmes - follow a well-designed model. Internal and external
evaluations show them to be highly effective. Anecdotal evidence supports this: a
government official told me about one of ‘his’ schools, for example, where the excessive
number of teenage pregnancies showed a marked reduction after Restless started its onsite
work. 120
For now, Restless’ work in health centres seems less effective. This might be because
volunteers are not present quite as frequently, as they are based in schools and colleges,
not in health centres. In the case of health centres, and in the out-of-school programmes, the
genuine buy-in of the duty bearers is particularly important, and my conversations with a few
of them suggest that this buy-in is not consistent and firm. The same applies to the
communities themselves: some are enthusiastic and willing to invest time and resources to
119
‘Effectiveness’ refers to how far a programme achieves its intended outcomes, using qualitative and
quantitative assessments of change.
120
Mr Wilfred Chilala, District Guidance and Counselling Officer of Chibombo District. He was not entirely sure of
the number of teenage pregnancies, but thought it had been between 25 and 28 in a single year.
Page 108 of 131
support Restless’ work, but I have also seen a community where the youth group is
insufficiently motivated to register itself and where the youth centre remains unbuilt because
the villages do not provide their part of the deal (i.e. 250 bricks each). Restless’ investigative
mentality suggests that the organisation will learn effectively, and will gradually strengthen
these programmes. All of them are new to an organisation that has, until 2010, only ever
focused on formal education and RSH.
Impact and value for money of PPA funding
Attributable impact of PPA funding
The impact of the PPA contribution is primarily one of scale rather than depth. Without this
contribution, Restless’ would be doing the same things, but with fewer staff, volunteers and
deployable assets, and would therefore not be reaching the same number of people.
Value for money assessment of PPA funding
The PPA contribution forms nearly 40% of the Restless Development Zambia budget, and a
discontinuation of the PPA contribution would require a dramatic cost-cutting and scalecutting exercise.
The utilisation of this contribution is as follows:
Office costs
Central costs
Governance costs
Programme support
Direct programme costs
12%
27%
26%
28%
7%
This is not strategic spending, and does not represent the best possible value for PPA
money. Restless Development is working towards a full cost recovery system. If successful,
this may enable the Zambia office to utilise the PPA contribution in a more strategic manner,
but progress could be much faster. Currently, even the first step towards full cost recovery –
appropriate time sheets that are systematically filled – has not yet been taken.
Recommendation 4
Restless Development Zambia should work towards a more strategic focus of its PPA
funding.
End of country report.
Page 109 of 131
Annex G3: Country programme report 3: War Child, Uganda
Final version, 19 August 2012
Contents
Context ............................................................................................................................ 111
Assignment ..................................................................................................................... 111
Results ............................................................................................................................. 111
Performance assessment against logframe ................................................................... 111
Effects on poor and marginalised groups and civil society ............................................. 112
Relevance ........................................................................................................................ 113
Representativeness ....................................................................................................... 113
Targeting strategy ......................................................................................................... 114
Effectiveness ................................................................................................................... 114
Learning ........................................................................................................................ 114
Innovation ...................................................................................................................... 115
Partnership working ....................................................................................................... 115
Sustainability ................................................................................................................. 117
Efficiency and value for money assessment ................................................................ 117
Impact and value for money of PPA funding ................................................................ 119
Attributable impacts of PPA funding .............................................................................. 119
Value for money assessment of PPA funding ................................................................ 119
Page 110 of 131
Context
War Child operates in challenging post-conflict environments. The people the organisation
aims to reach face multiple deprivations, have multiple vulnerabilities, and are often
‘invisible.’ Their habitats are hard to reach and district and sub-county authorities are, for a
range of reasons, unable or unwilling to meet responsibilities that are typically assigned to
public authorities. Harmful practices – courtship rape, the need for every family to have one
out-of-school warrior boy – are hard to change. Donor conditionalities are often stringent.
Assignment
This assessment is part of a PPA mid-term evaluation that covers the work of Restless
Development, War Child and Youth Business International . The assessment took five days,
excluding travelling time.
I first spent a day in Kampala, with the Country Director and some of his contacts in
UNICEF, UNFPA and the Ministry of Gender, Labour and Social Development (from now on:
‘Ministry of Gender’). I then visited the War Child offices and working environments in Kotido,
Kaabong and Pader – all in northern and north-eastern Uganda. I met with children, youths
and parents that do and sometimes do not belong 121 to War Child’s target group; some of
the authorities that cover child protection; school staff; War Child staff and facilitators; and
one donor partner (FAO). There were formal interviews and informal conversations with
individuals and groups; lots of documents; and a bit of observation (i.e. a REFLECT group in
action, children attending to their garden). I would like to thank all the many people who have
helped me understand War Child’s work in Uganda – and particularly those who travelled
with me, and who provided me with a wealth of insights while doing so.
This report follows the standard IPR structure.
Results
Performance assessment against logframe
No, I do not see my name on this list, but I do recognise the other names – they are all
members. Katerina, who enjoys full benefits of War Child’s ‘Junior Farmer Field and Life Schools’
(JFFLS) but is not reported on in War Child’s reports as she is, formally, only a ‘secondary
beneficiary.’
War Child Uganda does not implement programmes that are specifically PPA-funded. I have
therefore checked the organisation’s general reporting rigour by verifying, on random sample
basis, the reported numbers of three War Child projects. Specifically:
1. A JOAC-funded programme to improve education and protection for disabled
children in northern Uganda. I have visited the school, talked with the children and
seen the facilities that War Child has constructed, and have found the number of children
to be correct and the children to benefit from War Child’s investments.
2. War Child’s UNICEF-funded REFLECT groups. The groups exist, are active and, if my
brief observation is representative, very clearly useful. The War Child officer’s
121
An example of children that we interviewed but that do not belong to War Child’s direct target group: ablebodied pupils at Paipir Primary School, to verify their feelings about and treatment of their disabled class mates.
Page 111 of 131
confidence, when I randomly selected a group to visit, suggests that ‘my’ REFLECT
group was not performing better than other ones. The group’s session was attended by
both group members and others, suggesting that the number of people reached exceeds
the number of people targeted and reported on.
3. Its FAO-funded in- and out-of-school ‘Junior Farmer Field and Life Schools.’ War
Child underreports this programme’s reach when it reports 810 direct beneficiaries.
These were the initial beneficiaries – but I met several people who were registered as
secondary beneficiaries but who had the same rights and obligations as the other
members and who did not feel in the least bit ‘secondary.’
This random sample of three projects suggests that War Child reports honestly and is
meeting more than its share of output.
Effects on poor and marginalised groups and civil society
Then I used part of the profit I had made with the third bundle of clothes to buy flower – and
now I still sell clothes, but I also sell Mandas to the school children. Namongo Madalena, one
of War Child’s 36 community-selected livelihood support beneficiaries in Kaabong, selected randomly
for a visit and interview. A Manda is a Ugandan doughnut.
War Child’s remit is not to provide a little bit of support to many people throughout society,
but to zoom in on the hard-to-reach target group of particularly vulnerable and
disadvantaged children, and to facilitate a deep and lasting improvement in their lives.
This sometimes works and Namongo illustrates this. Namongo is a disabled 18 year-old
orphan girl, who I had selected randomly and visited. War Child had identified her in due
consultation with her village community. Conversations with Namongo, about possible
money-generating activities, led to a donation of a bundle of second-hand clothes, for her to
trade. Namongo traded and used her profits wisely, and forever changed the lives of herself
and her younger siblings (see quote above).
In other cases the long term impact cannot yet be measured but the early indicators are
positive. The REFLECT groups are likely to outlive the War Child involvement, as these
groups have built a strong community spirit and already started activities that were not in the
original plans (e.g. a saving scheme). The junior farmers’ schools are developing a similar
group spirit (‘these are my friends now’). The out-of-school group that I visited 122 is aware of
the temporary nature of War Child’s support and determined to continue the gardening: “We
talk about this all the time, and we are going to put money aside for next year’s seeds.”
I am less confident about the long term impact of the donation of dormitories and equipment
to a school, 123 meant to support this school’s inclusive education. This support has changed
the lives of 76 particularly disadvantaged children by enabling them to access education –
but the project will end soon and unless the sensitisation of the parents is extraordinarily
successful, many of these children are likely to return home without prospect of finishing
their primary education. I hope to be proven wrong but my expectations of the success of the
sensitisation are modest: War Child’s previous efforts to persuade parents to send their
disabled children to school has only been a partial success (only 78 out of the 100 available
122
This school was not randomly selected: it was the only school that proved reachable, after the rains of the
night before.
123
Note that construction is not at all War Child’s field of specialism and therefore probably more problematic
than strictly necessary.
Page 112 of 131
places were filled) – and this was at a time when War Child was still able to arrange
education that was entirely free of charge. 124
Lastly and in the very long term, War Child’s target group may benefit from a strengthened
government capacity in relation to OVCs, and from a functional legal process in relation to
gender-based violence. It is too early to assess if War Child’s investments in both areas are
worthwhile, and I fear that even the final evaluation will come too soon to assess impact.
Recommendations 1 and 2
• Some of the PPA funding is allocated to strategic M&E activities. War Child might want
to follow a selection of the War Child-supported disabled children that currently attend
Paipir primary school, to learn about the long term impact of its school infrastructure
investments.
• Such longitudinal research might prove me wrong, but until it does I suggest that War
Child is well-equipped to find particularly disadvantaged and disabled children and
persuade their parents or carers to allow them to go to school, but not the best
organisation to meet a school’s hardware needs.
Relevance
Representativeness
In my area of responsibility, none of the OVCs are still at home. They are all in school now.
Simon Mark, a Kaabong Town Council official responsible for child protection, who attributed this
achievement to War Child’s work.
Responding to the needs and priorities of some of the most marginalised and
vulnerable children in the world. The profile of all target persons I have met squarely fitted
within this description, and War Child does indeed provide intensive and tailor-made support
that meets their needs and priorities. War Child’s livelihood support, for example, is not a
matter of ‘everybody gets a goat.’ Instead, the selection of OVCs is done with the village
community and on the basis of ‘peer ranking.’ 125 This is followed by a careful and
individualised assessment, conducted jointly with the persons involved, to find the most
useful support possible. The only constraints are budgetary (support cannot far exceed UGX
350,000) and ethical (e.g. War Child will not support the production of liquor).
Representing some of the most marginalised and vulnerable children in the world.
War Child carefully matches its mandate with donor opportunities and I have seen no
evidence of mission drift. In terms of programme design, it may be possible to gradually turn
the tables from War Child implementing the programmes that follow the design instructions
of donor agencies such as FAO, UNICEF and the Ministry of Gender, to such agencies
replicating successful War Child practice. For this to happen, War Child needs to raise its
voice in the sector, and be particularly creative and innovative when utilising relatively
flexible grants (such as its current Comic Relief grant and, potentially, its PPA allocation).
The visibility of the organisation and its target groups.
124
The Head Teacher mentioned that the Ministry of Education has not responded to his request for support, and
that no other organisations had shown an interest in taking over War Child’s support role when War Child
withdraws in December 2012.
125
I.e. which criteria are most important in terms of vulnerability and deprivation, in any particular community.
Page 113 of 131
•
•
The organisation: War Child is not a banner-raising and sticker-pasting organisation.
On the road side, War Child’s logo does not feature among the many signs of
international organisations, and the logo is not visible in the communities it works with.
The War Child offices are modestly marked with a War Child sticker, and in Kotido even
this sticker was missing. This is good, as all these NGO and UN signs waste money and
reinforce the image of extreme donor dependency and lack of ownership of one’s own
future. If people know about War Child’s work, it is because of War Child’s work, not
because of its plaques and t-shirt distributions. 126
Its target groups: War Child mainstreams child protection in all its work and this is
visible in the way it depicts the children the organisation aims to support.
Targeting strategy
War Child is the only one. Sometimes Oxfam comes, to talk about gender based violence,
but War Child is the only one that stays. Group discussion with out-of-school children. Note that
these children have adopted INGO terminology: ‘gender-based violence’ was their term, not my
translation.
The children and their environments. War Child works in and out of school. In some of the
schools, War Child is one of many organisations – one school teacher in Kotido listed WFP,
UNICEF, 127 World Vision, ADRA, IRC, Human Rights, Red Cross, Tewpa and War Child.
Their activities are sometimes very comparable – I understood that Adra, ASB and War
Child all provide seeds, and that more than one organisation is providing drama kits. Out of
school, War Child is often the only organisation that reaches some of the most vulnerable
and disadvantaged children in a meaningful way. War Child identifies these children by
engaging with communities and, when selecting OVCs, allows sense to triumph over rigid
eligibility criteria. 128
Recommendation 3
In schools, War Child is one of many support organisations. Out of school, War Child is often
the only organisation with a meaningful presence. In terms of programme impact and as an
evidence-generating tool, War Child’s out-of-school work outperforms its in-school work.
This seems worth keeping in mind when designing future programmes.
Effectiveness
Learning
War Child is a dynamic organisation. There is ample evidence of learning-while-doing, and
the organisation is even willing to reconsider the entire programme design if the operational
context changes. 129
126
I did see one exception to this rule: an expensive-looking plaque that graced the dormitories of Paipir Primary
School.
127
These two UN agencies are the organisations that War Child’s facilitator - Mr Okongo – felt were most useful
to the school and its children.
128
A facilitator described how he had gone from one manjata to the other, asking community members to help
him identify OVCs. Sometimes, an OVC did not match the FAO eligibility criteria (on account of their age, mostly)
but he still included them, as ‘secondary beneficiaries,’ if they were particularly vulnerable. This facilitator had
obtained the right to farm a plot of land in the same way: he had asked a settlement chief for permission to take a
piece of unused land in use, and got it.
129
Just two of many examples of this: 1. When six women committed suicide in quick succession, in Sudok subcounty, War Child swiftly changed its target group from REFLECT groups only to the sub-county population in its
Page 114 of 131
The next step in the organisation’s maturation process is to make this learning explicit and
systematic, and to strengthen War Child’s institutional memory.
Recommendation 4
War Child UK in Uganda could improve its induction and handover processes, and would
benefit from more explicit information exchange, learning, staff development and filing
systems, and from more diverse and intensive contacts between the various types of
employees and their peers in London.
Progress in these areas would strengthen War Child’s institutional knowledge and, with this,
its potential to deepen its contributions to wider policy and practice-sharing forums.
Innovation
War Child Uganda tends to adopt and adapt innovation (such as the JFFLS model, which
originates in Zimbabwe and Mozambique), 130 rather than innovate itself – but there are
exceptions and the most obvious example of this is a (UNFPA-funded) video animation that
War Child used to facilitate discussions in REFLECT groups. It was produced for use in a
single sub-county, found successful, rolled out to all the district’s sub-counties, and
requested by ASB and UNICEF (which will hopefully lead to an even broader utilisation – but
I have not been able to verify this).
Partnership working
“War Child is deeper. Deeper in the community.” Simon Mark, a Kaabong Town Council official
responsible for child protection
With communities
Though War Child is often the only organisation doing meaningful work in remote areas, it
does not work in isolation. First, War Child very actively engages with its target communities,
and these communities have a very real say in the selection of beneficiaries. 131 Second, War
Child’s very presence in many communities in which no other organisations operate is an
illustration of its links with government authorities and other stakeholders: in consultation
with them, War Child seeks out such communities.
With donor agencies
War Child is itself being sought out as a programme implementing partner by at least two of
its funding agencies:
•
The Ministry of Gender pressured FOC-REV to partner with War Child as a Technical
Service Organisation’ (TSO), and
entirety. 2. When War Child noticed that armed forces joined JFFLS groups, the organisation asked FAO for
extra seeds, and now these armed forces are replicating the JFFLS plots inside the barracks.
130
The REFLECT methodology is also seen as innovative, but the method has been in Uganda for over 15 years,
albeit originally serving the slightly more limited purpose of adult literacy.
131
This direct engagement and impact is what War Child employees mentioned most frequently when I asked
them to tell me what made them proud to work for War Child.
Page 115 of 131
•
On the basis of UNICEF advice, FAO got a tender exemption in order to approach War
Child directly, to implement JFFLS in the Kaabong and Kotido districts.
UNFPA, UNICEF, the Ministry of Gender and FAO were all positive about War Child’s work,
reporting and level of engagement – also in comparison with peer organisations. An FAO
Programme Officer felt that War Child had achieved more in a single year of JFFLS than
many of its peers had achieved in two years. He attributed this partly to War Child’s staff, but
also partly to War Child’s organisational flexibility, which allows one person to take over if
another person is absent. 132
War Child’s programme and reporting performance creates goodwill, which in turn enables
War Child to, for example:
•
•
Provide critical feedback. The examples I came across had to do with FAO’s top down
imposition of seed choices for War Child’s JFFLS, and with the discrepancy between the
donated tomato seeds and the specifications printed on the bags.
Maintain focus on War Child’s identity and priorities. In a conversation with UNICEF I got
the impression that UNICEF does not stand much chance in its implicit attempts to
position War Child’s (yet-to-be-established) ‘One Stop Youth Centres’ as ‘UNICEF ICT
centres.’ War Child likes UNICEF’s ‘digital drum’ concept, and is happy to promote it, but
this will be in addition to rather than instead of War Child’s other plans.
With government authorities
War Child coordinates its activities with district and sub-country authorities, and has set up
monthly inter-agency district meetings that have subsequently been replicated at sub-county
level.
The contacts are frequent but challenging. In the five days in which I conducted the
assessment I met with a few able and motivated public servants, including one that had
taken the effort of coming to the War Child office in Kaabong to show me impressive
examples of his work in relation to the OVC mapping exercise and his tracking of cases of
GBV. Unfortunately, I also saw ample evidence of disengagement and a chronic lack of
resources.
This disengagement and lack of resources comes from the very top: the state ministry is
unable to secure funding for OVCs, and Uganda’s national priorities do not mention them. 133
This lack of prioritisation impacts upon War Child’s work to build the OVC-related capacity of
district and other authorities.
This work is implemented jointly with FOC-REV, is funded by USAID, and is coordinated by
the Ministry of Gender. It is a tough programme and success is uncertain as the challenges
are many and not all capacity-related. Not all public sector officials I talked with have a good
understanding of what the capacity building programme is meant to achieve. 134
132
Note that this contradicts with a statement made by one War Child staff member: “There is no back up. If you
go on leave or are sick, things stop.”
133
The four national priorities are water, education, health and roads. One of the side-objectives of the ministry’s
OVC mapping tool is that it would provide evidence of the number and location of OVCs, which in turn might
persuade the central government to allocate more funding to OVCs. This seems most unlikely, as the
insignificant budget allocation is not a consequence of the lack of data but of a lack of interest in OVCs. This is
not to say that sustained advocacy might not eventually have an impact, but that such advocacy would have to
focus on other things than mere numbers. War Child is too small to play an independent role in such advocacy
work, but might be able to gather and disseminate evidence related to the plight and potential of OVCs.
134
Specifically: I talked with a district Protection Officer and two Town Council officers, and they were all equally
hazy when describing the improvements that they thought this programme was meant to achieve. When
Page 116 of 131
The final evaluation could perhaps give an indication of the usefulness of this programme.
Even if it fails there is no real long term alternative to continuing such efforts as, ultimately,
the Ugandan government will have to take over the responsibilities that are currently
allocated to the NGO community. In the years to come, War Child will continue to struggle to
find the balance between ensuring that the authorities do their part of the work (which may
require providing government-set allowances, fuel and motorcycle maintenance services)
and keeping NGO dependency to a minimum.
Sustainability
This section covers War Child’s ability to secure and sustainably grow its funding base in
Uganda. The sustainability of War Child’s programme impact is covered in the section titled
‘Effects on poor and marginalised groups and civil society.’
The environment is tough. First, there are many NGOs in War Child’s regions of operation,
and they are competing for limited donor funding and struggling to make the transition from
emergency relief to developmental work. Second, the Ugandan authorities have mixed
feelings about the work of INGOs. On the one hand, they recognise that the government’s
own reach is limited and its problems are many, and they welcome NGOs that complement
the government’s work. On the other hand, the authorities feel that donor money is best
channelled through the government, rather than through NGOs, and the discrepancy
between government and NGO assets and salaries irritate.
Notwithstanding these challenges, War Child Uganda has managed to build a fairly diverse
donor portfolio, and half of its funding sources operate within Uganda. 135 Little of this funding
is unrestricted, and War Child is working towards but has not quite reached the point of full
cost recovery. The implication is that the organisation’s £100k unrestricted funding for 2012
(the majority of which is PPA-funded) is most welcome and is largely used to cover core
costs. 136
Efficiency and value for money assessment
“Our projects are providing intensive support to some of the most marginalised and
vulnerable children in the world. We’re cost-effective, but not cheap. We focus on quality, not
quantity.” From War Child’s website.
Many organisations specialise in one or only a few products or services (rural roads, school
feeding, health clinics) and produce or provide them at much lower unit costs than War Child
will ever do. These organisations are unlikely to reach War Child’s target group.
War Child is not one of the ‘save-a-life-for-50p’ organisations. War Child’s entry point is not
any particular product or service but the individual – the particularly vulnerable girl or boy,
unreached by any other organisation – and seeks to facilitate deep and lasting progress in
prompted, one Protection Officer said that he would much rather receive two days of focused training and a sum
of money that would enable him to do his work, than three years of coaching.
135
Recent in-country grants are from UNFPA, FAO, UNICEF, CSF and TSO. Recent international grants are
from the Barnes Femto Charitable Trust, Comic Relief, GOAC, GPEG, and a company that prefers not to be
named.
136
This PPA contribution of £90k does not imply that War Child would have had a mere £10k of unrestricted
funding without the PPA. In the last pre-PPA year, War Child Uganda had received an unrestricted allocation of
£80k.
Page 117 of 131
her or his live. This requires a costly and individualised approach, without which there would
probably be much less meaningful impact.
Some inefficiencies can be – and are being – rectified (e.g. more decentralised chequesigning; quicker dissemination of information), but most of War Child’s costs are inevitable.
Petrol is needed to cover vast distances. Cooperation with government authorities requires
several types of material support without which these authorities would not be able or willing
to leave their offices. A kilogram of second-hand clothes would be cheaper if War Child
bought several containers rather than a few small bundles, but trade in clothes does not befit
all livelihood beneficiaries and a one-size-fits-all approach would result in high economies of
scale but low impact on the ground.
There are a few options that would have indirect but significant benefits. Most prominently:


War Child Uganda would benefit from a more explicit ‘line of sight.’ Few staff members
know what War Child’s values are, or what the global strategy says, or how this global
strategy is linked with the national one, or how any of the above relates to individual
projects. The organisation is thoroughly focused on its hard-to-reach target group and I
did not see any evidence of mission creep – but I do think that there is the risk of mission
creep as long as there is not an explicit line of sight, from activities and projects to
strategies and values, of which all staff members are aware.
War Child Uganda operates in a volatile environment and needs a risk register that is
regularly reviewed and followed up on. Currently, there is no such register, and all early
warning systems are informal and implicit. 137
Recommendation 5
War Child Uganda would benefit from a well-maintained risk register, and a more widely
shared and explicit line of sight.
This recommendation is related to a previous set of recommendations about various types of
systems. The core message is that War Child Uganda has reached the size at which such
systems are a necessity. This is recognised and worked on, but the CD will not have the
time and peace of mind to give this his due attention unless some of his responsibilities –
and the time-consuming responsibility for fundraising in particular - are reassigned. I note
that in the UK, too, senior staff members end up spending a lot of time on minor
practicalities.
There are very significant cost savings to make, but making them requires action from other
stakeholders than War Child Uganda. The most significant examples of such cost savings
are:

A merger or at least much closer cooperation between War Child UK, War Child Holland
and War Child Canada. For UNICEF they formed a consortium and in Pader they share
an office, but these organisations do not pool their staff or even just their vehicles, 138 do
not share a single wireless internet system, and have three independent administrative
and salary systems. Internally, I am sure there are compelling reasons for the
137
One early warning indicator that people highlighted when I asked them to mention three War Child
weaknesses (which is a standard question I ask when conducting an organisational assessment) is related to the
low salaries and absence of extras such as hardship allowances and a providence fund. The current salaries are
based on a peer organisations’ survey that was conducted three year ago. This may be a temporary problem: a
War Child volunteer – Clare - is in the process of visiting all countries of operation in order to establish new salary
scales.
138
And cannot do this as each of the three organisations has its own insurance system.
Page 118 of 131

Headquarters of these three organisations to want to remain independent. From an
outsider’s perspective and considering the similarity in brand and image, and the scope
for cost savings, this seems wasteful. 139
Careful inter-agency planning. This rarely happens, but would save a lot of double work.
The most wasteful example I have come across is the two OVC mapping exercises, one
for FAO and one for the Ministry of Gender, the former being a sub-set of the latter.
Impact and value for money of PPA funding
Attributable impacts of PPA funding
The organisation has invested some of its PPA funding into the recruitment of a few systemfocused staff - and specifically of a worldwide M&E officer and a national HR officer.
These are important investments as War Child has reached a size that requires strong
systems to ensure good Value for Money. Without such systems, there is bound to be
tension, and there will continue to be expectations that are not met. Reporting is an example:
just insisting that ‘the reports are submitted in time’ may not result in more punctual reporting
and if it does it is likely to be at the cost of disproportional staff time. With such systems,
reporting would not take a disproportionate amount of time of national staff, and the London
office could spend less time on gap-filling and more time on recognising and celebrating War
Child’s considerable achievements. The way forward is a more consistent reporting system
that people are aware of and can work with. Similar arguments apply to other system gaps,
and some of the PPA funding is allocated to fill a number of such gaps.
The final evaluation would be able to assess the extent to which these PPA-funded staff
additions have led to the development of and adherence to such systems.
Value for money assessment of PPA funding
Some of the PPA funding has been invested in a few new strategic positions, and this
investment has potential to strengthen War Child’s Value for Money performance. The
remainder of the money – and this is most of it – has been utilised to cover core costs. This
does not represent the best possible value for money. Recognising this, War Child Uganda
is working towards a full cost recovery system. The final evaluation will be in the position to
assess the extent of success.
End of country report.
139
The three boards are at an early stage of developing a future vision for War Child International.
Page 119 of 131
Annex G4: Country programme report 4: KYBT, member of YBI,
Kenya
Final version, 17 August 2012
Contents
Assignment ..................................................................................................................... 121
Results ............................................................................................................................. 121
Performance assessment against logframe ................................................................... 121
Effects on poor and marginalised groups and civil society ............................................. 121
Relevance ........................................................................................................................ 122
Representativeness ....................................................................................................... 122
Targeting strategy ......................................................................................................... 122
Effectiveness ................................................................................................................... 123
Learning ........................................................................................................................ 123
Innovation ...................................................................................................................... 124
Partnership working ....................................................................................................... 124
Sustainability ................................................................................................................. 125
Efficiency and Value for Money assessment ................................................................ 126
Impact and value for money of PPA funding ................................................................ 126
Page 120 of 131
Assignment
This assessment covers the Nairobi-based operations of KYBT. It is part of a much
larger ‘PPA mid-term evaluation’ that covers the work of Restless Development, War
Child and Youth Business International. The assessment took three days. In the
course of these days I talked with all Nairobi-based KYBT staff; three trustees;
beneficiaries; some of their competitors; mentors; and people who had dropped out
in the various stages of the pipeline from the initial outreach activity to the KYBT
loan.
I would like to thank Eunice and her team for their hours of support, and for bearing
with me on that exceptionally long Tuesday 14th.
This report follows the standard IPR structure.
Results
Performance assessment against logframe
The logframe’s outcome indicator 1.1, milestone 1, covering ‘target countries’: 4% increase in young
people employed or setting up a successful business who are able to contribute to household income.
KYBT’s beneficiary portfolio as per 14 August 2012: 28 people, soon to be growing to 30.
KYBT is an organisation that very nearly collapsed between 2009 and 2011, and is now
recovering. The Nairobi-based part of the organisation only has one substantial grant. 140 It is
from Comic Relief, was facilitated by YBI, and amounts to £255k over three years. KYBT is
expected to use this grant to support 100 business start-ups, and is on track to achieving it.
If it does, this will contribute 0.1% to output indicator 1.1: the number of young people
accessing business or employment schemes and training through the consortium (i.e. 100
out of 84,444). It is reasonable to count these 100 entrepreneurs towards YBI and thus the
consortium, even though it is achieved by KYBT, as KYBT would not have survived without
YBI’s support.
Effects on poor and marginalised groups and civil society
My children got sick and I lost two months of income. Still, I will pay off my loan in four
months. After this, I will save money to buy an apartment. Ridah Akiny, who got a KYBT loan
and is now selling tomatoes on a Kibera market.
KYBT is re-establishing itself and does not yet have an effect on groups or civil society.
Instead, it focuses on its 28 individual beneficiaries, and on identifying others.
140
The Mombasa-based part of the organisation is implementing a larger programme, funded by the Islamic
Development Bank. I did not assess this programme.
Page 121 of 131
The effect of most of these 28 individuals has been significant. They have started – or
sometimes re-started – microbusinesses in the Kibera slums. These businesses tend to
create a profit of some £4-8 per day, before deducting rent and loan repayments. Three of
these starting entrepreneurs employ one person each. Their activities do not generate a
significant trickle down or multiplier effect, as the KYBT beneficiaries do not engage in
activities that add to the area’s economic activity – they merely spread this activity among
more people.
Life in the Kibera slums is volatile. In some cases, entrepreneurs adapt successfully: a shop
keeper successfully dealt with cooking oil price increases by diversifying into vegetables,
clothes and cooked beans. In other cases, beneficiaries failed to set up a business: one
beneficiary disappeared after being associated with a crime that involved a murder, and
another beneficiary did not open his bookshop and instead recorded music that failed to
generate an income. Yet other beneficiaries had to cope with disease and death of relatives,
and/or a husband with a drinking problem, and this affected their opening hours, investment
plans and repayment schedules. As long as KYBT feels that the struggles are genuine, the
organisation does not apply much pressure to defaulting borrowers. This is probably wise,
but also leads to the combination of intensive hands-on support and, nonetheless, a
relatively high default rate (currently standing at some 20 per cent 141, with the typical Kenyan
microfinance target of five per cent of less). 142
Relevance
Representativeness 143
The training was very good. Before this, I had an interest in opening a business, but no
knowledge. Now I have the knowledge. Mary Kuvuva, who finished her four-day KYBT training for
starting entrepreneurs, and produced a business plan. She is now awaiting her interview.
Understanding and responding to the needs and priorities of entrepreneurial Kibera
youths. KYBT maintains much stronger links with its beneficiaries than the typical
microfinance organisation. The road towards a loan is a thorough one that includes four days
of training and other support; and once KYBT has issued a loan it frequently monitors
progress and assigns a mentor for coaching and guidance. In the course of the six-month
repayment period, a beneficiary receives some ten visits on average, from his or her mentor
and from KYBT’s two loan officers. The mentors and a surprising number of the KYBT staff
have deep insight in the lives of the beneficiaries, and the challenges they face, and KYBT
tailors its support and expectations accordingly.
Representing the needs and priorities of entrepreneurial Kibera youths. A few years
down the road, KYBT might be able to develop advocacy messages on the basis of the
evidence it is currently creating. For now, this is not the case.
Targeting strategy
141
Eunice, could you check this percentage please, as it is not more than a gestimate?
I got this percentage from a conversation with Lydia, a Faulu microfinance supervisor and KYBT mentor.
143
Coffey’s note on this heading is that it is about “The degree to which the supported civil society organisations
[this refers to YBI, and possibly to its members] represent and respond to the needs and priorities of their
constituencies, (including where relevant the poorest and most marginalized).”
142
Page 122 of 131
We do not give them an application form right away. Instead, we ask them to wait for a week
and then get a form from somewhere. If we distributed our forms immediately, everybody
would be interested. Priscilla Kendi and Faith Kavivi, KYBT Field Officers.
KYBT is bound by the Comic Relief grant conditions, and this means that it focuses
exclusively on start-ups, and exclusively on young people living in Kibera.
First and in close consultation with the District Youth Officer, KYBT identifies appropriate
times and locations for its outreach activities. After this, KYBT employs a system of selfselection:
•
•
To ensure that the applicants are genuinely interested, application forms are not made
available immediately but only a week later.
People can only apply for a loan if they first attend at least three days of a four-day
training programme, and if they produce a business plan.
In each step on the way, people drop off. The remainder is motivated – though, as
mentioned, there are still people who do not utilise the loan in line with the business plan,
and who do not repay (and stopped answering KYBT phone calls).
Effectiveness
Learning
The YBI mentoring training was very useful – we learned a lot. Modesta Ongweso, KYBT’s
Training and Mentoring Manager.
A year ago, KYBT essentially started a new life with new staff and a new focus, and the
organisation is therefore inevitably learning while doing. This learning is supported by YBI’s
material, distance coaching and visits. This support led to:
•
•
•
•
A focus on the key issues that need to be addressed to turn KYBT into a fully functional
and low risk organisation. YBI’s external assessments and recommendations set much
of the Board of Trustees’ agenda.
Sound processes, with the help of swim lane diagrams. KYBT appreciates but does not
blindly follow YBI’s process-related advice, and modifies these swim lanes to befit
KYBT’s environment. 144
Sound (if sometimes a little excessive) online information gathering, on the basis of a
Salesforce OMS. This is one of the core benefits of YBI membership. Again, this OMS is
modified to befit KYBT’s environment. 145 A good next step would be to ensure equal
filing rigour of things that are not part of Salesforce. Procurement documentation, for
example, is not yet systematically filed in a single system.
Good products. KYBT’s mentors much appreciate the mentor training, and this training
is a YBI product.
144
For example: the original mentor recruitment process included interviews with candidates. KYBT took it out as
this step did not seem necessary in KYBT’s process. Over time, additional modifications are likely to be made –
in the case of Kibera, for example, the system of ‘one-mentor-one-mentee’ seems to be inefficient.
145
For example: KYBT’s Salesforce version distinguishes between ‘leads’ (those who attended an outreach
event) and ‘contacts’ (those who subsequently submitted an application form).
Page 123 of 131
KYBT benefits from and contributes to the YBI’s learning systems. For example: KYBT
participates in the periodic worldwide online YBI meetings in relation to Salesforce. It is now
time for KYBT to develop a few internal learning systems too – and these systems could
take a variety of forms (e.g. periodic learning gatherings or retreats, professional
development plans, an early warning system, a risk register, M&E plans, or formalised
information exchange.) There is clear eagerness to develop some of these systems in the
near future, and the PPA final evaluation is likely to see evidence of the results.
Sharing learning is not yet on KYBT’s radar, as this requires generating evidence that is then
translated into messages that could be useful to other stakeholders. KYBT is not in this
stage yet.
Recommendation 1
KYBT has a learning mentality, but no learning systems yet. Introducing such systems would
strengthen and accelerate learning processes.
Innovation
Membership is knowledge. The mentoring programme in the Dominican Republic worked
well – so I Skyped the person there, and she gave me some coaching. Eunice Ogolo, KYBT’s
Programme Manager.
KYBT is not – yet - an innovator. Instead, KYBT copies and modifies good practice that
exists already. It get inspiration, advice and products from YBI (e.g. YBI’s Salesforce OMS,
mentoring training) but KYBT does not limit itself to YBI-related products and has, for
example, adopted ILO’s entrepreneurship training model for East Africa.
Partnership working
KYBT invited me into their board. I go there almost every month for these interviews with
beneficiaries. I don’t mind: we are partners. Elector Opar, District Youth Officer Langata (of which
Kibera is part).
The section on ‘Learning’ covers YBI’s role in the capacity development of KYBT. In
addition, YBI has secured all of KYBT’s funding. Put bluntly: without YBI the organisation
would not exist today.
YBI is not KYBT’s only partner. Within Kibera, KYBT works closely with the District Youth
Department, and the department’s Head is part of KYBT’s Regional Board. This close
cooperation is appropriate as, without involvement of the District Youth Department, KYBT
would be working in isolation.
In the coming few years, there are three other partnership options that KYBT may want to
explore. Specifically:
1. There may well be scope for cooperation between KYBT and YBI’s members in
Tanzania and Uganda, once these members have been identified and accredited.
Similarly and already possible, KYBT may benefit from the years of learning of YBI’s
Page 124 of 131
members in Sri Lanka and India, where the beneficiaries have profiles that are
comparable to the beneficiary profiles in Kenya. 146
2. Within Kibera, KYBT could usefully join relevant inter-agency forums. In the immediate
future, KYBT is likely to be largely listening, but if operations continue and expand then
KYBT will soon be able to make useful contributions within the inter-agency discourse on
employment and entrepreneurship.
3. KYBT is not a lending institution and does not have the network required for the efficient
selection of beneficiaries. As soon as KYBT operates at a scale that might be of interest
to lending institutions, the organisation should spin off this part of its services, and focus
instead on training, coaching and mentoring.
Recommendation 2
Spinning off the lending activities to a specialised lending institution would enable KYBT to
achieve economies of scale and to concentrate on KYBT’s core competence: training and
mentoring of starting entrepreneurs. KYBT would have to pay this organisation or company a
substantial premium because of the risks related to working with start-up businesses in
disadvantaged communities – but this premium would be several times lower than KYBT’s
own overhead costs per borrower.
Sustainability
Without YBI, KYBT would be dead now. Eunice Ogolo, KYBT’s Programme Manager.
YBI wants its members to be financially sustainable. In the case of KYBT, this is not yet
realistic. To gain access to funding, KYBT requires at least two things:
•
•
A track record of success. This track record is currently being built.
Positive audit reports. This has been difficult, as the previous KYBT management
misappropriated KYBT funding, blocked its financial records, and disappeared. It has
taken time to retrace these records, and it is only now that KYBT feels ready to invite
auditors back into its office to try, again, to conduct an audit for 2009-2010 (which is a
condition to get the 2010-2011 and 2011-2012 audits done.)
For several more years, KYBT’s ability to continue and to grow its operations will very nearly
entirely depend on YBI’s ability to arrange funding. YBI will not find this an easy task. The
track record that KYBT is currently developing has an overhead costs that amounts to some
ten times the value of the lending portfolio (£225,000 over three years, for some 100 loans
with a total budgeted value of £80,000). A unit cost of some £2,250 for micro-business startups that have no more than 80% chance of survival is very high indeed, and a best-case
cost-benefit analysis (see annex C2) shows that a £1.00 investment will lead to not more
than some £1.44 in net income.
The only funding agency that might be interested in funding KYBT would either not make this
type of calculations or see strategic long term value in the existence of KYBT (as is the case
with YBI, which is the reason that YBI invests so disproportionately heavily in KYBT’s current
operations). 147
146
Already, KYBT has compared notes with YBI’s partner in the Dominican Republic, and found it useful.
The KYBT trustees presented the argument that part of KYBT’s usefulness lies in the training it provides to
people – even if these people do not subsequently develop a business plan and secure a loan. There is some
truth in this, but it would be incorrect to give this a significant monetary value as the barrier to KYBT’s loan facility
147
Page 125 of 131
Efficiency and Value for Money assessment
With that budget, we would reach 1,000 people, with a three-year business survival rate of
some 70-80 per cent. Brian Andanje, reflecting on KYBT’s costs effectiveness. Brian works for a
microfinance company and is a KYBT mentor.
Economy. 148 The basic checks and balances might be in place but are not yet consistently
documented. For example: the procurement files are not kept in a single location and do not
show a sufficiently clear rationale for the choice among procurement options. The audits that
will be conducted in the next few months are likely to suggest ways in which KYBT could
progress towards (or at least make more explicit) maximum economy of its operations.
Efficiency. 149 KYBT employs – and limits itself to - the typical YBI mix of training, mentoring
and loans. This is likely to change if and as the organisation matures and becomes less
dependent on YBI and a single donor agency. In the long run, KYBT will ideally explore
different options and select the most cost-effective one.
Within the current set-up, additional funding would enable KYBT to drive down its unit cost of
business support. The reason is that the overhead costs are set by an organisational
structure that is designed for operations that are much larger than the annual 33
beneficiaries that KYBT needs to reach. The operations could probably at least triple in size
without significant increase in overhead costs, which means that a tripling in size would
reduce the unit costs to much less than half what they are today (i.e. a little of £1,000
compared to £2,250, assuming that the cost of additional beneficiaries amounts to 150% of
the average loan value).
Effectiveness. 150 The figures and field visits conducted in the course of this evaluation
confirm that KYBT’s operations are on track to meet the requirements of the organisation’s
two grants. Following Coffey’s definition of effectiveness (“‘Effectiveness’ refers to how far a
programme achieves its intended outcomes”), this means that KYBT is entirely effective.
Impact and value for money of PPA funding
Without PPA, we would not have been working with KYBT quite so much. Sietske van der
Ploeg, Programme Manager, YBI
YBI has not allocated a portion of its PPA funding to KYBT. The implication is that, for this
assessment, the impact and value for money of YBI’s PPA funding depends on the PPA role
in YBI’s support to KYBT.
is probably lower than any barrier to financial services any Kibera resident will ever face, and if the trainees do
not access this facility now they are unlikely to access financial services in the future (and, moreover, if they do
they are likely to be trained again).
148
‘Economy’ refers to the costs of inputs and resources of an intervention. The extent of ‘economy’ is typically
measured by looking at procurement procedures and by calculating unit costs.
149
‘Efficiency’ refers to how much you get out in relation to what you put in. It’s about maximising an output for a
given input, or minimising input for an output.
150
‘Effectiveness’ refers to how far a programme achieves its intended outcomes, using qualitative and
quantitative assessments of change.
Page 126 of 131
Even if YBI’s support to KYBT has been inspired by the PPA and nothing else, this support
does not represent the best possible value for money if we look at the short term results, as
the investments have been considerable and the number of people reached has been small
(30, since the start of the PPA, with another 70 planned for the next two years). This picture
potentially changes if you consider the long term value of a bright future that may be in store
for KYBT – and this is for PPA’s final evaluation to assess.
Recommendation 3
YBI is not meeting all demographic targets that it lists in Appendix 3 of its Strategic Plan
2011-2014 (and particularly its gender target and its ‘developing countries’ target). Losing
KYBT would widen the discrepancy between targets and reality. YBI will lose KYBT if YBI
does not help the organisation to strengthen its funding base. Additional funding and funding
sources would have the additional advantage of driving down KYBT’s very high overhead
percentage and unit costs.
End of country report.
Page 127 of 131
Annex H: Details of the evaluation team
Lead consultant: Willem van Eekelen
Willem is a development economist who worked for a range of multilateral organisations and
NGOs. Most of his work has been programme-related, always in the fields of development
cooperation and humanitarian work and often with a focus on youth and livelihoods.
As an independent consultant, Willem regularly conducts evaluations of DFID-funded
programmes - in the past year he did this for Oxfam, Save the Children, ACF, CIDT, VSO
and this consortium. In addition to his consultancy work, Willem advises philanthropists,
lectures at the University of Birmingham and co-owns Green Visions, the Balkans’ liveliest
adventure tourism company.
Email: [email protected]
Tel: 079 3189 6359
Backstopper: Jill Edbrooke
Jill specialises in monitoring and evaluation of gender- and development-related
programmes. She has worked with and for a wide range of public and voluntary
organisations ranging from small NGOs and social movements to the world largest donors.
Before specialising in development cooperation, Jill worked in the fields of youth
empowerment and participation.
Email: [email protected]
Tel: 077 6218 4443
Division of responsibilities
Willem conducted the assessment and wrote the report. Jill served as Willem’s sparring
partners and reviewed his inception and assessment reports.
Page 128 of 131
Annex I: PPA organisation’s management response to report’s
findings (post-submission)
Restless Development (the lead Agency of the Consortium for this DfID PPA) has set out
this management response to the above report put together by Willem van Eekelen. The
Consortium partners, War Child and YBI, have inputted to each stage of the report and their
feedback into the facts and findings have been considered throughout.
Support for the Review:
From the outset Restless Development has actively and positively engaged in the
Independent Progress Review (IPR) and this consequent report, treating it as an intrinsic
part of our learning and growth (words that we are pleased to see are recognised by Mr Van
Eekelen throughout). In this spirit we:
- Have only minimal management responses below that aim at counter-balancing
some of the findings with our broader and internal perspective
- Have already invited Mr Van Eekelen to present his findings in a full session with the
Board of Trustees of Restless Development without any prior briefing from us the
Directorate. Mr Van Eekelen did that last month, setting out the strengths and
recommendations as he has found them in his review. The Trustees along with the
Executive will now be taking all of the recommendations throughout this report and
embedding them into our planning and management going forward
Summary:
In summary we are extremely pleased with the findings of the report. We note in particular:
- Transformative Impact of PPA: The transformative impact of the PPA on each of
the Consortium partners specifically in the benefit of flexible income to allow us to
over deliver against the outputs, innovate, increase our financial sustainability and to
invest in both core costs (though see “specific issues” moving forward below) and
programmatic achievements
- Broadening our influence: The exponential increase in our shaping of policy and
practice nationally (see the specific Restless Development reports in Uganda and
Zambia in Annexe G1 and 2) and internationally
- Line of sight: The repeated “line of sight” across Restless Development noted in this
report, specifically from the most junior staff and volunteers through to global senior
management, in the understanding and drive for the same goals and values
- Value for Money: The strong conclusion throughout, as Mr Van Eekelen states on
p6 that it is not just that for example, “There is something fundamentally low-cost
about the work of Restless Development” but that “overall, this consortium provides
very good value for money already”
Specific Issues
The following issues from the report do require particular responses:
- Throughout Mr Van Eekelen has been clear that his method for measuring
additionality of the PPA to the consortium has focused on traceable financial
investment. He has been upfront about this in the statement of his bias in this
manner on p16. We would argue very strongly that:
o From the outset with Restless Development’s lead the additionality of the
PPA has been the focus of all planning – including narrative plans, structures
and budgets
o The Consortium has set up a Working Group to monitor those plans of all
three organisations – acting as a fourth entity to monitor and approve plans of
all three organisations equally.
Page 129 of 131
The plans have been goals and outcomes focused. Mr Van Eekelen
mentions the four goals set up by the Consortium. The targets of success of
those four goals 151 have been exceeded in all cases – evident throughout the
report.
On page 5 Mr Van Eekelen proposes that Restless Development’s present reliance
on various Bi-lateral (institutional) funding risks the “independence” of the
organisation. Restless Development is extremely focused on diversification of its
funding and has a 5-year globally agreed business plan in place with targets already
being exceeded as noted in the report. However it is worth stating also that the
continued high levels of investment from a varied cross section of Bi-lateral investors
do not risk the independence of Restless Development. We approach each
partnership freely and with our own strategic parameters as non-negotiable aspects
of any agreement. Indeed strategic investment, pre-agreed and contracted over
multiple years (such as we have from DfID, Sida and so on) allows for more
independence and strategic focus often than very hands on personally driven
individuals and small trusts
Mr Van Eekelen notes that we have achieved or exceeded nearly all of the
consortium’s output targets and confirms that the M&E is mostly rigorous and
accurate. He does however note an example of double-counting of beneficiaries
through our monitoring processes. In this specific example we believe that this is to
do with the same young people accessing separate livelihoods, sexual health and
civic participation sessions. This will be looked into. It is important to add that Mr Van
Eekelen notes that even correcting for those examples, the PPA targets would have
been reached.
On Page 35 (and later in Annexe G1 and G2) Mr Van Eekelen recommends Restless
Development moves its financial year from its October to September cycle to the
calendar year, to fall in line with most of the rest of the world. We do not think this is
necessary as there are very few challenges faced by our financial year cycle and we
also have partnerships with organisations that also have financial years that do not
follow the calendar year – the British government being one. We will of course
always make sure we can negotiate contracts that work in our cycle as much as is
possible
Mr Van Eekelen also notes that we use multiple finance software – Pastel and Excel
– for our finance management for instance. While in the specific case that he
mentions in Uganda – of a staff member from a Karamoja field office having to spend
a day in the head office converting accounts from excel to Pastel, in fact globally this
is not an issue: Restless Development rather uniquely uses the same financial
software across all of its offices (Pastel), uses the same chart of accounts – that are
set up with codes related exactly to our Global Strategic Framework logframe so that
finances can easily be consolidated across all of our Country Programmes and
quickly show us costs for Value for Money assessments. The particular issue for the
field office in Karamoja, Uganda will be followed up.
On p89 in Annexe G1 and again on p109 in Annexe G2, Mr Van Eekelen rightly
notes the need for more long term strategic funding for Uganda and Zambia
respectively. I am pleased to announce that in both countries long term donors have
been secured in the past month including multi-year agreements with Marie Stopes
and USAID in Uganda, and ZGF (which Mr Van Eekelen mentions was a potential
transformative decision coming on line) in Zambia. We believe these have been as a
result of strategically investing PPA funding in each of these flagship Country
Programmes in the last year allowing them to deliver excellent results in programmes
o
-
-
-
-
-
151
To exceed the expectations of the logframe; to increase financial sustainability; to increase the presence of
the consortium as the go-to experts in youth development; and to increase the capacity and capabilities of the
three organisations
Page 130 of 131
-
while exploring these opportunities with partners. It will give us the opportunity to relook at investment of that aspect of the PPA in Year 3
On p92 Mr Van Eekelen mentions that 4 out of 6 of the Uganda Management
Committee are ex-patriots. This is true. It is relevant to state furthermore that
Restless Development has policies around this that are adhered to across the
organisation – and means that when the balance is weighted towards national or
international staff (as is the case in Uganda), concerted efforts will be made in future
recruitment, to which the Country Programme will be accountable. It is worth noting
as well that two of those international staff are Zimbabweans who are former
volunteer development professionals from our Zimbabwe programmes and so
standard bearers of our global strategic goals for our generation of leaders
programmes
Aside from these small, specific issues, Restless Development would like to re-iterate on
behalf of the consortium our pride and satisfaction at the first-year report that was submitted
by the consortium, expressing what we have achieved as a result of the PPA, and the
affirmation of those successes in this report from the Independent Progress Review. At the
same time we would like to re-iterate our determination to continue to grow and learn, and
work on behalf of young people who make up the largest demographic in relation to all
aspects of poverty, bar none. Our goal is to continue to place young people at the forefront
of change and development and, with the support of the UK government through DFID, we
will continue, as Mr Van Eekelen notes, to “punch above our weight”.
Finally, our gratitude to Mr Van Eekelen who has exceeded all expectations in the rigour and
interest with which he took on this review. His commitment to the cause, his desire to help
us learn and improve, and his professionalism in the completion of the report, have been
outstanding.
Page 131 of 131