Read the report - Clore Social Leadership

Transcription

The Clore Social Leadership Programme
IMPACT MEASUREMENT PRACTICES
OF LARGE CHARiTIES
Beth Green
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 20 21
IMPACT MEASUREMENT PRACTICES OF LARGE CHARITIES
Acknowledgements
Contents
I would like to thank the following people for their
time, support, insights and contributions to this
study:
Executive summary3
1
Introduction • Eibhlín Ní Ógáin and Tris Lumley from NPC;
1.1 Background
5
• Rhian Beynon and Sarah George from Family Action;
1.2 What is impact measurement?
6
1.3 Features of large charities that may
affect their impact measurement practices
6
2
Methodology
7
3
Findings9
• Ann Kendal and Lorna Jacques from The Children’s
Society;
• Karl Demian from WRVS;
• Sarah Hollaway and Beth Murphy from Mind;
• Liz Ward, Debby Mulling and Alison Roche from the
British Red Cross;
• Lindsay Hodgson, Louise Pritchard and Pamela Newman
from Action on Hearing Loss;
• Matt Barnard, Richard Cotmore and Ingrid Anson from
NSPCC;
• Siobhan Edwards from the Clore Social Leadership
Programme.
I would also like to thank the charities mentioned above
for allowing me to have access to their staff and expertise.
3.1 Why do large charities measure impact?
5
9
3.2 Challenges of impact measurement practices faced by large charities
11
3.3 Enablers in developing impact measurement practices
15
3.4 Benefits of measuring impact practices
18
4
19
Conclusions and Recommendations
Bibliography
21
Appendix
21
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Executive summary
Over the last decade an increasing number of
charities have realised the importance of effective
impact measurement. In the UK, large1 charities
account for 77% of the voluntary sector’s annual
income (NVCO, UK Civil Society Almanac, 2012).
This significant proportion gives large charities a
critical role in leading the sector in developing
impact measurement practices.
its mission and core functions and how it develops
its impact practices.
mission and approach and often the charity’s
theory of change had been a good mechanism to
achieve this coherency. Visible leadership in the
The study found that leadership is critical in
development and embedding of impact
overcoming organisational and culture inertia
measurement reinforced to front-line staff the
issues caused by a charity’s size and age. A charity’s
charity’s commitment to measuring impact.
size and age have to be taken into account as well
as external environment and funders in order to
Size
have success in developing and embedding impact
The greater distance between management and
The purpose of this study was to look at the impact measurement practices.
front-line staff mean large charities often struggle
measurement practices of large charities, with the
Leadership
to find an impact measurement approach that
aims of identifying and understanding the drivers,
balances the needs of front-line staff and senior
challenges, enablers and benefits of developing
Larger charities often have a greater number of
management. Front-line staff motivations differ
them. This study is based on interviews with staff in diverse services covering a larger geographical area
from management; they are primarily driven by a
seven large charities that have had success
and employ more staff (and volunteers) than their
desire to know whether services are improving
developing and embedding impact measurement
smaller counterparts. Large charities need
outcomes for beneficiaries whereas management
practices2. To ensure a fuller organisational
coherency between different functions and services
wanted to know the impact of the service and
perspective, interviews were conducted with frontin order to measure in a more meaningful way.
charity as a whole. Open dialogue and
line staff and ‘impact champions’ in senior
Leadership was found to be key to bringing these
communication ensures that the impact
positions from each charity.
services and employees together in order to drive
measurement approach meets both the needs of
the development of impact measurement practices.
Large charities are very diverse with different
front-line staff and management. Involving frontIn many of these charities the driver to measure
missions, services, geographical coverage and
line staff in the development of impact practices
impact had come after a change in chief executive
structures. However, by the nature of their size,
results in the measurement being more
or senior management; in all cases there had been
larger charities have a number of features that are
proportional and integrated with the day to day
a change in strategy. The new strategy had
more prevalent than in smaller charities. These
running of services.
supported coherency across the organisation’s
features are likely to affect how the charity delivers
Defined in this study as charities with incomes in excess of £10m.
Success is defined as routine use of impact measurement practices.
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Executive summary
Age
Recommendations
Large charities are generally older with more
established brands, systems and processes than
smaller charities and so longer histories make it
more difficult to change culture and adopt new
practices. Larger charities are more successful in
developing impact measurement practices when
they acknowledge this and ensure appropriate
resources. Organisational ‘buy in’ is obtained
through regular communication, appropriate
training, effective IT systems and realistic time
scales.
As a result of the finding of this study, it is
recommended that:
External environment and funders
• Chief executives, trustees and senior
managers of charities should commit to
measuring impact and offer clarity in
• Senior managers and front-line staff should
work together to design and embed impact
measurement practices.
• Charities should work with funders to agree
an impact measurement approach.
Charities reported that funders and other external
• Charities should take into account their
environment interest in impact measurement had
history, size and complexity when
grown over the last few years and this acted as a
developing and embedding impact
driver for charities to develop their impact
measurement practices. Large charities often have
a wider range of funding from a wider range of
sources than their smaller counterparts. This diverse
portfolio of funding results in different reporting
requirements, which makes it difficult for charities
to have a coherent approach to impact
measurement.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
1 Introduction
1
1.1 Background
Over the last decade an increasing number of charities
have realised the importance of effective impact
measurement. This practice is key to helping charities
demonstrate what they are achieving and learn how
to improve (NPC’s A Journey to Greater Impact,
2011). NPC found in its Making an Impact report
(2012) that large charities have different practices to
those of other sizes.
The purpose of this study, through qualitative and
secondary research, is to identify and understand the
drivers, challenges, enablers and benefits of
developing impact measurement practices in large
charities. It also offers recommendations, based on
these findings, for large charities wishing to develop
their impact practices.
In the UK there are over 170,000 charities, of which
474 are defined as large3. These large charities have a
combined annual income of 77% of the voluntary
sector’s income (NVCO, UK Civil Society Almanac,
2012). This significant proportion gives large charities a
critical role in leading the sector in developing impact
measurement practices. NPC’s Making an Impact
report (2012) showed that charity size affects impact
measurement practices. It demonstrated that 89% of
large charities are very positive about impact
measurement, which they believe makes them more
effective. However, the report also showed that even
3
Defined in this study as charities with incomes in excess of £10m.
Success is defined as routine use of impact measurement practices.
4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
though impact practices are more embedded and
advanced in larger charities, there are still those for
which impact measurement practices have not become
routine. Indeed, it claimed one in ten charities with
incomes over £10m did not measure impact at all.
This study is based on interviews with seven large
charities that have had success developing and
embedding impact measurement practices4.
Interviews were conducted with front-line staff and
‘impact champions’ in senior positions in each charity.
The interviews elicited their opinions on the drivers,
challenges, enablers and benefits they had
experienced when developing their impact
This report is written as part of the author’s Clore Social
Fellowship. Each Fellow is given the opportunity to
undertake a piece of practice-based research, as part of
their Clore Social Leadership Programme. The purpose
of the research is to help develop the evidence base for
the sector as a whole. NPC has supervised this study,
providing support in the areas of research design,
analysis of interviews and report writing.
Introduction
1.2 What is impact measurement?
Impact measurement can be defined as the set of
practices through which an organisation establishes
what difference its work makes. This term can be
used to define a broad set of practices that include
both measuring outputs and outcomes5.
The process of measuring impact includes
everything from gathering and analysing data to
demonstrating findings.
1.3 Features of large charities
that may affect their impact
measurement practices
Large charities are very diverse with different
missions, services, geographical coverage and
structures. However, by the nature of their size,
these charities have a number of features that are
more prevalent than in smaller charities. These
features are likely to affect how a charity delivers its
mission and core functions and how it develops its
impact practices. They include:
• Having a greater number of services, which are
There are many ways to measure impact. Charities
often more diverse in design and focus, and cover
interviewed used a huge variety of tools and
a larger geographical area.
methods to establish impact. These include random
control trials, distance travelled tools, follow-up
• Employing more staff and volunteers to deliver
questionnaires, SROI (social return on investment)
the work resulting in more complex staffing
studies and evaluation interviews.
structures.
• Being older with more established brands,
systems and processes.
• Having more unrestricted income.
• Having a wider range of funding, including
donations, earned income, grants and statutory
funding, from a number of different sources.
5
Taken from NPC’s 2012 Making an Impact report.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
2 Methodology
2
This research builds on the findings of NPC’s Making
crisis is over, it helps people to recover and move
an Impact report, which looked at impact
on with their lives. Income £213.8m
measurement among charities and social enterprises
•T
he Children’s Society: Dedicates itself to
in the UK. The report highlights different impact
making childhood better. Community
measurement practices in the sector, breaking down
programmes provide immediate and lasting help
its findings into small, medium and large charities.
for children who feel isolated or abandoned; who
Seven charities were selected to take part in this
are faced with a lifetime of disadvantaged in their
study. They were chosen on the basis that they each
daily lives and unable to find the support they
had an annual income in excess of £10m, and have
need anywhere else. It influences thinking, from
successfully developed and embedded impact
the general public to politicians about what
measurement (success being defined as routinely
needs to change to improve the lives of children.
using impact measurement practices within their
Income £42.8m
services). The charities were:
•F
amily Action: Preserves and protects the good
• Action on Hearing Loss: Promotes and
health (in particular the mental health) of
encourages the prevention and mitigation of
families, individuals and groups within the
deafness and the better treatment, education,
community and provides advice and financial
training, employment and welfare of people who
help for families in need. Income £18.5m
are deaf or hard of hearing, and generally aims to
•M
ind: Provides information and support,
promote, safeguard and protect the welfare of
campaigns to improve policy and attitudes and,
such people. Income £37.5m
in partnership with independent local Mind
• The British Red Cross: Helps people in crisis,
associations, develops and provides local services.
whoever and wherever they are. It is part of a
Income £29.9m
global network of volunteers, responding to
•N
SPCC: Prevents the public and private wrongs
natural disasters, conflicts and individual
of children and the corruption of their morals;
emergencies. It enables vulnerable people at home
takes action for the enforcement of laws for their
and overseas to prepare for and respond to
protection; provides and maintains an
emergencies in their own communities. When the
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Methodology
organisation for the above objectives; to do all
other such lawful things as are incidental or
conducive to the attainment of the above
objectives. Income £135.7m
• WRVS: Delivers practical support through the
power of volunteering so that older people can
get more out of life. WRVS aims to relieve poverty,
distress and suffering in various sections of the
community, primarily in the UK, by the provision
of various services for individuals; support to
victims of emergencies and disasters; and such
other purposes for the benefit of the community.
Income £74.7m
Each charity was asked to nominate at least two
individuals to be interviewed:
• A front-line member of staff who had regular
client contact and was involved in implementing
impact practices.
structured interviews with a total of 16 individuals.
All interviews lasted between 40 and 90 minutes.
They were recorded and transcribed, before being
examined using thematic analyses. A copy of the
interview schedule and questions can be found in
Appendix A.
Quotes in this report have been anonymised but
include reference to the contributor’s role (frontline staff or manager).
NPC has supervised this study, providing support in
the areas of research design, analysis of interviews
and report writing. NPC occupy a unique position
at the nexus between charities and funders, helping
them achieve the greatest impact. They find
solutions to the challenges they face, where they
are trying to work more effectively exploring new
ways of working, or setting out to prove their
worth.
• A senior member of staff based in the charity’s
head office who was deemed an ‘Impact
Champion’ (referred to as managers throughout
the report).
A series of questions were developed building
on NPC’s Making an Impact report. These
questions were used to form the basis of semi-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
3 Findings
3
3.1 Why do large charities
measure impact?
When the interviewees were asked what triggered
the charity to measure its impact, the following
drivers were identified:
• Change in leadership and strategy change;
• Desire to know whether they are making a
difference (front-line staff);
• Desire to show effectiveness to funders;
• Pressure from the external environment;
• Wish to lead/contribute to the debate about
what is working; and
• Moral obligation.
The main drivers for managers that were identified
through the interviews were leadership and
strategy changes. In many of the charities there
had been a change in chief executive or senior
management; in all cases there had been a change
in strategy.
‘The drive came from our board, they were
presented quarterly with lots of output data and
activity data, and kept asking questions like
“What’s the impact of all of that activity?” Then
it was given as a challenge to someone.’
Manager
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
In contrast to managers, stronger drivers for frontline staff were the desire to know if the services
were improving outcomes for clients and being
able to inform local decisions and agendas.
Managers also saw the impact measurement
practices as allowing them to lead or be part of
the debate about what worked.
‘From a project level the driver is to actually
show that we’re making a difference and
whether we are making an impact and to see
how we can improve and to share that with
the younger people that we work with.’
Front-line staff
‘It’s about two things, double-checking that
what we’re doing is what we think we’re doing;
we’re making assumptions about the benefits of
our information, so making sure that those
benefits are actually true, and also collecting
information to develop them and improve them
if we can see gaps.’ Front-line staff
NPC’s Making an Impact report also looked at the
reasons for developing impact measurement within
large charities. It cited that the main driver was a
‘change in funders’ requirements’.
FindingS: DRIVERS
This study reflected NPC’s finding that funding was
a key driver for charities with grant and statutory
funding. In particular it highlighted that charities
were keen to have impact measurement practices to
demonstrate to their funders that their services make
a difference and that they offer good value for money.
‘Part of the driver was coming from huge
competition around provision of services and
being able to evidence to commissioners our
value and our worth.’ Manager
‘We’re primarily driven by wanting to evolve our
impact measurement for the purposes of
improvement, but in reality, it’s the areas where
there are external pressures (commissioners or
funders) that are getting most support to develop
that quickly.’ Manager
‘We are living in a world today where actually
just ‘doing’ isn’t always enough. It’s about
not just delivering the programme, it’s about
measuring to see does it work, doesn’t it work.’
Manager
‘There’s been a bit of a shift change in the
last two to three years, both internally, in
our culture, and externally, in terms of the
growing discussion on outcome
measurements.’ Manager
Lastly the managers felt that the charities had
a moral obligation to measure impact, this was
particularly prevalent in charities whose work
was funded through unrestricted income.
‘My impression was there are two drivers, one
is an intellectual driver, which is about how can
The external environment, with the sector’s increasing we best use our money to help children, the
interest in impact, also acted as a driver. Interviewees best use of that is by learning. The other driver
talked about the growing debate and discussion that was “are our social workers really doing the
was occurring within the sector as well as within
best with their time”, is it a basic bunch of
government departments. For example, if working
cottage industries with no evidence base and
within the health field there is a need to engage with potentially poor practice and how can we
the NHS’s outcomes; and increasingly with statutory
professionalise it all or make it more robust?’
funding, payment is based on results (payment by
Manager
results) so there is a requirement to demonstrate
outcome achievement.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Key Drivers
• The main driver behind large charities
measuring impact is a change in strategy
or change in leadership.
• Front-line staff motivations differ from
management. Front-line staff are primarily
driven by a desire to know whether services
are improving outcomes for beneficiaries.
• Large charities are driven to measure to
maintain their relationship with funders.
• The external environment, particularly the
growing interest in and debate on impact
measurement, has influenced large
charities to start measuring impact.
• A moral obligation to clients is a key driver
in wanting to measure and demonstrate
impact.
FindingS: CHALLENGES
3.2 Challenges of impact measurement
practices faced by large charities
The charities interviewed were asked about the
challenges they had faced in developing their
impact measurement practices. Challenges were
identified that are likely to be more prevalent in
large charities, and these are outlined below:
Aligning the different needs of front-line staff and
management was seen as a challenge for many of
the charities. For example, charities delivering many
different services struggled to create a standardised
impact measurement tool that could be aggregated
and still be useful for front-line staff. Front-line staff
• Aligning the different needs of front-line staff and
often perceived the tools to be developed from a
management.
management-only perspective and stated that the
• Managing both the physical and operational
tools were too generic and created meaningless
distance between front-line staff and central
information for them to use. Front-line staff,
office functions.
working directly with clients, spoke at length about
the challenges of the evaluation being meaningful
• Dealing with cultural issues often associated with
and appropriate for their clients. Meaningful and
the age and history of the charity.
appropriate for front-line staff meant that impact
• Promoting measurement even when not required tools need to be in client-friendly language, the
to due to the high proportion of unrestricted
time required to collect the data proportional to the
income received.
service being offered and the timing of data
collection appropriate. Front-line staff felt that it
• The diversity of grant and statutory funders’
was often inappropriate to have time-consuming
requirements.
benchmarking very early on in the service. They also
• Mitigating reputational and brand risks if the
felt that it hindered the building of trust in the
service is shown not to have impact.
client-worker relationship. Another issue where
evaluation was seen as inappropriate was when a
A number of other challenges were also identified
client situation was critical, such as with emergency
that charities of all sizes experience when
first aid.
developing impact measurement, but this study has
focused on those more relevant to large charities.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
‘There is a discrepancy in what the leaders want
and what the staff want. It’s been decided that
we’ll use a form, but you can’t necessarily be
very local and specific with the form. That
distance between local operations and strategic
decision-making at a national level, definitely
does exist.’ Front-line staff
‘At an individual practitioner level you want to
know whether your service user has changed.
Then there’s “are services making an impact?”
Are we measuring and trying to make sure our
services are making a difference, and then
there’s another level which is about “can we
demonstrate impact in a scientific sense?” and
“can we demonstrate this approach has
impact?” The organisation is not entirely sure
about the difference between those things, and
is not entirely sure always what it’s asking for.’
Manager
The intrinsic characteristics of a large charity mean
that there is greater distance between front-line
staff and central functions of the organisation,
which adds challenges of communication,
consistency of services and establishing changes
when developing impact measurement practices.
‘We’re talking something in the region
nationally of 4,000 staff and 30,000 volunteers,
so in terms of the cultural change that we’re
leading, we’re talking to a lot of people before
you even reach the front-line service user
engagement.’ Manager
‘I would like to develop it in a more bottom-up
way, but the logistics of trying to do that, it’s
really hard work, and you’re not going to please
all of the people all of the time.’ Manager
‘Externally, the only significant barrier is the
sector not being consistent in the way it uses
language and the way it approaches these
things. It does make it unnecessarily hard work
in getting everyone on the same page, and
understanding what you’re talking about, what
you mean by the words you use. And it’s always
changing externally.’ Manager
Large charities tend to be older than smaller
charities and many of the charities interviewed have
been in existence for over 100 years. A charity’s
‘Implementing a national initiative was a
history is integral to its culture and its work. The
challenge within the team, because obviously
work may be more varied, staff may have been in
they felt that it hadn’t come from them, which it
post longer and the supporter base is likely to be
hadn’t, so it did feel like that was definitely a
well established. Many interviewees talked about
battle because the whole system has been
the challenge of needing to change the culture and
created in a way that seems like it hasn’t actually
overcome staff resistance in order to develop
taken on board from project-level staff how they
could see it working.’ Manager
‘Technically there are a lot of issues in how you
Establishing consistency across a charity’s work was
measure in all those wide range of contexts, and
further challenged by the lack of consistency in the
resources, of course, that’s always going to be
sector as a whole. For example interviewees talked
an issue no matter how much unrestricted funds
about how staff across the organisation often have
you might be lucky enough to have to throw at
different learning and development paths or are on
it, that’s not an inexhaustible source of money.’
different external groups, and these staff come
Manager
back from these sessions and introduce different
terminology.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
‘My challenge there was to try and get a culture
change and buy-in from people who are
historically very reluctant to do so from someone
like me who comes in from head office and
goes, ‘Hey, guys, how about doing this new
thing?’ and they’re rolling their eyes and going,
“Oh, this again, we’ve been here before,” so
that was my biggest challenge.’ Manager
Large charities often have a higher proportion of
unrestricted income than their smaller
counterparts. This can be both an opportunity and
challenge for impact measurement practices.
Unrestricted income gives charities more flexibility
in how they choose to spend their money so money
can be allocated more easily to impact practices.
However, without external funders, there can be
less pressure to be accountable to ‘prove’ their
impact.
Large charities that have restricted income, such as
grants and statutory funding, are likely to have an
array of different local authorities and grantmaking bodies to report to. Each funder may
require different impact measurement practices.
This creates a challenge for large charities when
trying to consolidate or create impact measurement
practices across the organisation, without creating
duplication for front-line staff.
Large charities often rely on their brand to support
their fundraising and exert influence on the media
and government policy. These brands have
generally developed over a number of years with
large investments. There is a fear, alluded to by
some interviewees, that if a charity is shown not to
be having a significant impact the brand may be
tarnished and this, in turn, will affect their
reputation and ability to influence and raise funds.
‘A lot of the commissioners ask for their own
tools, own forms, reports every month, which is
strenuous for the team already. If you have to do
the same type of report twice a month slightly
differently because that’s for in-house and that’s
for your commissioners.’ Front-line staff
‘There’s also an issue that needs to be addressed
and isn’t yet, which is what happens if you do
not show something is working, do you then
close it?’ Manager
In addition to the specific challenges relating to
larger charities, workers also talked about more
‘Linking (impact measurement practices) with
generic challenges. Few of the charities that were
reports to funders is quite difficult, because we interviewed were at the stage where they were fully
work with a number of different boroughs. Each utilising the information gained from their impact
borough might use different formats and expect measurement practices. Some charities had not yet
us to use theirs.’ Manager
collected enough information while others were
unsure how to use the information. Front-line staff
‘There’s the obvious challenge of different
talked about the fear that measurement tools were
funders asking for different things, and
being created without knowing how the
therefore they have a challenge in setting up
information will be used, and they wanted clear
databases that measure outcomes in different
guidance on how to interpret any results.
ways.’ Manager
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
‘People often say, well, if we get 70%, what
does that mean? People get worried that, there
seems to be a fear of really interrogating data
and sitting round and looking at numbers and
thinking about them. It’s almost like unless it’s
going to be really clear-cut, give me a yes or a
no, then actually what use is it?’ Manager
A manager [who was managing a telephone
advice service] was really disappointed as less
than 50% responded with ‘very strongly’, and he
was really upset, but actually I thought for a fiveminute phone call that seems perfectly
reasonable to me.’ Manager
‘There’s definitely a nervousness around being
perceived not to be doing a good enough job,
and there’s probably a lack of experience of
looking at this data and being able to draw
meaning from it, and therefore people saying
there’s not going to be much value in it because
we don’t know how to extract meaning from it
anyway.’ Front-line staff
Those that had established comparison groups
such as random control trials spoke about ensuring
that the cost is proportional to the benefits, while
those without comparison groups talked about the
challenges of benchmarking. Many interviewees
also highlighted that the impact measurement
tools focused on short-term impacts, and queried
if the information collected was reliable and if the
outcome that was being recorded could be fully
attributed to the service being offered.
‘A gap seems to be that we don’t seem to
have anything to compare ourselves against.’
Front-line staff
‘It’s quite easy [for clients] to tick and say
this information is going to help me to exercise
my rights, but the reality of whether that does
happen or not, it’s really hard to know, I think
it’s almost a longer-term evaluation that’s
needed, but that requires lots of resources.’
Front-line staff
Ensuring adequate resources was also seen as a
key challenge. Resources were needed both to
train staff with the necessary skills and give
them the understanding of their rationale and
processes and to equip services with the necessary
equipment and tools to carry out the impact
measurement practice. If resources were not
allocated to these it became a challenge.
‘The technical challenges are that our
computer system is a bit of a nightmare and
staff don’t have computer literacy skills.
We’ve got these new tools, but actually a lot
of people are scared to turn the computer on.’
Front-line staff
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Key CHALLENGES
• Large charities struggle to find an impact
measurement approach that balances the
needs of front-line staff and senior
management.
• Large charities need more coherency
between different functions and services
in order to measure in a more
meaningful way.
• Longer histories make it more difficult to
change culture and adopt new practices.
• A high proportion of unrestricted income
is both an asset and challenge to large
charities in trying to measure impact.
• Where charities do have grants and
contracts, different reporting
requirements of funders make it difficult
to have a coherent approach to impact
measurement.
• Brand awareness affects large charities’
willingness to measure impact. There is
risk that impact results may not match
the high level of brand awareness.
Findings: Enablers
3.3 Enablers in developing impact
measurement practices
The charities interviewed were asked what had
enabled them to develop their impact
measurement practices. The following were given
as key factors that had supported the concept,
design and embedding of impact measurement
practices:
Success strategies had a clear theory of change and
model of work. Many charities also found that a
critical first stage in developing their strategy was
ensuring consistency of language across the
organisation. Clarity on the work was gained
through the adoption of external programmes or
by adding rigour to internal models.
‘Our chief executive was really behind it, he
came to the training sessions and just sat in with
everybody else.’ Front-line staff
Those organisations that were able to balance the
needs of front-line staff and management in the
impact measurement practice had more success in
• Visible leadership at chief executive, board and
embedding processes. This is a critical point. Large
trustee level and clear strategy.
‘We were clear about why we asking the
charities generally are motivated to measure their
questions we were. It was so straightforward
impact to enable them to understand how a client
• Balancing front -line and management needs.
and adopted easily.’ Front-line staff
improves their own outcomes, the impact of the
• Adequate resources in both time and personnel.
service itself and the impact of the charity as a
‘A number of colleagues locally had begun their
whole. Front-line staff said they were motivated to
• Training of staff.
journey looking at the tools they were going to
collect and analyse data for local use and to see if a
use to demonstrate impact, rather than clarity
• Funders’ engagement.
service is making a difference to their clients;
on what the outcomes were that they wanted to
whereas the management said they wanted the
Leadership was seen as a critical enabler at trustee, measure. People try to fit external tools where
information to use strategically, to see the impact
chief executive, board and senior management
they don’t fit, so my tip is basically begin at the
the charity is having as a whole. Balancing these
level. All of the charities had a chief executive and/
beginning.’ Manager
two needs, by having an open dialogue and
or a trustee board committed to ensuring impact
‘There’s been a lot of support from senior
opportunities for front-line staff and senior
measurement took place, and who ensured that
managers and increasingly from the board of
managers to meet and share their ideas, was
the charity’s strategy reflected its vision for
trustees as well. They have had a dedicated
identified by both groups as a critical enabler. When
measuring impact. Front-line staff talked about the
conversations on outcomes and impact and both the impact measurement practices were designed
need for the leadership to be visible and how
groups, senior managers and trustees, wanted
and developed with the input of both front-line
leaders’ attendance at staff briefings and training
to be involved in that work along the way.’
staff and managers it was clear that the charity had
sessions had emphasised the charity’s commitment
Manager
greater buy-in from all.
to measuring impact practices.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Findings: Enablers
‘Lots of different departments were involved in
developing them [the impact practices] - that’s
been useful and we’ve been able to work
towards what we think is a better service. Frontline staff
measurement systems, this was shown to be a
strong enabler in ensuring success.
- Using the impact measurement tools
- Using any technical equipment that was needed to
‘We got together with users to help design and
record the data.
test the questions, to check that we were using
Acknowledging the challenges of the charity’s size,
language that made sense and would actually elicit
many leaders recruited champions to support them
‘Once project staff have had a chance to actually a response to the question.’ Manager
in developing and embedding the impact
add their ideas to how something can work,
Funder engagement was also seen as a key enabler,
measurement practices or rolling out the training.
their motivation increases a lot and they’ve got
funders who were interested in the impact data and
Another resource, that was identified as a critical
ownership (of the impact measurement
were willing to have an open dialogue helped drive
enabler by front-line staff was having adequate
practices). It removes the ‘them’ and ‘us’. Frontmeaningful and supportive impact measurement
time. Front-line staff said they needed time to be
line staff
practices.
allocated initially to understand what was required
Front-line staff claimed that impact measurement
‘It’s a two-way flow, our commissioners are driving of them and then an on-going basis to input the
tools better supported their work when developed
us and we’re helping our commissioners to see
data. For others, adequate resources meant having
in collaboration with managers. They said the tools
what it is they’re asking for.’ Front-line staff
an allocated impact measurement practice role that
linked more effectively into the service’s existing
they could approach for support and queries.
‘It might create the culture if we’re more used to
data recording system thus reducing the need for
funders as well as our board looking at this data.’
duplication of data collection. Front-line staff also
Manager
felt they were able to influence management in
ensuring that the resources and time that was
Those interviewed also stated that another key enabler
allocated to collecting impact measurement data
was to ensure that adequate resources were allocated
was proportional to the service that was delivered
to the embedding of the impact practices. This meant
and the results that would be gained.
a number of things to different charities. For some this
Some charities had developed impact measurement meant that staff received appropriate training. Key
tools that integrated into the service. Some charities areas that needed to be covered included:
also involved service users in developing the impact
- Understanding the needs and reasons for doing the
impact measurement
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Findings: Enablers
‘What has really helped us is these curiosity
building workshops. It got people to really
understand it [Impact measurement].’ Manager
‘I recruited about 65 outcomes champions all
over the country. I sent them off with a pack
of training to deliver to each of their service.’
Manager
IT systems. It was clear in successful charities that
resources had been allocated to this; these were
both in-house and external systems; these systems
allowed data collection to be consistent and
aggregated.
‘A post was created to try and support all of us
to find new ways to measure things. Having this
resource has been really useful for me just to
bounce ideas off, we’re thinking about doing
this, does this seem about right, that’s been
really useful, having someone in the
organisation who’s very good at research,
very clued-up.’ Front-line staff
The purpose of the training was to equip staff
with the relevant skills and knowledge, but a key
outcome was reduced resistance from staff and
greater support for the development of impact
‘We provided training where staff actually got
a better understanding of why [we were
developing impact measurement practices]
and then it was not so difficult.’ Manager
The last enabling resource highlighted through
the interviews was the development of appropriate
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Key ENABLERS
• Large charities need to have strong
visible leadership to implement impact
measurement. There needs to be a
coherency across the organisation’s
mission and strategy. A theory of change
can be a good method to achieve this.
• Open dialogue and communication is
needed to ensure that the impact
measurement approach meets both the
needs of front-line staff and senior
management.
• Measurement needs to be proportional
and integrated with the day to day
running of front-line services.
• Recruiting ‘impact champions’ and
providing training is crucial in getting
staff buy-in.
• Staff need to have adequate time and
resources to carry out impact
measurement.
• An appropriate IT solution is key to
ensuring that measurement is streamlined
with day to day work as well as being less
burdensome on resources.
Findings: BENEFITS
3.4 Benefits of measuring
impact practices
The charities interviewed were asked to specify the
benefits of their impact measurement practices.
Whilst few charities felt they were fully utilising the
data they still felt that they had seen a number of
benefits. These are outlined below:
T he process of developing impact practices has
improved services for beneficiaries through workers
questioning and clarifying the focus of the work,
developing clear models and theories of change.
This, alongside results from the impact
measurements was inspiring and increasing
motivation amongst front-line staff.
much easier [now] for people to say quite
clearly, “This is what we do and this is how we
do it and this is the outcome”.’ Front-line staff
Interviewees found that the impact measurement
results had enabled them to secure funding,
through renewing old or winning new contracts.
‘Since we developed the system and
implemented it we’ve won new contracts and
retained contracts as a result of the system.’
Manager
eveloping impact measurements had resulted in
D
the building of stronger networks across the
organisation and greater cross working and
shared learning.
‘Benefits are that people who are delivering
services say that using this outcomes system
makes it easier to do their job, which is brilliant.’
Front-line staff
‘There’s been very good, clear communication
across the different levels, a few years ago we
‘We are starting to reap the benefits of clearer
probably would have never spoken, so those
articulation of our intended outcomes.’ Manager
sorts of bridges are being built, people are no
At a national and local level charities have been able to longer just in their team, they’re actually more
use the data from their impact measurements to
aware that we’re part of a wider organisation,
influence the debate on what works. This has also
it connects the organisation up.’ Manager
enabled charities to raise their profile.
‘I have been able to use the results and share
‘We have made a special effort to go out and get them, I’ve been doing e-newsletters, I’ve
on to provider forums, steering groups and
been doing things within the organisation
service user involvement groups. It’s
like articles for the intranet.’ Front-line staff
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Key BENEFITS
• Improved services for beneficiaries
• Increased motivation amongst front
line staff
• Greater ability to influence the debate
• Raising profile
• Securing funding
• Greater cross-working and shared
learning across the organisation.
4 Conclusions and
Recommendations
4
This study set out to further identify and
understand the drivers, challenges, enablers and
benefits of developing impact measurements
practices in large charities. The study shows the
success that some large charities are having in
developing their impact practices and has
highlighted what has enabled this success.
This section offers recommendations for large
charities wishing to develop their impact practices.
Senior managers and front-line staff
should work together to design and
embed impact measurement
practices.
Working together enables the front-line staff and
management needs to be better balanced. The
tools developed are likely to be more effective and
suitable for the client group. Tools are also more
likely to link in with existing systems and not
duplicate work. Front-line staff will feel more
motivated both because they are able to express
their views and also because with more access and
visible leadership front-line staff will be able to see
for themselves their charity’s commitment to
measuring impact. Through open dialogue and
communication, front-line staff are also likely to
have their fears, such as what the data will be used
for, alleviated.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Conclusions and Recommendations
Take into account the history, size
and complexity that being large
brings.
Large charities are likely to be older and have a
greater history than their smaller counterparts. This
history needs to be acknowledged. In order to
counter the size and complexity of a large charity,
realistic time scales need to be set and adequate
resources need to be allocated for developing and
embedding impact practices. This will include
ensuring staff have training on the need to measure
impact and using impact measurement tools.
Work with funders to agree impact
and measurement approach.
Large charities funded by a variety local authorities
and grant-making bodies need to work with
funders to get an agreed impact measurement
approach. This may be a challenge, but the
alternative is for the charity to be led by the funders
impact measurement requirements, and where this
has occurred charities have found it difficult to
consolidate and create impact measurement
practices across the organisation, without creating
huge duplication for front-line staff.
Be clear about
what the impact measurement is
trying to capture.
A fundamental prerequisite to measuring impact is
to ensure services have clear theories of change,
models of work and consistency of language. It is
also critical to understand that within a charity
there may be different reasons for developing
impact measurements. These include
understanding how a client improves their own
outcomes; the impact of the service itself; and the
impact of the charity as a whole. The challenge of
aligning these different requirements is made easier
if there is clarity and transparency about these
different needs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Bibliography
Ní Ógáin, E., Lumley, T., and Pritchard, D. (2012)
Making an Impact: Impact measurement among
charities and social enterprises. New Philanthropy
Capital.
Smithson, K. and Tully, J.,(2012) Principles into
Practice: How Charities and social enterprises
communicate impact. Charity Finance Group,
ACEVO and New Philanthropy Capital.
Hornsby, A. (2012) The Good Analyst: Impact
Measurement and Analysis in the Social-Purpose
Universe. Investing for Good.
Lumley, T., Rickey, B. and Pike, M. (2011) Inspiring
Impact: Working together for a bigger impact in the
UK social sector. Views and New Philanthropy Capital.
Schorr, L. and Farrow, F. (2011) Expanding the
Evidence Universe: Doing Better by Knowing More.
Centre for the Study of Social Policy.
Rickey, B. Lumley, T and Ní Ógáin, E (2011) A Journey
to Greater Impact: Six Charities that learned to
measure better. New Philanthropy Capital.
Morino, M. (2011) Leap of Reason: Managing to
Outcomes in an Era of Scarcity. A Venture Philanthropy
Partners Publication.
Appendix A –
Interview Questions
and Schedules
INTERVIEW QUESTIONS:
1. Background questions: Details of charities: purpose, activities, size and funding
2. Interviewee: Role and length of time at organisation
3. What was driving your organisation to measure impact?
4. What challenges did you face when establishing impact measurement practices?
5. What enabled you to develop successful impact measurements?
6. What have been the main benefits of developing impact measurement practices?
INTERVIEW SCHEDULES:
Organisation Family Action
The Children’s Society
WRVS
Mind
British Red Cross
Action on Hearing Loss NSPCC
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Interviewees
Rhian Beynon
Sarah George
Ann Kendal Lorna Jacques Karl Demian
Sarah Hollaway
Beth Murphy
Liz Ward
Debby Mulling
Alison Roche
Lindsay Hodgson
Louise Pritchard
Pamela Newman
Matt Barnard Richard Cotmore
Ingrid Anson
Role
Manager/Impact Champion
Front-line
Front-line
Front-line
Front-line
Front-line
Manager/Impact Champion Manager/Impact Champion
Front-line
Front-line
Date
27/11/12
22/11/12
07/11/12
16/10/12
22/11/12
02/11/12
12/11/12
01/11/12
08/11/12
16/11/12
12/11/12
15/11/12
05/11/12
01/11/12
01/11/12
07/11/12

Read the report - Clore Social Leadership

Transcription

Similar documents

Bud Billiken Parade.indd

Open House Flyer 2015

For more information or tickets call (210)

Anniversary Campaign Raises Endowment Fund

Catholic Charities Brochure – English

www.lillypulitzer.com

November Hope Notes - Catholic Charities of Southern Nevada

Products with purpose

2009 Pre-Gala Press Release - Catholic Charities of the

CC Programs Report (part 1)