GMCR Monitoring and Evaluation Guide for Supply Chain Outreach

Transcription

FY2012
GMCR Monitoring and Evaluation Guide for
Supply Chain Outreach Funded Projects
Meryl Breton Olson
Ursula Georgeoglou, M.S.
V. Ernesto Mendez, Ph.D.
Marcela Pino
GMCR Monitoring and Evaluation Guide for Supply
Chain Outreach Funded Projects
FY2012
Table of contents
Table of contents .......................................................................................................................................... 1
Foreword....................................................................................................................................................... 3
Acknowledgements....................................................................................................................................... 2
1.
2.
3.
Introduction ...................................................................................................................................... 3
1.1
Objectives: Why monitor and evaluate? .................................................................................. 4
1.2
Definitions ................................................................................................................................. 4
1.3
Who should be involved in monitoring and evaluation? .......................................................... 6
1.4
How often should monitoring and evaluation take place?....................................................... 7
1.5
What types of information should be collected? ..................................................................... 7
1.6
Summary of indicators to be used for monitoring and evaluation........................................... 8
Methods of Data Collection for Monitoring and Evaluation ............................................................ 9
2.1
Interviews................................................................................................................................ 10
2.2
Focus groups ........................................................................................................................... 15
2.3
Timing of data collection......................................................................................................... 17
2.4
Designing a baseline study ...................................................................................................... 17
Quantitative Output Indicators ....................................................................................................... 19
3.1
4.
5.
Required output indicators for all projects ............................................................................. 20
Most Significant Change Methodology of Qualitative Project Evaluation ..................................... 22
4.1
Introduction ............................................................................................................................ 23
4.2
The Most Significant Change process ..................................................................................... 23
4.3
Practical tips for the MSC process .......................................................................................... 28
4.4
Adapting MSC to your organization ........................................................................................ 29
Quantitative Impact Indicators ....................................................................................................... 30
5.1
Introduction ............................................................................................................................ 31
5.2
Project objective: Build capacity ............................................................................................. 31
5.3
Project objective: Create employment opportunities ............................................................ 31
5.4
Project objective: Increase food security................................................................................ 32
5.5
Project objective: Increase crop yields ................................................................................... 41
© 2012 Green Mountain Coffee Roasters, Inc.
1
FY2012
5.6
Project objective: Improve access to credit ............................................................................ 42
5.7
Project objective: Increase income ......................................................................................... 42
5.8
Project objective: Improve educational access....................................................................... 44
5.9
Project objective: Improve educational quality ...................................................................... 45
5.10
Project objective: Improve access to health services ............................................................. 47
6.
Guidelines for Reporting ................................................................................................................. 54
7.
References and further reading ...................................................................................................... 57
Appendix 1: Sample Most Significant Change story collection format ................................................... 62
Appendix 2: Example facilitation guide for MSC story selection ............................................................ 64
Appendix 3: Example evaluation report to GMCR .................................................................................. 65
Story 1 ................................................................................................................................................. 70
Story 2 ................................................................................................................................................. 71
Story 3 ................................................................................................................................................. 72
2
FY2012
Foreword
Dear Friends,
It is with pleasure that we release Green Mountain Coffee Roaster’s (GMCR) Monitoring and
Evaluation (M&E) Guide for Supply Chain Outreach Funded Projects.
This journey started in the Agroecology and Rural Livelihoods Group lab at the University of
Vermont, where Rick and Professor Ernesto Mendez began discussing how GMCR could better
understand the impact of our coffee community outreach funding, and, at the same time, support our
grantees to engage in participatory evaluation processes and collect data that would be valuable to their
organizations. The idea was to create a standard monitoring and evaluation protocol that would include
quantitative and qualitative impact indicators and would be both useful and feasible to implement
across our portfolio of funded projects.
Being fans of the participatory research approach, we knew more brains were required to make
this work. The first GMCR Reporting Collaborative was held in May 2010 at our Headquarters in
Waterbury, Vermont, where a dozen experienced professionals, representing grantees of all sizes, came
together to help us craft a comprehensive M&E system for GMCR-funded projects. Tapping into the
expertise of the group and the rich discussion that emerged, UVM developed a draft M&E Guide which
was field tested by this pilot group in the first half of 2011. We reconvened the Reporting Collaborative
in June 2011 to gather feedback from the pilot testing and to make final enhancements to the guide,
which are reflected in the present version for FY2012.
We believe that the implementation of this M&E protocol will provide GMCR with a
standardized and meaningful way to measure our impact, while providing useful data for reflection and
reaction by our nonprofit and cooperative partners. We will use the information we receive in your
reports for three purposes: (1) Communication – providing aggregate information to our stakeholders
(our board, employees, consumers, suppliers, partners) across a growing portfolio of projects, (2)
Decision Making – identifying successful approaches, prioritizing outcomes over outputs, and (3)
Collaboration – sharing this information with grantees and other development actors in a spirit of
collective learning.
We are so fortunate to be partnering with some of the most innovative and effective
organizations working in rural human and economic development in the coffeelands. We honor the
time you spend on evaluating your programs and strive to mirror your commitment to impact and
continuous improvement in our own approach to supply chain grantmaking.
Thank you,
Rick Peyser, Director, Social Advocacy and Supply Chain Community Outreach
Colleen Bramhall, Manager, Coffee Community Outreach
Mary Beth Jenssen, Administrative Assistant, Supply Chain Community Outreach
3
FY2012
Acknowledgements
First, we would like to thank Dr. V. Ernesto Mendez, Assistant Professor at the University of Vermont in
the Agroecology and Rural Livelihoods Group (ARLG) for his vision and leadership in developing this
M&E program.
We would also like to thank Meryl Olson, PHD Candidate in the ARLG lab, for authoring the guide
through multiple drafts and for seeking out and incorporating diverse feedback from the team and
literature. We also appreciate the ongoing contributions of other members of the ARLG lab including
Marcela Pino, Ursula Georgeoglou, and Margarita Fernandez.
Our sincere thanks to all those who participated in the Reporting Collaborative and provided their time,
energy, and expertise to the development of this guide:
Michael Sheridan, Jefferson Shriver, and Sarah Cashore (Catholic Relief Services); Jose Luis Zarate
(Coffee Kids); Chris Bacon, Roberta Jaffe, and Heather Putnam (Community Agroecology Network);
August Burns, Elisa Vandervort, and Kayla Moore (Grounds for Health); Alvaro Cobo (Heifer
International); Lindsay Palanzuelos, Daniel Palazuelos, and Hugo Flores (Partners in Health); Beth Merrill
and Darryl Bloom (Planting Hope); Patty Devaney (Root Capital); Raphael Makonnen, Mario Roa
Romero, and Luciana Sette (Save the Children); Joanne Vincett (Tuck School of Business); and Shauna
Alexander Mohr (facilitator of both sessions).
This guide has also benefited from an extended review team at each of the above organizations,
including central and field teams engaging in thoughtful dialogue about monitoring and evaluation.
Thank you for your support!
2
GMCR Monitoring and Evaluation 1 Introduction
Guide for Supply Chain Outreach
Funded Projects
1. Introduction
3
Funded Projects
1.1
Objectives: Why monitor and evaluate?
The primary objective of this guide is to facilitate
more standardized reporting of progress, impacts, changes
Monitoring and evaluation
and/or needs of projects funded by GMCR’s Supply Chain
processes “help program teams
Community Outreach team. However, the protocols detailed
to learn what does and doesn’t
here are not intended to be an onerous set of requirements
work in their efforts to overcome
that create paperwork for GMCR-funded activities. Instead,
poverty and suffering and to
these protocols are intended to be flexible so that funded
adapt their programs in light of
organizations can incorporate them into existing Monitoring
what they find” (Oxfam, 2).
and Evaluation (M&E) programs, or use them to create a
simple but effective M&E program if none exists within the
project. Monitoring and evaluation is not just a way for reporting to funding agencies. A good
monitoring and evaluation system should allow for reflection and growth within an organization, and
ultimately foster more effective programs.
Another objective of this protocol, and of the process that created it, is to foster greater
communication between GMCR and the organizations that it funds, as well as collaboration between
these organizations. The information obtained from M&E activities will be useful for GMCR to better
communicate its funding approach to its employees, shareholders, the coffee industry and the general
public. We hope that this will ultimately lead to more sustainable impacts in the field and greater
recognition of the potential positive impact of effective and committed corporate social responsibility.
Ideally, this type of corporate social responsibility will come to be seen not as an “exotic” practice of a
few companies, but as the standard way of doing business.
1.2
Definitions
Project participant: A project participant (also referred to as a beneficiary) is a person who benefits
from (or is intended to benefit from) a project or program in some way.
Indicator: An indicator is a quantitative or qualitative variable or factor that provides a way to measure
achievements, assess performance, or reflect the changes resulting from a program activity (Fretheim,
Oxman, Lavis, & Lewin, 2009). Indicators can measure change directly or indirectly; indirect indicators
are known as “proxy” indicators (Oxfam). Indicators should be tied to specific program goals. In
participatory monitoring and evaluation, indicators are often developed with or by program participants
to allow them to define program success (Estrella & Gaventa, 1998).
Output indicator: Output indicators show how the project is progressing. They measure the amount of
work being done (e.g. number of people reached by a program, number of training sessions conducted)
and can also indicate if the work was high quality and completed according to schedule (Oxfam). Output
indicators are also called “process indicators” or “performance indicators.”
4
Funded Projects
Impact indicator: Impact indicators measure the ultimate impact of the project on the beneficiaries or
participants (e.g. change in income or nutrition status). These indicators can be quantitative and/or
qualitative, and can measure positive or negative
changes (Oxfam). Impact indicators are also called
“outcome indicators.”
Monitoring tracks achievement of
project objectives (“Have we done
Monitoring: Monitoring refers to the ongoing collection
what we said we would do?”).
of information on a project’s activities, performance, and
Usually associated with output or
outputs. It is intended to determine if the project is
performance indicators.
performing as expected, if funds are being allocated
correctly, and if any adjustments need to be made
Evaluation determines the value of
(Fretheim et al., 2009; Oxfam). Data collected during
the work (“Has our effort made a
monitoring activities may include more output indicators
difference?”). Usually associated
than impact indicators (Oxfam).
with outcome or impact indicators.
Evaluation: Evaluation refers to more in-depth, formal
(Mathie & Foster)
assessment of a project’s results at particular points in
time. It complements monitoring by trying to determine
why things are happening the way they are. An
evaluation can happen at any point during the project, but it generally occurs less frequently than
monitoring (Oxfam). It tends to focus more on impact indicators.
Quantitative methods: Quantitative methods of research are used to empirically and objectively
evaluate conditions and processes. Quantitative methods of data collection usually use specific, narrow
questions to gather numerical data from respondents.
Qualitative methods: Qualitative methods of data collection are designed to evaluate conditions and
processes as they are perceived by the people involved in the project. Qualitative methods tend to be
more open-ended. Qualitative data is often not “quantifiable;” it may, for example, be stories or
examples of changes resulting from the project. However, in some cases it can be quantified, such as by
having project beneficiaries rate their opinion about something on a numerical scale (Baker, 2000).
Malnutrition: Malnutrition is a nutritional disorder resulting from faulty or inadequate nutrition (Cogill,
2003). Some researchers do not distinguish between malnutrition and undernutrition, and group all
nutritional disorders of insufficient energy and nutrients under “malnutrition” (Svedburg, 2000).
Undernutrition: Undernutrition refers to the outcomes of a deficiency in food energy over a long period
(Svedburg, 2000). It is distinguished from malnutrition in that it indicates insufficient food quantity,
whereas it is possible to consume sufficient food calories and still be malnourished if the diet is of
insufficient quality. Undernutrition can also result from the body’s inability to use the food consumed,
due to disease or nutrient deficiencies.
5
Funded Projects
Underweight: Underweight is a condition of lower than average weight for one’s age. This can result
from either chronic or current acute undernutrition (Cogill, 2003).
Stunting: Stunting refers to a slowing of skeletal growth the results in reduced bodily height or length.
This results from long periods of inadequate food intake and/or disease, particularly during young
childhood (Cogill, 2003). Stunting is measured by height-for-age (see section 5.4.4).
Wasting: Wasting is a condition resulting from the loss of body fat and body tissue. It usually indicates
severely inadequate current food intake coupled with infection (Cogill, 2003). Wasting is measured by
weight-for-height (see section 5.4.4).
1.3
Who should be involved in monitoring and evaluation?
We encourage project administrators to aim for a participatory monitoring and evaluation
process. Participatory monitoring and evaluation seeks to involve all people who take part in or are
affected by a project in evaluating the success of the project (Mathie & Foster). Specifically, project
participants should have a voice in defining project success, in contrast to the more “conventional”
approach where donor representatives or external consultants determine what an effective project
looks like (Shah, Kambou, Goparaju, Adams, & Matarazzo, 2005). In participatory M&E, field staff and
project participants help collect, discuss, and analyze changes resulting from the project. Participatory
approaches also emphasize transparency in monitoring and evaluation by sharing the outcomes of M&E
activities with field staff and project beneficiaries. Such approaches allow the project to evolve around
the needs of the beneficiary community, promote community ownership of the project, and build
capacity of field staff and project beneficiaries as decision makers (Mathie & Foster).
Participatory approaches do, however, tend to be more time- and resource-intensive than more
conventional approaches, and therefore may not be within the reach of all GMCR-funded projects.
Additionally, in trying to standardize reporting, this guide likely removes some opportunities for
participation of project participants in designing monitoring and evaluation. It is, however, intended to
be flexible enough so that GMCR-funded organizations can find a balance of participatory and more
conventional methods. At the very least, GMCR expects grantees to share the results of monitoring and
evaluation activities with field staff (who are often the ones to collect data for M&E) and project
participants. This may be in written or verbal form. It is also extremely important to inform participants
during the data collection process why you are collecting the information and what you plan to do with
it (see Box 2.1).
Field staff involved in monitoring and evaluation will generally require training on how to
conduct interviews and collect data. This guide is a good starting point, and there are also several
excellent guides to interviewing and conducting focus groups, such as Flick (2007) and Stewart et al.
(2007). Funding for this training may be included in grant proposals to GMCR.
6
Funded Projects
1.4
How often should monitoring and evaluation take place?
Monitoring refers to keeping track of outputs and activities performed by the project. As such,
record keeping for monitoring activities must be performed on an ongoing basis. Analysis of this data to
examine if the project is meeting its goals should be performed at least every six months, beginning six
months after the date of the grant letter. A brief report on outputs and activities should be submitted to
GMCR at each six-month interval.
Evaluation examines if the project is having the desired impacts or outcomes within the
participant community. This requires a more in-depth process of data collection and analysis, often
through interviews with project participants. Evaluation should be performed on an annual basis, with
an evaluation report submitted to GMCR every year beginning one year after the date of the grant
letter. As this will also coincide with a six-month monitoring report, results from monitoring of outputs
and activities should be incorporated into the evaluation report rather than submitting separate reports.
As GMCR’s fiscal year ends in late September, any information received from grantees before
September can be considered for inclusion in the yearly Corporate Social Responsibility Report. M&E
reports received after September will be considered for the next year’s report.
1.5
What types of information should be collected?
This guide asks GMCR-funded projects to collect two types of information: quantitative
information and qualitative information. Both types of information are important for evaluating project
success. Qualitative information such as stories from project participants can explain the results of
quantitative indicators and reveal unexpected project impacts. Quantitative data can be used to
corroborate qualitative information.
Quantitative information to be collected for M&E can be further broken down into information
about project outputs and information about project impacts. Section 3 describes the quantitative
output indicators that GMCR requests from all projects, if they are applicable. Section 4 describes the
Most Significant Change methodology, which is the process by which GMCR asks grantees to
qualitatively evaluate the success of their projects. Section 5 describes sets of impact indicators specific
to different project objectives. Project leaders should select impact indicators as part of their grant
application or proposal to GMCR and use the indicators applicable to their project objectives. If none of
the indicators listed seem applicable to your project, this should be discussed with the GMCR team prior
to submitting your proposal to define acceptable alternatives. For those projects that started prior to
FY2012, please contact Colleen Bramhall if you feel that you cannot meet the reporting requirements in
this document.
7
Funded Projects
1.6
Summary of indicators to be used for monitoring and evaluation
Project objective
All projects
Required indicators
 Number of direct and indirect
participants
 Number of capacity building
activities
 Direct investments (number,
dollars)
 Most Significant Change
stories
Build capacity
 Continued use of new
knowledge
Create employment
opportunities
 Number of jobs created by
project activities
Increase food security
 Months of adequate
household food provisioning
 Dietary diversity
Increase crop yields
 Crop yields
Improve access to credit  Total revenue of funded
enterprises
Optional indicators
 Nutritional deficiencies among children
 Coping strategies index
 Net revenue of funded enterprises
 Loan repayment rates
Increase income
 New income generated by
project activities
 Annual household income
 Household savings
Improve educational
access
 Percent of eligible children
attending school
 Percent of reproductive-age women in
school
 Grade attainment
Improve educational
quality
 Performance on end-of-year tests
 Literacy rate
Improve access to health
services
 Treatment and resolution (acute
conditions)
 Detection, treatment, and loss to
follow-up (chronic conditions)
 Retention of health workers
 Effectiveness of health workers
8
GMCR Monitoring and Evaluation 2 Methods of Data Collection
Funded Projects
2. Methods of Data Collection for Monitoring
and Evaluation
9
Funded Projects
Not all M&E indicators necessitate talking with participants to gather information. Output
indicators and certain impact indicators such as loan repayment rates may simply be monitored by
project staff. However, at some point, most organizations will need to collect information from
participants in order to evaluate project success. The methods in this section are intended primarily for
collecting data for quantitative impact indicators; methods for the Most Significant Change methodology
of qualitative evaluation are explained in more detail in
section 4.
Box 2.1: Ethics of data collection
There are many methods to collect data for
monitoring and evaluation. Two of the primary
Attention must be paid to ethics
methods are interviews and focus groups. There are
when collecting information from
other ways of collecting this information as well, such
project participants, both for
as observing participants while working alongside
quantitative and qualitative data.
them; however interviews and focus groups are the
Ensure that the person collecting the
two simplest and most suitable methods of data
information has consent for the
collection for our purposes.
interview as well as for publishing the
story if there is a chance it will be
2.1 Interviews
used in publications. A good way to
do this is to include a box on the data
2.1.1 Designing and validating an interview
collection form that must be checked,
Interviews refer to one-on-one or group
indicating that the interviewer has
discussions with a project participant. Interviews can
obtained consent. For collection of
range from the more “journalistic” style where the
testimonials or stories, the person
interviewer already knows the information that he or
telling the story should also be asked
she wants and tries to elicit particular information
whether they would like their name
from the respondent, to the more “ethnographic” style
attached to the story. If not, names
where the interviewer knows little at the outset and
should be removed. Finally, special
explores a set of topics with the respondent in an
care must be taken when collecting
open-ended conversation (Leech, 2002). The style that
information from children. Parental
GMCR recommends is somewhere in between, a style
consent should be obtained before
called the semi-structured interview. This style of
children are interviewed. In some
interview has the advantage that it can elicit concrete,
countries this is required by law
reliable information that can be compared between
(Davies & Dart, 2005).
households as well as stories and experiences from the
respondent’s point of view. This style tends to combine
closed-ended questions where the interviewee is asked to choose between one or more answers with
open-ended questions that ask the interview to describe a condition or event in his or her own words.
10
Funded Projects
Turning indicators into interview questions is not a simple process. Think carefully about how
you will want to analyze the data and for what purpose it might be used in the future. For example, for
evaluating the effectiveness of capacity-building activities, one might ask respondents, “Have you used
the information that you learned at the training that you attended last month? “ This yields a yes or no
answer, which might be useful, but it may also provide little detail with which to compare from year to
year. Also, respondents may be likely to say “yes” simply out of courtesy. A better way to collect this
information would be to identify the specific behaviors or tools that your organization would like
participants to learn from the training and ask how often, in a specific time period, they carried out
those behaviors, such as “In the past week, how many times did you wash your hands before preparing
food?” or “In the past week, how many business transactions did you make? How many of those
transactions did you record?” In these examples, respondents are more likely to give a truthful answer,
and the interviewer gets more detail about the degree to which they are using the information that they
learned. These questions can then be followed up with “Why or why not did you use the skills that you
learned in the training?”
When organizing the interview, it is a good idea to put the least sensitive questions first and the
most sensitive questions (such as about income or personal health matters) at the end of the interview
(Leech, 2002). This gives the respondent time to become comfortable with the interviewer, and also
makes those questions seem less about the respondent and more about the subjects discussed in the
interview (Leech, 2002).
Box 2.2: Presuming questions
Asking questions that presume a certain answer is generally not a good idea, but there are
situations where it may be the only way to collect information that is sensitive or has stigma
attached. Leech (2002) relates an experience from her field work collecting reproductive
histories from women in Kenya:
“I began simply by asking women to tell me about all of their pregnancies. It was clear from the
first few interviews that no one was mentioning miscarriages, stillbirths, or deaths of children—
and I knew that could not be accurate. So I tried probing: ‘Tell me about any children who died.’
I used this question only once, and it caused the respondent to jump up, mutter that she must
go check on the goats, and run out the door. After some help from a language consultant, I did
two things. I made my language less threatening, and I asked the question in a presuming way.
‘How many children are the lost ones?’ I asked. My respondents’ faces would turn serious, they
would sigh, then they would tell me the details I was seeking (Leech, 2002, 666).
11
Funded Projects
Once an interview script has been written, it should be validated with a group of test
respondents. These should be people similar in cultural and educational background to the sample
group, but not actually part of the sample group. This will allow the interviewer to see if the questions
are understandable and if respondents will interpret them as intended. It also allows the interviewer to
develop additional “prompts” if a question will likely require further explanation to elicit the desired
information. For example, if in response to the question, “What do you do when you do not have any
food or any money to purchase more food?” the respondent answers “We try to get more food,” the
interviewer will need to follow up with a prompt for more detail. The interviewer could ask, “From
where do you try to get that food?” or “How do you get more food?” Interviewers should avoid putting
words in the mouth of the respondent. In the above example, “So you borrow more food?” would be a
leading prompt and one to be avoided. For more information on designing interview questions, consult
Fowler (1995).
2.1.2
Selecting participants for interviews
A number of questions arise when selecting participants to be part of interviews for project
evaluation. First, how many participants should be interviewed? A sample size of 20 is generally
considered sufficient to examine changes in one variable (Dytham, 2003). However, for very large
projects, or for a binary indicator (such as having savings or not), 20 may not be sufficient to ensure a
representative sample of all participants. For quantitative indicators, GMCR asks funded projects to
collect information from at least 10% of direct participants (see section 3.1.1), with a minimum of 25
individuals or households. Sample size requirements for the Most Significant Change methodology are
different; see section 4.1. A larger sample is nearly always preferable, but sample size must be balanced
against the resources available for monitoring and evaluation.
Second, how should participants be selected? Ideally, participants should be selected at
random. Random selection means that every participant has an equal chance of being selected for
inclusion in the sample. Generally, this is best accomplished by an automatic random number generator
(for example, www.random.org/integers) with participants each assigned a number, rather than by
haphazard human selection. Studies have shown that even when humans attempt to select a “random”
sample, they tend to be more systematic than truly random (Gotteli & Ellison, 2004). In some cases it
may also be useful to stratify the sample. Stratification means selecting a random sample from each
sub-unit of a larger population. For example, if a project works with three cooperatives, project staff
might choose to take a random sample of 25 participants from each cooperative rather than grouping all
participants together and taking a random sample of 75 people. This ensures a sample that is
representative of all participants and in some cases has been shown to be preferable to a simple
random sample (Dytham, 2003).
12
Funded Projects
Box 2.3: Tips for interviewing









Schedule enough time for the interview; tell the respondent ahead of time approximately
how long the interview will last.
Choose a location with as much privacy and as few interruptions as possible.
Obtain consent.
Greet in a culturally appropriate manner; reintroduce yourself and the purpose of the
interview.
Establish rapport by giving information about what will occur during the interview and
asking the respondent to begin by describing experiences of a non-controversial nature.
Use brief verbal cues or body language to indicate that you are listening and interested in
what the respondent is saying.
Before moving on to the next question, briefly summarize what the respondent has said,
in their own language, to indicate that you understood what was said.
Allow the respondent sufficient time to answer the question. Try to avoid jumping in with
a prompt if the respondent pauses to think.
At the end of the interview, ask if there is anything more the respondent would like to
convey. Inform the respondent how s/he can contact you for more information or if there
is anything further s/he would like to say.
(Leech, 2002; Pawar, 2004)
2.1.3
Organizing and analyzing interview data
Data management and analysis are equally important as data collection, but turning a pile of
interview sheets into meaningful data can seem overwhelming. It will seem less overwhelming if you
start with a well-designed table or set of tables for data entry. This can be a simple excel spreadsheet,
though organizations that collect more extensive data may wish to use a relational database program
such as Microsoft Access. Each row of the table corresponds to a project participant that was
interviewed. For data that pertains to households, businesses, or some other unit, each row may instead
correspond to a household, household member, or business. Each column corresponds to a variable or
piece of information about that participant/household/business. In general, these will also correspond
to questions in the interview (Fig. 2.1). In addition, it is good practice to maintain a separate table that
lists all of the variables (column headers) with descriptions of what they mean, along with any “coding”
that was used in entering the data (Fig. 2.2). An example of “coding” might be using 0 to represent a
“no” answer and 1 to represent a “yes.” This is commonly done to make it easier to sum the yes and no
answers.
13
Funded Projects
Figure 2.1. Example table for data entry and analysis
Notice that in Fig. 2.1, each individual is identified by a household code (the PAB indicates that these
households are from the town of Panajab) and a member number. Each combination of household code
and member number creates a unique identifier for each individual. While it would also be possible to
identify each individual by name, potential problems with different name spellings and concerns for
confidentiality make it more practical to use some sort of code. The code can be matched with each
individual’s name in a different table, separate from the data associated with that individual.
Figure 2.2. Example variable description table
14
Funded Projects
Once data is in this form, it becomes possible to answer questions and calculate statistics about
the group that was interviewed. For example, using the above data, we could figure out what
percentage of children under age 18 are attending school, the average grade level to which adults have
attended school, and the percentage of students who are at the appropriate grade level. These statistics
can then be incorporated into reports to GMCR.
2.2
Focus groups
2.2.1
Conducting a focus group
Focus groups involve a group of participants discussing a few common themes, with a facilitator
to direct the conversation. Focus groups are particularly useful for exploratory research when little is
known about a topic, such as for conducting an assessment of community needs. They are also useful
for collecting stories or experiences, as hearing others’ stories often elicits more stories from
participants. Focus groups can also be used to learn how respondents talk about a topic in order to
inform the design of interviews (Stewart et al., 2007).
Focus groups should last no more than three hours and focus on only a few key questions
(Butler, 1995). Longer focus groups or day-long workshops can be used with groups that may need
significant time to become comfortable enough to share their opinions, but it is important to provide
refreshments and schedule breaks to maintain the energy of everyone involved. Limit the number of
topics discussed; a question that might be answered in two minutes in a one-on-one interview may
generate a 30 minute discussion in a focus group! Questions should be open-ended and generally less
structured than interview questions. Words such as how, why, under what conditions and other similar
probes signal to the respondents that the interviewer is interested in complexity and in-depth discussion
(Steward & Shamdasani, 2006). As with semi-structured interviews, it is a good idea to pre-test and
validate questions for focus groups.
Eight to twelve participants is the ideal size; large enough to generate discussion but small
enough such that all participants get a chance to speak. Group dynamics can have a large influence on
the outcome of a focus group, and the facilitator should therefore foster a comfortable and nonevaluative atmosphere. The focus group should begin with a “get-acquainted” or “warm-up” session to
give participants time to get to know one another and become comfortable sharing thoughts and ideas,
particularly if they are strangers. In this early part of the focus group, the moderator can seek stories
and common experiences among group members before moving on to more difficult or controversial
topics. This will add to group cohesiveness and make the focus group more comfortable and enjoyable
for participants (Steward & Shamdasani, 2006). As the focus group progresses, the job of the moderator
is to keep the discussion on track and foster the active participation of everyone in the group. As with
individual interviews, the moderator can also probe for more information if a participant’s response is
unclear or if discussion stalls (Steward & Shamdasani, 2006).
15
Funded Projects
Simple visual aids such as a whiteboard or a paper flip chart can be useful for brainstorming or
recording information, but the literacy level of the group must be kept in mind. Using pictures to convey
information, whenever possible, can help get around this limitation.
Several options exist for recording the information gathered during a focus group. Audio or
video recordings are one option, but the equipment should be tested beforehand and used in a nonobtrusive way by someone other than the moderator. These methods also require someone to watch or
listen to the recording after the focus group, which can be quite time-consuming. Another option is to
simply have a note-taker in addition to the moderator, and this method can always be used as a fallback in case of audiovisual equipment problems.
2.2.2
Analyzing focus group data
Analysis of focus group data has been criticized because it can be easy to mold the data to fit
earlier assumptions. The sheer volume of data, combined with its complexity, can be overwhelming and
make rigorous analysis difficult (Krueger, 1994). However, when analyzed in a systematic way, a great
deal can be gleaned from focus group data. A general set of steps for data analysis is as follows (Rabiee,
2004):
Step 1: Go through the transcript (or notes) from the focus group and select those quotes that are
relevant to the question. There will likely be some responses that were irrelevant or unclear; set these
aside for now. This can be done either in a word processing program (be sure to save an original copy of
the transcript) or by hand using colored markers and scissors.
Step 2: Group similar quotes. If a particular quote or response is similar to one read earlier, put it in a
pile (or under a heading) with the earlier quote. If not, start a new pile (or heading).
Step 3: You now have a collection of relevant quotes grouped by topic, which can be interpreted and
analyzed for common themes and ideas. Consider the following factors when interpreting responses
(Rabiee, 2004):






Actual words used and their meaning. How do participants use particular words? How do they
define them?
Context. The way the moderator asked the question and previous comments made by other
group members influences the context in which the comments were made.
Frequency and extensiveness of comments. How often was a particular view expressed by the
same participant? How many participants expressed a particular view?
Intensity of the comments. Was the comment very emphatic? Did the respondent seem
emotional?
Internal consistency. Were there any changes in opinion or position by the participants?
Specificity of responses. Greater attention should be placed on responses referring to personal
experience than to those referring to hypothetical situations.
16
Funded Projects

2.3
Big ideas. What larger trends or concepts emerge from all of the responses? What about
responses from other focus groups?
Timing of data collection
In farming communities, little is constant throughout the year. Income often comes in a single
payment for the harvest, diet depends on the foods available in each season, and even certain diseases
may be more prevalent during the rainy season. Apparent project impacts may therefore vary
throughout the year with conditions created by these seasonal variations. Therefore, data for
monitoring and evaluation should ideally be collected throughout the year in order to reflect conditions
throughout the entire year. Constant monitoring and evaluation, however, is prohibitively expensive for
most projects. The best alternative is to pick a time of year in which to conduct certain M&E activities
and then collect information at the same time each year. Though data collected in this way may not
reflect conditions throughout the year, it will be more likely to capture true changes from year to year
than data that is collected at a different time each year.
2.4
Designing a baseline study
Baseline data is critical to monitoring and evaluation, especially for longer term projects (more
than five years). Without data collected before the beginning of the project (or shortly thereafter), it is
impossible to determine whether the changes in beneficiaries’ lives are due to the project, or if any
changes have happened at all (Shah et al., 2004). In essence, evaluation without baseline data is like
conducting an experiment without a “control” group. With this in mind, GMCR partners should make
every attempt to decide on indicators to use for monitoring and evaluation before the project begins. A
strong grant proposal will include baseline data on M&E indicators that has been collected from a preproject diagnostic, but GMCR understands that this is not always possible.
A baseline study is essentially the beginning of the monitoring and evaluation process. Baseline
studies should be carefully designed, as future evaluation efforts should use the same methodology and
the same indicators employed in the baseline study (Table 2.1).
Table 2.1. Conducting a baseline study, step by step
1. Clarify project objectives.
2. Determine the "unit of study" for monitoring and evaluation (e.g. households or individuals), based on
how your project works with participants.
3. Select the indicators that you will use for monitoring and evaluation based on your project objectives
(see sections 3 and 5) and those that are logical to use in a baseline study.
4. Choose methods of data collection (e.g. interviews).
5. Identify your group of direct project participants. You may not yet know who will end up participating
in the project, but at least identify the community in which the project will work an approximate
number of likely participants.
17
Funded Projects
6. Select an appropriate sample of project participants (see section 2.1.2).
7. Based on the indicators that you will use, create a comprehensive inventory of the data that will need
to be collected.
8. Use this inventory to create a questionnaire, interview guide, or other data collection tool.
9. Pre-test your data collection tool with a small sample of participants. Clarify and change the tool as
needed.
10. Determine a method for digitally organizing the data to be collected. If possible, set up a table or
database for data entry (see section 2.1.3).
11. Clarify data analysis methods. Draft a list of what data tables will be prepared and what indicators will
be calculated. This provides a final chance to alter the data collection tool if the information to be
collected does not align with the desired analyses.
12. Assemble a team to carry out data collection.
13. Train the data collection team in proper interview or focus group methods. Have the team conduct
practice interviews under the supervision of more experienced interviewers.
14. Establish a timeline for data collection.
15. Carry out data collection. If possible, have the data collection team enter information into a computer
soon after conducting the interview or focus group.
16. Once all data is collected and entered, review the data and "clean" it. Throw out any seemingly
erroneous or ambiguous responses.
17. Analyze data (see sections 2.1.3 and 2.2.2). Calculate indicators and draft tables for data analysis.
Create an outline for narrative information to be included in the baseline report.
18. Draft baseline report.
19. Ensure that data is safely stored and backed up.
20. Consolidate data collection materials/tools, and make a plan for the next iteration.
18
GMCR Monitoring and Evaluation 3 Quantitative Output Indicators
Funded Projects
3. Quantitative Output Indicators
19
Funded Projects
3.1
Required output indicators for all projects
While objectives differ between projects, monitoring outputs or performance can be conducted
in a similar way. Thus, what follows is a common set of output (performance) indicators for all GMCRfunded projects. Grantees should report on all indicators, or note if an indicator does not apply to their
project.
3.1.1
Numbers of direct and indirect project participants
All GMCR-funded projects should monitor the number of individuals or households reached by
each project activity. Each project defines “direct” and indirect” project participants differently. Direct
participants might, for example, be people who receive business loans; indirect participants might be
household members of these people, who indirectly benefit from the loan. Projects that work at the
household level might consider “direct” participants to be households that have received assistance in
establishing a vegetable garden; indirect participants might be other community members who benefit
from improved availability of produce in the local market. Not all projects have both direct and indirect
participants. It is up to each individual organization to describe, when reporting these numbers, how
their project defines and measures direct and indirect participants. These definitions should then be
used consistently for monitoring and evaluation purposes throughout the project. In reporting the
impact indicators described later in this guide, be clear about whether the impacts refer to the group of
direct or indirect participants. GMCR will assume that impact indicators were measured for direct
participants unless otherwise specified in a project’s M&E report.
3.1.2
Capacity building activities
Projects that provide trainings or classes (such as teacher trainings, volunteer health worker
trainings, or workshops in financial management or agricultural methods) should monitor the types of
training provided, the length of each training, the number of participants, and the number of trainings of
each type. Trainings should be reported as in Table 3.1
Table 3.1. Example table for reporting training activities
Type of training
Number of trainings
Length (hours) of
each training
Number of people
per training
Field Workshop
2
6
5
Classroom training on
production practices
Home visits to discuss
vaccination
6
2
25
20
1
1
20
Funded Projects
3.1.3
Direct Investments
Projects that provide direct financial support (e.g. loans, scholarships), or in-kind assistance (e.g.
books, cook stoves, water filters, radios, etc.) should monitor and report the amount of these direct
investments by the number of units and the total dollar value of such funds or materials distributed.
21
GMCR Monitoring and Evaluation 4 Most Significant Change Methodology
Funded Projects
4. Most Significant Change Methodology of
Qualitative Project Evaluation
22
Funded Projects
4.1
Introduction
During the initial meeting of the GMCR Reporting Collaborative in May, 2010 it was agreed that
in addition to quantitative monitoring and evaluation (i.e. collecting “numbers”), it is important for
GMCR partners to collect qualitative information about project impact as well. This frequently takes the
form of “stories” about how peoples’ lives have changed because of the project. Stories are a powerful
and important way to assess and communicate project impact, and a systematic method for collecting
and reviewing them allows for reflection and learning among project staff and funders.
Qualitative evaluation of GMCR-funded projects will be based on the Most Significant Change
(MSC) methodology developed by Davies & Dart (2005). The essence of the Most Significant Change
(MSC) protocol for qualitative monitoring and evaluation (Davies & Dart 2005) is the systematic
collection of stories about how peoples’ lives have changed due to the project, after which participants
and project staff at increasingly higher hierarchical levels narrow the pool of stories to those
representing the most significant changes resulting from the project. At each level of selection, the
reasons for selecting a particular story are then communicated back to the previous level. By this
method, staff at higher levels in the organization can learn the priorities and perceived positive and
negative impacts of participants and field staff, and vice versa (Davies & Dart 2005). The goal is a deeper
level of reflection and understanding of how projects are working on the ground and what adjustments
might need to be made.
GMCR nonprofit partners should collect one story for every 100 direct participants (as defined
in section 3.1.1) involved in the project, with a minimum of 10 stories collected, per year in
preparation for the annual report. If the direct participants include a number of different communities,
coffee cooperatives, or cultural groups, the sample should be stratified to include a representative
sample of beneficiaries. From the stories collected, choose the three stories that represent the most
significant changes to include in the annual evaluation report to GMCR.
4.2
The Most Significant Change process
4.2.1
Step 1: Define the collection period
Every organization must balance costs and benefits when defining a collection period for
monitoring and evaluation. For reporting to GMCR, we suggest collecting stories over a three-month
period on a yearly basis. Stories can also be collected on a continuous basis, as field workers interact
with beneficiaries. However, if stories are collected continuously, there must still be a defined period for
review and analysis of stories.
4.2.2
Step 2: Collect stories about significant changes
The core of the MSC method of qualitative monitoring and evaluation is an open question to
project participants, such as:
‘Looking back over the last year, what do you think was the most
23
Funded Projects
significant change in the quality of people’s lives in this community?’
This question contains six parts (Davies & Dart, 2005, 23):
1. ‘Looking back over the last y ea r …’ – It refers to a specific time period.
2. ‘…what do you think was...’ – It asks respondents to exercise their own judgment.
3. ‘…the most significant…’ – It asks respondents to be selective, not to try to comment on
everything, but to focus in and report on one thing.
4. ‘…change…’ – It asks respondents to be more selective, to report a change rather than static
aspects of the situation or something that was present in the previous reporting period.
5. ‘…in the quality of people’s lives…’ – It asks respondents to be even more selective, not to
report just any change but a change in the quality of people’s lives. If an organization wishes
to collect MSC stories around a different theme (people’s participation in that
organization’s activities, for example), this could be modified (‘…in people’s
participation’, for example).
6. ‘…in this community?’ – T his establishes some boundaries.
While the question used need not be identical to this one, the documentation of the story should
contain the following information (Davies & Dart, 2005):
1. Information about who collected the story and when the events occurred
2. Description of the story itself – what happened
3. Significance (to the storyteller) of events described in the story.
Depending on the scope of the project being evaluated, the most significant change in people’s lives
may not have anything to do with the project itself. It is always difficult to attribute causation to
qualitative changes. In order to prompt reflection on changes resulting from the project, an additional
qualifier may be added to the end of the question, such as:
‘Looking back over the last year, what do you think was the most
significant change in the quality of people’s lives in this community that
resulted from this project?’
Other modifications to the question may be made as well; see Box 3.2. To draw out the significance of
the story to the storyteller, an additional question may be asked at the end of the story, such as “Why is
this significant to you?” or “What difference has this made now or will make in the future?”
Most stories should be one to two pages in length. Shorter stories may be quicker and easier to read,
but they may leave out important information. Negative as well as positive changes may be
documented; much can be learned from negative changes as well. Information about the story should
be documented along with the story itself, including who collected the story, when the events
occurred, and the significance (to the storyteller) of the events described in the story. This gives some
24
Funded Projects
context for the story and enables follow-up inquiries about the story, if needed. An example story
template can be found in Appendix 1.
Box 3.2: Issues with phrasing the question
Eliciting good stories isn’t always easy. Difficulties sometimes have to do with how the question
has been translated. “Significance,” for example, does not always translate easily. In these cases,
it may help to break down the question into a series of questions. Jess Dart relates her
experience working in Papua New Guinea:
“I did not find it easy collecting the stories using the MSC question technique; people did not
immediately understand what I was trying to get at. This may be much easier in Tok Pisin, but in
English it needed more prompts to get an in-depth answer. In the end, I used a modified version
of MSC where I asked the following four questions:




How have you been involved in the project?
What are the important changes that have resulted from this project for you?
What are the important changes that have occurred in the community as a result of this
project?
What problems were there?
The story seemed to emerge from any of those four questions, depending on the experience of
the participants” (Davies & Dart, 2004, 46).
4.2.3
Step 3: Selecting the stories of most significant change
The Most Significant Change methodology is built on purposive sampling. This means that it is
selective rather than inclusive. It is not intended to represent the “average” condition of participants,
but rather to highlight particularly unusual or successful cases and learn from those. While some may
argue that this is not a reliable sampling methodology, it is a legitimate and widely used form of data
collection in qualitative research. In many cases, more can be learned from an in-depth study of a few
information-rich cases than from the average of a group (Davies & Dart, 2005).
The purposive step comes once all the stories have been collected. MSC uses a hierarchy of
selection processes. People discuss significant change stories within their level and then submit the
most significant to the level above, which then selects the most significant of all the stories submitted by
the lower levels and passes this on to the next level (if one exists) (Davies & Dart, 2005). GMCR
recommends using two levels of selection, but composition of the selection groups will vary depending
25
Funded Projects
on the structure of the organization. Most organizations have a hierarchical structure, and it makes
sense to use this existing structure to organize the selection process. The first level of selection, for
example, might be field staff, and the second level might be country office or HQ staff (Figure 4.1).
Figure 4.1. Flow of stories and feedback in MSC (adapted from Davies & Dart, 2005)
B
H
J
B
D
E
H
J
Stories selected by field staff
A
B
C
D
E
F
G
H
I
J
Flow of feedback
Flow of stories
Stories selected by organizational
headquarters
Stories from story tellers
At each level of story selection, a group sits down together with a pile of stories and selects one
or more stories as the most significant. The number of stories selected at each level will depend on the
size of the project and the total number of stories collected. The group may either decide beforehand on
a set of criteria for selecting stories, or allow the criteria emerge from the process. Often, selecting most
significant change stories without pre-set criteria clarifies the priorities for the project and the changes
that staff and beneficiaries want to see. A simple way to select stories is to discuss them, have each
person score them, and aggregate the scorings to choose the top stories. A facilitator can help move the
selection process along; an example facilitation guide can be found in Appendix 2. Regardless of the
method chosen, the key ingredients to story selection are that:



Everyone reads the stories
The group discusses which stories should be chosen
The group decides which stories are felt to be most significant
26
Funded Projects

The reasons for the group’s choices are documented (Davies & Dart, 2005).
The last step is particularly important as it is critical for reflection on project goals and accomplishments.
The reasons for selecting a significant change story should be documented during the selection process
(it helps to have a note taker to do this), as leaving it until the end creates a risk of eliminating or rushing
through this important step. This documentation should be attached to the story following the
explanations given by people who initially documented the story.
Box 3.3: The “conspiracy of courtesy”
Candid opinions can be difficult to come by. In many (perhaps most) cultures, people avoid
saying things that may be perceived as critical, particularly to outsiders. This is exacerbated
by the power dynamic between program staff (who may be seen as wealthy and educated)
and program participants (who may feel poor and uneducated by comparison). Participants
may know of problems with the program, such as corruption or poor staff performance,
but be reluctant to say anything (Bunch, 2000).
This hesitance may be difficult to overcome initially, but as participants become more
comfortable with program staff, they will likely become more open. Program staff
collecting significant change stories should ideally be people well-known to and trusted by
the beneficiaries. Most importantly, beneficiaries must know that openly criticizing the
program will not lead to repercussions or bad feelings, but instead that their suggestions
will be taken seriously and acted upon (Bunch, 2000). Candid opinions from participants
are a sign that they feel a sense of ownership of a program and want to see it improve.
4.2.4
Step 4: Reporting to GMCR on the Most Significant Change
The three stories selected during step 3 should be included in the yearly evaluation report to
GMCR. They may be formatted as in Appendix 1, with an additional section explaining why these stories
were selected and what process was used to select them.
4.2.5
Step 5: Feeding back the results of the selection process
GMCR does not require this step, but it is highly recommended as it is an integral part of the
MSC process. Once the most significant change story (or stories) is chosen, the results of the selection
process (what stories were chosen as most significant and why) should be communicated to those who
provided the MSC stories and carried out lower levels of selection. All stories, including those that were
not chosen, may also be shared with the community of participants, and this is an opportunity to learn
from all of the stories that were collected. There are several reasons to provide feedback. One is that
information about what stories were selected at higher hierarchical levels can help field staff target their
27
Funded Projects
search for stories and clarify the priorities of the organization. Providing feedback also shows that others
have read and analyzed the significant change stories. For project participants, hearing feedback on MSC
can provide motivation and ideas for meeting their own goals, as well as lift morale and make the
monitoring and evaluation process more transparent. There is a risk, however, that providing
information about what changes the organization does and does not value might be interpreted as the
organization telling communities how they should develop. (Davies & Dart, 2005). It is up to the
organization as to whether providing feedback on MSC story selection to participants is appropriate for
the organization’s structure and relationship with the community. Providing feedback to field staff,
however, is generally positive, and GMCR recommends this step.
There are several ways to provide feedback to field staff and project participants, and GMCR
leaves it up to each grantee to choose the method that works best for their organization. For providing
feedback to field staff, the results of MSC may be communicated verbally during a staff meeting, sent via
email, or written up in a more formal report. Some MSC users place the selected stories and the reasons
for their choice in a newsletter that is circulated to all field staff and/or participants. If an organization
chooses to provide feedback on MSC to participants, this feedback may need to be in verbal form,
depending on the literacy level of the participants (Davies & Dart, 2005).
4.3
Practical tips for the MSC process
How should stories be collected? There are several ways to identify and document significant
change stories. Field workers can write down unsolicited stories heard during the course of their work.
Fieldworkers can also more formally interview beneficiaries and write down their stories. This method is
most effective if the interviewer reads his or her notes back to the storyteller to ensure that they have
accurately captured the significant change story. Beneficiaries can also write their stories directly,
though this may not work with illiterate populations (Davies & Dart, 2005). GMCR suggests incorporating
story collection into normal field visits. Notes should then be reviewed soon after the field visit or
interview in order to capture as much detail as possible. However stories are collected, it is helpful to
have two people on the story collection team: one to ask questions and one to take notes so that all of
the information is captured. Another option is to record the interview with a good-quality recorder, but
keep in mind that this then requires each interview to later be transcribed, which can be a timeintensive process.
Who should collect stories? Who collects MSC stories will depend on the nature of the project.
This guide has been written assuming that project field staff will be collecting MSC stories. However, a
team from outside the project, or a group of participants from the community could also be trained to
collect stories. In general, it is best if the people collecting the stories speak the local language and
understand the local culture so as to avoid losing detail in interpretation. The sensitivity of the issues
that may come up in the stories is also a consideration; if the project deals with sexual occupations, for
example, the people collecting MSC stories must be known and trusted by project participants.
28
Funded Projects
How should people be selected to provide MSC stories? Not every participant has a story to tell.
Some MSC practitioners find that it helps to announce to the community ahead of time that project staff
will be looking for stories, emphasizing that suggestions about things to improve are also welcome. This
gives participants time to think about their stories, and allows those who have stories to share to come
forward. It is also important to ask for stories from more marginalized people, such as those in isolated
locations or those who are shy and not expecting to be asked.
4.4
Adapting MSC to your organization
GMCR grantees work with communities in a variety of ways; not all projects have a hierarchical
staff structure that easily lends itself to MSC. In particular, U.S.-based grantees that do not have field
staff “on the ground” and instead rely on partnerships with cooperatives or local organizations may
need to train staff from those organizations to carry out MSC. Such organizations may choose to
incorporate an additional level into the MSC selection process; stories may come from participants, be
first selected by field staff of the partner organization/cooperative, then by administration of the
partner organization/cooperative, then lastly by staff of the U.S.-based organization. It is the
responsibility of the GMCR grantee to provide whatever training is necessary to partner organizations
or cooperatives in order to carry out the MSC methodology.
Training is required to conduct MSC. A good place to start is by reading the entire MSC manual
by Davies & Dart (2005) which is available in English and Spanish on the web. Organizations unfamiliar
with MSC may want to hire a consultant to train upper-level staff and/or field staff. If necessary, costs
for MSC training may be included in the M&E budget in grants submitted to GMCR.
29
GMCR Monitoring and Evaluation 5 Quantitative Impact Indicators
Funded Projects
5. Quantitative Impact Indicators
30
Funded Projects
5.1
Introduction
The following indicators should be used for measuring how well a project is achieving its desired
objectives. The indicators are arranged by project objective; these objectives are intended to align with
GMCR’s objectives for its supply-chain outreach social responsibility funding. Grantees should identify
the objective(s) that are closest to their project’s objectives in their grant proposal and use the
indicator(s) described under those objectives for Monitoring and Evaluation. Some objectives have
required indicators associated with them; some objectives require you to select among several optional
indicators from which you should choose one or more that most closely fit your project.
5.2
Project objective: Build capacity
5.2.1
Required indicator: Continued use of new knowledge
Nearly all GMCR-funded projects involve capacity building in some form. The goal of any
capacity building activity is to teach someone a behavior or skill that they will then use in their personal
or livelihood activities. This indicator applies to a range of capacity building activities, such as financial
management training for business owners or cooperative members, workshops to train schoolteachers
in new teaching skills, or health promotion activities by community health providers.
Each capacity building program or activity should have defined
objectives around the skills that a participant should be able to
demonstrate at the end of the training. We suggest following up with
participants 3 months and 1 year after the training program to ask if they
have used (and are still using) the skills that were learned in the program.
For example, for a program teaching how to fill out a loan application, it
would be appropriate to ask participants if they have applied for a loan
and whether or not they received the loan, and report the percentage of
participants who have applied for a loan and the percentage of those who
have obtained a loan. Likewise, for home visits to encourage child
vaccination, report the percentage of parents who received home visits
who brought one or more of their children to be vaccinated.
Capacity-building
activities might include
financial management
training, health
promotion, or teacher
training.
As with other indicators, if the number of participants who received capacity building is very large,
grantees need only follow up with a sample of direct participants, using the guidelines in section 2.1.2. If
desired, reporting on capacity building may be broken down by type of capacity-building activity, as
shown in Table 3.1.
5.3
Project objective: Create employment opportunities
5.3.1
Required indicator: Number of jobs created by project activities
Small enterprises not only support the livelihoods of those that own them; they can also be an
engine for economic growth via employment of others in the community. The number of jobs created by
31
Funded Projects
such enterprises is one measure of the success of credit or farm diversification programs. In reporting
this indicator, be sure to distinguish between jobs directly and indirectly created by the project. For
example, a woman who receives a loan to keeping bees and selling honey is a job directly created by the
project. If that woman employs another person to help market her honey, that job has been indirectly
created by the project. In very small enterprises where employment may be informal or sporadic,
enterprise owners can be asked the number of people who worked for them in the past month (as
opposed to the number of full-time employees) (Dunn, 1999). Direct employment by the project (i.e. the
hiring of local field staff) should not be included in this indicator.
5.4
Project objective: Increase food security
5.4.1
Required indicator: Months of adequate household food provisioning (MAHFP)
The months of adequate household food provisioning (MAHFP) indicator was developed by
USAID as a way to measure household food access. The ability of households to access sufficient food
(though crop production, purchase, bartering, or food transfers from relatives or governments) may vary
throughout the year due to factors such as crop disease or loss of income (Bilinsky & Swindale, 2007).
This is especially true in coffee-growing communities, where income from coffee tends to be
concentrated in a short period rather than evenly distributed throughout the year. The goal of food
security programs is to reduce vulnerability to factors that result in insufficient food in order to increase
the time during the year that households are able to meet their food needs.
This indicator is based on asking two questions of the sample population:
1. In the past 12 months, were there months in which you did not have enough food to meet your
family’s needs?
2. If yes, which were the months (in the past 12 months) in which you did not have enough food to
meet your family’s needs?
These questions should be asked of the person who is responsible for meal preparation or
another adult (e.g. the household head) if that person is unavailable. The questions refer to the
household as a whole, not to any single person within the household (Bilinsky & Swindale, 2007).
The months of adequate household food provisioning (MAHFP) variable is calculated for each household
as:
(12) – (total number of months cited in question 2)
If the household cited no months when they were unable to meet their food needs, their
MAHFP variable is 12.
The months of adequate food provisioning indicator is the average of the MAHFP variable for all
the households in the sample population. Be sure to include even those households that responded
“no” to question 1 (In the past 12 months, were there months in which you did not have enough food to
meet your family’s food needs?), otherwise the estimate of food insecurity will be too high (Bilinsky &
32
Funded Projects
Swindale, 2007). The MAHFP indicator should be tracked over multiple years to assess changes in the
length of the “thin months.”
5.4.2
Required indicator: Dietary diversity
Dietary diversity is another measure of food access that captures the quality of the diet
consumed by an individual or household. Dietary diversity is significantly correlated with caloric and
protein adequacy in the diet (Swindale & Bilinsky, 2006) and childhood nutritional status (e.g. height-forage scores) (Arimond & Ruel, 2004). It is “among the most common and valid indicators of nutrient
adequacy and/or energy intake” (Maxwell, Caldwell, & Landworthy, 2008, 534). Measures of dietary
diversity are based on the number of food groups that a household or individual consumes during a
designated time period. For the purpose of reporting to GMCR, a modified version of the household
dietary diversity score (HDDS), an indicator developed by USAID, will be used.
Data for the HDDS is collected by asking the respondent (preferably the person who prepares
the food) the number of days that they consumed certain foods during the reference period. The
reference period should be the seven days prior to the interview. The consumption frequencies (number
of days a food was eaten) are then summed and divided by seven to obtain the household’s dietary
diversity score, which represents the average number of food groups eaten per day.
The dietary diversity scale uses 12 food groups to estimate dietary diversity. These food groups
are:
Group
A. Cereals
B. Root and tubers
C. Vegetables
D. Fruits
E. Meat, poultry, offal
F. Eggs
G. Fish and seafood
H. Pulses/legumes/nuts
I. Milk and milk products
J. Oil/fats
K. Sugar/honey/sweets
L. Miscellaneous
Table 5.1 shows an example of a questionnaire and calculation table for collecting data about
dietary diversity. This questionnaire should be modified to remove foods that are not eaten in the local
area and include the foods that are most common.
33
Funded Projects
Table 5.1. Example data collection format for household dietary diversity scale (adapted from
Swindale & Bilinsky, 2006)
I would like to ask you about the types of foods that you ate during the
past 7 days.
# days
(Read the list of foods. In the first box, write the number of days out of
the past seven days that someone in the household consumed that
food)
Any bread, rice, noodles, tortillas, or any other food made from maize,
rice, wheat, sorghum, millet, quinoa or (insert any other locally 7
available grain)?
Any potatoes, yams, yucca/manioc/cassava, or any other foods made
from roots or tubers?
5
Any vegetables?
3
Any fruits?
4
Any chicken, beef, pork, lamb, goat, rabbit, wild game, duck or other
birds, liver, kidney, heart, or other organ meats?
2
Any eggs?
5
Any fresh or dried fish or shellfish?
0
Any foods made from beans, peas, lentils, or nuts?
6
Any cheese, yogurt, milk, or other milk products?
3
Any foods made with oil, fat, or butter?
5
Any sugar, honey, or sweets (cookies, etc.)?
5
Any other foods such as condiments, coffee, or tea?
7
Total
Dietary diversity score (total above divided by 7)
52
7.4
34
Funded Projects
For tracking dietary diversity over many years, the dietary diversity score should be collected
during the same month each year, preferably during the usual time of lowest food security (“the thin
months”). This indicator can be modified to apply to the household level by beginning the questionnaire
with “I would like to ask you about the foods that you or anyone in your household ate during the past
seven days.”
In your monitoring and evaluation report to GMCR, please include both the average dietary
diversity score among the individuals or households that were interviewed and the average and range
of scores (lowest and highest) for each food group across the sample, as in Table 5.2. For example, of
20 families that were interviewed on dietary diversity, on average those families ate meat 4.2 times a
week, with the low end of the range reporting eating meat 0 times a week and the high end of the range
eating meat 6 times a week.
Table 5.2. Example reporting table for dietary diversity
Food Group
Number of days consumed
Average
Range
Lowest score Highest score
Cereals
Roots and tubers
Vegetables
Fruits
Meat, poultry, offal
4.2
Eggs
Fish and seafood
Pulses, legumes, nuts
Milk and milk products
Oils, fats
Sugar, honey, sweets
Miscellaneous
Average dietary diversity score:
0
6
8.2
5.4.3
Optional indicator: Coping Strategies Index
(CSI)
When people do not have enough food to eat, they
undertake a variety of behaviors to cope with the situation.
The ways that people cope with not having enough food
vary according to the local situation and the severity of the
food shortage. The Coping Strategies Index (Maxwell et al.,
2003) was developed as a way to assess the severity of food
shortages based on the idea that some coping behaviors
“What do you do when you don’t
have enough food, and don’t have
enough money to buy food?”
This question is the basis of the CSI
tool (Maxwell, Watkins, Wheeler, &
Collins 2003).
35
Funded Projects
are more “extreme” than others, i.e. they reflect a more severe food shortage. The Index is intended
primarily as a tool for rapid assessment of food shortages in emergency situations, however it correlates
well with other measures of food security (Coates et al., 2006; Maxwell et al., 2003) and can also be
used for evaluating the effectiveness of longer-term food security programs.
Food insecure households typically employ some of four types of coping strategies. Households
may change their diet by switching from preferred foods to those that are cheaper. Second, they may
try to increase food supplies through credit or borrowing, or by eating wild foods or seed stocks. Third,
they may decrease the number of people to be fed by sending children to a neighbor’s house to eat or
temporarily migrating to urban areas. Finally, households may simply ration food by skipping meals,
eating smaller meals, or even going entire days without eating (Maxwell et al., 2003). The Coping
Strategies Index only includes food consumption coping strategies, not livelihood coping strategies.
Consumption coping strategies are those that:





Are related specifically to food consumption
Can be done immediately, do not require long-term planning
Are reversible (i.e. can be reversed when no longer needed)
Can be used continuously (selling livestock, for example, is a one-time strategy and therefore
not a consumption strategy)
Do not depend on the initial asset holdings of the household (because, for example, a household
may not sell assets either because they do not need to, or because they have none to sell,
therefore making this an unreliable indicator of food insecurity) (Maxwell et al., 2003)
The felt severity of these coping strategies varies; changing diet to rely on less expensive foods,
for example, is generally employed at less severe levels of food insecurity than going entire days without
eating. The basic idea of the Coping Strategies Index is to measure the frequency of coping behaviors
(how often is the strategy used?) and the severity of those behaviors (what degree of food insecurity do
they suggest?) and combine this information into a single score (Coates et al., 2006).
Coping strategies and their felt severity may vary with cultural context, and ideally, a series of
focus groups is used to establish a list of common coping strategies and their severities specific to a
community. See Maxwell et al. (2003) for instructions on how to do this. However, establishing such a
list requires a substantial investment of resources, and research has shown that rural people, even in
different countries, tend to use similar strategies to cope with food insecurity (Coates et al., 2006).
GMCR suggests using the internationally-validated list of coping strategies and severity weights shown in
Table 5.3.
36
Funded Projects
Table 5.3. A generalized list of coping strategies (Coates et al., 2006; Maxwell et al., 2003)
How often in the past 30 days did you:
1. Dietary change
a. Rely on less preferred and less expensive foods?
2. Increase short-term household food availability
a. Borrow food, or rely on help from a friend or relative?
b. Purchase food on credit, or borrow money to buy food?
c. Gather wild food, hunt, or harvest immature crops?
d. Consume seed stock held for the next season?
3. Decrease numbers of people
a. Send children to eat elsewhere?
b. Send household members to beg?
4. Rationing strategies
a. Limit portion size at mealtimes?
b. Restrict consumption by adults in order for children to eat?
c. Reduce number of meals eaten in a day?
d. Skip entire days without eating?
Severity
1=least severe,
3=most severe
1.0
1.7
1.7
2.1
2.3
2.0
2.0
1.7
2.0
2.0
3.0
Data for calculating the Coping Strategies Index should be collected through household
interviews, and most practitioners recommend interviewing women. GMCR recommends using a
reference time period of the 30 days preceding the interview; as with other food security measures, the
index will thus be dependent on the time of year in which the information is collected, so it is a good
idea to collect the information around the same time each year. The frequency of strategies can be
counted in various ways, but the most common is to allow the respondent to choose whether they
employed a strategy every day, fairly often (3-6 times/week), once in a while (1-2 times/week), hardly at
all (<1/week), or never. The midpoint of those frequency ranges is used for the relative frequency score
(Maxwell et al., 2003).
37
Funded Projects
Table 5.4. Calculation table for coping strategies index (adapted from Maxwell et al., 2003)
In the past 30 days, if there
have been times when you did
Every
not have enough food or
day
money to buy food, how often
has your household had to:
3-6
times/
week
1-2
times/
week
<1 time/
Never
week
Relative frequency score
4.5
1.5
0.5
7
Rely on less preferred and less
expensive foods?
Frequency
Score
Severity
weight
Score=
Relative
frequency
X
weight
4.5
1.0
4.5
0
X
Borrow food, or rely on help
from a friend or relative?
X
1.5
1.7
2.55
Purchase food on credit, or
borrow money to buy food?
X
1.5
1.7
2.55
0
2.1
0
Gather wild food, hunt, or
harvest immature crops?
X
Consume seed stock held for
next season?
X
0.5
2.3
1.15
Send children to eat elsewhere?
X
0.5
2.0
1
0
2.0
0
7
1.7
11.9
0.5
2.0
1
Send household members to
beg?
Limit
portion
mealtimes?
sizes
at
X
X
Restrict consumption by adults
in order for children to eat?
X
Reduce number of meals eaten
in a day?
X
0
2.0
0
Skip entire days without eating?
X
0
3.0
0
CSI SCORE
Sum the totals for each individual strategy
24.65
The frequency score for each coping behavior is multiplied by the severity weight, and these
values are summed to calculate the Coping Strategies Index score. The household score means little in
absolute terms, but it is useful for tracking changes over time. For example, if a household had a score of
92 in year 1, a score of 75 in year 2, and a score of 63 in year 3, their level of food insecurity was clearly
decreasing (or, put another way, their level of food security was increasing). The CSI indicator is the
38
Funded Projects
average of the scores for all the households interviewed. This indicator is not recommended for use at
an individual level rather than a household level, because coping behaviors are by definition undertaken
at the household rather than the individual level.
5.4.4
Optional indicator: Nutritional deficiencies among children
Anthropometry refers to the measurement of the human body. Changes in body dimensions
reflect the health and welfare of a population, and anthropometric indicators can be used to measure
the general nutritional status of an individual or group. Anthropometric indicators are an inexpensive
and widely used tool, particularly to assess the nutritional status of young children.
The four basic pieces of any anthropometric indicator are age, sex, height (or length for children
under two years of age) and weight. With these variables, a variety of indices can be calculated that
provide a representation of a person’s nutritional status.
Box 4.1: Measuring Age
An accurate estimate of the child’s age is necessary for calculation of anthropometric indices. The
interviewer should ask the mother or primary caretaker should be asked the child’s birth date, and
cross-check with a birth or baptismal certificate if available (Cogill, 2003).
Box 4.2: Measuring Height
A number of devices are available for measuring height of infants and children at a range of prices.
Cogill (2003) provides a comprehensive listing of these devices. Regardless of the device used, it is
important that methods of measurement be uniform between individuals and years. Children should
look straight ahead while being measured, with shoulders level and hands at their sides. The child’s
caretaker should be asked to remove the child’s shoes and any hair styling (e.g. braids) that would
affect the measurement. For children under two years of age, length is measured instead of height,
with the child lying down (Cogill, 2003).
Box 4.3: Measuring Weight
The simplest way to record weight is with an electronic scale. First, the child’s caretaker steps onto
the scale holding the child and the weight of the caretaker plus the child is recorded. The child is then
handed to someone else and the mother’s weight alone is recorded. The mother’s weight can then
be subtracted from the first weight to obtain the weight of the child (Cogill, 2003). Older children can
be asked to step onto the scale by themselves. Hanging scales are also available (Cogill, 2003).
39
Funded Projects
It would be impossible to collect this information on every minor participant, so GMCR
recommends taking a sample of households or adult individuals and collecting anthropometric data on
each child under 5 in the household (or in the immediate family of the adult individual). If you are
unable to assess all children due to financial constraints, the youngest child from each household should
be assessed. When collecting this data, it is important that the purpose and content of the
anthropometric survey are explained in a non-threatening and culturally sensitive manner, and that the
person to be surveyed is given the opportunity to ask questions and decline participation (Cogill, 2003).
It is best to do anthropometric measurements at the end of an interview or survey (if one is taking
place) to allow the respondent to become comfortable with the data collector before the measurements
are taken.
From these measurements, three indicators can be inferred. Weight-for-age identifies the
condition of being underweight for a specific age. This can result from either acute or chronic
undernutrition, and is unable to distinguish between the two. Height-for-age identifies stunting
resulting from past growth failure and is therefore an indicator of past undernutrition or chronic
malnutrition, though not necessarily current or acute undernutrition. For evaluating improvements in
food security, it is preferable to use this indicator with children under two years of age, because the
effects of stunting may not be reversible, and therefore using this indicator with older children may
under-estimate gains in food security. Weight-for-height identifies children suffering from current or
acute undernutrition, called wasting. Wasting is the condition of a child falling significantly below the
weight expected for a child of the same height and can result from failure to gain weight or actual
weight loss. As this indicator does not require age, it is useful when exact age cannot be determined
(Cogill, 2003).
In order for these indicators to be meaningful, they must be compared to a reference value. The
World Health Organization (WHO) publishes internationally validated growth standards for healthy
children from birth to five years (World Health Organization, 2006). Comparison to reference standards
is usually done using standard deviation units, called Z-scores. The WHO growth standards tables give
the cutoff values for weight-for-age, height/length-for-age, and weight-for-height/length at which an
individual is one, two, or three standard deviations away from the median. For example, a one-monthold male child measuring 45 cm would have a Z-score of -2 on the length-for-age reference table (Table
5.5). Individuals one to two standard deviations below the median are considered mildly malnourished.
Those two to three standard deviations below the median are considered moderately malnourished,
and individuals three or more standard deviations below the median are considered severely
malnourished (
Table 5.6). These classifications should be used for reporting purposes. WHO growth standards
can be downloaded from www.who.int/childgrowth/standards/en/. Reports on anthropometric
indicators to GMCR should include the percent of children under five years old that are mildly,
moderately, and severely underweight, stunted, and wasted. Monitoring and evaluation teams should
40
Funded Projects
ensure that their sample is sufficiently large and representative of the beneficiary group to allow for
meaningful comparison over time.
Table 5.5. Example Z-score reference table
Table 5.6. WHO malnutrition classifications
Z-score range
Weight-for-age
Height-for-age
Weight-for-height
Mildly
underweight
Mildly
Mildly
-2 < Z-score < -1
stunted
wasted
-3 < Z-score < -2
Moderately
underweight
Moderately
stunted
Moderately
wasted
Severely
underweight
Severely
Severely
Z-score < -3
stunted
wasted
5.5
Project objective: Increase crop yields
5.5.1
Required indicator: Crop yields
Improvements in crop yields may benefit households by providing more food or by providing
more income with which to purchase food, send children to school, or invest in savings or farm
improvements. Projects that have increases in crop yields as an objective may use yields as an impact
indicator. Yields should be reported in per-hectare units, and international system (SI) units should be
used whenever possible. As with any indicator, yields should be monitored over many years and a
41
Funded Projects
representative sample of project participants should be chosen to increase the chance that the change is
attributable to project activities and not to external factors such as weather.
5.6
Project objective: Improve access to credit
5.6.1
Required indicator: Loan repayment rates
Loan repayment rates should be reported on an annual basis as:



Number of loans that were repaid fully or are in the process of being repaid, and for which the
client has been on time with all payments
Number of loans that were repaid or are in the process of being repaid, but for which the client
has been late with one or more payments, and
Number of loans for which the client defaulted
For reporting defaulted loans, use your organization’s definition of default, and include in the M&E
report what that definition is.
5.6.2
Required indicator: Total revenue of funded enterprises
For monitoring and evaluation purposes, total revenue of funded enterprises should be
reported according to years of funding or years since loan disbursement, as newly-funded enterprises
would be expected to have lower revenues than those with several years of growth. Larger enterprises
would also be expected to have larger revenues than smaller ones. As such, we suggested classifying
enterprises according to time in the program, and size reporting changes in revenue for each of these six
groups (small, newly assisted enterprises; small enterprises in loan repayment; large newly assisted
enterprises; etc.) (Table 5.7).
Table 5.7. Categories for classification of enterprises
Time in program
Newly assisted this year
In loan repayment
Loan repaid
Number of employees
Less than 10 (small)
10 or more (large)
5.6.3
Optional indicator: Net revenue of funded enterprises
Enterprise net revenue is generally defined as gross revenue minus the cost of goods sold minus
production expenses (Hyman & Dearden, 1998). However, different organizations use different metrics
for calculating net revenue. If you use to choose this indicator, please describe in the M&E report how
your organization calculated net revenue. As with total revenue, net revenue should be reported
according to the classifications in Table 5.6.
42
Funded Projects
5.7
Project objective: Increase income
5.7.1
Required indicator: New income generated by project activities
This indicator is appropriate for projects that seek to increase household income through new
economic activities such as diversification of farm products or by helping participants access markets for
their products. “New” income is defined as income resulting directly from the economic activity
supported by the project and additional to the individual’s or household’s income prior to the project.
Collecting this information will require participants, with the help of field staff, to estimate their sales
and the approximate prices that they received for their products. While participants may not keep track
of their total new income, they will usually have an idea of how many times in a month that they went
to market or engaged in the economic activity and how much money they brought home each time.
This should be extrapolated to calculate annual income.
Additionally, GMCR is interested in what income sources are available to households both pre
and post-project. The reason for evaluating all income sources, and not just new ones, is that income
diversification projects sometimes replace existing income sources with new ones rather than increasing
the overall diversity of sources. The income sources available to each household pre- and post-project
should be recorded as in Table 5.8. Report to GMCR the average number of net income sources added
by all households included in the evaluation.
Table 5.8. Recording table for net income sources
Income sources
Honey
Coffee
Fruit sales
Net income sources added
Pre-project
X
X
Post-project
X
X
X
1
5.7.2 Optional indicator: Gross annual household income
The collection of income data is controversial due to the difficulty involved in collecting reliable
data, particularly from rural farming families. Absolute measures of household income are difficult to
estimate accurately, however with some effort on the part of the interviewer, it is possible to measure
changes in household income from year to year, as long as the same methods of data collection are
used. For projects with the objective of increasing household income, GMCR encourages the use of this
indicator.
Household income data will be most accurate if as many household members as possible are
available during the interview. Ask the respondent (or respondents, depending on who is present) to list
each person in the household, whether they earn income, and how they earn that income. Because
43
Funded Projects
most earnings tend to be piecemeal rather than a regular salary, also ask respondents if they sell crops
throughout the year, approximately how much, and at approximately what price. For other earnings,
such as occasional work off-farm, respondents can usually recall about how many days they work per
week and how much they earn per day. In areas where remittances are common, also ask specifically
about these, as respondents may not list children living abroad as household members. It is helpful if the
interviewer is familiar with the respondents and the community, as he or she will know the common
income-generating activities in that community, the usual prices for products, and the usual salaries for
labor.
Income should be reported as dollars of income and number of income sources. If desired, income
from project-funded enterprises can be reported separately (see section 5.5.1).
5.7.3
Optional indicator: Household savings
Savings are an important way that households buffer themselves against financial crises and
seasonal variations in income. Savings can be in the form of money saved at home or in a bank account;
when asking participants if they have savings, the interviewer should ask the participant about both
types of savings. Projects should report the percentage of households in the project (or sample) that
have savings at the time of evaluation, and the change in this percentage since the last evaluation
period. When collecting this data, interviewers may also ask respondents about the reasons why their
households have been able to put away savings (or not), as this information will give context to the
numerical figure.
5.8
Project objective: Improve educational access
Educational indicators contain information about the status, performance, or quality of an
educational system (Greaney & Kellaghan, 1996). The indicators used to evaluate program impact will
vary based on whether the program objective is to improve school attendance or improve educational
quality (Rodríguez, Sánchez, & Armenta, 2010). Regardless of the indicator used, it is important to
collect information broken down by gender, as differences often exist between girls’ and boys’ school
enrollment and achievement (Grant & Behrman, 2010).
5.8.1
Required indicator: Percent of eligible children attending school
For the purposes of reporting to GMCR, “eligible children” should be defined as all children under
the age of 18 living in households that are direct participants in the project. As differences often exist
between girls’ and boys’ school enrollment (Grant & Behrman, 2010), and enrollment tends to decline
with age, school attendance data should be stratified by sex and by age and school level, as shown in
Table 5.9. Provide the usual grade range for primary, middle, and secondary school in the project
country.
The percent of children attending school is only a meaningful indicator when compared to baseline
data, because as the project gains more participants (for example, in the case of scholarship projects),
44
Funded Projects
the percent of eligible children attending school will likely remain constant, as these new eligible
children will be scholarship recipients.
Table 5.9. Categories for reporting school attendance
Percent of eligible children
attending school
Primary school
Middle school
Secondary school
Usual grade range
K-5 grade
6-9 grade
10-12 grade
Girls
92%
55%
44%
Boys
95%
68%
37%
Girls: Baseline
85%
33%
34%
Boys: Baseline
90%
60%
33%
5.8.2
Optional indicator: Grade attainment
Grade attainment refers to students being in the appropriate grade level for their age. This
information can be collected through surveys of participant households by asking parents to list all of
their children along with their ages and grade level in school. Be sure to ask whether each child is
currently attending school as well, as some parents may report the last grade that their child completed
without mentioning that the child is no longer in school.
Report grade attainment by school level and sex, as in Table 5.7. Report two figures: the percent
of children who are at the grade level appropriate for their age (as defined by the educational system incountry) and the percent of children who have progressed in grade level since the previous year.
5.8.3
Optional indicator: Percent of reproductive-age women in school
In some contexts, girls are more likely than boys to leave school after the onset of adolescence
(Biddlecom, Gregory, Lloyd, & Mensch, 2008). This may be due to a variety of factors, including
increased domestic responsibilities, early marriage, early pregnancy, or harassment from teachers or
male classmates. The percent of reproductive-age girls (between the ages of 10 and 18) in school is an
indicator of how effective projects are in helping girls overcome the barriers to completing their
education.
5.9
Project objective: Improve educational quality
Projects aimed at improving educational quality generally focus on building the capacities of
teachers. If your project aims to improve educational quality via teacher capacity building, use indicator
5.6.1 as a required indicator. Because it is difficult to measure changes in student learning as a result of
teacher capacity building activities, particularly in schools that do not administer standardized tests,
further indicators for this objective are optional.
45
Funded Projects
5.9.1
Optional indicator: Performance on end-of-year tests
Programs aimed at improving the quality of education in schools may use scores on
standardized tests to evaluate the improvement in learning at certain schools or among certain groups
of students (Rodriguez et al., 2010). Not all countries or local educational ministries administer
standardized tests every year, but most have some sort of criteria for advancement to the next grade.
Average scores on standardized tests (or, in their absence, grade advancement) among project
participants should be monitored from year to year to measure improvements in school performance. In
the absence of baseline information, scores among project participants can also be compared to school
or regional averages (if such data is available) to measure the impact of the project on school
performance.
5.9.2
Optional indicator: Literacy rate
Measurement of literacy rates is particularly applicable to adult education programs, and can
also be used to assess the quality of primary and secondary education. The United Nations Educational,
Scientific, and Cultural Organization defines literacy as “the ability to read and write, with
understanding, a short simple sentence about one’s daily life” (Terryn). The National Center for
Education Statistics defines literacy as “using printed and written information to function in society, to
achieve one’s goals, and to develop one’s knowledge and potential” (Kirsch, 2001).
Literacy assessment methods can be classified into school-based and household-based
assessments. School-based assessments rely on in-classroom testing of students’ ability to read and
write at grade level. Depending on the availability of results from these tests and the trustworthiness of
school officials in reporting the results, this is the least costly way of obtaining literacy data because it
does not require program staff to conduct literacy assessment directly. Such assessment methods are
most appropriate for programs that work through or with schools. Reports to GMCR should include, at
minimum, the percent of children who are reading and writing at grade level.
Household-based assessments require literacy assessment by program staff (often in the home
of the program participant), and are thus more time-consuming and expensive, but may be the only way
to measure the success of literacy programs that are not school-based. Most household surveys used in
developing countries elicit information on an individual’s literacy by asking the individual (or another
person in the household) to offer their opinion on whether that individual is literate. Recently, literacy
assessment has moved away from this method—which is thought to be quite inaccurate—to direct
assessment of the cognitive skills associated with reading and writing (Schaffner, 2005). The simplest
method of assessing “decoding” skills—which form the foundation for reading—is to have respondents
read aloud a sentence written on a flash card. This sentence should be something understandable and
universal across cultural contexts, such as “Parents love their children.” Care should be taken in
developing, pre-testing, and translating the sentence to be used (Schaffner, 2005).While this method
does not capture the spectrum of levels in literacy skills, it is the measure most comparable across
contexts and thus we encourage its use as a minimum in assessing GMCR-funded projects where school© 2012 Green Mountain Coffee Roasters, Inc.
46
Funded Projects
based assessment is not possible. More involved measures of literacy are also possible; a variety of
example surveys are available in Schaffner (2005) to assess reading, writing, and numeric skills (which
have recently begun to be included in literacy tests).
5.10 Project objective: Improve access to health services
Health-related projects in coffee communities differ in their objectives. Some may target
maternal and child health, others may deal with environmental health issues such as clean water and
sanitation, while still others may prevent and treat specific diseases such as malaria, AIDS, or cancer.
Methods of achieving those objectives also differ; some programs focus on education and preventative
behaviors, while others screen for and treat disease. As such, indicators for evaluating impacts of health
projects will differ according to the goals and methods of the project. Choose whichever of the two
following indicators most closely matches project objectives. For health promotion activities, indicator
5.6.1 may also be used to measure uptake of preventative health behaviors.
5.10.1 Optional indicator: Treatment and resolution (acute conditions)
Health projects that provide treatment for acute illnesses should measure the success of their
efforts by the number of patients seen who were ultimately able to resolve their illness. This
information should be reported as number of patients seeking treatment for acute illnesses, number of
patients who received treatment, and number of patients whose illness was resolved. Also report the
number of patients whose illness was not resolved due to the patient not returning for follow-up
treatment and the number of patients whose illness was not resolved due to a difficult-to-treat
condition. Care for patients with chronic or difficult-to-treat conditions should be evaluated using
indicator 5.8.2.
5.10.2 Optional indicator: Detection, treatment, and loss to follow-up (chronic conditions)
Health prevention work has several components. A population is screened for a particular health
issue. The program then aims to provide treatment for those at risk for the health issue. Success of the
project is then measured by the number of persons who receive treatment. For this indicator, report the
number of people who received screening (these are considered direct participants), the number that
were identified as needing treatment, the number of those identified that actually received the
treatment, and the number that were lost to follow up. This may be reported by health issue if the
project works with more than one health issue (Table 5.10). For each health issue, be sure to describe
(in narrative form) what screening and treatment activities were conducted. This indicator may also be
modified slightly to apply to prenatal care; the “identified as needing treatment” group would be
pregnant women and the “received treatment” group would be pregnant women who returned to the
clinic for prenatal care and birth (or received home visits).
Table 5.10. Example reporting table for detection, treatment, and loss to follow-up
Health issue
Cervical cancer
Vision screening
47
Funded Projects
Received screening
1000
1000
Identified as needing treatment
150
100
Received treatment
140
90
Lost to follow-up
10
10
5.10.3 Optional indicator: Retention of health workers
Training of community health workers, whether in the form of advanced instruction for existing
doctors or basic community health training for volunteer health promoters, is an important part of
strengthening weak health systems (Rahman et al., 2010). However, it is sometimes difficult to retain
health workers in community health programs. The first level of assessment of project impact is the
retention of such workers. For this indicator, report the number of health workers who have received
training or support from the project, and the number who are still (currently) providing health services
to the community. This information may be stratified according to the health worker’s role in the project
or the type of training they have received, as in Table 5.11.
Table 5.11. Example reporting table for retention of health providers
Type of health provider
Clinic nurse
Visiting health provider Health promoter
Number who have received
training or clinic assignment
15
11
26
Number continuing to provide
care after 3 months
Number continuing to provide
care after 12 months
14
9
19
14
8
18
5.10.4 Optional indicator: Effectiveness of health workers
The continued effectiveness of health workers can be evaluated in several ways. One method is
through self-assessment, whereby health workers provide their own opinions of their skills,
commitment, support, and tools. Other methods include external assessment of the competency of
health workers in performing certain skills, and measures of whether health workers continue to use the
skills that they learned through the project. This indicator is comprised of three parts: self-perception,
competency, and use of skills. If you choose to use effectiveness of health providers as an indicator,
select two of these three parts to include.
For projects with large numbers of health workers (over 25 workers), not all health workers
must be evaluated. Use the sample size guidelines in section 2.1.2.
48
Funded Projects
Part 1: Self-perception
A questionnaire, such shown in Table 5.11, can be used for community health providers to rate
their own sense of knowledge, capacity, and enthusiasm (Kim et al., 2009). Table 5.12 is merely an
example, and is specific to community health promoters; the questionnaire should be modified for use
with clinicians and health providers, who may be asked to report on their own perceived competency in
particular skills (e.g. screening for cervical cancer or giving vaccinations). Further explanation of the
items in the example questionnaire can be found in Woodard (2004). For reporting to GMCR, report the
average score of health workers on each item of the questionnaire that you use for assessing selfperception.
49
Funded Projects
Table 5.12. Example questionnaire for evaluating the self-perception of health workers (Woodard,
2004)
Knowledge
I have a holistic understanding of health and its determinants.
I understand the fundamental principles of (HIV prevention, prenatal
care, nutrition, etc.).
I am familiar with a variety of strategies for health promotion.
I am familiar with the conditions and cultures of the populations with
whom I work.
Skills
I am able to effectively plan, implement, and evaluate health promotion.
I communicate effectively with diverse audiences, using a variety of
means.
I work well with others, in a range of roles and contexts.
I systematically gather and use evidence to guide my practice.
I am able to build the capacity of communities and organizations with
whom I work.
I am strategic and selective in my practice.
Commitment
I have energy and persistence in my work.
I value empowerment, participation, and respect.
I learn from my experiences, and from those of others.
I am confident in my abilities.
I feel that my patients and my community respect my skills and
knowledge.
Resources
I have adequate time to engage in health promotion practice.
I have the infrastructure and tools that I need to practice health
promotion.
I have supportive managers, colleagues, and allies with whom to work.
I can access adequate financial resources for my health promotion
practice.
Strongly
agree
Agree
Disagree
Strongly
disagree
(circle one)
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
1
2
2
3
3
4
4
1
2
3
4
1
2
3
4
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
50
Funded Projects
Part 2: Competency
Measuring competency can be relevant both for health promoters and for clinical health
workers. The example described here is specific to assessing the skills of clinical health workers, but
could also be adapted to use with health promoters.
The first step in competency assessment is to establish the most important skills and
competencies in each area of care that is to be assessed. This list may encompass all of the areas in
which the health worker is expected to provide care, or only the areas in which your organization has
provided training and hopes to see improvement.
Table 5.13. Example competency standards
Area of care
Gastrointestinal
illnesses
Skills
Correctly applies diagnostic criteria for gastrointestinal illnesses
Screens for dehydration
Can administer rehydration treatment
Immunization
Knowledge of the proper age, site, and precautions and
contraindications
Prepares and administers immunizations correctly
Wound care
Practices aseptic technique
Screens for fever or infection
Properly applies or changes wound dressings
Next, a method of assessment should be selected. Two common methods are: (1) a job sample,
whereby the assessor observes the health worker treating an actual patient and evaluates the health
worker’s performance, and (2) a job simulation, in which the assessor observes and evaluates the health
worker’s performance treating a standardized client (a healthy person trained to provide a presentation
of an actual patient case) or performing clinical tasks at a set of rotations (Kak et al. 2001). Both of these
methods have their advantages and drawbacks. The job simulation method allows standardization of
assessment and can be performed at any time without waiting for a patient with a particular complaint.
This may be important for assessing skills that may be used rarely or only in emergency situations.
However, the job simulation method may not accurately replicate clinical conditions, and requires
training individuals to act as standardized patients or setting up situations in which to observe health
workers using their skills. It also requires health workers to take time away from seeing actual patients.
The job sample method has the advantages of allowing the assessor to observe the health worker in a
real patient situation and requiring time investment in set-up and assessment, but also requires waiting
for a clinical situation that accommodates testing a particular skill. Also, the difficulty of patient cases
varies, and a single observation may not be sufficient to accurately assess competency (Kak et al. 2001).
51
Funded Projects
Choose the method that is most appropriate to the area of care being evaluated, most culturally
appropriate for the area, and most feasible given the resources of your organization. Different areas of
care may be assessed using different methods.
Lastly, the assessment tool must be designed so as to allow conversion to a numerical score. The
clearest and most objective way of doing this is to design the assessment as a checklist of behaviors that
the assessor will determine to be at or below standard (Table 5.14). These should be closely linked to
the competency standards that your organization has established, but the listed skills should be more
specific such that, collectively, they describe the competent performance of that area of care in enough
detail that two different supervisors observing the same health worker would come to the same
conclusion about the health worker’s level of competency. The rating score is the percentage of skills
performed correctly (at standard).
Table 5.14. Example checklist for competency assessment
Area of care
Gastrointestinal
illnesses
Skills
1.
Asks patient or guardian what symptoms are present
2.
Takes patient’s temperature to determine if fever is
present
Takes patient’s vital signs
3.
4.
5.
6.
Score
Vaccine
administration
√
√
Checks for signs of dehydration: decreased frequency of
urination, concreted urine, dry mouth or tongue,
sunken eyes, skin does not return to normal when
pinched
Correctly determines if rehydration therapy is necessary
√
Prescribes antibiotics or de-worming medication if
appropriate
√
5 / 6
1. Knows correct storage and handling of vaccine serums
√
5
√
1
√
2.
Prepares vaccine correctly (including reconstitution)
3.
Double-checks: correct client, correct drug, correct dose
√
4.
√
6.
Demonstrates the age-appropriate injection sites and
proper client positioning used for immunization
Chooses the correct needle length and gauge for the
age and size of the client
Prepares the injection site correctly
7.
Uses appropriate technique for immunization
√
8.
Disposes of needle correctly
√
9.
Documents immunizations correctly
√
5.
Score
At standard Below standard
(check one)
√
6 / 9
√
√
6
3
52
Funded Projects
Competency assessment may be performed by peers of the health worker, supervisors, trainers,
or outside observers (such as U.S.-based staff). In general, the assessor should have some expertise in
the area of care to be assessed and have many opportunities to observe the health worker. The
advantage of using peers or supervisors is that they have more opportunities to observe performance,
but they may also have some bias if they have a personal relationship with the individual being assessed.
As with choosing a method of assessment, choose the type of assessor most appropriate for the
situation and the resources of your organization.
For reporting to GMCR, include a description of the assessment method used, a table of
competency standards, and the mean health worker score by area of care.
Part 3: Use of skills
A third measure of health worker effectiveness, and a good supplement to self-assessment and
competency assessment, is how frequently an individual health worker continues to use the skills that
he or she learned through training provided by your organization. This may be assessed either through
self-report by health workers or through observation of health workers by supervisors. Identify the list
of skills or activities that your training program promotes (for example, eye exams or cervical cancer
screenings) and report the number of such activities performed per health worker per month.
53
GMCR Monitoring and Evaluation 6 Guidelines for reporting
Funded Projects
6. Guidelines for Reporting
54
Funded Projects
Nonprofit partners should provide two reports to Green Mountain Coffee Roasters each year: a
mid-year “monitoring” report and a year-end “evaluation” report. The mid-year monitoring report
should be relatively brief and contain an outline of project objectives, notable activities completed
during the first six months of the year, and numerical figures for each of the quantitative output
indicators (sections II, IV, and V, described below).
Year-end reports should be more comprehensive and include the sections described below.
Beginning in early 2012, report submission will be in online form on the GMCR website.
I.
Executive summary
In a one-page summary, describe the project objectives and summarize the activities and impacts of
the past year. Describe any unusual events that affected the project.
II.
Introduction
Provide a summary of the grant that you received from GMCR, including amount, duration,
objectives, and geographic area. Be very clear in describing the project objectives; the objectives should
describe what impacts your organization expects the project to have, not just the activities that you plan
to carry out.
III.
Background and context
What is the socio-political and cultural environment in which the project operates? What
environmental factors affect the project? How is the project affected by economic or security conditions
within the country?
IV.
Methods
Describe the methods that were used for collecting information for monitoring and evaluation. How
many people were interviewed? How many focus groups were held? Who conducted monitoring and
evaluation activities?
V.
Activities and outputs
Summarize, in narrative form, the activities that the project carried out in the past year. Also include
results for the output indicators listed in section 3.1. These should be presented in table form (see Table
3.2 for an example)with accompanying text describing how your project defined direct and indirect
participants and what items you included in the direct investment indicators. .
If your project did not meet your projected targets for the year, describe the challenges that
prevented you from meeting your targets. Also describe activities or outputs that were not captured by
the required indicators.
VI.
Quantitative impacts
Explain what impact indicators you used to evaluate each of your project objectives. Present the
current status of these indicators, the change from the past year, and the change since baseline in one
55
Funded Projects
or more tables.. Also explain how your project measured these indicators. For example, if using
Effectiveness of Health Workers as an indicator, it would be appropriate to include the self-evaluation
questionnaire and the competency standards that were used to measure effectiveness. As with output
indicators, include a description of any challenges that prevented you from meeting your targets or from
improving since the previous year, as well as any impacts of the project not reflected in the indicators.
VII.
Most significant changes in the past year
Please include the three Most Significant Change stories from the past year. Why were these stories
selected? Use the format shown in Appendix 1. Also briefly outline the process that was used to select
the stories.
In addition to the Most Significant Change stories, please include any other project achievements
that cannot be captured by quantitative indicators. For example, has advocacy by your project staff led
to a more reliable medicine supply in local clinics, or more agricultural extension assistance in the
program area?
VIII.
Lessons learned and future directions
Describe what was learned from the evaluation. How did results from qualitative evaluation
reinforce or contradict the quantitative results? How will the results from this evaluation affect future
activities planned by your organization? Overall, did the project meet its goals? What was effective, and
what will change?
IX.
Financial report
Include the budget you provided in your original proposal and provide details of your expenditures
to date with a budget narrative. In some cases, this financial report will have more detailed line items
than what was available at grant application, but it should not be less detailed than your original budget.
Provide a brief explanation for any over or under-spending when the variance is 15% or greater.
X.
Photos
GMCR welcomes the inclusion of photos with your M&E report. Please keep in mind that your grant
contract gives us your permission to use photographs, logos, published/printed information, and any
other materials you supply, without further notice, in press releases and/or publications. Please do not
furnish any photos of which your organization does not have ownership or would not like GMCR to use
in press releases and/or publications.
56
GMCR Monitoring and Evaluation 7 References and further reading
Funded Projects
7. References and further reading
57
Funded Projects
Arimond, M., & Ruel, M. T. (2004). Dietary diversity is associated with child nutritional status: Evidence
from 11 demographic and health surveys. Journal of Nutrition, 134 (10), 2579-2585.
Baker, J. L. (2000). Evaluating the impact of development projects on poverty: A handbook for
practitioners.
Washington
D.C.:
The
World
Bank.
Available
from
http://siteresources.worldbank.org/INTISPMA/Resources/handbook.pdf.
Biddlecom, A., Gregory, R., Lloyd, C. B., & Mensch, B. S. (2008). Associations Between Premarital Sex and
Leaving School in Four Sub-Saharan African Countries. Studies in Family Planning, 39 (4), 337-350.
Bilinsky, P., & Swindale, A. (2007). Months of adequate household food provisioning (MAHFP) for
measurement of household food access: Indicator guide. Washington D.C.: Food and Nutrition
Technical Assistance Project, Academy for Educational Development. Available from
www.fantaproject.org/downloads/pdfs/MAHFP_Jun07.pdf.
Bunch, R. (2000). Two ears of corn: A guide to people-centered agricultural improvement (5th ed.).
Oklahoma City, OK: World Neighbors.
Butler, L. M., Dephelps, C., & Howell, R. E. (1995). Focus groups: A tool for understanding community
perceptions and experiences. Pullman, Washington: Washington State University.
Case, A., & Deaton, A. (1999). School Inputs and Educational Outcomes in South Africa. The Quarterly
Journal of Economics, 114 (3), 1047-1084.
Coates, J., Frongillo, E. A., Rogers, B. L., Webb, P., Wilde, P. E., & Houser, R. (2006). Commonalities in the
experience of household food insecurity across cultures: What are measures missing? Journal of
Nutrition, 136(5), 1438-1448.
Cogill, B. (2003). Anthropometric measurement indicators guide. Washington D.C.: Food and Nutrition
Technical Assistance Project, Academy for Educational Development. Available from
www.fantaproject.org/publications/anthropom.shtml.
Drescher, L.S., Thiele, S., Mensink, G.B.M., 2007. A new index to measure healthy food diversity better
reflects a healthy diet than traditional measures. Journal of Nutrition, 137, 647-651.
Davies, R. and Dart, J. (2005). The ‘Most Significant Change’ (MSC) Technique: A guide to its use.
Available from www.mande.co.uk/docs/MSCGuide.pdf.
Dunn, E. (1999). Microfinance clients in Lima, Peru: Baseline report for AIMS core impact assessment.
Washington,
D.C.:
Management
Systems
International.
Available
from
pdf.usaid.gov/pdf_docs/Pnacg045.pdf.
Dytham, C. (2003). Choosing and using statistics: A biologist's guide. Malden, MA: Blackwell Publishing.
58
Funded Projects
Estrella, M. and Gaventa, J. (1998). Who counts reality? Participatory monitoring and evaluation: A
literature review, Working Paper 70. Brighton, UK: Institute of Development Studies.
Flick, U. (Ed.). (2007). Doing Interviews. Thousand Oaks, CA: Sage Publications.
Floyd J. Fowler, J. (1995). Improving Survey Questions. Thousand Oaks, CA: Sage Publications.
Fretheim, A., Oxman, A. D., Lavis, J. N., & Lewin, S. (2009). SUPPORT Tools for evidence-informed
policymaking in health 18: Planning monitoring and evaluation of policies. Health Research Policy
and Systems, 7 (1).
Green Mountain Coffee Roasters (2008). Brewing a better world voices: Green Mountain Coffee
Roasters, Inc. corporate social responsibility report. Waterbury, VT.
Gotelli, N. J., & Ellison, A. M. (2004). A primer of ecological statistics. Sunderland, MA: Sinaur Associates.
Grant, M. J., & Behrman, J. R. (2010). Gender gaps in educational attainment in less developed
countries. Population and Development Review, 36 (1), 71-89.
Greaney, V. & Kellaghan, T. (1996). Monitoring the learning outcomes of education systems. Directions in
development. Washington, D.C.: The World Bank.
Hyman, E. L., & Dearden, K. (1998). Comprehensive impact assessment systems for NGO
microenterprise development programs. World Development, 26 (2), 261-276.
Kak, N., Burkhalter, B., & Cooper, M. (2001). Measuring the competence of healthcare providers.
Bethesda, MD: U.S. Agency for International Development (USAID).
Kim, J. M., Koh, K. W., Oak, C. H., Jung, W. H., Kim, S. H., & Park, D. H. (2009). Assessment of village
health worker training program in Tuguegarao, Philippine. Journal of Preventative Medicine and
Public Health, 42 (6), 377-385.
Kirsch, I. (2001). The International Adult Literacy Survey (IALS): Understand what was measured.
Princeton, NJ: Education Testing Service, Statistics and Research Division. Available from:
http://devdata.worldbank.org/phrd/imp_data/04_FRAMEWORK_ALL_Prose%20and%20Document
%20Framework.pdf.
Krueger, R. A. (1994). Focus groups: A practical guide for applied research (2nd ed.). Thousand Oaks, CA:
Sage Publications, Inc.
Leech, B. L. (2002). Asking questions: Techniques for semistructured interviews. Political Science &
Politics, 35 (4), 665-668.
59
Funded Projects
Mathie, A., & Foster, M. Participatory monitoring and evaluation: A manual for village organizers.
Antigonish, Canada: Coady International Institute, St. Francis Xavier University. Available from
www.coady.stfx.ca/tinroom/assets/file/resources/abcd/SEWA%20PME%20Manual.pdf.
Maxwell, D., Watkins, B., Wheeler, R., & Collins, G. (2003). The coping strategies index: A tool for rapidly
measuring food security and the impact of food aid programs in emergencies. Paper presented at
the FAO international workshop on "Food security in complex emergencies: building policy
frameworks to address longer-term programming challenges". Tivoli, Italy: Food and Agriculture
Organization
of
the
United
Nations.
Available
from
home.wfp.org/stellent/groups/public/documents/manual_guide_proced/wfp211058.pdf.
Maxwell, D., Caldwell, R., & Langworthy, M. (2008). Measuring food insecurity: Can an indicator based
on localized coping behaviors be used to compare across contexts? Food Policy, 33 (6), 533-540.
Oxfam.
Oxfam
GB
Evaluation
www.oxfam.org.uk/resources/evaluations/index.html
Guidelines.
Available
from
Rabiee, F. (2004). Focus-group interview and data analysis. Proceedings of the Nutrition Society, 63 (04),
655-660.
Rahman, S. M., Ali, N. A., Jennings, L., Seraji, M. H. R., Mannan, I., Shah, R., et al. (2010). Factors affecting
recruitment and retention of community health workers in a newborn care intervention in
Bangladesh. Human Resources for Health, 8 (12).
Rodríguez, C., Sánchez, F., & Armenta, A. (2010). Do interventions at school level improve educational
outcomes? Evidence from a rural program in Colombia. World Development, 38 (3), 415-428.
Schaffner, J. (2005). Measuring literacy in developing country household surveys: issues and evidence:
Paper commissioned for the Education for All Global Monitoring Report 2006, Literacy for Life of
the United Nations Educational, Scientific, and Cultural Organization. Available from
http://unesdoc.unesco.org/images/0014/001462/146285e.pdf.
Shah, M. K., Kambou, S. D., Goparaju, L., Adams, M. K., & Matarazzo, J. M. (2004). Participatory
monitoring and evaluation of community- and faith-based programs: A step-by-step guide for
people who want to make HIV and AIDS services and activities more effective in their community.
Washington,
D.C.:
CORE
Initiative,
USAID.
Available
from
www.coreinitiative.org/Resources/Publications/PME_2nd/index.php.
Stewart, D. W., Shamdasani, P. N., & Hook, D. W. (2007). Focus Groups: Theory and Practice. Thousand
Oaks, CA: Sage Publications.
60
Funded Projects
Svedburg, P. (2000). Poverty and undernutrition: Theory, measurement, and policy. New York: Oxford
Press.
Swindale, A., & Bilinsky, P. (2006). Household dietary diversity score for measurement of household food
access: Indicator guide (v. 2). Washington D.C.: Food and Nutrition Technical Assistance Project,
Academy
for
Educational
Development.
Available
from
www.fantaproject.org/publications/hdds_mahfp.shtml.
Terryn, B. Measuring Literacy in Developing Countries from an International Perspective. Montreal:
UNESCO
Institute
for
Statistics.
Available
from
http://www.stat.auckland.ac.nz/~iase/publications/3/TerrynI68.pdf.
Woodard, G. B. (2004). Health promotion capacity checklists: A workbook for individual, organizational,
and environmental assessment. Saskatoon, Canada: Prairie Region Health Promotion Research
Center,
University
of
Saskatchewan.
Available
from
www.prhprc.usask.ca/publications/finalworkbook.pdf.
World Health Organization (2006). The WHO
www.who.int/childgrowth/standards/en/.
child
growth
standards.
Available
from
61
GMCR Monitoring and Evaluation
Funded Projects
Appendices
Appendix 1: Sample Most Significant Change
story collection format
Below is an example of a Most Significant Change story from an organization called Osi Tanata. Adapted
from Davies & Dart (2005).
Appendix 2:
Osi Tanata Most Significant Change Story Collection Sheet
Significant Change Story
Contact Details
Do you the storyteller:
• want to have your name on the story? (check one)
Yes No 
• consent to us using your story for publication? (check one)
Yes No 
Name of person recording story: Wilson Kabui
Name of storyteller*: Sebastin Kakau
*Leave blank if the storyteller wishes to remain anonymous
Project: Organic project – cycle 3
Role of storyteller as a participant: Male gardening traineeLocation: Piva, Papua New
Guinea
Date of recording: 23th of March, 2010
When did it happen? Over the last year
Title of story: “Growing big”
Why was this story selected? The story teller transferred learned skills to his
community.
Tell me how you (the storyteller) first became involved with Osi Tanata, and what
your current involvement is:
I used to be a member of a community project. But I Ieft the community project in
anticipation of disputes that might occur within the community project.
However, upon hearing that Osi Tanata was giving training to grass roots, I
attended some of the Osi Tanata training of project management and book
keeping and TOT.
62
Funded Projects
Appendices
From your point of view, describe the most significant change that has resulted from
your involvement with Osi Tanata (training/support or funding).
After the training I went back to my village and mobilized my family members to
venture into organic gardening. I decided to set up my own family project on
organic gardening. Despite not having funding from any agency I ventured into
setting up this small project with only the knowledge that I got from Osi Tanata.
We set up our organic garden growing cabbages, capsicums, greens, tomatoes,
aibika, chillies and other things. Currently I am thankful for what I learned from
Osi Tanata, and am using it. Today my project is progressing well. We have sold
many of their produces from their organic farm. For example, for a bed of
cabbage, he is getting around K100. Now they have spent the money to buy
clothes and many other basic needs. Apart from generating income the families
and the surrounding villages have enough surplus to feed their family and others.
Also some of the money is being used to start other projects such as a trade
store.
Why is this significant to you?
It is significant to me because at first I had no knowledge to run a project.
Today I have a good project running and the income from this project is being
used to sustain the livelihood of my family.
63
Funded Projects
Appendices
Appendix 2: Example facilitation guide for MSC
story selection
Adapted from Davies & Dart (2005).
The facilitator writes all the titles of the stories on a whiteboard, leaving a space next to each story for
comments.
Title
Comments
My life is getting better
Strong, written by a project participant, but incomplete,
story not finished
Feeling empowered
Moving story, beginning middle and end. Attribution to
project is questionable. Great story, not sure if it is about
the project.
Better decisions for the Good solid story. Heard many times before. Small change
family
yet crucial. Not sure about the dates mentioned.
Now I understand
Okay, not enough information to really understand what is
going on.
1. The facilitator invites people from the selection group to read all of the stories out loud. After each
story, the facilitator asks:
 What is this story really about?
 What is our opinion of the story?
2. The facilitator writes any comments next to the title on the white board, as above.
3. When all the stories have been read out loud, the facilitator asks people to vote for the story that
they find most significant. Voting can be done by a show of hands.
4. When the votes have been cast, if there is not a consensus, the facilitator encourages participants to
discuss why they chose the story that they chose. Ask questions such as:
 Why did you choose this story above all the other stories?
 But some of you chose a different story—can you explain why you didn’t choose this story?
 What do you think of the stories in general?
5. Next to each story, the facilitator makes notes of the reasons why they were and were not selected.
6. Once everyone has heard why certain stories were voted for above others, the facilitator may call a
second vote. This time there may be consensus. However, the stories need not be chosen by
consensus. A majority vote may be used, as long as the reasons for choosing the stories are
documented.
64
Funded Projects
Appendices
Appendix 3: Example evaluation report to GMCR
This is a mock report adapted from actual reports submitted to GMCR in the past year, and includes
some real data and some fabricated data presented here for illustrative purposes.
I.
Executive Summary
II.
Introduction
In 2010, Health Care NGO received a one-year grant from GMCR in the amount of $100,000 to
continue health promotion, health care, and poverty alleviation activities in Chiapas, Mexico. The
objectives of this project are to improve access to health services for the 20 communities in southern
Chiapas where Health Care NGO works…
III.
Background and context
The residents of the southern Mexican state of Chiapas, including millions of indigenous Maya,
have long struggled with poverty, political violence, and dismal health conditions. Chiapas is burdened
with extremely high rates of maternal mortality, infant mortality, and tuberculosis when compared to
other states in Mexico. Our work aims to provide a more reliable, community-based alternative by
training and employing local community health promoters, called promotores….
Health Care NGO was established in 1985 by a small group of Mexican health promoters. They
initially worked with Guatemalan refugee communities in the Chiapas border region, and later expanded
their work to other marginalized people in Chiapas. Health Care NGO believes that "a life of dignity" is a
human right. This includes a strong public health system that responds to the most pressing health
needs of the population, and access to high quality health care…
Since 1989, Partner NGO has collaborated with Health Care NGO to improve medical
infrastructure in the region and to recruit and train hundreds of promotores. Over the past two decades,
Health Care NGO has partnered with dozens of indigenous and rural communities throughout Chiapas to
develop local health capacity. Recent work has focused on a network of communities in the area of
Huitiupan in the highlands and in the area of Amatan. Health Care NGO is dedicated to helping
communities build self-sufficiency and counts many successful community health groups throughout
Chiapas among its "alumni”…
With support from Green Mountain Coffee Roasters, Health Care NGO is deepening a longstanding partnership with a network of promotores living in more than 20 isolated farming communities
in the Sierra Madre Mountains of southern Chiapas. Faced with treacherous roads and a lack of
communications infrastructure, local families are frequently unable to afford transportation to
government or private health care facilities, let alone the cost of the consults and medicine. With
physician support, Community Health Promoters provide unprecedented access to treatment as well as
65
Funded Projects
Appendices
prevention information in their own communities at almost no cost. They are also able to provide crucial
follow-up for diseases such as diabetes, epilepsy, and tuberculosis.
IV.
Methods
In addition to output indicators, we assessed the impacts of our activities using indicators
5.10.1, 5.10.2 and 5.10.4 from GMCR’s Monitoring and Evaluation Guide. Monitoring and evaluation
activities were carried out by physicians and health promoter training staff in the communities of
Amatan, Huituipan, and Siltipec. Treatment and resolution of acute conditions (indicator 5.10.1) and
detection, treatment, and loss to follow-up of chronic conditions (indicator 5.10.2) were tracked by
physicians as part of their record-keeping activities. Effectiveness of health workers (indicator 5.10.4)
was evaluated by one health promoter trainer in each community (three trainers total). As Health Care
NGO has a large network of active health promoters, these three trainers interviewed a sample of 20-21
health promoters in each community for a total of 62 promoters. Most Significant Change stories were
collected during 6 of these interviews and from interviews with 4 patients treated by Health Care NGO
staff or volunteers. Selection of the three most significant stories was conducted by a panel of 6
physicians and trainers representing all three communities.
V.
Activities and outputs
1. Participants
Health Care NGO defines direct participants as individuals who have either participated in
training events or received health care (at clinics or during field visits). In the past year, Health Care
NGO trained 76 community health promoters, had 154 women participate in women’s groups, and
saw 212 patients for health issues, for a total of 442 direct participants.
2. Capacity building activities
Health Care NGO trained 76 health promoters in the past year: 16 in Amatan, 40 in Huitiupan,
and 20 in Siltepec. Health promoter trainings are 8-hour workshops that cover diagnosis and
treatment of common illnesses and injuries. Health Care NGO also held 26 women’s group meetings
in the Siltepec area. Women’s groups are small groups of women that meet to discuss health and
family issues.
Type of training
Health promoter training
Women’s group meeting
Number of people
per training
7-20
8-10
Length (hours) of
each training
8
1
Number of
trainings
7
26
3. Direct Investments
The majority of Health Care NGO’s direct investments are in the form of health and medical
supplies to community health promoters. These are disbursed in the form of basic medical kits.
Health Care NGO also provides medication for chronic illnesses without charge to patients. In the
66
Funded Projects
Appendices
past year, Partner NGO also purchased blood sugar monitors and blood pressure cuffs for all Health
Care NGO health promoters.
VI.
Description
Number of units
Total value
Basic medical kits
Medication for asthma, diabetes,
epilepsy, and hypertension
Blood sugar monitors
Blood pressure cuffs
76
42 monthly prescriptions
$5,400
$65,520
200
200
$12,600
$10,820
Quantitative impacts
1. Detection and resolution of acute conditions
Health Care NGO physicians and community health promoters saw 110 patients seeking
treatment for acute health conditions over the past year. Of these patients, all 108 received initial
treatment. Two did not return for treatment. Of the 108 that received treatment, 97 had their
illness resolved. Five patients did not return for follow-up treatment and Health Care NGO was
unable to determine if their illness was resolved. Six patients seeking treatment for acute illness
were diagnosed with a chronic illness and therefore their illness was not immediately resolved.
2. Detection, treatment, and loss to follow-up
Health Care NGO conducts screenings for asthma, diabetes, and hypertension. Patients may also
be diagnosed with these or other chronic illnesses when they receive visits from a health worker for
other health complaints.
Health issue
Asthma
Hypertension
Diabetes
Epilepsy
Received screening
125
525
103
(no regular
screening)
Identified as needing
treatment
42
38
12
10
Received treatment
35
23
7
10
Lost to follow-up
7
15
5
0
Overall, 64% of patients identified as needing treatment for a chronic health issue received the
needed treatment. This is an improvement over 2009 when 57% of identified patients received
67
Funded Projects
Appendices
treatment. A baseline study conducted prior to Health Care NGO’s work in Chiapas indicated that
only 5% of patients with likely chronic health issues received treatment, thus 64% is a substantial
improvement over baseline.
3. Effectiveness of health workers
Health Care NGO used parts 1 and 3 of indicator 5.10.4 to measure the effectiveness of
community health promoters. Part 2 (competency assessment) was not feasible because health
promoters work independently and are generally not under the supervision of physicians.
The questionnaire that Health Care NGO used to measure self-perception is shown below. NGO
trainers surveyed 62 community health promoters who were selected randomly from the pool of
health promoters in the three communities where the project works.
Question
1
2
3
4
5
6
7
8
9
Knowledge
I have a holistic
understanding of health and
its determinants.
I understand the
fundamental principles of
prevention and treatment.
I am familiar with a variety of
strategies for health
promotion.
I am familiar with the
conditions and cultures of
the populations with whom I
work.
Skills
I am able to effectively plan,
implement, and evaluate
health promotion.
I communicate effectively
with diverse audiences, using
a variety of means.
I work well with others, in a
range of roles and contexts.
I systematically gather and
use evidence to guide my
practice.
I am able to build the
capacity of communities and
organizations with whom I
work.
Strongly
Disagree
(1)
Disagree
(2)
Agree (3)
Strongly
Agree (4)
No
Answer
No.
No.
No.
No.
No.
%
%
%
%
3
4
2
3
16
20
6
8
2
3
6
8
12
16
6
8
2
3
7
9
14
19
4
5
2
2
2
2
9
11
12
15
2
3
7
9
12
16
6
8
3
4
2
3
13
19
6
9
4
5
7
9
6
8
10
13
1
1
6
8
10
14
7
10
2
3
8
13
10
17
3
5
Average
%
0
2.9
1
2.8
0
2.7
2
3.2
0
2.8
4
2.9
0
2.8
3
4
3.0
4
7
2.6
1
2
3
68
Funded Projects
Appendices
I am strategic and selective
10 in my practice.
11
12
13
1
Commitment
I have energy and
persistence in my work.
I value empowerment,
participation, and respect.
I learn from my experiences,
and from those of others.
16
17
18
19
3
13
13
13
13
2
2
3.3
0
4
5
13
16
9
11
1
1
3.2
0
2
2
9
10
14
16
2
2
3.5
0
10
10
17
17
0
3.6
0
3
4
11
15
9
12
4
5
3.3
0
1
1
13
17
9
12
4
5
3.3
1
1
5
6
12
15
8
10
1
1
3.0
3
4
7
10
12
17
4
6
1
1
2.7
2
2
3
4
11
14
10
12
1
1
3.1
2
3
7
9
10
13
8
10
0
2.9
I feel that my patients and
my community respect my
skills and knowledge.
Resources
I have adequate time to
engage in health promotion
practice.
I have the infrastructure and
tools that I need to practice
health promotion.
I have supportive managers,
colleagues, and allies with
whom to work.
I can access adequate
financial resources for my
health promotion practice.
3
0
14 I am confident in my abilities.
15
1
On average, Health Care NGO health promoters agreed with most statements on the self-perception
questionnaire tool. The area in which health promoters felt the strongest was their capacity to learn
from their own experiences and those of others, which indicates that health promoters are likely to
become more effective in their work over time. Health promoters felt weakest in their ability to build
capacity in their communities and in their access to tools and infrastructure for their health promotion
practice indicating that these are areas where Health Care NGO should focus its efforts so that health
promoters can work more effectively.
Use of skills (part 3 of the health worker effectiveness indicator) was measured by self-report of the
62 health promoters interviewed about their self-perception. Health Care NGO asked these health
promoters how often in the past year they used the six key skills taught in health promotion trainings
(shown below). Any interaction with a patient that involved the skill was counted as an incidence of use.
On average, health promoters reported 11 total patient interactions per year.
69
Funded Projects
Skill
1 Prenatal monitoring and care
Appendices
Incidences of use in the past year
Min
Max
Average
2
7
9
Care for childhood diarrhea (including
2 rehydration therapy)
3
4
5
6
Promotion of hygiene (hand washing,
food preparation)
Asthma diagnosis and/or care
Tuberculosis diagnosis and/or care
Hypertension diagnosis and/or care
1
5
3
7
0
0
0
16
6
8
3
9
4
2
1
Health Care NGO was pleased with the results from this evaluation. Health promoters are using their
skills with sufficient frequency to be effective in their communities and to improve their level of care.
VII.
Most Significant Change stories
Story 1
Project: Health Promoter Training
Role of storyteller: Women’s group participant and wife of health promoter
Location: Matazano
Date of recording: July 13, 2011
When did it happen? Over the past year
Title of story: “Now I say what I want to say”
What changes have you seen in your life from being married to a Health Promoter?
It’s changed, the truth is that we didn’t have an understanding. Now he lets me go out, and
when I return he doesn’t scold me, but on the contrary, we sit down and talk about what happened.
Before I didn’t leave the house, out of fear, my husband didn’t let me go to meetings, and if I did he’d be
angry. Now he supports me so that I don’t miss any trainings.
In the meetings or trainings you hear a lot of great things. I tell other women that we get
together so that we can share with one another. When I go to the meetings, on returning I tell them
about what we talked about, about decision making. It would be really great if both men and women
participated.
Now that [my husband] washes his own dishes and clothes, people say that I order him around
and that he isn’t in charge and is a pushover.
Do people trust in the two of you?
I would say that some people trust us, above all women trust me as another woman. My
husband shares with me what he learns. They already trust him because he calls town meetings, and
people attend. He’s now the Municipal Agent, and has also fulfilled other roles in the community. Here
70
Funded Projects
Appendices
people are ashamed to talk about their problems, most of all a woman to a man. I like learning about
childbirth, and I’ve tried to attend several pregnant women- my mother knows about childbirth- I still
have many doubts and this helps me to learn. I began as a health promotora but I stopped attending
trainings after I had my baby because it was difficult to continue.
You mentioned you were embarrassed? Of what?
Yes, I was embarrassed to even eat in front of the teacher, now I’m no longer timid and I have
talked with many other women about this change.
Do people seek you both out for health care?
People seek me out a great deal because I know about medicinal plants, how to give them to
children. I feel that I see more clearly, I thought that people who know more are better than us, but I’ve
learned that no one is better than anyone else. I changed my way of participating, before I was afraid to
speak, and now I’m not, I say what I want to say.
How do you feel about these changes?
I feel happy with these changes, for example, before when I would leave, when I came back I’d
get scolded, but not anymore. Now my husband receives me happily and we talk about what happened
when I was out, and this makes me feel happy. We thank you because if you hadn’t come we would
have continued on like before. I don’t like the machismo in the community, for example there is a
woman who wants to participate and her husband hits her if she does, he tells her that she mustn’t go
and that the workshops are worthless.
Story 2
Project: Human Rights Training
Role of storyteller: Community member and human rights promoter; husband of community health
promotora
Location: Town of Capitan Luis A. Vidal
Date of recording: May 4, 2011
When did it happen? Over the past year
Title of story: “I now believe I have the capacity”
What changes have you seen from your training?
A very interesting change, because the communities accepted my participation, and they
created groups to defend our human rights. When I began to speak out, there were confrontations
when I tried to have a dialogue. This process has helped me have greater confidence in myself, and in
my human rights knowledge. Now I can’t be easily manipulated because I have this knowledge.
What changes have you seen in your family?
71
Funded Projects
Appendices
I’ve come to understand the importance of education, within my family there is greater trust,
greater security, acceptance, with what we have, with what we do without, we help one another. When
she [my wife] or I go out to a meeting, the other stays to take care of the family.
Changes… greater understanding, better management of money. To acknowledge our reality,
it’s been a constant struggle to obtain our necessities. Based on my experience the reality we are living
is bad, above all in relation to human rights. That is why I’ve shared the knowledge and experiences that
I’ve learned to try to find a way to change our lives. I now believe I have the capacity to overcome.
What changes have you seen in yourself?
Not with everybody, but with many people I’ve gained their trust, we talk, we relate, are friends,
they ask me for help. My work is mostly with children, and the people know my work and recognize me
as a decent person.
What changes have you seen in your wife since she has become a health promoter?
She speaks up more, she does things more calmly, and she understands my work and my trips
[to trainings]. Before there were a lot of arguments, and now we support one another. Maybe she still
needs to say her piece more, and participate more.
Having allied ourselves with other organizations like Health Care NGO, your involvement has
been important in these changes. We need to ally ourselves with others more in order to have greater
strength.
Story 3
Project: Health Promoter Training
Role of storyteller: Health Promotora
Location: Town of Lagunita
Date of recording: Sept. 25, 2011
When did it happen? Over the past six years
Title of story: “Being a health promoter changed my life”
What changes have you seen since you began as a Health Promoter?
In the first place the health of our family has changed. In the workshops you’ve showed us how
to care for children, like when they get sick we give them medicine or ORS when it’s necessary. I’ve
learned to value my life, before I didn’t talk, I was afraid and timid, but now I participate, I am
motivated. Since my husband has been participating, a change is that now he washes his dishes after he
eats. I tell my family what is nutritious to eat and that’s what we practice. The community supports me,
they tell me to keep going, after each workshop I tell them what I learned. People seek me out, they say
I am a Health Promoter and that I’m learning. I’ve helped many people that seek me out, for example
one woman that came with a cough that brought up blood, I told her to get studies done because it
could be tuberculosis, based on what I found out about, but I wasn’t able to find out what happened
72
Funded Projects
Appendices
because she returned to Guatemala. I also helped a pregnant woman and because of that many people
trust in me. People seek me out more than before. People come to tell me about their problems and I
listen to them. They come from many towns, La Lucha, Piedra Parada, because of their illnesses. I’m
managing the care for a girl with epilepsy, and she’s now controlled.
When people visit me I ask them what they are feeling, I offer them water to drink, I ask them to
tell me what their problem is. First I listen and then I give advice. Some people that I care for come back
and say thanks, and others I’m not sure what happens to them. One advantage is that my husband and I
both are Health Promoters, so when a woman comes I will care for her, and when a man comes he will
care for him.
How do you feel as a Health Promoter?
I feel happy, I study my books, I feel happy because I can share with other people, I feel happy to
be able to help people that need it, I feel happy being a Health Promoter. Being a Health Promoter
changed my life, even though it can be tiring to go to the workshops. In our community meetings they
sometimes humiliate women that try to talk, and I tell the men they must respect us, I am motivated to
tell them so, and it’s not a common thing to do.
Anything else you’d like to say?
Thanks for the workshops and thanks to you all for getting us beyond where we were, for
sharing with us. I plan to continue.
VIII.
Lessons learned and future directions
Health Care NGO continues to improve the availability of care for communities in Chiapas.
Health promoters reported that they felt effective in their work and used their skills with sufficient
frequency to make an impact in their communities. An important impact of community health
promotion that was not captured in the quantitative impact indicators but came out strongly in MSC
stories was the impact of health promotion training in the lives of the promoters. Health promoters
frequently expressed greater satisfaction with their lives and an improved sense of self-worth.
Particularly in the case of female health promoters, health promotion activities often were seen as
improving family dynamics in the household of health promoters.
Areas in which Health Care NGO would like to improve its work are in building greater health
knowledge in the communities where health promoters work and improving the effectiveness of care
provided by health promoters. Health promoters felt least effective in their ability to build capacity
around health care and health knowledge in their communities. While this is not necessarily a primary
aim of Health Care NGO’s work, it is important to improving overall health in Chiapas. Health Care NGO
would also like to improve patient follow-up by health promoters, and we plan to implement a new
process for follow up in 2012.
73

GMCR Monitoring and Evaluation Guide for Supply Chain Outreach

Transcription

Similar documents

DeMarker Strategy

Open living room

Professors at Kaldi`s Getting Coffee

BatFink indicator

Overview Brochure

MountainHouse.net

DeMarker - ActFX Algorithmic Trading

Coffee Roasters Know Their Beans

Coopro Sanvito R.L.

brochure