Contents - Office for National Statistics

Transcription

No. 51
January 2003
Contents
Survey interviewer attitudes and demographic profile:
preliminary results from the 2001 ONS Interviewer Attitudes
Survey
Stephanie Freeth, Catherine
Kane & Allison Cowie
1
Proposals for an Intergrated Social Survey
Nikki Bennett
13
The National Identity Question: methodological investigations
Lucy Haselden & Richard J
Jenkins
18
Influencing response on the Family Resources Survey by using
incentives
Mark McConaghy and Roeland
Beerten
27
Quality Issues in Social Surveys (QUISS) seminar 11th April
2002 “Surveys of Children and Young People”
Karen Irving & Neil Park
36
Quality Issues in Social Surveys (QUISS) seminar 31st October
2002 - Quality Issues in Longitudinal Surveys
Trish McOrmond & Judith Bell
47
The National Statistical Methodology Series
58
Forthcoming conferences, seminars and courses
60
Recent Social Survey Division Publications
63
The Survey Methodology Bulletin is produced primarily to inform staff in the Office for
National Statistics (ONS) and the Government Statistical Service (GSS) about the work
on survey methodology carried out by the ONS. It is produced by the ONS, and staff in
the ONS are encouraged to write short articles about methodological projects or issues
of general interest. Articles in the bulletin are not professionally refereed, as this would
considerably increase the time and effort to produce the bulletin: they are working
papers and should be viewed as such.
The bulletin is published twice a year, in January and July, and is free to staff within the
ONS and the GSS. It is made available to others with an interest in survey methodology
at a small cost to cover production and postage.
The Office for National Statistics works in partnership with others in the Government
Statistical Service to provide Parliament, Government and the wider community with
the statistical information, analysis and advice needed to improve decision-making,
stimulate research and inform debate. It also registers key life events. It aims to provide
an authoritative and impartial picture of society and a window on the work and
performance of government, allowing the impact of government policies and actions to
be assessed.
Edited by: Joe Traynor and Roeland Beerten
Prepared by: Sarah Binfoh
Freeth, Kane and Cowie
Survey Interviewer attitudes and demographic profile
Survey interviewer attitudes and demographic profile:
preliminary results from the 2001 ONS Interviewer
Attitudes Survey
Stephanie Freeth, Catherine Kane and Allison Cowie
1. Background
Like all survey organisations in this country the Office for National Statistics (ONS),
which carries out many of the major government household surveys in Britain, has
been experiencing falling response rates in the last five to ten years. This has
prompted research to investigate reasons for non-response and to identify and
evaluate measures to improve response. The work on response issues ONS is carrying
out includes the Response Project which implements and evaluates response
improvement procedures, the Census-linked Study of Survey Non-response which
uses matched census-survey records to detect non-response bias and influences of
survey response (White, Freeth and Martin, 2001) and the Interviewer Attitudes
Survey ( which concentrates on the role of the interviewer in obtaining response. This
paper presents the preliminary results of the 2001 Interviewer Attitudes Survey.
The design of the lAS was built on the research of Groves and Couper (1998) which
in turn was developed from models originally proposed by Groves, Cialdini and
Couper (1992) and elaborated by Campanelli, Sturgis and Purdon (1997). Groves and
Couper have put forward conceptual models of the factors determining the likelihood
of the interviewer making contact with a sampled household and the likelihood of the
household agreeing to co-operate given contact. They list four broad categories of
influence:
• area characteristics
• household characteristics
• survey design features
• interviewer characteristics
Each of these combines with the others to affect both likelihood of contact, the
interaction between the household and the interviewer and hence the likelihood of co
operation given contact. Groves and Couper emphasise that it is not demographic
characteristics per se which determine non-response; rather that people with certain
characteristics are likely to lead lifestyles or hold attitudes which determine how easy
they are to find at home or persuade to take part in a survey. Interviewers have a huge
influence on response outcomes. The attitudes and strategies they bring to their work
and their detailed behaviour at the household have been shown to be major
determinants of response outcome.
ONS has carried out an lAS in 1998 and another one in 2001. The 1998 lAS was part
of an international study co-ordinated by Professor Joop Hox of the University of
Utrecht (Hox and de Leeuw, 2002). The international study aimed to address two
questions:
• Do interviewers in different countries have different attitudes towards the
interviewer role?
• Does interviewer attitude predict interviewer-level response rate internationally?
1
SMB 51 1/03
Participating countries used a standard questionnaire based on questionnaires
developed by Mick Couper from the Survey Research Centre at the University of
Michigan and Pam Campanelli from Survey Methods Centre of the SCPR (now the
National Centre for Social Research) so that results can be compared across countries.
In addition to those in the UK, survey organisations in Belgium, Canada, Finland,
Germany, The Netherlands, Slovenia, Sweden and the USA also took part.
The international study demonstrated that interviewers from different countries have
different views about the job of an interviewer. Interviewers from Germany and
Slovenia scored the highest on attitudes relating to the importance of persuading
reluctant respondents followed by interviewers from the UK, Canada and the USA.
Interviewers’ characteristics, attitudes and behaviour account for only a small
proportion of the variance between countries. However, Hox and de Leeuw pointed
out that variations in fieldwork conditions, survey subject matter and survey
organisations between the participating countries had introduced confounding factors
which made it more difficult to identify interviewer effects in the international study.
Using only the 1998 lAS data for the ONS, Beerten and Martin (2000) demonstrated
that interviewing experience was positively related to co-operation on the Survey for
English Housing, Family Expenditure Survey, Family Resources Survey and the ONS
Omnibus. In addition, they found that interviewers who believed that they could
convince the most reluctant respondents or those who believed that all respondents
could be dealt with in the same way obtained higher co-operation rates on the Survey
of English Housing.
In 2001, ONS repeated the lAS to find out, if interviewers’ attitudes and behaviour
have changed and to provide information to enable the 2001 Census-linked Study of
Survey Non-response to identify and allow for interviewer effects. As with the 1998
lAS, the 2001 data will be used to develop statistical models on interviewers’
influence on survey response. However, the analysis of the 2001 survey has included
more descriptive work on the attitudes and behaviour of interviewers compared with
1998. The work on developing models will begin in the autumn of 2002 but we have
already carried out a significant amount of descriptive analyses. Selected results from
the descriptive analyses are presented in this paper.
2. The Design of the 2001 lAS
The 2001 lAS was based on the design of the 1998 survey. ONS employs around
1,200 interviewers in total. The majority of interviewers (about 800) carry out face to
face interviews on household surveys and the remainder work as telephone
interviewers in the Telephone Unit or at air or seaports on the International
Passengers Survey. The lAS included only the interviewers who carried out face to
face interviews on household surveys. The lAS was a census in that all ONS’ face to
face interviewers and Field Managers working for ONS in June 2002 were included
and sent a questionnaire. Some interviewers were newly recruited and had not yet
started work but we decided to include them as well to see in due course whether
their attitudes are predictive of ability to gain response and to study how attitudes
change over time with experience of interviewing.
The interviewer questionnaire was based on the one developed for the 1998 lAS but
with some minor changes. Questionnaire topics included:
Demo graphic and background characteristics:
Sex
Age
Educational qualifications
2
SMB 51 1/03
Merit or pay band
No. of days per week available for work
Experience and work details:
Length of experience as SSD interviewer
Length of experience as interviewer for other organisations
Whether has other paid employment
Surveys worked on in past year
Amount of work done in past year
Attitude and views about:
ONS’ confidentiality pledge
Persuading reluctant respondents
The categories of people who are difficult to persuade
The categories of people who are difficult to find at home
Topics which are difficult to ask about (eg. income, investments, religion, drinking
smoking, drug use, religion, sexual behaviour, ethnicity)
Feelings after encountering a refusal
The ease of getting response on different surveys
Attitudes to working at different times and days of the week
Behaviour and strategies for:
Dealing with non-contacts
Introducing the survey at the doorstep
Gaining co-operation/persuading reluctant respondents
The questionnaire was delivered to the interviewers as a Computer Assisted SelfInterviewing instrument (developed using the questionnaire program Blaise). The
interviewers were asked to complete and return the questionnaire electronically.
Participation in the survey was voluntary and interviewers who participated in the
survey were paid up to an hour’s time for taking part. The questionnaires were sent
out in the middle of June 2001 and up to two reminders were sent to non-respondents
between the end of June and the beginning of August 2001.
The study requires that answers to the questionnaire are linked eventually to other
information held about the interviewers and so the questionnaires could not be
anonymous. However, interviewers were promised that the information provided
would be seen and used only by the research team and not by members of ONS’ Field
Branch responsible for interviewer management. We also consulted the trade union
representing the interviewers before the survey started.
In total, 669 questionnaires were returned representing a response rate of 84% (Table
1).
Table 1 Response to the 2001 Interviewer Attitudes Survey by field force
3
SMB 51 1/03
3. Preliminary results
3.1. Demographic characteristics
In contrast to the stereotype of interviewer being female, 60% of the interviewers
working for ONS in 2001 were men. The majority of ONS’ interviewers were more
than 50 years old and 44% of them had degree level qualifications (Table 2).
Table 2 Interviewer demographic characteristics by field force, 2001
3.2. Experience and performance grade
Half of the interviewers working in ONS in 2001 had worked as interviewers for two
or fewer years but a quarter of the field force had eight or more years of experience.
A quarter had previously worked for another survey organisation and 10% worked for
another survey organisation in financial year 2000/2001. (In the UK it is very
common for market research interviewers, who may also do social surveys, to work
for multiple companies as levels of demand for their services can be erratic. ONS
expects interviewers to commit to working more hours per week than most of its
competitors so multiple job holding is less common.)
All interviewers start off on the basic grade and progress to become advanced
interviewers once they have completed a probationary period of about a year.
Thereafter they can get further increases in their hourly rate of pay (merit grade
payments) based on their performance ratings. In 2001, 56% of interviewers were in
the basic or advanced interviewer grade and 38% were in the merit grade. The
remaining 5% were Field Managers who were interviewer supervisors responsible for
4
SMB 51 1/03
managing and training interviewers as well as carrying out occasional interview
assignments (Table 3).
Table 3 Experience and performance by field force, 2001
3.3. Attitudes and behaviour related to making contact with respondents
Barton (1999) and Campanelli, Sturgis and Purdon (1997) found that interviewers are
more likely to find respondents at home in the evening during the week and at
weekends than other times. It is therefore important for interviewers to be available
for work in the evening and at weekends. Eighty-four percent of interviewers were
happy to work regularly in the evenings and nearly all the interviewers were
agreeable to working occasionally at the weekend. The proportions cited here
represent interviewers’ views rather than behaviour so some of the interviewers who
said they were unhappy to work regularly in the evenings may be willing to do so.
Interviewers are trained in procedures that may increase their chance of making
contact with respondents. For example, interviewers are advised to leave a note
behind if no one is at home and to ask neighbours about the best time to call. They
are also instructed to make at least four calls to addresses where no one is at home.
The 2001 IAS showed that interviewers used these procedures to maximise their
5
SMB 51 1/03
chances of contacting respondents. Three quarters reported that they always or
frequently left a note behind or asked neighbours about the best time to call if they
found no one at an address. Nearly all interviewers (92%) would, on average, make at
least four more calls at the address if they found no-one at home when they first
called; 38% reported that, on average, they would make 10 or more calls at addresses
where they found no-one at home (Table 4).
Table 4 Attitudes and behaviour affecting contact by field force, 2001
3.4. Attitudes and behaviour related to securing co-operation
A fifth believed that they could convince the most reluctant of respondents. Nearly
half (47%) said that a refusal rarely or never affected their feelings at the next contact
and even more (77%) said a refusal rarely or never affected their behaviour at the
next contact.
6
SMB 51 1/03
Other researchers, including Morton-Williams (1993), who have studied interviewer
behaviour have found that effective interviewers do not use a set approach at every
household but are responsive to the situation and tailor their approach according to
cues they receive from the respondent. The lAS found that just over half the
interviewers said most respondents could not be dealt with in the same way and 70%
said they altered their doorstep introduction according to the household they visited
(Table 5).
Table 5 Attitudes and behaviour affecting co-operation by field force, 2001
7
SMB 51 1/03
3.5. Changes in interviewer characteristics, attitudes and behaviour
between 1998 and 2001
A comparison of the 2001 results with those from 1998 has shown that the
characteristics, attitudes and behaviour of the ONS field force have changed in the
following ways:
•
•
•
•
•
The proportion of men in the field force had grown from 51% in 1998 to 60%
in 2001.
The proportion of interviewers with two or fewer years’ experience had
increased from 36% in 1998 to 47% in 2001. This was because the size of the
field force was increased in response to the expansion of the business of
ONS’ Social Survey Division.
More interviewers were willing to work occasionally at weekends in 2001
(96%) compared with 1998 (91%).
Interviewers had increased the number of calls they made to an address if
they found no one at home. In 1998, 29% made ten or more calls to an
address if they found no one at home; the figure increased to 38% in 2001.
More interviewers in 2001 than in 1998 believed that a refusal rarely or never
had an effect on their feelings or behaviour at the next contact. The
proportion who said a refusal rarely or never affected their feelings at the next
contact rose from 34% to 47% while those who said a refusal rarely or never
affect their behaviour at the next contact grew from 60% to 77% (Table 6).
Table 6 Comparison of interviewer characteristics, attitudes and behaviour,
1998 and 2001
8
SMB 51 1/03
3.6. Variations in interviewer characteristics, attitudes and behaviour
The interviewers who took part in the lAS can be divided into two groups. The first
group (232 interviewers) worked exclusively on the Labour Force Survey (LFS)
while the second group (437 interviewers) worked on all the other households
surveys including the General Household Survey, Family Expenditure Survey,
Family Resources Survey, National Travel Survey, the ONS Omnibus and the
National Diet and Nutrition Survey. The second group is referred to as the General
Field Force (GFF). The work carried out by the LFS and GFF interviewers is slightly
different. The LFS interviewers work only on one survey while GFF interviewer
typically work on more than one survey at once and work on several surveys over the
course of a year. Due to the size of the LFS, LFS interviewers have more regular
work compared with their GFF colleagues. In addition, the LFS is a simple survey
compared with most of the other ONS surveys but LFS interviewers have a shorter
time period to complete an assignment of work; one week compared with a typical
field period of one month for most surveys carried out by GFF interviewers. Since the
nature of the work of LFS and GFF interviewers is slightly different, it is not
surprising that the analysis has revealed a number of differences in characteristics,
attitudes and behaviour between the LFS and GFF interviewers (Tables 2 — 5):
• The proportion of male interviewers was higher on the LFS (67%) than in the
GFF (56%).
• LFS interviewers had fewer years of experience as interviewers compared with
interviewers in the GFF (54% of LFS interviewers had two or less years’
experience while the figure for the GFF was 43%). This is because the expansion
of the LFS field took place earlier than the growth of the GFF.
• Compared with interviewers in the GFF, LFS interviewers were less likely to
have worked for another survey organisation (20% compared with 28%).
• LFS interviewers were more likely than GFF interviewers to leave a note at an
address when no one was at home or to ask neighbour about the best time to call
at an address. This may be due to the fact that LFS interviewers have only a week
to make contact with the household at a sampled address while GFF interviewers
typically have a month to contact the household.
• LFS interviewers tended to make fewer calls to addresses where they had found
no one at home than interviewers in the GFF. Again, this may be because LFS
interviewers have much less time to make contact with a household.
• LFS interviewers were more likely to feel that they could convince the most
reluctant of respondents and to say that a refusal never or rarely affect their
behaviour at the next contact.
• Interviewers on the LFS were more likely than GFF interviewers to agree that all
respondents can be dealt with in much the same way. This is likely to be due to
9
SMB 51 1/03
the fact that LFS interviewers work on a single survey rather than a number of
surveys covering a diverse range of topics.
3.7. Variations in behaviour and attitude by interviewer experience
Interviewers’ behaviour and attitude also varied by interviewing experience:
• The proportion saying they were happy to work evenings regularly rose from 76%
amongst interviewers with less than one year’s experience to 89% amongst those
with nine of more years of experience. However, it is not possible to determine
from the data whether this difference was due- to interviewers becoming more
agreeable to working in the evening as they gained experience or due to the fact
that people who were unhappy about working in the evening left the profession
after a short period.
• The percentage reporting they would, on average, make more than 10 calls to an
address if they found nobody at home increased from 26% amongst interviewers
with less than one year’s experience to 49% for those with nine or more years of
experience.
• Experienced interviewers were more likely to tailor their approach to fit the
respondent than to use a set style. Interviewers with three or more years’
experience were less likely to agree that most respondents could be dealt with in
much the same way (about 24% compared with about 40% amongst less
experienced interviewers). About three-quarters of interviewers with three or
more years’ experience said they altered their introduction to fit the household
compared with 66% or less amongst less experienced interviewers.
• The proportion who frequently or always left a note at an address if they found
nobody at home fell from 86% amongst interviewers with less than one year’s
experience to 62% amongst those with nine of more years of experience (Tables
7—11).
Table 7 Percentage who were happy to interview regularly in the evening by
interviewer experience, 2001
Table 8 Average number of calls made to an address before recording it as a
non- contact by interviewer experience, 2001
10
SMB 51 1/03
Table 9 Percentage agreeing that respondents can be dealt with in much the
same way by interviewing experience, 2001
Table 10 Percentage who altered the doorstep introduction to fit the household
by interviewing experience, 2001
Table 11 Percentage who left a note at contact address if no-one was home by
interviewing experience, 2001
4. Summary and plans for future analysis
The above results are a very preliminary look at the 2001 lAS data. They show that
interviewer attitudes and behaviour varied with fieldwork procedures such as the
length of the field period as well as interviewing experience. Interviewers with more
years of experience were more likely to use approaches known to be effective such as
working regularly in the evening, making a lot of calls to an address where nobody is
at home or be able to tailor their approach. We plan to carry out more detailed
analyses on these topics. Future analyses will involve linking the data to information
relating to the response rate each interviewer had achieved and developing multi-level
11
SMB 51 1/03
models to study the effect of the interviewer on survey contact and co-operation rates.
Multi-level modelling allows us to estimate the variance accounted for at the
interviewer, household and area levels separately. At every level, we will know how
much variance is accounted for by the variables in the model and how much is
unexplained. This type of statistically modelling will provide more information about
whether the differences in attitudes and behaviour between different groups of
interviewers presented above affect interviewer performance or not. We will also
continue to carry out descriptive analysis to provide information to help improve
fieldwork operations and to identify quality improvements opportunities.
References
Barton, J. (1999) “Effective calling strategies for interviewers on household surveys.”
Survey Methodology Bulletin, 44, 1/1999. London: Office for National Statistics.
Beerten, R. and Martin, J. (2000) ‘Interviewer Influences on Survey Non-response.’
Paper presented at the Conference on Social Science Methodology in 2000, Cologne,
3-4 October 2000.
Campanelli, P., Sturgis, P. and Purdon, S. (1997) Can You Hear Me Knocking: An
Investigation into the Impact of Interviewers on Survey Response Rates. London:
Social and Community Planning Research.
Groves, R. M., Cialdini, R. B. and Couper, M. P. (1992) Understanding the Decision
to Participate in a Survey. Public Opinion Quarterly, 65t4), 475-495.
Groves, R. M. and Couper, M. P. (1998) Nonresponse in Household interview
Surveys. New York: Wiley.
Hox, J. and de Leeuw, E. (2002) “ The Influence of Interviewers’ Attitudes and
Behaviour on Household Survey Nonresponse: An International Comparison,” in: R.
M. Groves, D.A. Dillman, J.L. Eltinge and R.J.A. Little (eds.) Survey Nonresponse.
New York: Wiley.
Morton-Williams, 1 (1993) Interviewer Approaches. Aldershot: Dartmouth
Publishing.
White, A., Freeth, S. and Martin, J. (2001) “Evaluation of survey data quality using
matched census-survey records.” Survey Methodology Bulletin, 49, 2/2001. London:
Office for National Statistics.
12
SMB 51 1/03
Nikki Bennett
Proposals for an Integrated Social Survey
Nikki Bennett
1. Introduction
There is a need for National Statistics to provide better information on key social and
socio-economic variables such as housing, employment, ethnicity, education and
health. This will improve inter-censal monitoring of key variables for a range of
policy purposes and small area statistics for the Neighbourhood Statistics Programme
(NeSS). At present, the Labour Force Survey (LFS) is the largest single source of
survey data available for monitoring purposes and produces estimates to a level of
precision not matched by any other UK household survey. Together with local LFS
boost samples in England (LLFS) and Wales (WLFS), the LFS sample provides
annual estimates for a range of indicators (mainly concerned with the labour market
and adult training targets), down to Local Education Authority (LEA) level (similar in
size to county). To provide a wider range of information at (the smaller) Local
Authority District level would be a big step forward for NS, but to achieve this a
much larger sample of households is needed. ONS is in the process of developing its
ideas for the integration of the ONS-led continuous household surveys to meet this
and related objectives.
The ONS surveys and the contribution they could make each year to an ISS are:
13
SMB 51 1/03
Nikki Bennett
In addition to these surveys, there is a proposed further addition to the sample to meet
the needs of Neighbourhood Statistics (NeSS survey). With the contribution of the
LFS boosts and the NeSS Survey, an integrated survey might achieve interviews with
more than 240,000 households and over 400,000 adults. If this development is
successful, the addition of other NS continuous household surveys at a second stage
would increase the sample size.
Proposals for integration were investigated previously in the 1990s. There are now
more business drivers. In brief:
•
there is increasing demand for small area statistics which cannot be met with
current arrangements: there is particular pressure for information on ethnic
group information and measures of income but other variables are also
needed;
•
the Neighbourhood Statistics work adds to this demand: larger samples
would also improve model-based estimates;
•
the investment in surveys should be exploited more fully between censuses;
•
there is a need for greater coherence in NS statistics, which integration of
sources will promote;
•
there is already a programme of harmonisation of questions and
classifications across government; integration would take advantage of, and
further develop, such standardisation;
•
there is increasing demand for more surveys: in particular, possible EU
Regulations obliging member states to conduct a survey of income and living
conditions (SILC) and to make design changes to the LFS;
2. Proposal
This proposal concentrates on integration of the continuous surveys on which ONS
leads: LFS. EFS, GHS and NS Omnibus. The design is compatible with continued
delivery of all current outputs of these surveys, including those which are not annual
estimates; this is particularly important for the LFS and its boosts. The working title
for the project is the Integrated Social Survey (ISS); a title with greater public appeal
will be needed if the development reaches the stage of trials.
The ISS will cover Great Britain and allow for the addition of Northern Ireland data
to provide UK results as at present.
The aim is to redesign the questionnaire content in terms of modules, covering all the
content of the existing surveys. A fixed core of key variables, e.g. demographic
information, household composition, economic status (employment, unemployment,
inactivity), ethnicity, tenure and banded household income, will be included in every
interview to provide a very large sample. Other modules will be asked of only part of
the sample, e.g. the labour market module would be very similar to the current LFS
preserving the current panel design required by Eurostat and other current users. The
income and expenditure module would be very similar to the existing EFS with diary
component. An illustration of how modules might be combined is shown
diagrammatically at Annex 1.
14
SMB 51 1/03
Nikki Bennett
The ISS design has taken note of relevant developments in other NSIs and of earlier
work by ONS towards an integrated household survey which has
demonstrated its operational feasibility.
The design will allow for addition of other continuous surveys at a later stage, if their
sponsors choose.
Preliminary thinking on the form of integration proposed for the initial stage of the
ISS involves:
•
a single sample of addresses drawn from the Postcode Address File;
•
a survey instrument comprising all the topic modules to meet the information
needs currently met in the separate surveys;
•
whole-sample topic modules which provide, for all ISS households and
persons, information on key variables which require samples of that size;
•
part-sample topic modules which provide information on variables for which
sufficient precision to meet policy needs can be obtained from just part of the
ISS sample;
•
combination of the whole-sample topic modules with selected part-sample
topic modules to make a viable interview combination, i.e. making up an
interview which flows acceptably for respondents; there will be a small
number of different interview combinations so that all part-sample topic
modules are covered (see Annex 1);
•
interpenetration of the interview combinations in interviewer workloads
(mixed workloads). It is essential to a cost-effective ISS that each interviewer
carries out the range of different combinations in a local area, to minimise
travel costs.
3. Benefits
The potential improvements from the ISS approach will include a greater precision of
estimates without increasing costs, resulting from:
•
a larger independent annual sample for the variables in the whole-sample
(core module) than is currently available from the LFS and boost surveys
alone
•
an unclustered design, which would be an improvement in the precision of
estimates for all variables except those particular to the LFS (which are
already based on an unclustered design) without the need for increased
interviewing costs.
The ISS also provides the opportunity for more regular, coherent reporting of
estimates from a single source. At present, even with harmonised questions across
surveys we have problems with coherence. For example, a key pioneering role of the
GHS was to monitor growth in one parent families and cohabitation. Over time, the
LFS has been used for the same purpose, given its larger sample size. Although the
differences in estimates are small, annual results from the two surveys give different
results.
The ISS could provide more regular estimates for whole sample topics because of
the sample size on which estimates are based ( the exception being the labour
15
SMB 51 1/03
Nikki Bennett
market variables). At the household level, indicators relating to tenure,
accommodation type, and amenities, for example, could be produced on a quarterly
basis with greater precision than the annual estimates from current ‘best sources
Likewise, at the person level, for indicators relating, for example, to marital status,
cohabitation, health, and provision of care.
For more regular reporting of key labour market indicators - economic activity,
employment and unemployment rates - even a large boost to the current sample size
would have a very limited impact on the variances of monthly change unless the
design was changed to a monthly panel.
The ISS whole sample module will also enable users to analyse a number of key
variables in combination, with a considerably larger sample than is possible from
existing surveys, such as the GHS and EFS. For example, there will be improved
precision for measures such as average, household income, % reporting a limiting
longstanding illness, % reporting health is not good, % living in flats or maisonettes,
by ethnic group using information from the ISS whole sample. Estimates of precision
for these types of output will be calculated as part of the development work for the
ISS.
As well as more regular reporting, there is increasing demand for small area statistics
which cannot be met by current survey arrangements. The existing surveys are
designed to provide regional/country estimates and only the LFS is large enough to
estimate for smaller areas. Together with local LFS boost samples in England (LLFS)
and Wales (WLFS), the LFS sample provides annual estimates for a range of
indicators (mainly concerned with the labour market and adult training targets), down
to Local Education Authority (LEA) level (similar in size to county). To provide a
wider range of information at (the smaller) Local Authority District level would be a
big step forward. The ISS would also provide a significantly larger sample to improve
small area estimates as part of the Neighbourhood Statistics Programme.
It will be possible to use the ISS whole sample and part sample output separately or
in combination to paint accessible and comprehensive pictures of society in general
and to explore specific topics in detail. There will be scope for short, topical outputs
from the ISS on a regular (quarterly) basis and for multi-topic analyses and articles
with the Web at the centre of providing the new look outputs.
4. Consultation and development work
ONS believes the arguments for developing in this direction are compelling.
Informal discussions suggest some OGDs will be attracted to join in at later stages.
For the initial stage, ONS is consulting with departments who pay into the four
ONS-led surveys.
If ONS decides to go ahead, resources will be needed to develop the fieldwork,
methodological and computing infrastructure for the ISS, and to test it thoroughly.
We will need to test the new style of interviewer work package and, in particular, to
check the potential impact on response. Much of the methodological work and testing
has already been done during previous exercises, but important new work remains to
be done on sample design and estimation.
Thorough testing will be required to ensure that there are no insurmountable
problems for the delivery of high quality data as a result of the transition from the
current survey designs to the ISS. In particular, ONS will need to maintain the
continuity of key time series, e.g. macro-economic time series produced from the
16
SMB 51 1/03
Nikki Bennett
LFS. Care will be taken at each appropriate stage of development to measure and to
minimise the impact of survey changes on the continuity of key time series. If, in
spite of these precautions, any unavoidable discontinuities arise, ONS will consult
fully with users about ways in which continuous historical time series for the key
variables can be estimated.
5. Timing
The ISS is at an early stage of development. A scoping paper has been presented to
the ONS Executive Board who have agreed on the importance of the development.
They requested more information, which is now being prepared.
ONS is developing a range of major improvements at present. Some of the officewide initiatives (such as the Statistical Infrastructure Development Programme,
Information Management Programme and Labour Market Re-engineering Project)
will deliver new methods, tools and infrastructure that ONS will be able to take full
advantage of in developing an ISS to provide world-class survey taking. Given these
developments, and the interaction between them, it is likely that April 2006 will be
the earliest possible start date for data collection on the ISS.
Annex 1
17
SMB 51 1/03
Lucy Haselden and Richard J Jenkins
The National Identity Question
The National Identity Question: methodological
Investigations
Lucy Haselden and Richard J Jenkins 1 Background
1 Background
A national identity question was introduced on the Labour Force Survey (LFS)
following the 2001 Census. The 2001 Census used a new ethnicity classification
which sub-divided the categories for “White” into “British”, “Irish” and “Other
White”. A number of groups expressed dissatisfaction with this question as it did not
provide a tick box for respondents to identify themselves as “Welsh”, “Scottish” or
“English” if they thought that was more appropriate than “British”. Furthermore,
there were no specific tick boxes for “British”, “English”, “Scottish”, “Irish” or
“Welsh” for those who were not White.
In order to address these concerns, National Statistics decided to introduce a separate
national identity question, which would be asked of all respondents and would allow
respondents to describe themselves as English, Scottish, Welsh, Irish, British or other.
This question was introduced on the Labour Force Survey in the March-May quarter
of 2001.
The aim of this paper is to evaluate the new national identity question and to highlight
any concerns that should be taken into account when analysing the data.
2 The Question
The national identity question is asked early on in the questionnaire, after respondents
have given details of their nationality, country of birth and citizenship status but
before the questions on Welsh language and ethnicity.
The question is not asked in Northern Ireland but is asked in all the other UK
countries. Although the same question is asked in England, Scotland and Wales, the
order of the answer categories varies according to the country in which the interview
takes place. This is detailed in the table below.
18
SMB 51 1/03
3 Order Effects
The national identity question was tested on the National Statistics Omnibus Survey
in January and March 2001 to see whether the positioning of the British category had
an effect on responses given.
The NS Omnibus Survey interviews approximately 1800 adults (aged 16 or over) in
private households in Great Britain each month. The survey uses the Postcode
Address File of ‘small users’ for the sampling frame and individual households are
identified at each address (usually one household per address). Within each
household, one randomly selected adult is interviewed. More details of the Omnibus
sample can be found in the “National Statistics Omnibus Survey - Technical Report”.
A split sample test was carried out, with respondents randomly allocated to one of
two-equal sized groups: those in group A were presented with a ‘British first’
showcard and those in Group B were presented with a ‘British last’ showcard. In each
case the question asked was:
“What do you consider your national identity to be? Please choose your answer from
this card, choose as many or as few as apply.”
The answer categories for groups A and B are shown in the table below.
Table 2 Lists of national identity options on showcards respondents in England,
Scotland and Wales
The test showed that changing the order of the answer categories had the following
effects:
• Respondents in England and Scotland were more likely to give a single answer of
“British” if they were presented with “British” as the first answer category on the
showcard (Group A). In England 38% of those in Group A were “British only”
compared with 26% in Group B. Iii Scotland 23% of those in Group A said they were
“British only” compared with 13% in Group B. This difference was not apparent in
Wales, however, the sample size in Wales was very small and this might have
masked such differences. (Table 3)
• Respondents in England were more likely to give a single answer of “English” if
they were presented with “English” as the first answer category on the showcard
(Group B). In England 39% of those in Group B were “English only” compared with
27% in Group A. The differences between the two groups in saying they were
“Scottish only” or “Welsh only” in Scotland or Wales was not so marked as in
England, probably reflecting a stronger nationalistic feeling in these countries.
(Table 3)
• The order of the answer categories on the showcard had very little impact on the
number of answers given by respondents in each of the three countries. (Table 4)
19
SMB 51 1/03
• Respondents in Group A (i.e. those who were presented with “British” as the first
answer category on the showcard were more likely than respondents in Group B
to give a response which included British. This difference occurred in all three
countries but was most notable in England where 65% of respondents in Group A
gave ‘British’ as one of their answers compared with only 52% of those in Group
B. (Table 5).
Table 3 - Answer combinations by whether British given as first or last option
20
SMB 51 1/03
Table 4 Number of National Identities given by whether British given as first or
last option
Table 5 National Identity by whether British given as first or last option
Note. Percentages do not sum to 100 as more than one answer could be given. Chi
square analysis not appropriate for multiple response data.
4 Variation in responses between the LFS and the National Statistics
Omnibus Survey
The national identity question asked on the LFS was very similar to the ‘British last’
option asked on the NS Omnibus Survey. This allows us to compare results from the
two surveys to see how sensitive responses are to small changes in the question
wording and to the survey/context effect. In order to do this, responses from the NS
Omnibus survey — Group B (British last) were compared with responses given on
the LFS by wave one respondents who were aged 16 or over for the year 2001/02
The wording of the question on the two surveys was exactly the same. . However, the
question was administered in a slightly different way on the two surveys:
• LFS respondents were not given a showcard whereas those on the NS Omnibus
Survey were. On the LFS the interviewer read out the list of answer categories.
21
SMB 51 1/03
• The LFS has a number of questions before the national identity question about
nationality and country of birth which are not asked on the NS Omnibus Survey. This
means that the context for the question is slightly different on the two surveys.
• Some LFS respondents were interviewed on the telephone. The effect of this
difference in mode is examined in section 5.
The results from the two surveys show:
• Respondents on the LFS were more likely to give just one answer than those on the
NS Omnibus Survey. For example, in England 92% of respondents on the LFS gave
just one answer compared to 72% on the NS Omnibus Survey. The pattern was the
same in Scotland and Wales. (see Table 6)
• Respondents to the LFS and were slightly less likely to say they were English,
Scottish or Welsh. (see Table 7). However, respondents to the NS Omnibus Survey
were considerably more likely than LFS respondents to include British as a second
answer.
• In all three countries, respondents to the LFS were more likely to describe
themselves as Fnglishl/Scottishl/Welsh only’ or British only’ reflecting the fact that
on the whole they only gave one answer. Consequently LFS respondents were less
likely to describe themselves as holding dual identities, Englishl/Scottishl/Welsh and
British’ for example. (Table 8)
Table 6 A comparison of the Number of answers given in response to the
national identity question on the LFS and NS Omnibus Survey
22
SMB 51 1/03
Table 7— A comparison of the answers given in response to the national identity
question on the LFS and the NS Omnibus Survey
Note: Chi square analysis not appropriate for multiple response data.
Table 8 - A comparison of the answer combinations given on the Omnibus
Survey and the LFS
1
This includes all cases where English and British answers categories were chosen,
even if others were also included.
23
SMB 51 1/03
5 Differences in mode
The Labour Force Survey uses both face-to-face and telephone interviewing. It does
not use showcards for either mode of interviewing and, as far as possible, the LFS
questionnaire is administered in exactly the same way, whether it is conducted faceto-face or over the telephone. It is therefore possible to compare the answers given to
the national identity question in face-to-face interviews with those given from
telephone interviews.
Table 9 shows that, in England, there was only a very small variation in the
proportion of respondents giving one answer to the national identity question with
mode of interview. Chi square analysis did reveal a significant association between
the number of answers given and mode of interview but this result can be explained
in terms of the large sample sizes provided by the Annual Local Area Labour Force
Survey. In Scotland and Wales, however, respondents were more likely to give just
one answer in a face-to-face interview than they were in a telephone interview
Table 9 Number of answers given by country and mode
Table 10 shows that in England, the proportion of respondents giving each answer
combination did not vary with mode of interview, although chi square analysis did
reveal a significant association due to the large sample sizes.. In Scotland and Wales,
however, respondents to face-to-face interviews were slightly less likely to say they
held dual identities than those interviewed by telephone.
24
SMB 51 1/03
Table 10 Answer combinations by country and mode
6 Conclusions
The question is sensitive to the order of the answer categories presented. Whether
“British” is presented as the first or last answer category affects the answers given.
This particularly makes a difference in England to the proportions saying they are
“English only” or “British only”. The effect in Scotland and Wales on the proportions
saying “Scottish only” or “Welsh only” is much less marked. However, it does affect
the proportions saying “British only” in Scotland.
The question is sensitive to the differences in the mode via which it is administered.
The pattern of answers given to the LFS national identity question is different to that
given to the NS Onmibus survey question, even though the question wording is the
same In particular, respondents are much more likely to give only one answer on the
LFS than they are on the NS Omnibus Survey. This in turn affects the estimates of
___________________________
25
SMB 51 1/03
each national identity. In England the proportions saying “English” and “British” are
different to the Omnibus estimates. In Scotland and Wales the proportions saying
“British” are different to the Omnibus estimates.
Although it is not clear why these differences occur, the following might have
contributed to this:
• the lack of showcards on the LFS;
• the different context of the questions on the LFS (i.e. the presence of questions
about country of birth and nationality immediately before the national identity
question).
Whether the survey was carried out face to face or over the telephone also has a very
small but significant effect on the answers given to this question.
Furthermore, there may have been some changes in responses over time. However,
this seems unlikely as the two surveys were carried out at broadly similar times time.
Thus when analysing or presenting data on national identity it is worth noting that the
question is sensitive to order effects of the answer categories, exactly how the
question is administered, changes in context, and changes in survey instrument, so
care should be taken when comparing results across surveys.
26
SMB 51 1/03
Mark McConaghy and Roeland Beerten
Influencing response on the FRS
Influencing response on the Family Resources Survey
by using incentives
1. Introduction
The Family Resources Survey (FRS) is a continuous survey of private households
carried out in Great Britain since 1992. The survey is commissioned by the
Department for Work and Pensions (DWP) and is primarily used to monitor living
standards in Great Britain, to examine people’s relationships and interactions with the
social security system and to model benefit uptake.
The FRS has been carried out throughout its life by a consortium comprising the
Office for National Statistics and the National Centre for Social Research. Each
organisation is responsible for half of the fieldwork. Combined data are delivered to
DWP.
Participation in the FRS is voluntary and since 1997/8 there has been a small drop in
response. To try to improve the response on this survey various initiatives were
undertaken. One initiative was to distribute a token incentive to the sampled
households.
The first part of this article discusses general response figures and trends on the FRS.
The second part describes two ONS incentive experiments. The figures in the first
part refer to the combined data, the figures on the incentive experiments refer to ONS
data only.
2. Background
2.1. Historical trend in FRS response rates
The FRS began in 1992 and following a six-month lead-in period has been running as
a continuous annual survey since April 1993.
Figure 1 shows the response rate (defined as the number of fully and partially co
operating households as a proportion of the number of households eligible for the
survey) has increased since 1993/4, when it was 68 percent, to a high of 71 percent in
1995/6. Since the high of 1995/96 the response rate has decreased, with a slowed
down decline from 1998/99.
It is difficult to explain the fall in response from 1995/96 onwards. This trend is not
unique to the FRS as other ONS surveys show a similar trend (Martin and Matheson,
1999). Various factors could have contributed to this decline (for an elaborate
discussion see Groves and Couper. 1998). One of the factors which could have
affected response rates is the practice of re-issuing the addresses of non-responding
households to (often more experienced) interviewers who try to convert these
households to a responding household. In 1997 the FRS contract was re-tendered by
the DWP and was successfully retained by the original consortium. A feature of the
new contract was that the financial provision to call back on households that had
initially refused or were not contacted after the first field period was removed
(although depending on available resources some reissues were undertaken).
Response declined further by one percentage point in the first year and by a further
two percentage points in 1998/9 and 1999/2000 by which time it stood at 67 percent.
In the last survey year for which results are available response declined another one
27
SMB 51 1/03
percentage point (despite a change to the contract which made additional provision
fdr a boost to response through reissues).
Figure 1 Response rate on the FRS (1993-2001)
Essentially non-response comprises two elements: (1) sampling units that are not
contacted and (2) those where contact is made but who refuse to participate. These
two components are known as the contact rate and the co-operation rate. The contact
rate is defined as the number of households that have been contacted as a proportion
of eligible households; the co-operation rate is defined as the number of co-operating
households as a proportion of the number of contacted households. Ineligible
households are not included in the calculation of response rates and have therefore no
effect on these rate calculations.
Figure 2 shows the overall response rate (which is the product of the contact rate and
the co-operation rate) for the FRS from 1993/94 to 2000/01. The contact rate has
remained fairly stable with a high of 97 percent in 1996/7 and a low of 95 percent in
2000/1. The highest level of co-operation was 74 percent in 1995/6 and the lowest
level of 69.6 percent in 2000/1. The last three years have seen levels of co-operation
lower than that observed in earlier years. As contact rates have remained fairly stable
variation in response has been primarily due to variation in the co-operation rate, with
the highest level of response (1995/96) being reached along the highest co-operation
rate.
28
SMB 51 1/03
Figure 2 Contact, co-operation and response rates on the FRS (1993-2001)
2.2. Regional variation in contact and co-operation rates
The figures discussed above show a decline in response and co-operation rates for the
FRS in Great Britain. However, previous research has shown survey response is
affected by characteristics of the areas where households are sampled (e.g. Beerten,
1999). In the survey literature several socio-environmental influences on survey
participation have been identified (Groves and Couper, 1998). The most important
correlates of survey participation are indicators of urbanicity, population density,
crime and social disorganisation. The size and strength of these correlates will vary
depending on the region in which a survey samples its addresses.
As on other surveys response rates (both contact and co-operation rates) differ
regionally on the FRS. Table 1 shows the FRS contact rate for the Government Office
Regions from 1997 to 2001.
Table 1 Contact rates by year and Government Office Region
29
SMB 51 1/03
Over the last four years the contact rate has been lowest in London and Scotland.
Between 1997/8 and 2000/1 the contact rates fell most for London (-3.0 percent) and
the West Midlands (-2.3 percent). The difference in contact rates between the region
with the highest proportion of contacts and the region with the lowest was 3.1 percent
in 1997/8 and 1998/9, but in 1999/2000 this increased to 5.2 percent and in 2000/1
was 5.0 percent.
Table 3 Co-operation rates by year and Government Office Region
The co-operation rate also shows regional variation. Between 1997/8 and 2000/1 the
most falls in co-operation rates were in London (-5.8 percent), East of England (-3.9
percent) and the South West (-3.7 percent). Regions with the highest level of co
operation have remained fairly consistent and include the North East, Wales and
Scotland. The only region which experienced an increase in co-operation was
Yorkshire and Humberside (+2 percent).
Overall, the difference in contact rates between the region with the highest level of
co-operation and that with the lowest in 1997/8 was 7.9 percent. This increased to 8.6
percent in 1998/9 and 9.3 percent in 1999/2000. In 2000/1 this difference increased
by over two percentage points to 11.8 percent.
3. Tackling non response through the use of incentives
There is a great deal of sometimes inconsistent literature about the effects of giving
incentives in surveys of the general population (see for example Church (1993),
Hubbard and Little (1988), Warriner et al. (1996), Tzamourani (2000) and Singer
(2002)). Incentives can be used in two ways, either to thank respondents for taking
part, when it is given on completion of the interview, or as a gift so that the incentive
is given regardless of whether or not the respondent participates. Most of the
literature suggests that the latter is more effective in improving response rates
(Singer, 2002).
The literature suggests that the increase in response rates will be significantly greater
if cash is offered in a mail survey rather than offered as a charitable donation. Nonmonetary incentives or goodwill gestures, such as small gifts or pens, are low cost but
there has been little evidence that they raise response in mail surveys.
In a face-to-face context the evidence suggests that offering a cash amount is the best
way to encourage survey co-operation, with improvements in response positively
30
SMB 51 1/03
correlated with the amount of cash offered (Singer, 2002). At ONS offering financial
incentives to survey respondents, either in cash or as vouchers, is most commonly
used on diary-keeping surveys because these have a relatively high respondent
burden. There is some evidence that offering a large financial incentive (for example
£20) might improve co-operation among groups such as younger people and those
with lower income (at least on literacy surveys) but that the majority of respondents
would take part in these surveys without payment (Carey, 2000).
In an attempt to improve response ONS distributed a small gift of a book of first class
postage stamps with the advance letter to all households sampled on the Family
Expenditure Survey in 1999-2000. Compared with earlier periods when no such
incentive was included, there appeared to be a slight increase in response. In order to
establish the net effect of this particular incentive a series of split-sample experiments
was set up on the FRS in 2000 and 2001.
3.1. The first split sample experiment (April to September 2000)
The households sampled for the FRS in April to September 2000 were randomly
assigned to an experimental group that received a gift of a booklet of stamps as a
gesture of appreciation for (their) time’ with the advance letter and a control group
that did not. As receipt of the incentive was not conditional on respondents’
agreement to take part in the survey, interviewers were instructed not to mention the
incentive unless respondents made a spontaneous conment or question. This set-up
represented an experimental design and ensured that area, interviewer and time
effects did not confound the results 4 .
A comparison of the co-operation rates for the two conditions (incentive vs. no
incentive) revealed that the respondents who received the incentive were significantly
more likely to co-operate than those who did not (70.4 percent versus 67 percent,
significance p<0.0l).
Figure 3 presents the difference between the control and experimental groups by
month. The graph indicates that although the co-operation rates are falling for both
groups, the group that received the incentive consistently has the higher co-operation
rate.
While use of this incentive appears to help to increase response overall, it is useful to
examine how this effect varied by region. We would expect the incentive to have a
greater effect in regions with lower co-operation, for example in London Figure 4
presents the co-operation rates for the two conditions by Government Office Regions
Some regions have been grouped to enable comparisons to be made The Figure 4
shows that the incentive appears to have had most impact in London (with a
difference of 7.8 percent between the expenmental groups) but little effect in the
North. The bases for these regional comparisons are quite small and only London
(p< 0.01) and South East & South West (p<0.05) showed significant differences
From these results it is clear that the incentive had a differential impact on regions
with those having lower response (e.g. London, South East and South West)
benefiting more than areas where co-operation was higher (e.g. North West and North
East, Scotland).
31
SMB 51 1/03
Figure 3 Co-operation rates by whether incentives given by month
Figure 4 Co-operation rate by region (grouped)
3.2. The second split-sample experiment (October — December 2001)
Response on the FRS rose by almost five percentage points at the time of the Census
of Population in April 2001 (data not shown) when there was a large amount of media
coverage about ONS. This ‘Census effect’ on survey response was attributed to the
increased visibility of ONS as a survey organisation which would have helped
respondents understand what the work of ONS was about. This in turn was expected
to have a positive effect on their willingness to co-operate.
A second split sample experiment with incentives was set up to see whether providing
additional information about the survey organisation has as strong an effect as
sending an incentive with the advance letter. It should be noted that a ‘Census effect’
is difficult to create in an experimental situation, not least because of the strong role
of the media in the true Census effect’. However, it was thought that providing extra
information to increase respondent co-operation could be introduced in an
experiment.
32
SMB 51 1/03
In the experiment, which was introduced on the FRS in October 2001, half of the
households in the sample received a booklet of stamps in addition to the advance
letter while the other half of the sample which received the ‘UK in Figures’ booklet
together with the advance letter. This booklet contains information about the statistics
that ONS produces and the surveys that it carries out.
The results of the experiment show that the group which received stamps continued to
show significantly better co-operation rates than the group which received the booklet
‘UK in Figures’ (71.9 vs. 66.7 percent, significance p<0.0l).
The results of this second experiment thus show that token incentives such as stamps
have a positive effect on response when compared with providing respondents with
additional information about the survey organisation.
4. Conclusions
In this article we have seen that the response rate on the FRS has fallen over the last
few years, mainly because of an increase in the proportion of households that refuse
to co-operate with the survey. The figures also showed evidence of differences
between the regions in survey co-operation on the FRS.
In order to try to reduce the nonresponse levels on the FRS an experiment was set up
to decide whether offering a small incentive would have an effect on survey co
operation. The figures showed this was the case: compared with sending households
the advance letter only sending out stamps with the advance letter had a positive
effect on co-operation.
After the positive ‘media effect’ on response at around the time of the Census, the
second experiment tried to find out whether providing extra information about the
survey organisation has as strong an influence as sending a monetary incentive such
as stamps. The experimental results did not confirm this: the stamps have a stronger
positive effect compared with providing extra information about the survey
organisation.
Although the experiments described here show that incentives have a positive effect
on response, it is clear that they only address part of the problem, and they come at a
cost. More research is needed to look into the balance between the effectiveness of
these incentives and their cost, and to find out about the comparative benefits of other
strategies to improve response. For example, in the first part of this article the
negative effect on response of removing a re-issuing policy in the survey contract was
described. The positive effects on response and the associated costs of making
extended efforts to obtain survey co-operation through re-issues could be compared
with the effect and costs of an incentive strategy (for a discussion of the former, see
Lynn et al., 2002 and Lynn and Clarke, 2002). Further research is also needed to
assess the effects of the different strategies on nonresponse bias, and the effect they
have on measurement error.
References
Beerten, R. (1999) “The Effect of Interviewer and Area Characteristics on Survey
Response Rates: an Exploratory Analysis”. in: Survey Methodology Bulletin, 45, July
1999. London: ONS.
Church A.H. (1993) Estimating the effects of incentives on mail survey response
33
SMB 51 1/03
rates: a meta-analysis. Public Opinion Quarterly 57: 62-79
- Carey S. (2000) Measuring adult literacy. ONS, London
Groves R.M. and Couper M.P. (1996) “Household level determinants of survey non
response”. In: Braverman M.T. and Slater J.K. (ed). Advances in survey research.
New directions for evaluation. San Francisco: Jossy-Bass
Groves R.M. and Couper M.P. (1998) Non-response in household interview surveys.
New York: John Wiley & Sons Inc.
Groves R.M. and McGonagle K.A. (2001) “A theory-guided interviewer training
protocol regarding survey participation”. In: Journal of Official Statistics. Vol 17, No
2, 2001 pp 249-265.
Harkness J. and Mohler P. “Two experiments with incentives on mail surveys in
Germany” in: Survey Methods Centre Newsletter Vol 18 No 2. London: Social &
Community Planning Research.
Hubbard R and Little EL (1988) ‘Promised contnbutions to charity and mail survey
responses” Public Opinion Quarterly, 52 223-230
Lynn, P, Clarke P Martin, J and Sturgis P (2002) “The Effects of Extended
Interviewer Efforts on Nonresponse Bias” pp 135-147 in Groves R M, Dillman
D A , Eltinge J L , Little R J (eds) Survey Nonresponse New York Wiley.
Lynn, P and Clarke, P (2002) “Separating Refusal Bias and Non-contact Bias
Evidence from UK National Surveys” in The Statistician, 51(3), pp 319-333
Martin, J and Matheson J (1999) “Responses to Declining Response Rates on
Government Surveys” in Survey Methodology Bulletin, 45, July 1999 London
ONS.
McConaghy M and Rowland M (2001) Family Resources Survey Annual technical
report Apr 1999 Mar 2000 London ONS.
Singer E (2002) ‘The use of incentives to reduce non response in household surveys”
pp 163-177 in Groves R M, Dillman D A, Eltinge J L, Little R J (eds) Survey
Nonresponse New York Wiley.
Singer E (1988) “The effects of incentives on response rates in interviewer- mediated
Surveys’ in Surve% Methods Centre Newsletter Vol 18 No 2 London SCPR
Tzamourani P (2000) “An experiment with a promised contnbution to charity as a
respondent incentive on a face-to-face survey’ in Survey Methods Newsletter Vol 20
No 2 London National Centre for Social Research.
Warriner K, Goyder J, Gjertsen H, Hohner P and McSpurren K (1996) Chanties, no,
Lottenes, no, Cash, yes Main effects and interactions in a Canadian incentives
expenment “p 542-562 in Public Opinion Quarter/v. 60
34
SMB 51 1/03
1
Formerly the Department of Social Security
2
Fully co-operating households are those where a full interview has been obtained
either in person or by proxy with every eligible member of the household. Partially
co-operating households are those where information has been obtained for the head
of household’s benefit unit but the interviewer has been unable to collect information
from other benefit units e.g. because of non contact or refusal. The proportion of
cases classified as partial is usually just over 1 percent.
3
The only mention of this was a sentence in the advance letter stating that ‘As a
gesture of our appreciation for your time, please find enclosed a book of postage
stamps’. Interviewers were informed about the experiment and that the sample had
been systematically split between odd and even serial numbers. Any households that
refused to participate could, of course, keep the stamps.
4
In the analysis which follows all outcomes are final outcomes after the first attempt
and do not include final outcomes to any initial refusal or initial non contact
addresses selected for reallocation.
35
SMB 51 1/03
Karen Irving and Neil Park
Quality Issues in Social Surveys (QUISS) seminar
April 2002
Surveys of Children and Young People
Karen Irving and Neil Park (ONS)
The eleventh in the series of QUISS seminars organised by the ONS was based around
surveys of children and young people. The seminar focused on a diverse range of issues
including design factors, new methodologies and cognitive interviewing techniques. The
seminar was conducted by Siobhán Carey from the Social Survey Division who emphasised
that over the last few years children and young people have become a major area of study for
Social Research covering a wide variety of topics. She therefore welcomed the opportunity to
hear from a number of people involved in surveys of children and young people and the work
they have carried out.
What the years may hold - designing a new longitudinal study of
young people.
lain Noble- Department for Education and Skills (DfES)
In his presentation, Iain discussed the changes in transition of young people from compulsory
education into the labour market and identified the need for a new longitudinal survey.
Iain began by identifying four quality issues that researchers need to consider in designing
a new survey of young people. Firstly, changes in policy environment, secondly, changes
in the labour market, thirdly, the changes in transition from compulsory education to
work and finally, changes that need to be made to research conducted about young
people:
Changes in policy — the new agenda
Changes in policy have resulted in fewer young people entering the labour market and
the policy for young people has becoming more integrated with changes in the funding
frame.
Changes in the labour market
Since 1985, young people are more likely to enter into further education, rather than the
labour market, which has lead to the collapse of the youth labour market. The increase in
young people entering into further education has increased the requirement for part-time
employment, with 70% of A’ level students having a part- time job. Iain raised concern that
some young people who enter into further education are being pushed towards obtaining a
vocational qualification. As a result of this, young people become concerned about the
dangers of polarisation within the workplace. This in turn has lead to the creation of Learning
Skill Councils (LSC’s).
Changes in transition from compulsory education to work
Young people are staying longer in compulsory education and consequently there is a
delay in them entering the labour market. In relation to this, it would be more
36
SMB 51 1/03
beneficial to collect data on the earlier years in young persons’ lives, as presently, surveys
only collect data on young people from the age of 16 to 18 years old.
Changes to research conducted about young people
Iain discussed several quality issues to consider when researching young people for a new
survey:
There is a need for consultation on whether it is feasible to conduct the survey, involving a
general evaluation of the effects of existing policy on young people and looking at how the
survey could support further policy development. In particular, how the survey would reflect
changes over time and establish the characteristics of people who were under performing,
focussing on what is causing under performance and how the survey data could be used to
further policy development.
Research needs to focus on covering a large number of policy options and also needs to be
flexible in terms of additional material that deals with any new circumstances.
Iain pointed out that current studies covered too short a period of time in a young person’s
life. For example, the Youth Cohort Study only covers 16 to 19 years olds and the British
Household Panel Survey (BHPS) only includes specific groups. In most cases samples of this
size and specificity are too small.
The presentation continued with a discussion of specific problems with sub-samples of key
policy interest. The Birth Cohort Studies was used as an example of how time lapse between
waves can be too long and how the topics covered for this and other surveys tend to be too
restrictive.
Iain suggested that the solution could be to commission a survey with extensive detail about
the parents and family circumstances at home and in work, in addition to assessing the
changes in transition of young people from compulsory schooling into the work place. It was
reasoned that this would then help to explain why some young people were delaying entering
the labour market.
Other problems identified included current surveys’ use of proxy data; this ‘second hand’
information could be a potential source of survey error. Iain also felt that there was a lack of
information collected by the surveys about the environment young people live in which could
have an effect on their delayed entry into the labour market. For example, in the United States
surveys collect data about the environment, the schools in different districts, families, and
social economical and geographical factors.
Finally, it was felt that current information provided on young people lacks robust
information for key groups, for example young people ‘at risk’ or those from ethnic
minorities.
So what are the needs of new research?
In summary, lain stated that new research needs to be similar to longitudinal studies which
look at transitions; that it should begin at an earlier age of young persons’ lives and carry on
for longer. In addition, new research should focus on specific groups of interest, take into
account family and environmental factors and should also be linked to administrative data.
Iain believed that the requirements of any new study should be to look at policy and how
it is likely to develop focusing on key factors and that ideally the study should be
longitudinally designed. He felt that such studies should allow young people to give
37
SMB 51 1/03
their own views on policy, in particular, what it is that is wanted, and what is actually
provided. This should include looking at whether young people are successful while in
education and when they have joined the labour market. New research should also enable
researchers to assess the effectiveness of policy and support its development.
Having advised using a longitudinal design, lain went on to state three specific advantages of
using a longitudinal method of research. Firstly, that it separates continuity and ‘churning’;
secondly, that it separates factors of influence and analysing causality and finally that
longitudinal methods are crucial where the sequencing of events or effects is key.
Iain described the development of a new study entitled the ‘Longitudinal Study of Young
People in England (LSYPE) currently taking place. The study will begin with young
persons aged 14 so that the data can be linked to administrative data that uses 14 year
olds as the starting age for courses. Annual interviews will be carried out until the age
of 25, with an additional interview at wave 1 with the parent/adult of each young
person.
The sample will be selected from English respondents only, with an initial set of 15,000
young persons. The target response rate at wave 1 is 75%. It will be a stratified sample
using deprivation as an indicator of risk of social exclusion. The sample will include an
ethnic boost and will be selected from either the Common Basic Data Set (CBDS), which
is derived from Pupil Level Annual Schools Census (PLASC) or a school based screening
survey. It would be preferable to use the CBDS for sampling but there are difficulties of
access/data protection for this data set. There is also concern that the key data may not
be fully verified and concern regarding over-sampling of ethnic minorities as, at present,
this data cannot be collected due to current ONS classification changes. An alternative to
using the CBDS to generate a sample would be to conduct a screening survey. The problem
with this approach is that it would require screening over 1,000 schools and heightens the
possibility of non-response by schools and individuals. However, a screening program
would be easier to implement and would allow for additional stratification and
disproportionate sampling.
The survey will require an ethnic minority sample boost of a minimum of 500 sample
members at wave 5, of Black Caribbean, Indian, Pakistani and Bangladeshi respondents.
The interview methods will be face-to-face at wave 1 with young persons and also interviews
with parents or responsible adults. There is a possibility of conducting a face-to-face
interview at wave 2 with the young persons. After which, data collection would take place
using telephone interviews and additional face-to-face interviews with key groups if
necessary, for example, if differential attrition is a possibility.
Researchers will attempt to link the survey data with administrative data about schools,
colleges and neighbourhoods using multi-level analysis. However, at present it is not possible
to merge individuals with colleges. It is planned that examination and training databases will
be linked further with individual level databases.
Iain emphasised that the survey is still in the development stage, involving scoping,
questionnaire development and testing. The development process started by consulting
with policy makers to see what they wanted to know. Department for Education and Skills
(DfES) spent several months in consultation with the National Centre for Social Research
to evaluate whether the study was feasible.
38
SMB 51 1/03
A dress rehearsal will be conducted to see if the survey works and following this, full funding
will be applied for. The target date for wave 1 is spring 2004.
Iain concluded the presentation by advocating the use of longitudinal methods, which he felt
provided excellent analytical opportunities. He pointed out that existing longitudinal surveys
do not meet the new needs of policy makers, that a new kind of study is needed and that this
is what LSYPE hopes to provide.
Interviewing children using audio-CASI (computer assisted self
interviewing)
Rebecca Gatward (ONS)
Rebecca Gatward began with an explanation of audio-CASI; describing how the respondent
hears questions and response categories through headphones and enters his or her own
answers into the laptop. This mode of data collection is often used for sensitive behaviourial
issues or when respondents have literacy problems. It can be utilised in conjunction with
Blaise (version 4.2), the standard questionnaire software used by Social Survey Division
(SSD).
Audio-CASI was first used by SSD during the pilot stage in a survey of the ‘Development
and Well-being of Children and Adolescents Looked After by Local Authorities.
Rebecca gave a brief description of the Survey, which is part of a series of surveys of
Children’s Mental Health, started in 1999. Results show 1 in 10 children aged 5 to 15 years
have a clinically recognisable mental disorder, which has a severe impact on the family. The
2001 survey sample was drawn from the Department of Health’s databases with permission
gained via local authorities for names and addresses to be passed onto SSD. The interview
was face-to-face with the current carer and an additional self questionnaire for children aged
11 to 15 years. A postal questionnaire was also sent to children’s teachers.
Why use audio-CASI?
The self-completion section of the survey contains a substantial set of sensitive questions; it
was believed that using audio-CASI would improve the reporting of sensitive behaviours. The
characteristics of the selected sample were more likely to include respondents with learning
difficulties and problems with concentration than those from a sample living in private
households. Research implies that audio-CASI is a suitable mode of administering surveys to
low literacy respondents and provides a way of increasing their level of attention and interest
in the interview.
Design of the audio-CASI questionnaire
Rebecca stated that the aim was to develop a questionnaire that could be easily used by young
people with learning difficulties, without appearing cumbersome or simplistic to young
people who could read. She pointed out that by adopting a step- by-step approach, it was
hoped that it would be possible to avoid alienating young persons and that they would feel
confident completing the section. The SSD interviewer was present during self-completion to
help the respondents and answer any questions.
39
SMB 51 1/03
The questionnaire consisted of 130 questions and six different response sets, including closed
questions with four response categories and open-ended questions requiring short text or
numerical responses to minimise changes in response. At the self-completion section of the
questionnaire, children were given the option of using CAST or CAPT (although it was hoped
that CAPT would only be used as a last resort). The interviewer then read a standard
introduction explaining how the audio-CASI instrument works. Interviewers were required to
demonstrate how to enter answers, move onto the next question and repeat the question. The
headphones were checked and the volume adjusted prior to completion and the first question
in the section asked respondents if the volume was OK. Respondents were encouraged to ask
for help if they needed it.
Screen layout
Rebecca discussed the importance of having an uncluttered screen so as not to distract
respondents. Only the response categories were displayed on-screen, with the question
identifier displayed for interviewers’ information. She discussed the usefulness of labelling
the navigator keys on the laptop for young people and suggested using paper stickers but to
bear in mind that some young people might be colour blind when selecting the colour of the
stickers.
During the Pilot, audio-guidance seemed sufficient and this aid helped to avoid the frustration
of repeating the instructions too often. The instructions were played on the first few questions
in the section and repeated after a change in response. New instructions were given when
respondents were expected to record a different type of response, for example, typing in a text
answer.
Recording the audio files
Rebecca described the process of recording the audio files. She explained how a colleague’s
voice was chosen making it easier to record any future audio files arid to make any essential
amendments. When using audio-CAST in surveys, Rebecca recommended that the colleague
selected should have interviewing experience and be familiar with the questionnaire. For this
particular survey, the audio files were recorded in a professional studio (130 questions,
90mins, approx. £200). The questions, response sets and instructions were all recorded as a
separate file/wave, the computer then picked up the relevant files. Using audio-CASI meant
that the questionnaire had to be finalised earlier than usual, allowing less flexibility for
change.
Other practicalities
The need to purchase screen wipes and headphones which are reasonably priced, robust and
compact.
Remembering to save the interview, before passing the laptop onto respondents.
Feedback from the pilot
All young persons in the sample used audio-CAST for the self-completion section of the
questionnaire. The self-completion section took 25 minutes.
40
SMB 51 1/03
Feedback was positive from the interviewers and respondents, the young people enjoyed
completing the audio-CASI section of the questionnaire and were able to do so independently.
Rebecca concluded her seminar saying that by using audio-CASI it was possible to obtain
information from respondents who otherwise might have refused to take part earlier on in the
interview.
Conducting cognitive interviews with children: an evaluation of
methods and techniques.
Kim Ursachi (ONS)
Kim Ursachi talked about the use of cognitive interviews to test questions with children
looked after in care, the (LAC) survey. She identified two areas for discussion, first asking
why researchers should conduct cognitive interviews and second, what special issues need to
be considered when the cohort is children.
Why conduct cognitive interviews?
Kim outlined briefly two reasons why researchers should conduct cognitive interviews, firstly
to test the reliability and validity of the survey question and secondly, to understand the
cognitive process behind how respondents answer questions. In conclusion, she felt it was
advantageous to conduct cognitive interviews as it is a quick method, cheap and improves
data quality.
The following 4 points cover Kim’s definition of cognitive interviewing:
It is a mental process respondents use to answer questions. For example, deciding what they
think the question is asking, making a judgement about the information to be retrieved and
then answering the question.
Cognitive interviewing is a one-to-one interview using a structured questionnaire and is often
referred to as a think aloud’ interview, as the respondent must discuss how they arrived at a
particular answer.
The interview can be concurrent or retrospective, and makes use of probing questions and
paraphrasing.
Cognitive interviewing is an in-depth interviewing technique.
What are the special issues to consider when the cohort are children?
There are various special issues that should be considered when interviewing children. Firstly,
it is important to explain to children who the client is. Secondly, it is necessary to keep
children focused and to establish some basic rules, for example allowing children to opt out of
answering sensitive questions if, at the first attempt, they feel uncomfortable answering a
particular question. Finally, it is important to build a rapport with children at the beginning so
that they feel comfortable with the interviewer. As respondents, children must feel secure
with interviewers and must understand the implications of what they disclose and that if
necessary, any information they disclose ma be passed on to the police.
Kim stressed the importance of speaking with children’s parents first, then telling both
parents and children that the interview will be taped and explaining the issue of
confidentiality, which Kim herself found easier to explain in terms of secrets.
41
SMB 51 1/03
It is also important to pay attention to the chi1dren language and cognitive abilities while
building a rapport. Kim suggested one way of doing this is was to focus on the child’s
development age and the language they are using, rather than the child’s chronological age.
Another important skill to develop involves trying to build in a sense of play when carrying
out interviews. During her own experience of cognitive testing with children, Kim used a
‘sentence finishing’ method to check understanding/probing if the child had not answered
previous questions. The key point was to tailor interviews to each child, bearing in mind that
the interview should not be any longer than one hour and 30 minutes.
For this particular set of cognitive interviews, children were given £10 as an incentive
payment before the start of the interview.
Kim concluded the presentation by mentioning the ethics of interviewing children. She
pointed out that it was difficult not being able to hug a child if they became distressed, and
that it was important to be clear and concise when talking to children to avoid any
misunderstandings. She stressed the necessity to be aware of changes in children behaviour,
for example withdrawing eye contact and evasiveness, and to remember to probe why this has
occurred. Finally, she pointed out that it is the responsibility of the interviewer to bring the
child out of the interview situation and leave the child feeling happy and content.
Scoping study and development work for a new cohort study of
young people (14-25)
John Bynner (Director of Centre for Longitudinal Studies at the Bedford Group
for Lifecourse and statistical studies, Institute of Education)
The Centre for Longitudinal Studies is responsible for three of the most prominent birth
cohort studies in Britain. The first of these, the National Child Development Study began in
1958, secondly, the British Cohort Study which was derived in 1970, and more recently, the
Centre has developed the Millennium Cohort Study. This new cohort study of young people
(14-25) fits in with the tradition of cohort studies undertaken by the Centre for Longitudinal
Studies and follows on from the Economic and Social Research Council’s 16-19 Initiative, on
which John Bynner acted as co-ordinator. The key strength of this new study is its’
recognition that the transition from youth to the labour market needs to be considered over a
far longer period than just 16-19 years of age.
The study has two aims, firstly to find out how young people achieve the transition from
youth into the labour market, in particular, looking at the routes that they take, be it by
academic means, vocational means or by no formal route. John commented that a third of
young people still leave school at 16 years old and that half of these have no qualifications.
The second aim of the study is to investigate social cohesion and integration, the key issue
being how the successful integration of all young people can be ensured. This second aim
necessitates the collection of a broad range of variables at levels other than those at the
individual and family level, for example, information about the community and wider
geographical environment in which young persons live.
The main study is due to begin in March 2004 and the sample will comprise young
people born in 1990. It will look at the experiences of these young people, who,
having grown up in the I will experience many of the changes of the modern world
and will be entering a labour market that is radically altered from that of the past.
Consequently, this cohort are of huge interest in terms of policy formation, and the data
42
SMB 51 1/03
collected will be used to help answer questions about what options the government can
make available to young people to ensure a smooth entry into adult life. A second part of the
study will look at the background of changing family life.
The timing of the research, with the first stage of fieldwork not due to start until March 2004,
affords the team a long period of time for development. Consequently, three months have
been allocated to the scoping study. During this time, an academic advisory group will be
established; contact will be made with experts within government who have policy interests
and those who hold keys to administrative information which can be used for sampling and
data linkage. In addition, about 80 academic experts from both the UK and the rest of the
world will be contacted so that a broad consensus of issues which need to be covered can be
realised.
A variety of specialist groups will also be involved in the study, such as an Ethnic Minority
Advisory Group, a Survey Methodology Group and sampling done in conjunction with the
University of Essex.
To achieve the aims of the research, it is intended that data will be collected on education and
qualifications, entry into the labour market, progress in employment, social consequences,
family, and social, psychological and educational development. The team is currently trying
to determine the most appropriate way of collecting each type of data.
It is intended that the exploratory fieldwork will utilise both quantitative and qualitative
research methods where necessary in order to ensure that the instrument works before it goes
live. This means that as well as extensive piloting of the instrument, the team will also use
cognitive and exploratory interviewing. A period of approximately seven months has been
assigned to questionnaire design, this will involve piloting of questions specialised pilots and
exploratory studies assessing, for example, what language to use, how to use it etc Following
completion of this extensive period of questionnaire development there will be a dress
rehearsal using the finalised instrument.
It is intended that the final sample frame will be based primarily on schools, but will also
look at individuals within schools. The main sample of the survey will comprise around
15,000 individuals this will be supplemented by an ethnic minority boost of about 5,000.
The sample will also be designed to allow analysis at other levels including communities,
housing estates, and the family. This will necessitate clustering in the sample design although
John stressed that this type of sample design will not compromise the key priority of the
study to produce robust population estimates.
Child Maltreatment
Sue Brooker (BMRB) and Pat Cawson (NSPCC)
Sue Brooker began the presentation with a discussion on public perception of child
maltreatment. The public image is almost entirely focussed on sensational media
coverage, high profile cases and themes such as stranger danger, paedophiles and the
abduction of children. Underlying this media coverage is the idea that our society is much
more dangerous for children than was the case in the past. The reality is that 70-80
children are killed each year (although this is likely to be an underestimate) and that
strangers kill just 5-10 of these children. However, these figures have been static for
decades.
43
SMB 51 1/03
This research was commissioned by the NSPCC, who intend that it will provide a benchmark
against which the progress of the ‘Full Stop Campaign’ can be measured. The Full Stop
Campaign has the ambitious objective of ending child abuse within a generation. Current
official figures on abuse were deemed unsuitable because they are known to considerably
under represent the true level. Other sources of data were also ruled out due to the general
lack of consistency in methods used and topics covered.
The study interviewed people from a single generation, those aged. between 18-24 years old.
The reasons for choosing this age group were:
• Childhood has been completed but is still recent enough to allow reasonable recollection of
experiences.
• It is unlikely that 18-24 year olds will have experienced adult traumas yet and so they are
therefore in a good position to isolate childhood experiences and their effects.
• Using people aged 18-24 does not require parental permission.
A study of this type presented several challenges to the researchers, amongst these were the
extreme sensitivity of the subject, the complexity of potential experiences and the need for
respondents to feel complete trust. These issues made the design of the questionnaire critical;
the breadth of experiences under consideration meant that the questionnaire would by
necessity be complex, and the need to for sensitivity and to gain trust required that the
questions be carefully constructed and ordered.
The final questionnaire used a tried and tested method of increasing question sensitivity. This
involved beginning the interview with questions about attitudes and family background before
asking about personal experiences. The questions aimed at identifying potential maltreatment
used a modular approach with a long series of questions about specific experiences to which
the respondent could reply ‘Yes’ or ‘No’. Previous research has shown that this approach
yields more accurate figures than asking about broad categories.
When a respondent answered positively to a question on potentially abusive
treatment, they were routed to a set of questions which aimed to establish:
• How often the abuse occurred,
• Who perpetrated it,
• Whether the effects lasted until the next day,
• Whether serious injuries were incurred,
• Whether it was reported to the authorities.
• Where it was thought that abuse might have occurred, the respondent was asked whether
they now regarded it as abuse. This enabled the researchers to compare “objectively defined
abuse” with the respondent’s personal interpretation.
The complexity of the question loops meant that CAPI (Computer Assisted Personal
Interviewing), was essential. The use of CAPT also enables self-completion by the respondent
directly on to the laptop computer; this is known to improve respondent’s perception of
confidentiality and increase the reliability of their answers.
44
SMB 51 1/03
Extensive piloting of the questionnaire was undertaken. This included a standard pilot that
used a sample of the general population, and a targeted pilot of people in hostels for the
homeless. It was anticipated that the incidence of child maltreatment in the general population
would be so low that the use of just a standard pilot would not test the questionnaire
thoroughly, hence the need for a targeted pilot as well. After each pilot interview, a short
qualitative in-depth interview was undertaken in order to establish the feelings of the
respondents towards the subject matter. Issues covered included:
• The sensitivity of the questions.
• The perceived intrusiveness of the questions.
• Whether respondents would complete the survey in their own home.
• Whether respondents understood why this survey was being conducted.
• Whether respondents would be willing to take part without an incentive.
The research findings
Sue briefly discussed the findings of the research in relation to three main areas:
physical abuse, emotional maltreatment and sexual abuse; areas not covered were bullying
and neglect. The total number of respondents in the survey sample was approximately 2,800
with a response rate of 69%.
A key challenge in interpreting the results was defining the various forms of abuse and
maltreatment. With respect to physical abuse the first thing that needed to be established
was when physical force or ‘discipline’ became abusive. Physical abuse was defined as
violent treatment from parents experienced by children regularly over the years, experienced
less often but leading to lasting effects on more than half the occasions, or if on any
occasion this form of treatment led to physical injuries. On this basis, the research found
that 7% of young people were physically abused during their childhood at the hands of their
parents or carers. A further 14% were treated undesirably (in a similar way, but for a shorter
period or with less frequency), and 3% reported behaviour that could be said to give cause
for concern. Altogether, almost a quarter of young persons reported experiencing some form
of maltreatment at the hands of their parents or carers that could be deemed to be
unacceptable.
The area of emotional or psychological maltreatment is the least well researched of all aspects
of child abuse with no existing consensus on its definition and nature. Yet there is growing
acknowledgement that this is an extremely damaging form of child abuse, and one that is
often an indicator of other forms of maltreatment. It is difficult to define because it covers
such a wide range of behaviours, many of them targeted directly at the individual child with
adults focussing on what they know will be most hurtful to them.
It was therefore decided that emotional maltreatment would need to cover a wide spectrum of
behaviours. These different behaviours were grouped into 7 broad categories:
• Psychological control and domination,
• Psycho/physical control and domination,
• Humiliation,
45
SMB 51 1/03
• Deliberate withdrawal of affection,
• Parents showing antipathy,
• Parents terrorising the child,
• Parents attacking the child by proxy.
In total, there were 32 items spanning the 7 categories included in the area of emotional
abuse. In each of the 7 categories, respondents were given a score of 0 if they had experienced
none of these behaviours, 1 if they had experienced a single item and 2 if they had
experienced 2 or more.
In order to ultimately define emotional maltreatment a relatively arbitrary total score had to
be selected. A score of 7 or more was chosen as this indicated that young people had
experienced some adverse treatment in at least four of the seven dimensions. Using this
criteria, 6% of young persons could be said to have been emotionally maltreated, this being
proportionally higher amongst girls (8%), than boys (4%).
Sue concluded the presentation with a discussion of the findings about sexual abuse. Sexual
abuse was said to have occurred if any sexual activities had taken place with a parent or carer
and if the child (aged 16 years old or younger) had any of these experiences against their
will. Sexual activities experienced by children aged 12 years old or younger with someone 5
years older or more, and the relationship of the abuser to the abused was also investigated.
On this basis, 1% of young persons had been abused by parents or carers, 3% by another
relative, 2% by a stranger or someone they had just met and just under one in ten had been
abused by other known people. In total, just over one in ten young persons could be
identified as having been sexually abused as children. Despite this, only a quarter of them had
told anyone about their abusive experiences at the time - if they did tell anyone, it was usually
a friend but sometimes a family member and on rare occasions the police or someone they
regarded as a professional.
46
SMB 51 1/03
Judith Bell and Trish McOrmond
Quality Issues in Social Surveys (QUISS) Seminar
31st October 2002
Quality Issues in Longitudinal Surveys
The twelfth in the series of QUISS seminars organised by ONS was based around quality
issues in longitudinal surveys. Longitudinal surveys are a valuable tool in measuring
change over time, and can be used in conjunction with other information as a predictor for
future events. Longitudinal surveys often suffer from attrition which diminishes their
precision. These issues, and how longitudinal surveys should be organised and analysed
formed a basis for the seminar. Siobhan Carey from the Social Survey Division chaired the
event.
Use of the Longitudinal Study in combination with General
Household Survey and National Fertility Statistics.
Michael Rendall, Office For National Statistics
In his presentation, Michael described how the General Household Survey and National
Fertility Statistics are being used in conjunction with the ONS Longitudinal Study in an
analysis that may aid in the prediction of “completed fertility” in the United Kingdom. This
refers to the total number of children born to a woman throughout her reproductive age. In
their present study, Michael and his colleague Steve Smallwood (also at Office for National
Statistics) estimated second and subsequent births to women born in England and Wales
between 1954 and 1958.
Michael began by explaining the growing difficulties in predicting completed fertility
statistics. The Period Total Fertility Rate (PTFR) is the usual statistical measure of completed
fertility and is determined by the sum of fertility rates at all reproductive ages. Thus the PTFR
serves as a means of predicting the average number of children a woman will have in her
lifetime, just as Life Expectancy serves as a means of predicting the average length of a
person’s life. The PTFR however is susceptible to changes in age patterns of fertility and has
thus, in the past decade become a poor predictor of completed fertility as fertility age pattems
have changed dramatically over this period. Prediction of completed fertility depends
increasingly upon being able to predict fertility at later ages, as women giving birth at later
ages account for a continually expanding proportion of completed fertility. The current
method of predicting completed fertility is based on age-specific fertility rates (ASFRs).
Michael discussed several potential methods of improving upon this. More demographic
variables could be added such as parity (the number of children a woman has had so far),
intervals between births and changes in mean age. In addition, the use of socio-economic
variables would make theoretical sense as delayed fertility is known to be related to a
woman’s alternative opportunities such as higher education and career advancement. These
same opportunities increase a woman’s economic resources and therefore the likelihood of
‘catching up’ once she has begun a family. In their present study, Michael and his colleague
have combined demographic and socio-econornic variables.
47
SMB 51 1/03
Estimation of fertility rates using the socio-economic variables mentioned and parity is a
difficult process, since such data is not available from the regular birth- registration estimates
of fertility. They are obtainable as survey estimates of fertility rates by age and socioeconomic variables but have sampling errors that are typically too great for prediction
purposes. This is particularly true for comparatively infrequent events such as further
childbearing (parity progression) following a late start to childbearing. Late childbirth in the
early cohorts was rarer.
Michael then described how his study complemented the ONS Longitudinal Study, with the
General Household Survey and national ASFRs from birth registration and population
estimates. Using all 3 sources of data it is possible to estimate the chance of having a second
and higher order birth in any given year after the age of 25, by current age, number of
previous births, duration since previous birth, and whether the woman has a higher education
qualification. The parity progression ratio (PPR), which refers to the proportion of women
having a next birth, can also be predicted, when taken from the first to the second parity and
the second to the next parity, by age at previous birth and by the presence or absence of a
higher education qualification.
The ONS Longitudinal Study is a valuable resource. It integrates an immense sample (one
percent of the population of England and Wales) with socio-economic variables recorded
at 3 previous censuses (1971, 1981 and 1991). The study also links a woman’s birth
records, thus enabling the measurement of birth orders to that woman i.e. the number of
births so far. However, this approach is not infallible as approximately 10% of births are
not successfully linked, leading to downwardly biased estimates of parity progression
(further childbearing), which in turn leads to the distortion of related estimates such as
the probability of further childbearing since a previous birth. For instance if a first and
third birth are linked but the second is missed out the interval between consecutive
births seems longer. This gives a false impression of less parity progression at shorter
durations and more at longer durations. As a first step towards correcting these biases
the study supplemented the ONS Longitudinal Study with the General Household
Survey and Population ASFRs. The 1998 and 2000 General Household Surveys were
compared to the Longitudinal Study’s annual birth probabilities by parity, time since
previous birth, age and education. The predicted ASFRs from the Longitudinal Study
were adjusted up to conform to the population ASFRs under the assumption that these are
not biased.
Michael gave the results of the comparisons between the Longitudinal Study (LS) and
General Household Study (GHS) by age, parity, birth interval and qualification. There is no
discernible difference in the pattern of older ages at first, second and third birth when
measured with the LS data and when measured with the GHS data. A distinct pattern of
lower and later parity progression in the LS is consistent with the non-linkage bias. This
pattern is found for women of both age groups who have had a previous birth (25-30 and
31-37 years) and for both education groups. As this pattern is found when looking at
women with and without higher educational qualifications and women having previous births
at a younger or older age, neither educational qualification level nor age at previous birth
has any clear relationship to the non-linkage bias caused by incomplete birth records (see
above) in the Longitudinal Study. Therefore there is no requirement to correct for age and
education.
The annual probability of having another child decreases with age at previous birth
and increases with youth at previous birth: This is the age effect. There is also an
48
SMB 51 1/03
educational effect; as the age of the mother increases so too does the proportion of mothers
who have a higher qualification, at both first and higher order births. The annual chance of
giving birth again increases with education with oldçr mothers being more likely to have
another child if they have a higher education qualification. The older the mother at
previous birth, the larger the effect education has. The education effect is greater for second
births than for third and higher order births. However, the effect of age at previous birth
(age effect) is stronger than the education effect. In other words, the decrease in
probability of a further childbirth when delaying childbearing is greater than the increase in
probability associated with higher education. Older women with higher educations are more
likely to give birth than older women without them. However, older women are less likely
to give birth than younger women whether or not they have a higher education
qualification. The net effect is that childbearing in later years results in smaller eventual
families.
In conclusion, Michael stated that supplementing demographic variables with socio economic
variables should significantly improve forecasts of completed fertility. Use of the
Longitudinal Study in studies of fertility requires correction for non-linkage by parity and
birth interval, but not by age or socio-economic variables.
Attrition: How much does it matter?
Ian Plewis, Bedford Group for Longitudinal and Statistical Studies, Institute of
Education, University of London
Ian Plewis’ presentation focused on longitudinal surveys, concentrating on the effects of
attrition and nonresponse patterns that should be considered during analysis.
Ian began by talking about the nature of attrition. He stressed attrition was an inevitable
occurrence in longitudinal studies resulting in a progressively smaller sample and a decrease
in precision. In addition attrition may increase the potential for bias, which intensifies with
what he termed non-ignorable attrition’. Non ignorable attrition occurs when those subjects
lost from the study would have exhibited different patterns of change from those who
remain, irrespective of observed differences between the two groups before the point of
dropping out. Analysing differences between respondents who remain in the study and
those who drop out is problematic, since only differences before respondents leave the
study can be analysed. Further, it is impossible to predict if these differences would have
shown different patterns of change. Ian reminded us that longitudinal studies also suffer
from first wave non-response in the same way that cross-sectional studies do. Attrition is
therefore not a longitudinal survey’s alternative to initial non-response but an additional
consideration.
Any analysis of attrition is dependent on a clear and specific definition of the dynamic
population. Ian used the Millennium Cohort Study (MCS) to illustrate the point. The MCS
defines the population of children in England and Wales as all children born between 1St
September 2000 and 31 August 2001, alive and living in England and Wales at nine
months old. The population is defined over time as including those who remain in the UK at
the chosen time of measurement. As people move in and out of the UK they move in and
out of the population and the sample. Any analysis of the data therefore needs to take
account of the changing nature of the population as well as the changing nature of the
sample.
49
SMB 51 1/03
Ian dispelled the assumption that once respondents missed a wave they dropped out of a study
altogether. He used the 1958 cohort (NCDS) to give an indication of the extent of movement
in and out of a survey that occurs over time in a longitudinal study. Thirty percent, a
substantial proportion of the sample, did not exhibit a monotonic pattern of response. Patterns
of attrition can be examined by looking at the risk of non-response as people in a cohort get
older. When the 1958 sample consisted of children during waves 1,2 and 3, the risk of non
response was very low. As the age of the sample increased into early adulthood with waves 4
and 5 the risk of becoming a non-respondent increased quite sharply. When the cohort entered
its forties during wave 6 the respondents were less likely to drop out of the study and so the
risk of losing them diminished. Subsequently risks and patterns of non-response need to be
considered during analysis.
A few studies have investigated attrition at the analysis stage. Ian referred to the Fitzgerald et
al’s Michigan Panel Study of Income (PS ID). This began in 1968 and by 1989 fifty percent
of the sample had been lost. Attrition was analysed in terms of both fixed characteristics and
changing properties, and was observed to be highly selective concentrating among lower
socio-economic status individuals. (NCDS data analysis supported these findings). Analysis
of changing properties found unstable earnings, marriages and migration patterns were also
associated with attrition. However, although such characteristics were associated with
attrition they explained very little. The selection that takes place is moderated by regressionto-the-mean effects from selection on transitory components which fade over time. This
suggests that even an enormous amount of attrition has not seriously distorted the ability of
the PSID to be representative. Furthermore even though the attrition is not random there is
considerable evidence that the cross-sectional sample is still representative and the efficacy
remains roughly intact.
Ian went on to give another example of a longitudinal study in which the effects of attrition
were taken into account and found to be negligible. He described how Burkam and Lee
(1988) examined the monotonic and non-monotonic participation patterns across four waves
of data in The High School and Beyond Longitudinal Study. The estimated effects on
parameter estimates in regression models suggested that particular demographic effects would
be biased due to sample attrition. However these effects were found to be neither pervasive
nor consistent and added support to the theory that attrition does not necessarily have any
serious effect.
Ian emphasised that although some studies have analysed attrition and found it to have only
slight effects further study is necessary. Attrition is an important phenomenon worthy of
investigation and cannot be ignored on the basis of a few case studies. He asserted that
attrition may eventually be disregarded in high quality longitudinal surveys that reduce
bias by putting a lot of effort into finding people ove r time, however, he stressed a lot
more evidence on attrition would need to be accumulated before this position was
reached. In the meantime attrition should be analysed and possibly corrected for. Several
methods were proposed. Data could be weighted for non-response (a method more suited
to cross-sectional surveys). Alternatively various imputation methods, specifically multiple
imputations could be used. This being particularly appropriate in situations of nonmonotonic nonresponse since information from both before and after missing a wave
could be employed to fill in the gaps. The final method of dealing with attrition
mentioned was the joint modelling of the attrition and the substantive processes of
interest.
50
SMB 51 1/03
Measuring Change in Labour Force Panel Surveys
Ian Knight, (ONS)
Labour force surveys measure the key features of the labour market but where they are
conducted on a regular and frequent basis their results are also used to provide indicators of
changes in national economic health. Advanced economies like the USA, Canada, Australia
and the UK have been using their labour force surveys like this for many years and now the
transitional economies are increasingly developing surveys for this purpose.
The focus on measurement of change affects all areas of labour force surveys, from fieldwork
to design and analysis, particularly in the adoption of panel designs. Ian’s presentation
focussed on challenges faced by labour force surveys using the Office for National Statistics
(ONS) Labour force Survey (LFS) as an example.
Issues in Survey Design
One problem with measuring change in these indicators is that the change can often be less
than the sampling error. It is therefore necessary to develop a methodology that provides a
reliable estimate, and one way of achieving this is to interview the same sample of people
over a short period of time.
Most labour force surveys have been designed to measure change on a monthly basis with the
same proportion of households entering and leaving the sample each month. The proportion
entering or leaving the sample each month varies between surveys and can range between
one-fifth and one-eighth of the overall sample. This approach prevents degradation in the of
the samples representation of the population; is as fixed as possible in the short term; and
changes completely over time.
The monthly (or in the UK quarterly) design allows for important economic changes to be
captured at the speed at which they occur. Such measures often indicate economic growth or
decline that may not be apparent, and are regarded as an alternative measure to key economic
variables gathered from administrative data. In addition the availability of such data allows
policy initiatives to be more responsive and have greater effect.
When the main elements of the UK survey were being designed in 1983, these features were
taken into consideration in assessing whether a monthly or quarterly approach should be
adopted. The pay off of a monthly survey was not thought to be worth the associated increase
in cost and a quarterly approach was adopted. (Indeed, when the decision was made to move
to a monthly survey in Australia there was much discussion about whether it as needed that
frequently.) The UK LFS was also required to meet Eurostat specifications to measure year
on year change, and the proposed quarterly design provided a 20 per cent overlap with the
first and fifth waves in different years.
Since the British LFS is based on a single stage sample design over each quarter. It
covers every part of the country, and it is currently being used to measure local change.
A boost sample across Wales and England (and extended to Scotland next year) has
been implemented to ensure each local education authority has a similar achieved
sample size. Waves 1 and 5 of the main LFS are combines with this boost sample to
allow changes in local area measures to be examined year on year. Respondents in the
boost are interviewed once a year over four years and are spread evenly across 52
weeks.
51
SMB 51 1/03
Since the British LFS interviews in every week of the year (unlike the labour force surveys in
most countries) problems associated with periodicity have been removed. The continuous
data collection means that any 13-week period could be considered a separate representative
sample, and compared to the 13-week period preceding it.
Patterns in labour markets emerge over time, and include seasonal variations not evident or
understandable without several years of data. These seasonal variations introduce
considerable short term volatility (for example the increase in people working part time in
shops during the Christmas season), and should not be interpreted as change in the indicator.
Consequently the main indicators have to be seasonally adjusted before publication.
Fieldwork
The LFS sample is a systematic sample of the whole country using the Postcode Address File
(PAF), spread in a geographically uniform way. The country is split into 220 interviewing
areas and, once selected, the sample is divided amongst these areas. Each area has 13 stints,
randomly assigned to the 13 weeks of the quarter and provides one weeks work to
interviewers. This approach results in one-thirteenth of the sample being interviewed each
week in a random sample of areas and over a consecutive 13 week period.
When measuring change it is necessary to ensure the changes noted reflect a real change in
the labour market and are not artefacts of fieldwork. However when we ask the same
respondent the same question at two moments in time there is a risk that (s)he may describe
his/her situation in slightly different words leading to a different classification of the answers
when the situation itself has not changed. This can be exacerbated when the subject of the
question answers it on one occasion and a proxy respondent answers it on the next occasion.
To avoid such spurious indicators of change the British LFS uses the dependent questioning
technique where the respondent is told how the question was answered last time and asked if
the situation has changed. This is used on key variables (such as job change) and works well
provided it is not overused.
Discontinuities in panel data
To measure change it is imperative to keep anything that may effect the results over time as
consistent as possible. In the short term it is convenient, and reasonable, to ensure that
nothing that is part of the measurement process changes over time. However, in the long term
there will be situations that present possible discontinuities in the data.
Discontinuities may grow out of grossing because the population figures used as control totals
(most notably the Census) are generally only available in arrears to the relative period.
Population control totals used in grossing are based on the data available at that time for the
period covered, but these are only model based projections. When the more reliable estimates
become available, the user databases need to be reissued for each quarter back to when the
last actual estimates were available, published tables have to be revised, and users informed
of the changes.
Fortunately, most population projections have not been significantly different from the
actual estimates when they become available, but the 2001 census may prove to be an
exception to this generalisation. Coding changes, for example the revised Standard
Occupational Codes issued just before each Census can also result in discontinuities as the
new classification systems cannot be mapped back to those originally used to provide a
revised back series.
52
SMB 51 1/03
Attrition and weighting
Panel surveys are also prone to bias caused by attrition since those respondents dropping Out
of the survey after the first interview may have different characteristics to those that remain in
the sample. When respondents are first interviewed for the LFS, economically active
respondents are under represented. On subsequent waves economically inactive respondents
have the highest attrition, which may in some way act as a counter balance. Using Census
data for the whole of the initially selected sample it is in fact possible to determine who is
more likely to be under represented at first entry. These are generally young, unmarried
individuals who work long hours, and are often not home when the interviewer calls. This is
partly compensated by the weighting element of the grossing procedures but that only covers
variations by age sex and locality. Traditionally we have undertaken this weighting as if it
involves a single sample but in fact each field period starts with an independent cohort with
its own sampling error, so we are planning to weight each wave separately in future.
If this is done wave specific questions can be asked (which reduce respondent burden) while
allowing the total number of questions asked to increase.
Panel conditioning
The literature of market research panels suggests that attitudes and buying behaviour of fast
moving consumer goods can be affected by panel membership. Research on panels has
indicated that some change in the middle waves (for example, waves 2 and 3) may exist,
though over the term of their participation respondents would revert to normal behaviour.
This raises questions like: Do people change their behaviour when asked the same question
over a period of time?
However it has been reassuring to find that there is surprisingly little change in the key
indicators for most LFS respondents, quarter on quarter, suggesting that panel conditioning
for this subject matter is of little concern.
53
SMB 51 1/03
Table 1 A comparison of Labour Force Surveys
54
SMB 51 1/03
Quality Initiatives for Academic Longitudinal Surveys
Peter Lynn, UK Longitudinal Studies Centre, University of Essex
The focus of Peter Lynn’s presentation was on the development of an appropriate quality
framework for evaluating longitudinal surveys, which could be used in the actual production
of quality profiles for specific surveys. A quality profile it is envisaged would be a document
or collection of documents that provides all the information a user would need about the
quality of a survey. The quality framework therefore needs to have a strong focus on
determining the users and their needs.
Context
Much of ESRC’ s funding is dispersed through one of two channels:
• Supporting research resources: making available data sets to academics, archiving data,
documenting it, providing metadata, training and tools, etc
• Research: analysing data and making substantive conclusions
The development of the Quality Profile Template is funded by the research resources board
and is an opportunity to improve the quality of longitudinal survey resources across the
academic world and the broader community. This project is being undertaken in consultation
with the Government Longitudinal Co-ordination Group as similar issues are faced by both
groups.
ESRC has an existing commitment to a number of (and spends a large proportion of its funds
on) longitudinal surveys. Due to this vested interest, long term strategic thinking about the
ESRC’s commitment to these surveys, and how to fund and structure them, is necessary, as is
the development of the quality initiative. The National Longitudinal Strategy Committee
(NLSC), working to the UK Longitudinal Studies Centre and ESRC, was created to represent
ESRC’s interest in these matters and provide the necessary academic background and skills
for strategic thinking.
Quality Framework
The first stage of the project was to develop a quality framework appropriate for all aspects
that were viewed critical by the ESRC. The framework was intended to inform the
development of quality profiles and therefore needs to be appropriate for multiple quality
discussions within the ESRC. For example, providing structure for considering quality when
assessing grant applications and determining what to ask grant applicants.
As most organisations already have a quality framework and guidelines, the NLSC began by
assessing the usual criteria of survey quality that are considered in other NSI’s and in
Eurostat. The criteria that make up the quality framework are therefore:
• Relevance: the longer a survey continues, the more likely that information collected
originally may no longer be relevant to the current social or political climates.
• Accuracy
• Timeliness
55
SMB 51 1/03
• Accessibility
• Interpretability
• Coherence
• Costs: a slightly contentious component. Many people think that cost is not a component of
quality surveys and that quality and cost are measured against each other. Others believe you
cannot consider quality without considering cost and that these should be considered in
tandem and therefore cost must be in the framework. As long as cost is considered at some
point, which is why it is included, that is sufficient.
When developing the framework, there were unique features of longitudinal surveys that
needed to be considered. For example, the many different population definitions of interest
and how these change as the population changes. In a survey with N waves of data collection,
any possible combination of waves (N waves — 1) is a potential dataset of interest. Ten
waves of data results in 1023 possible combinations of waves to analyse and 1023 different
target populations before considering subdomains.
Another unique feature was technology which can change dramatically and have implications
for planning and implementing longitudinal surveys. Longitudinal surveys need to ensure that
some or all of the data collected in early waves is not lost as technological changes occur, and
that the survey needs to be adaptable. In tandem with technological changes staff or
organisational responsibility changes may have a severe impact if the survey is not well
documented.
Finally, attrition was a key point of discussion for the seminar overall, and for any discussion
about quality of longitudinal surveys, because the impact on longitudinal surveys is more
complex than in cross-sectional surveys. Fairly standard item non- response, wherein two to
three per cent do not provide an answer in a cross-sectional survey, has a much larger impact
in a longitudinal survey. In a longitudinal survey, two to three per cent of respondents failing
to provide a usable answer over 10 waves may mean that about 20 per cent of the sample is
missing.
Quality Profile
The quality framework was developed, taking into account the aforementioned issues, and,
although the issues may be generic, the quality framework is specific to longitudinal surveys.
Using the framework, a template quality profile was the next development challenge.
The profile, ideally, is to provide information on the fitness for purpose of the survey. To
make this determination, it is necessary to know who the users are, for example, the
funding body; survey designers; implementers and analysts. The purpose of the survey
must also be known to plan subsequent waves and ensure that the approach fits its
purpose.
As there is a wide range of information needed, the template covers a variety of quality
issues. Such a broad variety of quality issues allows the template to cater to the needs of a
large user base. While it is unlikely that any one user will be interested in all available quality
indicators, all the information will be of interest to some of the users.
56
SMB 51 1/03
Next Step
The next step in the development of the quality framework is the finalising of the quality
profile for the BHPS and a need to understand the future of quality profiles, such as how to
update the profile over time. The current thinking is that the profiles will be primarily webbased documents that regularly self-update and have a clear audit trail.
The ESRC is committed to developing quality profiles for other existing ESRC longitudinal
surveys as soon as possible, however the mechanisms for this are not yet clear. For the future,
it is being considered that the development of the quality profile is undertaken at, and will be
a requirement of, the funding stage.
The underlying idea for the quality profile is that it will be an objective statement of fact
regarding the multiple dimensions of quality and will have varying degrees of importance to
different people using the survey. Subsequently, when the profile exists, the information will
be used by the ESRC to evaluate the survey and make objective observations, with other users
doing this, informally, all the time when they assess the data.
The profiles are not cast in stone and the NLSC is still looking for, and getting evaluations
and feedback from, other people. It is regarded as important that quality profiles are extended
to other surveys, both longitudinal and cross-sectional.
The necessary information on developing a quality profile, the actual draft Quality Profile
framework for the BHPS, the overall draft national strategy, template, and further information
is available on the website: www.iser.essex.ac.uk/natstrat/
Conclusion
The issues that are faced by longitudinal surveys need careful consideration as the surveys are
developed to ensure that the best quality data is collected over the long term. Monotonic and
non-monotonic patterns of response, which have little bearing in cross-sectional surveys, do
have a large impact in longitudinal surveys. In addition, survey design should take into
consideration possible changes, technological and staff, as well as social, political, and
economic, that may have an impact in the long term.
Finally, questions and the reasons for inclusion of various topics should be well documented
to ensure that the information remains appropriate over time. The quality framework that the
NLSC is developing should go a long way to encouraging best practice amongst longitudinal
surveys however, as Siobhan Carey noted at the end: “The only thing harder than getting a
question onto the Census is getting one off.”
57
SMB 51 1/03
The National Statistics Methodology Series
This new series, aimed at disseminating National Statistics methodology quickly and easily,
comprises of monographs with a substantial methodological interest produced by members of
the Government Statistical Service.
Currently available
1.
Software to weight and gross survey data, Dave Elliot.
2.
Report of Task Force on Seasonal Adjustment
3.
Report of the Task Force on Imputation
4.
Report of the Task Force on Disclosure
5.
Gross Domestic Product: Output methodological guide, Peter Sharp
6.
Interpolating annual data to monthly or quarterly data, Michael Baxter
7.
Sample design options for an integrated household survey, Dave Elliot and Jeremy
Barton
8.
Evaluating non-response on household surveys, Kate Foster
9.
Reducing statistical burdens on business, Andrew Machin
10. Statistics on Trade in Goods, David Ruffles
11. The 1997 UK pilot of the Eurostat Time Use Survey, Patrick Sturgis and Peter Lynn
12. Monthly statistics on Public Sector Finances, Jeff Golland, David Savage, Tim Pike and
Stephen Knight
13. A review of sample attrition and representativeness in three longitudinal surveys, Gad
Nathan
14. Measuring and Improving Data Quality, Vera Ruddock
15. Gross Domestic Product: Output Approach, Peter Sharp
16. Report of the Task Force on Weighting and Estimation, Dave Elliot
17. Methodological Issues in the Production and Analysis of Longitudinal Data from the
Labour Force Survey, P S Clarke and P F Tate (Winter 1999)
18. Comparisons of income data between the Family Expenditure Survey and the Family
Resources Survey, Margaret Frosztega and the Households Below Average Income team
(February 2000)
19. European Community Household Panel: Robustness Assessment Report for United
Kingdom Income Data, Waves 1 and 2 (March 2000)
20. Producer Price Indices: Principles and Procedures, Ian Richardson (March 2000)
58
SMB 51 1/03
21. Variance Estimation for Labour Force Survey Estimates of Level and Change, D J
Holmes and C J Skinner (May 2000)
22. Longitudinal Data for Policy Analysis, Michael White, Joan Payne and Jane Lakey (May
2000)
23. Report on GSS Survey Activity in 1997, Office for National Statistics, Survey Control
Unit. (December 2000)
24. Obtaining information about drinking through surveys of the general population, Eilen
Goddard (January 2001)
25. Methods for Automatic Record Matching and Linkage and their Use in National
Statistics, Leicester Gill (July 2001)
26. Designing surveys using variances calculated from Census data, Sharon Bruce, Charles
Lound, Dave Elliot (March 2001)
27. Report on GSS Survey Activity in 1998, ONS Survey Control Unit (September 2001)
28. Evaluation Criteria for Statistical Editing and Imputation, Ray Chambers (September
2001)
29. Report on GSS Survey Activity in 1 999,ONS Survey Control Unit (October 2001).
30. The Value and feasibility of a national survey of drug use among adults in the United
Kingdom, Eileen Goddard. (April 2002).
31. Report on GSS Survey Activity in 2000, Quality Centre. (April 2002)
32. Gross Domestic Product: Output Approach (Gross Value Added) – Revised
59
SMB 51 1/03
1. Centre for Applied Social Surveys
Web Survey Design
17-18 March 2003
Survey Data Analysis III Multivariate Data Analysis
2-4 April 2003
Survey Sampling
7-9 April 2003
Classifying and Scoring People in Surveys
8-9 May 2003
Telephone Survey Design
27-29 May 2003
For further information please contact:
Jane Schofield
Department of Social Statistics
Building 58
University of Southampton
Southampton S017 1BJ
Tel: 02380 593048
Fax
02380593846
2. Royal Statistical Society Seminars
Assessing the Effectiveness of Social, Behavioural 6 February 2pm at the RSS and
Educational Interventions
Two of the talks will focus on the Campbell Collaboration, established to undertake
systematic reviews of interventions in the social, behavioural and educational arenas. The
third will draw on Professor Greenberg’s experience of evaluating USA Government welfare
and training schemes.
GERALDINE MCDONALD (School for Social Policy Studies, Bristol University) DIANA
ELBOURNE (EPPI-Centre, Institute of Education)
Meta.analyses of evaluations of government-funded training programs
DAVID GREENBERG
Government-funded training programmes in the United States have often been subject to
rigorous evaluation. Indeed, many of these evaluations have been evaluated with random
assignment, although sophisticated quasi-experimental methods have been used as well. Until
very recently, however, there has been little systematic attempt to use the cumulative
information vested in these evaluation to attempt determine which kinds of program work
best in which setting and with respect to which types of client. Meta-analysis - a set of
statistical procedures for systematically synthesising findings from separate studies - can, in
theory at least, address these and other topics that evaluation of individual programs cannot.
60
SMB 51 1/03
This talk will discuss the steps in conducting such a synthesis, summarizes the results of three
recently conducted meta-analyses of training programs, identify limitations to the metaanalytic approach, and consider ways in which some of these limitations can be overcome.
For more information on the above or to book a place contact Nicola Emmerson
e-mail: [email protected]
3. The Cathie Marsh Centre for Census and Survey Research
Questionnaire Design (1 day) 2 May 2003
Introduction to Data Analysis Part 1 (1 day)
12 February 2003
Introduction to Data Analysis Part 2 (1 day)
26 February 2003
Analysing Hierarchical Surveys (1 day)
19 February 2003
An Introduction to Sampling Theory (1 day)
11 Decemeber 2002
Multiple Regression (2 days)
14-15 May 2003
Logistic Regression (1 day)
21 May 2003
Conceptualising Longitudinal Analysis (1 day)
19 March 2003
Introduction to Longitudinal Analysis (1 day)
20 March 2003
Data Management with STATA (1 day)
12 March 2003
Data Reduction and Classification (1 day)
2 April 2003
Multivariate Modelling (1 day)
30 April 2003
Statistical Modelling with STATA (1 day)
16 April 2003
Demographic Forecasting with POPGROUP
Date to be confirmed
Longitudinal Data Analysis (3 days)
22-24 April 2003
For further information on the above courses or to book a place contact Nasira Asghar
Tel 0161 275 4736
Or visit http://www.ccsr.ac.uk/courses.htm
4. Miscellaneous Courses and Conferences
The ASC’s 4th
The impact of Technology on the Survey Process
International
Conference
The Association for Survey Computing is pleased to announce that it will be hosting its fourth
International Conference on Survey and Statistical Computing at Warwick University
between Sept 17 and 19, 2003.
61
SMB 51 1/03
The central theme of the conference will be the Impact of Technology on the Survey Process,
and we will be seeking contributions from producers, consumers and commissioners of
survey research alike. The conference will, inter alia, explore the often- complex relationship
between the push and pull of technological changes and the expectations and demands created
by them. We will also be seeking to examine how outcomes feed back to affect the processes
which initially gave rise to them. Although a conference about technology, the topics to be
discussed will appeal to everyone with an interest in survey design, data collection, analysis,
reporting, or statistical computing.
The conference will consist of papers presented to both plenary and parallel sessions, as well
as poster sessions.
All papers will be published prior to the conference in a bound set of proceedings to be
distributed to delegates.
For complete conference details, please see:
http://www.asc.org.uk
Training of European Statisticians (TES)
Programme
Training for European statisticians is essentially a European training programme, which fills
existing gaps between national training schemes and the challenges of the European
Statistical System. The annual programme is developed by the Training of European
Statisticians (TES) Institute. The programme supports the quality of European statistics, their
harmonisation and comparability as well as the transfer of new tools, methods and technology
between the different European countries. It addresses demands, which cannot be met
nationally, gives access to international experts in relevant fields and promotes the exchanges
of skills and experiences.
In order to meet the different needs of training and to react to new challenges in an
appropriate way, the training programme covers various training activities and methods. The
annual core programme contains courses in Official Statistics, IT applications, Research and
Development and in Statistical Management.
If you would like an application form, which will need to be emailed/faxed back to the TES
Institute, please contact Gareth John on o207 75-6340 or email.
If you would like to discuss the Programme in more detail, please contact Wesley Woollard,
UK TES Correspondent on 0207 75-6336 or email.
ESDS Launch
30 June 2003
During the afternoon of 30 June 2003 the Economic and Social Data Service (ESDS) will be
having its inaugural launch, at which Ian Diamond, Len Cook and ESDS staff will be
speaking about the range of exciting possibilities and opportunities that the new ESRC/JISC
Data Service can offer to social scientists and beyond. There will be no charge for the day As
places are limited, those who would like to attend should register their interest to:
archive @essex.ac.uk.
62
SMB 51 1/03
Recent Socila Survey Division Publications
1. Recent Social Survey Division Publications
1.
Family Spending - A report on the 2000 - 2001 Family Expenditure Survey
Denis Down
ISBN 0116214783
2.
Non-fatal suicidal behaviour among Adults aged 16 to 74 in Great Britain
Howard Meltzer, Deborah Lader, Tania Corbin, Nicola Singleton, Rachel Jenkins, Troalach
Brugha
ISBN 0116215488
3.
Student achievement in England
Baijit Gill, Mark Dunn, Eileen Goddard
ISBN 011621550X
4.
Student achievement in Northern Ireland
Baijit Gill, Mark Dunn, Eileen Goddard
ISBN 0116215607
5.
Family Resources Survey, Annual Technical Report 2000/2001
Mark Rowland
ISBN 1 85774 485 3
6.
Mental Health of Carers
N Singleton, N Aye Maung, A Cowie, J Sparks, R Bumpstead, H Meltzer
ISBN 0116215542
7.
The Social and Economic Circumstances of Adults with Mental Disorders
Howard Meltzer, Nicola Singleton, Alison Lee, Paul Bebbington, Traolach Brugha, Rachel
Jenkins
ISBN 0 11 621563 1
8.
Travel Trends - A report on the 2001 International Passenger Survey
ISBN 0-11-621577-1
9.
The National Diet and Nutrition Survey: adults aged 19 to 64 years - Types and
quantities of foods consumed
63
SMB 51 1/03
Recent Socila Survey Division Publications
Lynne Henderson, Jan R Gregory. With Gillian Swan (FSA)
ISBN 0-11-621566-6
10. Adults Living with a Psychotic Disorder Living in Private Households, 2000
Maureen O Nicola Singleton, Howard Meltzer, Janet Sparks, Traolach Brugha
ISBN 0-11-6215642
11. Tobacco, Alcohol and Drug Use and Mental Health
Melissa Coulthard, Michael Farrell, Nicola Singleton, Howard Meltzer
ISBN 0-1 1-621578-X
12. Contraception and Sexual Health, 2000
Fiona Dawe, Howard Meltzer
ISBN 1 85774 505 1
13. Psychiatric Morbidity Among Adults Living in Private Households, 2000
N Singleton, A Lee, H Meltzer
ISBN 1 85774 508 6
14. Disadvantaged Households. Results from the 2000 General Household Survey
Supplement A
W Sykes, A Walker
Web Based Publication: http://www.statistics.gov.uk
15. Living in Britain. Results from the 2001 General Household Survey
A Walker, M O’Brien, C Fox, J Traynor
Web Based Publication: http://www.statistics.gov.uk
64
SMB 51 1/03
Subscriptions and Enquiries
The Survey Methodology Bulletin is published twice yearly, in January and July,
price £5 in the UK and EIRE or £6 elsewhere. Copies of the current and previous
editions are available from:
Ann McIntosh
Survey Methodology Bulletin Orders
Social Survey Division
Office for National Statistics
D1/15
1 Drummond Gate
London
SW1V 2QQ
[email protected]
www.statistics.gov.uk
Social Survey Division
Office for National Statistics
1 Drummond Gate
London SW1V 2QQ
www.statistics.gov.uk
ISBN 1 85774 5132
ISSN 0263 - 158X
Price £5

Contents - Office for National Statistics

Transcription

Similar documents

Survey Solutions with a Human Touch

youth participation in national agenda through new media

September 2015

Casino Slot Machine Gratuit Quarter Jeux Casino Gratuit Sans

frs_brz_sheet

The Business Value of Trust

2015 JUNE LINK VOL 32 ISSUE 2 - ccons

Song PredicZon System

November 2015