Minimum standards

Handout
Nick Saville: Munich 2011
Minimum standards
17 parameters for establishing Quality Profiles in ALTE examinations
The Quality Profile is created in each case, by explaining how the examination
meets the following minimum standards, and by providing adequate evidence.
The examination is based on a theoretical construct, e.g. on a model of
communicative competence.
2
You can describe the purpose and context of use of the examination, and the
population for which the examination is appropriate.
3
Test construction
1
You provide criteria for selection and training of test constructors and expert
judgement is involved both in test construction, and in the review and revision of
the examinations.
Parallel examinations are comparable across different administrations in terms of
content, stability, consistency and grade boundaries.
5
If you make a claim that the examination is linked to an external reference system
(e.g. Common European Framework), then you can provide evidence of alignment
to this system.
6
All centres are selected to administer your examination according to clear,
transparent, established procedures, and have access to regulations about how to
do so.
7
8
9
Administration & Logistics
4
Examination papers are delivered in excellent condition and by secure means of
transport to the authorized examination centres, your examination administration
system provides for secure and traceable handling of all examination documents,
and confidentiality of all system procedures can be guaranteed.
The examination administration system has appropriate support systems (e.g.
phone hotline, web services etc).
You adequately protect the security and confidentiality of results and certificates,
and data relating to them, in line with current data protection legislation, and
candidates are informed of their rights to access this data.
1
Handout
12
13
14
Test analysis
11
The examination system provides support for candidates with special needs.
Marking & Grading
10
17
Communication with
Stakeholders
15
16
Nick Saville: Munich 2011
Marking is sufficiently accurate and reliable for purpose and type of examination.
You can document and explain how marking is carried out and reliability estimated,
and how data regarding achievement of raters of writing and speaking
performances is collected and analysed.
You collect and analyse data on an adequate and representative sample of
candidates and can be confident that their achievement is a result of the skills
measured in the examination and not influenced by factors like L1, country of
origin, gender, age and ethnic origin.
Item-level data (e.g. for computing the difficulty, discrimination, reliability and
standard errors of measurement of the examination) is collected from an adequate
sample of candidates and analysed.
The examination administration system communicates the results of the
examinations to candidates and to examination centres (e.g. schools) promptly and
clearly.
You provide information to stakeholders on the appropriate context, purpose and
use of the examination, on its content, and on the overall reliability of the results of
the examination.
You provide suitable information to stakeholders to help them interpret results and
use them appropriately.
2
Handout
Nick Saville: Munich 2011
ALTE Membership
27 languages
32 full members, 1 associate member
organisations comprising government
consortia and examination boards
Basque
Bulgarian
Catalan
Czech
Danish
Dutch
Dutch
English
English
Estonian
Finnish
French
French
French
Galician
German
German
German
Greek
Hungarian
Irish
Italian
Latvian
Lithuanian
Luxembourgish
Norwegian
Polish
Portuguese
Russian
Slovenian
Spanish
Spanish
Swedish
Welsh
departments,
universities,
Eusko Jaurlaritza
University of Sofia "St Kliment Ohridski" Department for Language
Teaching and International Students (DLTIS)
Generalitat de Catalunya
Charles University, Prague
Danish Language Testing Consortium
CNVAT
CITO
University of Cambridge ESOL Examinations
Trinity College London
Riiklik Eksami - ja Kvalifikatsioonikeskus
Jyväskylän yliopisto ja Opetushallitus
Chambre de Commerce et d'industrie de Paris
Centre international d’études pédagogiques
Alliance Française
Xunta de Galicia
TestDaF-Institut
Goethe-Institut
telc GmbH
University of Athens
Institute of the Eötvös Loránd University (ELTE)
Language Centre, National University of Ireland, Maynooth
Università per Stranieri, Perugia (USP)
Latvijas Republikas Naturalizācijas pārvalde – membership temporarily
suspended.
LITUANISTINIŲ STUDIJŲ KATEDRA
Centre de Langues Luxembourg (CLL)
Studieforbundet Folkeuniversitetet
Uniwersytet Jagiellonski w Krakowie
Centro de Avaliação de Português Língua Estrangeira – CAPLE
The Russian Language Testing Consortium – consisting of Moscow
University and St Petersburg (Associate members)
Univerza v Ljubljani
Instituto Cervantes (IC)
Universidad de Salamanca (CIUS)
Stockholms universitet
Cyd-Bwyllgor Addysg Cymru (CBAC)
Plus many affiliated institutions and individual affiliates.
See www.alte.org
3
Handout
Nick Saville: Munich 2011
References:
AERA/APA/NCME (1985/1999). Standards for educational and psychological
testing. Washington: AERA.
ALTE (1993/2001). Principles of Good Practice for ALTE Examinations.
ALTE (1994). Code of Practice.
ALTE (1998). Handbook of Language Examinations and Examination Systems.
ALTE/Council of Europe (1997). Users Guide for Examiners. Strasbourg.
van Avermaet, P. Kuijper, H. and Saville, N. (2004). A Code of Practice and
Quality Management System for International Examinations. LAQ 1: 2&3.
Special Issue: The Ethics of Language Assessment. Guest Editor, Alan Davies.
Bachman L.F. (1990). Fundamental considerations in language testing. Oxford:
OUP.
Bachman, L. F. and Palmer, A. (1996). Language Testing in Practice. Oxford:
OUP.
ILTA (1995). Task Force on Testing Standards.
ILTA (2000). Code of Ethics. Language Testing Update, Issue 27.
Kane, M.T.1992. An argument-based approach to validity. Psychological
Bulletin. 112, 527-535.
Kunnan, A. (2000). Fairness and validation in language assessment.
Cambridge: UCLES/CUP.
Kunnan, A..J. 2004. Test Fairness. In Milanovic and Weir 2004.
Messick, S..A. 1989. Validity. In Linn (ed.) Educational Measurement. New York:
ACE / Macmillan.
Messick, S. 1994. The interplay of evidence and consequences in the validation
of performance assessments. Educational Researcher, 32(2), 13-23
Messick, S.A. (1980). Test validity and the ethics of assessment. American
Psychologist, 35.
Mislevy, R.J. 1994. Evidence and inference in educational assessment.
Psychometrika, 59, 439-483.
Mislevy, R.J., L.S. Steinberg, and R.G. Almond. 2003. On the structure of
educational assessments. Measurement: Interdisciplinary Research and
Perspectives, 1(1), 3-62.
Saville, N. Auditing the Quality Profile. Paper presented at LTRC 2008, Hang
Zhou, China.
Saville, N. Van Avermaet, P. and Kuijper, H. Setting and Monitoring
Professional Standards: a QMS approach. Paper presented at the 2nd EALTA
Conference, Voss, June 2005.
4