Testing the Four Skills: Part 2 Testing Writing and Reading

2015.04.23.
Testing the Four Skills: Part 2
Testing Writing and Reading skills
Language Testing and Assessment
Lecture 9
Brózik-Piniel, Katalin
Testing Writing
1
2015.04.23.
The nature of writing
 The ability to develop thoughts in writing and to convey
them accurately, effectively and appropriately with a
particular audience and communicative purpose in mind
(Kormos & Kontra, 2007, p. 96).
 Writing vs. Speaking
 Lexical density
 Spontaneity
 Structured
 The writing process is recursive
 Make candidates write!
Texts
 Variety
 Examiners and teachers have to be familiar with …
 Good writing task criteria
 Features of text types (e.g. of letters, articles,
compositions)
 Candidates should ideally have practice in different
text types (timed and untimed, peer review)
2
2015.04.23.
Task design
 Representative sample (more than one task type - fresh starts)
 Rubric- procedural information
 What the task will assess
 Steps
 Time
 Word limit
 Rubrics are not reading tasks
 Prompt: authentic situation, pictures
 Clear purpose, content-topic, audience, text type
 Clear layout
 Clear language
 Task types: forms, guided compositions, summary, letters, reports
 Piloting tasks
Rating scales
 Scale development:
 Criteria, bands, benchmarking, standardization - rater training
(reliability)
 Task specific
 Holistic
 Analytical :
 Grammar (accuracy)
 Organization (coherence, cohesion)
 Spelling, punctuation
 Vocabulary
 Content (task achievement)
3
2015.04.23.
Feedback
 Correction symbols (standard)
 Positive comments
Assessing writing in the classroom (1)
 Reasons for formative assessment of writing:
 Writing is a process
 A holistic view on the process
 Enhancing learning of writing
4
2015.04.23.
Assessing writing in the classroom (2)
Five principles of AFL adapted to the teaching and assessment of
writing:
 Share learning goals with students
 (e.g., story writing
 goals of story writing
 activities that realize these goals (for exampl identifying story structure)
 Help students understand the standards they are
working towards
 discuss criteria for assessment
 use sample texts, mini-text analysis tasks, and text improvement tasks where
they attempt to apply the criteria to evaluate the quality of the texts
(Lee, 2007, p. 204)
 Involving students in assessment
 self- and peer-assessmentindependent and reflective in learning
 focus not only on weaknesses but also strengths
 E.g., self-editing, self-assessment based on the assessment criteria
articulated, and self-inquiry, where students not only reflect on their
strengths and weaknesses in writing, but also formulate their own goals
and take initiatives to improve their own writing.(Lee, 2007, p. 204)
 Teachers should provide helpful feedback:
 effective feedback: focused, stimulates thinking, consists of comments
only, refers explicitly to success criteria, and provides concrete guidance
on how to improve (rather than giving complete solutions).
 Avoid giving vague comments (e.g., ‘under-developed ideas’)
 Make concrete suggestions to help revision (e.g.,‘Give one or two
examples to illustrate why smoking is harmful.’)
 Don’t write correct answers for students’ grammatical errors,
 Students should be asked to correct their own errors – if these are errors
amenable to self-correction (Ferris, 2003).
5
2015.04.23.
 Provide opportunities for students to act upon feedback
 Create a classroom culture where mistakes are a natural part
of learning and where everyone can improve
 E.g., devise strategies and design materials that help students
work on relevant aspects of their grammar.
 Integrating teaching, learning, and assessment in writing in
the classroom
Guidelines for AFL - Writing
 Use a variety of tasks
 Empower students:
 Using self- and peer evaluation
 Writing journals in pairs or groups
 Compiling their own portfolios
 Keeping error logs
 Writing reflective journals or progress logs on how they can improve their
future compositions
 Suggesting areas of error feedback for the teacher
 Participating in the development of feedback forms or checklists
 Withhold scores till final draftsand just have comments on previous
drafts
6
2015.04.23.
Testing Reading
Reading
 Reading comprehension means understanding a written text,
and extracting the information required to fulfill the reading
purpose (Kormos & Kontra, 2007, p. 85).
 Process-product (difficult to isolate)
7
2015.04.23.
The nature of reading
 Different levels of understanding (literal, inferred, critical)
 Multi-level process






Recognition, selection
Vocab+structure
Discourse
Content(background)
Synthesis and evaluation
Meta-cognitive knowledge (e.g. skimming, scanning)
 Sub-skills: skimming and scanning
 Approaches
 bottom-up: sequential, unidirectional
 top-down: overall meaning, schema theory (making use of
existing knowledge)
 Interactive model: reader-text-social context
Task types and techniques
To match authentic purposes, variety of text types and setting
 Multiple-choice
 True/false
 Short answer questions (L1/L2)
 Matching (headings, sentences etc.)
 Ordering
 Information transfer
 Gap filling
8
2015.04.23.
Caution
Reading+other skills
 Cloze tests
 Editing
 Summary
Candidates should be familiar with task types.
When testing language proficiency, avoid testing memory and
background knowledge.
Authenticity-an issue (task determines level of difficulty)
ITEM WRITING
 Familiarizing with Specifications and Guidelines
 Writing items
 Small-scale pre-testing followed by revision of items
 Receiving feedback from pre-editing
 Revising items
 Receiving acceptance or rejection from an Editing Committee
The second part of this process normally includes the following steps:
 Pre-testing items on a larger scale
 Analyzing items statistically
 Revising items centrally
 Optional further pre-testing
 Building item banks
 Constructing test papers
(Alderson & Cseresznyés, 2003, p. 299)
9
2015.04.23.
Text selection
 Use texts that are authentic and suitable for the selected task type.
 Select texts whose topic is accessible to the students’ age group.
 Try to select texts that are likely to be interesting for students. Be aware that
humorous texts might be enjoyable and easy for you but very difficult for students.
 Avoid offensive or distressing topics.
 Avoid well-known novels as text sources.
 Select texts of suitable length.
 Do not use more illustrations than necessary. When illustrations are used, they should be
clear and photocopiable.
 Do not use published teaching materials.
 Do not use texts with content covered in detail in any textbook. You do not want to test
knowledge of content, but ability to understand texts.
 Do not make any changes to the original text. Do not delete words, sentences
or paragraphs from the selected body of text. If the text contains any offensive words
that you think should be replaced, only change these with great care and always seek the
advice of a fellow teacher or a native speaker as to the acceptability of the changes you
have made.
 If the text used is not the beginning part of a longer text, make sure it stands alone
and does not contain references to previous parts of the original text.
(Alderson & Cseresznyés, 2003, p. 303)
Task selection
 Have a clear idea about the purpose of each reading task,
which reading skill is tested in a particular task, and why a
particular task type is used.
 The task should reflect the way in which people usually read
the type of text used.
 Consider applying more difficult texts with easier tasks and
vice versa.
10
2015.04.23.
Task design
 There should be a minimum of five items in a task.
 Do not design in any one task more than 10 items for a passage of approximately
100 words.
 The wording of the items should be such that it is easy to understand, is below
the language level of the text and does not require more reading than the text
itself.
 Make sure that no item can be answered correctly without reading the
text.
 Make sure that each question can be answered independently.
 Make sure there is only one possible answer to each question. If more than
one answer is required, this fact must be clearly indicated, and they must count as
different items.
 Avoid questions which demand much imagination.
 Sequence items in the order they appear in the original text (other than sequencing
tasks!).
 In any gap-filling task (e.g. modified cloze, banked cloze, discourse cloze) there
should be a sentence at the beginning (before the example) for lead-in and a
sentence at the end for lead-out, which do not have any items.
 In a gapped text, where you take out paragraphs, do not take out the first or the
last paragraph, in order to leave enough context for comprehension.
 In gap-filling tasks there should be at least five words between gaps to provide
enough context.
 “Not mentioned” must not be used as an optional answer,






because this can be misinterpreted. Many things “not mentioned” can
still be inferred.
Provide more choices than matches in matching tasks and banked gapfilling tasks (minimum 1, maximum 2 distractors to be provided).
Make sure that ambiguous matches are excluded.
Make sure that in sequencing tasks there is only one correct order
and no more than 6-7 items for intermediate students, or 8-9 items for
advanced students. Be careful if using newspaper articles in
sequencing tasks: they are rarely in chronological order.
Clearly indicate the required length of the answers (maximum 3
words) in open-ended (short-answer) questions.
Design tasks with clear and consistent layout. The input text
should retain the original format as much as possible. Paragraph texts
clearly with indentations.
Provide answer keys reflecting the form in which answers are
expected. Give all possible or acceptable answers. Unacceptable
answers that are likely to occur should also be indicated. In tasks where
the correct answer may be phrased in various ways, the answer key
should provide content-related instructions concerning acceptable
answers.
11
2015.04.23.
Rubrics
 Indicate what type of text students are going to read and/or
the text source.
Assessing reading in the classroom
 Reading logs
 Reading reports
 Self-assessment scales for reading (using can-do statements)
12