- ATS2020-Mahara E-Portfolio

EUfolio – EU classroom ePortfolios
Work Package 4: Pilot design
Deliverable 12
EUfolio Quality Assurance Plan
for Assessment Operations
and
Guidance Materials for Participants
EUfolio has been funded with support from the European Commission. This publication reflects the
views only of the authors, and the Commission cannot be held responsible for any use which may be
made of the information contained therein.
Edited by Andrea Ghoneim
Department of Interactive Media and Educational Technologies,
Danube University Krems, Austria
Final Version: 30 April 2015
Summary
The quality assurance plan aims at ensuring a common understanding of terms and
assessment practices and the question how they can fit into the ePortfolio process.
As an ePortfolio is a learner centered tool and can be used both to accompany the
whole learning process (actually: life-long) but also to collect evidence around one
project that is part of a certain subject in a certain grade, ePortfolio assessment can
contribute to different extents to the overall classroom assessment. However, the
ways in which assessment can be carried out and the parts of the ePortfolio process
to which they can be linked are similar.
In the first part of this Quality Assurance Plan, readers will be guided through the
ePortfolio process and will be shown which forms of assessment can be carried out
in which phases of the ePortfolio process. Forms of assessment and feedback have
to be shown in order to ensure the same understanding of assessment. Furthermore,
the fact that the creator owns the ePortfolio provides a challenge when it comes to
store the artefacts and/or ePortfolio pages/collections that are the basis of
assessment (recording of evidence).
In the second part, short guidance materials for participants of the EUfolio pilot target
different stakeholders in the process in order to ensure the same understanding of
ePortfolios, assessment and quality standards.
Thanks for valuable input to
Ben Murray, National Council for Curriculum and Assessment (Ireland) and
Bernhard Ertl, Danube University Krems (Austria)
3
Table of contents
Preface: Aims ............................................................................................................. 5
1. Quality Assurance Plan .......................................................................................... 6
1.1 Assessment in the ePortfolio Process ............................................................... 6
1.2 Assessment operations ..................................................................................... 7
1.2.1 Context Definition ....................................................................................... 8
1.2.2 (Peer) feedback + formative assessment ................................................. 11
1.2.3 Final submission / presentation ................................................................ 14
1.2.4 Summative assessment (with rubrics) ...................................................... 15
1.3 Assessment and training ................................................................................. 17
1.4 Assessment and selection/storage of evidence .............................................. 18
1.4.1 Assessment process using the ePortfolio platform Mahara ...................... 18
1.4.2 Assessment via the Microsoft ePortfolio platform ..................................... 22
2. Guidance Materials for Participants ...................................................................... 25
2.1 For pilot partners ............................................................................................. 25
2.2 For principals................................................................................................... 25
2.3 For teachers .................................................................................................... 26
2.4 For support staff .............................................................................................. 27
2.5 For students .................................................................................................... 28
Glossary ................................................................................................................... 29
References ............................................................................................................... 30
4
Preface: Aims
The aim of this deliverable is to offer Quality Assurance guidelines for assessment
operations for ePortfolio works.
With respect to the manifold types of ePortfolios (c. EUfolio Process specification =
Del. 10), this Quality Assurance Plan for Assessment Operations does not aim at
limiting the ePortfolio activities to an elaboration or compilation of assessment
portfolios.
The quality assurance plan simply aims at ensuring a common understanding of
terms and assessment practices and how they can fit into the ePortfolio process. As
an ePortfolio is a learner centered tool and can be used both to accompany the
whole learning process (actually: life-long) but also just to collect evidence around
one project that is part of a certain subject in a certain grade, ePortfolio assessment
can contribute to different extents to the overall assessment. However, the ways in
which assessment can be carried out and the parts of the ePortfolio process to which
they can be linked are similar.
In order to ensure a common understanding of terms, a short glossary at the end of
this paper collects some terms with definitions.
5
1. Quality Assurance Plan
1.1 Assessment in the ePortfolio Process
As “e-Portfolio based learning offers a real potential for autonomous and
personalised learning“ (JISC 2008), the assessment process has to take this
autonomy into consideration. While in conventional teaching processes learning
outcomes are defined by the teacher as a basis for assessment, in teaching based
on ePortfolios, the learning targets have to be negotiated between teachers and
students taking into consideration the requirements of the curriculum as well as the
aims of the class. Additionally, the teacher also has to decide, whether s/he bases all
classroom activities on ePortfolios or whether the ePortfolio is only accompanying
single projects. If the latter is the case, the assessment of the ePortfolio (or the parts
of it that are handed in for assessment) will only be a part of the overall grade.
Nevertheless, the process for assessment will be the same – no matter if it is only
part of the overall assessment or if it is the basis of it, no matter if it is formative
assessment (assessment for learning) or if it is summative (assessment of learning).
We start with a model of the ePortfolio process:
According to Klaus Himpsl-Gutermann’s process model
(2012, p.260-268) the Context Definition is the starting
point for the work with ePortfolios in the classroom. It
clarifies the aim and the context of ePortfolio work, it is
mainly elaborated by the authorities (policymakers,
teachers) in cooperation with the students.
The Context Definition in the beginning of the educational
process and the Final Submission in the end of the
process are – according to this model – the activities
within the core process in which the authorities are mainly
involved. Furthermore, the Feedback activity also may
involve authorities or peers who give feedback to a
student’s ePortfolio. Hence, the dark blue layers are the
periods, in which ePortfolio views are made visible to
peers and/or authorities (teachers, assessment
commissions) while the lighter coloured areas stand for
activities that are mainly done by the learner alone.
(Himpsl-Gutermann 2012: 262)
(N.B.: This model and the explanation were also part of EUfolio Deliverable 10 =
Process specification.)
A further model (Himpsl-Gutermann 2012: 239) explains the layers of ePortfolios.
This helps us to understand that ePortfolios cannot be assessed as a whole. The
showcase ePortfolio will usually consist of a collection of pages (views) that are
6
organized by the learner, containing selected artefacts (products of learning) that the
student put into context in order to represent her/his learning and in order to show
how s/he has tried/managed to reach her/his learning goals.
The middle layer of this model shows the process of learning which is mainly
consisting of reflections – usually done in a diary (blog). This process – as far as it is
made accessible by the student – can be part of the assessment.
Himpsl-Gutermann 2012: 239
1.2 Assessment operations
It is a principle of ePortfolio that the creator is the owner of the portfolio. It is the
owner’s (= the student's) decision which parts of the ePortfolio are shared with whom.
S/he is also the one who can decide which parts of her ePortfolio are handed in for
assessment.
Del. 10 (“EUfolio process specification”) displays in chapter 3.6 a table of the Scottish
Qualifications Authority (SQA) that shows the process of creating an ePortfolio for
assessment. The table is addressed to the students (“you”), but it also shows where
the students can expect advice by their teacher/assessor.
7
(Table taken from SQA 2012, p.26)
1.2.1 Context Definition
“If an eportfolio can embed the necessary range of learning artifacts, allow them
to be added at anytime, allow students, teachers, friends and parents to provide
effective feedback on these artifacts from anywhere, engage and motivate the
learner through control of their learning and using technology, and celebrate
learning, then the eportfolio will support the process of assessment for learning.
[…] However one must note that it is not the technology that will make effective
learning happen. Whether the eportfolio supports formative assessment
ultimately depends on how the teacher facilitates its use.” (Rate 2008, p.22)
8
Defining the context means for the assessment authority to propose and negotiate a
clear scheme for assessment with the student(s).
Following a learner centered model, the students have to be made acquainted with
the skills they can acquire and possible ways to do so. Following these steps, the
student will define her/his learning aims with help (coaching) by the teacher and/or
peers. The learning aims (targeting “learning outcomes”) and the learning path are
the basis for assessment.
The Irish National Council for Curriculum and Assessment (NCCA) emphasizes:
“Teachers are usually quite clear about the objectives for a unit of study.
Students may understand what to do for individual tasks but be less clear about
how these tasks fit into 'the big picture'. They can be greatly helped when the
teacher explains the learning goals--what she/he hopes the students will learn
and why they are learning it--in words that they can understand.”
(NCCA 2014. AfL – Key principles. http://www.ncca.ie/en/Curriculum_and_Assessment/PostPrimary_Education/Junior_Cycle/Assessment_for_Learning_AfL_/Key_principles/AfL__Key_principles.html#sthash.WJogjb5n.dpuf)
The basis for defining the context of setting up an ePortfolio can be skills-based. The
context will also be in relation to the curriculum and the subject.
Another basis defining the context of learning with ePortfolios can be selfassessment, which can be done with the same catalogue of criteria in the beginning
of the ePortfolio process in order to help the student in setting goals and to be aware
of her/his status quo at the beginning of her/his ePortfolio journey. This selfassessment can help both teachers/assessors/coaches and students to define the
context in which the ePortfolio should be initiated.
The benefits of self-assessment are highlighted by the NCCA (2014, as above)
“Students need to develop the capacity for self-assessment so that they can
become independent learners with the ability to seek out and gain new skills,
knowledge and understandings. Teachers can encourage this by providing
opportunities for students to assess their own and one another's work, and to
review and record their own progress. This helps the students to understand
their achievements, identify the gaps in their knowledge and plan for better
learning.”
For self-assessment, Education Services Australia (2008) compiled the following
bullet points:
The very important role of the teacher involves:
• sharing with students the success criteria for each assessment activity
• ensuring that students understand the success criteria
• explicitly teaching students how to apply those criteria to their own work
• providing students with feedback to help them improve; and
9
•
helping students to set learning targets to achieve that improvement.
Students who use self-assessment:
• recognise that learning is associated with a very positive kind of difficulty, which
increases motivation rather than destroying it
• experience an increase in self-esteem
• experience an improvement in their learning because they come to know how they
learn rather than just what they learn.
Teachers who encourage students to self-assess:
• see the responsibility for learning shifting from them to their students
• recognise an increase in student motivation and enthusiasm for learning and a
corresponding decrease in behavioural problems
• are able to use feedback from their students about how they learn to shape lessons
to individual and group needs rather than teaching to the mythical class as whole
(Education Services Australia: Assessment for Learning. Background:
http://www.assessmentforlearning.edu.au/professional_learning/student_selfassessment/student_research_background.html)
A possible basis for self-assessment can be a set of the can-do statements for a
range of 21st century skills (referred to as “Key Skills”) as given by the NCCA (2012),
which are reproduced in the EUfolio Process Specification (Del. 10). However, a
checklist could also be the basis for self-assessment, as well as rubrics (see chapter
1.2.2).
My Learning
For the EUfolio project, the Mahara-plugin “My Learning” was the basis of some of
the ePortfolio pilots. The plugin, created by Gregor Anželj, supports self assessment
and self-guided learning. The following screenshot shows an overview over a user’s
“learnings”, which are different learning projects.
When a “new learning” is created, the user has to fill forms for the following topics:
Prior knowledge
10
Here, the learner can analyze what s/he already knows and/or can do in
connection with his/her subject
Setting goals
This part of the form serves the setting of a student’s learning goals supported by her/his teacher or mentor
Strategies
The student should ask her-/himself: “How can I reach the learning goals
determined above?” and write the answers into the field provided
Evidence
This filed serves the question: “How will I document the outcomes of my
learning?” the student can for example aim at keeping evidence by a collection
of artefacts, a diary etc.
Self-evaluation
Finally, the student looks back at her/his learning experience. How does it
match with her/his goals? How was the overall process? S/He can ask her/himself: “How do I want to proceed?”
The rubrics “Prior Knowledge”, “Setting Goals”, “Strategies”, and “Evidence” are filled
at the beginning of the learning (project) and thus help to define the context for the
learning.
The field with the title “Self-evaluation” is used at the end of the learning project for
self-evaluation which can lead to new learning targets.
The contents of all mentioned fields document and define the context for the learning
and thus form a basis for assessment.
1.2.2 (Peer) feedback + formative assessment
Avraamidou and Economou (2013, p. 7-8) describe the ePortfolio in 3 stages:
storage (level 1), workspace/process (level 2), showcase and assessment/product
(level 3). If we look at this description while still keeping the process model of HimpslGutermann (cf. chapter 1.1) in mind, the stage of (peer) feedback and formative
assessment would correspond to level 2. The student hands in a product (ePortfolio
collection or page) for assessment that can still be considered work in progress, but
is already mature enough to be shown to a peer or a teacher for assessment.
The assessment in this stage of the ePortfolio process needs to be formative
(Assessment for Learning) in order to allow the student to ameliorate his ePortfolio
with respect to the feedback.
11
It is recommended to base the feedback/assessment on the learning targets as
defined by the student in the context definition. Again, a possible basis are can-dostatements like the ones defined by the NCCA (2012) for the following competences
and skills.
(Source: NCCA 2012, p.10)
The following table from NCCA 2014 shows how a competence (here as one
example the communication competence) can be described in can-do-statements.
12
(Source: NCCA 2014)
The skills and learning targets (basing on can-do-statements) should be chosen upon
agreement between the student and the teacher/assessor. They could also be put in
rubrics in order to allow a more distinct feedback.
Formative feedback/assessment for learning has to provide developmental feedback,
i.e. to give the learner hints, what to do next to ameliorate/redesign their ePortfolio in
order to continue their way towards their learning targets.
Therefore, for peer-feedback in lower secondary school (but also above this level) it
can be helpful to encourage the peers to a more distinct feedback with open
sentences that have to be completed by the peers.
Examples:
“I would like to know more about…..”
“I particularly enjoyed ….. and hope that you …. “
Education Services Australia recommend the following “strategies to enhance peer
feedback”, part of which can also be included in the peer feedback process:
Two stars and a wish
Students identify two positive aspects of the work of a peer and then express a wish about what
the peer might do next time in order to improve another aspect of the work. 'I want to give you a
star for the start of your story and a star for the way you described the house. I wish that you will
tell us more about Billy.'
13
Teachers model this strategy several times, using samples of student work, before asking the
students to use the strategy in pairs on their own. They check the process and ask pairs who
have implemented the strategy successfully to demonstrate it to the whole group.
Plus, minus and what's next?
Students comment on what was done well in relation to the success criteria, and also on what
could be done better. This strategy may be better used after the students have become adept at
using Two stars and a wish. This strategy can also be used as part of self-assessment, where
students use 'What's next?' to set a personal learning target.
Warm and cool feedback
When students comment on the positive aspects of a peer's work, they are said to be giving
warm feedback, and when they identify areas that need improvement, they are providing cool
feedback. They provide hints on 'how to raise the temperature' when they give advice about
how their peer could improve their work.
Traffic lights
Students green-light (using a green highlighter on the margin of the work) the work of their peer
to indicate where the success criteria have been achieved, or amber-light where improvement is
needed.
This strategy is best used on a work-in-progress, although it could also be used, with coloured
sticky notes, to provide feedback on a final piece of work. The suggestions for improvement
would then relate to the next occasion on which the students undertook work which required
similar skills - writing or number skills, for example.
(http://www.assessmentforlearning.edu.au/professional_learning/peer_feedback/peer_strategies
_enhance.html)
The peer feedback should be trained and discussed in classroom; the teacher should
complete this feedback in order to ensure suitable assessment for learning. This
guarantees that the student is well coached for his/her way towards final submission.
Both peer feedback and formative assessment by the teacher can also be given
orally. If this is the case, the student who receives this feedback has to be invited to
summarize it – for example in a blog – and to link it to the submitted state of her/his
work.
It is further recommended that the student keeps a copy of the ePortfolio
pages/collection, s/he handed in for assessment before proceeding with redesign.
This ensures that both the student and – if necessary – the teacher/assessors can
determine the progress from the preliminary version (selection of workspace/progress
ePortfolio) and the final version (showcase ePortfolio).
1.2.3 Final submission / presentation
The final submission of the ePortfolio (showcase) should be accompanied by a
presentation. This allows the student to prove that the collected and contextualized
artefacts are her/his own and also gives room for another round of feedback, both by
peers and the teacher/assessor. Before the presentation the student has already
14
submitted her/his showcase ePortfolio to the teacher/assessor, and this version of it
is the basis for the presentation. The showcase should contain a self-evaluation by
the student, which can be also the basis for the final assessment by the teacher.
If there are the same learning targets for a group, collective self-assessment is also
possible.
Group assessment in
form of an analysis of
Student’s reflection
on the use of the
ePortfolio platform
Mahara (presentation
from an EUfolio pilot
class from OS Polje,
picture taken by
Andrea Ghoneim on
15 April 2014)
The following picture above shows an example of a class evaluation on working with
the ePortfolio platform Mahara.
1.2.4 Summative assessment (with rubrics)
Both, the showcase ePortfolio and the presentation are the basis for the final
assessment. This can be formative (aiming at the student’s further progress) and/or
summative. Rubrics are a reliable basis for summative assessment – especially if
they contain and/or refer to the learning targets the student(s) and the teacher
agreed upon.
According to a publication of the Educational Research Service (based in Arlington,
USA, 2004) a rubric can define standards and helps students to achieve them “by
providing criteria with which they can evaluate their own work”. The rubric has to be
simple and easy to use and should provide all students with an opportunity to
succeed at some level. If a rubric is well set up and evaluated (for example through a
discussion with the students about their learning targets) it should be able to “yield
consistent results, even when administered by different scorers” (p.10).
15
With reference to Andrade (2009), Avraamidou and Economou (2013, p.18) state:
A rubric is a tool that is developed in order to set criteria for assessing certain
skills or knowledge. Andrade et al. (2009), researched middle school writing
with the use of rubrics and found that children that went through the process of
reviewing a sample rubric, developing a rubric by generating criteria and then
using it for self-assessment performed better in their writing and had better
group discussions. Going through the process of creating a rubric is a valuable
activity for both the teacher and the student. It is a way for the teacher to assess
whether students have learnt the content-knowledge (in order to generate
assessment criteria, one needs to first be aware of what to assess) and whether
they have developed certain skills (in order to develop i.e. a collaborative rubric,
one needs to develop collaboration, communication, critical thinking, creative
and learning to learn skills).
Rubrics can be scaled into 3 to 5 grades. They should be extended by a field for
each item in which a short explanation for the assessment is given.
Here is an example for a five grade scale (for adults). This example can be adapted
for school purposes.
Explanation
Illustration taken from: Radziwill (2012). http://qualityandinnovation.com/2012/02/02/the-rubric-as-ageneral-purpose-quality-tool/. Extended by column for explanation of the score by the authors.
16
In the blogpost that contains this example, advantages and disadvantages of rubrics
(as a quality tool) are discussed:
Advantages of applying a well developed rubric include:
•
Provides a common language for sharing expectations and feedback
•
Helps to clarify and distinguish the differences between various performance levels
•
Helps to focus an individual or group’s ATTENTION on relevant aspects of each desired
quality characteristic or skill area
•
Provides a mechanism to more easily identify strengths and opportunities for improvement
• Helps lend objectivity to an evaluation process that might otherwise be subjective
Disadvantages:
•
Different rubrics may need to be devised for the different activities or artifacts that are to be
evaluated using the rubric
•
Not all evaluators will apply the rubric in exactly the same way – there is a subjective element
at work here – so people may need to be trained in the use of a rubric, or perhaps it would be
more effective in a group consensus context where inter-rater variability can be interactively
discussed and resolved
•
Creating a rubric can be time consuming
•
The rubric may limit exploration of solutions or modes of presentation that do not conform to
the rubric
(Source: Radzwili 2012 as quoted above)
1.3 Assessment and training
In order to ensure reliability and validity of the rubrics, assessment training is
necessary, both for peer review (to be done in classroom) and for teachers (to be
done in CPD). If external assessors are involved, they need to see the learning
targets of the students (as part of their ePortfolio or stored in a different way) together
with the rubric in order to make sure that the assessment rubric can serve as a
reliable tool. Validity is again ensured by training of the assessors.
A further measure to increase validity and reliability is assessment by two (trained)
assessors who do double blind assessment and then discuss the outcomes until they
agree on one grade and the reason for it. However, this is a time consuming process
and therefore can be only done for ePortfolios that are part of high-stakes
assessments (for example as part of state examinations, final examinations in order
to finish a school cycle, …).
There is a range of literature available on reliability and validity. A recommendable
resource – because of its brevity and obvious targeting of teachers/assessors – could
be Charles Darr: “A hitchhiker’s guide to reliability” and “A hitchhiker’s guide to
validity” (2005, available via: http://assessment.tki.org.nz/Using-evidence-forlearning/Concepts/Concept/Reliability-and-validity).
17
1.4 Assessment and selection/storage of evidence
1.4.1 Assessment process using the ePortfolio platform Mahara
In an ePortfolio process, the basis of learning is owned by the learner. S/he takes the
decision what to share with whom. For assessment, s/he has to select, which
learning artefacts are chose for assessment. Usually they are collected and put into
context in a view/page. Contextualisation is done by a cover letter (an introduction)
and further texts that illustrate the reasons for selection and refer to the target and
the chosen audience (for example: the assessment authority). The illustration from
the Mahara User Manual below shows the framework of an ePortfolio.
Source (illustration and text):
Mahara 1.8 User Manual:
http://manual.mahara.org/en/1.8/intro/introduction.html#the
-mahara-framework
The description of the Mahara framework
(addressed to the ePortfolio owner) also
applies to other ePortfolio platforms:
„With Mahara, you control which items
and what information within your
portfolio other users see. Such items
and information are termed artefacts. To
facilitate this access control, all artefacts
you wish to show to other users need to
be arranged into one area. In Mahara
this compilation of selected artefacts is
called a View [Page]. You can have as
many Views as you like, each with a
different collection of artefacts, and
intended purpose and audience. Your
audience, or the people you wish to give
access to your View, can be added as
individuals or as a member of a Group.
It can even be made publicly available.“
SQA’s “Guidance on Using E-portfolios” (2012) also emphasizes (addressing the
teacher/coach of the creators of the ePortfolios):
it’s important to remind learners that, to achieve a qualification, the e-portfolio they submit for
assessment should contain the items that clearly recognise and record their achievement.
Evidence for assessment could include:
♦ written notes, letters or reports — based on surveys, experiments, investigations
♦ entries from the learner’s blog
♦ photographs, and scanned documents - pictures, posters, maps, graphs, diagrams
♦ narratives recording reflective accounts, witness testimonies, assessor observations and other
commentaries, discussions and interviews that are relevant to the Learner’s work or evidence —
presented as written or audio/video records
♦ online presentations showing images and providing information about the learner’s work —
products, displays, events, exhibitions
♦ records of the learner’s contributions to electronic forums and social networking sites
♦ online questionnaires or surveys the learner has used, with summaries of the responses
♦ links to other websites and resources relevant to the learner’s work
18
♦ audio or video clips of learners carrying out assessment-related activity
♦ formal feedback from Assessors on summative assessments activities
E-portfolios for assessment should also include, or link to:
♦ assessment plans for Units
♦ Unit checklists/summaries
♦ records of progress — Units achieved
External Verifiers would also expect to see comments from Assessors about the judgements
made on learners’ evidence. Ideally, this would include positive feedback and comments on
areas the learner needs to develop further.
(SQA 2012, p. 4-5)
SQA also mentions additional items that could be included for assessment (SQA
2012, p.5)
Submissions for assessment and storage of evidence
The ePortfolio platform Mahara, which is used for EUfolio as http://mahara.eufolio.eu/
allows the following procedure for submission of ePortfolio views for assessment,
described as “Soft Assessment Procedure” by Gavin Henrick (2010):
1. The teacher creates a controlled group
2. S/he invites her/his students to the group
3. Users submit their view/page to the group
4. The group admin (usually the teacher) evaluates the view/page, gives
feedback to the user and releases the view/page, which remains
“frozen” from the moment it was submitted until is released by the group
admin.
19
This is a screenshot
showing the group
settings for assessment:
Make sure that the
checkboxes “controlled”
and “allow submissions”
are checked.
This procedure, however, allows storage of evidence only by taking snapshots of the
page that was the basis of assessment. These snapshots can be taken either as
screenshots or by printing the page virtually (as pdf). These storage procedures,
however, do not allow adequate storage of non-printable artefacts (such as videos,
podcasts etc.).
Henrik (2010) understands the “Soft Assessment Process” as a process that serves
formative assessment – given both by peers and/or a teacher/tutor (“group admin”).
As “Hard Assessment Process” he describes the process that leads to summative
assessment – “users” – i.e. the learners – create an ePortfolio view/page which is
pre-structured (via the use of a template) and graded by an authority.
(Henrik 2010)
20
For the “Hard Assessment Process”, the user exports the page her-/himself. The
export function allows the learner to save the page/view in HTML format. It is done
with the “Export” option of Mahara:
In both cases, the evidence for assessment has to be stored in a storage space that
is not the ePortfolio of the learner, as s/he can continue working with the page/view
that forms the evidence, as soon as it was assessed and released by the group
admin/teacher.
The description of these processes shows the strenghts and weaknesses of the
ePortfolio platform Mahara. Mahara allows submission of pages/views/collections via
a group. Upon submission, this evidence cannot be changed any more by the
creator/owner until the assessor (be it the teacher or peer assessor) has assessed
and “released” the submitted items. This is a strength of Mahara, because it ensures
that the evidence remains unchanged during the assessment process.
The weakness of Mahara (and a range of other ePortfolio platforms) is, that after
assessment there is no space provided to store the evidence. The reason of this is
the philosophy behind ePortfolios that make the creator its owner.
However, in most educational settings (like in schools), ePortfolios or parts thereof
have to be stored as evidence for assessment, especially if the ePortfolio forms an
important part of a final and/or state exam. As Henrik’s description of the “Hard
Assessment Process” shows, Mahara ePortfolio views/pages that are substantial part
21
of (summative) assessment, are uploaded to Moodle (a Learning Management
System – cf.: https://moodle.org/) where the evidence for assessment can be stored.
Aaron Wells, Mahara developer, dreams of a “Mahara LMS [=Learning Management
System] plugin” to overcome the described weakness. Among the functionalities he is
envisioning, there are the following:
“- The plugin would come with blocks for different LMS content -- quizzes,
assignments, polls, etc.
- The teacher would create pages in the group, put LMS blocks into them, and then the
students would visit the page to access the LMS content (perhaps linking them from
the group homepage)
- The interaction plugin would also add a "gradebook" tab to the group, from which the
teacher and student could see their grades” (Wells 2014)
NB: Even the creation of a plugin as described above needs an additional LMS (like
Moodle) as a space to store the evidence. (cf. Wells 2014b)
1.4.2 Assessment via the Microsoft ePortfolio platform
The Microsoft ePortfolio platform is based on Office 365 (O365).
Students can elaborate assignments with a range of O365 programs. They can store
the work they want to keep for themselves on OneDrive (cloud storage space).
The basis for classroom work are virtual classrooms on O365. They can be reached
via a login URL and the click on a “tile” with the school’s name. After choosing the
school, the student is asked to choose a class. The tiles of the chosen virtual
classroom are leading to the following activities/offers:
•
•
•
•
•
Notes + Files: This is the place where teachers and students can post their
notes or files, everybody has access to it.
Useful Sites: This is a place where teachers and students can create
discussions and interact with each other.
Assignment DropBox: In this place, students can post their assignments to be
reviewed by the teacher only. Each student can only see their own assignment
but teachers can see everybody’s work.
OneDrive: This is a link to your personal online storage (cloud) [this cloud
storage is connected to the O365 classroom by a link, only]
Pages: This is a place where teachers can create new pages with specific
content, web parts, apps, etc.
Francony/Russell (2014)
If students want to hand in their assignment they upload it to the Assignment
Dropbox. Assignments there can be reviewed by the teacher only. See below the
teacher’s “library” with assignments from her/his students.
22
Submissions for assessment and storage of evidence
Assignment dropbox (teacher’s view): Screenshot taken from Microsoft/H2 Learning (2013): ePortfolio.
[Youtube Video]. https://www.youtube.com/watch?v=hFYC28LyILQ.
When a new document is uploaded, the teacher gets a notification:
Screenshot taken from Microsoft/H2 Learning (2013): ePortfolio. [Youtube Video].
https://www.youtube.com/watch?v=hFYC28LyILQ.
For assessment, s/he has a good interface for placing comments that are directly
placed next to the content s/he wants to comment/to be corrected.
Documents can be shared via OneDrive as well. This allows sharing with internal or
external users by entering their eMail addresses and could be used – for example –
for peer feedback.
23
A more exhaustive user scenario from the teacher’s point of view is given in the
Microsoft in Education publication “Office 365 for education” (2014).
rd
Screenshot taken from Microsoft Office 365 by Julie-Ann Russell (platform role = teacher) on 23 April
2015.
With OneDrive both teachers and students can store and share classroom files.
OneDrive allows teachers and students to create, edit, store and share files including
class assignments and projects to which many students and teachers need access.
Teachers and students can manage which files and folders are shared or kept private
and easily store and share photos, videos, documents and more – anywhere, on any
connected device.
24
2. Guidance Materials for Participants
2.1 For pilot partners
Guidance for EUfolio pilot partners is given in the EUfolio Process Specification (Del.
10) and through the CPD resources elaborated for EUfolio by the CPI. An example:
Avraamidou/Economou (2013). EUfolio – EU classroom ePortfolios. Trainers’ booklet
(Deliverable 16).
2.2 For principals
Principals have to be made acquainted with the ePortfolio process in general. In
addition they have to know the guidance material for teachers in order to train them
or to support CPD done by support staff.
Crucial for principals are furthermore the following legal considerations (taken from
JISC 2008/2012) which are also important for the assessment process:
•
•
•
Who owns the e-portfolio? (see Ownership and Intellectual Property Rights
section as well!)
What data will be added to the e-portfolio by the institution eg from student
database? Who will be responsible for this?
What advice will learners be given about what files not to upload into an eportfolio?
Furthermore the following considerations concerning ownership and IPR (Intellectual
property rights) have to be taken – again following the abovementioned JISC
checklist:
•
•
•
•
•
How will the e-portfolio system authenticate that all the work,
documentation and demonstrations were created by the author?
Who is the real owner of the artefacts in the e-portfolio file repository?
How will intellectual property used in an e-portfolio be protected?
What can or cannot be included in an e-portfolio?
Who owns the learner record (transcript)?
In order to ensure the quality of implementation and to make sure that all
considerations are well discussed, a workshop with principals is recommended. A
slide presentation for a workshop with principals can be downloaded at:
http://mahara.eufolio.eu/artefact/file/download.php?file=12972&view=3991
25
2.3 For teachers
Teachers also have to be made acquainted with the ePortfolio process through
workshops and other CPD measures. They have to understand the changing role in
teaching (from teacher to coach) as stated in the EUfolio Process Specification and in
Avraamidou/Economou (2013). EUfolio – EU classroom ePortfolios. Trainers’ booklet
(Deliverable 16). Furthermore, they should take the following into consideration (Cf.
JISC 2008/2012):
•
What is the purpose of the e-portfolio for learners? Who is going to explain
this to the learner and when?
• How prescriptive will tutors be regarding the use of the e-portfolio and
regarding the artefacts used by learners?
• What effect will it have on the curriculum?
• What programme re-design, and possibly re-validation, will be required?
• What aspects of the e-portfolio will be assessed and at which levels:
module, programme or institutional?
• Will the e-portfolio be integrated within programmes or an additional
optional activity?
• Will it be mandatory?
Source: JISC „Implementing e-portfolios checklist“
(http://www.jiscinfonet.ac.uk/checklists/implementing-e-portfolios/)
Another helpful checklist focusing on the ePortfolio process from the assessment
perspective is compiled on the basis of SQA 2012 (pp.11/12). It can be both a basis
for the work with students and for workshops with teachers:
Teachers should:
• Outline the curriculum and/or the part of it that should be accompanied by
ePortfolio work
• Discuss learning aims, ways to reach these aims and possible evidence for
skills with their students
• Advise learners to generate evidence also out of prior learning (if
applicable) and across different subjects/areas (if applicable)
• Accompany the ePortfolio process of the of the learner as a coach
• Manage peer feedback and provide the students with assessment for
learning
• Help the students, if necessary, to select and organize evidence for
assessment
• Provide a clear scheme for assessment basing on the learning targets that
were elaborated together with the students
• Manage or undertake final assessment (summative and/or formative)
The following illustration shows the model adopted for the Irish EUfolio pilots:
26
(Source: EUfolio: Pilot Schools Update Ireland, April 2014)
2.4 For support staff
ePortfolios require both technical support (for the platform administration and support
for teachers and students) and expert support (usually given by the national teacher
CPD teams)
Technical staff could use the following part of the JISC checklist (2008/2012) as
guidance material:
Hardware and software considerations
• Integration – how will the issues of integrating an institutional MIS and/or the VLE
with the e-portfolio be dealt with and by whom?
• Server performance and storage – scaling up to cope with increasing numbers of
e-portfolio users and the growing size of the e-portfolios as users expand them
over time
• What plug-ins, file formats and browsers will be required or supported?
• What technologies will be used to implement an offline, portable e-portfolio that
authors can take with them? (Extensible Markup Language (XML), content
packaging, etc)
• Service level agreements for future software releases – once the system is being
used on a basis wider than a pilot study and a resilient and reliable delivery
becomes paramount
• What back up systems are in place to ensure operational integrity and disaster
recovery?
27
Support and scalability considerations
• Can the system scale adequately as its usage grows and storage expands?
• Will there be adequate staff to develop, deploy and maintain the system?
• Will there be an infrastructure in place to properly train learners and
administrators how to use the e-portfolio system?
• Will there be adequate online help or will a staffed help desk be required?
CPD support staff can use the training materials as provided by
Avraamidou/Economou (2013) which are available online through the EUfolio trainers
community of practice (http://mahara.eufolio.eu/group/view.php?id=5; for logged-in users
only). The community also maintains forums in which open issues can be discussed.
2.5 For students
Students have to be made acquainted with skills-based teaching in general, with
Portfolios and with the ePortfolio platform chosen. It has to be made sure that
students have both the necessary ICT skills and ICT infrastructure in order to
elaborate their ePortfolios.
As ePortfolios are learner centered, students should be able to negotiate their
learning targets with the teacher and to determine the way in which tey want to reach
them. Once the targets are set, a self-assessment tool (see above) can help the
students to determine their status quo (in the beginning of the process) and to gain
awareness their progress in a later stage or as a kick off for the final assessment.
Furthermore, the table of the SQA as shown above (p. 8 of this publication; table
taken from SQA 2012, p.26) can serve as a good guideline for students’ assessment
operations.
28
Glossary
Assessment For
Learning
The process of seeking and interpreting evidence of a learner’s performance
for use by learners and their teachers to identify where the learners are in
their learning, where their next learning goals are, and what to do next to
achieve them.
Assessor
The person who assesses a candidate’s work.
Formative
Assessment
Assessment that provides developmental feedback to a learner (and perhaps
also their teacher) about an item, a group of items or the topic(s) to which
they relate so that they can adjust their plan for future learning. As such, it
usually takes place during the learning programme (rather than at the end –
summative, or beginning – diagnostic).
In many contexts (as also in the one of EUfolio), formative assessment is
used synonymously with “Assessment for Learning”.
LMS
Learning Management System (for example: Moodle)
Peer Assessment
Assessment of a student by a fellow student or students typically following the
same programme of study. Peer assessors apply criteria and standards of
judgement as other assessors do. This term is usually applied when each
partner in a pair of students assesses the other’s work.
Reliability
In an assessment context, the extent to which a test’s results are repeatable
and fair from one candidate to the next, and from one occasion to the next
(for example with a different set of candidates). I.e. a measure of the
accuracy of the score achieved, with respect to the likelihood that the score
would be constant if the test were re-taken or the same performance were
rescored by another marker, or if another test from a test bank of ostensibly
equivalent items is used.
Many factors affect the reliability of an assessment. Ambiguous instructions to
candidates can make an assessment unreliable since candidates may be
unclear about what it is they are required to do. Badly worded questions may
be interpreted differently by different candidates or even by the same
candidate on different occasions. Vague marking instructions may result in
different markers awarding marks for different reasons or the same marker
awarding marks inconsistently between candidates.
Self Assessment
A judgement a candidate makes about his/her work or level of attainment in
relation to the stated learning outcomes for the activity/programme. Selfassessment is generally used to develop the candidate’s ability to think
critically about his/her learning.
Summative
Assessment
An assessment generally undertaken at the end of a learning activity or
programme of learning which is used to make a judgment on the candidate’s
overall achievement. A key purpose of summative assessment is to record,
and often grade, the candidate’s performance in relation to the stated learning
objectives of the programme.
Validity
The extent to which an assessment tests the actual abilities in the candidate
that it is supposed to test. The appropriateness of the interpretation and use
of the results for any assessment instrument. (E.g. a driving test where a
candidate is observed driving is highly valid. A test where the candidate
describes how they would drive is less valid). There are many different
measures of validity.
If not stated otherwise, definitions are taken from JISC/QCA 2007 (many of them with reference to
“SQA Draft e-Assessment Glossary of Terms“, which could not be retrieved).
29
References
Avraamidou, Antri and Anastasia Economou (2013). EU classroom ePortfolios. Trainer’s booklet.
http://mahara.eufolio.eu/artefact/file/download.php?file=36421&view=9802.
Baumgartner, Peter (2011). Educational Scenarios with E-portfolios - a Taxonomy of Application
Patterns. In: SCO 2011, ed. by Petr Sojka and Martin Kvizda. Masarykova univerzita, Brno, muni
Press, pp. 3-12. Preprint, WWW: http://peter.baumgartner.name/publikationen/listeabstracts/abstracts-2011/educational-scenarios-with-e-portfolios/?aid=1579&sa=1
Slides: http://peter-baumgartner.at/material/slides/eportfolio-taxonomy-brno.pdf
Darr, Charles (2005): A hitchhiker’s guide to reliability. WWW:
http://toolselector.tki.org.nz/content/download/370/4472/version/2/file/Reliability++Hitchhiker%5C%27s+guide.pdf
Darr, Charles (2005): A hitchhiker’s guide to validity. WWW:
http://toolselector.tki.org.nz/content/download/370/4472/version/2/file/Reliability++Hitchhiker%5C%27s+guide.pdf
ePistle (2007). e-Portfolios Informing and Supporting Teaching, Learning and Evaluation. WWW:
https://jiscinfonetcasestudies.pbworks.com/w/page/59135878/ePistle%20Case%20Study
Educational Research Service (2004). Developing and using instructional rubrics. (= ERS Focus on
Research and practices to bring key issues into focus for educators across the spectrum of K12education) WWW:
http://www.nesacenter.org/uploaded/conferences/FLC/2012/handouts/Arpin/ArpinReadingDeveloping
UsingInstructionalRubrics.pdf
Education Services Australia (2008). Assessment for Learning.
http://www.assessmentforlearning.edu.au/
EUfolio. Team Ireland Project Update Cyprus 2014. http://prezi.com/p1dq9vzfwqgk/team-irelandproject-update-cyprus-2014/?utm_campaign=share&utm_medium=copy
EUfolio Trainers – Community of Practice (2013ff). WWW:
http://mahara.eufolio.eu/group/view.php?id=5 (for logged-in users only).
European Communities (2007). Key competences for Lifelong Learning. A reference framework.
Luxembourg: Office for Official Publications of the European Communities. WWW:
http://ec.europa.eu/dgs/education_culture/publ/pdf/ll-learning/keycomp_en.pdf. Last accessed: 2014-09-22.
EUfolio Functional Specification, Generic Functional Specification – see:
Herber, Erich and Andrea Ghoneim (eds., 2013)…
EUfolio Process Specification. 2014 (= EUfolio deliverable 10, final draft).
http://mahara.eufolio.eu/view/view.php?id=9951
Francony, Tim and Julie-Ann Russell (2014). O365 Workshop in Cyprus. 24th September 2014.
/Microsoft. [handout, print, no page numbers. Archived at the Department of Interactive Media and
Edicational Technologies, Danube University Krems in the folder “EUfolio”].
Hallissy, Michael, Deirdre Butler, John Hurley and Kevin Marshall (2013). Redesigning Education:
st
Meeting the Challenges of the 21 Century. WWW:
http://www.teachnet.ie/Blog/Lists/Posts/Attachments/478/MSLearningPaperMay13_1_1B3354D7.pdf
Henrick, Gavin (2010): Some examples of Mahara usage. Presentation at IMOOT 2010. WWW:
http://www.slideshare.net/ghenrick/a-case-studies-presentation (2013-07-15)
Herber, Erich and Andrea Ghoneim (eds., 2013). EPortfolio for EUfolio. Generic functional
Specification (= EUfolio deliverable 11, version 1.0). http://mahara.eufolio.eu/view/view.php?id=10270
Himpsl-Gutermann, Klaus (2012). E-Portfolios in der universitären Weiterbildung. Studierende im
Spannungsfeld von Reflexivem Lernen und Digital Career Identity. Boizenburg: Verlag Werner
Hülsbusch.
ITL Research/Microsoft: 21CLD Learning Activity Rubrics. o.J.
http://www.itlresearch.com/images/stories/reports/21cld%20learning%20activity%20rubrics%202012.p
df
JISC/QCA (2006). e-Assessment Glossary (Short).
http://www.jisc.ac.uk/uploaded_documents/eAssess-Glossary-Short-v1-01.pdf. Last accessed: 201409-22.
30
JISC (2008). Effective Practice with e-Portfolios. Supporting 21st century learning.
http://www.jisc.ac.uk/media/documents/publications/effectivepracticeeportfolios.pdf
JISC (2008/2012). Implementing e-portfolios checklist
http://www.jiscinfonet.ac.uk/checklists/implementing-ePortfolios/. Last accessed: 2014-09-22.
Microsoft/H2 Learning (2013): ePortfolio. [Youtube Video].
https://www.youtube.com/watch?v=hFYC28LyILQ.
Microsoft in Education (2014). Office 365 for Education. User Scenario:
Using SharePoint and Office 365 for class projects and returning/marking assignments.
http://www.schoolnet.org.za/PILP/office365/sharepoint-education5-submissions.html
NCCA = National Council for curriculum and Assessment (2012). A Framework for Junior Cycle.
WWW: https://www.education.ie/en/Publications/Policy-Reports/A-Framework-for-Junior-Cycle-FullReport.pdf.
NCCA (2014). Key Skills of Junior Cycle.
http://www.juniorcycle.ie/NCCA_JuniorCycle/media/NCCA/Documents/Key/Key_Skills_2014.pdf
Process Specification (Del. 10) – see: Ghoneim, Andrea and Erich Herber (eds., 2013).
QUT – Queensland University of Technology (2011). Australian ePortfolio Toolkit.
http://www.eportfoliopractice.qut.edu.au/information2/toolkit/index.jsp (and sub-pages, in the text
quoted with the URL).
Rate, Nick (2008). Assessment for Learning & ePortfolios. What are the formative benefits of
ePortfolios? http://www.core-ed.org/sites/efellows.org.nz/files/nick-eportfolios.pdf
Radziwill, Nicole (2012). The rubric as a general purpose quality tool. In: Quality and Innovation [blog].
WWW: http://qualityandinnovation.com/2012/02/02/the-rubric-as-a-general-purpose-quality-tool/
SQA – Scottish Qualifications Authority (2007). E-Assessment. Guide to effective practice. WWW:
https://www.sqa.org.uk/sqa/files_ccc/guide_to_best_practice.pdf
SQA – Scottish Qualifications Authority (2012). Guidance of Using E-portfolios. WWW:
http://www.sqa.org.uk/sqa/files_ccc/E-portfolios%20guidance.pdf.
Vogt, Joke and Natalie Pareja Roblin (2010). 21st century skills. Discussion paper. Enschede, Univ.
Twente. WWW:
http://www.internationalsymposiumoneducationalreform.com/storage/21st%20Century%20Skills.pdf.
Last accessed: 2014-09-22.
Wells, Aaron (2014a). Mahara LMS plugin (a modest proposal). In: Mahara Community. Forums->
Developers. 11 April 2014, 12:37 PM, https://mahara.org/interaction/forum/topic.php?id=6242
Wells, Aaron (2014b). Assessing Mahara portfolios in Moodle. Changes to the Mahara assignment
plugin. http://de.slideshare.net/aaronwells52/moodle-moot-nz-2014-aaron-wells
All Internet sources last accessed 2015-04-14, unless mentioned otherwise.
31
Project partners:
Project no.
535910-LLP-1-2012-1-IE-KA1-KA1ECETB
Deliverable number
12
Work Package
4
Work Package Leader
Department of Interactive Media and Educational Technologies,
Danube University Krems (Austria)
Dissemination level
Internal document
Delivery date
30/04/2015
Status
Final
(which is, however, made accessible to the public)
Andrea Ghoneim (Danube University Krems)
Author(s)
with contributions from:
EUfolio project partners
(Special thanks to Ben Murray, National Curriculum and Assessment Commission,
Ireland, Tanja Rupnik Vec, Zavod RS za šolstvo, Slovenia, and Bernhard Ertl,
Danube University Krems)
Location (URL)
This document is embedded in the ePortfolio showcasing
EUfolio’s WP4: Pilot design: http://bit.ly/1DKM15S
This project has been funded with support from the European Commission.
This publication [communication] reflects the views only of the author, and the Commission cannot be held responsible for any use which
may be made of the information contained therein.
32