Evaluation report from external testing of services

Transcription

iQVET
Innovative Peer Learning Assessment System for
Evaluation of trainers and Quality VET professional
programs
!
LEONARDO DA VINCI Transfer of innovation
2012-1-NO1-LEO05-04776
WP 5 – Validation of methodologies and learning
activities
D 10 - EVALUATION REPORT FROM EXTERNAL
TESTING OF SERVICES AND METHODS BY
PROFESSIONALS
Authors: Liviu MOLDOVAN, Stela BATAGA, Ofelia BACULEA,
Eugenia NOVAC, Anca GIURGIU, Iwona Kostrzewa,
Hariklia Tsalapatas, Audun Grøm, john B. Stav
Co-Authors: Aleksandra Mykowska
!
Date: 30.11.2014
"!
Deliverable
10 – Evaluation report from external testing
of services and methods by professionals
This project has been funded with support from the European Commission. This publication reflects the views
only of the authors, and the Commission cannot be held responsible for any use, which may be made of the
information contained therein.
!
!
!
iQVET - Project „Innovative Peer Learning Assessment System for
Evaluation of trainers and Quality VET professional programs”
LdV-Project number –2012-1-NO1-LEO05-04776
!
!
Work package number: 5
Work package title: Validation of methodologies and learning activities utilizing
"Pele Professional"
Start month: 01/2013
End month: 09/2014
Package leader: Chamber of Commerce and Industry Mures, Romania
Language of the report: English
This document may not be copied, reproduced, or modified in whole or in part for any
purpose without written permission from the iQVET consortium. In addition to such
written permission to copy, reproduce, or modify this document in whole or part, an
acknowledgement of the authors of the document and all applicable portions of the
copyright notice must be clearly referenced. All rights reserved.
This document may change without notice, but consortium members should be
informed, and number of version, stage and date should be given
Project consortium:
P0 – HiST – Sør-Trondelag University College Trondheim - Norway
P1 – HCR – HiST Contract Research Trondheim - Norway
P2 – DOM - Dom Szkole! i Doradztwa Krakow – Poland
P3 – UTH - University of Thessaly Volos - Greece
P4 – CCI – Chamber of Commerce and Industry of Mures County – Romania
!
!
2
Deliverable 10 – Evaluation report from external testing
!
!
!
TABLE OF CONTENTS
1 About the deliverable
2 iQvet project executive summary
3 Target groups
4 Evaluation and validation products
5 Evaluation at Sor-Trondelag University College
6 Evaluation at Dom Szkolen I Doradztwa
7 Evaluation al University of Thessaly
8 Evaluation at Chamber of Commerce and Industry of Mures County
!
!
!
3
!
!
1. ABOUT THE DELIVERABLE
The deliverable reports on the delivery of various activities conducted by the
entire partnership during the lifetime of the project in relation to the validation of the
project results.
Validation activities have a particular weight and cover a wide geographical
area in south, central, eastern, and northern Europe.
The aim of this deliverable is to ensure that the evaluation of the project
activities resulted in:
•
•
•
•
•
•
•
•
•
a continuous feedback during the project lifecycle on the activities that
are carried out
that the software development is done according to the training needs
identified by consortium partners
that the planned courses are being delivered and evaluated
professionally according to the proposed methodology
a summary of future actions based on the findings in this project
a consistent recommendation for future actions
a sound and unbiased evaluation of the different tasks
a project report that can give all stakeholders recommendations for
future actions
a feedback to lecturers for future developments in train-the -trainer
programmes
recommendations for future enhancements of the PeLePro employment
Internal Evaluation has been achieved through regular progress monitoring by
consortium partners. Implementation milestones have been reviewed during each
consortium meeting and have been used as a means for progress monitoring.
During the bi-annual consortium meetings the partners set interim (6-month)
implementation goals and review interim results. Interim project results were further
made available to the consortium in the restricted areas on the project web site,
allowing partners to review and provide feedback based on their specific expertise.
Based on the above findings the final reports have circulated between the project
partners for their evaluation and comments as well as circulated to external experts
for their comments.
4
!
!
2. iQVET PROJECT EXECUTIVE SUMMARY
In nowadays VET during short VET courses addressing the World of Work
teachers are currently unable to judge and assess on the fly professionals trainer`s
reproduction, demonstration and performance of learning. Instructors can't answer
questions like
- How successful is/was my teaching?
- How well do/did the students learn the subject taught?
- How much do/did the students learn of the subject taught?
In class communication, interaction and collaboration processes that may
enhance effective peer learning processes, where students learn from their peers,
are difficult to measure by use of existing ICT technologies
The brand new Peer Learning Assessment Service (PeLE) system for
Smartphones, PAD, PC and Mac developed in the KA3 Done-IT project (2011-2012)
gives the teacher a new tool, allowing him/her to either give verification or elaborative
feedback to individual students or groups of students immediately after a test. When
they still remember the questions in the test, students will learn why the answers are
correct or incorrect. Thus, mobile technology provides new evaluation, assessment
and testing criteria within education and training
The aims of the iQVET project are:
•
•
to provide instant evaluation in industry oriented short, intense 2-3 days VET
professionals courses, by adapting, testing and validating the new online
service "PeLe Professional".
extend Kirkpatrick`s learning outcome evaluation model by using modern,
easy to use, cheap mobile response technology, to include Assessment for
Learning in competence based systems
3. TARGET GROUPS
•
VET providers or universities that run short 1-3 days skills upgrading courses
in order to train professionals in the World of Work, with their trainers and
teachers
•
companies and organizations that provide on site (inside the organization) or
off site (at an external site like at a hotel) short courses to personnel, with their
instructors
•
instructors, teachers, students, trainees in technical and vocational secondary
education
5
!
!
The profile of the different groups participating in the iQVET project may be
quite different depending on their roles, tasks and activities in the project. They may
also have different interest of use of the project product PeLePro, creating different
biases to the content development and the delivery structure of the content
•
•
•
•
Trainers, teachers in VET: They are personnel working in the school system
covering both traditional education and also lifelong learning with adult
personnel for training. Their competence profile is specialist with documented
competence from fabrication.
Instructors, trainers in industry: They are personnel working in professional
training centres either organized as a training institute or as a training branch
in an industrial company. Their competence profile is industrial specialist or
engineer with special education or experience in production.
Trainees, students in VET: Most students are in the age group of 18 – 30
years. Background will be from the school system or from public supported
educational programmes. Their reason for seeking education and training is to
get an education or being promised a job.
Trainees in industry: A mixture of male and female students in the age group
of 25 – 45 years. They have little or no background from industry or from other
educational areas and weak background from industry. Their reason for
seeking education and training is promotion or seeking a job.
4. EVALUATION AND VALIDATION PRODUCTS
Main validated products are:
•
•
The One2act service solution on using the immediate feedback on test results
from short and intense industry oriented VET courses targeting professionals
adapted to include support for the 4 level Kirkpatrick's evaluation model
The 4 level Kirkpatrick's evaluation model of courses
6
5. EVALUATION AT SØR-TRØNDELAG UNIVERSITY COLLEGE
Students have been engaged in the evaluation during the whole project. Students
have tested the software, feedback has been collected and used to upgrade and
improve the software, and then a new round with testing in classes. This has been
repeated many times. When most of the errors have been removed, partners have
been invited to test the latest version.
The figures below shows an example from use of EVAL in a VET course for staff that
works in companies that are members of Norwegian Industry. The courses were
thought during fall 2014, and consist of a combination of e-learning and on-site
training. It is necessary to e-learning since the students are in full job in their
companies. During a period of three years they study to become an engineer while
they are in full job. This requires flexible training paths, and high effect of the on-site
training part. All students have at least 5-10 years of experience in the company.
The examples below use the final version of EVAL in a VET class with 57 students:
Figure 5.1. An overview of the evaluation results. 50 out of 57 students have
answered.
Each gathering lasts 2-3 days, and there is ca one gathering per month. During this
on-site training component, the lectures include 2-3 different subjects. In the example
shown here, the teacher who teaches a course in economy for engineers, starts the
#!
Deliverable
!
!
!
gathering by asking a few knowledge questions such that the teacher knows what the
class is good at. In this why he may focus more of his time towards the problematic
areas. At the end of the gathering, he runs a knowledge-oriented evaluation in the
class. The figures below show the distribution on each question:
8
!
!
9
!
!
The figures below show the open text responses.
10
!
!
The figure above shows the matrix view.
The figure above shows the results view.
11
6. EVALUATION AT DOM SZKOLEN I DORADZTWA
Dom Szkole! i Doradztwa undertaken some activities in order to validate the methodology and
support the evaluation process of the trainings and courses based on 4 levels Kirkpatrick evaluation
methodology. We also took actions in the adaptation of the innovative system to the new training
method. We have been involved in the testing phase of the tool. Thanks to it, we have gathered many
ideas how to support the trainings activities, their attractiveness and mobility as well as the evaluation
process.
!"#$%#&'()*+,*#)&+*
-.+*$+#/)')0*1/(2+33*#)4*
()*0(')0*+"#$%#&'()*
!"#$%#&'()*+,*1(3&*
5(61#/'3()*(7*&.+*
/+3%$&3*#)4*2()2$%3'()3*
The trainings with the use of the EVAL tool were organized by Dom Szkole! i Doradztwa and were
used at Jagiellonian University, Language school, train the trainers workshop, training of a career
counselor and the methodology was also used as a part of an e-learning course. Explanation below
refers to only 3 of 5 courses as the conclusions are the same.
During the first training from psychology course and we have used the EVAL tool and we have tested
the Kirkpatrick methodology. Approximately 23 students took part in the pre-test and the same
amount of students managed to answer the questions in the post-test. Questions have been carefully
"$!
Deliverable
!
!
!
prepared by the teacher in order to measure the knowledge before the lecture. Check the knowledge
after the lecture and to measure the changes in behavior.
pre-test:
post-test:
pre-test:
post-test:
pre-test:
post-test:
13
!
!
As it’s visible in the print screens the level of the knowledge increased. The lecturer observed that not
only the learning effectiveness and the students’ motivation increased but also the teacher noticed the
change of attitudes and behaviors of students of the processed material. This has been observed in
subsequent teacher’s observations of the students taking part in the evaluation process. After the
knowledge implementation that students have acquired during the lecture, they were asked the simple
question of what they have learnt? It forced them to recall the acquired knowledge and made them
focused on the reflection of what they managed or failed to implement in real life. It also forced them
to rethink what they remember and thereby increased the efficiency of the learning process.
Second course during which we used the tool and we have included the methodology was the language
course. 5 students took part in it. The course was constructed in such way that users first were asked to
do pre-test, next the course was conducted, students then were asked to write post-test and after 5 days
their change of attitudes and behaviors was checked through roll plays exercises.
14
!
!
pre-test :
post-test:
pre-test:
post-test:
Third course during which we have tested the methodology was part of an e-learning course. Users had
15
!
!
to do pre-test on mobile devices using EVAL which was done before they got access to the e-learning
lesson and post-test which was done after the e-learning course.
16
!
!
pre-test:
pre-test:
post-test:
post-test:
17
!
!
People who conducted the training were workers of a big company. Unfortunately it was additional
task for the employees and only 7 out of 10 people finished the course. The supervisor who noticed
some changes in the employees’ behavior, checked the change of attitudes and behaviors during his
evaluation observations. We need to remember that measurement of behavior change is less easy to
quantify and interpret than reaction and learning evaluation.
18
7. EVALUATION AL UNIVERSITY OF THESSALY
University of Thessaly exploiting existing working ties engaged different groups of
end users in training sessions with the different available versions of EVAL software
(ONE2ACT). More precisely, training sessions with the EVAL software were
organised in the Department of Agriculture, Crop Production and Rural Environment
of the University of Thessaly, in the Department of Electrical and Computer
Engineering of the University of Thessaly, in the Vocational School of Agria in Volos,
in the Vocational School of Sparta and in the 2nd Vocational School of Karditsa. It is
estimated that 23 teachers/trainers organised or participated in trainning sessions
with the EVAL software and more than 350 trainnees used EVAL software in the
context of scheduled training sessions.
In overal it was opted that EVAL software contributes to the modernization of
education through the introduction of innovative solutions that combine well accepted
pedagogical methodologies, such as the Kirckpatrtick learning design evaluation
model, with emerging technology, namely mobile learning towards better addressing
the learning needs of trainnees.
TRAINING SESSIONS AT THE DEPARTMENT OF AGRICULTURE CROP
PRODUCTION AND RURAL ENVIRONMENT, UNIVERSITY OF THESSALY!
Evaluation activities at the Department of Agriculture Crop Production and Rural
Environment of the University of Thessaly started in the Spring of 2014. More
precisely, EVAL software was used in the context of the undergraduate course “Postharvest handling facilities for products of crop production”, which is offered in the 4th
year of studies. The training sessions were carried out by Prof. Constantinos Kittas,
Dinos Ferefinos, Nikolaos Katsiulas, Christos Lykas and Thomas Batzianas who are
all active in research and training.
The trainers prepared quizes that aimed at exploring students’ knowledge and
skills gained. Each quiz was conducted twice, initially at the beginning of the class
and then at the end of the class. The goal was to compare the results of the two
repetitions of each quiz and record students' improvement in the specific class
material after the lecture was given, so that valuable conclusions could be drawn
regarding knowledge absorption by the students, knowledge transmissibility by the
instructor, quality of the lecture presentation, etc. In addition, possible problematic
areas of the class material could be identified, so that they could be improved in
future lectures.
"%!
Deliverable
!
!
!
Quiz no. 1:
The first quiz was given on the second section (out of three) of the material of the
course that concerned the programming language of MATLAB (a computational tool
for solving the courses exercises). There were about 25 students attending the class,
but only 10 had smart-phones. From them, only 6 managed to login into the system,
as those running Symbian and Windows Mobile did not get in log-in screen. These 6
students completed both quizzes and the results (Figures 1 and 2) show a 33.5%
improvement in their performance (from 16.5% correct answers at the beginning of
the class, to 50% correct answers at the end of the class). The 5 questions of the
quiz (with 4 possible answers each) concerned introductory programming material
and specific syntax rules of MATLAB.
Figure 1. Results of 1st repetition of Quiz no.1 (16.5% correct / 67% wrong / 16.5%
unanswered)
Figure 2. Results of 2nd repetition of Quiz no.1 (50% correct / 33.5% wrong / 16.5%
unanswered)
Quiz no. 2:
The second quiz was given on the third (and final) section of the material of the
course that concerned the programming language of MATLAB. There were about 25
20
!
!
students attending the class, but only 10 had smart-phones. From them, only 6
managed to login into the system and complete the first repetition of the quiz, while 5
of them completed the second repetition at the end of the class. The results (Figures
3 and 4) show a 14.5% improvement in their performance (from 33.5% correct
answers at the beginning of the class, to 48% correct answers at the end of the
class). The 5 questions of the quiz (with 4 possible answers each) concerned
decision structures and functions of MATLAB.
Figure 3. Results of 1st repetition of Quiz no.2 (33.5% correct / 66.5% wrong)
Figure 4. Results of 2nd repetition of Quiz no.2 (48% correct / 52% wrong)
Quiz no. 3:
The third quiz was given on the “Storage of fresh production” section of the course.
There were about 20 students attending the class, but only 4 had smart-phones and
managed to login into the system and complete both repetitions of the quiz. The
results (Figures 5 and 6) show a 30% improvement in their performance (from 60%
correct answers at the beginning of the class, to 90% correct answers at the end of
the class). The 5 questions of the quiz (with 4 possible answers each) concerned the
characteristics of storage facilities for fresh fruit and vegetables, which is one of the
main sections of the course.
21
!
!
Figure 5. Results of 1st repetition of Quiz no.3 (60% correct / 40% wrong)
Quiz no. 4:
The fourth quiz was given on the “Calculation of vapor displacement in walls” section
of the course. There were about 20 students attending the class, but only 6 had
smart-phones and managed to login into the system. Four of them completed the first
repetitions of the quiz and all six of them completed the second repetition. The
results (Figures 7 and 8) did no show any improvement in their performance. From
60% correct answers at the beginning of the class, it dropped down to 53.3% correct
answers at the end of the class, while there was a 30% of unanswered questions in
that second repetition (all questions were answered in the first repetition). The
introduction of 2 new students in the second repetition of the quiz (an increase of
50% on the initial participation) makes the comparison of the results and its
interpretation rather problematic. The 5 questions of the quiz (with 4 possible
answers each) concerned the analytical calculations of the displacement of vapor in
several materials inside the walls of storage facilities.
22
!
!
Figure 8. Results of 2nd repetition of Quiz no.4 (53.3% correct / 16.7% wrong / 30%
unanswered)
Quiz no. 5:
The fifth quiz was given on the “Energy and mass exchange in storage chambers”
section of the course. There were about 15 students attending the class, but only 4
had smart-phones and managed to login into the system and complete both
repetitions of the quiz. The results (Figures 9 and 10) show a 35% improvement in
their performance (from 55% correct answers at the beginning of the class, to 90%
correct answers at the end of the class). The 5 questions of the quiz (with 4 possible
answers each) concerned the calculations required to estimate energy and mass
(water and CO2) exchanges in storage chambers.
23
!
!
Quiz no. 6:
The sixth quiz was given on the “Packaging of products of crop production” section of
the course. There were about 10 students attending the class, but only 2 had smartphones and managed to login into the system and complete the first repetition of the
quiz, while only one of them comleted the second repetition. The results (Figures 11
and 12), even though they show a 25% improvement in the performance (from 75%
correct answers at the beginning of the class, to 100% correct answers at the end of
the class), they cannot be considered reliable, because of the very small sample
size. The 4 questions of the quiz (with 4 possible answers each) concerned general
material on packaging methodologies for agricultural products.
24
!
!
Figure 12. Results of 2nd repetition of Quiz no.6 (100% correct)
Quiz no. 7:
The seventh quiz was given on the “Controlled environment for storage” section of
the course. There were about 10 students attending the class, but only 3 had smartphones and managed to login into the system and complete both repetitions of the
quiz. The results (Figures 13 and 14) show a 13% improvement in their
performance (from 27% correct answers at the beginning of the class, to 40% correct
answers at the end of the class). The 5 questions of the quiz (with 4 possible
answers each) concerned the conditions and the required equipment for the storage
of agricultural products in controlled environment.
25
!
!
Figure 14. Results of 2nd repetition of Quiz no.7 (40% correct / 10% wrong / 50%
unanswered)
Table 1 summarizes the results of the seven quizzes. Some basic statistical results
are shown in Table 2. It can be seen that 85.7% of the quizzes gave an improvement
in the second repetition of each quiz, while if the problematic quizzes are excluded,
there was always an improvement in the second repetition. The average
improvement between first and second repetition of all quizzes was 20.7% (25.3%, if
the problematic quizzes are not taken into account)
26
!
!
Table 1. Summary of quizzes' results (red font represents problematic situations, as
explained in the text).
Quiz
1
2
3
4
5
6
7
1st
repetition
participan
ts
6
6
4
4
4
2
3
2nd
repetition
participant
s
6
5
4
6
4
1
3
1st repetition
correct ans.
(%)
2nd repetition
correct ans. (%)
Improvement
(%)
16.5
33.5
60.0
60.0
55.0
75.0
27.0
50.0
48.0
90.0
53.3
90.0
100.0
40.0
33.5
14.5
30.0
-6.5
35.5
25.0
13.0
Table 2. Overall statistical results
Improved quizzes
Average improvement
Over all quizzes
(%)
85.7
20.7
Problematic quizzes
excluded (%)
100
25.3
!
The software was also used in Autumn 2014 with additional group of trainnees.
The trainers that were involved enjoyed the breadth of available functionality ranging
from different question types to exploiting statistical information on learner
responses. The Department of Agriculture of the University of Thessaly considers
that Eval software offers the potential of enhancing existing educational practices
through the generation and analysis of real-time information provided by learners
towards better linking the course offerings to learning needs related to the world of
work.!!
!
TRAINING SESSIONS WITH EVAL IN THE DEPARTMENT OF ELECTRICAL AND
COMPUTER ENGINEERING OF THE UNIVERSITY OF THESSALY
The different available versions of EVAL software (ONE2ACT) were used in an
ongoing manner in the Department of Electrical and Computer Engineering of the
University of Thessaly. More precisely EVAL software was used
1. in the context of ‘Educational Technologies’ course in the fall 2014 engaging
130 higher education students and 2 trainers Prof. Catherine Houstis and
Hariklia Tsalapatas
2. in the context of ‘Programming I’ course engaging almost 200 higher
education students and two trainers Prof. Christos Antonopoulos and Vana
Ntoufexi.
27
!
!
3. in the context of ‘Mathematics I’ course engaging almost 200 higher education
students and one trainer Prof. Giota Tsopanopoulou
It is worth noting that the attendance rate of the three courses is very high. However,
not all the students had access to smart-phones and managed to login into the
system at the same time.
The following figure demonstrates the codes of indicative training sessions with
EVAL software in the context of the course ‘Educational technologies’.
!!
!
"#$%&'!(!)*+',!*-!#.+#/01#2'!1&0#..#.$!,',,#*.,!3#14!4#$4'&!'+%/01#*.!,1%+'.1,5
The questions that were raised aimed at gaining an insight into learners’ reactions
in relation to their training. This information is particularly important for the
instructor/trainer as it can inform future improvements, including for example new or
updated thematic topics, more examples, additional practical tasks and more. Some
of the questions focused also on the knowledge and skills gained. An additional set
of questions aimed at exploring any change in behaviour or view. Measuring
behaviors is not a straightforward process.Identifying changes in behaviour usually
requires time and specific conditions. If a change in a behaviour has not been
observed this does not mean that skill building and training was innefective or that a
change may not occur in the future.
28
!
!
In the context of the ‘Educational technologies’ course the questions that were raised
regarding skills and knowledge gained were related to usability issues,
characteristics of educational software, pedagogical theories and trends (i.e.
gamification). They were also questions about specific educational software in the
area of science education and language learning. Multiple choices and open text
questions were addressed.
!!!!
!
!
!
"#$%&'!6!71%+'.1,8!&',9*.,',!1*!,':'/1'+!;%',1#*.,5
The questions raised also aimed at setting a basis whereupon discussion in the class
could occur. Exploiting EVAL software, the instructor was raising questions that
required both knowledge and critical thinking. Indicative questions include ‘In your
opinion what makes a good educational system?’ or ‘To what extent Modelus
supports secondary science education?’ and more. The instructor then based on the
submitted answers was raising a discussion in the class, introducing more
pedagogical theories, ideas and educational dilemmas. Through this process a basis
was being set whereupon the student could reconsider existing views and
perceptions.
Undisputedly, in more practical and technical subjects such as ‘Mathematics I’ and
‘Programming I’ a theoretical debate is not in line with the training objectives.
However, there was an agreement among the trainers that the feedback gained was
ideal for evaluating training effectiveness. The trainers could gain a direct
feedback on the quality of training that they offer.
29
!
!
In overal, it seems that EVAL software provides an excellent medium for
organizing training sessions in a manner that best meets learners’ needs
through on-going and post completion evaluation.
TRAINING SESSIONS WITH EVAL IN THE VOCATIONAL SCHOOL OF SPARTA
EVAL software was used in the context of ‘programming courses’ in the Vocational
school of Sparta. Two ICT teachers, Ilias Nikolaros and Anna Vasala, integrated the
software into their teaching practices. Approximately 18 vocational students used
the software.
The ICT teachers used the EVAL software to gain an insight into students’ reactions,
learning and attitudes. In overall, EVAL was seen as an innovative way to maximise
students’ participation and meaningful engagement in the class. In addition, useful
feedback was generated that could directly inform the adjustment of the learning
process.
!
"#$%&'!<!=4'!#.1'&-0/'!*-!14'!1'0/4'&!>:#0,!?#@*:0&*,5
The quizzes that were initially released were aimed at generating feedback on
students’ needs and feelings about their learning experience so far. Multiple
choices and open text questions were addressed. Indicative questions were:
1. In which of the following subject you feel that you need more support?
2. Would you be interested in designing programs in visual environments?
3. Which of the following techniques can help you better understand the subject
taught?
4. I feel that “programming” is a useful tool for describing, analysing and solving
problems.
30
!
!
"#$%&'!A!B',%:1,!-*&!14'!A!;%',1#*.,!099'0&!0C*2'5
Students’ responses (see figures above) show that there is an agreement that
programming courses are the most challenging and thus additional support is needed
towards this direction. The majority of the students also feel that practical tasks in the
school laboratory help them better understand programming concepts. The students
are interested in further exploring the area of visual programming. Last, the students
are moderately convinced about the value of programming as a process for
describing, analysing and solving problems.
The next set of questions that were addressed focused on the knowledge gained.
The aim was to identify problem areas or misunderstandings and to support students
in overcoming cognitive difficulties. It was important for the teachers to see whether
knowledge or capacities were increased and to identify issues and concepts that
raise students’ confusion. Did the trainees understand the programming concepts
introduced? This was the core question. The teachers raise relevant questions aims
at exploring students’ understanding on core concepts and basic theoretical aspects.
The figures below show students’ responses (see figures below). The teachers
based on students’ answers re-organised the course, provided additional examples
and explanation and enriched the teaching process with more practical tasks. It was
opted that few students faced difficulty in explaining how a variable operates; other
problem areas included the TDM technique and the characteristics of Visual Basic
programming language.
31
!
!
!
"#$%&'!D!"*/%,!*.!14'!:'0&.#.$E!7':'/1'+!;%',1#*.,!0.+!,1%+'.1,!&',9*.,',5
The last set of questions aims at exploring the extent to which the trainees changed
their views. An attempt is also made to see whether learners closely associate
the knowledge gained with real life. Can the learners identify links among what
they are being taught and the reality? Can they recognise the applicability of
knowledge in daily activities?
Noteworthy, it was opted that participants identify links to reality. They can see
programming concepts in many software applications that they use in daily life. The
students are referred to a range of computer games that are based on algorithms
and programs. They also stress the fact that algorithms and the process of
programming is behind social networks and media. The students also recognise the
value of logical operations for making claims about logical statements.
32
!
!
!
"#$%&'!F!71%+'.1,8!&',9*.,',
The students also closely relate programming to the development of digital skills,
problem solving skills, analytical skills and creative skills that are useful both in daily
life but also in the world of work. A change in their view as far as the value of
programming is concerned was also observed. More precisely, all the students
appear now convinced about the value of programming, as a tool for describing,
analysing and solving problems (see figure below).
!
"#$%&'!G!71%+'.1,8!&',9*.,',5!
The two school ICT teachers expressed the idea that EVAL software triggers
mechanisms towards higher student participation in the class. It also allows
teachers to monitor more closely the learning process and to gain an insight into
33
!
!
students’ thoughts, experiences, needs and perceptions. Last EVAL software results
to increased student satisfaction, as students think that interactive tools like EVAL
make learning more relevant and meaningful to them.
TRAINING SESSIONS WITH EVAL IN THE VOCATIONAL SCHOOL OF AGRIA
EVAL software was used in the Vocational school of Agria in the area of Volos.
More precisely, the project members trained the ICT teacher of the school, Dr. Eleni
Tousidou, on how to use the EVAL software. The class teacher recognised that the
EVAL software can potentially enhance the learning process and agreed in using it in
her class. 18 vocational students were exposed to EVAL software in the context of
the course: ‘programming in Visual Basic’. Some students work in pairs due to limited
number of desktop computers.
The ICT teacher used the EVAL software to gain an insight into students’
expectations, knowledge and perceptions. Multiple choice questions and open text
questions were formed.
!
!!
!
"#$%&'!H5!7/&''.,4*1!*-!1'0/4'&8,!#.1'&-0/'
The class teacher commented positively on EVAL software. She also observed that
a fruitful dialogue was on among the students that were working in pairs. The class
teacher stresses the fact that the software encourages learners to participate
actively in the learning process while also showing to the teacher possible
directions for improvement of the teaching practices.
34
!
!
EVAL SOFTWARE USE IN THE INSTRUCTOR TRAINING WORKSHOP
An instructor training workshop was organised in the vocational school of Karditsa on
October 24, 2014 attracting the interest of 10 trainers (Konstantina Galani, Ioanna
Koutouva, Aikaterini Giannakoudi, Nikos Ioannou, Ourania Petropoulou, Vasilis
Argyris, Kon/nos Karydas, Euaggelia Pappa, Charalabos Liatos and Ioannis
Papaioannou) from the city of Karditsa and the city of Trikala. 10 vocational
students (Paulos Mantziaris, Christos Karagiannakis, Kon/nos Tsiamantas, Denis
Rousta, Egkesin Miftari, Stefanos Ntourakis, Lambros Noukas, Mary Ververa, Donka
Paikova, Semili Gionous). The questions that were raised were related to foreign
language learning and the use of ICT tools given the fact that many foreign language
teachers attended the workshop.
!
"#$%&'!I!=&0#.'&,!0.+!1&0#..'',!0&'!%,#.$!JKLM!#.!14'!/*.1'N1!*-!14'!>.,1&%/1*&!=&0#.#.$!O*&@,4*95
Although this was not a longitudinal study, it allowed the project members to gather
qualitative feedback on the relevance of the EVAL software. Aiming at gaining an
insight into participants’ views questions related to the use of mobile applications for
educational purposes were raised prior and in the end of the seminar.
Participants were initially sceptical about the use of mobile applications in the class.
However, by the end of the workshop and after the practical exposure to EVAL
software the participants were engaged in a dialogue about the educational value of
EVAL software and its potential integration in the class. This fact is also exemplified
by the following statement.
‘The EVAL software provides space for communication at several levels as it can
support you in understanding not only students’ needs but also students’ views
and perception in relation to the training that you offer’
Participant teacher
It is worth noting that the participants in the study stressed the fact that EVAL
software offers many advantages for the teacher for the ultimate benefit of learners
as it allows the continuous adaptation of the instructional method to the needs of the
learners. It also provides students with the opportunity to regulate the learning
35
!
!
process so that to be in line with their needs, expectations and desires. The
participant teachers were very willing to apply the EVAL software in their class.
36
8. EVALUATION AT CHAMBER OF COMMERCE AND
INDUSTRY OF MURES COUNTY
a)-Employment of Eval software at CCIM
CCIM has used the different available versions of EVAL software (ONE2ACT),
in ordinary training sessions of different course delivered in Vocational Education and
Training, on subjects like: quality management, quality auditors, quality professionals,
project management, environmental inspector, human resources inspector etc.
CCIM.
Figure 1 demonstrates the codes of sessions employing EVAL software at
!
!
&#!
Deliverable
!
!
!
!
!
!
Figure 1: Codes of sessions employing EVAL software at CCIM
Most of the sessions are pictured and the archive is available for project
evaluators. In figure 2 are presented some pictures but also the directors containing
pictures.
2013 – The initial version of software
!
38
!
!
!
!
2014 – The current version of EVAL software
!
!
Figure 2: Pictures of
ONE2ACT sessions !
!!
!
CCIM has collected a large database of results after testing as seen in figure 3.!!
39
!
!
!
Figure 3: Files with results after testing
Figure 4 demonstrates one of the results.
!
40
!
!
Figure 4: Example of results
During the two years employment of EVAL software CCIM has collected a
large number of tests (see figure 5), in various subjects taught in VET courses.
!
41
!
!
!
Figure 5: Saved tests in EVAL software by CCIM
Most of the tests are multiple choices, as seen in figure 6.
!
Figure 6: Example of a multiple choice test
42
!
!
VET teacher is setting parameters of the test (as seen in figure 7)
!
Figure 7: Example of settings for a test
Then questions are uploaded. Figure 8 is demonstrating 3 examples of
questions (no. 1, 4, 20) composing a test with a total of 22 questions
!
!
!
!!!!!!!
Figure 8: 3 Examples of questions (1, 4, 20) in a test with a total of 22 questions
43
!
!
Observations and conclusion:
a)-the number of alternatives and the number of characters allowed by the
EVAL software fulfils the CCIM course delivery/evaluation needs;
b)-it is allowed to use special characters required by Romanian language.
The progress in monitored during testing (Figure 9).
!
!
Figure 9: Example of progress during testing
At the final of testing the teacher evaluates the percentage bar (Figure 10) and
the results (Figure 11).
!
Figure 10: Example of results percentage bar
44
!
!
!
Figure 11: Example of results
!
!
b)-The evaluation model used by CCIM
CCIM is currently employing the new learning and evaluation model that is
including four evaluation-learning activities (Figure 12): (1) pre-tests, (2) learning, (3)
post-tests, (4) comparison.
(1)
Pre-test
(3)
Post-test
(2)
LEARNING
(4)
Comparison
Training course
Figure 12. The process of evaluation
This methodology of evaluation is applied by CCIM in two ways:
A) - In each module of the training course (M1,...,M7) delivered to students, when the
comparison is done on the same test (Level 1 in figure 13)
B) - With the amount of the results database after each module and a selection of
questions for the final exam (Level 2 in figure13).
45
'0,
+
*,)*!!
!
!
.,/01213!
-"!
'()*
'()*+
+*,)*!
*,)*!!
-"!
-"!
'0,
+
*,)*!!
!
-(=<9,!
!
4(56/
02+)(1!!
!
.,/01213!!
'0,+
*,)*!!
$7
!
'()
*+
*,)*!!
4(56/02
+)(1!!
" #!
!!!!!.,8,9!$!!!!!!!!!!!!!!!.,8,9!"!
!
:0/21213!
!
Figure 13: The two level process of evaluation
A) – Evaluation level 1- in each module of the training course (M1,...,M7), at the
beginning (pre-test) and end (post-test)
In order to have an objective evaluation it is necessary to fulfil two conditions:
a)-students participating in the pre-test and post-test have to be the same;
b)-questions composing the test have to be the same.
Example for the first level evaluation.
Figure 14 is demonstrating a test applied to quality professionals at the
beginning of Module 1 – Conformity Certification.
46
!
!
Figure 14: Test for Conformity certification, module 1
The students participating in the pre-test are shown in figure 15.
!
Figure 15: Students participating in pre-test, level 1
The results after the pre-test are presented in figure 16.
47
!
!
!
Figure 16: Results after pre-test
Than the teacher is delivering the course module, for example M1. At the end
of the course students are invited for post-test (Figure 17), and the results are shown
in figure 18.
!
Figure 17: Students participating in post-test, level 1
48
!
!
Figure 18: Results after post-test
After that is done the comparison for level 1 evaluation, as follows:
a)- number of points: pre-test / post-test = 42,5 / 73 = 0,58 (71,76% better)
b)- correct answers: pre-test / post-test = 36/61 = 0,59 (69,44% better)
c)- wrong answers: pre-test / post-test = 64/39 = 1,64 (64,1% better)
Comparison can be presented as a histogram.
B) – Evaluation level 2- in each module of the training course (M1,...,M7), at the end
(pre-test) and end of the course (post-test).
In this situation post-test from level 1 evaluation is pre-test for level 2 evaluation.
Post-test is done at the final exam and comprises a selection of questions from pretests. In order to have o comparison it is necessary to collect the data manually
which is time consuming.
Without exemplifying with pictures we have collected data for 10 participants on the 3
criteria
a)- number of points: pre-test / post-test = 64 / 92 = 0,69 (14,37% better)
b)- correct answers: pre-test / post-test = 58 /91 = 0,63 (15,68% better)
c)- wrong answers: pre-test / post-test = 61 /37 = 1,64 (64,86% better)
Conclusion and recommendation: in order to have an easy comparison of
the individual results it is necessary to ensure the traceability of students
and tests (questions) employed in pre-test and post-test, level 2, but also
to define inside EVAL software the calculus formulas.
49
!
!
c)- Kirkpatrick four level model evaluation
at CCIM
Kirkpatrick described four levels of evaluation in which the complexity of the
behavioural change increases as evaluation strategies ascend to each higher level.
According to the Kirkpatrick methodology, the teaching innovation of the Quality
auditors delivered by CCIM is evaluated at levels of (Fig. 2):
1- Reaction (course evaluation and trainee feedback about course content and
instructor);
2- Learning (the resulting increase in knowledge or capability certified by formative
and summative exam grades);
3- Behaviour (extent of behaviour and capability improvement and
implementation/application effect of the course in organisation);
4- Evaluation of results (product quality in society - the effects on the business or
environment resulting from the trainee's performance).
@HD!E8/9</*2(1!(A!0,)<9*)!
!
! !
@&D!E8/9</*2(1!(A!B,G/82(<0!
! ! !
! !
@$D!E8/9</*2(1!(A!9,/01213!
!
!
!
(1) Evaluation of reaction:
@;(<0),!,8/9</*2(1!/1=!)*<=,1*!A,,=B/;CD!
Figure 19. Kirkpatrick’s hierarchy of evaluation levels.
E8/9</*2(1!9,8,9!
@HD+K,)<9*)!
E569(J,0!
@&D+L,G/82(<0!
E8/9</*2(1!
@$D+.,/01213!
!
@"D+K,/;*2(1!
!
:,/;G,0!,8/9</*2(1!
!
4(569,F2*J!(A!
,8/9</*2(1!
Figure 20: Complexity of evaluation hierarchy.
50
!
!
Complexity of evaluation increases as evaluation of intervention ascends the
hierarchy, as demonstrated in figure 20. At the first two levels the trainee evaluation
is dominant in relevance, while at the last two levels the employer evaluation is
dominant in relevance.
CCIM has developed four questionnaires for each level evaluation. The results
in a VET group trained in the quality management course, with duration of 40 hours,
in synthesis are shown below.
Questionnaire 1-Reaction evaluation
Name: """"""""""""."""""
Gender: # Male # Female
1. Previous experience in quality assurance
My theoretical instruction in quality assurance at
licence or continuous education degree is
I have practiced quality assurance
Very
NeuHigh
Low
high
tral
Very
low
2
4
14
6
1
1
2
10
8
6
&M!
$N!
O,0J!G23G!
$M!
P23G!
"N!
Q,<*0/9!
"M!
.(R!
O,0J!9(R!
N!
M!
'0,82(<)!,F6,02,1;,!21!I</92*J!/))<0/1;,!
2. Courseware
51
!
!
The clarity of the handouts, power point
presentation was
The course material and presentation provided
have followed the course subject
The overall rating of the course material provided
during training
NM!
HN!
HM!
&N!
&M!
$N!
$M!
"N!
"M!
N!
M!
High
Neutral
Low
Very
low
4
14
8
1
0
6
17
4
0
0
5
16
6
0
0
O,0J!G23G!
P23G!
Q,<*0/9!
.(R!
O,0J!9(R!
4(<0),R/0,!
3. Personal benefit of the course
To understand the subject of the course was
The content of the course for my requirements
was
My knowledge after the thought subject after
the course are
Overall rating of the course
NM!
HN!
HM!
&N!
&M!
$N!
$M!
"N!
"M!
N!
M!
Very
high
Very
high
3
15
Neutral
8
1
Very
low
0
2
9
10
4
2
3
14
9
1
0
8
8
10
1
0
High
Low
O,0J!G23G!
P23G!
Q,<*0/9!
.(R!
O,0J!9(R!
',0)(1/9!B,1,S*!(A!*G,!;(<0),!
52
!
!
4. Teacher
Knowledge of the subject of training by the
teacher
Practical examples offered during training
Presentation abilities, clarity of expression and
speech
The pedagogical method, instructiveness with
participants, discussions
Openness to suggestions from participants
Ability to respond appropriately to questions
Very
high
High
NeuLow
tral
Very
low
9
12
6
0
0
6
17
3
1
0
9
13
5
0
0
8
15
3
1
0
12
14
13
12
2
1
0
0
0
0
%M!
UM!
#M!
O,0J!G23G!
TM!
P23G!
NM!
HM!
Q,<*0/9!
&M!
.(R!
$M!
O,0J!9(R!
"M!
M!
:,/;G,0!
5. Further comments
Your opinions and suggestions stated below will help to improve training delivery.
Therefore, please give us the complete and honest answers to the following
questions:
Which lecture/module/unit/information do you find most useful in your work and why?
....Process approach is giving a clear picture about how to use indicators
Which lecture/module/unit/information do you consider least useful in your work and
why?
....Qualty awars because we will newer apply for
What kind of the issues you missed in training which have high importance for
you/your job and why?
........Quality in automotive industry, we are currently using
53
!
!
Questionnaire 2-Learning evaluation
Name: """"""""""""""""
1. Which of the following quality management standards are part of the latest edition:
a) 9000:2006
b) 9000:2008
c) 9001:2006
d) 9001:2008
2. To implement QMS in an organization wishing to certify the system is used:
a) 9000
b) 9001
c) 9004
3. Principles embodied in ISO 9000 are the numbers:
a) 4
b) 6
c) 8
4. Quality pyramid consists of:
a) characteristics
b) delivery time
c) required resources
d) product price
e) after-sales service
5. ISO 9001 allows exclusion of chapters:
a) 6
b) 7
c) 8
6. An outsourced process is:
a) the process that the organization needs and the organization chooses to be
conducted by an external party
b) a process that organization does not need and give up performing it
7. Quality manual must include:
a) scope
b) QMS procedures or reference to them
c) description of interaction between processes
8. Quality management representative may be appointed:
a) a person within the organization
b) outside the organization
9. Between input elements of the management analysis are:
a) results of audits
54
!
!
b) need of resources
c) recommendations to improve
10. The scope of the QMS internal audit is:
a) to determine whether QMS conforms to the regulations for quality
management
b) to provide management information on the results of audits
c) determination of nonconformities
11. Nonconformities audit report comprises:
a) description of the nonconformity
b) proposed correction
c) correction verification
d) proposed preventive correction
12. To the audit report is attached:
a) nonconformities report
b) corrective action reports
c) audit questionnaires
13. Audit objective is documented in:
a) audit plan
b) audit questionnaire
c) audit report
14. Active listening technique assumes the auditor to speak:
a) 80% of the time
b) 20% of the time
15. Audit questions must be:
a) open
b) answers yes or no
16. The available standard for quality audit is:
a) ISO 10011
b) ISO 19011
c) ISO 14001
17. Nonconformities can be:
a) minor
b) critical
c) major
18. In the audit are the following roles:
a) client
b) patron
c) audited
d) auditors
55
!
!
19. Necessity and audit purposes are determined by:
a) auditors
b) client
c) audited
20. Horizontal audit follows:
a) a certain aspect of the audited organization in all departments
b) all the aspects of audit object in every audited department
21. According to 19011, audit activities include:
a) audit initiating
b) on site activities
c) auditor's assessment
d) audit report preparation, approval and distribution
22. Sources for collection of audit evidences are:
a) review of activities
b) records
c) interviews with employees
23. An auditor who audits is performing certification audits should be instructed with:
a) 40 hours
b) 30 hours
24. According to ISO 19011 auditors knowledge and specific skills refer to:
a) techniques and methods in the field of quality
b) environmental science and technology
c) technical and environmental aspects related to operating
d) processes and products, including services
25. According to ISO 19011 auditor competence comprises:
a) generic knowledge and skills
b) quality domain-specific knowledge and skills
c) knowledge and skills specific to the environment
d) personal skills
Points
Students
25-23
6
22-20 19-17
12
7
16-14 13-11
2
0
10-8
7-0
0
0
56
!
!
"H!
"$!
"M!
$N+$&!
U!
$$+$M!
T!
"%+"#!
H!
"T+"H!
$!
"&???"H!
M!
.,/01213!,8/9</V(1!
Questionnaire 3-Behaviour evaluation
Name: """"""""""""."""""
3a) Trainee evaluation
I work in quality assurance in the position of :
a) Quality professional (14);
b) Quality auditor (4);
c) Quality manager (2);
d) I do not work in quality assurance (6)
Very
high
NeuHigh
Low
tral
Ver
y NA
low
The course helped me to understand
current issues of quality related to the work
place
3
13
5
2
After the course I have identified the
requirements of ISO 9001 on the
organization's quality management system
and the application in the organization
5
10
4
3
1
4
After the course I have identified the
requirements of ISO 19001 for auditing
quality management system and the
application in the organization
4
11
5
3
0
4
4
57
!
!
During internal audit of the
company/workplace I understand better the
quality issues and feel more comfortable in
the discussions
4
14
4
1
0
4
During certification/surveillance audit of the
company/workplace I understand better the
quality issues and feel more comfortable in
the discussions
5
10
8
0
0
4
I developed my skills in quality assurance
8
10
4
1
0
4
UM!
#M!
O,0J!G23G!
TM!
P23G!
NM!
HM!
Q,<*0/9!
&M!
.(R!
$M!
O,0J!9(R!
"M!
QW!
M!
:0/21,,!,8/9</V(1!
3b) Employer evaluation
Very
high
High
NeuVery
Low
NA
tral
low
The employee should be able to select
what kind of responsibility and task will take
in the job
1
5
8
4
2
7
The employee is able to select the activities
in order to perform a task related to quality
2
4
11
2
1
7
The employee should be able to select the
best methods, tools he/she like to use in the
job
1
5
10
3
1
7
Appointing a task the employee likes a
particular type of work
1
5
10
2
2
7
The employee is able to plan their own
work
3
6
8
2
1
7
58
!
!
NM!
HN!
HM!
&N!
&M!
$N!
$M!
"N!
"M!
N!
M!
O,0J!G23G!
P23G!
Q,<*0/9!
.(R!
O,0J!9(R!
QW!
E569(J,0!,8/9</V(1!
Questionnaire 4-Results evaluation
Name: """"""""""""."""""
4a) Trainee evaluation
I work in quality assurance in the position of :
a) Quality professional;
b) Quality auditor;
c) Quality manager;
d) I do not work in quality assurance
Very
NeuVery
High
Low
high
tral
low
NA
Graduating the course I found a new
job in the profile/related to the profile
1
2
12
5
3
4
Graduating the course I was promoted
in the company/the manager attitude is
changed
2
2
4
11
4
4
My results in the work are better/I can
perform quality tasks easier
3
3
6
8
3
4
Are there any changes in the relation with the employer? How? - Open-ended
question
.......To some extend after the course I have new duties regarding quality
59
!
!
&M!
$N!
O,0J!G23G!
$M!
P23G!
"N!
Q,<*0/9!
.(R!
"M!
O,0J!9(R!
N!
M!
QW!
:0/21,,!,8/9</V(1!
4b) Employer evaluation
Very
High
high
Neutral
Low
Very
low
NA
Employee degree is fulfilling a legal
requirement for the company
2
4
7
6
0
8
The number of non conformities has
decreased in the area of employee
intervention
0
3
8
7
1
8
Quality of the product/service is better
in the area of employee intervention
1
3
5
8
2
8
Activities related to quality are more
efficient
2
4
4
6
3
8
Are there any changes in the relation with the employees? How? - Open-ended
question
......I consider I can use him better in relation with the certification body
60
!
!
&N!
&M!
O,0J!G23G!
$N!
P23G!
$M!
Q,<*0/9!
"N!
.(R!
"M!
O,0J!9(R!
N!
M!
QW!
E569(J,0!,8/9</V(1!
Observation and conclusion:
Questionnaire 1-Reaction evaluation and Questionnaire 2-Learning evaluation have
been distributed to trainees during the course.
Questionnaire 3-Behaviour evaluation has been distributed to trainees and
employers after a period of 3 month the course was ended.
Questionnaire 4-Results evaluation has been distributed to trainees and employers
after a period of 5 month the course was ended.
NA is the number of trainees/employers we could not get.
There is a difficulty in collecting data and the number of employers we could not get
/were not available, is larger than of the trainees.
The course has benefits upon trainees but also organization.
The benefits are better appreciated from the trainees’ perspective than from
employers.
There are some errors in filling questionnaires, but the data can be validated due to
the shape of distribution.
61

Evaluation report from external testing of services

Transcription

Similar documents

here - YOUTHCOACH project

www.nycacc.org - Animal Care Centers of NYC

Jaltemba Bay Animal Rescue Newsletter Fall 2015

EU-WISE Conference Southampton: A. Kennedy

1 Selling Innovative products

CleverAdvice Company Profile_en 214x301mm

Located in the heart of South Lake Union, PRO Sports Club offers a

Guinea Pigs are Great! - Bellevue Cavy Critters