Done-IT - histproject.no

Transcription

LPP KA3-‐ICT Project 2011-‐13 511485-‐LLP-‐1-‐2010-‐1-‐NO-‐KA3-‐KA3MP Done- IT
Develop of open systems services for smartphones that facilitates new evaluation methods, and
enhances use of immediate feedback on evaluation results obtained in tests as a creative learning
tool.
WP 3: Develop new evaluation models
D3.1: New Peer Learning assessment methods
Author and editor:
Co-Authors:
Version:
Date:
Start month
End month
Package leader
Language of the report
Regine Ringdal, Gabrielle Hansen, Trond
Morten Thorseth, John B. Stav and Liviu
Moldovan
Bela Gayer, Miro Uran, Gabor Markus, Per
Bergegard
Final
31.03.2013
1
27
HiST Contract Research
English
This project has been funded with support from the European Commission. This publication
reflects the views only of the authors, and the Commission cannot be held responsible for any use,
which may be made of the information contained therein.
Done-IT
LLP-Project Nr. 511485-LLP-1-2010-1-NO-KA3-KA3MP
This document may not be copied, reproduced, or modified in whole or in part for any purpose without written permission from the Done-‐IT consortium. In addition to such written permission to copy, reproduce, or modify this document in whole or part, an acknowledgement of the authors of the document and all applicable portions of the copyright notice must be clearly referenced. All rights reserved. This document may change without notice, but consortium members should be informed, and number of version, stage and date should be given. Project consortium •
•
•
•
•
•
Sør-‐Trøndelag Unievrsity College, Faculty of Technology, Trondheim, Norway Centrum for Flexible Learning, Söderhamn, Sweden Petru Maior University of Targu-‐Mures, Tirgu Mures, Romania Magyar Hegesztéstechnikai és Anyagvizsgálati Egyesülés (MHtE), Budapest,Hungaria Institut za Varilstvo (IzV), Ljubljana, Slovenia HiST Contract Research, Trondheim, Norway The summary is written in English, Norwegian, Swedish, Hungarian, Slovenian and Rumanian. 2 Done-IT
Table of Contents
Done-IT ................................................................................................................. 1 1. Summary in EN, NO, SE, HU, SL and RO ...................................................... 4 2. Introduction.................................................................................................... 14 2.1 Assessment -‐ the situation today ........................................................................... 14 3. The DoneIT-approach: Peer Learning Assessment System ..................... 17 4. Requirements for use.................................................................................... 19 5. Initiating .......................................................................................................... 20 6. The questions ................................................................................................ 21 6.1 The nature of Multiple Choice questions ............................................................... 21 7. Rules settings ................................................................................................ 23 8. Identifying problems ..................................................................................... 24 9. Learning activities ......................................................................................... 25 9.1 Feedback ................................................................................................................ 25 10. Practical possibilities with use of the new Peer Learning Assessment System
(PELE). ................................................................................................................ 28 10.1 Group activity ....................................................................................................... 28 10.2 Immediate feedback ............................................................................................. 29 10.3 Active learning ...................................................................................................... 30 11. Learning mechanisms involved ................................................................. 31 11.1 Group discussion .................................................................................................. 31 11.2 Verification of a group discussion ........................................................................ 32 12. Feedback to the teacher ............................................................................. 33 13. Big versus small assessments .................................................................. 34 14. The second chance ..................................................................................... 35 15. Experiences with new Peer Learning Assessment methods in Romania36
15.1 Teaching in a new way ......................................................................................... 36 15.2 Evaluating in a new way ....................................................................................... 37 15.3 New type of evaluation supported by infrastructure ........................................... 41 15.4 Conclusions .......................................................................................................... 42 16. Literature references ................................................................................... 47 3 Done-IT
1. Summary in EN, NO, SE, HU, SL and RO
How may the teacher or instructor turn PELE into a new evaluation method? PELE in a nutshell online video is found at http://histproject.no/node/856. PELE may be used in assignments, exercises, laboratory work, tests and exams that are structured as multiple-‐choice questions with a certain number of alternatives. When your teacher account for PELE has been created (D8.1 in WP 8), you may do this: 1. Develope a number of assignments, exercises, and laboratory related quizzez, tests of exams containing closed multiple choice questions. The tests should include the whole curriculumn, and utilize a number of conceptual questinos. The later is important for use in assessment for learning processes. Upload them into PELE 2. In each course, organize a number of short tests. It is better to provide many, but short assessment activities. This could for instance involve one test every 3-‐4 weeks. Each assessment activitity may contribute with a certain percentage to the final score/grade that each student obtain at the end of a course. 3. Distribute the assignments, exercises, laboratory related quizzez, tests or exam to the class on paper. With only a few questions, just project them on the wall. 4. Use PELE to collect the result from each test.. Students may use hand held devices like laptops, tablets or modern smartphones to answer the test. 5. Provide immediate feedback after closure of each test. It is recommended to do that after having a short break in class. The teahcer may for some of the questions, give the students a second chance by starting up (from PeLe) the online Student Response System. 6. PELE may be used to collect resutls from a final exam. This requires that the questionnaries are using closed multiple choice questions. The analytical part of an exam must still be organized by using traditional hand written solutions. With PELE, the traditional "large" final exam at the end of the course may be replaced with a set of smaller tests during the semester, and a reduced/smaller final exam that addresses the analytical part. The analytcal part usually requires reflection processes, whereby multiple choice questions cannot be used. It is a key issue that the students them self register their data (inclusive the student name), making it is easy for the teacher or instructor to handle several courses, each containing several tests in large classes. 4 Done-IT
It is not necessary to reconstruct rooms or buildings in order to start using PeLe. Access to a good wireless network is enough. Consider to use the Mobile PeLe Service Unit (MPSU) if the WI-‐FI network is overloaded (has too limited capacity), security rules limit the access from students hand held devices in the class room, if you don't have any WI-‐FI network infrastructure, or no access to Internet. MPSU is a server and built-‐in Access Point (AP), setting up a local WI-‐FI network. Additional AP and/or Wireles Routers may be connected in order to increase the capacity of the local wireless network. During the assignments, exercises, tests or exams the students may use their hand held device to connect to Internet based resources. PeLe may not be used to collect analytical descriptions from assignments, exercises, tests or exams, since PeLe dosent yet support free text input. PeLe gives the teacher a new learning tool, allowing him/her to either give verification or elaborative feedback to individual students or groups of students immediately after a test. Students will, when they still remember the test questions, learn why the correct answer is correct and why the other ones are incorrect. Thus, mobile technology provides new assessment and testing criteria for education and training. Translation of summary into Norwegian: Oppsummering Hvordan kan læreren eller instruktøren bruke PELE som en ny evalueringsmetode? PELE i et nøtteskal-‐videoen kan finnes på nettsiden: http://histproject.no/node/856. PELE kan brukes til oppgaver, øvelser, laboratoriumsarbeid, prøver og eksamener som er strukturert som flervalgsspørsmål med et visst antall alternativer. Når din lærerkonto for PELE er opprettet (D8.1 i WP 8), kan du gjøre følgende: 1. Utvikle en rekke oppgaver, øvelser, laboratorium-‐quizer og eksamenstester som består av lukkede flervalgsspørsmål. Testene skal omfatte hele pensumet og bruke en rekke konseptuelle spørsmål som det er viktig å bruke i vurdering av læringsprosesser. Du kan laste dem inn i PELE. 2. Organisere en del korte tester på hvert kurs. Det er best å ha flere korte evalueringeringsrunder. Dette kan for eksempel innebære en prøve hver 3-‐4 uker. Hver evalueringsrunde kan bidra med en viss prosent til sluttresultatet/ sluttkarakteren som hvert elev oppnår i slutten av kurset. 5 Done-IT
3. Dele ut oppdragene, øvelsene, laboratorium-‐testene, prøvene eller eksamensoppgavene til klassen på papir. Når de består av noen få spørsmål, kan du bare projisere dem på veggen. 4. Bruke PELE til å samle resultater fra enhver test. Studentene kan bruke håndholdte enheter som bærbare PC-‐er, tablet-‐PC-‐er eller moderne smarttelefoner til å svare på prøven. 5. Gi umiddelbar tilbakemelding til studenetene etter slutten av hver test. Det anbefales å gjøre det etter å ha en kort pause i klassen. Læreren kan i tillegg gi elevene en ny sjanse til å svare noen av spørsmålene ved å starte opp online Student Response System som finnes i PELE. 6. PELE kan brukes til å samle resultater fra en slutteksamen. Dette krever at spørreskjemaene bruker lukkede flervalgsspørsmål. Den analytiske delen av eksamen må fortsatt være organisert ved hjelp av tradisjonelle håndskrevne løsninger. Med PELE kan den tradisjonelle "store" avslutningseksamen i slutten av kurset bli erstattet med et sett av mindre tester i løpet av semesteret og en redusert/mindre avslutningseksamen som inneholder den analytiske delen. Den analytiske delen krever vanligvis refleksjonsprosesser der flervalgsspørsmål ikke kan brukes. Det som står sentralt er at studentene selv registrerer sine data (også navnet på studenten) slik at det blir lett for læreren eller instruktøren å håndtere flere kurs som alle inneholder flere tester i store klasser. Det er ikke nødvendig å rekonstruere rom eller bygninger for å begynne å bruke PELE. Det er nok med tilgang til et godt trådløst nettverk. Du kan vurdere å bruke Mobile PELE service Unit (MPSU) dersom WI-‐FI-‐nettverket er overbelastet (har for dårlig kapasitet), om sikkerhetsregler begrenser tilgang fra studenters håndholdte enheter i klasserommet eller hvis du ikke har noen WI-‐FI-‐nettverkinfrastruktur eller tilgang til Internett. MPSU er en server med innebygd aksesspunkt (AP) som setter opp et lokalt trådløst nettverk. Ytterligere AP-‐routere og/eller trådløse routere kan bli forbundet for å øke kapasiteten til det lokale trådløse nettverket. I løpet av oppdrag, øvelser, prøver eller eksamener kan studentene bruke sine håndholdte enheter til å koble seg til Internett-‐baserte ressurser. PELE kan ikke brukes til å samle analytiske beskrivelser fra oppdrag, øvelser, tester eller eksamener siden PELE ennå ikke støtter fri skriving. PELE gir læreren et nytt læringsverktøy som det gjør det mulig for ham/henne å gi både verifiserende og elaborerende tilbakemelding til enkeltelever eller elevgrupper rett etter prøven. Studentene vil lære hvorfor det riktige svaret er riktig og hvorfor de andre er gale mens de fortsatt husker testspørsmålene. Dermed gir mobilteknologi ny vurderingskriterier og prøvekriterier for utdanning og opplæring. 6 Done-IT
Translation of summary into Swedish: Sammanfattning Hur kan läraren eller instruktören göra PELE till en ny bedömningsmetod? Internetvideon ”PELE i ett nötskal” är tillgänglig på http://histproject.no/node/856. PELE kan användas i olika uppdrag, övningar, laboratorieaktiviteter, test och prov som består av flervalsfrågor med ett visst antal alternativ. När ditt PELE-‐lärarkonto är skapat kan du göra följande: 1. Skapa uppdrag, övningar och laboratoriefrågesporter, -‐test och -‐prov som består av slutna flervalsfrågor. Test borde innehålla den hela läroplanen och ett antal konceptuella frågor. Det viktigt att använda konceptuella frågor i ett uppdrag för inlärningens skull. Överför dem till PELE. 2. Ordna flera korta test i varje kurs. Det är bättre att skapa fler korta uppdrag och aktiviteter, t. ex. ett test varje 3-‐4 veckor. Varje aktivitet kan med ett visst antal procent bidra till slutpoängvärdena/-‐betyget varje student uppnår vid kursslutet. 3. Dela ut uppdrag, övningar, laboratoriefrågesporter, test eller prov i pappersform. De ska bestå av bara några frågor och visas på skärm. 4. Använd PELE för att samla in svaren. Elever kan använda handhållna enheter, som t. ex. bärbara datorer, surfplattor eller moderna smartphones för att svara. 5. Ge direkt feedback efter slutet på test, helst efter en kort rast. För några frågor kan läraren ge elever andra chans genom att sätta igång (från PELE) online Elevresponssystemet (Student Response System). 6. PELE kan användas för att samla in resultat på slutprovet. Det kräver att frågeformulär som består av slutna flervalsfrågor. Den analytiska delen av provet måste fortfarande anordnas med hjälp av traditionella handskrivna lösningar. Med hjälp av PELE kan traditionella ”stora” prov på kursslut ersättas av en uppsättning små test under terminens lopp och en begränsat/mindre slutprov som riktar sig åt den analytiska delen. Den analytiska delen kräver ofta reflektionsprocesser, som inte kan dra nytta av flervalsfrågor. Det är mycket viktigt att elever själva registrerar sina uppgifter (inklusive elevens namn), vilket gör det lätt för läraren eller instruktören att samtidigt ta hand om flera kurser, med flera test i stora grupper. 7 Done-IT
Det är inte nödvändigt att inreda om rum eller byggnader för att börja använda PELE. Det räcker till med tillgång till ett bra trådlöst nätverk. Tänk på att använda Mobila PELE Serviceenheten (MPSU) om WI-‐FI nätverket är överbelastat (har för begränsad kapacitet), om säkerhetsregler begränsar elevernas tillgång från handhållna enheter i klassrummet, om du inte har någon WI-‐FI nätverksutrustning eller Internettillgång. MPSU är en server med en inbyggd anslutningspunkt (AP) som skapar ett lokalt WI-‐FI nätverk. Ytterligare AP och/eller en trådlös router kan också kopplas till för att förbättra det lokala trådlösa nätverkets kapacitet. Elever kan under uppdrag, övningar, test eller prov använda sina handhållna enheter för att koppla till Internetbaserade resurser. PELE kan inte användas för att samla in analytiska beskrivningar i uppdrag, övningar, test eller prov eftersom PELE inte stöder fria textinsatser än. PELE ger läraren ett nytt undervisningsverktyg och gör det möjligt för honom/henne att antingen ge verifikation eller utvecklad feedback till enskilda elever eller elevgrupper direkt efter ett test. Elever ska lära sig varför det rätta svaret är rätt och alla andra är fel medan de fortfarande kommer ihåg testfrågorna. På så sätt skapar den mobila teknologin nya uppdrags-‐ och testkriterier inom utbildning och yrkesträning. Translation of summary into Hungarian: Összefoglalás Hogyan lehet a tanárnak, vagy az oktatónak a PELE rendszert egy új értékelési módszerként alkalmazni a munkája során? PELE-‐ről dióhéjban egy online videó található http://histproject.no/node/856. web lapon. PELE rendszer felhasználható több területen, például feladatok, gyakorlatok, laboratóriumi munka során, tesztek és vizsgák alkalmával, amelyek felépítése a feleletválasztós kérdésekre épül egy bizonyos számú alternatíva megadásával. Ha az oktatónak érvényes belépési kódja van a PELE rendszerhez, (D8.1 a WP 8), akkor ezeket a feladatokat végezheti el: 1.. Fejleszthet számos feladatot, gyakorlatot és laboratóriumi munkával kapcsolatos vetélkedőket, teszteket vizsgákhoz, zárt feleletválasztós kérdésekkel. A kérdéseknek célszerű kiterjednie az egész tananyagra, így használhat számos koncepcionális kérdést is. A későbbiekben fontos olyan értékelés elkészítése is, ami a tanulási folyamatokra világít rá. 8 Done-IT
Ezek feltölthetőek a PELE rendszerbe is. 2.. Minden kurzushoz, készíthet számos rövid tesztet. Jobb ha sok, de rövid értékelési tevékenységet végez az oktató. Például adott testet 3-‐4 hetente ismételnek. Az egyes értékelési tevékenységek hozzájárulhatnak egy bizonyos százalékban, hogy a végeredmény / rangsor, pontos legyen és minden diák megkaphassa azt végén. 3.. Osszuk ki a feladatokat, gyakorlatokat, laboratóriumi munkával kapcsolatos feladatokat, tesztek és vizsga anyagokat az osztálynak papíron. Csak néhány kérdést, amiket kivetíthetünk a falra is. 4.. Használja PELE rendszert begyűjteni az eredményeket minden teszt végén. A hallgatók használhatnak kézi eszközöket persze, mint a laptopok, Tablet vagy modern okos telefonok a válaszok megadásához. 5. Azonnali visszajelzést ad a lezárása után minden PELE teszt. Javasoljuk, hogy miután egy rövid szünetet tartottunk az osztályban. A tanár néhány kérdést a hibásan megválaszoltak közül vissza ad szavazásra, hogy a diákok egy második esélyt kapjanak és használják a válaszokhoz az SRS Hallgatói Válaszadó Rendszert. 6. PELE-‐t lehet egy záróvizsga eredményének gyűjtésére is használni. Ez megköveteli, hogy a kérdőívek segítségével zárt feleletválasztós kérdésekre adjanak választ a hallgatók. A vizsga analitikai részéhez kell még a hagyományos kézzel írott megoldásokat is alkalmazni persze. A Pelé rendszer használatával a hagyományos "nagy" vizsgák a kurzus végén helyettesíthetőek egy sor kisebb teszt használatával a félév során, és egy redukált / kisebb záróvizsgával, amely foglalkozik az elemző részekkel. Az analitikai rész általában csak gondolkodás folyamatait méri, amely feleletválasztós kérdések esetében nem használhatók. Az egyik legfontosabb kérdés, hogy a tanulók saját adataikat regisztrálják (beleértve a hallgató nevét is), így könnyen megtalálhatja a tanár, vagy az oktató, és könnyen tudja kezelni több tanfolyam esetében is. Nem baj ha mindenki több vizsgát akar tenni vagy nagy osztálylétszámok vannaka vizsgázó csorportoknál. Nem szükséges átépíteni semmit a helyiségekben vagy épületekben a PELE használatának megkezdése érdekében. A használathoz egy jó vezeték nélküli hálózat is elég. Fontoljuk meg, hogy a Mobil PELE Service Unit (MPSU)használható-‐e, ha a Wi-‐Fi hálózat túlterhelt (túl korlátozott kapacitással rendelkezik), a biztonsági szabályok korlátozzák a hozzáférést a diákok kézi eszközeihez a tanteremben, vagy ha nincs WI-‐FI hálózati infrastruktúra, illetve nincs internet hozzáférés. MPSU egy szerver és a beépített hozzáférési pont (AP), amely létrehozta a helyi Wi-‐Fi hálózatot. Kiegészítő AP-‐vel és / vagy vezeték 9 Done-IT
nélküli routerel lehet csatlakoztatni a készülékeket annak érdekében, hogy növeljék a kapacitását a helyi vezeték nélküli hálózat terhelése nélkül. A feladatok, gyakorlatok, tesztek vagy vizsgák alkalmával a diákok használhatják a kézi eszközöket, csatlakozhatnak az internetes forrásokhoz. PELE nem alkalmas gyűjteni az analitikus leírásokat feladatokat, gyakorlatokat, teszteket és az ilyen vizsgákhoz sem jó, mert PELE még nem támogatja a szabad szövegbevitelt. PELE ad a tanárnak egy új tanulási eszközt, amely lehetővé teszi számára / neki, hogy vagy ellenőrzéshez használja, vagy elaborative visszajelzést ad az egyes tanulóknak vagy csoportoknak a teszt után. A hallgatók, ha még emlékeznek a vizsgálati kérdésekre, hogy miért a helyes válasz helyes, és helytelen helytelen, akkor újra tudják gondolni a megoldásaikat. Így a mobil technológia új értékelési kritériumokat és vizsgálati módszert ad az oktatáshoz és a képzéshez. Translation of summary into Slovenian: Povzetek Kako lahko učitelj ali inštruktor uvede metodo PeLe (Peer Learning Assessment Service) kot novo metodo ocenjevanja? Spletni prikaz PeLe metode je na voljo na http://histproject.no/node/856 Metodo PeLe lahko uporabljamo pri vajah, delu v laboratoriju, reševanju vprašalnikov, testov in izpitov, ki so sestavljeni na način reševanja z več možnimi odgovori in z določenim številom alternativ. Ko ustvariš svoj PeLe učiteljski račun (D8.1 v DP 8), ti program zagotavlja dostopanje do sledečih možnosti: 1. Razvoj številnih nalog, vaj in laboratorijsko povezanih vprašalnikov, testnih pol in izpitov z več možnimi odgovori. Testi morajo vključevati celoten učni načrt, vključno s številnimi konceptualnimi vprašanji. Zlasti slednje je pomembno za uporabo v procesih ocenjevanja za učenje. Naložite jih v v PeLe sistem. 2. V vsakem ciklu izobraževanja organizirajte več kratkih testov. Dosedanja testiranja so namreč pokazala, da so dobljeni rezultati boljši, če zagotovimo pogostejše a hkrati krajše ocenjevanje. To lahko na primer vključuje en preizkus vsake 3-‐4 tedne. To na končni ravni lahko rezultira z višjim odstotkom uspešnosti. 3. Razdelite naloge, vaje, vprašalnike, teste ali izpite za razred v papirni obliki. Z le nekaj vprašanji jih projicirajte na steno. 10 Done-IT
4. Uporabite metodo PeLe za analizo posameznega testa. Slušatelji lahko za reševanje uporabljajo ročne naprave, kot so prenosni računalniki, tablica ali sodobni pametni telefon. 5. Po zaključku posameznega testa ali izpita zagotovite takojšnjo povratno informacijo Priporočljivo je, da to naredite po kratkem odmoru. Učitelj lahko da študentom pri reševanju nekaterih vprašanj možnost ponovnega odgovarjanja, tako da zažene v programu PeLe aplikacijo online Student Response System (“second chance” -‐ druga možnost). 6. Sistem PeLe lahko uporabimo za zbiranje rezultatov zaključnih izpitov. Vprašalniki morajo biti sestavljeni na način z zaprtimi večmožnostnimi odgovori. Analitični del moramo še vedno narediti ročno. S sistemom PeLe lahko običajni zaključni izpit nadomestimo z nizom manjših testov med semestrom. Tako zmanjšamo analitični del zaključnega izpita. Ta običajno zahteva procese refleksije, pri čemer uporaba sistema več možnih odgovorov ni mogoča Ključno je, da študenti sami vnesejo svoje podatke (vključujoč ime študenta), zaradi česar je postopek za učitelja enostavnejši, še posebno kadar učijo več različnih predmetov z več predvidenimi testiranji in v velikih razredih. Za uspešen začetek dela po PeLe metodi ni potrebno preurejati dosedanjih prostorov. Zagotoviti je potrebno le dostop do dobrega brezžičnega omrežja. V primeru, da je brezžično omrežje preobremenjeno (ima tudi omejene zmogljivosti), ali varnostna pravila omejujejo dostop študentom z ročnimi napravami v učilnicah, če nimate vzpostavljenega brezžičnega omrežja, ali nimate sploh dostopa do interneta, lahko uporabite Mobile PeLe storitev (MPSU). MPSU je v tem primeru strežnik in vgrajen Access Point (AP) je nastavljen kot nosilec lokalnega brezžičnega omrežja. Dodatne AP in /ali brezžični Router-‐ji so lahko povezani, da povečamo zmožnost lokalnega brezžičnega omrežja. Med reševanjem nalog, vaj, testov ali izpitov lahko slušatelji uporabljajo svoje ročne elektronske naprave, da se brezžično povežejo z osnovnimi bazami v PeLe sistemu. PeLe ne moremo uporabiti za zbiranje analitičnih tolmačenj nalog, vaj, testov ali izpitov, ker sistem PeLe še ne podpira brezplačnega vnosa besedil. Sistem PeLe daje učitelju na voljo novo učno orodje. Omogoča mu sprotno preverjanje in mu daje povratne informacije za posameznike ali skupine študentov takoj po opravljenem testu. Za študente je pomembno, da takoj po zaključku testa izvedo, zakaj je nek odgovor pravilen in zakaj so ostali napačni. 11 Done-IT
Translation of summary into Romanian: Cum poate profesorul sau instructorul transforma PELE într-‐o nouă metodă de evaluare? Videoclip-‐ul despre PELE se găsește la adresa http://histproject.no/node/856. Pele pot fi folosit în evaluări, exerciții, lucrări de laborator, teste și examene, care sunt structurate sub formă de întrebări cu răspunsuri multiple, cu un anumit număr de alternative. Dacă vă creați contul de profesor (D8.1 în WP 8), cu ajutorul PeLe puteți face următoarele: 1. Dezvolta o serie de evaluări, exerciții, chestionare legate de laborator, teste de examene care conțin întrebări cu răspunsuri multiple închise. Testele trebuie să cuprindă întregul curriculum și să utilizeze o serie de întrebări conceptuale. Sub această formă pot fi utilizate în evaluarea procesului de învățare. Încărcați-‐le în PELE 2. In fiecare curs, se pot efectua o serie de teste scurte. Activitățile de evaluare este mai bine să le efectuați mai multe și de scurtă durată. De exemplu, se poate efectua un test la fiecare 3-‐4 săptămâni. Fiecare activitate de evaluare poate contribui cu un anumit procent la nota pe care studentul o obține la finalizarea cursului. 3. Se distribuie cursanților evaluările, exercițiile, chestionarele de laborator, testele sau examenele pe suport de hârtie. Dacă acestea constau din câteva întrebări, se pot proiecta pe perete. 4. Se utilizează PELE pentru colectarea rezultatelor testelor. Studenții pot folosi dispozitive portabile cum ar fi: laptop-‐uri, tablete sau telefoane inteligente moderne, pentru a răspunde la test. 5. Se comunică rezultatele imediat după finalizarea testului. Acest lucru este recomandat după o scurtă pauză. Pentru anumite întrebări profesorul poate să le ofere studenților încă o șansă prin activarea (din Pele) Sistemului de Răspuns al Studenților. 6. PELE pot fi folosit pentru colectarea rezultatelor de la un examen final. Acest lucru necesită utilizarea de chestionare cu întrebări cu răspunsuri multiple închise. În continuare, partea analitică a examenului trebuie să fie susținută sub forma clasică, scrisă de mână. 12 Done-IT
Cu Pele, examenul tradițional "amplu" de la finalizarea cursului, poate fi înlocuit cu un set de teste mai mici susținute pe parcursul semestrului și un examen final restrâns / mai mic, care se referă la partea analitică a cursului. Aceasta necesită, de obicei, reflecție, în cursul căreia nu se pot utiliza întrebări cu mai multe variante de răspuns. Este o problemă-‐cheie ca studenții să-‐și înregistreze datele de autentificare (inclusiv numele), ceea ce permite profesorului / instructorului să susțină mai multe cursuri in clase mari, fiecare dintre acestea conținând mai multe teste. Nu este necesar că construiți laboratoare sau clădiri, pentru a utiliza Pele. Accesul la o rețea wireless este suficient. Puteți să folosiți unitatea mobilă Pele Service (MPSU) dacă rețeaua Wi-‐Fi este supraîncărcată (are o capacitate limitată), normele de securitate limitează accesul dispozitivelor mobile ale studenților în sala de clasă, dacă nu aveți nicio infrastructură WI-‐FI de rețea FI, sau dacă nu aveți acces la Internet. MPSU este un server construit în Access Point (AP) pentru crearea unei rețele locale Wi-‐Fi. AP suplimentare și / sau rutere wireless pot fi conectate în vederea extinderii capacității rețelei locale wireless. În timpul evaluării, exercițiilor, chestionarelor de laborator, testelor sau examenelor studenții pot folosi dispozitivele mobile proprii pentru conectare la resursele de pe Internet. PeLe nu poate fi folosit pentru colectarea de descrieri analitice de evaluări, exerciții, teste sau examene deoarece PeLe nu permite introducerea textului liber. PeLe oferă profesorului un nou instrument de învățare, care îi permite să examineze și să colecteze impresii de la studenți sau grupuri de studenți, imediat după un test. Studenții, când încă își mai amintesc întrebările din test, au posibilitatea să învețe de ce anumite răspunsuri sunt corecte, iar celelalte sunt incorecte. În acest fel, tehnologia mobilă oferă noi metodologii de evaluare și criterii de testare care se pot folosi în educația și formarea profesională. 13 Done-IT
2. Introduction
The aim of this report is to give some pedagogical arguments for why teaching and assessments with the assessment system developed through the DoneIT project, might work and to some extent how. We have gathered some pedagogical argumentation for why the Done-‐IT approach might be beneficial in students learning processes, in addition to the traditional assessment of student’s level of understanding. This picture illustrates how a written exam is done in higher education in Norway. 2.1 Assessment - the situation today
Within the research field on assessment it is normal to distinguish between two types of assessment, summative and formative. The first of these measures and controls that defined learning objectives are achieved. Feedback information is provided after students work is completed, often in form of a final core or a grade. The purpose of the assessment is to rank, endorse or verify students' competence (Sadler, 1998). Assessments can also generate feedback information provided during a learning process, as information students can use to improve their own learning and further performances. This feedback information may also be a tool for teachers. Teachers can adjust their own teaching practice in relation to students' needs. When assessment serves these purposes it is called a formative assessment (Sadler, 1998). Within the research literature on assessment, it is promoted a need to explore how higher education institutions, and others, can use assessment more effective in promoting student learning. Feedback is barely conceptualized in the theoretical literature on higher education, and 14 Done-IT
other places, making it difficult to design effective feedback practices, and to evaluate their effectiveness in relation to students' learning (Sadler, 1998; Yorke, 2003). In higher education, formative assessment and feedback is often a teacher-‐led process, and is often perceived as the teacher's responsibility. Feedback is still considered a mediation process, where teachers communicate feedback information to students about their academic strengths and weaknesses, which students are supposed "to decode" and convert to concrete actions to improve their understanding and academic progress (Nicol and Macfarlane-‐Dick, 2006). The traditional and most common feedback practice generally involves correction of errors. Traditional classroom assessments/tests are a widely used feedback practice for this. Great emphasis is placed on the adequacy of scores, and less emphasis on the importance of them (Hattie & Timperley, 2007). There is little research evidence that such traditional classroom tests / tests assist students in their learning process. A meta-‐study by Black & William (1998), examined 578 studies that related classroom tests to the students' learning, and concluded the following: Classroom assessment typically encourages superficial and rote learning, concentrating on recall of isolated details, usually items of knowledge which pupils soon forget . . . teachers do not generally review the assessment questions that they use and do not discuss them critically with peers, so there is little reflection on what is being assessed. (p. 17) Assessments are in a way the main focus in a student’s life. It is the judgment from the final exam or test that decides their future and possibilities for further studies and a possible job. A school diploma, with an evaluation of their professional qualification is the only proof they can use after the education or course. This makes it natural for many students to focus on the performance on the assessment rather than learning focusing on learning the subject. The most common assessment activities students participate in throughout a semester are academic tests and/or a final examination. Students perform through answering a number of academic questions one day, and get a verifying feedback, often in form of a grade or pass/fail response, days or perhaps weeks after their performance. Elaborative feedback information, like in-‐depth focus on what is right or wrong and why, is rarely included, reducing the learning effect of the feedback information significantly. For a professional feedback to influence students' learning process in a positive way, it needs to include both verifying and elaborative information (Kulhavy & Stock 1989). With the New Peer Learning Assessment System (PeLe), developed through the done IT project, each student can respond to test questions using a smartphone. An automatic grading system, integrated in the assessment system, assess their performance. This assessment technology gives teachers immediate access to student’s performances, and makes it possible for him/her to localize student’s problem areas in a fast, simple and efficient manner. This further enable teacher`s to give students verifying and/or elaborative feedback information immediately after the test is finished (post assessment activity). Students can get feedback on their performance, 15 Done-IT
while their performance and thought processes behind, is fresh in their memory. Through various elaborative feedback activities, such as in-‐depth teacher explanation or peer learning through student collaboration, students can get an opportunity to achieve a greater understanding of, and learn from their mistakes. 16 Done-IT
3. The DoneIT- approach: Peer Learning
Assessment System
The following figures in this report are taken at various points in the project and therefore do not represent necessarily the finished product. The DoneIT-‐approach to an assessment is based on the following way of performing an assessment. Details about how the system can be used are found in the user manual. This is a general approach to what is done: 1. Preparation. The teacher prepares questions for the assessment and creates a printout that can be handed out to the class. The assessment has to be prepared as a multiple choice question assessment. 2. Registration. When the assessment is finished and ready, the setup has to be prepared for the DoneIT system. The teacher has to start DoneITC and create a new assessment setup document. Here the system is fed information about the assessment, the number of questions, number of alternatives for each question and what alternative(s) is correct. 3. Rules. When the assessment is set up, the rules have to be set. The rules tell the system how the assessment is to be evaluated. As a default value, a correct alternative give 1p. while a wrong alternative gives 0p. Not responding to an alternative gives 0p. 4. End preparation. When the assessment is set up correctly, the assessment can be uploaded to the DoneIT server and gets an assessment code. The assessment can also be stored to file and brought to the classroom upon the assessment day. 5. Start. You start the assessment by uploading the setup document, and get an assessment code. 6. Login. Students login with their username and password to the given assessment identified by the code. 7. Handout. When everybody has logged in, something the teacher can monitor in DoneITC, the paper version is handed out and the students can start working with the assignment. 8. Monitor. While the students respond, the situation is monitored and the teacher can prepare for what to do now after the assessment is over. 9. Submission. Students submit their responses, and take a break. 10. Identification. When everybody has submitted, the teacher can start to identify their problem areas, based on what the students have responded. The interface gives a picture of how the students have responded. 17 Done-IT
11. Post assessment activity (PAA). When the questions causing problems are selected and the class is gathered again, the main interactivity can begin. The approach here is individually and will be variable from teacher to teacher or based upon what the students have responded. The main purpose of the DoneIT project is to open up for the PAA and to give the teacher freedom to choose from a set of different approaches. 18 Done-IT
4. Requirements for use
In order to run the system you need a good WiFi coverage over the area with sufficient access for all students to participate with their response unit. A student can use a smartphone, an iPad or iPod or a computer that is WebKit1 compatible. With the first prototype the system requires webKit browsers and works only on iPhone, iPad or iPod, but a student interface that is more generic html+javascript is under construction and we are planning for a solution where most kind of browsers can be used. You also need a computer with the DoneIT Controller software DoneITC installed on a computer where the screen can be projected on the wall for the class. More information about the system and how can be used is found in the DoneIT User manual. 1
http://en.wikipedia.org/wiki/WebKit
19 Done-IT
5. Initiating
When you start to use the system, it’s important to give the students a thoroughly introduction; what is a Peer Learning Assessment System, why should they use it and how does it work? Based on initial testing of the assessment system with three students groups, we learned that for students, use of technology in an assessment situation represents something new and slightly frightening. They are used to answering on paper, as this is what they have done throughout their academic careers. Initially, they therefore felt a bit insecure about the technology, and the first test we had with them introduced considerable stress because of this. The uncertainty was rooted in fear of the iPod suddenly turning off, or breaking down, or that their responses would be lost. When we enter a classroom with the assessment system, one of our primary focus areas will therefore be on making the students feel confident and comfortable with the technology. The system should be presented with a thorough introduction followed by a sample test, so that the students get used to the technology as soon as possible. Creating a simple assessment where you force the students to do certain activities just to get to know the system might be one way of getting students used to navigation and responding. You should explain why you want to use it and how things will work, and how the students should use their interface, what the buttons do and so on. Do not expect that everyone understands how they should use the system. We have spent hours with students testing out and trying to make the student interface user friendly and intuitive and made corrections based on feedback from the students. We have also followed up with interviews of the students trying to identify parts of the interface that is not intuitive. The best way will be to give an introduction and let the students try before any kind of assessment is performed. Some people might fear technology and don’t trust the technology when participating. 20 Done-IT
6. The questions
In order to use the DoneIT system you have to prepare you assessment as a Multiple Choice Question assessment. When you have a set of multiple choice questions, you want to measure something or create something in the post assessment activity. You have to make sure that your questions target what you want to measure and meets to some extent what you want to in the PAA. However Multiple Choice Questions (MCQ) can be limited, and there is a lot of critics towards the use of such assessments. The results of your use depends on the context you use the questions (Nicol 2007). The DoneIT approach, with the Peer Learning Assessment System, tries to utilize a twist in the way that assessments are done. We want to change the assessment context. Academic assessments, like test, students typically perceive as a performance situation, their focus is on achievement. We want an assessment to be perceived as a learning situation, where students can learn from their mistakes. We want to turn an assessment into an arena for learning. 6.1 The nature of Multiple Choice questions
In a world where the access to computers have improved, resources to education reduced, and the number of students is increasing, assessments in the form of a Multiple Choice Question (MCQ) assessment has increased. If the assessment is done electronically online, the results can be provided immediately, and since this approach saves time for the teacher removing the time spend to correct and give feedback, and giving time to teach even more, the trend is that the use of MCQ has increased. However as stated by Nicol, the nature of MCQ provide some pedagogical limitations by nature (Nicol, 2007). In the literature it is argued that MCQ promote memorization and factual recall (Airasian, 1994; Scouller,1998) and does not encourage students to think and understand the big picture. The situation can be improved if the questions are made better, requiring students to really think before they submit an answer. This however requires a lot of work for the teacher to construct assessments that has good questions. Secondly, the feedback provided through MCQs is usually quite limited as it is predetermined during test construction. Hence there is little scope for personalization of feedback based on different student needs. For subjects like mathematics and physics, where a significant target of the course is to teach the students practical calculation procedures and reasoning skills, feedback from a MCQ does not provide information about where an error was made in the reasoning or calculation. The students get a feedback that the calculation was wrong or correct. If an activity should enhance learning, the feedback must be in such a way that the student can change the reasoning where the logical brake is, understand what went wrong and also understand what is 21 Done-IT
the correct way to reason or calculate. At HiST we often see students fail in their reasoning at different levels in the same course, and feedback is needed on an individual level. With the use of MCQ this is not possible using online MCQ assessments. Thirdly MCQs are often used, not for pedagogical purposes, but to improve teacher efficiency, freeing time and for more teaching. The aim is not effective learning but effective teaching. The focus of learning might be lost in the search for cost effective teaching. For the students, the outcome of the course is the results from the assessment and the grade from their performance. The MCQ type of assessment, encourage students to optimize the assessment procedure. They learn to recognize alternatives, exclude unlikely alternatives and to memorize facts, rather than constructing a response based on their own understanding of the subject. Often students don’t have a possibility to unravel the problem given and to get an understanding of the problem, something that often lead to guessing and optimizing results by excluding the most unlikely alternatives. Recent publications states that students should play a more active and participating part in the assessment procedures (Boud 2000; Yorke 2003). The way that we try to compensate for these drawbacks is to change the context. We apply MCQ with an activity after the assessment where the teacher can work with the group and possibly create different activities that benefit learning and give the students a chance to learn from their own mistakes. 22 Done-IT
7. Rules settings
Grading MCQs has to be done by a set of grading rules. The system allows the teacher to set rules within certain limits. You can give the same rules for all questions, or you can override the rules on individual questions. If a teacher experiments too much with the rules, the outcome might become unpredictable for the students, and this might have negative effect on the students. If your assessment contains few questions, the possible outcome of the assessment is limited. If you run an assessment with ten questions, using standard rules, you will get results ranging from 0p to 10p. By changing the rules you might alter the outcome. You can introduce multiple cardinality (more than one correct answer) with the questions, and you can give different score to the alternatives. This will generate a larger span in the outcome. You can also introduce negative score on wrong alternatives, and even allow the questions to bleed (allowing questions to give a negative score that is subtracted from the total). If you use punishment on wrong alternatives and does not allow the questions to bleed, you might end up with a minimum of zero points in score. If you allow bleeding, it would be an idea to use an initial score, such that the outcome will be positive. This will give the chance to extend the outcome space and give a better ranging of the results, if used carefully. However, an understanding of how the outcome space of the assessment looks like might be challenging when the rules are complex. To help the teacher in designing the assessment we have introduced the “monkey cage” that will simulate a random response statistically looks like. By letting the some thousand simulated “monkeys” respond, you can get a feeling of the probability to get a score at pure random. If these simulations can be used pedagogically with the class is difficult to predict, but it will give you as a teacher the possibility to see how well a student should respond to be better than just guessing. 23 Done-IT
8. Identifying problems
When the assessment is finished, you will be able to see what questions caused problems. You then may select what questions you want to introduce to the post assessment activity phase. If you have a high spread in the responses, this means that the class has a big problem understanding the question, and might have guessed their response. The spread factor is a factor ranging from 0% to 100% where 0% means that all students have responded to 1 question. A spread factor of 100% means that there is equal number of responses to each alternative. If you have more than one correct alternative, you will not get a spread factor of 0%, even if all students respond correctly to your alternatives. The distribution might help you to select what kind of activity you want to choose. If the group is divided, it might be an indication that those who know can teach the others that don’t know. We lack the experience to predict what you should do. You simply have to gain experience and listen to what you students can tell you! 24 Done-IT
9. Learning activities
For a student, learning activities can consist of several types of activities. •
•
•
•
•
Lectures, seminars, courses Practical learning, problem based activities Reading, tutorials, podcasts Discussions, peer learning and peer support Reflection, engagement and interaction Often lectures only serves as an information source for learning objectives. “This is what you should learn this week!” Fitting the assessment as a natural part in this set if activities, is difficult, since it traditionally only provides the teacher with some feedback on his/her teaching. It is up to the student to find ways to learn the subject, with the aid of some guided exercises and practice. Upon how well a student has learned some material, Bloom’s taxonomies may serve as a guide for arranging the level of knowledge. Bloom divided this domain into six levels of understanding as given in the table below, revised by (Anderson et al. 2001). Acquisition of facts or having knowledge is only the initial stat of understanding. The facts must be understood (comprehension) before they can be applied to new situations (application). Knowledge must be organized and patterns recognized (analysis) before it can be used to create new ideas (synthesis). At the end, being able to judge and consider competing or evidence, the student needs to be able to assess (evaluation) the relative value and validity of information or ideas. Level of thinking Knowlegde Comprehension Application Analysis Synthesis Evaluation What it represents Facts Understand meanings Apply to new situations See organization and pattern Generalize and create new ideas Assess value of evidence 9.1 Feedback
Feedback is important for learning. In any learning situation, feedback in one form or the other is the mechanism that regulates the learning process. Some use the model of building a brick wall, where feedback is the cement holding the wall together. If no cement is used, the wall can reach an instability and collapse. Feedback serves as the verifying glue that the knowledge build in a student’s head holds. 25 Done-IT
Today, traditional assessments give a the students a chance to respond to a set of questions and get a verifying feedback, either in the form of a degree or a passed/ not passed, often days and weeks later. This feedback gives little or no useful feedback to the students when they don’t know exactly what caused the degree. The problems that were solved are out of the head and the assessment is done. If the students could get an immediate feedback and a possible input on how to think, reason, or argue to solve the problems, the feedback could act as a tool for the student to adjust or correct thinking patterns. With the correct / wrong judgment, there is no such feedback to the student. If the results appear days later, the judgment is not connected to the problem solving thoughts and hence no corrections are made. By giving immediate feedback, the students can have a chance to connect this minimal feedback to the actual performance done in the assessment. According to (Race 2005) feedback is most effective when: •
•
•
•
•
•
Timing: Feedback is given as soon as possible. Personalizing: Feedback fit each student's achievement, individual nature, and personality. Empowering: the feedback intended to strengthen and consolidate learning, and not the opposite. Analytical: Explain carefully what was good or excellent in detail, so that the message is clear. If the only feedback is excellent, the student does not know what was excellent. Constructive: the feedback gives guidance on how to improve. Managable: the amount of information has to be manageable for both parts. One last important point is that feedback has to be manageable for both for students and the teacher. It is impossible to provide the perfect feedback, since there just is no time. For students the amount of feedback has to be manageable and the feedback should be focused on the points that have the highest impact on possible improvement. With electronic MCQ assessments this kind of feedback is a challenge. Timing can be met with immediate feedback, but the other points are difficult to find practical ways to solve. But Phil Race (Race 2005) also suggests some payoff feedback methods that can be provided. Some of these are: •
•
•
•
•
students comparing work peer-‐marking with feedback constructive questioning within groups verbal feedback to whole class or groups group peer review 26 Done-IT
Common for all these is that the activity can be done in the class, and using the group as a whole, either in groups or in class. In the review of Nicol 2007, he looks closer at different contexts where MCQ has been applied. He also presents seven principles for good feedback practice that could support the development of learner self-‐regulation (Nicol & MacFarlane-‐Dick 2006). 1 2 3 4 5 6 7 Good feedback practice helps clearify what good performace is (gols, criteria, standards); facititates the development of self-‐assessment and reflection in learning; delivers high quality information to students about their learning; encourages teacher and peer dialogue around learning; encourages positive motivational beliefs and self-‐esteem; Provides opportunities to close the gap between current and desired performance; Provides information to the teachers that can be used to shape teaching. 27 Done-IT
10. Practical possibilities with use of the
new Peer Learning Assessment System
(PELE).
On an academic test students use their former knowledge to answer questions. By using the new assessment system, the teacher can in a simple way use students test results to identify their problem areas, and make a methodological choice about where he/she should focus further teaching in the post assessment face. The purpose is to give students more detailed feedback in relation to their own achievements. By linking feedback to other classroom activities, the goal is to improve the quality of feedback students receive about their own performance. The way students perform, will place large constraints on the methodological choices teacher makes after the test is completed. It`s their performances that will decide whether the teacher encourages students to participate in group discussions, or others activities. Teacher's choice of method will be related to the response distribution on the various test questions. Teachers can, for example: •
Give students a verifying feedback by showing them the response distribution from different questions and emphasize which alternative is correct. •
Give students an elaborative feedback, by giving them a detailed explanation about what is right, less right, or wrong and why. •
Give students the opportunity to learn from their mistakes by cooperating with their fellow students. •
Initiate a larger class discussion / debate •
Give students a hint and ask them to work on a specific question one again, either individually or in groups, and answer the question again using the response system. 10.1 Group activity
The Peer Learning Assessment System has been designed in such a way that teachers can immediately use students’ results from an assessment and initiate numerous learning activities in the classroom using several methodological approaches. One of these approaches is peer learning through small group discussions and student collaboration. An interesting question is how a group activity can benefit students learning? 28 Done-IT
Collaboration between students is linked to both better understanding of the learning material and better performances (Collier 1980; Ames 1984, 1992; Slavin 1983, 1985; Graham & Weiner 1996). According to Collier (1980), this can be explained by the fact that the learning materials often appear clearer to students when it is "translated" by, and illustrated using examples from fellow students rather than teacher led explanations. Students use a language (words and concepts) between themselves that they understand, rather than more advanced terminology. Discussions within the student groups can also help create a more relaxed and calm atmosphere which in turn can lead to a greater willingness to admit uncertainty (Collier 1980) Group discussion also has the advantage of involving students more active in the learning process. Further, it can be used effectively with younger students for short periods of time. However, unless you require participation and use a system to mark each student who speaks, it can be challenging for the teacher to get everyone involved. There has been evidence for an increased learning and a more favorable attitude towards learning when undergraduate courses in science have been taught in small groups (Springer, Stanne, and Donovan 1999). Group learning does not necessary work with every assignment. Cohen (1994) found that small group learning can be productive for conceptual learning, under certain conditions. The most important of these conditions is that the task is a real group task. Problems that are context-‐rich, many meet these requirements. Heller et al. (1992) also found that in well-‐functioning cooperative groups a better problem solution emerged than was achieved by individuals working alone and the instructional approach improved the problem-‐solving performance of students at all ability levels. The success for creating an activity and engagement in the class is the nature of the question. A simple factual question might have little effect on creating a discussion. There is one answer, if the student does not know it or remember any argument for it, there is nothing to discuss. The challenge is to find questions and problems that are within students reach, but at the same time it is not crystal clear what might be the correct answer. If several alternatives are partly correct but one is most correct, this might cause discussion and lead to a deeper understanding. The goal is to get students to go more in depth on topics. 10.2 Immediate feedback
Feedback is an important contribution in relation to students' learning (Mory 1992; Laurillard 1993; Kluger & DeNisi 1996; Nelson & Pearson 1999), and it is one of the most important principles for ensuring good and effective teaching (Rose Shine & First 1971; Chick Call & Gamson 1987; Book 1999). The message is really very simple: students learn best when they receive feedback on their understanding and performance (Book 1999). 29 Done-IT
Through use of new assessment system students can gain immediate feedback on their own performances. Immediate feedback is important for the students to be able to connect the feedback information they receive to the assessment and their own performance done. 10.3 Active learning
Learning cannot be compared with some sort of spectator sport (Chick Ring & Gamson 1987; Nelson & Pearson, 1999), although perhaps misconceptions about the term "active learning" has suggested otherwise. According to Bonwell & Eison (1991), the term "active learning" has been used a little too random and perhaps most of intuitive understanding rather than fixed definitions, which has led to a perception that all learning is active and that students are also actively involved when they just sit quietly and listen (Bonwell & Eison 1991). Analysis of research (Chick Ring & Gamson 1987), suggest that students must do more than just sit quietly and listen to be actively involved in their own learning process, they must read, write, discuss, or be engaged in some form of problem solving. In addition, they should also engage in deeper thought processes as analysis, synthesis and evaluation (Bonwell & Eison 1991). Such thoughts about learning are in line with a constructivist approach, which emphasizes the active role of students in relation to the development of personal knowledge. Active learning involves students doing things and thinking about the things they are doing. Typical activities where: •
•
•
•
•
•
•
Students are involved in more than passive listening Students are engaged in activities (e.g., reading, discussing, writing) There is less emphasis placed on information transmission and greater emphasis placed on developing student skills There is greater emphasis placed on the exploration of attitudes and values Student motivation is increased (especially for adult learners) Students can receive immediate feedback from their instructor Students are involved in higher order thinking (analysis, synthesis, evaluation) Here the task might range from simple to complex tasks, and still be within the active learning part. The Done-‐IT project aims to develop a new assessment model in which students' academic performance can be used as a tool to promote active learning. Immediate access to students' academic performance allows for quick receipt of verifying feedback, and use of more elaborate feedback methods involving active participation from students. 30 Done-IT
11. Learning mechanisms involved
11.1 Group discussion
“Students learn what they care about and remember what they understand” Ericksen, 1984, p. 51. This citation covers a lot of what good teaching is about. If you as a teacher just continues to talk and a student do not understand, it’s impossible to remember. You have to make sure that they understand and that the content of what you are teaching makes sense. One way of doing that is to allow students to rephrase and organize their own structure of understanding. When they have got the understanding, remembering is no longer an impossible task. Group activity and group discussion serves as a way to restructure and formulate orally an understanding of a concept. It will also immediately give feedback to the students wither they understand or not. In order to trigger a group discussion, several approaches can be done from the teacher with the Done-‐IT system. He/she might display the question and talk about the question, identifying the target of the problem. The application is made transparent such that the questions can appear through the application and results can be displayed on top. It is possible to display the percentage right or wrong chart to the class, giving no leading information to the correct alternative. Figure 1 Percentage right or wrong bar chart from the DoneIT prototype.
It is also possible to show the distribution of the responses, but without showing what alternative is correct. 31 Done-IT
Figure 2 Histogram showing the distribution of responses from the DoneIT prototype.
11.2 Verification of a group discussion
Any form of group discussion should always end up with a teachers led explanation afterwards. Smith et al., (2011) shows that for peer instruction, a method applied when using student response systems, where students first answer individually, and then again later after a group discussion, the final teacher explanation is important. The combination of peer discussion followed by instructor explanation improved average student performance substantially when compared with either alone. Research from HiST regarding use of student response system in classes also highlights the importance of thoroughly teacher explanation after a session of student discussions and voting. For the students, the teacher's explanation was perceived as a constructive feedback, and was highlighted as critically important for their own experience of learning. Through the teacher's explanation, the students got an understanding of why the various options were correct or incorrect. One thing is to cast a vote that turns out to be right or wrong; another matter entirely is to be able to understand why it is right or wrong. If they achieve such an understanding, they feel that they really learn something through the questions. The students are keen to point out that the teacher should give adequate explanations for why the wrong options are incorrect. For the students, this is a way for those who answered incorrectly to understand why they got it wrong. The same methodological approach will be important using the new Peer Learning Assessment System in the classroom. Any student led activity should be followed by a teacher led explanation at the end. 32 Done-IT
12. Feedback to the teacher
The DoneIT system has a built inn functionality to allow students to “mark” a question. This marking will appear to the teacher and he/she will be able to get statistics of how many marks are given on a question. Students have argued for making this anonymous. The teacher should not see who marks a question. The idea is that the teacher can define what the mark represents. The students can also use it to keep track of questions that are difficult and needs more attention. Possible approaches can be used: •
•
•
Teacher doesn’t use the marking function: Free to use at will for the students. Signal for “Please Explain” from the students. The teacher encourages students to mark those questions they want the teacher to explain carefully after the assessment. I am sure flagging of the questions. Students can mark the question they feel certain on they know. This to make them think more deeply about the questions being asked. Figure 3 The students interface when question 1 has been “marked”.
The “Please Explain” will give the students some influence of what the teacher should spend time on after the assessment, and may trigger positive motivation towards wanting to understand a problem. The system now will not allow students to benefit from an “I am sure” definition in such a way that the score can be increased when the students have responded correctly and flagged that response. The suggestion is here to choose an approach and stick to it. If you change the rules, the use might cause confusion. 33 Done-IT
13. Big versus small assessments
One question that is not answered here is the size of the assessment. What way of using the New Peer Learning Assessment System works best for the students? Should the assessments be given with relative high frequency, and with a few questions, or should the assessments be less frequent but bigger in size? What is the optimal size of the assessments, when trying to trigger learning activities? One suggestion here is to make the assessments large enough such that some problems can be identified. An assessment with questions ranging from simple fact questions to more analytical problems such that some everyone reaches a problem they can’t solve. 34 Done-IT
14. The second chance
The Peer Learning Assessment System also introduce the possibility to answer a question a second time, and therefor get a second chance on questions that are difficult to understand or by any reason was not answered correctly. How this will be used is not easy to predict. If you allow the students to get a pure second chance you might get unwanted effects. The Peer Learning Assessment System allows teachers to send the question out to the students again after some information, student discussion or a hint, and opens the possibility to respond to the same question after a teacher-‐ student interaction or in a student response system approach where the results are stored. When the response is given as a part of an assessment, Dickinson and Flick 1998 observed how grading might undermine the pedagogical goals of the instruction. James (2006) observed, working with student response systems, that when students responses were graded, this cause some unwanted effects on the students. James also observed that when the score for correct answers were given weak students started focusing on stronger student and their response rather than arguing for their own meaning. The score focus turned the students from a learning mode towards performance mode. So finding a way to use the second chance data is essential. Pure performance mode might be leading the group activity away from what you are trying to create in the class. (James 2006) An analysis of conversation transcripts revealed that conversation partners with a large disparity in student knowledge in the high stakes classroom focused most of their discussion around the dominant student’s answer preference. Such was not the case in the low stakes classroom where conversations were more balanced, including ideas put forth evenly from both partners. In future versions we have to focus on what to do with the score from a second chance data. It can be that we introduce a second set of questions, at the same level and let the second question count as a standard question. It might also be that the second chance score counts 50% or 20% of the assessment score. Another possibility is to let the collective response control the weight of the second chance data. If everybody answers correctly or participates, then the second chance data counts 100%. The system has no way to handle this approach now but it might be done in a later stage of the project. 35 Done-IT
15. Experiences with new Peer Learning
Assessment methods in Romania
15.1. Teaching in a new way
Due to the involvement in European projects UPM is currently using new pedagogical methods, the blended learning solutions, video solutions, mobile learning, etc. In the Done-‐IT project UPM has investigated new features for training, by using adequate equipment bought for this purpose. UPM extended the features of the traditional teaching, and investigated a new innovative dimension of teaching based up on using 2D or 3D multimedia material for illustration of defects within material testing, but also 3D reconstruction of defects, using non-‐destructive techniques. This has come as a consequence of the teaching innovation on which, during material testing courses delivered by UPM on which students organised in collaborative learning groups, solved practical tasks, specially designed by the teacher with the aid of various video, 2D and 3D resources. The practical sessions are structured in two parts: first part under instructor’s guidance and second part is self directed by students. In the last part of the practical session, selected groups have performed a peer teaching by demonstrating key concepts to the whole group with available material testing resources: pictures, videos, documents, web resources, samples with defects, but also 3D images with defects of materials and process that generating it. In the material testing courses and practical applications at UPM it is used: −
−
−
2D images with defects projected on a high resolution monitor, in order students to be able to have a good appreciation of the defect (nature, shape, characteristics); 3D images, where the defects can be illustrated from different orientations; non-‐destructive testing of materials with the standard x-‐ray analysis, by using a 3D reconstruction algorithm, regions of interest around the defects is possible to be reconstructed. 36 Done-IT
15.2 Evalauting in a new way
After the theoretical/practical presentations a peer learning assessment is performed by engaging large student groups in experiences of practical material testing skills in combination with appropriate real time feedback in order to improve their deep understanding and learning of the engineering aspects related to material testing. Most of the higher education institutions in Europe have a few computer laboratories equipped most of them with maximum 24 computers, on which is possible to run Learning Management Systems with digital multiple-‐choice tests/exams for students. Currently is used LMS Moodle. From another perspective, Smartphones with high resolution become more popular to the students, the access being conditioned by their price, which becomes cheaper. The characteristics of the various devices are so different that the issues related to delivering information and services on the web involve not only presentational aspects, but also structural and navigational aspects. It is estimated that mobile device could replace in the very near future the access to evaluation tests instead of computer interface, with the advantage that it is not necessary to have a lecture in a room equipped with computers. The traditional way of teaching face to face does not allow participants in the teaching process: teachers and students to verify their learning achievements during partial or final exams in a short period of time and the results are published after a few days, usually 10 days. Students are interested to be evaluated during semester in order to retract a part of the curriculum from the final exam. In this context UPM has participated as higher education institution in the Done-‐IT with the main deliverable Peer Learning Assessment System (PELE) for Smartphones. It is an assessment system, which is designed to enhance students learning. It is an advanced educational method based on Peer Learning processes, where students learn from their peers. The efficiency of such processes is difficult to measure by use of existing Information and Communication Technologies. Also a challenging problem is to consider using of assessment for learning as a training method, because students are unable to immediately verify their learning during tests/exams. Universities have access to computer science laboratories, but these cannot be used during evaluation period for all the campus students. On the other hand, within a few years ahead a lot of students will have access to cheap mobile phones with high resolution pressure sensitive screens. 37 Done-IT
Since 2010, UPM has used in higher education training delivery and evaluation of tests/exams the mobile devices like iPodTouch in quality assurance training processes. Usually, assessment is a process that measures student’s performance of learning by reproduction content. The assessment provides teacher information about: -
The successes of the teaching; -
The measure of learning the thought subject by the students; -
The degree of understanding the thought subject by the students. This information is collected by the teachers in order to improve instructional methods of teaching from one generation of students to another, for the benefit of the educational process in the school/university. The collected information, affects the subsequent classes, helping them to be where the teacher want to be at the end of the course. Usually in education, courses are finished with a final exam for assessment of knowledge, as displayed in figure 1. The dotted line indicates the skills obtained by the students. Such courses are usually terminated with a traditional (handwritten) exam. Fig. 1. Acquisition of knowledge in a traditional education course
Evaluation is a process through which instructors/teachers appreciate the quality of teaching by achievements of the students. Considering the moment of evaluation can be: formative evaluation by monitoring continuously students learning through ongoing feedback and summative evaluation, conducted at the end of the course. Usually, summative evaluations have a high contribution to the final degree, while formative assessments have low or no contribution to the final degree. Summative evaluation may consist in: final exam, project, essay, report. Results from summative evaluations may be used to improve teaching and learning efforts in the next generation of students. 38 Done-IT
Done-‐IT implements peer learning assessment solutions through verification or elaborative learning processes that utilize immediate feedback after tests and/or exams. The traditional exam (Fig. 1) is replaced with a number of tests (Fig. 2) that provide feedback from the student to the teacher and from the teacher to student. Fig. 2. Acquisition of knowledge in a course using several smaller tests in combination with PeLe The course may be completed with a smaller final exam since the tests contribute to the final score from the course. PeLe is a new evaluation model where test results for a class are turned into an active, creative and collaborative learning process by the use of immediate feedback: •
•
•
The verification feedback leaded by the teacher allows demonstrate the students why a particular answer is correct and why the others are incorrect; An elaborate feedback discussion run by students: the answers are displayed but they don’t know which are the (in)correct ones; An elaborative feedback discussion led by one student: the deviation from the correct answer without addressing why this is correct and the other ones are incorrect. The mobile student evaluation system for Smartphones gives the teachers a new tool to provide verification or elaborative feedback to students immediately after a test or exam. Students still remember questions after testing, and in this way they have the opportunity to learn the reason an answer is correct or not. This method is a collaborative supported learning that helps students to improve their knowledge in the study subject. Thus, mobile technology provides new evaluation and testing criteria for education and training. There are two arenas where one demands special attention regarding the design of the controller interface: 39 Done-IT
•
•
In classroom, Out of classroom. The in classroom arena is the most important since this is where one should start and the teacher should always be in control. Everything is simple and easy to use, just single click events from the teacher’s finger on the blackboard. The out of classroom comes later when the teacher wants to administrate several assessments and design course, model for the course and so on.The traditional reason to introduce electronic assessment is to save time for correcting answers. In the Done-‐IT project this aspect has been changed slightly by focusing on the learning perspective: ● establishing a connection between the assessment and learning; ● using an assessment as a part of a learning activity performed by the teacher in such a way that student learn from the assessments and the faults that are made. When an assessment has been performed, the students has been working on a problem for a time and given to the teacher a digitally delivery of the results. The teacher can see the results immediately and can use these results to create an arena for learning. In figure 3 is depicted the assessment arena: to the left, a traditional assessment, with the learning arena to the right where information from the assessment, information from the teacher or peer information can be used to work out an understanding of the problems assessed. Figure 3: Relation between assessment arena and learning arena
If the information from an assessment system to the teacher is correct, dependent upon the results, the teacher can organise teaching in many different ways. With the Done-‐IT assessment solution the target is to provide the teacher with instant feedback on the status of the assessment. Teacher analyses what questions did the students solve correct and what questions caused more problems. Than he can apply the following scenarios: 40 Done-IT
a) Continue as usual and only give the results; b) Give the students verifying feedback and explain what has been misunderstood; c) Give the students a hint of what might be the problem for this question, but not the actual solution; d) Give the students the results, “this is what you voted” and allow the students the possibility to discuss the problem; e) Or to find other ways. In case of c) and d) the students could be allowed to take part of the test again, or could be after a group / peer discussion be able to renegotiate their response. There are several possibilities that teacher is allowed to do: a) Pick one question, as set up in the assessment, and send it out prepared for an SRS session on the question; b) Collect several of the questions after a pedagogical treatment, and allow the students to get a second chance their answer to parts of the assessment. The main advantage of entering the test subject just after the test is the problem is fresh in mind since the student has just been working with it. The student might have spent time on parts of the assessment but have just not found or understand the right answer. Dependent upon the nature of the subject being thought, there might be just a hint from a peer student or a teacher that might solve the problem. The entire idea here is to make the assessment an arena for learning. There are many methodologically questions rising in such an arena, regarding: how the teachers will use the system; how the students will like to respond electronically; how the immediate feedback will change the view on assessment; the frequency and dimension of the assessments; the focus of the learning culture more on learning than on assessment; the relevance of continuous evaluation during a course in comparison with the final exam result; the category of students that benefit from the change in the assessment, etc. The key issue in this approach is to provide a set of tools that the teacher can use and feel free to apply different educational path in the learning activity. 15.3 New type of evaluation supported by
infrastructure UPM extended the features of the Peer Learning Assessment Services, and investigated a new innovative dimension of a test or an exam based up on using 2D or 3D multimedia material for illustration of defects within material testing, but also 3D reconstruction of defects, using non-‐
destructive techniques. 41 Done-IT
Questionars designed by the teacher are not only plain text, they require analysis of a real world situation related to materials defects, that is projected on the 2D or 3D monitors In the assessment phase picture of the material with the defects is projected by the teacher on a dedicated 2D high resolution monitor. Differences between some defects consist in minor details, that have to be evaluated carefully, on high 2D quality images that are projected on the dedicated 2D monitors. But, there are situations when students cannot appreciate the shape of the defect from the 2D view. In order to overcome this inconvenieint, in the assessment phase, on the 3D monitor are presented graphical pictures which illustrate defects in materials that can be watched from different orientations, in order students to be able to fully understand the shape of the defect. In a few seconds, rotating the object in different 3D positions they are able to understand the material defects. The material testing laboratory at UPM is equipped with x-‐ray equipment that can be used for non-‐destructive testing of materials. The 2D images from the equipment are used in a 3D reconstruction algorithm that reconstructs regions of interest around the defects inside materials. Then, the defects are evaluated on the 3D monitor on views from different orientations. 15.4 Conclusions
The first employment of PELE for mobile devices was done over a period of 4 weeks in June 2012 in engineering courses. In the introductory phase of PeLe teachers have read the user manual to understand how to use the system, and how to log at the system, than have projected the Online Instructional videos that demonstrate how to: a) Create teacher and student accounts on the PeLe assessment system (3 minutes long); b) Log on to the assessment services from students' mobile devices (4 minutes long); c) Use the PeLe (Peer Learning) Assessment Services for Smartphones, Pad`s, iPod, PC and Mac (the video contains 13 sections and is 11,5 minutes long). – These videos are available on the webpage of the project at http://www.histproject.no/node/197. These activity lead to a very good satisfaction for the introductory phase. Students’ feedbacks on the system were collected from a survey given at the end of the test period. A selection of the results obtained from 97 students appreciates that: 42 Done-IT
Q9) To what extent do you think the system, as it has been used, may integrate as a natural part of an exam?: Excellent (28); Very good (38); Satisfactory (22); Not very good (7); Bad (2); Q9) Integrate system in exam 40 20 Q9) Integrate system in exam 0 Q11) By using PeLe I had a chance to learn from the mistakes: I totally agree (37); I agree (45); Neutral (9); I slightly disagree (6); I totally disagree (0); Q11) Learn from the mistakes 50 40 30 20 10 0 Q11) Learn from the mistakes 43 Done-IT
Q12) By using PeLe I had a chance to learn from peers: I totally agree (33); I agree (48); Neutral (15); I slightly disagree (1); I totally disagree (0); Q12) Learn from peers 60 50 40 30 20 10 0 Q12) Learn from peers Q14) To what extent do you feel that the PeLe engages and activate you during evaluation?: Very large (31); Large (43); Neutral (14); Small (7); Poor (2); Q14) PeLe acgvate during evaluagon 50 40 30 20 10 0 Q14) PeLe ac„vate during evalua„on Very Large Neutral Small Poor large 44 Done-IT
Q15) To what extent do you think the PeLe can aid students learning of the course curriculum?: Very large (32); Large (41); Neutral (15); Small (9); Poor (0); Q15) PeLe aid students learn 50 40 30 20 10 0 Q15) PeLe aid students learn Very large Large Neutral Small Poor Q16) PeLe should be used in all classes/lectures: I totally agree (39); I agree (42); Neutral (11); I slightly disagree (5); I totally disagree (0). Q16) PeLe should be used in classes 60 40 20 Q16) PeLe should be used in classes 0 45 Done-IT
Q17) To what extent you consider the 2D and 3D ilustrations help understanding the theoretical problem/assessment question: I totally agree (54); I agree (42); Neutral (1); I slightly disagree (5); I totally disagree (0). Q17) 2D and 3D ilustragons help understanding 60 50 40 30 20 10 0 Q17) 2D and 3D ilustra„ons help understanding Totally I agree Neutral Slightly Totally agree agree disagree Q18) The 2D and 3D illustrations facilitate peer learning/assessment processes: I totally agree (59); I agree (38); Neutral (0); I slightly disagree (0); I totally disagree (0). Q18) 2D and 3D illustragons facilitate peer learning/assessment 70 60 50 40 30 20 10 0 Q18) 2D and 3D illustra„ons facilitate peer learning/assessment Totally I agree Neutral Slightly Totally agree agree disagree 46 Done-IT
As a general conclusion we appreciate that: -
-
the impression for using Peer Learning Assessment Software (PeLe) is very positive and students agree to use it as an advanced educational tool; the new model for course delivery and assessment, supported by the ICT equipment presenting materials characteristics and defects in 2D and 3D views, is largely accepted by students. The new innovative dimension of a test or an exam based up on using 2D or 3D multimedia material for illustration of defects within material testing, but also 3D reconstruction of defects, using non-‐destructive techniques, on which UPM extended the features of the Peer Learning Assessment Services, are largely accepted by students. 16. Literature references
Anderson, L.W. (2001), Krathwohl, D.R., Airasia, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J., Wittrock, M.C., (eds), 2001, A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman, New York. Nicol, David (2007). E-‐assessment by design: using multiple-‐choice tests to good effect. Journal of Further and Higher Education, Vol. 31. No. 1, Feb. 2007, pp 53-‐64 Nicol, D. J. & Macfarlane-‐Dick, D. (2006) Formative assessment and self-‐regulated learning: a model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 198–218. Boud, D. (2000) Sustainable assessment: rethinking assessment for the learning society, Studies in Continuing Education, 22(2), 151–167. Airasian, P. W. (1994) Classroom assessment (2nd edn) (New York, McGraw-‐Hill). Scouller, K. (1998) The influence of assessment method on students’ learning approaches: multiple choice question examination versus assignment essay, Higher Education, 35,453–
472. Yorke, M. (2003) Formative assessment in higher education: moves towards theory and the enhancement of pedagogic practice, Higher Education, 45(4), 477–501. Race, P. (2005). Making learning happen: a guide for post-‐compulsory education. London: Sage Publications Springer, L., Stanne, M. E. & Donovan, S. S. (1999). Effects of small-‐group learning on undergraduates in science, mathematics, engineering, and technology: a meta-‐analysis. Review of Educational Research 69 (1), 21-‐51. Barnes, D. & Todd, F. (1995). Communication and Learning Revisited. Portsmouth: Boynton/Cook Publishers. 47 Done-IT
Lemke, J. L. (1990). Talking Sience: Language, Learning and Values. Norwood, New Jersey: Ablex Publishing Corporation. Mortimer, E. F. & Scott, P. H. (2003). Meaning Making in Secondary Science Classrooms. Maidenhead: Open University Press. Cohen, E.G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64(1), 1-‐35. M.K. Smith, W.B.Wood, K. Krauter, and J.K. Knight, Combining Peer Discussion with Instructor Explanation Increases Student Learning from In-‐Class Concept Questions, CBE—Life Sciences Education Vol. 10, 55–63, 2011 Andreas Gegenfurtner, Motivation and transfer in professional training: A meta-‐analysis of the moderating effects of knowledge type, instruction, and assessment conditions, Educational Research Review 6 (2011) 153–168 Ericksen, S. (1984). The essence of good teaching. San Francisco: Jossey-‐Bass. James, Mark C., The effect of grading incentive on student discourse in Peer Instruction, Am. J. Phys. 74 (8), August 2006 Dickinson, V. L. and L. B. Flick, Beating the system: Course structure and student strategies in a traditional introductory undergraduate physics course for nonmajors, Sch. Sci. Math. 98(5), 238–246, 1998. 48

Done-IT - histproject.no

Transcription

Similar documents

Clarins Face Treatments Tratamentos para o Rosto Clarins

1-877-914-MOM2 You`re caring for your special needs

May 2009 Walnut Creek Clay Arts Guild

QUICK INSTALLATION GUIDE SPECIFICATIONS

Item 7 (14-233)

Rehabilitation Programs Division

sangliste firma - Music and More

March 2011 - Value Behavioral Health of Pennsylvania

Presentation by Mr. Gonzalo Gortázar, CFO of CaixaBank

Drug Enforcement Administration Ministry of Interior Tel: (+974) 44